Blog

Posted April 21, 2023

Code Coverage vs. Test Coverage: What’s Better for Software Development?

Learn more about the similarities and differences between code coverage and test coverage, and how they help you create better software.

All software developers strive to ensure their code will perform as expected and is free of bugs. In doing so, the terms code coverage and test coverage will appear frequently.

While related, code coverage refers to the percentage of code executed during testing, whereas test coverage refers to the extent to which the tests cover the software's requirements or functionality.

Code coverage tells you what areas of code have and have not been executed.
Test coverage tells you what risks have been examined, from the user's point of view.

Both are essential because they measure the quality and completeness of the testing process. Both are also incomplete, in and of themselves, because they only tell a partial story.

In this article, you'll learn more about the similarities and differences between code coverage and test coverage. You'll look specifically at how they're measured, what their purposes are, as well as what their advantages and limitations are. By the end of the article, we hope you'll understand how both can help you create better software.

What is Code Coverage?

As previously stated, code coverage is a measure of how much of a program's source code has been executed during testing, and it can be used to guide the development of test cases to ensure that all parts of the code are tested.

To find an error in your code, you need to execute it and observe the results. This is especially important when considering the internal structure of the code, or white-box testing, where you look at how the code is structured and how it will impact the output.

Measuring Code Coverage

It's common practice to express code coverage as a percentage of the total number of lines of code, statements, or functions executed during testing. You can measure this in several different ways—statement, branch, condition, and function coverage.

Statement coverage

Statement coverage, also referred to as line coverage, is a measurement used during testing to determine how many lines of code within a program are executed during testing.

Here's the general formula for calculating statement coverage:

Statement coverage = (Number of executed statements / Total number of statements) * 100%

For example, consider the following Python program that translates color names to hex codes:

1
color_dict = {
2
    "aliceblue": "#f0f8ff",
3
    "antiquewhite": "#faebd7",
4
    "antiquewhite1": "#ffefdb",
5
    # More colors here
6
}
7

8
 def translate_color(color):
9
    if color in color_dict:
10
        return color_dict[color]
11
    elif color == "":
12
        return "No color entered"
13
    else:
14
        return "Color not found"

And here's a test case for the previous program:

1
from app.codecov import translate_color
2

3
 def test_translate_color():
4
    assert translate_color("aliceblue") == "#f0f8ff"
5
    assert translate_color("antiquewhite") == "#faebd7"
6
    assert translate_color("antiquewhite1") == "#ffefdb"

In this example, the program has seven lines or statements; however, the test case only covers the condition that first checks if color exists in a dictionary called color_dict. If it does, it returns the value of the color key in the dictionary.

You can retrieve your statement coverage results using the pytest-cov coverage tool, which shows you only have 57 percent statement coverage:

In this case, you have seven lines of code; three of which have yet to be tested (lines 17–20).

To obtain 100 percent statement coverage, you need to write a test case that exercises each of the seven lines in this program's source code, as shown here:

1
def test_translate_color():
2
    assert translate_color("aliceblue") == "#f0f8ff"
3
    assert translate_color("antiquewhite") == "#faebd7"
4
    assert translate_color("antiquewhite1") == "#ffefdb"
5
    assert translate_color("") == "No color entered"
6
    assert translate_color("not a color") == "Color not found"

After rerunning the coverage analysis with the modified test case, you should observe the following:

Branch coverage

A branch is a point in the code that allows the program flow to be directed to one of two or more different paths. Branch coverage is a metric that indicates how fully a testing process has covered the various branches present in a software's source code. It's often stated as a percentage, with 100 percent branch coverage suggesting that every potential branch in the code has been executed at least once during testing.

An example of this would be an if-else statement that contains two branches: TRUE and FALSE. The formula for calculating branch coverage is as follows:

Branch coverage = (Number of branches executed / Total number of branches) * 100%

To better understand branch coverage, consider the previous example again, which included four branches:

1
def translate_color(color):
2
    if color == "":
3
        return "No color entered"
4
    elif color in color_dict:
5
        return color_dict[color]
6
    else:
7
        return "Color not found"

In the program, there are two decision points: the first is when the program checks whether the color is provided or not. When no color is specified, or the input is an empty string, this evaluates to true, and the program terminates. However, when a color is supplied, this condition is false, and the program execution continues.

The second decision point occurs when a color is checked; if it exists, the color code is printed; otherwise, the program prints "Color not found" and stops, resulting in a false evaluation.

If you run the program to check for branch coverage, you will get the following result: Branch coverage = (4 / 4) * 100% = 100%. You achieve 100 percent branch coverage because the test case traverses every branch in the code. This means that every conceivable branch in your code has been executed by the tests at least once and has, thus, been somewhat validated.

Here's a diagram of the branching and branches in this example:

Notably, obtaining 100 percent branch coverage also entails 100 percent statement coverage, as every statement would be part of a branch. However, the converse is not necessarily true; reaching 100 percent statement coverage does not necessarily imply that all branches have been covered.

For instance, here is an example of a Python code that demonstrates 100 percent statement coverage but doesn't always imply that all branches have been covered:

1
def divide(a, b):
2
    if b == 0:
3
        raise ValueError("Cannot divide by zero")
4
    return a / b

And here's an example of a pytest test that achieves 100 percent statement coverage but does not cover all branches:

1
def test_divide():
2
    assert divide(10, 2) == 5
3
    with pytest.raises(ValueError):
4
        divide(10, 0)

In this example, the test test_divide covers all statements in the divide function, but it does not test the case where b is a positive, non-zero value. As a result, even though 100 percent statement coverage is achieved, the branches of the if statement are not fully covered.

All the example codes for this tutorial can be found on this GitHub repo.

Condition coverage

Condition coverage, also referred to as expression coverage, is a code coverage metric measuring that each condition must evaluate to true and false at least once. A condition is a Boolean expression that evaluates to true or false and is frequently used in control flow expressions, such as if statements, for loops, and while loops.

The formula for calculating condition coverage is as follows:

Condition coverage = (Number of conditions tested / Total number of conditions) * 100%

The program in the previous example had only two conditions that branched into four paths. The last case test ensured that all the branches and conditions were exercised, so in this case, the condition coverage would look like this: Condition coverage = (2/2) * 100% = 100%.

For further understanding, here's an illustration of the conditions present in the example program:

To obtain 100 percent condition coverage, you must ensure that your tests exercise all available true and false pairings for each condition in your code.

Function coverage

Function coverage is a code coverage metric used to assess how test cases exercise the functions in the program when tested. It responds to the question, "How many functions in my code have been called at least once?"

The resulting ratio is then reported as a percentage, indicating the proportion of functions that have been tested. The formula for calculating function coverage is as follows:

Function coverage = (Number of called functions / Total number of functions) * 100%

If a function is not invoked during testing, it's probably not being exercised correctly and may have flaws.

Now that you understand code coverage and its assessment, it's time to discuss why it's crucial in software development.

The Purpose of Code Coverage

There are several reasons why code coverage is an essential tool for ensuring the quality and reliability of code, including the following:

You can identify code that isn't being used or tested by unit tests.
You can assess the test suite's quality and identify parts of the code that require extra testing.
You can identify portions of code that may contain bugs and assist in determining which areas of code should be examined first. Untested code is a potential bug hideout because the code has not been tested and validated.
You can recognize dead code, which is no longer needed in the program and can be safely removed.

Advantages of Code Coverage

Code coverage is a valuable measure for assessing the quality and completeness of tests. Other advantages of code coverage include the following:

Increased software reliability as a result of extensive testing.
Shorter development cycles because developers can focus on testing critical portions of the code.
Easier debugging because code coverage can assist in detecting bug-infested portions of code.
Better code quality since code coverage identifies portions of code that may contain errors.
Increased confidence in the codebase, as code coverage identifies untested code sections.

Limitations of Code Coverage

Code coverage is a helpful metric that developers can use to evaluate their code's quality. However, it can have downsides that must be considered while utilizing it:

Code coverage may not always reflect code quality. Just because a codebase has a high code coverage does not imply that the code is of excellent quality.
The absence of flaws is not guaranteed by code coverage. A test suite may have excellent code coverage but still contain errors or fail to exercise all the code's functionality.
Code coverage can be deceptive. A test suite can have excellent code coverage while containing poorly written or inefficient tests.
Code coverage only measures the execution of code. It doesn't consider other factors, such as the quality of the tests themselves or the system's overall design.
Different code coverage tools may be needed for various programming languages, which can be inconvenient and increase the complexity of implementing code coverage.
Code coverage can be time-consuming and costly to implement.

Code coverage fundamentally implies that you reduce the likelihood of having faults by increasing the number of possible code executions.

Now that you know all about code coverage, compare it to test coverage.

What is Test Coverage?

Test coverage is an essential factor in software development and testing. It quantifies how much of the features being tested are covered by the test, which can be a good indicator of risk. This is different from code coverage, as code coverage measures the percentage of code executed during testing. Achieving adequate test coverage is critical for assuring software quality and reducing potential defects.

Test coverage is also about perspective: code coverage is about the internal workings of the software and where to point future efforts. Test coverage is from the user's point of view.

Why Test Coverage?

The goal of measuring test coverage is to ensure that all critical business requirements are tested and that there are no significant gaps in testing. This includes determining whether all relevant scenarios and edge cases have been considered, as well as whether the tests are sufficiently robust to detect potential issues. This assists in identifying any gaps in testing and areas for software enhancement.

How to Measure Test Coverage

Unlike code coverage, test coverage is a qualitative metric that indicates how fully the business requirements have been tested. It's a subjective evaluation of how well the tests cover the business requirements (and risks) rather than a quantitative measure that can be expressed as a percentage or number.

Some test coverage metrics are as follows:

Specifications coverage ensures that all the software's requirements have been tested and met.
Product coverage ensures that the entire product, including all its features and components, has been tested.
Risk coverage identifies potential software risks and ensures that adequate tests have been performed to mitigate these risks. This helps to reduce the potential impact of any issues that may arise while using the software.
Functional coverage ensures that all the software's functional requirements have been tested and adequately covered.
Test execution coverage provides valuable information about the test results and helps to ensure that the software has been thoroughly tested before it's released to the public.

Advantages of Test Coverage

There are several advantages to using test coverage, including the following:

It helps to enhance software quality by offering an in-depth assessment of the software's operability.
It reduces the risk of introducing new defects or weaknesses in the code while ensuring that current faults are discovered and resolved.
As a black-box testing method, test coverage is simpler to implement because it doesn't necessitate in-depth technical knowledge of the code under test.
It assists in identifying portions of a product that have yet to be tested and can ensure that all product elements have been validated.
It saves time and money by lowering the need for costly bug fixes since the likelihood of bugs getting through is reduced.
It contributes to product confidence by ensuring that all aspects of the product have been evaluated.
Any changes made to the code can be tested quickly, allowing for a faster release cycle.

Limitations of Test Coverage

As with code coverage, test coverage has some limitations, including the following:

Limited scope: The capacity of test coverage to detect flaws, such as security vulnerabilities or performance concerns, is limited.
False sense of security: While test coverage can create the image of a high-quality codebase, it does not guarantee that the code is bug-free.
Test coverage is only as adequate as the quality of the tests: Incorrect or incomplete tests may not identify errors.

Without adequate test coverage, you might as well be driving blindfolded down a winding mountain road—you might get there, but it's not worth the risk. Don't skimp on testing!

Key Differences Between Code Coverage and Test Coverage

There are several key differences between code coverage and test coverage. For instance, code coverage is a quantitative measurement, expressed as a percentage, focusing on the number of codes tested. In contrast, test coverage is a qualitative measurement that focuses on the quality and completeness of the testing process, and it is not expressed numerically.

Another critical distinction is that code coverage is concerned with the lines of code executed and signifies the completeness of testing in terms of code execution. In comparison, test coverage is concerned with the entire scope of testing, which includes not only code execution but also the quality and effectiveness of the tests. It evaluates the tests' depth and breadth to ensure that all possible scenarios and test cases are covered.

Code coverage is a white-box testing approach, whereas test coverage examines the program's external behavior, which is a black-box testing methodology.

And finally, test coverage is usually done in integration, system, or acceptance tests, whereas code coverage is generally done in unit tests.

Should You Choose Code Coverage or Test Coverage?

Choosing between code coverage and test coverage is dependent on your specific situation, but both are essential in ensuring that a software system has been thoroughly tested and that any issues have been identified and corrected.

In general, code coverage and test coverage should be used as part of a comprehensive testing strategy.

Code coverage is usually preferred when determining the extent to which a software application's source code has been tested. For instance, during the development stage, code coverage is the preferred choice when you want to ensure that all lines of code have been tested and are functioning as expected. Code coverage is also the better option if you need to verify that a specific portion of the code is run at least once during testing or when you want to prioritize testing to areas of code that have not attained the needed coverage threshold.

In contrast, test coverage is more suitable when assessing a test's effectiveness. For example, when testing for performance, test coverage is more effective in determining the effectiveness of the code. Moreover, in cases where a quick evaluation of the overall quality of the code is required, test coverage allows for faster problem resolution.

It's necessary to consider the specific requirements of a software system to determine which type of coverage is required and when. Depending on the project, you might need both code coverage and test coverage or only one.

Final Thoughts

Both code and test coverage are critical metrics for determining software correctness. Code coverage verifies and validates code quality by evaluating the number of codes executed while running automated tests. It ensures that all parts of the code have been tested and that there are no defects or bugs present.

In comparison, test coverage is a measure of the overall quality of the testing process. Both metrics are important, and deciding which to use depends on your specific scenario.