Troubleshooting Coverage

Overview

Talking about coverage the most important question is: what coverage your are talking about? There are different kinds of coverage measurement and each tool has a slightly different way doing it and displaying the results. If you are looking for a very good introduction read Code Coverage Analysis on the Bullseye page.

SonarQube API provides the possibility to set the covered lines (setHits) and covered conditions (setConditions). All other metrics are derived from these settings:

• Line coverage: On a given line of code, Line coverage simply answers the following question: Has this line of code been executed during the execution of the unit tests?
• Conditions by line / Covered conditions by line: These are the number of conditions by line and number of covered conditions by line.
• Branch coverage: On each line of code containing some boolean expressions, the branch coverage simply answers the following question: 'Has each boolean expression been evaluated both to true and false?'. This is the density of possible branches in flow control structures that have been followed during unit tests execution.
• Coverage: It is a mix of Line coverage and Branch coverage. Its goal is to provide an even more accurate answer to the following question: How much of the source code has been covered by the unit tests?

Coverage = (CT + CF + LC)/(2*B + EL)

where
    CT = conditions that have been evaluated to 'true' at least once
    CF = conditions that have been evaluated to 'false' at least once
    LC = covered lines = linestocover - uncovered_lines
    B = total number of conditions
    EL = total number of executable lines (lines_to_cover)

For more details and exact calculation see Metric Definitions on the SonarQube page.

Hint: Absolute coverage numbers may vary slightly depending on the tool used. This is because the tools provide different coverage metrics and these must also be mapped differently to the SonarQube metrics.

Excluding source code from coverage metric

Parts to be excluded from the coverage metric can be configured via Narrow the Focus. In addition, the parameter sonar.coverage.exclusions is available. In the SonarQube UI this can be found under Administration > Configuration > Analysis Scope.

Merging coverage results

Coverage results from multiple tools, multiple programming languages, or multiple test runs can be combined into one overall result. In most cases, the coverage tools themselves already provide methods for merging reports. In newer SonarQube versions, the data can also be aggregated by SonarQube itself. The question is how SonarQube handles missing or conflicting data.

Missing coverge data

Coverage information must be available for all source files defined under sonar.sources. If coverage information is missing, "not covered" is used as default. Parts to be excluded from the coverage metric can be configured via Narrow the Focus. In addition, the parameter sonar.coverage.exclusions is available. In the SonarQube UI this can be found under Administration > Configuration > Analysis Scope.

Conflicting coverge data

When multiple sources provide divergent coverage data, they are aggregated. Since SonarQube 6.2 you can save several reports for the same file and reports will be merged using the following "additive" strategy:

• Reports should contain coverage data only for executable lines.
• The coverage data can be spread over several reports.
• Multiple sensors can be used to read coverage data.
• Source code and test code do not have to be in the same programming language.
• The data is merged with the existing coverage information in each case.
  • Line hits are cumulated.
  • The maximum value for the condition coverage is used.

     Examples:
        2/4 + 2/4 = 2/4
        2/4 + 3/4 = 3/4

Release versus Debug build

In release build, your code can be optimized aggressively. You can have strange behaviors when debugging a release build. Same kinds of issues occur with code coverage. If you need to use a release build, you may expect some inaccuracies.

Hint: Coverage Tools should always be be used with a debug build!