One way of measuring the effectiveness of automated tests, is by measuring how many lines of code are touched while the tests are being executed. This is called test coverage. The idea is that the more lines are being executed by your tests, the more of your code’s behaviour is being exercised, thus yielding a greater probability of bugs surfacing and being detected or prevented.

Note, however, that line test coverage only measures whether a line of code is executed. This does not mean that the result or side-effect of that line’s execution is being assessed for correctness.

Additionally, one line may cause two different pathways through your application, e.g., an if-statement. When testing one such path, line test coverage will show that the if-statement was covered, yet it does not always show that only one of the possible paths through your application has been exercised. This can especially occur in complex one-line operations. For this use-case, there is also the concept of branch coverage, though mllint currently does not assess this though.

Furthermore, for testing ML systems, there is also academic discussion as to whether line coverage or branch coverage makes sense, or whether different forms of coverage are required. While mllint currently does not check or support any of these novel forms of test coverage for ML, we are looking for suggestions on what novel forms of ML code coverage should be assessed and how these can be measured.

While mllint will not run your tests as part of its static analysis, mllint expects you to run these on your own terms and provide a the filenames to a JUnit-compatible XML test report and a Cobertura-compatible XML coverage report in your project’s mllint configuration. Specifically for this rule, the Cobertura-compatible coverage report is analysed.

Generating a test coverage report with pytest can be done by adding and installing pytest-cov as a development dependency of your project. Then use the following command to run your tests and generate both a test report as well as a coverage report:

pytest --junitxml=tests-report.xml --cov=path_to_package_under_test --cov-report=xml

You can then configure mllint to pick up your test report as follows:

testing:
  coverage:
    report: coverage.xml
    targets:
      line: 80 # percent line coverage. Default is 80%

or equivalent TOML:

[tool.mllint.testing.coverage]
report = "coverage.xml"
targets = { line = 80.0 } 

# Note: unlike YAML, TOML distinguishes between floats and integers, so be sure to use 80.0 instead of 80