wiki:CodeCoverage

Version 2 (modified by gz, 7 years ago) (diff)

try again

The information below describes a feature currently available only in the working-0711 branch in subversion

Overview

Code coverage provides information about which paths through generated code have been executed and which haven't. For each source form, it can report one of three possible outcomes:

  • Not covered: This form was never entered.

  • Partly covered: This form was entered, and some parts were executed and some weren't

  • Fully covered: Every bit of code generated from this form was executed.

Limitations

While the information gathered for coverage of generated code is complete and precise, the mapping back to source forms is of necessity heuristic, and depends a great deal on the behavior of macros and the path of the source forms through compiler transforms. Source information is not recorded for variables, which further limits the source mapping. In practice, there is often enough information scattered about a partially covered function to figure out which logical path through the code was taken and which wasn't. If that doesn't work, you can try disassembling to see which parts of the compiled code were not executed.

Sometimes the situation can be improved by modifying macros to try to preserve more of the input forms, rather than destructuring and rebuilding them.

Coverage information for inlined functions is not collected. Currently there is no workaround short of removing the inline declarations.

Because the code coverage information is associated with compiled functions, load-time toplevel expressions do not get reported on. You can work around this by creating a function and calling it. I.e. instead of

(progn
  (do-this)
  (setq that ...) ...))

do:

(defun init-this-and-that ()
  (do-this)
  (setq that ...)  ...)
(init-this-and-that)

Then you can see the coverage information in the definition of init-this-and-that.

Usage

In order to gather code coverage information, you first have to recompile all your code to include code coverage instrumentation. Compiling files will generate code coverage instrumentation if CCL:*COMPILE-CODE-COVERAGE* is true:

(setq ccl:*compile-code-coverage* t) 
(recompile-all-your-files) 

The compilation process will be many times slower than normal, and the fasl files will be many times bigger.

When you execute function loaded from instrumented fasl files, they will record coverage information every time they are executed. The system keeps track of which instrumented files have been loaded.

The following functions can be used to manage the coverage data:

CCL:REPORT-COVERAGE index-file &key external-format statistics

This will generate an HTML report, consisting of an index file and one html file for each instrumented source file that has been loaded in the current session. The individual source file reports are stored in the same directory as the index file. If :statistics is non-nil, a comma-separated file is also generated with the summary of statistics. You can specify a filename for the statistics argument, otherwise "statistics.csv" is created in the output directory.

So for example if you've loaded "foo.lx64fsl" and "bar.lx64fsl" and have run some tests, you could do

(CCL:REPORT-COVERAGE "/my/dir/coverage/report.html" :statistics t)

and this would generate "report.html", "foo_lisp.html" and "bar_lisp.html", and "statistics.csv" all in /my/dir/coverage/.

CCL:RESET-COVERAGE

Resets all coverage data back to the 'Not executed' state.

CCL:CLEAR-COVERAGE

Gets rid of the information about which instrumented files have been loaded, so ccl:report-coverage will not report any files, and ccl:save-coverage-in-file will not save any info, until more instrumented files are loaded.

CCL:SAVE-COVERAGE-IN-FILE pathname

Saves all coverage info in a file, so you can restore the coverage state later. This allows you to combine multiple runs or continue in a later session.

CCL:RESTORE-COVERAGE-FROM-FILE pathname

Restores the coverage data previously saved with CCL:SAVE-COVERAGE-IN-FILE, for the set of instrumented fasls that were loaded both at save and restore time. I.e. coverage info is only restored for files that have been loaded in this session. I.e. if in a previous session you had loaded "foo.lx86fsl" and then saved the coverage info, in this session you must load the same "foo.lx86fsl" before calling ccl:restore-coverage-from-file in order to retrieve the stored coverage info for "foo".

CCL:*COMPILE-CODE-COVERAGE*

This variable controls whether functions are instrumented for code coverage. Files compiled while this variable is true will contain code coverage instrumentation.

Example Output

The output produced after loading CL-PPCRE and running (cl-ppcre-test:test) is  here.

The corresponding statistics file is here Download.

Attachments