wiki:CodeCoverage

Version 6 (modified by gz, 5 years ago) (diff)

Now in the trunk

Overview

Code coverage provides information about which paths through generated code have been executed and which haven't. For each source form, it can report one of three possible outcomes:

  • Not covered: This form was never entered.

  • Partly covered: This form was entered, and some parts were executed and some weren't

  • Fully covered: Every bit of code generated from this form was executed.

Limitations

While the information gathered for coverage of generated code is complete and precise, the mapping back to source forms is of necessity heuristic, and depends a great deal on the behavior of macros and the path of the source forms through compiler transforms. Source information is not recorded for variables, which further limits the source mapping. In practice, there is often enough information scattered about a partially covered function to figure out which logical path through the code was taken and which wasn't. If that doesn't work, you can try disassembling to see which parts of the compiled code were not executed: in the disassembled code there will be references to #<CODE-NOTE [xxx] ...> where xxx is NIL if the code that follows was never executed and non-NIL if it was.

Sometimes the situation can be improved by modifying macros to try to preserve more of the input forms, rather than destructuring and rebuilding them.

Because the code coverage information is associated with compiled functions, load-time toplevel expressions do not get reported on. You can work around this by creating a function and calling it. I.e. instead of

(progn
  (do-this)
  (setq that ...) ...))

do:

(defun init-this-and-that ()
  (do-this)
  (setq that ...)  ...)
(init-this-and-that)

Then you can see the coverage information in the definition of init-this-and-that.

Usage

In order to gather code coverage information, you first have to recompile all your code to include code coverage instrumentation. Compiling files will generate code coverage instrumentation if CCL:*COMPILE-CODE-COVERAGE* is true:

(setq ccl:*compile-code-coverage* t) 
(recompile-all-your-files) 

The compilation process will be many times slower than normal, and the fasl files will be many times bigger.

When you execute function loaded from instrumented fasl files, they will record coverage information every time they are executed. The system keeps track of which instrumented files have been loaded.

The following functions can be used to manage the coverage data:

CCL:REPORT-COVERAGE index-file &key (external-format :default) (statistics t) (html t)

If :html is non-nil, this will generate an HTML report, consisting of an index file and one html file for each instrumented source file that has been loaded in the current session. The individual source file reports are stored in the same directory as the index file.
The :external-format argument controls the external format of the html files.
If :statistics is non-nil, a comma-separated file is also generated with the summary of statistics. You can specify a filename for the statistics argument, otherwise "statistics.csv" is created in the output directory. See documentation of ccl:coverage-statistics below for a description of the values in the statistics file.

So for example if you've loaded "foo.lx64fsl" and "bar.lx64fsl" and have run some tests, you could do

(CCL:REPORT-COVERAGE "/my/dir/coverage/report.html")

and this would generate "report.html", "foo_lisp.html" and "bar_lisp.html", and "statistics.csv" all in /my/dir/coverage/.

CCL:RESET-COVERAGE

Resets all coverage data back to the 'Not executed' state.

CCL:CLEAR-COVERAGE

Gets rid of the information about which instrumented files have been loaded, so ccl:report-coverage will not report any files, and ccl:save-coverage-in-file will not save any info, until more instrumented files are loaded.

CCL:SAVE-COVERAGE-IN-FILE pathname

Saves all coverage info in a file, so you can restore the coverage state later. This allows you to combine multiple runs or continue in a later session. Equivalent to (ccl:write-coverage-to-file (ccl:save-coverage) pathname).

CCL:RESTORE-COVERAGE-FROM-FILE pathname

Restores the coverage data previously saved with CCL:SAVE-COVERAGE-IN-FILE, for the set of instrumented fasls that were loaded both at save and restore time. I.e. coverage info is only restored for files that have been loaded in this session. For example if in a previous session you had loaded "foo.lx86fsl" and then saved the coverage info, in this session you must load the same "foo.lx86fsl" before calling ccl:restore-coverage-from-file in order to retrieve the stored coverage info for "foo".
Equivalent to (ccl:restore-coverage (ccl:read-coverage-from-file pathname)).

CCL:SAVE-COVERAGE

Returns a snapshot of the current coverage data. A snapshot is a copy of the current coverage state. It can be saved in a file with ccl:write-coverage-to-file, reinstated back as the current state with ccl:restore-coverage, or combined with other snapshots with ccl:combine-coverage.

CCL:RESTORE-COVERAGE snapshot

Reinstalls the coverage snapshot as the current coverage state.

CCL:COMBINE-COVERAGE snapshots

Takes a sequence of snapshots and returns a new snapshot that is the union of all of them. I.e. in the combined snapshot an instruction is marked as covered if it is marked as covered in at least one of the input snapshots, otherwise it is marked as uncovered.

CCL:WRITE-COVERAGE-TO-FILE snapshot pathname

Saves the coverage snapshot in a file. The snapshot can be loaded back with ccl:read-coverage-from-file or loaded and restored with ccl:restore-coverage-from-file. Note that the file created is actually a lisp source file and can be compiled for faster loading.

CCL:READ-COVERAGE-FROM-FILE pathname

Returns the snapshot saved in pathname. Doesn't affect the current coverage state. pathname can be the file previously created with ccl:write-coverage-to-file or ccl:save-coverage-in-file, or it can be the name of the fasl created from compiling such a file.

CCL:COVERAGE-STATISTICS

Returns a sequence ccl:coverage-statistics objects, one for each source file, containing the same information as that written to the statistics file by ccl:report-coverage. The following accessors are defined for ccl:coverage-statistics objects:

  • ccl:coverage-source-file - the name of the source file corresponding to this information
  • ccl:coverage-expressions-total - the total number of expressions
  • ccl:coverage-expressions-entered - the number of source expressions that have been entered (i.e. at least partially covered)
  • ccl:coverage-expressions-covered - the number of source expressions that were fully covered
  • ccl:coverage-unreached-branches - the number of conditionals with one branch taken and one not taken
  • ccl:coverage-code-forms-total - the total number of code forms. A code form is an expression in the final stage of compilation, after all macroexpansion and compiler transforms and simplification
  • ccl:coverage-code-forms-covered - the number of code forms that have been entered
  • ccl:coverage-functions-total - the total number of functions
  • ccl:coverage-functions-fully-covered - the number of functions that were fully covered
  • ccl:coverage-functions-partly-covered - the number of functions that were partly covered
  • ccl:coverage-functions-not-entered - the number of functions never entered

CCL:*COMPILE-CODE-COVERAGE*

This variable controls whether functions are instrumented for code coverage. Files compiled while this variable is true will contain code coverage instrumentation.

CCL:WITHOUT-COMPILING-CODE-COVERAGE &body body

This macro arranges so that body doesn't record internal details of code coverage. It will be considered totally covered if it's entered at all. The Common Lisp macros ASSERT and CHECK-TYPE use this macro.

Attachments