Index: /trunk/source/doc/src/using.xml =================================================================== --- /trunk/source/doc/src/using.xml (revision 13105) +++ /trunk/source/doc/src/using.xml (revision 13106) @@ -2311,3 +2311,562 @@ + +Code Coverage +Overview + + In Clozure CL 1.4 and later, code coverage provides information + about which paths through generated code have been executed and + which haven't. For each source form, it can report one of three + possible outcomes: + + + + + Not covered: this form was never entered. + + + + + Partly covered: This form was entered, and some parts were + executed and some weren't. + + + + + Fully covered: Every bit of code generated from this form was + executed. + + + + + +Limitations + + While the information gathered for coverage of generated code is + complete and precise, the mapping back to source forms is of + necessity heuristic, and depends a great deal on the behavior of + macros and the path of the source forms through compiler + transforms. Source information is not recorded for variables, which + further limits the source mapping. In practice, there is often + enough information scattered about a partially covered function to + figure out which logical path through the code was taken and which + wasn't. If that doesn't work, you can try disassembling to see which + parts of the compiled code were not executed: in the disassembled + code there will be references to #<CODE-NOTE [xxx] ...> where xxx + is NIL if the code that follows was never executed and non-NIL if it + was. + + + Sometimes the situation can be improved by modifying macros to try + to preserve more of the input forms, rather than destructuring and + rebuilding them. + + + Because the code coverage information is associated with compiled + functions, load-time toplevel expressions do not get reported + on. You can work around this by creating a function and calling + it. I.e. instead of + +(progn + (do-this) + (setq that ...) ...)) + + do: + +(defun init-this-and-that () + (do-this) + (setq that ...) ...) +(init-this-and-that) + +Then you can see the coverage information in the definition of +init-this-and-that. + + + +Usage + + In order to gather code coverage information, you first have to + recompile all your code to include code coverage + instrumentation. Compiling files will generate code coverage + instrumentation if CCL:*COMPILE-CODE-COVERAGE* + is true: + +(setq ccl:*compile-code-coverage* t) +(recompile-all-your-files) + + + + The compilation process will be many times slower than normal, and + the fasl files will be many times bigger. + + + When you execute function loaded from instrumented fasl files, they + will record coverage information every time they are executed. The + system keeps track of which instrumented files have been loaded. + + + The following functions can be used to manage the coverage data: + + + + + report-coverage + + + + REPORT-COVERAGE + Generate code coverage report + Function + + + + report-coverage &key; + (external-format :default) (statistics t) (html t) + + + + Arguments and Values + + + html + + + If non-nil, this will generate an HTML report, consisting of + an index file and one html file for each instrumented source + file that has been loaded in the current session. The + individual source file reports are stored in the same + directory as the index file. + + + + + external-format + + + Controls the external format of the html files. + + + + + statistics + + + If :statistics is non-nil, a comma-separated file is also + generated with the summary of statistics. You can specify a + filename for the statistics argument, otherwise + "statistics.csv" is created in the output directory. See + documentation of ccl:coverage-statistics below for a + description of the values in the statistics file. + + + + + + Example + + If you've loaded foo.lx64fsl and + bar.lx64fsl, and have run some tests, you could + do + +(CCL:REPORT-COVERAGE "/my/dir/coverage/report.html") + + and this would generate report.html, + foo_lisp.html and + bar_lisp.html, and + statistics.csv all in + /my/dir/coverage/. + + + + + + + reset-coverage + + + + reset-coverage + + Resets all coverage data back to the "Not Executed" state + + Function + + + Summary + + Resets all coverage data back to the "Not Executed" state + + + + + + + clear-coverage + + + + clear-coverage + + Forget about all instrumented files that have been loaded. + + Function + + + Summary + + Gets rid of the information about which instrumented files have + been loaded, so ccl:report-coverage will not report any files, + and ccl:save-coverage-in-file will not save any info, until more + instrumented files are loaded. + + + + + + + save-coverage-in-file + + + + save-coverage-in-file + + Save all coverage into to a file so you can restore it later. + + Function + + + + save-coverage-in-file pathname + + + + Summary + + Saves all coverage info in a file, so you can restore the + coverage state later. This allows you to combine multiple runs + or continue in a later session. Equivalent to + (ccl:write-coverage-to-file (ccl:save-coverage) pathname). + + + + + + + restore-coverage-from-file + + + + restore-coverage-from-file + + Load coverage state from a file. + + Function + + + + restore-coverage-from-file pathname + + + + Summary + + Restores the coverage data previously saved with + CCL:SAVE-COVERAGE-IN-FILE, for the set of instrumented fasls + that were loaded both at save and restore time. I.e. coverage + info is only restored for files that have been loaded in this + session. For example if in a previous session you had loaded + "foo.lx86fsl" and then saved the coverage info, in this session + you must load the same "foo.lx86fsl" before calling + ccl:restore-coverage-from-file in order to retrieve the stored + coverage info for "foo". Equivalent to (ccl:restore-coverage + (ccl:read-coverage-from-file pathname)). + + + + + + + save-coverage + + + + save-coverage + + Returns a snapshot of the current coverage data. + + Function + + + Summary + + Returns a snapshot of the current coverage data. A snapshot is a + copy of the current coverage state. It can be saved in a file + with ccl:write-coverage-to-file, reinstated back as the current + state with ccl:restore-coverage, or combined with other + snapshots with ccl:combine-coverage. + + + + + + + restore-coverage + + + + RESTORE-COVERAGE + + Reinstalls a coverage snapshot as the current coverage state. + + Function + + + + restore-coverage snapshot + + + + Summary + + Reinstalls a coverage snapshot as the current coverage state. + + + + + + + write-coverage-to-file + + + + WRITE-COVERAGE-TO-FILE + + Save a coverage snapshot in a file. + + Function + + + + write-coverage-to-file snapshot pathname + + + + Summary + + Saves the coverage snapshot in a file. The snapshot can be + loaded back with ccl:read-coverage-from-file or loaded and + restored with ccl:restore-coverage-from-file. Note that the file + created is actually a lisp source file and can be compiled for + faster loading. + + + + + + + read-coverage-from-file + + + + READ-COVERAGE-FROM-FILE + + Return the coverage snapshot saved in a file. + + Function + + + + read-coverage-from-file pathname + + + + Summary + + Returns the snapshot saved in pathname. Doesn't affect the + current coverage state. pathname can be the file previously + created with ccl:write-coverage-to-file or + ccl:save-coverage-in-file, or it can be the name of the fasl + created from compiling such a file. + + + + + + + coverage-statistics + + + + COVERAGE-STATISTICS + + Returns a sequence of coverage-statistics objects, one per source file. + + Function + + + + coverage-statistics + + + + Summary + + Returns a sequence ccl:coverage-statistics objects, one for each + source file, containing the same information as that written to + the statistics file by ccl:report-coverage. The following + accessors are defined for ccl:coverage-statistics objects: + + + ccl:coverage-source-file + + + the name of the source file corresponding to this information + + + + + ccl:coverage-expressions-total + + + the total number of expressions + + + + + ccl:coverage-expressions-entered + + + the number of source expressions that have been entered + (i.e. at least partially covered) + + + + + ccl:coverage-expressions-covered + + + the number of source expressions that were fully covered + + + + + ccl:coverage-unreached-branches + + + the number of conditionals with one branch taken and one not taken + + + + + ccl:coverage-code-forms-total + + + the total number of code forms. A code form is an + expression in the final stage of compilation, after all + macroexpansion and compiler transforms and simplification + + + + + ccl:coverage-code-forms-covered + + + the number of code forms that have been entered + + + + + ccl:coverage-functions-total + + + the total number of functions + + + + + ccl:coverage-functions-fully-covered + + + the number of functions that were fully covered + + + + + ccl:coverage-functions-partly-covered + + + the number of functions that were partly covered + + + + + ccl:coverage-functions-not-entered + + + the number of functions never entered + + + + + + + + + + + *COMPILE-CODE-COVERAGE* + + + + *COMPILE-CODE-COVERAGE* + + When true, instrument functions for code coverage. + + Variable + + + + *compile-code-coverage* + + + + Summary + + This variable controls whether functions are instrumented for + code coverage. Files compiled while this variable is true will + contain code coverage instrumentation. + + + + + + + without-compiling-code-coverage + + + + WITHOUT-COMPILING-CODE-COVERAGE + + Don't record code coverange for forms within the body. + + Macro + + + + without-compiling-code-coverage + + + + Summary + + This macro arranges so that body doesn't record internal details + of code coverage. It will be considered totally covered if it's + entered at all. The Common Lisp macros ASSERT and CHECK-TYPE use + this macro. + + + + + +