Changes between Version 6 and Version 7 of CodeCoverage


Ignore:
Timestamp:
May 13, 2011, 8:47:27 PM (8 years ago)
Author:
gz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • CodeCoverage

    v6 v7  
    1 === Overview ===
    2 Code coverage provides information about which paths through generated code have been executed
    3 and which haven't.  For each source form, it can report one of three possible outcomes:
    4  
    5   - Not covered: This form was never entered.
    6  
    7   - Partly covered: This form was entered, and some parts were executed and some weren't
    8  
    9   - Fully covered: Every bit of code generated from this form was executed.
    10  
    11 === Limitations ===
    12  
    13 While the information gathered for coverage of generated code is complete and precise, the
    14 mapping back to source forms is of necessity heuristic, and depends a great deal on the
    15 behavior of macros and the path of the source forms through compiler transforms.  Source
    16 information is not recorded for variables, which further limits the source mapping.  In
    17 practice, there is often enough information scattered about a partially covered function
    18 to figure out which logical path through the code was taken and which wasn't.  If that
    19 doesn't work, you can try disassembling to see which parts of the compiled code were
    20 not executed: in the disassembled code there will be references to `#<CODE-NOTE [`xxx`] ...>`
    21 where xxx is NIL if the code that follows was never executed and non-NIL if it was.
     1Code Coverage documentation is now in the CCL manual: http://ccl.clozure.com/ccl-documentation.html#code-coverage
    222
    23 Sometimes the situation can be improved by modifying macros to try to preserve more of the
    24 input forms, rather than destructuring and rebuilding them.
    25 
    26 Because the code coverage information is associated with compiled functions, load-time
    27 toplevel expressions do not get reported on. You can work around this by creating a
    28 function and calling it.  I.e. instead of
    29 
    30 {{{
    31 (progn
    32   (do-this)
    33   (setq that ...) ...))
    34 }}}
    35 
    36 do:
    37 
    38 {{{
    39 (defun init-this-and-that ()
    40   (do-this)
    41   (setq that ...)  ...)
    42 (init-this-and-that)
    43 }}}
    44 
    45 Then you can see the coverage information in the definition of `init-this-and-that`.
    46 
    47 === Usage ===
    48 
    49 In order to gather code coverage information, you first have to recompile all your
    50 code to include code coverage instrumentation.  Compiling files will generate code coverage
    51 instrumentation if `CCL:*COMPILE-CODE-COVERAGE*` is true:
    52 
    53 {{{
    54 (setq ccl:*compile-code-coverage* t)
    55 (recompile-all-your-files)
    56 }}}
    57 
    58 The compilation process will be many times slower than normal, and the fasl files will be
    59 many times bigger.
    60 
    61 When you execute function loaded from instrumented fasl files, they will record coverage
    62 information every time they are executed.  The system keeps track of which instrumented
    63 files have been loaded.
    64 
    65 The following functions can be used to manage the coverage data:
    66 
    67 '''`CCL:REPORT-COVERAGE index-file &key (external-format :default) (statistics t) (html t)`'''
    68 
    69 If `:html` is non-nil, this will generate an HTML report, consisting of an index file and one html file for each
    70 instrumented source file that has been loaded in the current session.   The individual
    71 source file reports are stored in the same directory as the index file.  [[br]]
    72 The `:external-format` argument controls the external format of the html files. [[br]]
    73 If `:statistics`
    74 is non-nil, a comma-separated file is also generated with the summary of statistics. You can specify
    75 a filename for the statistics argument, otherwise `"statistics.csv"` is created in the output directory.
    76 See documentation of `ccl:coverage-statistics` below for a description of the values in the statistics file. [[br]]
    77 
    78 So for example
    79 if you've loaded `"foo.lx64fsl"` and `"bar.lx64fsl"` and have run some tests, you could do
    80 {{{
    81 (CCL:REPORT-COVERAGE "/my/dir/coverage/report.html")
    82 }}}
    83 and this would generate `"report.html"`, `"foo_lisp.html"` and `"bar_lisp.html"`, and `"statistics.csv"` all in `/my/dir/coverage/`.
    84 
    85 '''`CCL:RESET-COVERAGE`'''
    86 
    87 Resets all coverage data back to the 'Not executed' state.
    88 
    89 '''`CCL:CLEAR-COVERAGE`'''
    90 
    91 Gets rid of the information about which instrumented files have been loaded, so
    92 `ccl:report-coverage` will not report any files, and `ccl:save-coverage-in-file` will
    93 not save any info, until more instrumented files are loaded.
    94 
    95 '''`CCL:SAVE-COVERAGE-IN-FILE pathname`'''
    96 
    97 Saves all coverage info in a file, so you can restore the coverage state later.  This
    98 allows you to combine multiple runs or continue in a later session.  Equivalent to `(ccl:write-coverage-to-file (ccl:save-coverage) pathname)`.
    99 
    100 '''`CCL:RESTORE-COVERAGE-FROM-FILE pathname`'''
    101 
    102 Restores the coverage data previously saved with `CCL:SAVE-COVERAGE-IN-FILE`, for 
    103 the set of instrumented fasls that were loaded both at save and restore time.  I.e. coverage info is only restored for files that have been loaded in this session.  For example if in a previous session you had loaded `"foo.lx86fsl"` and then saved the coverage info, in this session you must load the same `"foo.lx86fsl"` before calling `ccl:restore-coverage-from-file` in order to retrieve the stored coverage info for `"foo"`.[[br]]
    104 Equivalent to `(ccl:restore-coverage (ccl:read-coverage-from-file pathname))`.
    105 
    106 '''`CCL:SAVE-COVERAGE`'''
    107 
    108 Returns a snapshot of the current coverage data.  A snapshot is a copy of the current coverage state.  It can be saved in a file with `ccl:write-coverage-to-file`, reinstated back as the current state with `ccl:restore-coverage`, or combined with other snapshots with `ccl:combine-coverage`.
    109 
    110 '''`CCL:RESTORE-COVERAGE snapshot`'''
    111 
    112 Reinstalls the coverage snapshot as the current coverage state.
    113 
    114 '''`CCL:COMBINE-COVERAGE snapshots`'''
    115 
    116 Takes a sequence of snapshots and returns a new snapshot that is the union of all of them.  I.e. in the combined snapshot an instruction is marked as covered if it is marked as covered in at least one of the input snapshots, otherwise it is marked as uncovered.
    117 
    118 '''`CCL:WRITE-COVERAGE-TO-FILE snapshot pathname`'''
    119 
    120 Saves the coverage snapshot in a file.  The snapshot can be loaded back with `ccl:read-coverage-from-file` or loaded and restored with `ccl:restore-coverage-from-file`.  Note that the file created is actually a lisp source file and can be compiled for faster loading.
    121 
    122 '''`CCL:READ-COVERAGE-FROM-FILE pathname`'''
    123 
    124 Returns the snapshot saved in `pathname`.  Doesn't affect the current coverage state.  `pathname` can be the file previously created with `ccl:write-coverage-to-file` or `ccl:save-coverage-in-file`, or it can be the name of the fasl created from compiling such a file.
    125 
    126 '''`CCL:COVERAGE-STATISTICS`'''
    127 
    128 Returns a sequence `ccl:coverage-statistics` objects, one for each source file, containing the same information as that written to the statistics file by `ccl:report-coverage`.  The following accessors are defined for `ccl:coverage-statistics` objects:
    129    * `ccl:coverage-source-file` - the name of the source file corresponding to this information
    130    * `ccl:coverage-expressions-total` - the total number of expressions
    131    * `ccl:coverage-expressions-entered` - the number of source expressions that have been entered (i.e. at least partially covered)
    132    * `ccl:coverage-expressions-covered` - the number of source expressions that were fully covered
    133    * `ccl:coverage-unreached-branches` - the number of conditionals with one branch taken and one not taken
    134    * `ccl:coverage-code-forms-total` - the total number of ''code forms''.  A code form is an expression in the final stage of compilation, after all macroexpansion and compiler transforms and simplification
    135    * `ccl:coverage-code-forms-covered` - the number of code forms that have been entered
    136    * `ccl:coverage-functions-total` - the total number of functions
    137    * `ccl:coverage-functions-fully-covered` - the number of functions that were fully covered
    138    * `ccl:coverage-functions-partly-covered` - the number of functions that were partly covered
    139    * `ccl:coverage-functions-not-entered` - the number of functions never entered
    140 
    141 
    142 '''`CCL:*COMPILE-CODE-COVERAGE*`'''
    143 
    144 This variable controls whether functions are instrumented for code coverage.  Files compiled
    145 while this variable is true will contain code coverage instrumentation.
    146 
    147 '''`CCL:WITHOUT-COMPILING-CODE-COVERAGE &body body`'''
    148 
    149 This macro arranges so that `body` doesn't record internal details of code coverage.  It will be considered totally covered if it's entered at all.  The Common Lisp macros `ASSERT` and `CHECK-TYPE` use this macro.