Changes between Version 6 and Version 7 of CodeCoverage

Show
Ignore:
Timestamp:
05/13/11 15:47:27 (2 years ago)
Author:
gz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • CodeCoverage

    v6 v7  
    1 === Overview === 
    2 Code coverage provides information about which paths through generated code have been executed  
    3 and which haven't.  For each source form, it can report one of three possible outcomes:  
    4   
    5   - Not covered: This form was never entered.  
    6   
    7   - Partly covered: This form was entered, and some parts were executed and some weren't  
    8   
    9   - Fully covered: Every bit of code generated from this form was executed.  
    10   
    11 === Limitations === 
    12   
    13 While the information gathered for coverage of generated code is complete and precise, the  
    14 mapping back to source forms is of necessity heuristic, and depends a great deal on the  
    15 behavior of macros and the path of the source forms through compiler transforms.  Source  
    16 information is not recorded for variables, which further limits the source mapping.  In  
    17 practice, there is often enough information scattered about a partially covered function  
    18 to figure out which logical path through the code was taken and which wasn't.  If that  
    19 doesn't work, you can try disassembling to see which parts of the compiled code were  
    20 not executed: in the disassembled code there will be references to `#<CODE-NOTE [`xxx`] ...>` 
    21 where xxx is NIL if the code that follows was never executed and non-NIL if it was. 
     1Code Coverage documentation is now in the CCL manual: http://ccl.clozure.com/ccl-documentation.html#code-coverage 
    222 
    23 Sometimes the situation can be improved by modifying macros to try to preserve more of the  
    24 input forms, rather than destructuring and rebuilding them.  
    25  
    26 Because the code coverage information is associated with compiled functions, load-time  
    27 toplevel expressions do not get reported on. You can work around this by creating a  
    28 function and calling it.  I.e. instead of  
    29  
    30 {{{ 
    31 (progn 
    32   (do-this) 
    33   (setq that ...) ...)) 
    34 }}} 
    35  
    36 do:  
    37  
    38 {{{ 
    39 (defun init-this-and-that () 
    40   (do-this) 
    41   (setq that ...)  ...) 
    42 (init-this-and-that) 
    43 }}} 
    44  
    45 Then you can see the coverage information in the definition of `init-this-and-that`.  
    46  
    47 === Usage === 
    48  
    49 In order to gather code coverage information, you first have to recompile all your  
    50 code to include code coverage instrumentation.  Compiling files will generate code coverage  
    51 instrumentation if `CCL:*COMPILE-CODE-COVERAGE*` is true:  
    52  
    53 {{{ 
    54 (setq ccl:*compile-code-coverage* t)  
    55 (recompile-all-your-files)  
    56 }}} 
    57  
    58 The compilation process will be many times slower than normal, and the fasl files will be  
    59 many times bigger.  
    60  
    61 When you execute function loaded from instrumented fasl files, they will record coverage  
    62 information every time they are executed.  The system keeps track of which instrumented  
    63 files have been loaded. 
    64  
    65 The following functions can be used to manage the coverage data:  
    66  
    67 '''`CCL:REPORT-COVERAGE index-file &key (external-format :default) (statistics t) (html t)`''' 
    68  
    69 If `:html` is non-nil, this will generate an HTML report, consisting of an index file and one html file for each  
    70 instrumented source file that has been loaded in the current session.   The individual  
    71 source file reports are stored in the same directory as the index file.  [[br]] 
    72 The `:external-format` argument controls the external format of the html files. [[br]] 
    73 If `:statistics` 
    74 is non-nil, a comma-separated file is also generated with the summary of statistics. You can specify 
    75 a filename for the statistics argument, otherwise `"statistics.csv"` is created in the output directory. 
    76 See documentation of `ccl:coverage-statistics` below for a description of the values in the statistics file. [[br]] 
    77  
    78 So for example  
    79 if you've loaded `"foo.lx64fsl"` and `"bar.lx64fsl"` and have run some tests, you could do  
    80 {{{ 
    81 (CCL:REPORT-COVERAGE "/my/dir/coverage/report.html") 
    82 }}} 
    83 and this would generate `"report.html"`, `"foo_lisp.html"` and `"bar_lisp.html"`, and `"statistics.csv"` all in `/my/dir/coverage/`. 
    84  
    85 '''`CCL:RESET-COVERAGE`''' 
    86  
    87 Resets all coverage data back to the 'Not executed' state.  
    88  
    89 '''`CCL:CLEAR-COVERAGE`''' 
    90  
    91 Gets rid of the information about which instrumented files have been loaded, so  
    92 `ccl:report-coverage` will not report any files, and `ccl:save-coverage-in-file` will  
    93 not save any info, until more instrumented files are loaded.  
    94  
    95 '''`CCL:SAVE-COVERAGE-IN-FILE pathname`''' 
    96  
    97 Saves all coverage info in a file, so you can restore the coverage state later.  This  
    98 allows you to combine multiple runs or continue in a later session.  Equivalent to `(ccl:write-coverage-to-file (ccl:save-coverage) pathname)`. 
    99  
    100 '''`CCL:RESTORE-COVERAGE-FROM-FILE pathname`''' 
    101  
    102 Restores the coverage data previously saved with `CCL:SAVE-COVERAGE-IN-FILE`, for   
    103 the set of instrumented fasls that were loaded both at save and restore time.  I.e. coverage info is only restored for files that have been loaded in this session.  For example if in a previous session you had loaded `"foo.lx86fsl"` and then saved the coverage info, in this session you must load the same `"foo.lx86fsl"` before calling `ccl:restore-coverage-from-file` in order to retrieve the stored coverage info for `"foo"`.[[br]] 
    104 Equivalent to `(ccl:restore-coverage (ccl:read-coverage-from-file pathname))`. 
    105  
    106 '''`CCL:SAVE-COVERAGE`''' 
    107  
    108 Returns a snapshot of the current coverage data.  A snapshot is a copy of the current coverage state.  It can be saved in a file with `ccl:write-coverage-to-file`, reinstated back as the current state with `ccl:restore-coverage`, or combined with other snapshots with `ccl:combine-coverage`. 
    109  
    110 '''`CCL:RESTORE-COVERAGE snapshot`''' 
    111  
    112 Reinstalls the coverage snapshot as the current coverage state. 
    113  
    114 '''`CCL:COMBINE-COVERAGE snapshots`''' 
    115  
    116 Takes a sequence of snapshots and returns a new snapshot that is the union of all of them.  I.e. in the combined snapshot an instruction is marked as covered if it is marked as covered in at least one of the input snapshots, otherwise it is marked as uncovered. 
    117  
    118 '''`CCL:WRITE-COVERAGE-TO-FILE snapshot pathname`''' 
    119  
    120 Saves the coverage snapshot in a file.  The snapshot can be loaded back with `ccl:read-coverage-from-file` or loaded and restored with `ccl:restore-coverage-from-file`.  Note that the file created is actually a lisp source file and can be compiled for faster loading. 
    121  
    122 '''`CCL:READ-COVERAGE-FROM-FILE pathname`''' 
    123  
    124 Returns the snapshot saved in `pathname`.  Doesn't affect the current coverage state.  `pathname` can be the file previously created with `ccl:write-coverage-to-file` or `ccl:save-coverage-in-file`, or it can be the name of the fasl created from compiling such a file. 
    125  
    126 '''`CCL:COVERAGE-STATISTICS`''' 
    127  
    128 Returns a sequence `ccl:coverage-statistics` objects, one for each source file, containing the same information as that written to the statistics file by `ccl:report-coverage`.  The following accessors are defined for `ccl:coverage-statistics` objects: 
    129    * `ccl:coverage-source-file` - the name of the source file corresponding to this information 
    130    * `ccl:coverage-expressions-total` - the total number of expressions 
    131    * `ccl:coverage-expressions-entered` - the number of source expressions that have been entered (i.e. at least partially covered) 
    132    * `ccl:coverage-expressions-covered` - the number of source expressions that were fully covered 
    133    * `ccl:coverage-unreached-branches` - the number of conditionals with one branch taken and one not taken 
    134    * `ccl:coverage-code-forms-total` - the total number of ''code forms''.  A code form is an expression in the final stage of compilation, after all macroexpansion and compiler transforms and simplification  
    135    * `ccl:coverage-code-forms-covered` - the number of code forms that have been entered 
    136    * `ccl:coverage-functions-total` - the total number of functions 
    137    * `ccl:coverage-functions-fully-covered` - the number of functions that were fully covered 
    138    * `ccl:coverage-functions-partly-covered` - the number of functions that were partly covered 
    139    * `ccl:coverage-functions-not-entered` - the number of functions never entered 
    140  
    141  
    142 '''`CCL:*COMPILE-CODE-COVERAGE*`''' 
    143  
    144 This variable controls whether functions are instrumented for code coverage.  Files compiled 
    145 while this variable is true will contain code coverage instrumentation. 
    146  
    147 '''`CCL:WITHOUT-COMPILING-CODE-COVERAGE &body body`''' 
    148  
    149 This macro arranges so that `body` doesn't record internal details of code coverage.  It will be considered totally covered if it's entered at all.  The Common Lisp macros `ASSERT` and `CHECK-TYPE` use this macro.