Changeset 13106


Ignore:
Timestamp:
Oct 27, 2009, 11:12:34 AM (15 years ago)
Author:
R. Matthew Emerson
Message:

Document code coverage in manual.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/source/doc/src/using.xml

    r13104 r13106  
    23112311  </sect2>
    23122312</sect1>
     2313
     2314<sect1 id="code-coverage"><title>Code Coverage</title>
     2315<sect2 id="code-coverage-overview"><title>Overview</title>
     2316<para>
     2317  In Clozure CL 1.4 and later, code coverage provides information
     2318  about which paths through generated code have been executed and
     2319  which haven't. For each source form, it can report one of three
     2320  possible outcomes:
     2321</para>
     2322<itemizedlist>
     2323  <listitem>
     2324    <para>
     2325      Not covered: this form was never entered.
     2326    </para>
     2327  </listitem>
     2328  <listitem>
     2329    <para>
     2330      Partly covered: This form was entered, and some parts were
     2331      executed and some weren't.
     2332    </para>
     2333  </listitem>
     2334  <listitem>
     2335    <para>
     2336      Fully covered: Every bit of code generated from this form was
     2337      executed.
     2338    </para>
     2339  </listitem>
     2340</itemizedlist>
     2341</sect2>
     2342
     2343<sect2 id="code-coverage-limitations"><title>Limitations</title>
     2344<para>
     2345  While the information gathered for coverage of generated code is
     2346  complete and precise, the mapping back to source forms is of
     2347  necessity heuristic, and depends a great deal on the behavior of
     2348  macros and the path of the source forms through compiler
     2349  transforms. Source information is not recorded for variables, which
     2350  further limits the source mapping. In practice, there is often
     2351  enough information scattered about a partially covered function to
     2352  figure out which logical path through the code was taken and which
     2353  wasn't. If that doesn't work, you can try disassembling to see which
     2354  parts of the compiled code were not executed: in the disassembled
     2355  code there will be references to #&lt;CODE-NOTE [xxx] ...> where xxx
     2356  is NIL if the code that follows was never executed and non-NIL if it
     2357  was.
     2358</para>
     2359<para>
     2360  Sometimes the situation can be improved by modifying macros to try
     2361  to preserve more of the input forms, rather than destructuring and
     2362  rebuilding them.
     2363</para>
     2364<para>
     2365  Because the code coverage information is associated with compiled
     2366  functions, load-time toplevel expressions do not get reported
     2367  on. You can work around this by creating a function and calling
     2368  it. I.e. instead of
     2369  <programlisting>
     2370(progn
     2371  (do-this)
     2372  (setq that ...) ...))
     2373  </programlisting>
     2374  do:
     2375  <programlisting>
     2376(defun init-this-and-that ()
     2377  (do-this)
     2378  (setq that ...)  ...)
     2379(init-this-and-that)
     2380  </programlisting>
     2381Then you can see the coverage information in the definition of
     2382init-this-and-that.
     2383</para>
     2384</sect2>
     2385
     2386<sect2 id="code-coverage-usage"><title>Usage</title>
     2387<para>
     2388  In order to gather code coverage information, you first have to
     2389  recompile all your code to include code coverage
     2390  instrumentation. Compiling files will generate code coverage
     2391  instrumentation if <literal>CCL:*COMPILE-CODE-COVERAGE*</literal>
     2392  is true:
     2393  <programlisting>
     2394(setq ccl:*compile-code-coverage* t)
     2395(recompile-all-your-files)
     2396  </programlisting>
     2397</para>
     2398<para>
     2399  The compilation process will be many times slower than normal, and
     2400  the fasl files will be many times bigger.
     2401</para>
     2402<para>
     2403  When you execute function loaded from instrumented fasl files, they
     2404  will record coverage information every time they are executed. The
     2405  system keeps track of which instrumented files have been loaded.
     2406</para>
     2407<para>
     2408  The following functions can be used to manage the coverage data:
     2409</para>
     2410
     2411<refentry id="f_report-coverage">
     2412  <indexterm zone="f_report-coverage">
     2413    <primary>report-coverage</primary>
     2414  </indexterm>
     2415 
     2416  <refnamediv>
     2417    <refname>REPORT-COVERAGE</refname>
     2418    <refpurpose>Generate code coverage report</refpurpose>
     2419    <refclass>Function</refclass>
     2420  </refnamediv>
     2421
     2422  <refsynopsisdiv>
     2423    <synopsis><function>report-coverage</function> &key;
     2424    (external-format :default) (statistics t) (html t)
     2425    </synopsis>
     2426  </refsynopsisdiv>
     2427
     2428  <refsect1><title>Arguments and Values</title>
     2429  <variablelist>
     2430    <varlistentry>
     2431      <term>html</term>
     2432      <listitem>
     2433        <para>
     2434          If non-nil, this will generate an HTML report, consisting of
     2435          an index file and one html file for each instrumented source
     2436          file that has been loaded in the current session. The
     2437          individual source file reports are stored in the same
     2438          directory as the index file.
     2439        </para>
     2440      </listitem>
     2441    </varlistentry>
     2442    <varlistentry>
     2443      <term>external-format</term>
     2444      <listitem>
     2445        <para>
     2446          Controls the external format of the html files.
     2447        </para>
     2448      </listitem>
     2449    </varlistentry>
     2450    <varlistentry>
     2451      <term>statistics</term>
     2452      <listitem>
     2453        <para>
     2454          If :statistics is non-nil, a comma-separated file is also
     2455          generated with the summary of statistics. You can specify a
     2456          filename for the statistics argument, otherwise
     2457          "statistics.csv" is created in the output directory. See
     2458          documentation of ccl:coverage-statistics below for a
     2459          description of the values in the statistics file.
     2460        </para>
     2461      </listitem>
     2462    </varlistentry>
     2463  </variablelist>
     2464  </refsect1>
     2465  <refsect1><title>Example</title>
     2466    <para>
     2467      If you've loaded <filename>foo.lx64fsl</filename> and
     2468      <filename>bar.lx64fsl</filename>, and have run some tests, you could
     2469      do
     2470    <programlisting>
     2471(CCL:REPORT-COVERAGE "/my/dir/coverage/report.html")
     2472    </programlisting>
     2473    and this would generate <filename>report.html</filename>,
     2474    <filename>foo_lisp.html</filename> and
     2475    <filename>bar_lisp.html</filename>, and
     2476    <filename>statistics.csv</filename> all in
     2477    <filename>/my/dir/coverage/</filename>.
     2478    </para>
     2479  </refsect1>
     2480</refentry>
     2481
     2482<refentry id="f_reset-coverage">
     2483  <indexterm zone="f_reset-coverage">
     2484    <primary>reset-coverage</primary>
     2485  </indexterm>
     2486 
     2487  <refnamediv>
     2488    <refname>reset-coverage</refname>
     2489    <refpurpose>
     2490      Resets all coverage data back to the "Not Executed" state
     2491    </refpurpose>
     2492    <refclass>Function</refclass>
     2493  </refnamediv>
     2494
     2495  <refsect1><title>Summary</title>
     2496    <para>
     2497      Resets all coverage data back to the "Not Executed" state
     2498    </para>
     2499  </refsect1>
     2500</refentry>
     2501
     2502<refentry id="f_clear-coverage">
     2503  <indexterm zone="f_clear-coverage">
     2504    <primary>clear-coverage</primary>
     2505  </indexterm>
     2506 
     2507  <refnamediv>
     2508    <refname>clear-coverage</refname>
     2509    <refpurpose>
     2510      Forget about all instrumented files that have been loaded.
     2511    </refpurpose>
     2512    <refclass>Function</refclass>
     2513  </refnamediv>
     2514
     2515  <refsect1><title>Summary</title>
     2516    <para>
     2517      Gets rid of the information about which instrumented files have
     2518      been loaded, so ccl:report-coverage will not report any files,
     2519      and ccl:save-coverage-in-file will not save any info, until more
     2520      instrumented files are loaded.
     2521    </para>
     2522  </refsect1>
     2523</refentry>
     2524
     2525<refentry id="f_save-coverage-in-file">
     2526  <indexterm zone="f_save-coverage-in-file">
     2527    <primary>save-coverage-in-file</primary>
     2528  </indexterm>
     2529 
     2530  <refnamediv>
     2531    <refname>save-coverage-in-file</refname>
     2532    <refpurpose>
     2533      Save all coverage into to a file so you can restore it later.
     2534    </refpurpose>
     2535    <refclass>Function</refclass>
     2536  </refnamediv>
     2537
     2538  <refsynopsisdiv>
     2539    <synopsis><function>save-coverage-in-file</function> pathname
     2540    </synopsis>
     2541  </refsynopsisdiv>
     2542
     2543  <refsect1><title>Summary</title>
     2544    <para>
     2545      Saves all coverage info in a file, so you can restore the
     2546      coverage state later. This allows you to combine multiple runs
     2547      or continue in a later session. Equivalent to
     2548      (ccl:write-coverage-to-file (ccl:save-coverage) pathname).
     2549    </para>
     2550  </refsect1>
     2551</refentry>
     2552
     2553<refentry id="f_restore-coverage-from-file">
     2554  <indexterm zone="f_restore-coverage-from-file">
     2555    <primary>restore-coverage-from-file</primary>
     2556  </indexterm>
     2557 
     2558  <refnamediv>
     2559    <refname>restore-coverage-from-file</refname>
     2560    <refpurpose>
     2561      Load coverage state from a file.
     2562    </refpurpose>
     2563    <refclass>Function</refclass>
     2564  </refnamediv>
     2565
     2566  <refsynopsisdiv>
     2567    <synopsis><function>restore-coverage-from-file</function> pathname
     2568    </synopsis>
     2569  </refsynopsisdiv>
     2570
     2571  <refsect1><title>Summary</title>
     2572    <para>
     2573      Restores the coverage data previously saved with
     2574      CCL:SAVE-COVERAGE-IN-FILE, for the set of instrumented fasls
     2575      that were loaded both at save and restore time. I.e. coverage
     2576      info is only restored for files that have been loaded in this
     2577      session. For example if in a previous session you had loaded
     2578      "foo.lx86fsl" and then saved the coverage info, in this session
     2579      you must load the same "foo.lx86fsl" before calling
     2580      ccl:restore-coverage-from-file in order to retrieve the stored
     2581      coverage info for "foo".  Equivalent to (ccl:restore-coverage
     2582      (ccl:read-coverage-from-file pathname)).
     2583    </para>
     2584  </refsect1>
     2585</refentry>
     2586
     2587<refentry id="f_save-coverage">
     2588  <indexterm zone="f_save-coverage">
     2589    <primary>save-coverage</primary>
     2590  </indexterm>
     2591 
     2592  <refnamediv>
     2593    <refname>save-coverage</refname>
     2594    <refpurpose>
     2595      Returns a snapshot of the current coverage data.
     2596    </refpurpose>
     2597    <refclass>Function</refclass>
     2598  </refnamediv>
     2599
     2600  <refsect1><title>Summary</title>
     2601    <para>
     2602      Returns a snapshot of the current coverage data. A snapshot is a
     2603      copy of the current coverage state. It can be saved in a file
     2604      with ccl:write-coverage-to-file, reinstated back as the current
     2605      state with ccl:restore-coverage, or combined with other
     2606      snapshots with ccl:combine-coverage.
     2607    </para>
     2608  </refsect1>
     2609</refentry>
     2610
     2611<refentry id="f_restore-coverage">
     2612  <indexterm zone="f_restore-coverage">
     2613    <primary>restore-coverage</primary>
     2614  </indexterm>
     2615 
     2616  <refnamediv>
     2617    <refname>RESTORE-COVERAGE</refname>
     2618    <refpurpose>
     2619      Reinstalls a coverage snapshot as the current coverage state.
     2620    </refpurpose>
     2621    <refclass>Function</refclass>
     2622  </refnamediv>
     2623
     2624  <refsynopsisdiv>
     2625    <synopsis><function>restore-coverage</function> snapshot
     2626    </synopsis>
     2627  </refsynopsisdiv>
     2628
     2629  <refsect1><title>Summary</title>
     2630    <para>
     2631      Reinstalls a coverage snapshot as the current coverage state.
     2632    </para>
     2633  </refsect1>
     2634</refentry>
     2635
     2636<refentry id="f_write-coverage-to-file">
     2637  <indexterm zone="f_write-coverage-to-file">
     2638    <primary>write-coverage-to-file</primary>
     2639  </indexterm>
     2640 
     2641  <refnamediv>
     2642    <refname>WRITE-COVERAGE-TO-FILE</refname>
     2643    <refpurpose>
     2644      Save a coverage snapshot in a file.
     2645    </refpurpose>
     2646    <refclass>Function</refclass>
     2647  </refnamediv>
     2648
     2649  <refsynopsisdiv>
     2650    <synopsis><function>write-coverage-to-file</function> snapshot pathname
     2651    </synopsis>
     2652  </refsynopsisdiv>
     2653
     2654  <refsect1><title>Summary</title>
     2655    <para>
     2656      Saves the coverage snapshot in a file. The snapshot can be
     2657      loaded back with ccl:read-coverage-from-file or loaded and
     2658      restored with ccl:restore-coverage-from-file. Note that the file
     2659      created is actually a lisp source file and can be compiled for
     2660      faster loading.
     2661    </para>
     2662  </refsect1>
     2663</refentry>
     2664
     2665<refentry id="f_read-coverage-from-file">
     2666  <indexterm zone="f_read-coverage-from-file">
     2667    <primary>read-coverage-from-file</primary>
     2668  </indexterm>
     2669 
     2670  <refnamediv>
     2671    <refname>READ-COVERAGE-FROM-FILE</refname>
     2672    <refpurpose>
     2673      Return the coverage snapshot saved in a file.
     2674    </refpurpose>
     2675    <refclass>Function</refclass>
     2676  </refnamediv>
     2677
     2678  <refsynopsisdiv>
     2679    <synopsis><function>read-coverage-from-file</function> pathname
     2680    </synopsis>
     2681  </refsynopsisdiv>
     2682
     2683  <refsect1><title>Summary</title>
     2684    <para>
     2685      Returns the snapshot saved in pathname. Doesn't affect the
     2686      current coverage state. pathname can be the file previously
     2687      created with ccl:write-coverage-to-file or
     2688      ccl:save-coverage-in-file, or it can be the name of the fasl
     2689      created from compiling such a file.
     2690    </para>
     2691  </refsect1>
     2692</refentry>
     2693
     2694<refentry id="f_coverage-statistics">
     2695  <indexterm zone="f_coverage-statistics">
     2696    <primary>coverage-statistics</primary>
     2697  </indexterm>
     2698 
     2699  <refnamediv>
     2700    <refname>COVERAGE-STATISTICS</refname>
     2701    <refpurpose>
     2702      Returns a sequence of coverage-statistics objects, one per source file.
     2703    </refpurpose>
     2704    <refclass>Function</refclass>
     2705  </refnamediv>
     2706
     2707  <refsynopsisdiv>
     2708    <synopsis><function>coverage-statistics</function>
     2709    </synopsis>
     2710  </refsynopsisdiv>
     2711
     2712  <refsect1><title>Summary</title>
     2713    <para>
     2714      Returns a sequence ccl:coverage-statistics objects, one for each
     2715      source file, containing the same information as that written to
     2716      the statistics file by ccl:report-coverage. The following
     2717      accessors are defined for ccl:coverage-statistics objects:
     2718      <variablelist>
     2719      <varlistentry>
     2720        <term><function>ccl:coverage-source-file</function></term>
     2721        <listitem>
     2722          <para>
     2723            the name of the source file corresponding to this information
     2724          </para>
     2725        </listitem>
     2726      </varlistentry>
     2727      <varlistentry>
     2728        <term><function>ccl:coverage-expressions-total</function></term>
     2729        <listitem>
     2730          <para>
     2731            the total number of expressions
     2732          </para>
     2733        </listitem>
     2734      </varlistentry>
     2735      <varlistentry>
     2736        <term><function>ccl:coverage-expressions-entered</function></term>
     2737        <listitem>
     2738          <para>
     2739            the number of source expressions that have been entered
     2740            (i.e. at least partially covered)
     2741          </para>
     2742        </listitem>
     2743      </varlistentry>
     2744      <varlistentry>
     2745        <term><function>ccl:coverage-expressions-covered</function></term>
     2746        <listitem>
     2747          <para>
     2748            the number of source expressions that were fully covered
     2749          </para>
     2750        </listitem>
     2751      </varlistentry>
     2752      <varlistentry>
     2753        <term><function>ccl:coverage-unreached-branches</function></term>
     2754        <listitem>
     2755          <para>
     2756            the number of conditionals with one branch taken and one not taken
     2757          </para>
     2758        </listitem>
     2759      </varlistentry>
     2760      <varlistentry>
     2761        <term><function>ccl:coverage-code-forms-total</function></term>
     2762        <listitem>
     2763          <para>
     2764            the total number of code forms. A code form is an
     2765            expression in the final stage of compilation, after all
     2766            macroexpansion and compiler transforms and simplification
     2767          </para>
     2768        </listitem>
     2769      </varlistentry>
     2770      <varlistentry>
     2771        <term><function>ccl:coverage-code-forms-covered</function></term>
     2772        <listitem>
     2773          <para>
     2774            the number of code forms that have been entered
     2775          </para>
     2776        </listitem>
     2777      </varlistentry>
     2778      <varlistentry>
     2779        <term><function>ccl:coverage-functions-total</function></term>
     2780        <listitem>
     2781          <para>
     2782            the total number of functions
     2783          </para>
     2784        </listitem>
     2785      </varlistentry>
     2786      <varlistentry>
     2787        <term><function>ccl:coverage-functions-fully-covered</function></term>
     2788        <listitem>
     2789          <para>
     2790            the number of functions that were fully covered
     2791          </para>
     2792        </listitem>
     2793      </varlistentry>
     2794      <varlistentry>
     2795        <term><function>ccl:coverage-functions-partly-covered</function></term>
     2796        <listitem>
     2797          <para>
     2798            the number of functions that were partly covered
     2799          </para>
     2800        </listitem>
     2801      </varlistentry>
     2802      <varlistentry>
     2803        <term><function>ccl:coverage-functions-not-entered</function></term>
     2804        <listitem>
     2805          <para>
     2806            the number of functions never entered
     2807          </para>
     2808        </listitem>
     2809      </varlistentry>
     2810      </variablelist>
     2811    </para>
     2812  </refsect1>
     2813</refentry>
     2814
     2815<refentry id="v_compile-code-coverage">
     2816  <indexterm zone="v_compile-code-coverage">
     2817    <primary>*COMPILE-CODE-COVERAGE*</primary>
     2818  </indexterm>
     2819 
     2820  <refnamediv>
     2821    <refname>*COMPILE-CODE-COVERAGE*</refname>
     2822    <refpurpose>
     2823      When true, instrument functions for code coverage.
     2824    </refpurpose>
     2825    <refclass>Variable</refclass>
     2826  </refnamediv>
     2827
     2828  <refsynopsisdiv>
     2829    <synopsis><varname>*compile-code-coverage*</varname>
     2830    </synopsis>
     2831  </refsynopsisdiv>
     2832
     2833  <refsect1><title>Summary</title>
     2834    <para>
     2835      This variable controls whether functions are instrumented for
     2836      code coverage. Files compiled while this variable is true will
     2837      contain code coverage instrumentation.
     2838    </para>
     2839  </refsect1>
     2840</refentry>
     2841
     2842<refentry id="v_without-compiling-code-coverage">
     2843  <indexterm zone="v_without-compiling-code-coverage">
     2844    <primary>without-compiling-code-coverage</primary>
     2845  </indexterm>
     2846 
     2847  <refnamediv>
     2848    <refname>WITHOUT-COMPILING-CODE-COVERAGE</refname>
     2849    <refpurpose>
     2850      Don't record code coverange for forms within the body.
     2851    </refpurpose>
     2852    <refclass>Macro</refclass>
     2853  </refnamediv>
     2854
     2855  <refsynopsisdiv>
     2856    <synopsis><function>without-compiling-code-coverage</function>
     2857    </synopsis>
     2858  </refsynopsisdiv>
     2859
     2860  <refsect1><title>Summary</title>
     2861    <para>
     2862      This macro arranges so that body doesn't record internal details
     2863      of code coverage. It will be considered totally covered if it's
     2864      entered at all. The Common Lisp macros ASSERT and CHECK-TYPE use
     2865      this macro.
     2866    </para>
     2867  </refsect1>
     2868</refentry>
     2869
     2870</sect2>
     2871</sect1>
    23132872</chapter>
Note: See TracChangeset for help on using the changeset viewer.