source: release/1.2/source/doc/src/build.xml @ 9200

Last change on this file since 9200 was 9200, checked in by gb, 12 years ago

synch with trunk

File size: 25.2 KB
1<?xml version="1.0" encoding="utf-8"?>
2<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" ""[
3          <!ENTITY rest "<varname>&amp;rest</varname>">
4          <!ENTITY key "<varname>&amp;key</varname>">
5          <!ENTITY optional "<varname>&amp;optional</varname>">
6          <!ENTITY body "<varname>&amp;body</varname>">
7          <!ENTITY aux "<varname>&amp;aux</varname>">
8          <!ENTITY allow-other-keys "<varname>&amp;allow-other-keys</varname>">
9          <!ENTITY CCL "Clozure CL">
10          ]>
11<chapter><title>Building &CCL; from its Source Code</title>
12  <anchor id="Building-CCL"/>
13  <para>&CCL;, like many other Lisp implementations, consists of a
14    kernel and a heap image.  The kernel is an ordinary C program, and
15    is built with a C compiler.  It provides very basic and
16    fundamental facilities, such as memory management, garbage
17    collection, and bootstrapping.  All the higher-level features are
18    written in Lisp, and compiled into the heap image.  Both parts are
19    needed to have a working Lisp implementation; neither the kernel
20    nor the heap image can stand alone.</para>
22  <para>You may already know that, when you have a C compiler which
23    is written in C, you need a working C compiler to build the
24    compiler. Similarly, the &CCL; heap image includes a Lisp
25    compiler, which is written in Lisp. You therefore need a working
26    Lisp compiler in order to build the Lisp heap image.</para>
28  <para>Where will you get a working Lisp compiler?  No worries; you
29    can use a precompiled copy of a (slightly older and compatible)
30    version of &CCL;. This section explains how to do all this.</para>
32  <para>In principle it should be possible to use another
33    implementation of Common Lisp as the host compiler, rather than an
34    older &CCL;; this would be a challenging and experimental way to
35    build, and is not described here.</para>
37  <sect1 id="building-definitions"><title>Building Definitions</title>
38    <para>The following terms are used in subsequent sections; it
39      may be helpful to refer to these definitions.</para>
41    <para><indexterm><primary>fasl
42          files</primary></indexterm><glossterm linkend="fasl-file">fasl
43        files</glossterm> are the object files produced
44      by <literal>compile-file</literal>.  fasl files store the
45      machine code associated with function definitions and the
46      external representation of other lisp objects in a compact,
47      machine-readable form. fasl is short for
48      &ldquo;<literal>FAS</literal>t
49      <literal>L</literal>oading&rdquo;. &CCL; uses different pathname
50      types (extensions) to name fasl files on different platforms;
51      see
52      <xref linkend="Platform-specific-filename-conventions"/> </para>
54    <para>The <indexterm><primary>lisp
55          kernel</primary></indexterm> <glossterm linkend="lisp_kernel">Lisp
56        kernel</glossterm> is a C program with a fair amount of
57      platform-specific assembly language code. Its basic job is to
58      map a lisp heap image into memory, transfer control to some
59      compiled lisp code that the image contains, handle any
60      exceptions that occur during the execution of that lisp code,
61      and provide various other forms of runtime support for that
62      code. &CCL; uses different filenames to name the lisp kernel
63      files on different platforms; see
64      <xref linkend="Platform-specific-filename-conventions"/>.</para>
66    <para>A <indexterm><primary>heap
67          image</primary></indexterm> <glossterm linkend="lisp_image">heap
68        image</glossterm> is a file that can be quickly mapped into a
69      process' address space. Conceptually, it's not too different
70      from an executable file or shared library in the OS's native
71      format (ELF orMach-O/dyld format); for historical reasons,
72      &CCL;'s own heap images are in their own (fairly simple)
73      format. The term <literal>full heap image</literal> refers to a
74      heap image file that contains all of the code and data that
75      comprise &CCL;. &CCL; uses different filenames to name the
76      standard full heap image files on different platforms; see
77      <xref linkend="Platform-specific-filename-conventions"/>.</para>
79    <para>A <indexterm><primary>bootstrapping
80          image</primary></indexterm> bootstrapping image is a minimal
81      heap image used in the process of building &CCL; itself.  The
82      bootstrapping image contains just enough code to load the rest
83      of &CCL; from fasl files.  It may help to think of the
84      bootstrapping image as the egg and the full heap image as the
85      chicken. &CCL; uses different filenames to name the standard
86      bootstrapping image files on different platforms; see
87      <xref linkend="Platform-specific-filename-conventions"/>
88      .</para>
90    <para>Each supported platform (and possibly a few
91      as-yet-unsupported ones) has a uniquely named subdirectory of
92      <literal>ccl/lisp-kernel/</literal>; each such
93      <indexterm><primary>kernel build directory</primary></indexterm>
94      contains a Makefile and may contain some auxiliary files (linker
95      scripts, etc.) that are used to build the lisp kernel on a
96      particular platform.The platform-specific name of the kernel
97      build directory is described in
98      <xref linkend="Platform-specific-filename-conventions"/>.</para>
100    <sect2 id="filename_conventions">
101      <title>Platform-specific filename conventions</title>
102      <table id ="Platform-specific-filename-conventions">
103            <title>Platform-specific filename conventions</title>
104            <tgroup cols="6">
105              <thead>
106            <row>
107              <entry>Platform</entry>
108              <entry>kernel</entry>
109              <entry>full-image</entry>
110              <entry>boot-image</entry>
111              <entry>fasl extension</entry>
112              <entry>kernel-build directory</entry>
113                </row>
114              </thead>
115              <tbody>
116                <row>
117                  <entry>DarwinPPC32</entry>
118              <entry>dppccl</entry>
119              <entry>dppccl.image</entry>
120              <entry>ppc-boot.image</entry>
121              <entry>.dfsl</entry>
122              <entry>darwinppc</entry>
123                </row>
124                <row>
125                  <entry>LinuxPPC32</entry>
126              <entry>ppccl</entry>
127                      <entry>PPCCL</entry>
128              <entry>ppc-boot</entry>
129              <entry>.pfsl</entry>
130              <entry>linuxppc</entry>
131                </row>
132                <row>
133                  <entry>DarwinPPC64</entry>
134                  <entry>dppccl64</entry>
135                  <entry>dppccl64.image</entry>
136                  <entry>ppc-boot64.image</entry>
137                  <entry>.d64fsl</entry>
138                  <entry>darwinppc64</entry>
139            </row>
140            <row>
141                      <entry>LinuxPPC64</entry>
142              <entry>ppccl64</entry>
143              <entry>PPCCL64</entry>
144              <entry>ppc-boot64</entry>
145              <entry>.p64fsl</entry>
146              <entry>linuxppc64</entry>
147            </row>
148                <row>
149                      <entry>LinuxX8664</entry>
150              <entry>lx86cl64</entry>
151              <entry>LX86CL64</entry>
152              <entry>x86-boot64</entry>
153              <entry>.lx64fsl</entry>
154              <entry>linuxx8664</entry>
155            </row>
156                <row>
157                      <entry>DarwinX8664</entry>
158                      <entry>dx86cl64</entry>
159              <entry>dx86cl64.image</entry>
160              <entry>x86-boot64.image</entry>
161              <entry>.dx64fsl</entry>
162              <entry>darwinx8664</entry>
163            </row>
164                <row>
165                      <entry>FreeBSDX8664</entry>
166              <entry>fx86cl64</entry>
167              <entry>FX86CL64</entry>
168              <entry>fx86-boot64</entry>
169              <entry>.fx64fsl</entry>
170              <entry>freebsdx8664</entry>
171            </row>
172              </tbody>
173            </tgroup>
174      </table>
175    </sect2>
176  </sect1>
178  <sect1 id="Setting-Up-to-Build">
179    <title>Setting Up to Build</title>
180    <para>There are currently three versions of &CCL; that you
181      might want to use (and therefore might want to build from
182      source):</para>
183    <itemizedlist>
184      <listitem><para>Version 1.0 - the more stable version</para></listitem>
185      <listitem><para>Version 1.1 - the more recent version, which
186          runs on more platforms (including x86-64 platforms) and
187          supports Unicode</para></listitem>
188          <listitem><para>Version 1.2 - supports (at least) all of the
189              features and platforms as 1.1, but is distributed and updated
190              differently</para></listitem>
191    </itemizedlist>
192    <para>All versions are available for download from the &CCL;
193      website in the form of archives that contain everything you need
194      to work with &CCL;, including the complete sources, a full
195      heap image, and the foreign-function interface database.</para>
196    <para>Version 1.0 archives are named
197      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-all-1.0.tar.gz</literal>,
198      where <replaceable>platform</replaceable> is either
199      <literal>darwinppc</literal>, <literal>darwinppc64</literal>, or
200      <literal>linuxppc</literal>.  Because version 1.0 is no longer
201      undergoing active development, you won't ever need to update
202      these sources.</para>
203    <para>Version 1.1 archives are named
204      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-snapshot-</literal><replaceable>yymmdd</replaceable><literal>.tar.gz</literal>,
205      where <replaceable>platform</replaceable> is either
206      <literal>darwinppc</literal>, <literal>darwinx8664</literal>,
207      <literal>linuxppc</literal>, <literal>linuxx8664</literal>, or
208      <literal>freebsdx8664</literal>, and where
209      <replaceable>yymmdd</replaceable> is the year, month, and day
210      the snapshot was released.</para>
211    <para>Because version 1.1 is undergoing active development,
212      there may be times when you want to get sources that are more
213      recent than the most recent snapshot and use them to build
214      yourself a new bleeding-edge &CCL;.  In that case, you should
215      download and install the latest snapshot, and then update your
216      sources via CVS.  At that point you can rebuild and you'll have
217      the latest and greatest &CCL;.  The snapshot has CVS
218      working-copy information in it, so all you need to do to update
219      is</para>
220    <programlisting>
221$ cd ccl
222$ cvs login             # password is "cvs"
223# this step only needs to be done once,
224# that'll store the trivially encrypted
225# password in ~/.cvspas
226$ cvs update
227    </programlisting>
228    <para>Unless you tell it to, cvs won't delete ("prune") empty
229      directories or create new ones when the repository changes.
230      It's generally a good habit to use</para>
231    <programlisting>
232$ cvs update -d -P      # create dirs as needed, prune empty ones
233    </programlisting>
234    <para>Version 1.2 archives follow naming conventions that are
235      similar to those used by 1.0 (though more platforms are supported.)
236      However, rather than containing CVS working-copy information, the
237      1.2 (and, presumably, later) archives contain metainformation used
238      by the Subversion (svn) source-code control system.</para>
239    <para>Subversion client programs are pre-installed on OSX 10.5 and
240      later and are typically either pre-installed or readily available
241      on Linux and FreeBSD platforms.  The <ulink url="">Subversion web page</ulink> contains links to subversion client programs
242      for many platforms; users of OSX versions 10.4 and earlier can also
243      install Subversion clients via Fink or MacPorts.</para>
245  </sect1>
247  <sect1 id="Building-Everything">
248    <title>Building Everything</title>
249    <para>Given that you now have everything you need, do the
250      following in a running &CCL; to bring your Lisp system
251      completely up to date.</para>
252    <programlisting>
253? (ccl:rebuild-ccl :full t)
254    </programlisting>
255    <para>That call to the function <literal>rebuild-ccl</literal>
256      performs the following steps:</para>
257    <itemizedlist>
258      <listitem>
259            <para>Deletes all fasl files and other object files in the
260              <literal>ccl</literal> directory tree</para>
261          </listitem>
262      <listitem>
263            <para>Runs an external process that does a
264              <literal>make</literal> in the current platform's kernel
265              build directory to create a new kernel</para>
266          </listitem>
267      <listitem>
268            <para>Does <literal>(compile-ccl t)</literal> in the running
269              lisp, to produce a set of fasl files from the &ldquo;higher
270              level&rdquo; lisp sources.</para>
271          </listitem>
272      <listitem>
273            <para>Does <literal>(xload-level-0 :force)</literal> in the
274              running lisp, to compile the lisp sources in the
275              &ldquo;ccl:level-0;&rdquo; directory into fasl files and
276              then create a bootstrapping image from those fasl
277              files.</para>
278          </listitem>
279      <listitem>
280            <para>Runs another external process, which causes the newly
281              compiled lisp kernel to load the new bootstrapping image.
282              The bootsrtrapping image then loads the &ldquo;higher
283              level&rdquo; fasl files and a new copy of the platform's
284              full heap image is then saved.</para>
285          </listitem>
286    </itemizedlist>
287    <para>If all goes well, it'll all happen without user
288      intervention and with some simple progress messages.  If
289      anything goes wrong during execution of either of the external
290      processes, the process output is displayed as part of a lisp
291      error message.</para>
292    <para><literal>rebuild-ccl</literal> is essentially just a short
293      cut for running all the individual steps involved in rebuilding
294      the system.  You can also execute these steps individually, as
295      described below.</para>
296  </sect1>
298  <sect1 id="Building-the-kernel">
299    <title>Building the kernel</title>
300    <para>The Lisp kernel is the executable that you run to use
301      Lisp.  It doesn't actually contain the entire Lisp
302      implementation; rather, it loads a heap image which contains the
303      specifics&mdash;the "library", as it might be called if this was a C
304      program.  The kernel also provides runtime support to the heap
305      image, such as garbage collection, memory allocation, exception
306      handling, and the OS interface.</para>
308    <para>The Lisp kernel file has different names on different
309      platforms. See
310      <xref linkend="Platform-specific-filename-conventions"/>. On all
311      platforms the lisp kernel sources reside
312      in <literal>ccl/lisp-kernel</literal>.</para>
314    <para>This section gives directions on how to rebuild the Lisp
315      kernel from its source code.  Most &CCL; users will rarely
316      have to do this.  You probably will only need to do it if you are
317      attempting to port &CCL; to a new architecture or extend or enhance
318      its kernel in some way.  As mentioned above, this step happens
319      automatically when you do
320      <programlisting>
321? (rebuild-ccl :full t)
322      </programlisting>
323    </para>
326    <sect2 id="Kernel-build-prerequisites">
327      <title>Kernel build prerequisites</title>
328          <para>The &CCL; kernel can be bult with the following widely
329            available tools:</para>
330      <itemizedlist>
331        <listitem><para>cc or gcc- the GNU C compiler</para></listitem>
332        <listitem><para>ld - the GNU linker</para></listitem>
333        <listitem><para>m4 or gm4- the GNU m4 macro processor</para></listitem>
334        <listitem><para>as - the GNU assembler (version 2.10.1 or later)</para></listitem>
335            <listitem><para>make - either GNU make or, on FreeBSD, the default BSD make program</para></listitem>
336          </itemizedlist>
337          <para> In general, the more recent the versions of those
338            tools, the better; some versions of gcc 3.x on Linux have
339            difficulty compiling some of the kernel source code correctly
340            (so gcc 4.0 should be used, if possible.)  On OSX, the
341            versions of the tools distributed with XCode should work fine;
342            on Linux, the versions of the tools installed with the OS (or
343            available through its package management system) should work
344            fine if they're "recent enough".  On FreeBSD, the installed
345            version of the <literal>m4</literal> program doesn't support
346            some features that the kernel build process depends on; the
347            GNU version of the m4 macroprocessor (called
348            <literal>gm4</literal> on FreeBSD) should be installed.
350          </para>
351    </sect2>
352    <sect2 id="kernel-build-command">
353          <title>Using "make" to build the lisp kernel</title>
354      <para>With those tools in place, do:
355        <programlisting>
356shell> cd ccl/lisp-kernel/<replaceable>PLATFORM</replaceable>
357shell> make
358            </programlisting>
359          </para>
360      <para>That'll assemble several assembly language source files,
361        compile several C source files, and link
362        ../../<replaceable>the kernel</replaceable>.
363          </para>
364    </sect2>
365  </sect1>
366  <sect1 id="Building-the-heap-image">
367    <title>Building the heap image</title>
368    <para>The initial heap image is loaded by the Lisp kernel, and
369      provides most of the language implementation The heap image
370      captures the entire state of a running Lisp (except for external
371      resources, such as open files and TCP sockets).  After it is
372      loaded, the contents of the new Lisp process's memory are
373      exactly the same as those of the old Lisp process when the image
374      was created.</para>
375    <para>The heap image is how we get around the fact that we can't
376      run Lisp code until we have a working Lisp implementation, and
377      we can't make our Lisp implementation work until we can run Lisp
378      code.  Since the heap image already contains a fully-working
379      implementation, all we need to do is load it into memory and
380      start using it.</para>
381    <para>If you're building a new version of &CCL;, you need to
382      build a new heap image.</para>
383    <para>(You might also wish to build a heap image if you have a
384      large program that is very complicated or time-consuming to
385      load, so that you will be able to load it once, save an image,
386      and thenceforth never have to load it again. At any time, a heap
387      image capturing the entire memory state of a running Lisp can be
388      created by calling the function
389      <literal>ccl:save-application</literal>.)</para>
391    <sect2 id="Development-cycle">
392          <title>Development cycle</title>
393      <para>Creating a new &CCL; full heap image consists of the
394        following steps:</para>
395      <orderedlist>
396        <listitem><para>Using your existing &CCL;, create a
397            bootstrapping image</para></listitem>
398        <listitem><para>Using your existing &CCL;, recompile your
399            updated &CCL; sources</para></listitem>
400        <listitem><para>Invoke &CCL; with the bootstrapping image
401            you just created (rather than with the existing full heap
402            image).</para></listitem>
403          </orderedlist>
404          <para>When you invoke &CCL; with the bootstrapping image, it
405            starts up, loads all of the &CCL; fasl files, and saves out a
406            new full heap image.  Voila.  You've created a new heap
407            image.</para>
408      <para>A few points worth noting:</para>
409          <itemizedlist>
410        <listitem>
411              <para>There's a circular dependency between the full heap
412                image and the bootstrapping image, in that each is used to
413                build the other.</para>
414            </listitem>
415        <listitem>
416              <para>There are some minor implementation
417                differences, but the environment in effect after the
418                bootstrapping image has loaded its fasl files is essentially
419                equivalent to the environment provided by the full heap
420                image; the latter loads a lot faster and is easier to
421                distribute, of course.</para>
422            </listitem>
423        <listitem>
424              <para>If the full heap image doesn't work (because
425                of an OS compatibilty problem or other bug), it's very likely
426                that the bootstrapping image will suffer the same
427                problems.</para>
428            </listitem>
429          </itemizedlist>
430      <para>Given a bootstrapping image and a set of up-to-date fasl
431        files, the development cycle usually involves editing lisp
432        sources (or updating those sources via cvs update),
433        recompiling modified files, and using the bootstrapping image
434        to produce a new heap image.</para>
435    </sect2>
437    <sect2 id="Generating-a-bootstrapping-image">
438      <title>Generating a bootstrapping image</title>
439      <para>The bootstrapping image isn't provided in &CCL;
440        distributions. It can be built from the source code provided
441        in distributions (using a lisp image and kernel provided in
442        those distributions) using the procedure described
443        below.</para>
445      <para>The bootstrapping image is built by invoking a special
446        utility inside a running &CCL; heap image to load files
447        contained in the <literal>ccl/level-0</literal> directory. The
448        bootstrapping image loads several dozen fasl files.  After
449        it's done so, it saves a heap image via
450        <literal>save-application</literal>. This process is called
451        "cross-dumping".</para>
453      <para>Given a source distribution, a lisp kernel, and a heap
454        image, one can produce a bootstrapping image by first invoking
455        &CCL; from the shell:</para>
456      <programlisting>
457shell&gt; openmcl
458Welcome to &CCL; .... !
460          </programlisting>
461          <para>then calling <literal>ccl:xload-level-0</literal> at the
462            lisp prompt</para>
463          <programlisting>
464? (ccl:xload-level-0)
465          </programlisting>
466      <para>This function compiles the lisp sources in the ccl/level-0
467        directory if they're newer than the corresponding fasl files
468        and then loads the resulting fasl files into a simulated lisp
469        heap contained in data structures inside the running
470        lisp. That simulated heap image is then written to
471        disk.</para>
472      <para><literal>xload-level-0</literal> should be called
473        whenever your existing boot image is out-of-date with respect
474        to the source files in <literal>ccl:level-0;</literal>
475        :</para>
476      <programlisting>
477? (ccl:xload-level-0 :force)
478      </programlisting>
479      <para>forces recompilation of the level-0 sources.</para>
480    </sect2>
482    <sect2 id="Generating-fasl-files">
483      <title>Generating fasl files</title>
484          <para> Calling:</para>
485      <programlisting>
486? (ccl:compile-ccl)
487          </programlisting>
488          <para>at the lisp prompt compiles any fasl files that are
489            out-of-date with respect to the corresponding lisp sources;
490            <literal>(ccl:compile-ccl t)</literal> forces
491            recompilation. <literal>ccl:compile-ccl</literal> reloads
492            newly-compiled versions of some files;
493            <literal>ccl:xcompile-ccl</literal> is analogous, but skips
494            this reloading step.</para>
495      <para>Unless there are bootstrapping considerations involved, it
496        usually doesn't matter whether these files are reloaded after
497        they're recompiled.</para>
498      <para>Calling <literal>compile-ccl</literal> or
499        <literal>xcompile-ccl</literal> in an environment where fasl
500        files don't yet exist may produce warnings to that effect
501        whenever files are <literal>require</literal>d during
502        compilation; those warnings can be safely ignored. Depending
503        on the maturity of the &CCL; release, calling
504        <literal>compile-ccl</literal> or
505        <literal>xcompile-ccl</literal> may also produce several
506        warnings about undefined functions, etc. They should be
507        cleaned up at some point.</para>
508    </sect2>
510    <sect2 id="Building-a-full-image-from-a-bootstrapping-image">
511          <title>Building a full image from a bootstrapping image</title>
512          <para>To build a full image from a bootstrapping image, just
513            invoke the kernel with the bootstrapping image as an
514            argument</para>
515      <programlisting>
516$ cd ccl                        # wherever your ccl directory is
518          </programlisting>
519      <para>Where <replaceable>KERNEL</replaceable> and
520        <replaceable>BOOT_IMAGE</replaceable> are the names of
521        the kernel and boot image appropriate to the platform you are
522        running on.  See <xref linkend="Platform-specific-filename-conventions"/></para>
523      <para>That should load a few dozen fasl files (printing a
524        message as each file is loaded.) If all of these files
525        successfully load, the lisp will print a prompt. You should be
526        able to do essentially everything in that environment that you
527        can in the environment provided by a "real" heap image. If
528        you're confident that things loaded OK, you can save that
529        image.</para>
530      <programlisting>
531? (ccl:save-application "<replaceable>image_name</replaceable>") ; Overwiting the existing heap image
532          </programlisting>
533          <para>Where <replaceable>image_name</replaceable> is the name of
534        the full heap image for your platform. See
535        <xref linkend="Platform-specific-filename-conventions"/>.</para>
536      <para>If things go wrong in the early stages of the loading
537        sequence, errors are often difficult to debug; until a fair
538        amount of code (CLOS, the CL condition system, streams, the
539        reader, the read-eval-print loop) is loaded, it's generally
540        not possible for the lisp to report an error.  Errors that
541        occur during these early stages ("the cold load") sometimes
542        cause the lisp kernel debugger (see ) to be invoked; it's
543        primitive, but can sometimes help one to get oriented.</para>
544    </sect2>
545  </sect1>
Note: See TracBrowser for help on using the repository browser.