source: trunk/source/doc/src/build.xml @ 9071

Last change on this file since 9071 was 9071, checked in by mikel, 12 years ago

mechanical and style edits in install and build chapters

File size: 24.9 KB
1<?xml version="1.0" encoding="utf-8"?>
2<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" ""[
3<!ENTITY rest "<varname>&amp;rest</varname>">
4<!ENTITY key "<varname>&amp;key</varname>">
5<!ENTITY optional "<varname>&amp;optional</varname>">
6<!ENTITY body "<varname>&amp;body</varname>">
7<!ENTITY aux "<varname>&amp;aux</varname>">
8<!ENTITY allow-other-keys "<varname>&amp;allow-other-keys</varname>">
9<!ENTITY CCL "Clozure CL">
11  <chapter><title>Building &CCL; from its Source Code</title>
12    <anchor id="Building-CCL"/>
13    <para>&CCL;, like many other Lisp implementations, consists of a
14    kernel and a heap image.  The kernel is an ordinary C program, and
15    is built with a C compiler.  It provides very basic and
16    fundamental facilities, such as memory management, garbage
17    collection, and bootstrapping.  All the higher-level features are
18    written in Lisp, and compiled into the heap image.  Both parts are
19    needed to have a working Lisp implementation; neither the kernel
20    nor the heap image can stand alone.</para>
22    <para>You may already know that, when you have a C compiler which
23    is written in C, you need a working C compiler to build the
24    compiler. Similarly, the &CCL; heap image includes a Lisp
25    compiler, which is written in Lisp. You therefore need a working
26    Lisp compiler in order to build the Lisp heap image.</para>
28    <para>Where will you get a working Lisp compiler?  No worries; you
29    can use a precompiled copy of a (slightly older and compatible)
30    version of &CCL;. This section explains how to do all this.</para>
32    <para>In principle it should be possible to use another
33    implementation of Common Lisp as the host compiler, rather than an
34    older &CCL;; this would be a challenging and experimental way to
35    build, and is not described here.</para>
37    <sect1 id="building-definitions"><title>Building Definitions</title>
38      <para>The following terms are used in subsequent sections; it
39      may be helpful to refer to these definitions.</para>
41      <para><indexterm><primary>fasl
42      files</primary></indexterm><glossterm linkend="fasl-file">fasl
43      files</glossterm> are the object files produced
44      by <literal>compile-file</literal>.  fasl files store the
45      machine code associated with function definitions and the
46      external representation of other lisp objects in a compact,
47      machine-readable form. fasl is short for
48      &ldquo;<literal>FAS</literal>t
49      <literal>L</literal>oading&rdquo;. &CCL; uses different pathname
50      types (extensions) to name fasl files on different platforms;
51      see
52      <xref linkend="Platform-specific-filename-conventions"/> </para>
54      <para>The <indexterm><primary>lisp
55      kernel</primary></indexterm> <glossterm linkend="lisp_kernel">Lisp
56      kernel</glossterm> is a C program with a fair amount of
57      platform-specific assembly language code. Its basic job is to
58      map a lisp heap image into memory, transfer control to some
59      compiled lisp code that the image contains, handle any
60      exceptions that occur during the execution of that lisp code,
61      and provide various other forms of runtime support for that
62      code. &CCL; uses different filenames to name the lisp kernel
63      files on different platforms; see
64      <xref linkend="Platform-specific-filename-conventions"/>.</para>
66      <para>A <indexterm><primary>heap
67      image</primary></indexterm> <glossterm linkend="lisp_image">heap
68      image</glossterm> is a file that can be quickly mapped into a
69      process' address space. Conceptually, it's not too different
70      from an executable file or shared library in the OS's native
71      format (ELF orMach-O/dyld format); for historical reasons,
72      &CCL;'s own heap images are in their own (fairly simple)
73      format. The term <literal>full heap image</literal> refers to a
74      heap image file that contains all of the code and data that
75      comprise &CCL;. &CCL; uses different filenames to name the
76      standard full heap image files on different platforms; see
77      <xref linkend="Platform-specific-filename-conventions"/>.</para>
79      <para>A <indexterm><primary>bootstrapping
80      image</primary></indexterm> bootstrapping image is a minimal
81      heap image used in the process of building &CCL; itself.  The
82      bootstrapping image contains just enough code to load the rest
83      of &CCL; from fasl files.  It may help to think of the
84      bootstrapping image as the egg and the full heap image as the
85      chicken. &CCL; uses different filenames to name the standard
86      bootstrapping image files on different platforms; see
87      <xref linkend="Platform-specific-filename-conventions"/>
88      .</para>
90      <para>Each supported platform (and possibly a few
91      as-yet-unsupported ones) has a uniquely named subdirectory of
92      <literal>ccl/lisp-kernel/</literal>; each such
93      <indexterm><primary>kernel build directory</primary></indexterm>
94      contains a Makefile and may contain some auxiliary files (linker
95      scripts, etc.) that are used to build the lisp kernel on a
96      particular platform.The platform-specific name of the kernel
97      build directory is described in
98      <xref linkend="Platform-specific-filename-conventions"/>.</para>
100      <sect2 id="filename_conventions">
101       <title>Platform-specific filename conventions</title>
102       <table id ="Platform-specific-filename-conventions">
103         <title>Platform-specific filename conventions</title>
104         <tgroup cols="6">
105           <thead>
106            <row>
107                <entry>Platform</entry>
108                <entry>kernel</entry>
109                <entry>full-image</entry>
110                <entry>boot-image</entry>
111                <entry>fasl extension</entry>
112                <entry>kernel-build directory</entry>
113            </row>
114           </thead>
115           <tbody>
116             <row>
117               <entry>DarwinPPC32</entry>
118                <entry>dppccl</entry>
119                <entry>dppccl.image</entry>
120                <entry>ppc-boot.image</entry>
121                <entry>.dfsl</entry>
122                <entry>darwinppc</entry>
123             </row>
124             <row>
125               <entry>LinuxPPC32</entry>
126                <entry>ppccl</entry>
127                <entry>PPCCL</entry>
128                <entry>ppc-boot</entry>
129                <entry>.pfsl</entry>
130                <entry>linuxppc</entry>
131             </row>
132             <row>
133               <entry>DarwinPPC64</entry>
134               <entry>dppccl64</entry>
135               <entry>dppccl64.image</entry>
136               <entry>ppc-boot64.image</entry>
137               <entry>.d64fsl</entry>
138               <entry>darwinppc64</entry>
139             </row>
140              <row>
141                <entry>LinuxPPC64</entry>
142                <entry>ppccl64</entry>
143                <entry>PPCCL64</entry>
144                <entry>ppc-boot64</entry>
145                <entry>.p64fsl</entry>
146                <entry>linuxppc64</entry>
147              </row>
148              <row>
149                <entry>LinuxX8664</entry>
150                <entry>lx86cl64</entry>
151                <entry>LX86CL64</entry>
152                <entry>x86-boot64</entry>
153                <entry>.lx64fsl</entry>
154                <entry>linuxx8664</entry>
155              </row>
156              <row>
157                <entry>DarwinX8664</entry>
158                <entry>dx86cl64</entry>
159                <entry>dx86cl64.image</entry>
160                <entry>x86-boot64.image</entry>
161                <entry>.dx64fsl</entry>
162                <entry>darwinx8664</entry>
163              </row>
164              <row>
165                <entry>FreeBSDX8664</entry>
166                <entry>fx86cl64</entry>
167                <entry>FX86CL64</entry>
168                <entry>fx86-boot64</entry>
169                <entry>.fx64fsl</entry>
170                <entry>freebsdx8664</entry>
171              </row>
172           </tbody>
173         </tgroup>
174       </table>
175      </sect2>
176    </sect1>
178    <sect1 id="Setting-Up-to-Build">
179      <title>Setting Up to Build</title>
180      <para>There are currently three versions of &CCL; that you
181      might want to use (and therefore might want to build from
182      source):</para>
183      <itemizedlist>
184        <listitem><para>Version 1.0 - the more stable version</para></listitem>
185        <listitem><para>Version 1.1 - the more recent version, which
186        runs on more platforms (including x86-64 platforms) and
187        supports Unicode</para></listitem>
188        <listitem><para>Version 1.2 - supports (at least) all of the
189        features and platforms as 1.1, but is distributed and updated
190        differently</para></listitem>
191      </itemizedlist>
192      <para>All versions are available for download from the &CCL;
193      website in the form of archives that contain everything you need
194      to work with &CCL;, including the complete sources, a full
195      heap image, and the foreign-function interface database.</para>
196      <para>Version 1.0 archives are named
197      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-all-1.0.tar.gz</literal>,
198      where <replaceable>platform</replaceable> is either
199      <literal>darwinppc</literal>, <literal>darwinppc64</literal>, or
200      <literal>linuxppc</literal>.  Because version 1.0 is no longer
201      undergoing active development, you won't ever need to update
202      these sources.</para>
203      <para>Version 1.1 archives are named
204      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-snapshot-</literal><replaceable>yymmdd</replaceable><literal>.tar.gz</literal>,
205      where <replaceable>platform</replaceable> is either
206      <literal>darwinppc</literal>, <literal>darwinx8664</literal>,
207      <literal>linuxppc</literal>, <literal>linuxx8664</literal>, or
208      <literal>freebsdx8664</literal>, and where
209      <replaceable>yymmdd</replaceable> is the year, month, and day
210      the snapshot was released.</para>
211      <para>Because version 1.1 is undergoing active development,
212      there may be times when you want to get sources that are more
213      recent than the most recent snapshot and use them to build
214      yourself a new bleeding-edge &CCL;.  In that case, you should
215      download and install the latest snapshot, and then update your
216      sources via CVS.  At that point you can rebuild and you'll have
217      the latest and greatest &CCL;.  The snapshot has CVS
218      working-copy information in it, so all you need to do to update
219      is</para>
220      <programlisting>
221$ cd ccl
222$ cvs login             # password is "cvs"
223                        # this step only needs to be done once,
224                        # that'll store the trivially encrypted
225                        # password in ~/.cvspas
226$ cvs update
227      </programlisting>
228      <para>Unless you tell it to, cvs won't delete ("prune") empty
229      directories or create new ones when the repository changes.
230      It's generally a good habit to use</para>
231      <programlisting>
232$ cvs update -d -P      # create dirs as needed, prune empty ones
233      </programlisting>
234      <para>Version 1.2 archives follow naming conventions that are
235      similar to those used by 1.0 (though more platforms are supported.)
236      However, rather than containing CVS working-copy information, the
237      1.2 (and, presumably, later) archives contain metainformation used
238      by the Subversion (svn) source-code control system.</para>
239      <para>Subversion client programs are pre-installed on OSX 10.5 and
240      later and are typically either pre-installed or readily available
241      on Linux and FreeBSD platforms.  The <ulink url="">Subversion web page</ulink> contains links to subversion client programs
242      for many platforms; users of OSX versions 10.4 and earlier can also
243      install Subversion clients via Fink or MacPorts.</para>
245    </sect1>
247    <sect1 id="Building-Everything">
248      <title>Building Everything</title>
249      <para>Given that you now have everything you need, do the
250      following in a running &CCL; to bring your Lisp system
251      completely up to date.</para>
252      <programlisting>
253? (ccl:rebuild-ccl :full t)
254      </programlisting>
255      <para>That call to the function <literal>rebuild-ccl</literal>
256      will perform the following steps:</para>
257      <itemizedlist>
258        <listitem>
259          <para>Deletes all fasl files and other object files in the
260          <literal>ccl</literal>directory tree</para>
261        </listitem>
262        <listitem>
263          <para>Runs an external process which does a
264          <literal>make</literal> in the currentplatform's kernel
265          build directory to create a new kernel</para>
266        </listitem>
267        <listitem>
268          <para>Does <literal>(compile-ccl t)</literal> in the running
269          lisp, to produce aset of fasl files from the &ldquo;higher
270          level&rdquo; lisp sources.</para>
271        </listitem>
272        <listitem>
273          <para>Does <literal>(xload-level-0 :force)</literal> in the
274          running lisp, to compile thelisp sources in the
275          &ldquo;ccl:level-0;&rdquo; directory into fasl files and
276          then createa bootstrapping image from those fasl
277          files.</para>
278        </listitem>
279        <listitem>
280          <para>Runs another external process, which causes the newly
281          compiled lispkernel to load the new bootstrapping image.
282          The bootsrtrapping image then loadsthe &ldquo;higher
283          level&rdquo; fasl files and a new copy of the platform's
284          full heap imageis then saved.</para>
285        </listitem>
286      </itemizedlist>
287      <para>If all goes well, it'll all happen without user
288      intervention and with some simple progress messages.  If
289      anything goes wrong during execution of either of the external
290      processes, the process output is displayed as part of a lisp
291      error message.</para>
292      <para><literal>rebuild-ccl</literal> is essentially just a short
293      cut for running all the individual steps involved in rebuilding
294      the system.  You can also execute these steps individually, as
295      described below.</para>
296    </sect1>
298    <sect1 id="Building-the-kernel">
299      <title>Building the kernel</title>
300      <para>The Lisp kernel is the executable which you run to use
301      Lisp.  It doesn't actually contain the entire Lisp
302      implementation; rather, it loads a heap image which contains the
303      specifics - the "library", as it might be called if this was a C
304      program.  The kernel also provides runtime support to the heap
305      image, such as garbage collection, memory allocation, exception
306      handling, and the OS interface.</para>
308      <para>The Lisp kernel file has different names on different
309      platforms. See FIXTHIS . On all platforms the lisp kernel sources reside
310      in <literal>ccl/lisp-kernel</literal>.</para>
312      <para>This section gives directions on how to rebuild the Lisp
313      kernel from its source code.  Most &CCL; users will rarely
314      have to do this.  You probably will only need to do it if you are
315      attempting to port &CCL; to a new architecture or extend or enhance
316      its kernel in some way.  As mentioned above, this step happens
317      automatically when you do
318      <programlisting>
319? (rebuild-ccl :full t)
320      </programlisting>
321      </para>
324      <sect2 id="Kernel-build-prerequisites">
325        <title>Kernel build prerequisites</title>
326        <para>The &CCL; kernel can be bult with the following widely
327        available tools:</para>
328        <itemizedlist>
329          <listitem><para>cc or gcc- the GNU C compiler</para></listitem>
330          <listitem><para>ld - the GNU linker</para></listitem>
331          <listitem><para>m4 or gm4- the GNU m4 macro processor</para></listitem>
332          <listitem><para>as - the GNU assembler (version 2.10.1 or later)</para></listitem>
333          <listitem><para>make - either GNU make or, on FreeBSD, the default BSD make program</para></listitem>
334        </itemizedlist>
335        <para> In general, the more recent the versions of those
336        tools, the better; some versions of gcc 3.x on Linux have
337        difficulty compiling some of the kernel source code correctly
338        (so gcc 4.0 should be used, if possible.)  On OSX, the
339        versions of the tools distributed with XCode should work fine;
340        on Linux, the versions of the tools installed with the OS (or
341        available through its package management system) should work
342        fine if they're "recent enough".  On FreeBSD, the installed
343        version of the <literal>m4</literal> program doesn't support
344        some features that the kernel build process depends on; the
345        GNU version of the m4 macroprocessor (called
346        <literal>gm4</literal> on FreeBSD) should be installed
348        </para>
349      </sect2>
350      <sect2 id="kernel-build-command">
351        <title>Using "make" to build the lisp kernel</title>
352        <para>With those tools in place, do:
353        <programlisting>
354shell> cd ccl/lisp-kernel/<replaceable>PLATFORM</replaceable>
355shell> make
356        </programlisting>
357        </para>
358        <para>That'll assemble several assembly language source files,
359        compile several C source files, and link
360        ../../<replaceable>the kernel</replaceable>.
361        </para>
362      </sect2>
363    </sect1>
364    <sect1 id="Building-the-heap-image">
365      <title>Building the heap image</title>
366      <para>The initial heap image is loaded by the Lisp kernel, and
367      provides most all of the language implementation The heap image
368      captures the entire state of a running Lisp (except for external
369      resources, such as open files and TCP sockets).  After it is
370      loaded, the contents of the new Lisp process's memory are
371      exactly the same as those of the old Lisp process when the image
372      was created.</para>
373      <para>The heap image is how we get around the fact that we can't
374      run Lisp code until we have a working Lisp implementation, and
375      we can't make our Lisp implementation work until we can run Lisp
376      code.  Since the heap image already contains a fully-working
377      implementation, all we need to do is load it into memory and
378      start using it.</para>
379      <para>If you're building a new version of &CCL;, you need to
380      build a new heap image.</para>
381      <para>(You might also wish to build a heap image if you have a
382      large program which it is very complicated or time-consuming to
383      load, so that you will be able to load it once, save an image,
384      and thenceforth never have to load it again. At any time, a heap
385      image capturing the entire memory state of a running Lisp can be
386      created by calling the function
387      <literal>ccl:save-application</literal>.)</para>
389      <sect2 id="Development-cycle">
390        <title>Development cycle</title>
391        <para>Creating a new &CCL; full heap image consists of the
392        following steps:</para>
393        <orderedlist>
394          <listitem><para>Using your existing &CCL;, create a
395          bootstrapping image</para></listitem>
396          <listitem><para>Using your existing &CCL;, recompile your
397          updated &CCL; sources</para></listitem>
398          <listitem><para>Invoke &CCL; with the bootstrapping image
399          you just created (rather than with the existing full heap
400          image).</para></listitem>
401        </orderedlist>
402        <para>When you invoke &CCL; with the bootstrapping image, it
403        will start up, load al of the &CCL; fasl files, and save out
404        a new full heap image.  Voila.  You've created a new heap
405        image.</para>
406        <para>A few points worth noting:</para>
407        <itemizedlist>
408          <listitem>
409            <para>There's a circular dependency between the full heap
410            image and thebootstrapping image, in that each is used to
411            build the other.</para>
412          </listitem>
413          <listitem>
414            <para>There are some minor implementation
415            differences, but the environment in effect after the
416            bootstrapping image has loaded its fasl files is essentially
417            equivalent to the environment provided by the full heap
418            image; the latter loads a lot faster and is easier to
419            distribute, of course.</para>
420          </listitem>
421          <listitem>
422            <para>If the full heap image doesn't work (because
423            of an OScompatibilty problem or other bug), it's very likely
424            that thebootstrapping image will suffer the same
425            problems.</para>
426          </listitem>
427        </itemizedlist>
428        <para>Given a bootstrapping image and a set of up-to-date fasl
429        files, the development cycle usually involves editing lisp
430        sources (or updating those sources via cvs update),
431        recompiling modified files, and using the bootstrapping image
432        to produce a new heap image.</para>
433      </sect2>
435      <sect2 id="Generating-a-bootstrapping-image">
436        <title>Generating a bootstrapping image</title>
437        <para>The bootstrapping image isn't provided in &CCL;
438        distributions. It can be built from the source code provided
439        in distributions (using a lisp image and kernel provided in
440        those distributions) using the procedure described
441        below.</para>
443        <para>The bootstrapping image is built by invoking a special
444        utility inside a running &CCL; heap image to load files
445        contained in the <literal>ccl/level-0</literal> directory. The
446        bootstrapping image loads several dozen fasl files.  After
447        it's done so, it saves a heap image via
448        <literal>save-application</literal>. This process is called
449        "cross-dumping".</para>
451        <para>Given a source distribution, a lisp kernel, and aheap
452        image, one can produce a bootstapping image by first invoking
453        &CCL; from the shell:</para>
454        <programlisting>
455shell&gt; openmcl
456Welcome to &CCL; .... !
458        </programlisting>
459        <para>then calling <literal>ccl:xload-level-0</literal> at the
460        lisp prompt</para>
461        <programlisting>
462? (ccl:xload-level-0)
463        </programlisting>
464        <para>This will compile the lisp sources in the ccl/level-0
465        directory if they're newer than the corresponding fasl files
466        and will then load the resulting fasl files into a simulated
467        lisp heap contained inside data structures inside the running
468        lisp. That simulated heap image is then written to
469        disk.</para>
470        <para><literal>xload-level-0</literal> should be called
471        whenever your existing boot image is out-of-date with respect
472        to the source files in <literal>ccl:level-0;</literal>
473        :</para>
474        <programlisting>
475? (ccl:xload-level-0 :force)
477        <para>will force recompilation of the level-0 sources.</para>
478      </sect2>
480      <sect2 id="Generating-fasl-files">
481        <title>Generating fasl files</title>
482        <para> Calling:</para>
483        <programlisting>
484? (ccl:compile-ccl)
485        </programlisting>
486        <para>at the lisp prompt will compile any fasl files that are
487        out-of-date with respect to the corresponding lisp sources;
488        <literal>(ccl:compile-ccl t)</literal> will force
489        recompilation. <literal>ccl:compile-ccl</literal> will reload
490        newly-compiled versions of some files;
491        <literal>ccl:xcompile-ccl</literal> is analogous, but skips
492        this reloading step.</para>
493        <para>Unless there are bootstrapping considerations involved,
494        it usually doesn't matter whether these files reloaded after
495        they're recompiled.</para>
496        <para>Calling <literal>compile-ccl</literal> or
497        <literal>xcompile-ccl</literal> in an environment where fasl
498        files don't yet exist may produce warnings to that effect
499        whenever files are <literal>require</literal>d during
500        compilation; those warnings can be safely ignored. Depending
501        on the maturity of the &CCL; release, calling
502        <literal>compile-ccl</literal> or
503        <literal>xcompile-ccl</literal> may also produce several
504        warnings about undefined functions, etc. They should be
505        cleaned up at some point.</para>
506      </sect2>
508      <sect2 id="Building-a-full-image-from-a-bootstrapping-image">
509        <title>Building a full image from a bootstrapping image</title>
510        <para>To build a full image from a bootstrapping image, just
511        invoke the kernel with the bootstrapping image is an
512        argument</para>
513        <programlisting>
514$ cd ccl                        # wherever your ccl directory is
516        </programlisting>
517        <para>Where <replaceable>KERNEL</replaceable> and
518        <replaceable>BOOT_IMAGE</replaceable> are the names of
519        the kernel and boot image appropriate to the platform you are
520        running on.  See FIXTHIS</para>
521        <para>That should load a few dozen fasl files (printing a
522        message as each file is loaded.) If all of these files
523        successfully load, the lisp will print a prompt. You should be
524        able to do essentially everything in that environment that you
525        can in the environment provided by a "real" heap image. If
526        you're confident that things loaded OK, you can save that
527        image.</para>
528        <programlisting>
529? (ccl:save-application "<replaceable>image_name</replaceable>") ; Overwiting the existing heap image
530        </programlisting>
531        <para>Where <replaceable>image_name</replaceable> is the name
532        of the full heap image for your platform. See FIXTHIS.</para>
533        <para>If things go wrong in the early stages of the loading
534        sequence, errors are often difficult to debug; until a fair
535        amount of code (CLOS, the CL condition system, streams, the
536        reader, the read-eval-print loop) is loaded, it's generally
537        not possible for the lisp to report an error.  Errors that
538        occur during these early stages ("the cold load") sometimes
539        cause the lisp kernel debugger (see ) to be invoked; it's
540        primitive, but can sometimes help one to get oriented.</para>
541      </sect2>
542    </sect1>
543  </chapter>
Note: See TracBrowser for help on using the repository browser.