source: trunk/source/doc/src/build.xml @ 8606

Last change on this file since 8606 was 8606, checked in by gb, 13 years ago

Rename chapter-level files to not include chapter numbers (so that it'd be
less confusing to change chapter order, remove/introduce chapters, etc.)

Ensure that all chapter-level files have ENTITY definitions in their
DOCTYPE forms.

Define a CCL entity (currently just <literal>CCL</literal>; change all
references to "OpenMCL" to "&CCL;"

File size: 24.4 KB
Line 
1<?xml version="1.0" encoding="utf-8"?>
2<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[
3<!ENTITY rest "<varname>&amp;rest</varname>">
4<!ENTITY key "<varname>&amp;key</varname>">
5<!ENTITY optional "<varname>&amp;optional</varname>">
6<!ENTITY body "<varname>&amp;body</varname>">
7<!ENTITY aux "<varname>&amp;aux</varname>">
8<!ENTITY allow-other-keys "<varname>&amp;allow-other-keys</varname>">
9<!ENTITY CCL "<literal>CCL</literal>">
10]>
11  <chapter><title>Building &CCL; from its Source Code</title>
12
13    <para>&CCL;, like many other Lisp implementations, consists of a
14    kernel and a heap image.  The kernel is an ordinary C program, and
15    is built with a C compiler.  It provides very basic and
16    fundamental facilities, such as memory management, garbage
17    collection, and bootstrapping.  All the higher-level features are
18    written in Lisp, and compiled into the heap image.  Both parts are
19    needed to have a working Lisp implementation; neither the kernel
20    nor the heap image can stand on their own.</para>
21
22    <para>You may already know that, when you have a C compiler which
23    is written in C, to build the compiler, you need to already have a
24    C compiler available.  The heap image includes a Lisp compiler,
25    which is written in Lisp.  Therefore, you need a working Lisp
26    compiler in order to build the heap image!</para>
27   
28    <para>Where will you get a working Lisp compiler?  No worries; you
29    can use a precompiled copy of a (slightly older and compatible)
30    version of &CCL;.  For help, read on.</para>
31
32    <para>In principle it would be possible to use another Lisp as the
33    host compiler, rather than an old &CCL;; this would be a
34    challenging and experimental way to build, and is not described
35    here.</para>
36
37    <sect1 id="building-definitions"><title>building definitions</title>
38      <para>The following terms are used in subsequent sections; it
39      may be helpful to refer to these definitions.</para>
40
41      <para><indexterm><primary>fasl files</primary></indexterm> are
42      the object files produced by<literal>compile-file</literal>.
43      fasl files store the machine codeassociated with function
44      definitions and the external representationof other lisp objects
45      in a compact, machine-readable form. fasl is short for
46      &ldquo;<literal>FAS</literal>t
47      <literal>L</literal>oading&rdquo;.&CCL; uses different
48      pathname types (extensions) to name faslfiles on different
49      platforms; see <xref
50      linkend="Platform-specific-filename-conventions"/> </para>
51
52      <para>The <indexterm><primary>lisp kernel</primary></indexterm> is
53      a C program (with a fair amount ofplatform-specific assembly
54      language code as well.)  Its basic job isto map a lisp heap
55      image into memory, transfer control to somecompiled lisp code
56      that the image contains, handle any exceptionsthat occur during
57      the execution of that lisp code, and provide variousother forms
58      of runtime support for that code.&CCL; uses different
59      filenames to name the lisp kernel fileson different platforms;
60      see FIXTHIS.</para>
61
62      <para>A <indexterm><primary>heap image</primary></indexterm> is
63      a file that can be quickly mapped into aprocess's address space.
64      Conceptually, it's not too different from anexecutable file or
65      shared library in the OS's native format (ELF orMach-O/dyld
66      format); for historical reasons, &CCL;'s own heap images are in
67      their own (fairly simple) format.The term <literal>full heap
68      image</literal> refers to a heap image file thatcontains all of
69      the code and data that comprise &CCL;.&CCL; uses different
70      filenames to name the standard full heapimage files on different
71      platforms; see FIXTHIS .</para>
72
73      <para>A <indexterm><primary>bootstrapping
74      image</primary></indexterm> is a minimal heap image used in
75      the process of building &CCL; itself.  The bootstrapping image
76      containsjust enough code to load the rest of &CCL; from fasl
77      files.  It mayhelp to think of the bootstrapping image as the
78      egg and the full heapimage as the chicken...&CCL; uses
79      different filenames to name the standardbootstrapping image
80      files on different platforms; see FIXTHIS .</para>
81
82      <para>Each supported platform (and possibly a few
83      as-yet-unsupported ones) has a uniquely named subdirectory of
84      <literal>ccl/lisp-kernel/</literal>; each such
85      <indexterm><primary>kernel build directory</primary></indexterm>
86      contains a Makefile and may contain some auxiliary files (linker
87      scripts, etc.) that are used to build the lispkernel on a
88      particular platform.The platform-specific name of the kernel
89      build directory is described in FIXTHIS.</para>
90
91      <sect2 id="filename_conventions">
92       <title>Platform-specific filename conventions</title>
93       <table id ="Platform-specific-filename-conventions">
94         <title>Platform-specific filename conventions</title>
95         <tgroup cols="6">
96           <thead>
97            <row>
98                <entry>Platform</entry>
99                <entry>kernel</entry>
100                <entry>full-image</entry>
101                <entry>boot-image</entry>
102                <entry>fasl extension</entry>
103                <entry>kernel-build directory</entry>
104            </row>
105           </thead>
106           <tbody>
107             <row>
108               <entry>DarwinPPC32</entry>
109                <entry>dppccl</entry>
110                <entry>dppccl.image</entry>
111                <entry>ppc-boot.image</entry>
112                <entry>.dfsl</entry>
113                <entry>darwinppc</entry>
114             </row>
115             <row>
116               <entry>LinuxPPC32</entry>
117                <entry>ppccl</entry>
118                <entry>PPCCL</entry>
119                <entry>ppc-boot</entry>
120                <entry>.pfsl</entry>
121                <entry>linuxppc</entry>
122             </row>
123             <row>
124               <entry>DarwinPPC64</entry>
125               <entry>dppccl64</entry>
126               <entry>dppccl64.image</entry>
127               <entry>ppc-boot64.image</entry>
128               <entry>.d64fsl</entry>
129               <entry>darwinppc64</entry>
130             </row>
131              <row>
132                <entry>LinuxPPC64</entry>
133                <entry>ppccl64</entry>
134                <entry>PPCCL64</entry>
135                <entry>ppc-boot64</entry>
136                <entry>.p64fsl</entry>
137                <entry>linuxppc64</entry>
138              </row>
139              <row>
140                <entry>LinuxX8664</entry>
141                <entry>lx86cl64</entry>
142                <entry>LX86CL64</entry>
143                <entry>x86-boot64</entry>
144                <entry>.lx64fsl</entry>
145                <entry>linuxx8664</entry>
146              </row>
147              <row>
148                <entry>DarwinX8664</entry>
149                <entry>dx86cl64</entry>
150                <entry>dx86cl64.image</entry>
151                <entry>x86-boot64.image</entry>
152                <entry>.dx64fsl</entry>
153                <entry>darwinx8664</entry>
154              </row>
155              <row>
156                <entry>FreeBSDX8664</entry>
157                <entry>fx86cl64</entry>
158                <entry>FX86CL64</entry>
159                <entry>fx86-boot64</entry>
160                <entry>.fx64fsl</entry>
161                <entry>freebsdx8664</entry>
162              </row>
163           </tbody>
164         </tgroup>
165       </table>
166      </sect2>
167    </sect1>
168
169    <sect1 id="Setting-Up-to-Build">
170      <title>Setting Up to Build</title>
171      <para>There are currently three versions of &CCL; that you
172      might want to use (and therefore might want to build from
173      source):</para>
174      <itemizedlist>
175        <listitem><para>Version 1.0 - the more stable version</para></listitem>
176        <listitem><para>Version 1.1 - the more recent version, which
177        runs on more platforms (including x86-64 platforms) and
178        supports Unicode</para></listitem>
179        <listitem><para>Version 1.2 - supports (at least) all of the
180        features and platforms as 1.1, but is distributed and updated
181        differently</para></listitem>
182      </itemizedlist>
183      <para>All versions are available for download from the &CCL;
184      website in the form of archives that contain everything you need
185      to work with &CCL;, including the complete sources, a full
186      heap image, and the foreign-function interface database.</para>
187      <para>Version 1.0 archives are named
188      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-all-1.0.tar.gz</literal>,
189      where <replaceable>platform</replaceable> is either
190      <literal>darwinppc</literal>, <literal>darwinppc64</literal>, or
191      <literal>linuxppc</literal>.  Because version 1.0 is no longer
192      undergoing active development, you won't ever need to update
193      these sources.</para>
194      <para>Version 1.1 archives are named
195      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-snapshot-</literal><replaceable>yymmdd</replaceable><literal>.tar.gz</literal>,
196      where <replaceable>platform</replaceable> is either
197      <literal>darwinppc</literal>, <literal>darwinx8664</literal>,
198      <literal>linuxppc</literal>, <literal>linuxx8664</literal>, or
199      <literal>freebsdx8664</literal>, and where
200      <replaceable>yymmdd</replaceable> is the year, month, and day
201      the snapshot was released.</para>
202      <para>Because version 1.1 is undergoing active development,
203      there may be times when you want to get sources that are more
204      recent than the most recent snapshot and use them to build
205      yourself a new bleeding-edge &CCL;.  In that case, you should
206      download and install the latest snapshot, and then update your
207      sources via CVS.  At that point you can rebuild and you'll have
208      the latest and greatest &CCL;.  The snapshot has CVS
209      working-copy information in it, so all you need to do to update
210      is</para>
211      <programlisting>
212$ cd ccl
213$ cvs login             # password is "cvs"
214                        # this step only needs to be done once,
215                        # that'll store the trivially encrypted
216                        # password in ~/.cvspas
217$ cvs update
218      </programlisting>
219      <para>Unless you tell it to, cvs won't delete ("prune") empty
220      directories or create new ones when the repository changes.
221      It's generally a good habit to use</para>
222      <programlisting>
223$ cvs update -d -P      # create dirs as needed, prune empty ones
224      </programlisting>
225      <para>Version 1.2 archives follow naming conventions that are
226      similar to those used by 1.0 (though more platforms are supported.)
227      However, rather than containing CVS working-copy information, the
228      1.2 (and, presumably, later) archives contain metainformation used
229      by the Subversion (svn) source-code control system.</para>
230      <para>Subversion client programs are pre-installed on OSX 10.5 and
231      later and are typically either pre-installed or readily available
232      on Linux and FreeBSD platforms.  The <ulink url="http://subversion.tigris.org">Subversion web page</ulink> contains links to subversion client programs
233      for many platforms; users of OSX versions 10.4 and earlier can also
234      install Subversion clients via Fink or MacPorts.</para>
235     
236    </sect1>
237
238    <sect1 id="Building-Everything">
239      <title>Building Everything</title>
240      <para>Given that you now have everything you need, do the
241      following in a running &CCL; to bring your Lisp system
242      completely up to date.</para>
243      <programlisting>
244? (ccl:rebuild-ccl :full t)
245      </programlisting>
246      <para>That call to the function <literal>rebuild-ccl</literal>
247      will perform the following steps:</para>
248      <itemizedlist>
249        <listitem>
250          <para>Deletes all fasl files and other object files in the
251          <literal>ccl</literal>directory tree</para>
252        </listitem>
253        <listitem>
254          <para>Runs an external process which does a
255          <literal>make</literal> in the currentplatform's kernel
256          build directory to create a new kernel</para>
257        </listitem>
258        <listitem>
259          <para>Does <literal>(compile-ccl t)</literal> in the running
260          lisp, to produce aset of fasl files from the &ldquo;higher
261          level&rdquo; lisp sources.</para>
262        </listitem>
263        <listitem>
264          <para>Does <literal>(xload-level-0 :force)</literal> in the
265          running lisp, to compile thelisp sources in the
266          &ldquo;ccl:level-0;&rdquo; directory into fasl files and
267          then createa bootstrapping image from those fasl
268          files.</para>
269        </listitem>
270        <listitem>
271          <para>Runs another external process, which causes the newly
272          compiled lispkernel to load the new bootstrapping image.
273          The bootsrtrapping image then loadsthe &ldquo;higher
274          level&rdquo; fasl files and a new copy of the platform's
275          full heap imageis then saved.</para>
276        </listitem>
277      </itemizedlist>
278      <para>If all goes well, it'll all happen without user
279      intervention and with some simple progress messages.  If
280      anything goes wrong during execution of either of the external
281      processes, the process output is displayed as part of a lisp
282      error message.</para>
283      <para><literal>rebuild-ccl</literal> is essentially just a short
284      cut for running all the individual steps involved in rebuilding
285      the system.  You can also execute these steps individually, as
286      described below.</para>
287    </sect1>
288
289    <sect1 id="Building-the-kernel">
290      <title>Building the kernel</title>
291      <para>The Lisp kernel is the executable which you run to use
292      Lisp.  It doesn't actually contain the entire Lisp
293      implementation; rather, it loads a heap image which contains the
294      specifics - the "library", as it might be called if this was a C
295      program.  The kernel also provides runtime support to the heap
296      image, such as garbage collection, memory allocation, exception
297      handling, and the OS interface.</para>
298
299      <para>The Lisp kernel file has different names on different
300      platforms. See FIXTHIS . On all platforms the lisp kernel sources reside
301      in <literal>ccl/lisp-kernel</literal>.</para>
302
303      <para>This section gives directions on how to rebuild the Lisp
304      kernel from its source code.  Most &CCL; users will rarely
305      have to do this.  You probably will only need to do it if you are
306      attempting to port &CCL; to a new architecture or extend or enhance
307      its kernel in some way.  As mentioned above, this step happens
308      automatically when you do
309      <programlisting>
310? (rebuild-ccl :full t)
311      </programlisting>
312      </para>
313
314
315      <sect2 id="Kernel-build-prerequisites">
316        <title>Kernel build prerequisites</title>
317        <para>The &CCL; kernel can be bult with the following widely
318        available tools:</para>
319        <itemizedlist>
320          <listitem><para>cc or gcc- the GNU C compiler</para></listitem>
321          <listitem><para>ld - the GNU linker</para></listitem>
322          <listitem><para>m4 or gm4- the GNU m4 macro processor</para></listitem>
323          <listitem><para>as - the GNU assembler (version 2.10.1 or later)</para></listitem>
324          <listitem><para>make - either GNU make or, on FreeBSD, the default BSD make program</para></listitem>
325        </itemizedlist>
326        <para> In general, the more recent the versions of those
327        tools, the better; some versions of gcc 3.x on Linux have
328        difficulty compiling some of the kernel source code correctly
329        (so gcc 4.0 should be used, if possible.)  On OSX, the
330        versions of the tools distributed with XCode should work fine;
331        on Linux, the versions of the tools installed with the OS (or
332        available through its package management system) should work
333        fine if they're "recent enough".  On FreeBSD, the installed
334        version of the <literal>m4</literal> program doesn't support
335        some features that the kernel build process depends on; the
336        GNU version of the m4 macroprocessor (called
337        <literal>gm4</literal> on FreeBSD) should be installed
338
339        </para>
340      </sect2>
341      <sect2 id="kernel-build-command">
342        <title>Using "make" to build the lisp kernel</title>
343        <para>With those tools in place, do:
344        <programlisting>
345shell> cd ccl/lisp-kernel/<replaceable>PLATFORM</replaceable>
346shell> make
347        </programlisting>
348        </para>
349        <para>That'll assemble several assembly language source files,
350        compile several C source files, and link
351        ../../<replaceable>the kernel</replaceable>.
352        </para>
353      </sect2>
354    </sect1>
355    <sect1 id="Building-the-heap-image">
356      <title>Building the heap image</title>
357      <para>The initial heap image is loaded by the Lisp kernel, and
358      provides most all of the language implementation The heap image
359      captures the entire state of a running Lisp (except for external
360      resources, such as open files and TCP sockets).  After it is
361      loaded, the contents of the new Lisp process's memory are
362      exactly the same as those of the old Lisp process when the image
363      was created.</para>
364      <para>The heap image is how we get around the fact that we can't
365      run Lisp code until we have a working Lisp implementation, and
366      we can't make our Lisp implementation work until we can run Lisp
367      code.  Since the heap image already contains a fully-working
368      implementation, all we need to do is load it into memory and
369      start using it.</para>
370      <para>If you're building a new version of &CCL;, you need to
371      build a new heap image.</para>
372      <para>(You might also wish to build a heap image if you have a
373      large program which it is very complicated or time-consuming to
374      load, so that you will be able to load it once, save an image,
375      and thenceforth never have to load it again. At any time, a heap
376      image capturing the entire memory state of a running Lisp can be
377      created by calling the function
378      <literal>ccl:save-application</literal>.)</para>
379
380      <sect2 id="Development-cycle">
381        <title>Development cycle</title>
382        <para>Creating a new &CCL; full heap image consists of the
383        following steps:</para>
384        <orderedlist>
385          <listitem><para>Using your existing &CCL;, create a
386          bootstrapping image</para></listitem>
387          <listitem><para>Using your existing &CCL;, recompile your
388          updated &CCL; sources</para></listitem>
389          <listitem><para>Invoke &CCL; with the bootstrapping image
390          you just created (rather than with the existing full heap
391          image).</para></listitem>
392        </orderedlist>
393        <para>When you invoke &CCL; with the bootstrapping image, it
394        will start up, load al of the &CCL; fasl files, and save out
395        a new full heap image.  Voila.  You've created a new heap
396        image.</para>
397        <para>A few points worth noting:</para>
398        <itemizedlist>
399          <listitem>
400            <para>There's a circular dependency between the full heap
401            image and thebootstrapping image, in that each is used to
402            build the other.</para>
403          </listitem>
404          <listitem>
405            <para>There are some minor implementation
406            differences, but the environment in effect after the
407            bootstrapping image has loaded its fasl files is essentially
408            equivalent to the environment provided by the full heap
409            image; the latter loads a lot faster and is easier to
410            distribute, of course.</para>
411          </listitem>
412          <listitem>
413            <para>If the full heap image doesn't work (because
414            of an OScompatibilty problem or other bug), it's very likely
415            that thebootstrapping image will suffer the same
416            problems.</para>
417          </listitem>
418        </itemizedlist>
419        <para>Given a bootstrapping image and a set of up-to-date fasl
420        files, the development cycle usually involves editing lisp
421        sources (or updating those sources via cvs update),
422        recompiling modified files, and using the bootstrapping image
423        to produce a new heap image.</para>
424      </sect2>
425
426      <sect2 id="Generating-a-bootstrapping-image">
427        <title>Generating a bootstrapping image</title>
428        <para>The bootstrapping image isn't provided in &CCL;
429        distributions. It can be built from the source code provided
430        in distributions (using a lisp image and kernel provided in
431        those distributions) using the procedure described
432        below.</para>
433
434        <para>The bootstrapping image is built by invoking a special
435        utility inside a running &CCL; heap image to load files
436        contained in the <literal>ccl/level-0</literal> directory. The
437        bootstrapping image loads several dozen fasl files.  After
438        it's done so, it saves a heap image via
439        <literal>save-application</literal>. This process is called
440        "cross-dumping".</para>
441
442        <para>Given a source distribution, a lisp kernel, and aheap
443        image, one can produce a bootstapping image by first invoking
444        &CCL; from the shell:</para>
445        <programlisting>
446shell&gt; openmcl
447Welcome to &CCL; .... !
448?
449        </programlisting>
450        <para>then calling <literal>ccl:xload-level-0</literal> at the
451        lisp prompt</para>
452        <programlisting>
453? (ccl:xload-level-0)
454        </programlisting>
455        <para>This will compile the lisp sources in the ccl/level-0
456        directory if they're newer than the corresponding fasl files
457        and will then load the resulting fasl files into a simulated
458        lisp heap contained inside data structures inside the running
459        lisp. That simulated heap image is then written to
460        disk.</para>
461        <para><literal>xload-level-0</literal> should be called
462        whenever your existing boot image is out-of-date with respect
463        to the source files in <literal>ccl:level-0;</literal>
464        :</para>
465        <programlisting>
466? (ccl:xload-level-0 :force)
467</programlisting>
468        <para>will force recompilation of the level-0 sources.</para>
469      </sect2>
470
471      <sect2 id="Generating-fasl-files">
472        <title>Generating fasl files</title>
473        <para> Calling:</para>
474        <programlisting>
475? (ccl:compile-ccl)
476        </programlisting>
477        <para>at the lisp prompt will compile any fasl files that are
478        out-of-date with respect to the corresponding lisp sources;
479        <literal>(ccl:compile-ccl t)</literal> will force
480        recompilation. <literal>ccl:compile-ccl</literal> will reload
481        newly-compiled versions of some files;
482        <literal>ccl:xcompile-ccl</literal> is analogous, but skips
483        this reloading step.</para>
484        <para>Unless there are bootstrapping considerations involved,
485        it usually doesn't matter whether these files reloaded after
486        they're recompiled.</para>
487        <para>Calling <literal>compile-ccl</literal> or
488        <literal>xcompile-ccl</literal> in an environment where fasl
489        files don't yet exist may produce warnings to that effect
490        whenever files are <literal>require</literal>d during
491        compilation; those warnings can be safely ignored. Depending
492        on the maturity of the &CCL; release, calling
493        <literal>compile-ccl</literal> or
494        <literal>xcompile-ccl</literal> may also produce several
495        warnings about undefined functions, etc. They should be
496        cleaned up at some point.</para>
497      </sect2>
498
499      <sect2 id="Building-a-full-image-from-a-bootstrapping-image">
500        <title>Building a full image from a bootstrapping image</title>
501        <para>To build a full image from a bootstrapping image, just
502        invoke the kernel with the bootstrapping image is an
503        argument</para>
504        <programlisting>
505$ cd ccl                        # wherever your ccl directory is
506$ ./KERNEL BOOT_IMAGE
507        </programlisting>
508        <para>Where <replaceable>KERNEL</replaceable> and
509        <replaceable>BOOT_IMAGE</replaceable> are the names of
510        the kernel and boot image appropriate to the platform you are
511        running on.  See FIXTHIS</para>
512        <para>That should load a few dozen fasl files (printing a
513        message as each file is loaded.) If all of these files
514        successfully load, the lisp will print a prompt. You should be
515        able to do essentially everything in that environment that you
516        can in the environment provided by a "real" heap image. If
517        you're confident that things loaded OK, you can save that
518        image.</para>
519        <programlisting>
520? (ccl:save-application "<replaceable>image_name</replaceable>") ; Overwiting the existing heap image
521        </programlisting>
522        <para>Where <replaceable>image_name</replaceable> is the name
523        of the full heap image for your platform. See FIXTHIS.</para>
524        <para>If things go wrong in the early stages of the loading
525        sequence, errors are often difficult to debug; until a fair
526        amount of code (CLOS, the CL condition system, streams, the
527        reader, the read-eval-print loop) is loaded, it's generally
528        not possible for the lisp to report an error.  Errors that
529        occur during these early stages ("the cold load") sometimes
530        cause the lisp kernel debugger (see ) to be invoked; it's
531        primitive, but can sometimes help one to get oriented.</para>
532      </sect2>
533    </sect1>
534  </chapter>
Note: See TracBrowser for help on using the repository browser.