source: trunk/source/doc/src/02-build.xml @ 8567

Last change on this file since 8567 was 8567, checked in by rme, 14 years ago

Break up monolithic openmcl-documentation.xml file into chapters. Add file
top.xml, which includes all the chapters via xi:xinclude.

File size: 24.2 KB
1<?xml version="1.0" encoding="utf-8"?>
2<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "">
3  <chapter><title>Building OpenMCL from its Source Code</title>
5    <para>OpenMCL, like many other Lisp implementations, consists of a
6    kernel and a heap image.  The kernel is an ordinary C program, and
7    is built with a C compiler.  It provides very basic and
8    fundamental facilities, such as memory management, garbage
9    collection, and bootstrapping.  All the higher-level features are
10    written in Lisp, and compiled into the heap image.  Both parts are
11    needed to have a working Lisp implementation; neither the kernel
12    nor the heap image can stand on their own.</para>
14    <para>You may already know that, when you have a C compiler which
15    is written in C, to build the compiler, you need to already have a
16    C compiler available.  The heap image includes a Lisp compiler,
17    which is written in Lisp.  Therefore, you need a working Lisp
18    compiler in order to build the heap image!</para>
20    <para>Where will you get a working Lisp compiler?  No worries; you
21    can use a precompiled copy of a (slightly older and compatible)
22    version of OpenMCL.  For help, read on.</para>
24    <para>In principle it would be possible to use another Lisp as the
25    host compiler, rather than an old OpenMCL; this would be a
26    challenging and experimental way to build, and is not described
27    here.</para>
29    <sect1 id="building-definitions"><title>building definitions</title>
30      <para>The following terms are used in subsequent sections; it
31      may be helpful to refer to these definitions.</para>
33      <para><indexterm><primary>fasl files</primary></indexterm> are
34      the object files produced by<literal>compile-file</literal>.
35      fasl files store the machine codeassociated with function
36      definitions and the external representationof other lisp objects
37      in a compact, machine-readable form. fasl is short for
38      &ldquo;<literal>FAS</literal>t
39      <literal>L</literal>oading&rdquo;.OpenMCL uses different
40      pathname types (extensions) to name faslfiles on different
41      platforms; see <xref
42      linkend="Platform-specific-filename-conventions"/> </para>
44      <para>The <indexterm><primary>lisp kernel</primary></indexterm> is
45      a C program (with a fair amount ofplatform-specific assembly
46      language code as well.)  Its basic job isto map a lisp heap
47      image into memory, transfer control to somecompiled lisp code
48      that the image contains, handle any exceptionsthat occur during
49      the execution of that lisp code, and provide variousother forms
50      of runtime support for that code.OpenMCL uses different
51      filenames to name the lisp kernel fileson different platforms;
52      see FIXTHIS.</para>
54      <para>A <indexterm><primary>heap image</primary></indexterm> is
55      a file that can be quickly mapped into aprocess's address space.
56      Conceptually, it's not too different from anexecutable file or
57      shared library in the OS's native format (ELF orMach-O/dyld
58      format); for historical reasons, OpenMCL's own heap images are in
59      their own (fairly simple) format.The term <literal>full heap
60      image</literal> refers to a heap image file thatcontains all of
61      the code and data that comprise OpenMCL.OpenMCL uses different
62      filenames to name the standard full heapimage files on different
63      platforms; see FIXTHIS .</para>
65      <para>A <indexterm><primary>bootstrapping
66      image</primary></indexterm> is a minimal heap image used in
67      the process of building OpenMCL itself.  The bootstrapping image
68      containsjust enough code to load the rest of OpenMCL from fasl
69      files.  It mayhelp to think of the bootstrapping image as the
70      egg and the full heapimage as the chicken...OpenMCL uses
71      different filenames to name the standardbootstrapping image
72      files on different platforms; see FIXTHIS .</para>
74      <para>Each supported platform (and possibly a few
75      as-yet-unsupported ones) has a uniquely named subdirectory of
76      <literal>ccl/lisp-kernel/</literal>; each such
77      <indexterm><primary>kernel build directory</primary></indexterm>
78      contains a Makefile and may contain some auxiliary files (linker
79      scripts, etc.) that are used to build the lispkernel on a
80      particular platform.The platform-specific name of the kernel
81      build directory is described in FIXTHIS.</para>
83      <sect2 id="filename_conventions">
84       <title>Platform-specific filename conventions</title>
85       <table id ="Platform-specific-filename-conventions">
86         <title>Platform-specific filename conventions</title>
87         <tgroup cols="6">
88           <thead>
89            <row>
90                <entry>Platform</entry>
91                <entry>kernel</entry>
92                <entry>full-image</entry>
93                <entry>boot-image</entry>
94                <entry>fasl extension</entry>
95                <entry>kernel-build directory</entry>
96            </row>
97           </thead>
98           <tbody>
99             <row>
100               <entry>DarwinPPC32</entry>
101                <entry>dppccl</entry>
102                <entry>dppccl.image</entry>
103                <entry>ppc-boot.image</entry>
104                <entry>.dfsl</entry>
105                <entry>darwinppc</entry>
106             </row>
107             <row>
108               <entry>LinuxPPC32</entry>
109                <entry>ppccl</entry>
110                <entry>PPCCL</entry>
111                <entry>ppc-boot</entry>
112                <entry>.pfsl</entry>
113                <entry>linuxppc</entry>
114             </row>
115             <row>
116               <entry>DarwinPPC64</entry>
117               <entry>dppccl64</entry>
118               <entry>dppccl64.image</entry>
119               <entry>ppc-boot64.image</entry>
120               <entry>.d64fsl</entry>
121               <entry>darwinppc64</entry>
122             </row>
123              <row>
124                <entry>LinuxPPC64</entry>
125                <entry>ppccl64</entry>
126                <entry>PPCCL64</entry>
127                <entry>ppc-boot64</entry>
128                <entry>.p64fsl</entry>
129                <entry>linuxppc64</entry>
130              </row>
131              <row>
132                <entry>LinuxX8664</entry>
133                <entry>lx86cl64</entry>
134                <entry>LX86CL64</entry>
135                <entry>x86-boot64</entry>
136                <entry>.lx64fsl</entry>
137                <entry>linuxx8664</entry>
138              </row>
139              <row>
140                <entry>DarwinX8664</entry>
141                <entry>dx86cl64</entry>
142                <entry>dx86cl64.image</entry>
143                <entry>x86-boot64.image</entry>
144                <entry>.dx64fsl</entry>
145                <entry>darwinx8664</entry>
146              </row>
147              <row>
148                <entry>FreeBSDX8664</entry>
149                <entry>fx86cl64</entry>
150                <entry>FX86CL64</entry>
151                <entry>fx86-boot64</entry>
152                <entry>.fx64fsl</entry>
153                <entry>freebsdx8664</entry>
154              </row>
155           </tbody>
156         </tgroup>
157       </table>
158      </sect2>
159    </sect1>
161    <sect1 id="Setting-Up-to-Build">
162      <title>Setting Up to Build</title>
163      <para>There are currently three versions of OpenMCL that you
164      might want to use (and therefore might want to build from
165      source):</para>
166      <itemizedlist>
167        <listitem><para>Version 1.0 - the more stable version</para></listitem>
168        <listitem><para>Version 1.1 - the more recent version, which
169        runs on more platforms (including x86-64 platforms) and
170        supports Unicode</para></listitem>
171        <listitem><para>Version 1.2 - supports (at least) all of the
172        features and platforms as 1.1, but is distributed and updated
173        differently</para></listitem>
174      </itemizedlist>
175      <para>All versions are available for download from the OpenMCL
176      website in the form of archives that contain everything you need
177      to work with OpenMCL, including the complete sources, a full
178      heap image, and the foreign-function interface database.</para>
179      <para>Version 1.0 archives are named
180      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-all-1.0.tar.gz</literal>,
181      where <replaceable>platform</replaceable> is either
182      <literal>darwinppc</literal>, <literal>darwinppc64</literal>, or
183      <literal>linuxppc</literal>.  Because version 1.0 is no longer
184      undergoing active development, you won't ever need to update
185      these sources.</para>
186      <para>Version 1.1 archives are named
187      <literal>openmcl-</literal><replaceable>platform</replaceable><literal>-snapshot-</literal><replaceable>yymmdd</replaceable><literal>.tar.gz</literal>,
188      where <replaceable>platform</replaceable> is either
189      <literal>darwinppc</literal>, <literal>darwinx8664</literal>,
190      <literal>linuxppc</literal>, <literal>linuxx8664</literal>, or
191      <literal>freebsdx8664</literal>, and where
192      <replaceable>yymmdd</replaceable> is the year, month, and day
193      the snapshot was released.</para>
194      <para>Because version 1.1 is undergoing active development,
195      there may be times when you want to get sources that are more
196      recent than the most recent snapshot and use them to build
197      yourself a new bleeding-edge OpenMCL.  In that case, you should
198      download and install the latest snapshot, and then update your
199      sources via CVS.  At that point you can rebuild and you'll have
200      the latest and greatest OpenMCL.  The snapshot has CVS
201      working-copy information in it, so all you need to do to update
202      is</para>
203      <programlisting>
204$ cd ccl
205$ cvs login             # password is "cvs"
206                        # this step only needs to be done once,
207                        # that'll store the trivially encrypted
208                        # password in ~/.cvspas
209$ cvs update
210      </programlisting>
211      <para>Unless you tell it to, cvs won't delete ("prune") empty
212      directories or create new ones when the repository changes.
213      It's generally a good habit to use</para>
214      <programlisting>
215$ cvs update -d -P      # create dirs as needed, prune empty ones
216      </programlisting>
217      <para>Version 1.2 archives follow naming conventions that are
218      similar to those used by 1.0 (though more platforms are supported.)
219      However, rather than containing CVS working-copy information, the
220      1.2 (and, presumably, later) archives contain metainformation used
221      by the Subversion (svn) source-code control system.</para>
222      <para>Subversion client programs are pre-installed on OSX 10.5 and
223      later and are typically either pre-installed or readily available
224      on Linux and FreeBSD platforms.  The <ulink url="">Subversion web page</ulink> contains links to subversion client programs
225      for many platforms; users of OSX versions 10.4 and earlier can also
226      install Subversion clients via Fink or MacPorts.</para>
228    </sect1>
230    <sect1 id="Building-Everything">
231      <title>Building Everything</title>
232      <para>Given that you now have everything you need, do the
233      following in a running OpenMCL to bring your Lisp system
234      completely up to date.</para>
235      <programlisting>
236? (ccl:rebuild-ccl :full t)
237      </programlisting>
238      <para>That call to the function <literal>rebuild-ccl</literal>
239      will perform the following steps:</para>
240      <itemizedlist>
241        <listitem>
242          <para>Deletes all fasl files and other object files in the
243          <literal>ccl</literal>directory tree</para>
244        </listitem>
245        <listitem>
246          <para>Runs an external process which does a
247          <literal>make</literal> in the currentplatform's kernel
248          build directory to create a new kernel</para>
249        </listitem>
250        <listitem>
251          <para>Does <literal>(compile-ccl t)</literal> in the running
252          lisp, to produce aset of fasl files from the &ldquo;higher
253          level&rdquo; lisp sources.</para>
254        </listitem>
255        <listitem>
256          <para>Does <literal>(xload-level-0 :force)</literal> in the
257          running lisp, to compile thelisp sources in the
258          &ldquo;ccl:level-0;&rdquo; directory into fasl files and
259          then createa bootstrapping image from those fasl
260          files.</para>
261        </listitem>
262        <listitem>
263          <para>Runs another external process, which causes the newly
264          compiled lispkernel to load the new bootstrapping image.
265          The bootsrtrapping image then loadsthe &ldquo;higher
266          level&rdquo; fasl files and a new copy of the platform's
267          full heap imageis then saved.</para>
268        </listitem>
269      </itemizedlist>
270      <para>If all goes well, it'll all happen without user
271      intervention and with some simple progress messages.  If
272      anything goes wrong during execution of either of the external
273      processes, the process output is displayed as part of a lisp
274      error message.</para>
275      <para><literal>rebuild-ccl</literal> is essentially just a short
276      cut for running all the individual steps involved in rebuilding
277      the system.  You can also execute these steps individually, as
278      described below.</para>
279    </sect1>
281    <sect1 id="Building-the-kernel">
282      <title>Building the kernel</title>
283      <para>The Lisp kernel is the executable which you run to use
284      Lisp.  It doesn't actually contain the entire Lisp
285      implementation; rather, it loads a heap image which contains the
286      specifics - the "library", as it might be called if this was a C
287      program.  The kernel also provides runtime support to the heap
288      image, such as garbage collection, memory allocation, exception
289      handling, and the OS interface.</para>
291      <para>The Lisp kernel file has different names on different
292      platforms. See FIXTHIS . On all platforms the lisp kernel sources reside
293      in <literal>ccl/lisp-kernel</literal>.</para>
295      <para>This section gives directions on how to rebuild the Lisp
296      kernel from its source code.  Most OpenMCL users will rarely
297      have to do this.  You probably will only need to do it if you are
298      attempting to port OpenMCL to a new architecture or extend or enhance
299      its kernel in some way.  As mentioned above, this step happens
300      automatically when you do
301      <programlisting>
302? (rebuild-ccl :full t)
303      </programlisting>
304      </para>
307      <sect2 id="Kernel-build-prerequisites">
308        <title>Kernel build prerequisites</title>
309        <para>The OpenMCL kernel can be bult with the following widely
310        available tools:</para>
311        <itemizedlist>
312          <listitem><para>cc or gcc- the GNU C compiler</para></listitem>
313          <listitem><para>ld - the GNU linker</para></listitem>
314          <listitem><para>m4 or gm4- the GNU m4 macro processor</para></listitem>
315          <listitem><para>as - the GNU assembler (version 2.10.1 or later)</para></listitem>
316          <listitem><para>make - either GNU make or, on FreeBSD, the default BSD make program</para></listitem>
317        </itemizedlist>
318        <para> In general, the more recent the versions of those
319        tools, the better; some versions of gcc 3.x on Linux have
320        difficulty compiling some of the kernel source code correctly
321        (so gcc 4.0 should be used, if possible.)  On OSX, the
322        versions of the tools distributed with XCode should work fine;
323        on Linux, the versions of the tools installed with the OS (or
324        available through its package management system) should work
325        fine if they're "recent enough".  On FreeBSD, the installed
326        version of the <literal>m4</literal> program doesn't support
327        some features that the kernel build process depends on; the
328        GNU version of the m4 macroprocessor (called
329        <literal>gm4</literal> on FreeBSD) should be installed
331        </para>
332      </sect2>
333      <sect2 id="kernel-build-command">
334        <title>Using "make" to build the lisp kernel</title>
335        <para>With those tools in place, do:
336        <programlisting>
337shell> cd ccl/lisp-kernel/<replaceable>PLATFORM</replaceable>
338shell> make
339        </programlisting>
340        </para>
341        <para>That'll assemble several assembly language source files,
342        compile several C source files, and link
343        ../../<replaceable>the kernel</replaceable>.
344        </para>
345      </sect2>
346    </sect1>
347    <sect1 id="Building-the-heap-image">
348      <title>Building the heap image</title>
349      <para>The initial heap image is loaded by the Lisp kernel, and
350      provides most all of the language implementation The heap image
351      captures the entire state of a running Lisp (except for external
352      resources, such as open files and TCP sockets).  After it is
353      loaded, the contents of the new Lisp process's memory are
354      exactly the same as those of the old Lisp process when the image
355      was created.</para>
356      <para>The heap image is how we get around the fact that we can't
357      run Lisp code until we have a working Lisp implementation, and
358      we can't make our Lisp implementation work until we can run Lisp
359      code.  Since the heap image already contains a fully-working
360      implementation, all we need to do is load it into memory and
361      start using it.</para>
362      <para>If you're building a new version of OpenMCL, you need to
363      build a new heap image.</para>
364      <para>(You might also wish to build a heap image if you have a
365      large program which it is very complicated or time-consuming to
366      load, so that you will be able to load it once, save an image,
367      and thenceforth never have to load it again. At any time, a heap
368      image capturing the entire memory state of a running Lisp can be
369      created by calling the function
370      <literal>ccl:save-application</literal>.)</para>
372      <sect2 id="Development-cycle">
373        <title>Development cycle</title>
374        <para>Creating a new OpenMCL full heap image consists of the
375        following steps:</para>
376        <orderedlist>
377          <listitem><para>Using your existing OpenMCL, create a
378          bootstrapping image</para></listitem>
379          <listitem><para>Using your existing OpenMCL, recompile your
380          updated OpenMCL sources</para></listitem>
381          <listitem><para>Invoke OpenMCL with the bootstrapping image
382          you just created (rather than with the existing full heap
383          image).</para></listitem>
384        </orderedlist>
385        <para>When you invoke OpenMCL with the bootstrapping image, it
386        will start up, load al of the OpenMCL fasl files, and save out
387        a new full heap image.  Voila.  You've created a new heap
388        image.</para>
389        <para>A few points worth noting:</para>
390        <itemizedlist>
391          <listitem>
392            <para>There's a circular dependency between the full heap
393            image and thebootstrapping image, in that each is used to
394            build the other.</para>
395          </listitem>
396          <listitem>
397            <para>There are some minor implementation
398            differences, but the environment in effect after the
399            bootstrapping image has loaded its fasl files is essentially
400            equivalent to the environment provided by the full heap
401            image; the latter loads a lot faster and is easier to
402            distribute, of course.</para>
403          </listitem>
404          <listitem>
405            <para>If the full heap image doesn't work (because
406            of an OScompatibilty problem or other bug), it's very likely
407            that thebootstrapping image will suffer the same
408            problems.</para>
409          </listitem>
410        </itemizedlist>
411        <para>Given a bootstrapping image and a set of up-to-date fasl
412        files, the development cycle usually involves editing lisp
413        sources (or updating those sources via cvs update),
414        recompiling modified files, and using the bootstrapping image
415        to produce a new heap image.</para>
416      </sect2>
418      <sect2 id="Generating-a-bootstrapping-image">
419        <title>Generating a bootstrapping image</title>
420        <para>The bootstrapping image isn't provided in OpenMCL
421        distributions. It can be built from the source code provided
422        in distributions (using a lisp image and kernel provided in
423        those distributions) using the procedure described
424        below.</para>
426        <para>The bootstrapping image is built by invoking a special
427        utility inside a running OpenMCL heap image to load files
428        contained in the <literal>ccl/level-0</literal> directory. The
429        bootstrapping image loads several dozen fasl files.  After
430        it's done so, it saves a heap image via
431        <literal>save-application</literal>. This process is called
432        "cross-dumping".</para>
434        <para>Given a source distribution, a lisp kernel, and aheap
435        image, one can produce a bootstapping image by first invoking
436        OpenMCL from the shell:</para>
437        <programlisting>
438shell&gt; openmcl
439Welcome to OpenMCL .... !
441        </programlisting>
442        <para>then calling <literal>ccl:xload-level-0</literal> at the
443        lisp prompt</para>
444        <programlisting>
445? (ccl:xload-level-0)
446        </programlisting>
447        <para>This will compile the lisp sources in the ccl/level-0
448        directory if they're newer than the corresponding fasl files
449        and will then load the resulting fasl files into a simulated
450        lisp heap contained inside data structures inside the running
451        lisp. That simulated heap image is then written to
452        disk.</para>
453        <para><literal>xload-level-0</literal> should be called
454        whenever your existing boot image is out-of-date with respect
455        to the source files in <literal>ccl:level-0;</literal>
456        :</para>
457        <programlisting>
458? (ccl:xload-level-0 :force)
460        <para>will force recompilation of the level-0 sources.</para>
461      </sect2>
463      <sect2 id="Generating-fasl-files">
464        <title>Generating fasl files</title>
465        <para> Calling:</para>
466        <programlisting>
467? (ccl:compile-ccl)
468        </programlisting>
469        <para>at the lisp prompt will compile any fasl files that are
470        out-of-date with respect to the corresponding lisp sources;
471        <literal>(ccl:compile-ccl t)</literal> will force
472        recompilation. <literal>ccl:compile-ccl</literal> will reload
473        newly-compiled versions of some files;
474        <literal>ccl:xcompile-ccl</literal> is analogous, but skips
475        this reloading step.</para>
476        <para>Unless there are bootstrapping considerations involved,
477        it usually doesn't matter whether these files reloaded after
478        they're recompiled.</para>
479        <para>Calling <literal>compile-ccl</literal> or
480        <literal>xcompile-ccl</literal> in an environment where fasl
481        files don't yet exist may produce warnings to that effect
482        whenever files are <literal>require</literal>d during
483        compilation; those warnings can be safely ignored. Depending
484        on the maturity of the OpenMCL release, calling
485        <literal>compile-ccl</literal> or
486        <literal>xcompile-ccl</literal> may also produce several
487        warnings about undefined functions, etc. They should be
488        cleaned up at some point.</para>
489      </sect2>
491      <sect2 id="Building-a-full-image-from-a-bootstrapping-image">
492        <title>Building a full image from a bootstrapping image</title>
493        <para>To build a full image from a bootstrapping image, just
494        invoke the kernel with the bootstrapping image is an
495        argument</para>
496        <programlisting>
497$ cd ccl                        # wherever your ccl directory is
499        </programlisting>
500        <para>Where <replaceable>KERNEL</replaceable> and
501        <replaceable>BOOT_IMAGE</replaceable> are the names of
502        the kernel and boot image appropriate to the platform you are
503        running on.  See FIXTHIS</para>
504        <para>That should load a few dozen fasl files (printing a
505        message as each file is loaded.) If all of these files
506        successfully load, the lisp will print a prompt. You should be
507        able to do essentially everything in that environment that you
508        can in the environment provided by a "real" heap image. If
509        you're confident that things loaded OK, you can save that
510        image.</para>
511        <programlisting>
512? (ccl:save-application "<replaceable>image_name</replaceable>") ; Overwiting the existing heap image
513        </programlisting>
514        <para>Where <replaceable>image_name</replaceable> is the name
515        of the full heap image for your platform. See FIXTHIS.</para>
516        <para>If things go wrong in the early stages of the loading
517        sequence, errors are often difficult to debug; until a fair
518        amount of code (CLOS, the CL condition system, streams, the
519        reader, the read-eval-print loop) is loaded, it's generally
520        not possible for the lisp to report an error.  Errors that
521        occur during these early stages ("the cold load") sometimes
522        cause the lisp kernel debugger (see ) to be invoked; it's
523        primitive, but can sometimes help one to get oriented.</para>
524      </sect2>
525    </sect1>
526  </chapter>
Note: See TracBrowser for help on using the repository browser.