wiki:UpdatingFromSource

Version 11 (modified by rme, 5 years ago) (diff)

--

Updating from Subversion and rebuilding Clozure CL from source

Getting Clozure CL via Subversion (as described on the main page) creates a "working copy" of the CCL source code and binaries on your local system. It is also possible to get an initial working copy (including the svn metainformation in .svn directories) from a release tarball. These are periodically made available from  ftp://ftp.clozure.com/pub/release/.

With a working copy in hand, you can now use Subversion to get updates.

  1. In a shell, enter the "ccl" directory that contains the working copy of the sources.
    shell> cd ccl
    
  1. Tell svn to update the sources, creating new directories and removing empty ones as needed.
    shell> svn update
    

This will try to reconcile changes made in the svn repository with the local, working copy. If you've made changes to files in the working copy and changes have been made in the repository, svn will attempt to merge those changes into an updated copy of the local file. It's often able to do so reasonably well (for text files); sometimes, the repository changes and local changes can't be resolved automatically and the affected file is said to be "in conflict." svn has no good way to even try to merge changes to binary files (like the CCL kernel and heap image).

As "svn update" reconciles changes to the repository with any changes made to the working copy, it prints a line consisting of a one-letter code, some whitespace, and the pathname of the affected item. These codes include:

A path The file or directory at path was added to the repository and has been added to the working copy
D path The file or directory at path was deleted from the repository and has been deleted from the working copy
U file The file at file was changed in the repository and the local copy has been updated
G file The file at file was changed in both the repository and working copy, and the local copy contains the merged content
C file The file at file was changed in both the repository and working copy, and svn was unable to resolve conflicts

Binary files that have been locally modified (a rebuilt lisp kernel or heap image counts as a modification) generally can't be merged with newer versions from the repository. The lisp kernel and heap image generally don't change often in a nominally stable "release", but when they do the changes are often significant. Therefore, if you notice that the lisp kernel or heap image have been updated, and the line is marked with a "C", the correct response is to revert the file:

shell> svn revert some-file

will effectively discard any local changes to some-file and replace it with the repository version.

  1. Once svn has finished creating local copies of files that have changed on the svn server since the last time you did this (or since the tarballs were created), use the currently installed Clozure CL to compile the updated sources and build a new kernel and heap image:
shell> ccl  # or "ccl64", or m-x slime, or whatever you do to start Clozure CL
Welcome to whatever version this is!
? (ccl:rebuild-ccl :full t)

(Windows users: use (ccl:rebuild-ccl :clean t). See WindowsNotes for more information.)

That'll recompile the lisp kernel and all lisp sources used to build a heap image and build that image. Once that completes (usually takes 1 minute +/- 30 seconds) there should be a new heap image and new lisp kernel. There may be compiler warnings during compilation, usually related to calls to functions that will be defined when the process finishes but which are not yet defined during compilation (or which take a different number of arguments at compile time than they will when the image is built, etc.) Once in a while, some constants get redefined and a continuable error is signaled; continue from the CERROR to use the new value.

After a while, this (simple and simple-minded) bootstrapping process will cease to work: the newer source code will diverge from the code used to build the lisp image enough that it'll have to be bootstrapped in a more complicated (and usually totally ad-hoc) way. When this happens, new heap images and/or lisp kernels are checked in to svn (as soon as possible after the affected sources are checked in.) As noted above, this shouldn't happen too often in the "release" tree (where most changes are simple, relatively isolated bug fixes.) If it ever does happen, it's worth being aware of the fact that this may result in those files being marked as being in conflict and that "svn revert" can be used to resolve that conflict.