wiki:WindowsNotes

Version 15 (modified by omah, 5 years ago) (diff)

Fix issues with updating and rebuilding Clozure CL on Windows.

Notes on installing and using Clozure CL on Windows

Installation

The files that make up a Clozure CL distribution for a "platform" (an OS/CPU architecture combination) are contained in a directory that's canonically called "ccl". (You can rename this if you want to, but it's more concise and convenient to refer to this as "the CCL directory" than to use a longer term.) The ccl directory for Windows contains:

  • wx86cl.exe - the kernel for the 32-bit implementation
  • wx86cl.image - a heap image for the 32-bit implementation
  • wx86cl64.exe, wx86cl64.image - the 64-bit versions of the above
  • a directory named "win32-headers", which contains database files that describe win32 foreign functions, constants, and data structures
  • a directory named "win64-headers", which contain 64-bit FFI info
  • several other directories containing sources, examples, and auxiliary files.

The canonical versions of these files are maintained in a "subversion repository"; subversion (aka "svn") is a revision control system (also known as a "source code control system.") We use svn both as a way of maintaining and tracking changes to the CCL sources and as a way of distributing CCL sources and binaries.

An svn repository can be accessed via an "http" URL and it's possible to access individual files by downloading them in a web browser, but it's generally not desirable to do so. It's preferable to use an svn client program to obtain a "working copy" of a directory from the repository. This will ensure that you get all of the distribution (sources, binaries, interfaces, and documentation) and will allow you to periodically update your working copy to obtain bug fixes and other changes.

So:

1) get a subversion (svn) client for Windows. There are several choices available; the "certified binaries" available at

 http://subversion.tigris.org/getting.html#windows

should work well.

If you're not comfortable with the Windows command-line, you may prefer TortoiseSVN (< http://tortoisesvn.tigris.org>), which adds svn-related entries to the context menu (right-click) in Windows Explorer. There are some things in the lisp build process that work better if a command-line version of svn is installed, but if you're shell-averse you might find it easiest to get started with something like TortoiseSVN and switch to a command-line client after you're more comfortable with the general idea of using svn.)

If you use Cygwin ( http://www.cygwin.com), you'll find that the Cygwin svn client generally works fine.

There are several other options available. Some GUI svn clients don't understand an svn feature called "svn:externals properties", and the CCL repository makes heavy use of this feature. The clients listed above do understand this.

2) read

 http://trac.clozure.com/openmcl/wiki/TrackingTheTrunk

3) check out the windows version of CCL by following the instructions on that page. The "PLATFORM" in question is called "windows", so the command to check out a fresh copy of the current release would be:

shell> svn co http://svn.clozure.com/publicsvn/openmcl/release/1.3/windows/ccl 

and the trunk (day-to-day development version) can be obtained from a slightly different URL:

shell> svn co http://svn.clozure.com/publicsvn/openmcl/trunk/windows/ccl

either of which would create and populate a directory called "ccl".

In tortoisesvn, right-clicking on the desktop will activate a popup menu with an "SVN Checkout ..." entry. Choose this entry; in the resulting dialog, enter the URL above as "URL of repository", select a directory where you want to keep the "ccl" directory (it'll be created if it doesn't exist when you start the checkout), leave other options with default values (checkout depth = fully recursive, "omit externals" unchecked, HEAD revision.) Click the "OK" button when ready; that'll confirm the creation of the CCL directory and then start downloading the distribution.)

If you later right-click on the CCL directory that was created above, the popup menu will contain an additional entry named "SVN Update" and a "TortoiseSVN" submenu which contains entries for many less-frequently-used options. "SVN Update" corresponds (not suprisingly) to the shell

shell> svn update

command described in the wiki page. However it's invoked, "svn update" checks to see if newer versions of files are available in the repository and downloads them into your working copy if so. The wiki page describes some of the issues related to "svn update" when local copies of files have been modified since they were last obtained from the repository.

As the wiki page also notes: we don't usually update the binaries every time a small change is made to the source; it's assumed that you can (usually) rebuild the lisp image yourself. The wiki page recommends doing that via:

? (rebuild-ccl :full t)

which would rebuild the lisp kernel, delete any FASL files, recompile all lisp sources, and build a new heap image. This is not so straightforward on Windows. In order to rebuild the kernel (wx86cl.exe or wx86cl64.exe) the GNU toolchain (Cygwin or MinGW), must be present on the path (see below.)

Also, because of Windows file-sharing issues it's not possible to overwrite an executable file (.exe) while it's running, so you must make copies of the wx86cl.exe and wx86cl.image files, and run the (rebuild-ccl :full t) command from the copy. This will update and rebuild the original.

Sometimes, especially on the development version of Clozure, the compilation process will hang. When this happens, open a Cygwin or MSYS shell, navigate to ccl/lisp-kernel/win32 and type 'make clean' (this will remove all C compiled files including the wx86cl.exe - so make sure you still have your copy .) Run the (rebuild-ccl :full t) command again in order to rebuild all.

You can do everything that the :FULL T option does besides rebuilding the kernel via:

? (rebuild-ccl :clean t)

Since it's likely that relatively few users will have the GNU toolchain for Windows installed, we're probably a little better about keeping the kernel up-to-date (e.g., we try to check in an updated wx86cl(64).exe soon after a significant change is made to its sources). As is noted on the wiki page above, rebuilding the lisp image from its sources should usually work, but there may be some warnings and continuable errors (when the value of a constant changes) during the build process.

REBUILD-CCL tries to obtain the current svn version (by using RUN-PROGRAM to run "svnversion" and capturing its output) and stores this version number in the value returned by LISP-IMPLEMENTATION-VERSION (it's also printed in the Welcome! banner.) If the "svnversion" program doesn't exist (or can't be found on the shell search path), the version string will just be the emtpy string (and there won't be anything too useful in LISP-IMPLEMENTATION-VERSION.)

Windows-specific issues

The 64-bit Windows port is probably more mature than the 32-bit Windows port. It only runs on 64-bit versions of Windows; it's been tested/used some on XP64 and quite a bit more extensively on 64-bit Vista. (Don't laugh: as of this writing, a large percentage of new Windows boxes sold at retail outlets - like BestBuy in the US - come with 64-bit versions of Vista installed. It's not clear if "a large percentage" is "a majority" yet, but it's much more common than it was even 6 months ago.)

The 32-bit port should run on 32-bit versions of XP and Vista. (For obscure reasons, the 32-bit port doesn't run on 64-bit OS versions at this point. We intend to address that, if only to avoid having to explain that it doesn't and why it doesn't.)

Windows pathnames can use either forward-slash or backward-slash characters as directory separators. As of the 1.3 release, CCL should handle namestrings which use either forward- or backward-slashes; some prereleases and release-candidates generally had difficulty with backslashes.

Windows namestrings generally contain a "drive letter" preceded by a colon; the drive letter of a pathname derived from such a namestring is accessible via PATHNAME-DEVICE.

The value returned by USER-HOMEDIR-PATHNAME depends on OS version. The "home" logical host has its translations set up so that "home:" references this directory, and the lisp will try to load "home:ccl-init" (as a lisp or fasl file) on startup, just as on other platforms.

Windows example

There's a very simple (but heavily commented) example program in "ccl:examples;mswin.lisp". Doing:

? (require "MSWIN")

and then doing

? (cl-user::make-simple-ms-window)

will create a (very) simple window and process events related to that window until its close box is pressed. You might prefer to have the window created (and the event loop run in) a separate thread:

? (process-run-function "window" #'cl-user::make-simple-ms-window)

The thrill of seeing a window appear onscreen will likely wear off pretty quickly; a window that contains interesting content or which has interesting behavior might be less boring.

Building the lisp kernel on Windows

The Windows kernel Makefiles (in ccl/lisp-kernel/win32 and ccl/lisp-kernel/win64) assume that the Cygwin tools ( http://www.cygwin.com) are installed. (It's likely that small changes to the Makefiles would allow the use of the MinGW ( http://www.mingw.org) tools instead.)

A full discussion of Cygwin installation and configuration issues is *way* beyond the scope of this document. Fortunately, the net is full of helpful and accurate documents that discuss these issues in great detail. For the sake of argument, let's assume that Cygwin is installed in C:\cygwin (which is the default) and that it's configured to use Unix-style line-termination.

For both the win32 and win64 platforms, it's necessary to install the following Cygwin packages (if you notice any dependencies not listed here, please edit this page or let me (gb) know.)

  • make (Unix "make" program)
  • m4 (m4 macro processor)

You'll probably find life easier if you also install standard utilities ('coreutils', 'fileutils', 'tar', ...) but these things aren't used by the build process.

To build the win32 kernel, you'd additionally need:

  • gcc-core (32-bit C compiler)
  • gcc-mingw (mingw integration)
  • gcc-mingw-core
  • mingw-runtime (static C runtime library)
  • w32-api (win32 headers)
  • binutils (assembler/linker/object file utilities.)

64-bit toolchains for the Cygwin/MinGW environments are available from the Mingw-w64 project ( http://sourceforge.net/projects/mingw-w64/). In the "Downloads" section of that site, find a "complete toolchain snapshot" for Cygwin and download the compressed tar archive (it's largish - probably > 70MB). In a Cygwin shell, extract the the tar archive into /usr :

shell> tar xvf mingw-w64-bin-i686-cygwin-VERSION-YYYYMMDD.tar.bz2 -C /usr

This archive contains everything that you need (64-bit toolchain, libraries, headers) to build the lisp kernel for win64. (You still need the 'make' and 'm4' packages.)

Actually building a Windows kernel

Ensure that no copies of the lisp are running. (Windows doesn't like to overwrite an executable file while it's in use.)

From a Cygwin shell, cd to .../ccl/lisp-kernel/win32 (or win64) and do:

shell> make

If all of the tools/headers/libraries are installed, that should create a working kernel executable in the ccl directory.