Index: /trunk/source/doc/src/openmcl-documentation.xml =================================================================== --- /trunk/source/doc/src/openmcl-documentation.xml (revision 8516) +++ /trunk/source/doc/src/openmcl-documentation.xml (revision 8516) @@ -0,0 +1,11371 @@ + +&rest"> +&key"> +&optional"> +&body"> +&aux"> +&allow-other-keys"> +]> + + + OpenMCL Documentation + + + Obtaining, Installing, and Running OpenMCL + + Releases and System Requirements + + There are three active versions of OpenMCL. Version 1.0 was + a stable release (released in late 2005); it is no longer under + active development. Version 1.1 has been under active development + since shortly after 1.0 was released; it's been distributed as a + series of development "snapshots" and CVS updates. 1.1 snapshots + have introduced support for x86-64 platforms, internal use of + Unicode, and many other features, but have presented something of a + moving target. Version 1.2 (being released in early 2008) is + intended to both a more stable and predictable release schedule and + to make it a bit easier for users who wish to track the "bleeding + edge" of development to do so. + + Version 1.0 is available for three platform configurations: + + + Linux on PowerPC (32-bit implementation) + + + Mac OS X on PowerPC (32-bit implementation) + + + Mac OS X on PowerPC (64-bit implementation) + + + + Versions 1.1 and 1.2 are available for five platform + configurations: + + + Linux on PowerPC (32-bit and 64-bit implementations) + + + >Mac OS X on PowerPC (32-bit and 64-bit implementations) + + + Linux on X86-64 (64-bit implementation) + + + Mac OS X on X86-64 (64-bit implementation) + + FreeBSD on X86-64 (64-bit implementation) + + + A 64-bit version of OpenMCL requires a 64-bit processor + (obviously) running a 64-bit OS variant. + + There are ongoing efforts to port OpenMCL to the Windows + operating system and 32-bit x86 processors. + + + Additional platform-specific information is given in the + following subsections. + + LinuxPPC + + OpenMCL versions 1.0 and later run under relatively + recent versions of LinuxPPC. All versions of OpenMCL require + version 2.2.13 (or later) of the Linux kernel and version + 2.1.3 (or later) of the GNU C library (glibc) at a bare + minimum. + + + LinuxX8664 + + Version 1.1 and later of OpenMCL runs on relatively + recent Linux distributions for the x86-64 architecture. It + requires a Linux with Thread Local Storage support in the + toolchain and standard libraries, and the New Posix Thread + Library (NPTL). Fortunately, these features seem to be + present in all current Linux distributions for x86-64, though + there may be some problems with early Linux distributions for + x86-64. (Some GCC versions older than 4.0 on Linux have been + known to have problems compiling some of the C code in the + kernel, some very old Linux distributions don't follow the + current ABI standards wrt segment register usage, some early + Linux kernels for x86-64 had problems mapping large regions of + the address space ... it's difficult to enumerate exactly what + versions of what Linux distributions have what problems. A + rule of thumb is that - since much of the development of + OpenMCL for x86-64 took place in that time frame - Linux + distributions released earlier than early 2006 may have + problems running OpenMCL.) + + + FreeBSD-amd64 + Versions 1.1 and later of OpenMCL runs on FreeBSD on + x86-64 (FreeBSD releases generally call the platform "and64") + OpenMCL should run under FreeBSD 6.0 or later; as of this + writing, FreeBSD 7.0 is about to be released and it may be + necessary for FreeBSD 7 users to install the "compat6x" + package in order to use a version of OpenMCL built on FreeBSD + 6.x under FreeBSD 7. . + + + DarwinPPC-MacOS-X + + OpenMCL 1.0 runs on MacOS X versions 10.2, 10.3 and + 10.4 on the PowerPC. + + Current development on version 1.1 and later takes place + under OS X versions 10.4 and 10.5 and requires at least + version 10.3.9 + + The 64-bit DarwinPPC version of OpenMCL requires + functionality introduced in OSX 10.4 (namely, the ability to + run 64-bit binaries). It also, obviously, requires a G5 + processor. + + OpenMCL hasn't been tested under Darwin proper, but + OpenMCL doesn't intentionally use any MacOS X features beyond + the Darwin subset and therefore it seems likely that OpenMCL + would run on PPC Darwin versions that correspond to recent OSX + versions. + + + Darwinx8664-MacOS-X + Versions 1.1 and later of OpenMCL runs on 64-bit + DarwinX86 (MacOS on Intel). + + OpenMCL Darwinx8664/MacOS X requires a 64-bit processor. + All Macintoshes currently sold by Apple (as of early 2008) and + all Macintoshes introduced by Apple since August 2006 have + such processors. However, the original MacBooks, MacBook Pros + and Intel iMacs (models introduced in early 2006) used 32-bit + Core Duo processors, and so OpenMCL will not (yet) run on + them. + + + + + Installation + Installing OpenMCL consists of + + + Downloading an appropriate distribution of OpenMCL; + + + If necessary, extracting the archive somewhere on your + computer; + + + Configuring a shell script so OpenMCL can locate + necessary library files and other auxilliary files. + + + + Downloading and untarring OpenMCL binary source + releases OpenMCL releases and snapshots are + distributed as tarballs (compressed tar archives). + + Tarballs of version 1.0 for supported platforms are + available from the download page of the OpenMCL website. + + Tarballs of the latest development snapshots of version + 1.1, along with release notes, are available from the testing + directory on Clozure.com. + Both the release and snapshot archives contain a directory + named ccl, which in turn contains a number of + files and subdirectories. The ccl directory + can reside anywhere in the filesystem, assuming appropriate + permissions. If you wanted the ccl directory + to reside in ``~/openmcl/ccl'' and the + directory ``~/openmcl/'' already existed, you + could do anything equivalent to: + +shell> cd ~/openmcl +shell> tar xvzf path-to-downloaded-openmcl-archive.tar.gz + + Important Note for Macintosh Users: + Double-clicking the archive in the Finder may work, but it also + may have unintended side-effects. In some versions of the Mac + OS double-clicking an archive will invoke Stuffit, which may try + to replace linefeeds with carriage returns as it extracts + files. Also, tar can be used to merge the archive contents into + an existing ccl directory, whereas + double-clicking in the Finder will create a new directory named + ccl 2 (or ccl 3, or...) + Bottom line is that you're better off using tar from the + shell. + + Once the ccl directory is installed, + it's necessary to install and configure a shell script + distributed with OpenMCL before using it. + + + The-openmcl-Shell-Script" + OpenMCL needs to be able to find the + ccl directory in order to support features + such as require and + provide, access to foreign interface + information (see ) and the lisp build process (see + ). Specifically, it needs to set up logical pathname + translations for the "ccl:" logical host. If + this logical host isn't defined (or isn't defined correctly), + some things might work, some things might not... and it'll + generally be hard to invoke and use OpenMCL productively. + + OpenMCL uses the value of the environment variable + CCL_DEFAULT_DIRECTORY to determine the + filesystem location of the ccl directory; + the openmcl shell script is intended to provide a way to + invoke OpenMCL with that environment variable set + correctly. + There are two versions of the shell script: + "ccl/scripts/openmcl" is used to invoke + 32-bit implementations of OpenMCL and + "ccl/scripts/openmcl64" is used to invoke + 64-bit implementations. + To use the script: + + + Edit the definition of + CCL_DEFAULT_DIRECTORY near the + beginning of theshell script so that it refers to your ccl + directory. Alternately,set + CCL_DEFAULT_DIRECTORY in your .cshrc, + .tcshrc, .bashrc,.bash_profile, .MacOSX/environment.plist, + or wherever you usually set environment variables. If + there is an existing definition of thevariable, the + openmcl script will not override it.The shell script sets + a local variable (OPENMCL_KERNEL) to + the standard name of the OpenMCL kernel approprate for the + platform (asdetermined by 'uname -s'.) You might prefer to + set this variable manually in the shell script + + + + Ensure that the shell script is executable, for + example:$ chmod +x + ~/openmcl/ccl/scripts/openmcl64This command + grants execute permission to the named script. If you + areusing a 32-bit platform, substitute "openmcl" in place + of "openmcl64". + + The above command won't work if you are not the + owner of the installed copy of OpenMCL. In that case, + you can use the "sudo" command like this: + $ sudo chmod +x + ~/openmcl/ccl/scripts/openmcl64 + Give your password when prompted. + If the "sudo" command doesn't work, then you are + not an administrator on the system you're using, and you + don't have the appropriate "sudo" permissions. In that + case you'll need to get help from the system's + administrator. + + + + Install the shell script somewhere on your shell's + search path + + + + Once this is done, it should be possible to invoke + OpenMCL by typing openmcl at a shell + prompt: + +> openmcl [args ...] +Welcome to OpenMCL Version whatever (DarwinPPC32)! +? + + The openmcl shell script will pass all of its arguments + to the OpenMCL kernel. See for + more information about "args". When invoked this way, the + lisp should be able to initialize the + "ccl:" logical host so that its + translations refer to the "ccl" + directory. To test this, you can call + probe-file in OpenMCL's read-eval-print + loop: + +? (probe-file "ccl:level-1;level-1.lisp") ;will return the physical pathname of the file +#P"/Users/alms/my_lisp_stuff/ccl/level-1/level-1.lisp" + + + + + Personal Customization with the Init File + By default OpenMCL will try to load the file + "home:openmcl-init.lisp" or the compiled + + "home:openmcl-init.fasl" upon starting + up. It does this by executing (load + "home:openmcl-init"). If it is unable to load the + file (for example because it does not exist) no action is + taken; an error is not signalled. + The "home:" prefix to the filename is + a Common Lisp logical host, which OpenMCL initializes to refer + to your home directory, so this looks for either of the files + ~/openmcl-init.lisp or + ~/openmcl-init.fasl. + Since the init file is loaded the same way as normal + Lisp code is, you can put anything you want in it. For + example, you can change the working directory, and load + packages which you use frequently. + To suppress the loading of this init-file, invoke OpenMCL with the +--no-init option. + + + + Some (hopefully) useful options + The exact set of command-line arguments accepted by + OpenMCL may vary slightly from release to release; + openmcl --help will provide a definitive + (if somewhat terse) summary of the options accepted by the + current implementation and then exit. Some of those options + are described below. + + + -S (or --stack-size). Specify the size of the initial process + stack. + + + + -b (or --batch). Execute in "batch mode". End-of-file + from *STANDARD-INPUT* will cause OpenMCL to exit, as will attempts to + enter a break loop. + + + + -n (or --no-init). If this option is given, the init file + is not loaded. This is useful if OpenMCL is being invoked by a + shell script which should not be affected by whatever + customizations a user might have in place. + + + + + -e <form> (or --eval <form>). An expression is + read (via READ-FROM-STRING from the string <form> and + evaluated. If <form> contains shell metacharacters, it may be + necessary to escape or quote them to prevent the shell from + interpreting them. + + + + -l >path> (or --load <path>). Executes (load + "<path>"). + + + + The --load and + --eval options can each be provided + multiple times. They're executed in the order specified on + the command line, after the init file (if there is one) is + loaded and before the toplevel read-eval-print loop is + entered. + + + + + Using OpenMCL with GNU Emacs and SLIME + A very common way to use OpenMCL is to run it within the + GNU Emacs editor, using a Lisp interface called SLIME ("Superior + Lisp Interaction Mode for Emacs"). SLIME is an Emacs package + designed to provide good support within Emacs for any of several + Common Lisp implementations; one of the supported + implementations is OpenMCL. This page describes how you can + download SLIME and set it up to work with your OpenMCL + installation. + Why use SLIME? With SLIME, you can do the following things from within +an Emacs editing session: + + run and control Lisp + evaluate, compile, and load files or expressions + macroexpand expressions + fetch documentation and source code for Lisp symbols + autocomplete symbols and package names + cross-reference function calls + examine stack traces and debug errors + + + There is an excellent SLIME tutorial video available at +Common-Lisp.net. For +complete information about SLIME, see the +SLIME home page. + + + Assumptions and Requirements + In order to simplify these instructions, we'll make + several assumptions about your system. Specifically, we + assume: + + + You have a working installation of GNU Emacs. If you + don't have a working copy of GNU Emacs, see the web page on + obtaining Emacs. If you prefer to use XEmacs instead of + GNU Emacs,these instructions should still work; SLIME + supports XEmacs Version21. Mac OS X includes an Emacs + installation. If you want to look into different versions, + you can check out theEmacsWiki, whichmaintains a + page,EmacsForMacOS,that provides much more information + about using Emacs on the Mac. + + A popular version of Emacs among Mac users is + Aquamacs. This application is a version of GNU Emacs + with a number of customizations meant to make it behave + more like a standard Maciontosh application, with + windows, a menubar, etc. Aquamacs includes SLIME; if + you like Aquamacs then you can use SLIME right away, + without getting and installing it separately. You just + need to tell SLIME where to find your installation of + OpenMCL. (See FIXTHIS.) + + + + + You have a working copy of OpenMCL, installed in + "~/openmcl/ccl"If you prefer to install + OpenMCL in some directory other + than"~/openmcl/ccl" then these + instructions still work, but you must remember to use your + path to your ccl directory instead of theone that we give + here. + + + You install emacs add-ons in the folder + "~/emacs/site/"If this directory + doesn't exist on your system, you can just create it.If + you prefer to install Emacs add-ons in some place other + than"~/emacs/site/" then you must + remember to use your path toEmacs add-ons in place of + ours. + + + + + + Getting SLIME + + You can get SLIME from the SLIME Home Page. Stable + releases and CVS snapshots are available as archive files, or + you can follow the instructions on the SLIME Home Page to + check out the latest version from their CVS repository. + + It's worth noting that stable SLIME releases happen very + seldom, but the SLIME developers often make changes and + improvements that are available through CVS updates. If you + asked the SLIM developers, they would most likely recommend + that you get SLIME from their CVS repository and update it + frequently. + + Whether you get it from CVS, or download and unpack one + of the available archives, you should end up with a folder + named "slime" that contains the SLIME distribution. + + + Installing SLIME + + Once you have the "slime" folder described in the + previous section,installation is a simple matter of copying + the folder to the proper place. You can drag it into the + "~/emacs/site/" folder, or you can use a terminal command to + copy it there. For example, assuming your working directory + contains the unpacked "slime" folder: $ + cp -R slime ~/emacs/site/ That's all it + takes. + + + + + Telling Emacs About SLIME + Once SLIME and OpenMCL are installed, you just need to + add a line to your "~/.emacs" file that tells SLIME where to + find the script that runs OpenMCL: + (setq inferior-lisp-program "~/openmcl/ccl/scripts/openmcl64") + or + (setq inferior-lisp-program "~/openmcl/ccl/scripts/openmcl") + + Aquamacs users should add this line to the file "~/Library/Preferences/Aquamacs Emacs/Preferences.el". + + + + + Running OpenMCL with SLIME + Once the preparations in the previous section are + complete, exit Emacs and restart it, to ensure that it reads + the changes you made in your ".emacs" file (alternatively, you + could tell Emacs to reload the ".emacs" file). If all went + well, you should now be ready to run OpenMCL using + SLIME. + To run OpenMCL, execute the command "M-x slime". SLIME + should start an OpenMCL session in a new buffer. (If you are + unfamiliar with the Emacs notation "M-x command", see the GNU + Emacs FAQ; specifically, take a look at questions 1, 2, and + 128.) + + + + What if a New Version of OpenMCL Breaks SLIME? + Sometimes you'll get a new version of OpenMCL, set up + Emacs to use it with SLIME, and SLIME will fail. Most likely + what has happened is that the new version of OpenMCL has a + change in the output files produced by the compiler (OpenMCL + developers will say "the fasl version has changed." fasl + stands for “fast load” aka compiled files). This + problem is easy to fix: just delete the existing SLIME fasl + files. The next time you launch Emacs and start SLIME, it will + automatically recompile the Lisp files, and that should fix + the problem. + SLIME's load process stores its fasl files in a hidden + folder inside your home folder. The path is + ~/.slime/fasl + You can use a shell command to remove the fasl files, or + remove them using your system's file browser. + Note for Macintosh + Users: The leading "." character in the ".slime" + folder's name prevents the Finder from showing this folder to + you. If you use the "Go To Folder" menu item in the Finder's + "Go" menu, you can type in "~/.slime" and the Finder will show + it to you. You can then drag the "fasl" folder to the trash. + + + + + Known Bugs + SLIME has not been updated to account for recent changes + made in OpenMCL to support x86-64 processors. You may run into + bugs running on those platforms. + The SLIME backtrace sometimes shows incorrect information. + return-from-frame and + apply-in-frame do not work reliably. (If + they work at all, it's pure luck.) + Some versions of Emacs on the Macintosh may have trouble + finding the shell script that runs OpenMCL unless you specify + a full path to it. See the above section "Telling Emacs About + SLIME" to learn how to specify the path to the shell + script. + For more help with OpenMCL on Mac OS X, consult the + OpenMCL mailing lists. + + + + + Example Programs + Beginning with release 0.9, a number (ok, a + small number, at least initially) of + example programs are distributed in the "ccl:examples;" + directory of the source distribution. See the + README-OPENMCL-EXAMPLES text file in that directory for + information about prerequisites and usage. + Some of the example programs are derived from C examples + in textbooks, etc.; in those cases, the original author and work + are cited in the source code. + It may be a stretch to imply that the examples have a + great deal of value as "intellectual property", but I might as + well say this: Unless the original author or contributor claims + other rights, you're free to incorporate any of this example + code or derivative thereof in any of you're own works without + restriction. In doing so, you agree that the code was provided + "as is", and that no other party is legally or otherwise + responsible for any consequences of your decision to use + it. + If you've developed OpenMCL examples that you'd like to + see added to the distribution, please send mail to let me + know. Any such contributions would be welcome and appreciated + (as would bug fixes and improvements to the existing + examples.) + + + + Building OpenMCL from its Source Code + + OpenMCL, like many other Lisp implementations, consists of a + kernel and a heap image. The kernel is an ordinary C program, and + is built with a C compiler. It provides very basic and + fundamental facilities, such as memory management, garbage + collection, and bootstrapping. All the higher-level features are + written in Lisp, and compiled into the heap image. Both parts are + needed to have a working Lisp implementation; neither the kernel + nor the heap image can stand on their own. + + You may already know that, when you have a C compiler which + is written in C, to build the compiler, you need to already have a + C compiler available. The heap image includes a Lisp compiler, + which is written in Lisp. Therefore, you need a working Lisp + compiler in order to build the heap image! + + Where will you get a working Lisp compiler? No worries; you + can use a precompiled copy of a (slightly older and compatible) + version of OpenMCL. For help, read on. + + In principle it would be possible to use another Lisp as the + host compiler, rather than an old OpenMCL; this would be a + challenging and experimental way to build, and is not described + here. + + >building definitions + The following terms are used in subsequent sections; it + may be helpful to refer to these definitions. + + fasl files are + the object files produced bycompile-file. + fasl files store the machine codeassociated with function + definitions and the external representationof other lisp objects + in a compact, machine-readable form. fasl is short for + “FASt + Loading”.OpenMCL uses different + pathname types (extensions) to name faslfiles on different + platforms; see + + The lisp kernel is + a C program (with a fair amount ofplatform-specific assembly + language code as well.) Its basic job isto map a lisp heap + image into memory, transfer control to somecompiled lisp code + that the image contains, handle any exceptionsthat occur during + the execution of that lisp code, and provide variousother forms + of runtime support for that code.OpenMCL uses different + filenames to name the lisp kernel fileson different platforms; + see FIXTHIS. + + A heap image is + a file that can be quickly mapped into aprocess's address space. + Conceptually, it's not too different from anexecutable file or + shared library in the OS's native format (ELF orMach-O/dyld + format); for historical reasons, OpenMCL's own heap images are in + their own (fairly simple) format.The term full heap + image refers to a heap image file thatcontains all of + the code and data that comprise OpenMCL.OpenMCL uses different + filenames to name the standard full heapimage files on different + platforms; see FIXTHIS . + + A bootstrapping + image is a minimal heap image used in + the process of building OpenMCL itself. The bootstrapping image + containsjust enough code to load the rest of OpenMCL from fasl + files. It mayhelp to think of the bootstrapping image as the + egg and the full heapimage as the chicken...OpenMCL uses + different filenames to name the standardbootstrapping image + files on different platforms; see FIXTHIS . + + Each supported platform (and possibly a few + as-yet-unsupported ones) has a uniquely named subdirectory of + ccl/lisp-kernel/; each such + kernel build directory + contains a Makefile and may contain some auxiliary files (linker + scripts, etc.) that are used to build the lispkernel on a + particular platform.The platform-specific name of the kernel + build directory is described in FIXTHIS. + + + Platform-specific filename conventions + + Platform-specific filename conventions + + + + Platform + kernel + full-image + boot-image + fasl extension + kernel-build directory + + + + + DarwinPPC32 + dppccl + dppccl.image + ppc-boot.image + .dfsl + darwinppc + + + LinuxPPC32 + ppccl + PPCCL + ppc-boot + .pfsl + linuxppc + + + DarwinPPC64 + dppccl64 + dppccl64.image + ppc-boot64.image + .d64fsl + darwinppc64 + + + LinuxPPC64 + ppccl64 + PPCCL64 + ppc-boot64 + .p64fsl + linuxppc64 + + + LinuxX8664 + lx86cl64 + LX86CL64 + x86-boot64 + .lx64fsl + linuxx8664 + + + DarwinX8664 + dx86cl64 + dx86cl64.image + x86-boot64.image + .dx64fsl + darwinx8664 + + + FreeBSDX8664 + fx86cl64 + FX86CL64 + fx86-boot64 + .fx64fsl + freebsdx8664 + + + +
+ + + + + Setting Up to Build + There are currently three versions of OpenMCL that you + might want to use (and therefore might want to build from + source): + + Version 1.0 - the more stable version + Version 1.1 - the more recent version, which + runs on more platforms (including x86-64 platforms) and + supports Unicode + Version 1.2 - supports (at least) all of the + features and platforms as 1.1, but is distributed and updated + differently + + All versions are available for download from the OpenMCL + website in the form of archives that contain everything you need + to work with OpenMCL, including the complete sources, a full + heap image, and the foreign-function interface database. + Version 1.0 archives are named + openmcl-platform-all-1.0.tar.gz, + where platform is either + darwinppc, darwinppc64, or + linuxppc. Because version 1.0 is no longer + undergoing active development, you won't ever need to update + these sources. + Version 1.1 archives are named + openmcl-platform-snapshot-yymmdd.tar.gz, + where platform is either + darwinppc, darwinx8664, + linuxppc, linuxx8664, or + freebsdx8664, and where + yymmdd is the year, month, and day + the snapshot was released. + Because version 1.1 is undergoing active development, + there may be times when you want to get sources that are more + recent than the most recent snapshot and use them to build + yourself a new bleeding-edge OpenMCL. In that case, you should + download and install the latest snapshot, and then update your + sources via CVS. At that point you can rebuild and you'll have + the latest and greatest OpenMCL. The snapshot has CVS + working-copy information in it, so all you need to do to update + is + +$ cd ccl +$ cvs login # password is "cvs" + # this step only needs to be done once, + # that'll store the trivially encrypted + # password in ~/.cvspas +$ cvs update + + Unless you tell it to, cvs won't delete ("prune") empty + directories or create new ones when the repository changes. + It's generally a good habit to use + +$ cvs update -d -P # create dirs as needed, prune empty ones + + Version 1.2 archives follow naming conventions that are + similar to those used by 1.0 (though more platforms are supported.) + However, rather than containing CVS working-copy information, the + 1.2 (and, presumably, later) archives contain metainformation used + by the Subversion (svn) source-code control system. + Subversion client programs are pre-installed on OSX 10.5 and + later and are typically either pre-installed or readily available + on Linux and FreeBSD platforms. The Subversion web page contains links to subversion client programs + for many platforms; users of OSX versions 10.4 and earlier can also + install Subversion clients via Fink or MacPorts. + + + + + Building Everything + Given that you now have everything you need, do the + following in a running OpenMCL to bring your Lisp system + completely up to date. + +? (ccl:rebuild-ccl :full t) + + That call to the function rebuild-ccl + will perform the following steps: + + + Deletes all fasl files and other object files in the + ccldirectory tree + + + Runs an external process which does a + make in the currentplatform's kernel + build directory to create a new kernel + + + Does (compile-ccl t) in the running + lisp, to produce aset of fasl files from the “higher + level” lisp sources. + + + Does (xload-level-0 :force) in the + running lisp, to compile thelisp sources in the + “ccl:level-0;” directory into fasl files and + then createa bootstrapping image from those fasl + files. + + + Runs another external process, which causes the newly + compiled lispkernel to load the new bootstrapping image. + The bootsrtrapping image then loadsthe “higher + level” fasl files and a new copy of the platform's + full heap imageis then saved. + + + If all goes well, it'll all happen without user + intervention and with some simple progress messages. If + anything goes wrong during execution of either of the external + processes, the process output is displayed as part of a lisp + error message. + rebuild-ccl is essentially just a short + cut for running all the individual steps involved in rebuilding + the system. You can also execute these steps individually, as + described below. + + + + Building the kernel + The Lisp kernel is the executable which you run to use + Lisp. It doesn't actually contain the entire Lisp + implementation; rather, it loads a heap image which contains the + specifics - the "library", as it might be called if this was a C + program. The kernel also provides runtime support to the heap + image, such as garbage collection, memory allocation, exception + handling, and the OS interface. + + The Lisp kernel file has different names on different + platforms. See FIXTHIS . On all platforms the lisp kernel sources reside + in ccl/lisp-kernel. + + This section gives directions on how to rebuild the Lisp + kernel from its source code. Most OpenMCL users will rarely + have to do this. You probably will only need to do it if you are + attempting to port OpenMCL to a new architecture or extend or enhance + its kernel in some way. As mentioned above, this step happens + automatically when you do + +? (rebuild-ccl :full t) + + + + + + Kernel build prerequisites + The OpenMCL kernel can be bult with the following widely + available tools: + + cc or gcc- the GNU C compiler + ld - the GNU linker + m4 or gm4- the GNU m4 macro processor + as - the GNU assembler (version 2.10.1 or later) + make - either GNU make or, on FreeBSD, the default BSD make program + + In general, the more recent the versions of those + tools, the better; some versions of gcc 3.x on Linux have + difficulty compiling some of the kernel source code correctly + (so gcc 4.0 should be used, if possible.) On OSX, the + versions of the tools distributed with XCode should work fine; + on Linux, the versions of the tools installed with the OS (or + available through its package management system) should work + fine if they're "recent enough". On FreeBSD, the installed + version of the m4 program doesn't support + some features that the kernel build process depends on; the + GNU version of the m4 macroprocessor (called + gm4 on FreeBSD) should be installed + + + + + Using "make" to build the lisp kernel + With those tools in place, do: + +shell> cd ccl/lisp-kernel/PLATFORM +shell> make + + + That'll assemble several assembly language source files, + compile several C source files, and link + ../../the kernel. + + + + + Building the heap image + The initial heap image is loaded by the Lisp kernel, and + provides most all of the language implementation The heap image + captures the entire state of a running Lisp (except for external + resources, such as open files and TCP sockets). After it is + loaded, the contents of the new Lisp process's memory are + exactly the same as those of the old Lisp process when the image + was created. + The heap image is how we get around the fact that we can't + run Lisp code until we have a working Lisp implementation, and + we can't make our Lisp implementation work until we can run Lisp + code. Since the heap image already contains a fully-working + implementation, all we need to do is load it into memory and + start using it. + If you're building a new version of OpenMCL, you need to + build a new heap image. + (You might also wish to build a heap image if you have a + large program which it is very complicated or time-consuming to + load, so that you will be able to load it once, save an image, + and thenceforth never have to load it again. At any time, a heap + image capturing the entire memory state of a running Lisp can be + created by calling the function + ccl:save-application.) + + + Development cycle + Creating a new OpenMCL full heap image consists of the + following steps: + + Using your existing OpenMCL, create a + bootstrapping image + Using your existing OpenMCL, recompile your + updated OpenMCL sources + Invoke OpenMCL with the bootstrapping image + you just created (rather than with the existing full heap + image). + + When you invoke OpenMCL with the bootstrapping image, it + will start up, load al of the OpenMCL fasl files, and save out + a new full heap image. Voila. You've created a new heap + image. + A few points worth noting: + + + There's a circular dependency between the full heap + image and thebootstrapping image, in that each is used to + build the other. + + + There are some minor implementation + differences, but the environment in effect after the + bootstrapping image has loaded its fasl files is essentially + equivalent to the environment provided by the full heap + image; the latter loads a lot faster and is easier to + distribute, of course. + + + If the full heap image doesn't work (because + of an OScompatibilty problem or other bug), it's very likely + that thebootstrapping image will suffer the same + problems. + + + Given a bootstrapping image and a set of up-to-date fasl + files, the development cycle usually involves editing lisp + sources (or updating those sources via cvs update), + recompiling modified files, and using the bootstrapping image + to produce a new heap image. + + + + Generating a bootstrapping image + The bootstrapping image isn't provided in OpenMCL + distributions. It can be built from the source code provided + in distributions (using a lisp image and kernel provided in + those distributions) using the procedure described + below. + + The bootstrapping image is built by invoking a special + utility inside a running OpenMCL heap image to load files + contained in the ccl/level-0 directory. The + bootstrapping image loads several dozen fasl files. After + it's done so, it saves a heap image via + save-application. This process is called + "cross-dumping". + + Given a source distribution, a lisp kernel, and aheap + image, one can produce a bootstapping image by first invoking + OpenMCL from the shell: + +shell> openmcl +Welcome to OpenMCL .... ! +? + + then calling ccl:xload-level-0 at the + lisp prompt + +? (ccl:xload-level-0) + + This will compile the lisp sources in the ccl/level-0 + directory if they're newer than the corresponding fasl files + and will then load the resulting fasl files into a simulated + lisp heap contained inside data structures inside the running + lisp. That simulated heap image is then written to + disk. + xload-level-0 should be called + whenever your existing boot image is out-of-date with respect + to the source files in ccl:level-0; + : + +? (ccl:xload-level-0 :force) + + will force recompilation of the level-0 sources. + + + + Generating fasl files + Calling: + +? (ccl:compile-ccl) + + at the lisp prompt will compile any fasl files that are + out-of-date with respect to the corresponding lisp sources; + (ccl:compile-ccl t) will force + recompilation. ccl:compile-ccl will reload + newly-compiled versions of some files; + ccl:xcompile-ccl is analogous, but skips + this reloading step. + Unless there are bootstrapping considerations involved, + it usually doesn't matter whether these files reloaded after + they're recompiled. + Calling compile-ccl or + xcompile-ccl in an environment where fasl + files don't yet exist may produce warnings to that effect + whenever files are required during + compilation; those warnings can be safely ignored. Depending + on the maturity of the OpenMCL release, calling + compile-ccl or + xcompile-ccl may also produce several + warnings about undefined functions, etc. They should be + cleaned up at some point. + + + + Building a full image from a bootstrapping image + To build a full image from a bootstrapping image, just + invoke the kernel with the bootstrapping image is an + argument + +$ cd ccl # wherever your ccl directory is +$ ./ + + Where and + are the names of + the kernel and boot image appropriate to the platform you are + running on. See FIXTHIS + That should load a few dozen fasl files (printing a + message as each file is loaded.) If all of these files + successfully load, the lisp will print a prompt. You should be + able to do essentially everything in that environment that you + can in the environment provided by a "real" heap image. If + you're confident that things loaded OK, you can save that + image. + +? (ccl:save-application "image_name") ; Overwiting the existing heap image + + Where image_name is the name + of the full heap image for your platform. See FIXTHIS. + If things go wrong in the early stages of the loading + sequence, errors are often difficult to debug; until a fair + amount of code (CLOS, the CL condition system, streams, the + reader, the read-eval-print loop) is loaded, it's generally + not possible for the lisp to report an error. Errors that + occur during these early stages ("the cold load") sometimes + cause the lisp kernel debugger (see ) to be invoked; it's + primitive, but can sometimes help one to get oriented. + + + + + + + Questions and Answers + + + How can I do nonblocking (aka "unbuffered" and "raw") IO? + There's some code for manipulating TTY modes in + "ccl:library;pty.lisp". + +? (require "PTY") + +? (ccl::disable-tty-local-modes 0 #$ICANON) +T + + will turn off "input canonicalization" on file descriptor + 0, which is at least part of what you need to do here. This + disables the #$ICANON mode, which tells the OS not to do any + line-buffering or line-editing. Of course, this only has any + effect in situations where the OS ever does that, which means + when stdin is a TTY or PTY. + If the #$ICANON mode is disabled, you can do things like: + +? (progn (read-char) (read-char)) +a +#\a + + (where the first READ-CHAR consumes the newline, which + isn't really necessary to make the reader happy anymore.) So, + you can do: + +? (read-char) +#\Space + + (where there's a space after the close-paren) without + having to type a newline. + + + + I'm using the graphics demos. Why doesn't the menubar + change? + When you interact with text-only OpenMCL, you're either + in Terminal or in Emacs, running OpenMCL as a subprocess. When + you load Cocoa or the graphical environment, the subprocess does + some tricky things that turn it into a full-fledged Application, + as far as the OS is concerned. + So, it gets its own icon in the dock, and its own menubar, + and so on. It can be confusing, because standard input and + output will still be connected to Terminal or Emacs, so you can + still type commands to OpenMCL from there. To see the menubar + you loaded, or the windows you opened, just click on the OpenMCL + icon in the dock. + + + + I'm using Slime and Cocoa. Why doesn't *standard-output* + seem to work? + This comes up if you're using the Slime interface + to run OpenMCL under Emacs, and you are doing Cocoa programming + which involves printing to *standard-output*. It seems as + though the output goes nowhere; no error is reported, but it + doesn't appear in the *slime-repl* buffer. + + For the most part, this is only relevant when you are + trying to insert debug code into your event handlers. The SLIME + listener runs in a thread where the standard stream varaiables + (like *STANDARD-OUTPUT* and and + *TERMINAL-IO* are bound to the stream used to + communicate with Emacs; the Cocoa event thread has its own + bindings of these standard stream variables, and output to these + streams goes to the *inferior-lisp* buffer instead. Look for it + there. + + + + + Programming with Threads + + + Threads Overview + + OpenMCL provides facilities which enable multiple threads + of execution (threads, sometimes called + lightweight processes or just + processes, though the latter term shouldn't + be confused with the OS's notion of a process) within a lisp + session. This document describes those facilities and issues + related to multitheaded programming in OpenMCL. + + Wherever possible, I'll try to use the term "thread" to + denote a lisp thread, even though many of the functions in the + API have the word "process" in their name. A + lisp-process is a lisp object (of type + CCL:PROCESS) which is used to control and communicate with an + underlying native thread. Sometimes, the + distinction between these two (quite different) objects can be + blurred; other times, it's important to maintain. + Lisp threads share the same address space, but maintain + their own execution context (stacks and registers) and their own + dynamic binding context. + + Traditionally, OpenMCL's threads have been + cooperatively scheduled: through a + combination of compiler and runtime suppport, the currently + executing lisp thread arranged to be interrrupted at certain + discrete points in its execution (typically on entry to a + function and at the beginning of any looping construct). This + interrupt occurred several dozen times per second; in response, + a handler function might observe that the current thread had + used up its time slice and another function (the lisp + scheduler) would be called to find some other thread + that was in a runnable state, suspend execution of the current + thread, and resume execution of the newly executed thread. The + process of switching contexts between the outgoing and incoming + threads happened in some mixture of Lisp and assembly language + code; as far as the OS was concerned, there was one native + thread running in the Lisp image and its stack pointer and other + registers just happened to change from time to time. + Under OpenMCL's cooperative scheduling model, it was + possible (via the use of the CCL:WITHOUT-INTERRUPTS construct) + to defer handling of the periodic interrupt that invoked the + lisp scheduler; it was not uncommon to use WITHOUT-INTERRUPTS to + gain safe, exclusive access to global data structures. In some + code (including much of OpenMCL itself) this idiom was very + common: it was (justifiably) believed to be an efficient way of + inhibiting the execution of other threads for a short period of + time. + + The timer interrupt that drove the cooperative scheduler + was only able to (pseudo-)preempt lisp code: if any thread + called a blocking OS I/O function, no other thread could be + scheduled until that thread resumed execution of lisp code. Lisp + library functions were generally attuned to this constraint, and + did a complicated mixture of polling and "timed blocking" in an + attempt to work around it. Needless to say, this code is + complicated and less efficient than it might be; it meant that + the lisp was a little busier than it should have been when it + was "doing nothing" (waiting for I/O to be possible.) + + For a variety of reasons - better utilization of CPU + resources on single and multiprocessor systems and better + integration with the OS in general - threads in OpenMCL 0.14 and + later are preemptively scheduled. In this + model, lisp threads are native threads and all scheduling + decisions involving them are made by the OS kernel. (Those + decisions might involve scheduling multiple lisp threads + simultaneously on multiple processors on SMP systems.) This + change has a number of subtle effects: + + + + it is possible for two (or more) lisp threads to be + executingsimultaneously, possibly trying to access and/or + modify the same datastructures. Such access really should + have been coordinated throughthe use of synchronization + objects regardless of the scheduling modelin effect; + preemptively scheduled threads increase the chance ofthings + going wrong at the wrong time and do not offer + lightweightalternatives to the use of those synchronization + objects. + + + even on a single-processor system, a context switch + can happenon any instruction boundary. Since (in general) + other threads mightallocate memory, this means that a GC can + effectively take place atany instruction boundary. That's + mostly an issue for the compilerand runtime system to be + aware of, but it means that certain practices(such as trying + to pass the address of a lisp object to foreign code)that + were always discouraged are now discouraged + ... vehemently. + + + there is no simple and efficient way to "inhibit the + scheduler"or otherwise gain exclusive access to the entire + CPU. + + + There are a variety of simple and efficient ways + tosynchronize access to particular data + structures. + + + As a broad generalization: code that's been aggressively + tuned to the constraints of the cooperative scheduler may need + to be redesigned to work well with the preemptive scheduler (and + code written to run under OpenMCL's interface to the native + scheduler may be less portable to other CL implementations, many + of which offer a cooperative scheduler and an API similar to + OpenMCL (< 0.14) 's.) At the same time, there's a large + overlap in functionality in the two scheduling models, and it'll + hopefully be possible to write interesting and useful MP code + that's largely independent of the underlying scheduling + details. + The keyword :OPENMCL-NATIVE-THREADS is on *FEATURES* in + 0.14 and later and can be used for conditionalization where + required. + + + + (Intentionally) Missing Functionality + Much of the functionality described above is similar to + that provided by OpenMCL's cooperative scheduler, some other + parts of which make no sense in a native threads + implementation. + + + PROCESS-RUN-REASONS and PROCESS-ARREST-REASONS were + SETFable process attributes; each was just a list of + arbitrary tokens. A thread was eligible for scheduling + (roughly equivalent to being "enabled") if its arrest-reasons + list was empty and its run-reasons list was not. I don't + think that it's appropriate to encourage a programming style + in which otherwise runnable threads are enabled and disabled + on a regular basis (it's preferable for threads to wait for + some sort of synchronization event to occur if they can't + occupy their time productively.) + + + There were a number of primitives for maintaining + process queues;that's now the OS's job. + + + Cooperative threads were based on coroutining + primitivesassociated with objects of type + STACK-GROUP. STACK-GROUPs no longerexist. + + + + + + Implementation Decisions and Open Questions + As of August 2003: + + + It's not clear that exposing + PROCESS-SUSPEND/PROCESS-RESUME is a good idea: it's not clear + that they offer ways to win, and it's clear that they offer + ways to lose. + + + It has traditionally been possible to reset and enable + a process that's "exhausted" . (As used here, the + term"exhausted" means that the process's initial function + hasrun and returned and the underlying native thread has + beendeallocated.) One of the principle uses of PROCESS-RESET + is to "recycle" threads; enabling an exhausted process + involves creating a new native thread (and stacks and + synchronization objects and ...),and this is the sort of + overhead that such a recycling scheme is seeking to avoid. It + might be worth trying to tighten things up and declare that + it's an error to apply PROCESS-ENABLE to an exhausted thread + (and to make PROCESS-ENABLE detect this error.) + + + When native threads that aren't created by OpenMCL + first call into lisp, a "foreign process" is created, and + that process is given its own set of initial bindings and set + up to look mostly like a process that had been created by + MAKE-PROCESS. The life cycle of a foreign process is + certainly different from that of a lisp-created one: it + doesn't make sense to reset/preset/enable a foreign process, + and attempts to perform these operations should be + detectedand treated as errors. + + + + + + Porting Code from the Old Thread Model + Older versions of OpenMCL used what are often called + "user-mode threads", a less versatile threading model which does + not require specific support from the operating system. This + section discusses how to port code which was written for that + mode. + It's hard to give step-by-step instructions; there are certainly + a few things that one should look at carefully: + + + It's wise to be suspicious of most uses + of WITHOUT-INTERRUPTS; there may be exceptions, but + WITHOUT-INTERRUPTS is often used as shorthand for + WITH-APPROPRIATE-LOCKING. Determining what type of locking + is appropriate and writing the code to implement it is + likely to be straightforward and simple most of the + time. + + + I've only seen one case where a process's "run reasons" + were used to communicate information as well as tocontrol + execution; I don't think that this is a common idiom, but may + be mistaken about that. + + + + It's certainly possible that programs written + for cooperatively scheduled lisps that have run reliably for + a long timehave done so by accident: resource-contention + issues tend to be timing-sensitive, and decoupling thread + scheduling from lisp program execution affects timing. I know + that there is or was code in both OpenMCL and commercial MCL + that was written under the explicit assumption that certain + sequences of open-coded operations were uninterruptable; it's + certainly possible that the same assumptions have been made + (explicitly or otherwise) by application developers. + + + + + + Background Terminal Input + + + Overview + + Unless and until OpenMCL provides alternatives (via window + streams, telnet streams, or some other mechanism) all lisp + processes share a common *TERMINAL-IO* stream (and therefore + share *DEBUG-IO*, *QUERY-IO*, and other standard and + internal interactive streams.) + It's anticipated that most lisp processes other than + the "Initial" process run mostly in the background. If a + background process writes to the output side of + *TERMINAL-IO*, that may be a little messy and a little + confusing to the user, but it shouldn't really be + catastrophic. All I/O to OpenMCL's buffered streams goes + thru a locking mechanism that prevents the worst kinds of + resource-contention problems. + Although the problems associated with terminal output + from multiple processes may be mostly cosmetic, the question + of which process receives input from the terminal is likely + to be a great deal more important. The stream locking + mechanisms can make a confusing situation even worse: + competing processes may "steal" terminal input from each + other unless locks are held longer than they otherwise need + to be, and locks can be held longer than they need to be (as + when a process is merely waiting for input to become + available on an underlying file descriptor). + Even if background processes rarely need to + intentionally read input from the terminal, they may still + need to do so in response to errors or other unanticipated + situations. There are tradeoffs involved in any solution to + this problem. The protocol described below allows background + processes which follow it to reliably prompt for and receive + terminal input. Background processes which attempt to + receive terminal input without following this protocol will + likely hang indefinitely while attempting to do so. That's + certainly a harsh tradeoff, but since attempts to read + terminal input without following this protocol only worked + some of the time anyway, it doesn't seem to be an + unreasonable one. + In the solution described here (and introduced in + OpenMCL 0.9), the internal stream used to provide terminal + input is always locked by some process (the "owning" + process.) The initial process (the process that typically + runs the read-eval-print loop) owns that stream when it's + first created. By using the macro WITH-TERMINAL-INPUT, + background processes can temporarily obtain ownership of the + terminal and relinquish ownership to the previous owner when + they're done with it. + In OpenMCL, BREAK, ERROR, CERROR, Y-OR-N-P, + YES-OR-NO-P, and CCL:GET-STRING- FROM-USER are all defined + in terms of WITH-TERMINAL-INPUT, as are the :TTY + user-interfaces to STEP and INSPECT. + + + + An example + +? Welcome to OpenMCL Version (Beta: linux) 0.9! +? + +? (process-run-function "sleeper" #'(lambda () (sleep 5) (break "broken"))) +# + This example was run under ILISP; ILISP often gets confused if one + tries to enter input and "point" doesn't follow a prompt. + Entering a "simple" expression at this point gets it back in + synch; that's otherwise not relevant to this example. + +() +NIL +? (:y 1) +;; +;; process sleeper(1) now controls terminal input +;; +> Break in process sleeper(1): broken +> While executing: # Type :GO to continue, :POP to abort. +> If continued: Return from BREAK. +Type :? for other options. +1 > :b +(30C38E30) : 0 "Anonymous Function #x3063B276" 52 +(30C38E40) : 1 "Anonymous Function #x304984A6" 376 +(30C38E90) : 2 "RUN-PROCESS-INITIAL-FORM" 340 +(30C38EE0) : 3 "%RUN-STACK-GROUP-FUNCTION" 768 +1 > :pop +;; +;; control of terminal input restored to process Initial(0) +;; +? + + + + + A more elaborate example. + If a background process ("A") needs access to the terminal + input stream and that stream is owned by another background process + ("B"), process "A" announces that fact, then waits until + the initial process regains control. + +? Welcome to OpenMCL Version (Beta: linux) 0.9! +? + +? (process-run-function "sleep-60" #'(lambda () (sleep 60) (break "Huh?"))) +# Break in process sleep-5(2): quicker +> While executing: #x3063CFDE> +> Type :GO to continue, :POP to abort. +> If continued: Return from BREAK. +Type :? for other options. +1 > ;; Process sleep-60(1) will need terminal access when +;; the initial process regains control of it. +;; +() +NIL +1 > :pop +;; +;; Process sleep-60(1) needs access to terminal input. +;; +;; +;; control of terminal input restored to process Initial(0) +;; + +? (:y 1) +;; +;; process sleep-60(1) now controls terminal input +;; +> Break in process sleep-60(1): Huh? +> While executing: #x3063BE5E> +> Type :GO to continue, :POP to abort. +> If continued: Return from BREAK. +Type :? for other options. +1 > :pop +;; +;; control of terminal input restored to process Initial(0) +;; + +? + + + + + Summary + This scheme is certainly not bulletproof: imaginative + use of PROCESS-INTERRUPT and similar functions might be able + to defeat it and deadlock the lisp, and any scenario where + several background processes are clamoring for access to the + shared terminal input stream at the same time is likely to be + confusing and chaotic. (An alternate scheme, where the input + focus was magically granted to whatever thread the user was + thinking about, was considered and rejected due to technical + limitations.) + The longer-term fix would probably involve using network or + window-system streams to give each process unique instances of + *TERMINAL-IO*. + Existing code that attempts to read from *TERMINAL-IO* + from a background process will need to be changed to use + WITH-TERMINAL-INPUT. Since that code was probably not working + reliably in previous versions of OpenMCL, this requirement + doesn't seem to be too onerous. + Note that WITH-TERMINAL-INPUT both requests ownership of + the terminal input stream and promises to restore that + ownership to the initial process when it's done with it. An ad + hoc use of READ or READ-CHAR doesn't make this promise; this + is the rationale for the restriction on the :Y command. + + + + + The Threads which OpenMCL Uses for Its Own Purposes +In the "tty world", OpenMCL starts out with 2 lisp-level threads: + +? :proc +1 : -> listener [Active] +0 : Initial [Active] + + If you look at a running OpenMCL with a debugging tool, + such as GDB, or Apple's Thread Viewer.app, you'll see an + additional kernel-level thread on Darwin; this is used by the + Mach exception-handling mechanism. + The initial thread, conveniently named "initial", is the + one that was created by the operating system when it launched + OpenMCL. It maps the heap image into memory, does some + Lisp-level initialization, and, when the Cocoa IDE isn't being + used, creates the thread "listener", which runs the top-level + loop that reads input, evaluates it, and prints the + result. + After the listener thread is created, the initial thread + does "housekeeping": it sits in a loop, sleeping most of the + time and waking up occasionally to do "periodic tasks". These + tasks include forcing output on specified interactive streams, + checking for and handling control-C interrupts, etc. Currently, + those tasks also include polling for the exit status of external + processes and handling some kinds of I/O to and from those + processes. + In this environment, the initial thread does these + "housekeeping" activities as necessary, until + ccl:quit is called; + quitting interrupts the initial thread, which + then ends all other threads in as orderly a fashion as possible + and calls the C function #_exit. + The short-term plan is to handle each external-process in + a dedicated thread; the worst-case behavior of the current + scheme can involve busy-waiting and excessive CPU utilization + while waiting for an external process to terminate in some + cases. + The Cocoa features use more threads. Adding a Cocoa + listener creates two threads: + +? :proc +3 : -> Listener [Active] +2 : housekeeping [Active] +1 : listener [Active] +0 : Initial [Active] + + The Cocoa event loop has to run in the initial thread; + when the event loop starts up, it creates a new thread to do the + "housekeeping" tasks which the initial thread would do in the + terminal-only mode. The initial thread then becomes the one to + receive all Cocoa events from the window server; it's the only + thread which can. + It also creates one "Listener" (capital-L) thread for each + listener window, with a lifetime that lasts as long as the + thread does. So, if you open a second listener, you'll see five + threads all together: + +? :proc +4 : -> Listener-2 [Active] +3 : Listener [Active] +2 : housekeeping [Active] +1 : listener [Active] +0 : Initial [Active] + + Unix signals, such as SIGINT (control-C), invoke a handler + installed by the Lisp kernel. Although the OS doesn't make any + specific guarantee about which thread will receive the signal, + in practice, it seems to be the initial thread. The handler + just sets a flag and returns; the housekeeping thread (which may + be the initial thread, if Cocoa's not being used) will check for + the flag and take whatever action is appropriate to the + signal. + In the case of SIGINT, the action is to enter a break + loop, by calling on the thread being interrupted. When there's + more than one Lisp listener active, it's not always clear what + thread that should be, since it really depends on the user's + intentions, which there's no way to divine programmatically. To + make its best guess, the handler first checks whether the value + of ccl:*interactive-abort-process* is a + thread, and, if so, uses it. If that fails, it chooses the + thread which currently "owns" the default terminal input stream; + see . + In the bleeding-edge version of the Cocoa support which is + based on Hemlock, an Emacs-like editor, each editor window has a + dedicated thread associated with it. When a keypress event + comes in which affects that specific window the initial thread + sends it to the window's dedicated thread. The dedicated thread + is responsible for trying to interpret keypresses as Hemlock + commands, applying those commands to the active buffer; it + repeats this in a loop, until the window closes. The initial + thread handles all other events, such as mouse clicks and + drags. + This thread-per-window scheme makes many things simpler, + including the process of entering a "recursive command loop" in + commands like "Incremental Search Forward", etc. (It might be + possible to handle all Hemlock commands in the Cocoa event + thread, but these "recursive command loops" would have to + maintain a lot of context/state information; threads are a + straightforward way of maintaining that information.) + Currently (August 2004), when a dedicated thread needs to + alter the contents of the buffer or the selection, it does so by + invoking methods in the initial thread, for synchronization + purposes, but this is probably overkill and will likely be + replaced by a more efficient scheme in the future. + The per-window thread could probably take more + responsibility for drawing and handling the screen than it + currently does; -something- needs to be done to buffer screen + updates a bit better in some cases: you don't need to see + everything that happens during something like indentation; you + do need to see the results... + When Hemlock is being used, listener windows are editor + windows, so in addition to each "Listener" thread, you should + also see a thread which handles Hemlock command + processing. + The Cocoa runtime may make additional threads in certain + special situations; these threads usually don't run lisp code, + and rarely if ever run much of it. + + + + Threads Dictionary + + + + all-processes + + + + ALL-PROCESSES + Obtain a fresh list of all known Lisp + threads. + Function + + + + + all-processes => result + + + + + Values + + + + result + + a list of all lisp processes (threads) + known to OpenMCL. + + + + + + Description + + Returns a list of all lisp processes (threads) known + to OpenMCL as of the precise instant it's + called. It's safe to traverse this list and to modify + the cons cells that comprise that list (it's freshly + consed.) Since other threads can create and kill threads at + any time, there's generally no way to get an + "accurate" list of all threads, and (generally) no + sense in which such a list can be accurate. + + + + See Also + + + + + + + + + + + make-process + + + + MAKE-PROCESS + Creates and returns a new process. + Function + + + + + make-process + name &key; + persistent priority class stack-size vstack-size + tstack-size initial-bindings use-standard-initial-bindings + => process + + + + + Arguments and Values + + + + name + + + a string, used to identify the process. + + + + + persistent + + + if true, requests that information about the process + be retained by SAVE-APPLICATION so that an equivalent + process can be restarted when a saved image is run. The + default is nil. + + + + + priority + + + ignored. It + shouldn't be ignored of course, but there are + complications on some platforms. The default is 0. + + + + + class + + + the class of process object to create; + should be a subclass of CCL:PROCESS. The default is + CCL:PROCESS. + + + + + stack-size + + + the size, in bytes, of the newly-created process's + control stack; used for foreign function calls and to save + function return address context. The default is + CCL:*DEFAULT-CONTROL-STACK-SIZE*. + + + + + vstack-size + + + the size, in bytes, of the newly-created process's + value stack; used for lisp function arguments, local + variables, and other stack-allocated lisp objects. + The default is CCL:*DEFAULT-VALUE-STACK-SIZE*. + + + + + tstack-size + + + the size, in bytes, of the newly-created process's + temp stack; used for the allocation of dynamic-extent + objects. The default is CCL:*DEFAULT-TEMP-STACK-SIZE*. + + + + + use-standard-initial-bindings + + + when true, the global "standard initial + bindings" are put into effect in the new thread before. See + DEF-STANDARD-INITIAL-BINDING. "standard" initial bindings + are put into effect before any bindings specified by + :initial-bindings are. The default is t. + + + + + initial-bindings + + + an alist of (symbol . + valueform) pairs, which can be + used to initialize special variable bindings in the new + thread. Each valueform is used to + compute the value of a new binding of + symbol in the execution environment of + the newly-created thread. The default is nil. + + + + + process + + + the newly-created process. + + + + + + + Description + + Creates and returns a new lisp process (thread) with the + specified attributes. process will not begin + execution immediately; it will need to be + preset (given + an initial function to run, as by + ) and + enabled + (allowed to execute, as by ) + before it's able to actually do anything. + + If valueform is a function, it is + called, with no arguments, in the execution environment of the + newly-created thread; the primary value it returns is used for + the binding of the corresponding symbol. + + Otherwise, valueform is evaluated in the + execution + environment of the newly-created thread, and the resulting value + is used. + + + + See Also + + + + + + + + + + + PROCESS-SUSPEND + process-suspend + Name + PROCESS-SUSPEND — Suspends a specified process. + Function + Synopsis + +process-suspend process + => result + + Arguments and Values + process + a lisp process (thread). + result + T if process had been runnableand is now suspended; NIL otherwise. That is, T ifprocess'stransitioned from 0 to 1. + + + Description + Suspends process, preventing it from +running, and stopping it if it was already running. This is a fairly +expensive operation, because it involves a few +calls to the OS. It also risks creating deadlock if used +improperly, for instance, if the process being suspended owns a +lock or other resource which another process will wait for. + Each +call to process-suspend must be reversed by +a matching call to +before process is able to run. What +process-suspend actually does is increment +the of +process. + A process cannot suspend itself (although that was possible in +some older OpenMCL releases and this documentation claimed that +it still was.) + See Also + , + Notes + process-suspend was previously called +process-disable. + now names a function for which there is no +obvious inverse, so process-disable +is no longer +defined. + + + + PROCESS-RESUME + process-resume + Name + PROCESS-RESUME — Resumes a specified process which had previously +been suspended by process-suspend. + Function + Synopsis + +process-resume process + => result + + Arguments and Values + process + a lisp process (thread). + result + T if process had been suspendedand is now runnable; NIL otherwise. That is, T ifprocess'stransitioned from to 0. + + + Description + Undoes the effect of a previous call to +; if +all such calls are undone, makes the process runnable. Has no +effect if the process is not suspended. What +process-resume actually does is decrement +the of +process, to a minimum of 0. + See Also + , + Notes + This was previously called PROCESS-ENABLE; + now does something slightly +different. + + + + PROCESS-SUSPEND-COUNT + process-suspend-count + Name + PROCESS-SUSPEND-COUNT — Returns the number of currently-pending suspensions +applicable to a given process. + Function + Synopsis + + + process-suspend-count + process => result + + + Arguments and Values + process + a lisp process (thread). + result + The number of "outstanding" calls onprocess, or NIL ifprocess has expired. + + + Description + An "outstanding" call +is one which has not yet been reversed by a call to +. A process expires when +its initial function returns, although it may later be +reset. + A process is runnable when it has a +process-suspend-count of 0, has been +preset as by , and has been +enabled as by . Newly-created +processes have a process-suspend-count of +0. + See Also + , + + + + PROCESS-PRESET + process-preset + Name + PROCESS-PRESET — Sets the initial function and arguments of a specified +process. + Function + Synopsis + +process-preset + process function &rest args + => result + + Arguments and Values + process + a lisp process (thread). + function + a function, designated by itself or by a symbolwhich names it. + args + a list of values, appropriate as arguments tofunction. + result + undefined. + + + Description + Typically used to initialize a newly-created or newly-reset +process, setting things up so that when process +becomes enabled, it will begin execution by +applying function to args. +process-preset does not enable +process, +although a process must be process-preset +before it can be enabled. Processes are normally enabled by +. + See Also + , , + + + + PROCESS-ENABLE + process-enable + Name + PROCESS-ENABLE — Begins executing the initial function of a specified +process. + Function + Synopsis + +process-enable + process &optional timeout + + + Arguments and Values + process + a lisp process (thread). + timeout + a time interval in seconds. May be anynon-negative real number the floor ofwhich fits in 32 bits. The default is 1. + result + undefined. + + + Description + Tries to begin the execution of process. +An error is signaled if process has never +been . Otherwise, +process invokes its initial function. + process-enable attempts to +synchronize with process, which is presumed +to be reset or in the act of resetting itself. If this attempt +is not successful within the time interval specified by +timeout, a continuable error is signaled, +which offers the opportunity to continue waiting. + A process cannot meaningfully attempt to enable itself. + See Also + , , + Notes + It would be nice to have more discussion of what it means +to synchronize with the process. + + + + PROCESS-RUN-FUNCTION + process-run-function + Name + PROCESS-RUN-FUNCTION — Creates a process, presets it, and enables it. + Function + Synopsis + +process-run-function + process-specifier function &rest args => process + + process-specifier + name |(&key namepersistentpriorityclassstack-sizevstack-sizetstack-size) + + + Arguments and Values + name + a string, used to identify the process.Passed to make-process. + function + a function, designated by itself or by a symbolwhich names it. Passed topreset-process. + persistent + a boolean, passed to make-process. + priority + ignored. + class + a subclass of CCL:PROCESS. Passed tomake-process. + stack-size + a size, in bytes. Passed tomake-process. + vstack-size + a size, in bytes. Passed tomake-process. + tstack-size + a size, in bytes. Passed tomake-process. + process + the newly-created process. + + + Description + Creates a lisp process (thread) via +, +presets it via , and +enables it via . This means +that process will immediately begin to +execute. +process-run-function is +the simplest way to create and run a process. + See Also + , , + + + + PROCESS-INTERRUPT + process-interrupt + Name + PROCESS-INTERRUPT — Arranges for the target process to invoke a +specified function at some point in the near future, and then +return to what it was doing. + Function + Synopsis + +process-interrupt + process function &rest args => result + + Arguments and Values + process + a lisp process (thread). + function + a function. + args + a list of values, appropriate as arguments tofunction. + result + the result of applying functionto args if proceessis the current-process, otherwiseNIL. + + + Description + Arranges for process to apply function to args at +some point in the near future (interrupting whatever process +was doing.) If function returns normally, process +resumes execution at the point at which it was interrupted. + process must be in an enabled state in order to respond to a +process-interrupt request. It's perfectly legal for a process +to call process-interrupt on itself. + process-interrupt uses asynchronous POSIX signals to interrupt +threads. If the thread being interrupted is executing lisp code, it +can respond to the interrupt almost immediately (as soon as it has +finished pseudo-atomic operations like consing and stack-frame +initialization.) + If the interrupted thread is blocking in a system call, that system +call is aborted by the signal and the interrupt is handled on return. + Beginning with the version 1.1 prereleases of OpenMCL interrupts are +disabled in unwind-protect cleanup forms and in any stack-unwinding +code between the point of the throw and the corresponding CATCH +target. If interrupts were enabled at the time that the CATCH was +established, then any interrupt that had been deferred during +unwinding will be taken just before the transfer to catch target +(after all of that unwinding is complete.) + If an unwind-protect cleanup form actually does something that +needs to be interruptible, it's necessary to use +with-interrupts-enabled. (This actually happens somewhere in +the code which waits for external processes to complete; some of that +waiting occurred within an unwind-protect cleanup, and interrupts +needed to be explicitly enabled in that case in order to make the wait +interruptible.) + Note that (without-interrupts (throw ...)) wouldn't have the intended +effect, since the throw would cause execution to exit the extent of +the without-interrupts. + In versions prior to 1.1, interrupts could occur at arbitrary times +during the process of unwinding the stack and executing intervening +cleanup forms. + It is still difficult to reliably interrupt arbitrary foreign code +(that may be stateful or otherwise non-reentrant); the interrupt +request is handled when such foreign code returns to or enters lisp. + See Also + + Notes + It would probably be better for result to always be NIL, since +the present behaviour is inconsistent. + Process-interrupt works by sending signals between threads, via +the C function #_pthread_signal. It could be argued that it +should be done in one of several possible other ways under Darwin, to +make it practical to asynchronously interrupt things which make heavy +use of the Mach nanokernel. + + + + *CURRENT-PROCESS* + *current-process* + Name + *CURRENT-PROCESS* — Bound in each process, to that process +itself. + Variable + Value Type + A lisp process (thread). + Initial Value + Bound separately in each process, to that process itself. + Description + Used when lisp code needs to find out what process it is +executing in. Shouldn't be set by user code. + See Also + + + + + PROCESS-RESET + process-reset + Name + PROCESS-RESET — Causes a specified process to cleanly exit from +any ongoing computation. + Function + Synopsis + +process-reset + process &optional kill-option => result + + Arguments and Values + process + a lisp process (thread). + kill-option + a generalized boolean. The default is T. + result + undefined. + + + Description + Causes process to cleanly exit +from any ongoing computation. If kill-option +is true, process then exits. Otherwise, it +enters a state where it can be +. This +is implemented by signaling a condition of type PROCESS-RESET; +user-defined condition handlers should generally refrain from +attempting to handle conditions of this type. + A process can meaningfully reset itself. + There is in general no way to know precisely when +process +has completed the act of resetting or killing itself; a process +which has either entered the limbo of the reset state or exited +has few ways of communicating either fact. + can reliably determine when a process has entered +the "limbo of the reset state", but can't predict how long the +clean exit from ongoing computation might take: that depends on +the behavior of unwind-protect cleanup +forms, and of the OS scheduler. + Resetting a process other than + involves the +use of . + See Also + , + + + + PROCESS-KILL + process-kill + Name + PROCESS-KILL — Causes a specified process to cleanly exit from any +ongoing computation, and then exit. + Function + Synopsis + +process-kill process + => result + + Arguments and Values + process + a lisp process (thread). + result + undefined. + + + Description + Entirely equivalent to calling +(PROCESS-RESET PROCESS T). Causes process +to cleanly exit from any ongoing computation, and then exit. + See Also + , + + + + PROCESS-ABORT + process-abort + Name + PROCESS-ABORT — Causes a specified process to process an abort +condition, as if it had invoked +abort. + Function + Synopsis + +process-abort process + &optional condition + => NIL + + Arguments and Values + process + a lisp process (thread). + condition + a lisp condition. The default is NIL. + + + Description + Entirely equivalent to calling +( process +(lambda () +(abort condition))). +Causes process to transfer control to the +applicable handler or restart for abort. + If condition is non-NIL, +process-abort does not consider any +handlers which are explicitly bound to conditions other than +condition. + See Also + , + + + + *TICKS-PER-SECOND* + *ticks-per-second* + Name + *TICKS-PER-SECOND* — Bound to the clock resolution of the OS +scheduler. + Variable + Value Type + A positive integer. + Initial Value + The clock resoluton of the OS scheduler. Currently, +both LinuxPPC and DarwinPPC yeild an initial value of 100. + Description + This value is ordinarily of marginal interest at best, +but, for backward compatibility, some functions accept timeout +values expressed in "ticks". This value gives the number of +ticks per second. + See Also + + + + + PROCESS-WHOSTATE + process-whostate + Name + PROCESS-WHOSTATE — Returns a string which describes the status of +a specified process. + Function + Synopsis + +process-whostate process + => whostate + + process + a lisp process (thread). + whostate + a string which describes the "state" ofprocess. + + + Description + This information is primarily for the benefit of +debugging tools. whostate is a terse report +on what process is doing, or not doing, +and why. + If the process is currently waiting in a call to + or +, its +process-whostate will be the value +which was passed to that function as whostate. + See Also + , , + Notes + This should arguably be SETFable, but doesn't seem to +ever have been. + + + + PROCESS-ALLOW-SCHEDULE + process-allow-schedule + Name + PROCESS-ALLOW-SCHEDULE — Used for cooperative multitasking; probably never +necessary. + Function + Synopsis + +process-allow-schedule + + Description + Advises the OS scheduler that the current thread has nothing +useful to do and that it should try to find some other thread to +schedule in its place. There's almost always a better +alternative, such as waiting for some specific event to +occur. For example, you could use a lock or semaphore. + See Also + , , , , , + Notes + This is a holdover from the days of cooperative +multitasking. All modern general-purpose operating systems use +preemptive multitasking. + + + + PROCESS-WAIT + process-wait + Name + PROCESS-WAIT — Causes the current lisp process (thread) to wait for +a given +predicate to return true. + Function + Synopsis + +process-wait + whostate function &rest args => result + + Arguments and Values + whostate + a string, which will be the value ofwhile the process is waiting. + function + a function, designated by itself or by a symbolwhich names it. + args + a list of values, appropriate as arguments tofunction. + result + NIL. + + + Description + Causes the current lisp process (thread) to repeatedly +apply function to +args until the call returns a true result, then +returns NIL. After +each failed call, yields the CPU as if by +. + As with , it's almost +always more efficient to wait for some +specific event to occur; this isn't exactly busy-waiting, but +the OS scheduler can do a better job of scheduling if it's given +the relevant information. For example, you could use a lock +or semaphore. + See Also + , , , , , , , + + + + PROCESS-WAIT-WITH-TIMEOUT + process-wait-with-timeout + Name + PROCESS-WAIT-WITH-TIMEOUT — Causes the current thread to wait for a given +predicate to return true, or for a timeout to expire. + Function + Synopsis + +process-wait-with-timeout + whostate ticks function args => result + + Arguments and Values + whostate + a string, which will be the value ofwhile the process is waiting. + ticks + either a positive integer expressing a durationin "ticks" (see ),or NIL. + function + a function, designated by itself or by a symbolwhich names it. + args + a list of values, appropriate as arguments tofunction. + result + T if process-wait-with-timeoutreturned because its function returnedtrue, or NIL if it returned because the durationticks has been exceeded. + + + Description + If ticks is NIL, behaves exactly like +, except for returning T. +Otherwise, function will be tested repeatedly, +in the same +kind of test/yield loop as in > +until either function returns true, +or the duration ticks has been exceeded. + Having already read the descriptions of + and +, the +astute reader has no doubt anticipated the observation that +better alternatives should be used whenever possible. + See Also + , , , , , , , , + + + + WITHOUT-INTERRUPTS + without-interrupts + Name + WITHOUT-INTERRUPTS — Evaluates its body in an environment in which +process-interrupt requests are deferred. + Macro + Synopsis + +without-interrupts + &body body => result + + Arguments and Values + body + an implicit progn. + result + the primary value returned bybody. + + + Description + Executes body in an environment in which + requests are deferred. As noted in the +description of , this has nothing to do with +the scheduling of other threads; it may be necessary to inhibit + handling when (for instance) modifying some +data structure (for which the current thread holds an appropriate +lock) in some manner that's not reentrant. + Beginning with the version 1.1 prereleases of OpenMCL interrupts are +disabled in unwind-protect cleanup forms and in any +stack-unwinding code between the point of the throw and the +corresponding CATCH target. If an unwind-protect cleanup form +actually does something that needs to be interruptible, it's necessary +to use with-interrupts-enabled. In versions prior to 1.1, +interrupts can occur at arbitrary times during the process of +unwinding the stack and executing intervening cleanup forms. + See Also + + + + + MAKE-LOCK + make-lock + Name + MAKE-LOCK — Creates and returns a lock object, which can +be used for synchronization between threads. + Function + Synopsis + +make-lock &optional + name => lock + + Arguments and Values + name + any lisp object; saved as part oflock. Typically a string or symbolwhich may appear in the sof threads which are waiting for lock. + lock + a newly-allocated object of type CCL:LOCK. + + + Description + Creates and returns a lock object, which can +be used to synchronize access to some shared resource. +lock is +initially in a "free" state; a lock can also be +"owned" by a +thread. + See Also + , , , , , , , , + + + + WITH-LOCK-GRABBED + with-lock-grabbed + Name + WITH-LOCK-GRABBED — Waits until a given lock can be obtained, then +evaluates its body with the lock held. + Macro + Synopsis + +with-lock-grabbed + (lock) &body body + + Arguments and Values + lock + an object of type CCL:LOCK. + body + an implicit progn. + result + the primary value returned bybody. + + + Description + Waits until lock is either free or +owned by the calling +thread, then excutes body with the +lock owned by the calling thread. If lock +was free when with-lock-grabbed was called, +it is restored to a free state after body +is executed. + See Also + , , , , , , , , + + + + GRAB-LOCK + grab-lock + Name + GRAB-LOCK — Waits until a given lock can be obtained, then +obtains it. + Function + Synopsis + +grab-lock lock + + Arguments and Values + lock + an object of type CCL:LOCK. + + + Description + Blocks until lock is owned by the +calling thread. + The macro +could be defined in +terms of grab-lock and +, but it is actually +implemented at a slightly lower level. + See Also + , , , , , , , , + + + + RELEASE-LOCK + release-lock + Name + RELEASE-LOCK — Relinquishes ownership of a given lock. + Function + Synopsis + +release-lock lock + + Arguments and Values + lock + an object of type CCL:LOCK. + + + Description + Signals an error of type CCL:LOCK-NOT-OWNER if +lock +is not already owned by the calling thread; otherwise, undoes the +effect of one previous +. If this means that +release-lock has now been called on +lock the same number of times as + has, lock +becomes free. + See Also + , , , , , , , , + + + + TRY-LOCK + try-lock + Name + TRY-LOCK — Obtains the given lock, but only if it is not +necessary to wait for it. + Function + Synopsis + +try-lock lock => result + + Arguments and Values + lock + an object of type CCL:LOCK. + result + T if lock has been obtained,or NIL if it has not. + + + Description + Tests whether lock +can be obtained without blocking - that is, either +lock is already free, or it is already owned +by . If it can, +causes it to +be owned by the calling lisp process (thread) and returns T. +Otherwise, the lock +is already owned by another thread and cannot be obtained without +blocking; NIL is returned in this case. + See Also + , , , , , , , , + + + + MAKE-READ-WRITE-LOCK + make-read-write-lock + Name + MAKE-READ-WRITE-LOCK — Creates and returns a read-write lock, which can +be used for synchronization between threads. + Function + Synopsis + +make-read-write-lock + => read-write-lock + + Arguments and Values + read-write-lock + a newly-allocated object of typeCCL:READ-WRITE-LOCK. + + + Description + Creates and returns an object of type CCL::READ-WRITE-LOCK. +A read-write lock may, at any given time, belong to any number +of lisp processes (threads) which act as "readers"; or, it may +belong to at most one process which acts as a "writer". A +read-write lock may never be held by a reader at the same time as +a writer. Intially, read-write-lock has +no readers and no writers. + See Also + , , , , , , + Notes + There probably should be some way to +atomically "promote" a reader, making it a writer without +releasing the lock, which could otherwise cause delay. + + + + WITH-READ-LOCK + with-read-lock + Name + WITH-READ-LOCK — Waits until a given lock is available for +read-only access, then evaluates its body with the lock +held. + Macro + Synopsis + +with-read-lock + (read-write-lock) &body body => result + + Arguments and Values + read-write-lock + an object of typeCCL:READ-WRITE-LOCK. + body + an implicit progn. + result + the primary value returned bybody. + + + Description + Waits until read-write-lock has no +writer, +ensures that is a +reader of it, then executes body. + After executing body, if + was not a reader of +read-write-lock before +with-read-lock was called, the lock is +released. If it was already a reader, it remains one. + See Also + , , , , , , + + + + WITH-WRITE-LOCK + with-write-lock + Name + WITH-WRITE-LOCK — Waits until the given lock is available for write +access, then executes its body with the lock held. + Macro + Synopsis + +with-write-lock + (read-write-lock) &body body + + Arguments and Values + read-write-lock + an object of typeCCL:READ-WRITE-LOCK. + body + an implicit progn. + result + the primary value returned bybody. + + + Description + Waits until read-write-lock has no +readers and no writer other than , +then ensures that is the +writer of it. With the lock held, executes body. + After executing body, if + was not the writer of +read-write-lock before +with-write-lock was called, the lock is +released. If it was already the writer, it remains the +writer. + See Also + , , , , , , + + + + MAKE-SEMAPHORE + make-semaphore + Name + MAKE-SEMAPHORE — Creates and returns a semaphore, which can be used +for synchronization between threads. + Function + Synopsis + +make-semaphore + => semaphore + + Arguments and Values + semaphore + a newly-allocated object of type CCL:SEMAPHORE. + + + Description + Creates and returns an object of type CCL:SEMAPHORE. +A semaphore has an associated "count" which may be incremented +and decremented atomically; incrementing it represents sending +a signal, and decrementing it represents handling that signal. +semaphore has an initial count of 0. + See Also + , , , , , , , + + + + SIGNAL-SEMAPHORE + signal-semaphore + Name + SIGNAL-SEMAPHORE — Atomically increments the count of a given +semaphore. + Function + Synopsis + +signal-semaphore + semaphore => result + + Arguments and Values + semaphore + an object of type CCL:SEMAPHORE. + result + an integer representing an error identifierwhich was returned by the underlying OS call. + + + Description + Atomically increments semaphore's +"count" by 1; this +may enable a waiting thread to resume execution. + See Also + , , , , , , , + Notes + result should probably be interpreted +and acted on by signal-semaphore, because +it is not likely to be meaningful to a lisp program, and the +most common cause of failure is a type error. + + + + WAIT-ON-SEMAPHORE + wait-on-semaphore + Name + WAIT-ON-SEMAPHORE — Waits until the given semaphore has a positive +count which can be atomically decremented. + Function + Synopsis + +wait-on-semaphore + semaphore => result + + Arguments and Values + semaphore + an object of type CCL:SEMAPHORE. + result + WAIT-ON-SEMAPHORE always returns T. + + + Description + Waits until semaphore +has a positive count that can be +atomically decremented; this will succeed exactly once for each +corresponding call to SIGNAL-SEMAPHORE. + See Also + , , , , , , , + Notes + + + + TIMED-WAIT-ON-SEMAPHORE + timed-wait-on-semaphore + Name + TIMED-WAIT-ON-SEMAPHORE — Waits until the given semaphore has a postive +count which can be atomically decremented, or until a timeout +expires. + Function + Synopsis + +timed-wait-on-semaphore + semaphore timeout => result + + Arguments and Values + semaphore + An object of type CCL:SEMAPHORE. + timeout + a time interval in seconds. May be anynon-negative real number the floor ofwhich fits in 32 bits. The default is 1. + result + T if timed-wait-on-semaphorereturned because it was able to decrement the count ofsemaphore; NIL if it returned becausethe duration timeout has beenexceeded. + + + Description + Waits until semaphore +has a positive count that can be +atomically decremented, or until the duration +timeout has +elapsed. + See Also + , , , , , , + + + + PROCESS-INPUT-WAIT + process-input-wait + Name + PROCESS-INPUT-WAIT — Waits until input is available on a given +file-descriptor. + Function + Synopsis + +process-input-wait + fd &optional timeout + + Arguments and Values + fd + a file descriptor, which is a non-negative integerused by the OS to refer to an open file, socket, or similarI/O connection. See CCL::STREAM-DEVICE. + timeout + either NIL, or a time interval in seconds. May be anynon-negative real number the floor ofwhich fits in 32 bits. The default is NIL. + + + Description + Wait until input is available on fd. +This uses the select() system call, and is +generally a fairly +efficient way of blocking while waiting for input. More +accurately, process-input-wait +waits until it's possible to read +from fd without blocking, or until timeout, if +it is not NIL, has been exceeded. + Note that it's possible to read without blocking if +the file is at its end - although, of course, the read will +return zero bytes. + See Also + , , , , + Notes + process-input-wait has a timeout parameter, +and + does not. This +inconsistency should probably be corrected. + + + + PROCESS-OUTPUT-WAIT + process-output-wait + Name + PROCESS-OUTPUT-WAIT — Waits until output is possible on a given file +descriptor. + Function + Synopsis + +process-output-wait fd + + Arguments and Values + fd + a file descriptor, which is a non-negative integerused by the OS to refer to an open file, socket, or similarI/O connection. See CCL::STREAM-DEVICE. + + + Description + Wait until output is possible on fd. +This uses the select() system call, and is +generally a fairly +efficient way of blocking while waiting to output. + If process-output-wait is called on +a network socket which has not yet established a connection, it +will wait until the connection is established. This is an +important use, often overlooked. + See Also + , , , , + Notes + has a timeout parameter, +and +process-output-wait does not. This +inconsistency should probably be corrected. + + + + WITH-TERMINAL-INPUT + with-terminal-input + Name + WITH-TERMINAL-INPUT — Executes its body in an environment with exclusive +read access to the terminal. + Macro + Synopsis + +with-terminal-input + &body body => result + + Arguments and Values + body + an implicit progn. + result + the primary value returned bybody. + + + Description + Requests exclusive read access to the standard terminal +stream, *terminal-io*. Executes +body in an environment with that access. + See Also + , :Y, , , , , + + + + *REQUEST-TERMINAL-INPUT-VIA-BREAK* + request-terminal-input-via-break + Name + *REQUEST-TERMINAL-INPUT-VIA-BREAK* — Controls how attempts to obtain ownership of +terminal input are made. + Variable + Value Type + A boolean. + Initial Value + NIL. + Description + Controls how attempts to obtain ownership of terminal input +are made. When NIL, a message is printed on *TERMINAL-IO*; +it's expected that the user will later yield +control of the terminal via the :Y toplevel command. When T, a +BREAK condition is signaled in the owning process; continuing from +the break loop will yield the terminal to the requesting process +(unless the :Y command was already used to do so in the break +loop.) + See Also + , :Y, , , , , + + + + :Y + :y + Name + :Y — Yields control of terminal input to a specified +lisp process (thread). + Toplevel Command + Synopsis + +(:y p) + + Arguments and Values + p + a lisp process (thread), designated either byan integer which matches itsprocess-serial-number,or by a string which is equal toits process-name. + + + Description + :Y is a toplevel command, not a function. As such, it +can only be used interactively, and only from the initial +process. + The command yields control of terminal input to the +process p, which must have used + to request access to the +terminal input stream. + See Also + , , , , , , + + + + + + Programming with Sockets + + + OverviewOpenMCL supports the socket abstraction for + interprocess communication. A socket represents a connection to + another process, typically (but not necessarily) a TCP/IP + network connection to a client or server running on some other + machine on the network. + All symbols mentioned in this chapter are exported from + the CCL package. As of version 0.13, these symbols are + additionally exported from the OPENMCL-SOCKET package. + OpenMCL supports three types of sockets: TCP sockets, UDP + sockets, and Unix-domain sockets. This should be enough for all + but the most esoteric network situations. All sockets are + created by . The type of socket depends on the arguments to it, + as follows: + tcp-stream + A buffered bi-directional stream over a TCP/IP connection.tcp-stream is a subclass of stream, and you can read and write to itusing all the usual stream functions. Created by (make-socket:addess-family :internet :type :stream :connect :active ...) or by(accept-connection ...). + file-socket-stream + A buffered bi-directional stream over a "UNIX domain"connection. file-socket-stream is a subclass of stream, and you canread and write to it using all the usual stream functions. Createdby (make-socket :address-family :file :type :stream :connect :active...) or by (accept-connection ...), + listener-socket + A passive socket used to listen for incoming TCP/IPconnections on a particular port. A listener-socket is not a stream.It doesn't support I/O. It can only be used to create newtcp-streams by accept-connection. Created by (make-socket :type:stream :connect :passive ...) + file-listener-socket + A passive socket used to listen for incoming UNIX domainconnections named by a file in the local filesystem. Alistener-socket is not a stream. It doesn't support I/O. It canonly be used to create new file-socket-streams by accept-connection.Created by (make-socket :address-family :file :type :stream :connect:passive ...) + udp-socket + A socket representing a packet-based UDP/IP connection. Audp-socket supports I/O but it is not a stream. Instead, you mustuse the special functions send-to and receive-from to read and writeto it. Created by (make-socket :type :datagram ...) + + + + + + Sockets Dictionary + + + MAKE-SOCKET + make-socket + Name + MAKE-SOCKET — + Function + Synopsis + +make-socket + &key address-family type connect eol format + remote-host remote-port local-host local-port local-filename + remote-filename keepalive reuse-address nodelay broadcast linger + backlog + + Arguments and Values + address-family + The address/protocol family of this socket. Currentlyonly :internet (the default), meaning IP, and :file,referring to UNIX domain addresses, are supported. + type + One of :stream (the default) to request aconnection-oriented socket, or :datagram to request aconnectionless socket. The default is :stream. + connect + This argument is only relevant to sockets of type:stream. One of :active (the default) to request a :passiveto request a file or TCP listener socket. + eol + This argument is currently ignored (it is accepted forcompatibility with Franz Allegro). + format + One of :text (the default), :binary, or :bivalent.This argument is ignored for :stream sockets for now, as:stream sockets are currently always bivalent (i.e. theysupport both character and byte I/O). For :datagram sockets,the format determines the type of buffer created byreceive-from. + remote-host + Required for TCP streams, it specifies the host toconnect to (in any format acceptable to lookup-hostname).Ignored for listener sockets. For UDP sockets, it can beused to specify a default host for subsequent calls tosend-to or receive-from. + remote-port + Required for TCP streams, it specifies the port toconnect to (in any format acceptable to lookup-port).Ignored for listener sockets. For UDP sockets, it can beused to specify a default port for subsequent calls to forsubsequent calls to send-to or receive-from. + remote-filename + Required for file-socket streams, it specifies thename of a file in the local filesystem (e.g., NOT mountedvia NFS, AFP, SMB, ...) which names and controls access to aUNIX-domain socket. + local-host + Allows you to specify a local host address for alistener or UDP socket, for the rare case where you want torestrict connections to those coming to a specific localaddress for security reasons. + local-port + Specify a local port for a socket. Most useful forlistener sockets, where it is the port on which the socketwill listen for connections. + local-filename + Required for file-listener-sockets. Specifies the nameof a file in the local filesystem which is used to name aUNIX-domain socket. The actual filesystem file should notpreviously exist when the file-listener-socket is created;its parent directory should exist and be writable by thecaller. The file used to name the socket will be deletedwhen the file-listener-socket is closed. + keepalive + If true, enables the periodic transmission of"keepalive" messages. + reuse-address + If true, allows the reuse of local ports in listenersockets, overriding some TCP/IP protocol specifications. Youwill need this if you are debugging a server.. + nodelay + If true, disables Nagle's algorithm, which triesto minimize TCP packet fragmentation by introducingtransmission delays in the absence of replies. Try settingthis if you are using a protocol which involves sending asteady stream of data with no replies and are seeingsignificant degradations in throughput. + broadcast + If true, requests permission to broadcast datagrams ona UDP socket. + linger + If specified and non-nil, should be the number ofseconds the OS is allowed to wait for data to be pushedthrough when a close is done. Only relevant for TCP sockets. + backlog + For a listener socket, specifies the number ofconnections which can be pending but not accepted. Thedefault is 5, which is also the maximum on some operatingsystems. + + + Description + Creates and returns a new socket + + + + ACCEPT-CONNECTION + accept-connection + Name + ACCEPT-CONNECTION — + Function + Synopsis + +accept-connection + (socket listener-socket) &key wait + + Arguments and Values + socket + The listener-socket to listen on. + wait + If true (the default), and there are no connectionswaiting to be accepted, waits until one arrives. If false,returns NIL immediately. + + + Description + Extracts the first connection on the queue of pending +connections, accepts it (i.e. completes the connection startup +protocol) and returns a new tcp-stream or file-socket-stream +representing the newly established connection. The tcp stream +inherits any properties of the listener socket that are relevant +(e.g. :keepalive, :nodelay, etc.) The original listener socket +continues to be open listening for more connections, so you can +call accept-connection on it again. + + + + DOTTED-TO-IPADDR + dotted-to-ipaddr + Name + DOTTED-TO-IPADDR — + Function + Synopsis + +dotted-to-ipaddr + dotted &key errorp + + Arguments and Values + dotted + A string representing an IP address in the"nn.nn.nn.nn" format + errorp + If true (the default) an error is signaled if dottedis invalid. If false, NIL is returned. + + + Description + Converts a dotted-string representation of a host address to +a 32-bit unsigned IP address. + + + + IPADDR-TO-DOTTED + ipaddr-to-dotted + Name + IPADDR-TO-DOTTED — + Function + Synopsis + +ipaddr-to-dotted + ipaddr &key values + + Arguments and Values + ipaddr + A 32-bit integer representing an internet host address + values + If false (the default), returns a string in the form"nn.nn.nn.nn". If true, returns four valuesrepresenting the four octets of the address as unsigned8-bit integers. + + + Description + Converts a 32-bit unsigned IP address into octets. + + + + IPADDR-TO-HOSTNAME + ipaddr-to-hostname + Name + IPADDR-TO-HOSTNAME — + Function + Synopsis + +ipaddr-to-hostname + ipaddr &key ignore-cache + + Arguments and Values + ipaddr + a 32-bit integer representing an internet host address + ignore-cache + This argument is ignored (it is accepted forcompatibility with Franz Allegro) + + + Description + Converts a 32-bit unsigned IP address into a host name +string + + + + LOOKUP-HOSTNAME + lookup-hostname + Name + LOOKUP-HOSTNAME — + Function + Synopsis + +lookup-hostname + host + + Arguments and Values + host + Specifies the host. It can be either a host namestring such as "clozure.com", or a dotted addressstring such as "192.168.0.1", or a 32-bit unsignedIP address such as 3232235521. + + + Description + Converts a host spec in any of the acceptable formats into a +32-bit unsigned IP address + + + + LOOKUP-PORT + lookup-port + Name + LOOKUP-PORT — + Function + Synopsis + +lookup-port + port protocol + + Arguments and Values + port + Specifies the port. It can be either a string, such as"http" or a symbol, such as :http, or an unsignedport number. Note that a string is case-sensitive. A symbolis lowercased before lookup. + protocol + Must be one of "tcp" or "udp". + + + Description + Finds the port number for the specified port and protocol + + + + RECEIVE-FROM + receive-from + Name + RECEIVE-FROM — + Function + Synopsis + +receive-from + (socket udp-socket) size &key buffer + extract offset + + Arguments and Values + socket + The socket to read from + size + Maximum number of bytes to read. If the packet islarger than this, any extra bytes are discarded. + buffer + If specified, must be either a string or a byte vectorwhich will be used to read in the data. If not specified, anew buffer will be created (of type determined bysocket-format). + extract + If true, the subsequence of the buffer correspondingonly to the data read in is extracted and returned as thefirst value. If false (the default) the original buffer isreturned even if it is only partially filled. + offset + Specifies the start offset into the buffer at whichdata is to be stored. The default is 0. + + + Description + Reads a UDP packet from a socket. If no packets are +available, waits for a packet to arrive. Returns four values: + + The buffer with the data + The number of bytes read + The 32-bit unsigned IP address of the sender of the data + The port number of the sender of the data + + + + + SEND-TO + send-to + Name + SEND-TO — + Function + Synopsis + +send-to + (socket udp-socket) buffer size &key remote-host + remote-port offset + + Arguments and Values + socket + The socket to write to + buffer + A vector containing the data to send. It must beeither a string or a byte vector (either one is acceptableregardless of the stream format). + size + Number of bytes to send + remote-host + The host to send the packet to, in any formatacceptable to lookup-hostname. The default is the remotehost specified in the call to make-socket. + remote-port + The port to send the packet to, in any formatacceptable to lookup-port. The default is the remote portspecified in the call to make-socket. + offset + The offset in the buffer where the packet data starts + + + Description + Send a UDP packet over a socket. + + + + SHUTDOWN + shutdown + Name + SHUTDOWN — + Function + Synopsis + +shutdown + socket &key direction + + Arguments and Values + socket + The socket to shut down (typically a tcp-stream) + direction + One of :input to disallow further input, or :output todisallow further output. + + + Description + Shuts down part of a bidirectional connection. This is +useful if e.g. you need to read responses after sending an +end-of-file signal. + + + + SOCKET-OS-FD + socket-os-fd + Name + SOCKET-OS-FD — + Function + Synopsis + +socket-os-fd + socket + + Arguments and Values + socket + The socket + + + Description + Returns the native OS's representation of the socket, or +NIL if the socket is closed. On Unix, this is the Unix 'file +descriptor', a small non-negative integer. Note that it is +rather dangerous to mess around with tcp-stream fd's, as there +is all sorts of buffering and asynchronous I/O going on above the +OS level. listener-socket and udp-socket fd's are safer to +mess with directly as there is less magic going on. + + + + REMOTE-HOST + remote-host + Name + REMOTE-HOST — + Function + Synopsis + +remote-host + socket + + Arguments and Values + socket + The socket + + + Description + Returns the 32-bit unsigned IP address of the remote host, +or NIL if the socket is not connected. + + + + REMOTE-PORT + remote-port + Name + REMOTE-PORT — + Function + Synopsis + +remote-port + socket + + Arguments and Values + socket + The socket + + + Description + Returns the remote port number, or NIL if the socket is not +connected. + + + + LOCAL-HOST + local-host + Name + LOCAL-HOST — + Function + Synopsis + +local-host + socket + + Arguments and Values + socket + The socket + + + Description + Returns 32-bit unsigned IP address of the local host. + + + + LOCAL-PORT + local-port + Name + LOCAL-PORT — + Function + Synopsis + +local-port + socket + + Arguments and Values + socket + The socket + + + Description + Returns the local port number + + + + SOCKET-ADDRESS-FAMILY + socket-address-family + Name + SOCKET-ADDRESS-FAMILY — + Function + Synopsis + +socket-address-family + socket + + Arguments and Values + socket + The socket + + + Description + Returns :internet or :file, as appropriate. + + + + SOCKET-CONNECT + socket-connect + Name + SOCKET-CONNECT — + Function + Synopsis + +socket-connect + socket + + Arguments and Values + socket + The socket + + + Description + Returns :active for tcp-stream, :passive for +listener-socket, and NIL for udp-socket + + + + SOCKET-FORMAT + socket-format + Name + SOCKET-FORMAT — + Function + Synopsis + +socket-format + socket + + Arguments and Values + socket + The socket + + + Description + Returns the socket format as specified by the :format +argument to make-socket. + + + + SOCKET-TYPE + socket-type + Name + SOCKET-TYPE — + Function + Synopsis + +socket-type + socket + + Arguments and Values + socket + The socket + + + Description + returns :stream for tcp-stream and listener-socket, and +:datagram for udp-socket. + + + + SOCKET-ERROR + socket-error + Name + SOCKET-ERROR — + Class + Description + The class of OS errors signaled by socket functions + Superclasses + simple-error + + + + SOCKET-ERROR-CODE + socket-error-code + Name + SOCKET-ERROR-CODE — + Function + Synopsis + +socket-error-code + socket-error + + Arguments and Values + socket-error + the condition + + + Description + The OS error code of the error + + + + SOCKET-ERROR-IDENTIFIER + socket-error-identifier + Name + SOCKET-ERROR-IDENTIFIER — + Function + Synopsis + +socket-error-identifier + socket-error + + Arguments and Values + socket-error + the condition + + + Description + A symbol representing the error code in a more +OS-independent way. + One of: :address-in-use :connection-aborted :no-buffer-space +:connection-timed-out :connection-refused :host-unreachable +:host-down :network-down :address-not-available :network-reset +:connection-reset :shutdown :access-denied or :unknown. + + + + SOCKET-ERROR-SITUATION + socket-error-situation + Name + SOCKET-ERROR-SITUATION — + Function + Synopsis + +socket-error-situation + socket-error + + Arguments and Values + socket-error + the condition + + + Description + A string describing the context where the error happened. On +Linux, this is the name of the system call which returned the +error. + + + + CLOSE + close + Name + CLOSE — + Method + Synopsis + +close + (socket socket) &key abort + + Arguments and Values + socket + The socket to close + abort + If false (the default), closes the socket in anorderly fashion, finishing up any buffered pending I/O,before closing the connection. If true, aborts/ignorespending I/O. (For listener and udp sockets, this argument iseffectively ignored since there is never any buffered I/O toclean up). + + + Description + The close generic function can be applied to sockets. It +releases the operating system resources associated with the +socket. + + + + WITH-OPEN-SOCKET + with-open-socket + Name + WITH-OPEN-SOCKET — + Macro + Synopsis + +with-open-socket + (var . make-socket-args) &body body + + Arguments and Values + var + variable to bind + make-socket-args + arguments suitable for passing to make-socket + body + body to execute + + + Description + executes body with var bound to the result of applying +make-socket to make-socket-args. The socket gets closed on exit. + + + + + + Running Other Programs as Subprocesses + + + Overview + OpenMCL provides primitives to run external Unix programs, + to select and connect Lisp streams to their input and output + sources, to (optionally) wait for their completion and to check + their execution and exit status. + All of the global symbols described below are exported + from the CCL package. + This implementation is modeled on - and uses some code + from - similar facilities in CMUCL. + + + + Examples + +;;; Capture the output of the "uname" program in a lisp string-stream +;;; and return the generated string (which will contain a trailing +;;; newline.) +? (with-output-to-string (stream) + (run-program "uname" '("-r") :output stream)) +;;; Write a string to *STANDARD-OUTPUT*, the hard way. +? (run-program "cat" () :input (make-string-input-stream "hello") :output t) +;;; Find out that "ls" doesn't expand wildcards. +? (run-program "ls" '("*.lisp") :output t) +;;; Let the shell expand wildcards. +? (run-program "sh" '("-c" "ls *.lisp") :output t) + + These last examples will only produce output if OpenMCL's + current directory contains .lisp files, of course. + + + + Limitations and known bugs + + OpenMCL and the external processs may get + confused about whoowns which streams when input, output, or + error are specified as T and wait is specified as + NIL. + External processes that need to talk to a + terminal device may not work properly; the environment (SLIME, + ILISP) under which OpenMCL is run can affect + this. + + + + + + External-Program Dictionary + + + RUN-PROGRAM + run-program + Name + RUN-PROGRAM — Invokes an external program as an OS subprocess +of lisp. + Function + Synopsis + +run-program + program args &key (wait t) pty input + if-input-does-not-exist output (if-output-exists :error) (error + :output) (if-error-exists :error) status-hook + + Arguments and Values + program + A string or pathname which denotes an executable file.The PATH environment variable is used to find programs whosename doesn't contain a directory component. + args + A list of simple-strings + wait + Indicates whether or not run-program should wait forthe EXTERNAL-PROCESS to complete or should returnimmediately. + pty + This option is accepted but currently ignored;it's intended to make it easier to run external programsthat need to interact with a terminal device. + input + Selects the input source used by the EXTERNAL-PROCESS.May be any of the following: + + NIL Specifies that a null input stream (e.g.,/dev/null) should be used. + T Specifies that the EXTERNAL-PROCESS should usethe input source with which OpenMCL was invoked. + A string or pathname. Specifies that theEXTERNAL-PROCESS should receive its input from the namedexisting file. + :STREAM Creates a Lisp stream opened for characteroutput. Any data written to this stream (accessible asthe EXTERNAL-PROCESS-INPUT-STREAM of theEXTERNAL-PROCESS object) appears as input to theexternal process. + A stream. Specifies that the lisp stream shouldprovide input to the EXTERNAL-PROCESS. + + + + if-input-does-not-exist + If the input argument specifies the name of anexisting file, this argument is used as theif-does-not-exist argument to OPEN when that file is opened. + output + Specifies where standard output from the externalprocess should be sent. Analogous to input above. + if-output-exists + If output is specified as a string or pathname, thisargument is used as the if-exists argument to OPEN when thatfile is opened. + error + Specifies where error output from the external processshould be sent. In addition to the values allowed foroutput, the keyword :OUTPUT can be used to indicate thaterror output should be sent where standard output goes. + if-error-exists + Analogous to if-output-exists. + status-hook + A user-defined function of one argument (theEXTERNAL-PROCESS structure.) This function is calledwhenever OpenMCL detects a change in the staus of theEXTERNAL-PROCESS. + + + Description + Runs the specified program in an external (Unix) process, +returning an object of type EXTERNAL-PROCESS if successful. + + + + SIGNAL-EXTERNAL-PROCESS + signal-external-process + Name + SIGNAL-EXTERNAL-PROCESS — + Function + Synopsis + +signal-external-process + proc signal-number + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + signal + A small integer. + + + Description + Sends the specified "signal" to the specified +external process. (Typically, it would only be useful tocall +this function if the EXTERNAL-PROCESS was created with :WAIT +NIL. ) Returns T if successful; signals an error otherwise. + + + + EXTERNAL-PROCESS-ID + external-process-id + Name + EXTERNAL-PROCESS-ID — Returns the "process ID" of an OS subprocess, +a positive integer which identifies it. + Function + Synopsis + +external-process-id + proc + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + + + Description + Returns the process id assigned to +the external process by the operating system. This is typically +a positive, 16-bit number. + + + + EXTERNAL-PROCESS-INPUT-STREAM + external-process-input-stream + Name + EXTERNAL-PROCESS-INPUT-STREAM — Returns the lisp stream which is used to write +input to a given OS subprocess, if it has one. + Function + Synopsis + +external-process-input-stream + proc + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + + + Description + Returns the stream created when the input argument to +run-program is specified as :STREAM. + + + + EXTERNAL-PROCESS-OUTPUT-STREAM + external-process-output-stream + Name + EXTERNAL-PROCESS-OUTPUT-STREAM — Returns the lisp stream which is used to read +output from an OS subprocess, if there is one. + Function + Synopsis + +external-process-output-stream + proc + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + + + Description + Returns the stream created when the output argument to +run-program is specified as :STREAM. + + + + EXTERNAL-PROCESS-ERROR-STREAM + external-process-error-stream + Name + EXTERNAL-PROCESS-ERROR-STREAM — Returns the stream which is used to read +"error" output from a given OS subprocess, if it has +one. + Function + Synopsis + +external-process-error-stream + proc + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + + + Description + Returns the stream created when the error argument to +run-program is specified as :STREAM. + + + + EXTERNAL-PROCESS-STATUS + external-process-status + Name + EXTERNAL-PROCESS-STATUS — Returns information about whether an OS +subprocess is running; or, if not, why not; and what its +result code was if it completed. + Function + Synopsis + +external-process-status + proc + + Arguments and Values + proc + An EXTERNAL-PROCESS, as returned by RUN-PROGRAM. + + + Description + Returns, as multiple values, a keyword denoting the status +of the external process (one of :running, :stopped, :signaled, or +:exited), and the exit code or terminating signal if the first +value is other than :running. + + + + + + Creating Your Own Stream Classes with Gray Streams + + + Overview + This chapter is still being written and revised, because + it is woefully incomplete. The dictionary section currently + only lists a couple functions. Caveat lector. + Gray streams are an extension to Common Lisp. They were + proposed for standardization by David Gray (the astute reader + now understands their name) quite some years ago, but not + accepted, because they had not been tried sufficiently to find + conceptual problems with them. + They have since been implemented by quite a few modern + Lisp implementations. However, they do indeed have some + inadequacies, and each implementation has addressed these in + different ways. The situation today is that it's difficult to + even find out how to get started using Gray streams. This is + why standards are important. + Here's a list of some classes which you might wish for + your new stream class to inherit from: + + + + + + fundamental-stream + + + + fundamental-input-stream + + + + fundamental-output-stream + + + + fundamental-character-stream + + + + fundamental-binary-stream + + + + fundamental-character-input-stream + + + + fundamental-character-output-stream + + + + fundamental-binary-input-stream + + + + fundamental-binary-output-stream + + + + ccl::buffered-stream-mixin + + + + ccl::buffered-input-stream-mixin + + + + ccl::buffered-output-stream-mixin + + + + ccl::buffered-io-stream-mixin + + + + ccl::buffered-character-input-stream-mixin + + + + ccl::buffered-character-output-stream-mixin + + + + ccl::buffered-character-io-stream-mixin + + + + ccl::buffered-binary-input-stream-mixin + + + + ccl::buffered-binary-output-stream-mixin + + + + ccl::buffered-binary-io-stream-mixin + + + + file-stream + + + + file-input-stream + + + + file-output-stream + + + + file-io-stream + + + + file-character-input-stream + + + + file-character-output-stream + + + + file-charcter-io-stream + + + + file-binary-input-stream + + + + file-binary-output-stream + + + + file-binary-io-stream + + + + ccl::fd-stream + + + + ccl::fd-input-stream + + + + ccl::fd-output-stream + + + + ccl::fd-io-stream + + + + ccl::fd-character-input-stream + + + + ccl::fd-character-output-stream + + + + ccl::fd-character-io-stream + + + + ccl::fd-binary-input-stream + + + + ccl::fd-binary-output-stream + + + + ccl::fd-binary-io-stream + + + + + + All of these are defined in ccl/level-1/l1-streams.lisp, except for +the ccl:file-* ones, which are in ccl/level-1/l1-sysio.lisp. + According to the original Gray streams proposal, you should inherit +from the most specific of the +fundamental-* classes which applies. Using OpenMCL, though, if you +want +buffering for better performance, which, unless you know of some +reason you wouldn't, you do, +you should instead inherit from the appropriate ccl::buffered-* class +The buffering you get this way is exactly the same as the +buffering which is used on ordinary, non-Gray streams, and force-output +will work properly on it. + Notice that -mixin suffix in the names of all the ccl::buffered-* +classes? +The suffix means that this class +is not "complete" +by itself; you still need to inherit from a fundamental-* stream, +even if you also inherit from a *-mixin stream. You might consider +making your own class like this. .... Except that they do +inherit from the fundamental-* streams, that's weird. + If you want to be able to create an instance of your class with the +:class argument to (open) and (with-open-file), you should make it +inherit from one of the file-* classes. If you do this, it's not +necessary to inherit from any of the other classes (though it won't +hurt anything), since the file-* classes already do. + When you inherit from the file-* classes, you can use +(call-next-method) in any of your methods to get the standard +behaviour. This is especially useful if you want to create a class +which performs some simple filtering operation, such +as changing everything to uppercase or to a different character +encoding. If you do this, you will definitely need to specialize +ccl::select-stream-class. Your method on ccl::stream-select-class +should accept an instance of the class, but pay no attention to its +contents, and return a symbol naming the +class to actually be instantiated. + If you need to make your functionality generic across all the +different types of stream, probably the best way +to implement it is to make it a mixin, define classes with all the +variants of input, output, io, character, and binary, which inherit +both from your mixin and from the appropriate other class, then +define a method on ccl::select-stream-class which chooses from among +those classes. + Note that some of these classes are internal +to the CLL package. If you try to inherit from those ones without +the ccl:: prefix, you'll get an error which may confuse you, calling +them "forward-referenced classes". That just means you used the +wrong symbol, so add the prefix. + Here's a list of some generic functions which you might wish to +specialize for your new stream class, and which ought to be +documented at some point. + + + + + + stream-direction stream => + + + + stream-device stream direction => + + + + stream-length stream &optionalnew => + + + + stream-position stream &optionalnew => + + + + streamp stream => boolean + + + + stream-write-char output-stream char => + + + + stream-write-entire-string output-stream string => + + + + stream-read-char input-stream => + + + + stream-unread-char input-stream char => + + + + stream-force-output output-stream => nil + + + + stream-maybe-force-output output-stream => nil + + + + stream-finish-output output-stream => nil + + + + stream-clear-output output-stream => nil + + + + close stream &keyabort => boolean + + + + stream-fresh-line stream => t + + + + stream-line-length stream => length + + + + interactive-stream-p stream => boolean + + + + stream-clear-input input-stream => nil + + + + stream-listen input-stream => boolean + + + + stream-filename stream => string + + + + ccl::select-stream-class instance in-p out-p char-p =>class + + + + + + The following functions are standard parts of Common Lisp, but +behave in special ways with regard to Gray streams. + + + + + + open-stream-p stream => generalized-boolean + + + + input-stream-p stream => generalized-boolean + + + + output-stream-p stream => generalized-boolean + + + + stream-element-type stream => + + + + stream-error-stream => + + + + open + + + + close + + + + with-open-file + + + + + + Specifically, (open) and (with-open-file) accept a new keyword +argument, :class, which may be a symbol naming a class; the class +itself; or an instance of it. The class so given must be a subtype +of 'stream, and an instance of it with no particular contents will +be passed to ccl::select-stream-class to determine what class to +actually instantiate. + The following are standard, and do not behave specially with +regard to Gray streams, but probably should. + + + + + + stream-external-format + + + + + + + + + Extending READ-SEQUENCE and WRITE-SEQUENCE + + + Overview + The "Gray Streams" API is based on an informal proposal that was + made before ANSI CL adopted the READ-SEQUENCE and WRITE-SEQUENCE + functions; as such, there is no "standard" way for the author of a Gray + stream class to improve the performance of these functions by exploiting + knowledge of the stream's internals (e.g., the buffering mechanism it + uses.) + In the absence of any such knowledge, READ-SEQUENCE and + WRITE-SEQUENCE are effectively just convenient shorthand for a + loop which calls READ-CHAR/READ-BYTE/WRITE-CHAR/WRITE-BYTE as + appropriate. The mechanism described below allows subclasses + of FUNDAMENTAL-STREAM to define more specialized (and + presumably more efficient) behavior. + + + + Notes + READ-SEQUENCE and WRITE-SEQUENCE do a certain amount of + sanity-checking and normalization of their arguments before + dispatching to one of the methods above. If an individual + method can't do anything particularly clever, CALL-NEXT-METHOD + can be used to handle the general case. + + + + Example + +(defclass my-string-input-stream (fundamental-character-input-stream) + ((string :initarg :string :accessor my-string-input-stream-string) + (index :initform 0 :accessor my-string-input-stream-index) + (length))) + +(defmethod stream-read-vector ((stream my-string-input-stream) vector start end) + (if (not (typep vector 'simple-base-string)) + (call-next-method) + (with-slots (string index length) + (do* ((outpos start (1+ outpos))) + ((or (= outpos end) + (= index length)) + outpos)) + (setf (schar vector outpos) + (schar string index)) + (incf index))))) + + + + + + Multibyte I/O + All heap-allocated objects in OpenMCL that cannot contain + pointers to lisp objects are represented as + ivectors. OpenMCL provides low-level + functions, and , to efficiently transfer data between buffered + streams and ivectors. There's some overlap in functionality + between the functions described here and the ANSI CL + READ-SEQUENCE and WRITE-SEQUENCE functions. + As used here, the term "octet" means roughly the same + thing as the term "8-bit byte". The functions described below + transfer a specified sequence of octets between a buffered + stream and an ivector, and don't really concern themselves with + higher-level issues (like whether that octet sequence is within + bounds or how it relates to the logical contents of the + ivector.) For these reasons, these functions are generally less + safe and more flexible than their ANSI counterparts. + + + + Gray Streams Dictionary + + + CCL:STREAM-READ-LIST + stream-read-list + Name + CCL:STREAM-READ-LIST — + Generic Function + Synopsis + +stream-read-list + stream list count + + Arguments and Values + stream + a stream, presumably a fundamental-input-stream. + list + a list. When a STREAM-READ-LIST method is called byREAD-SEQUENCE, this argument is guaranteed to be a properlist. + count + a non-negative integer. When a STREAM-READ-LIST methodis called by READ-SEQUENCE, this argument is guaranteed notto be greater than the length of the list. + + + Description + Should try to read up to count elements from stream into the +list list, returning the number of elements actually read (which +may be less than count in case of a premature end-of-file.) + + + + CCL:STREAM-WRITE-LIST + stream-write-list + Name + CCL:STREAM-WRITE-LIST — + Generic Function + Synopsis + +stream-write-list + stream list count + + Arguments and Values + stream + a stream, presumably a fundamental-ouput-stream. + list + a list. When a STREAM-WRITE-LIST method is called byWRITE-SEQUENCE, this argument is guaranteed to be a properlist. + count + a non-negative integer. When a STREAM-WRITE-LISTmethod is called by WRITE-SEQUENCE, this argument isguaranteed not to be greater than the length of the list. + + + Description + should try to write the first count elements of list to +stream. The return value of this method is ignored. + + + + CCL:STREAM-READ-VECTOR + stream-read-vector + Name + CCL:STREAM-READ-VECTOR — + Generic Function + Synopsis + +stream-read-vector + stream vector start end + + Arguments and Values + stream + a stream, presumably a fundamental-input-stream + vector + a vector. When a STREAM-READ-VECTOR method is calledby READ-SEQUENCE, this argument is guaranteed to be a simpleone-dimensional array. + start + a non-negative integer. When a STREAM-READ-VECTORmethod is called by READ-SEQUENCE, this argument isguaranteed to be no greater than end and not greater thanthe length of vector. + end + a non-negative integer. When a STREAM-READ-VECTORmethod is called by READ-SEQUENCE, this argument isguaranteed to be no less than end and not greater than thelength of vector. + + + Description + should try to read successive elements from stream into +vector, starting at element start (inclusive) and continuing +through element end (exclusive.) Should return the index of the +vector element beyond the last one stored into, which may be less +than end in case of premature end-of-file. + + + + CCL:STREAM-WRITE-VECTOR + stream-write-vector + Name + CCL:STREAM-WRITE-VECTOR — + Generic Function + Synopsis + +stream-write-vector + stream vector start end + + Arguments and Values + stream + a stream, presumably a fundamental-output-stream + vector + a vector. When a STREAM-WRITE-VECTOR method is calledby WRITE-SEQUENCE, this argument is guaranteed to be asimple one-dimensional array. + start + a non-negative integer. When a STREAM-WRITE-VECTORmethod is called by WRITE-SEQUENCE, this argument isguaranteed to be no greater than end and not greater thanthe length of vector. + end + a non-negative integer. When a STREAM-WRITE-VECTORmethod is called by WRITE-SEQUENCE, this argument isguaranteed to be no less than end and not greater than thelength of vector. + + + Description + should try to write successive elements of vector to stream, +starting at element start (inclusive) and continuing through +element end (exclusive.) + + + + CCL::STREAM-DEVICE + stream-device + Name + CCL::STREAM-DEVICE — Returns the OS file descriptor associated with a +given lisp stream. + Generic Function + Synopsis + +ccl::stream-device + s direction + + Method Signatures + +ccl::stream-device + (s stream) direction => fd + + Arguments and Values + s + a stream. + direction + either :INPUT or :OUTPUT. + fd + a file descriptor, which is a non-negative integerused by the OS to refer to an open file, socket, or similarI/O connection. NIL if there is no file descriptor associatedwith s in the direction given bydirection. + + + Description + Returns the file descriptor associated with +s in the direction given by +direction. It is necessary to specify +direction because the input and output +file descriptors may be different; the most common case is when +one of them has been redirected by the Unix shell. + + + + STREAM-READ-IVECTOR + stream-read-ivector + Name + STREAM-READ-IVECTOR — + Generic Function + Synopsis + +stream-read-ivector + stream ivector start-octet max-octets + + Description + Reads up to max-octets octets from stream into ivector, +storing them at start-octet. Returns the number of octets actually +read. + Arguments + stream + An input stream. The method defined onBUFFERED-INPUT-STREAMs requires that the size in octets ofan instance of the stream's element type is 1. + ivector + Any ivector. + start-octet + A non-negative integer. + max-octets + A non-negative integer. The return value may be lessthan the value of this parameter if EOF was encountered. + + + + + + STREAM-WRITE-IVECTOR + stream-write-ivector + Name + STREAM-WRITE-IVECTOR — + Generic Function + Synopsis + +stream-write-ivector stream + ivector start-octet max-octets + + Description + Writes max-octets octets to stream from ivector, starting at +start-octet. Returns max-octets. + Arguments + stream + An input stream. The method defined onBUFFERED-OUTPUT-STREAMs requires that the size in octets ofan instance of the stream's element type is 1. + ivector + Any ivector + start-octet + A non-negative integer. + max-octet + A non-negative integer. + + + Examples + +;;; Write the contents of a (SIMPLE-ARRAY(UNSIGNED-BYTE 16) 3) +;;; to a character file stream. Read back the characters. +(let* ((a (make-array 3 + :element-type '(unsigned-byte 16) + :initial-contents '(26725 27756 28449)))) + (with-open-file (s "junk" + :element-type 'character + :direction :io + :if-does-not-exist :create + :if-exists :supersede) + ;; Write six octets (three elements). + (stream-write-ivector s a 0 6) + ;; Rewind, then read a line + (file-position s 0) + (read-line s))) + +;;; Write a vector of DOUBLE-FLOATs. Note that (to maintain +;;; alignment) there are 4 octets of padding before the 0th +;;; element of a (VECTOR DOUBLE-FLOAT) in 32-bit OpenMCL. +;;; (Note that (= (- ppc32::misc-dfloat-offset +;;; ppc32::misc-data-offset) 4)) +(defun write-double-float-vector + (stream vector &key (start 0) (end (length vector))) + (check-type vector (vector double-float)) + (let* ((start-octet (+ (* start 8) + (- target::misc-dfloat-offset + target::misc-data-offset))) + (num-octets (* 8 (- end start)))) + (stream-write-ivector stream vector start-octet num-octets))) + + + + + + + Writing Portable Extensions to the Object System using the MetaObject Protocol + + + Overview + OpenMCL supports a fairly large subset of the + semi-standard MetaObject Protocol (MOP) for CLOS, as defined in + chapters 5 and 6 of "The Art Of The Metaobject Protocol", + (Kiczales et al, MIT Press 1991, ISBN 0-262-61074-4); this + specification is also available online at + http://www.alu.org/mop/index.html. + + + + Implementation status + The keyword :openmcl-partial-mop is on *FEATURES* to + indicate the presence of this functionality. + + All of the symbols defined in the MOP specification + (whether implemented or not) are exported from the "CCL" package + and from an "OPENMCL-MOP" package. + constructstatusaccessor-method-slot-definition+add-dependent+add-direct-method+add-direct-subclass+add-method+class-default-initargs+class-direct-default-initargs+class-direct-slots+class-direct-subclasses+class-direct-superclasses+class-finalized-p+class-prototype+class-slots+compute-applicable-methods-compute-applicable-methods-using-classes-compute-class-precedence-list+compute-direct-initargs+compute-discriminating-function-compute-effective-method+compute-effective-slot-definition+compute-slots+direct-slot-definition-class+effective-slot-definition-class+ensure-class+ensure-class-using-class+ensure-generic-function-using-class+eql-specializer-object+extract-lambda-list+extract-specializer-names+finalize-inheritance+find-method-combination+funcallable-standard-instance-access+generic-function-argument-precedence-order+generic-function-declarations+generic-function-lambda-list+generic-function-method-class+generic-function-method-combination+generic-function-methods+generic-function-name+intern-eql-specializer+make-method-lambda-map-dependents+method-function+method-generic-function+method-lambda-list+method-qualifiers+method-specializers+reader-method-class+remove-dependent+remove-direct-method+remove-direct-subclass+remove-method+set-funcallable-instance-function-slot-boundp-using-class+slot-definition-allocation+slot-definition-initargs+slot-definition-initform+slot-definition-initfunction+slot-definition-location+slot-definition-name+slot-definition-readers+slot-definition-type+slot-definition-writers+slot-makunbound-using-class+slot-value-using-class+specializer-direct-generic-functions+specializer-direct-methods+standard-instance-access+update-dependent+validate-superclass+writer-method-class+ + + Note that those generic functions whose status is "-" in + the table above deal with the internals of generic function + dispatch and method invocation (the "Generic Function Invocation + Protocol".) Method functions are implemented a bit differently + in OpenMCL from what the MOP expects, and it's not yet clear if + or how this subprotocol can be well-supported. + Those constructs that are marked as "+" in the table above + are nominally implemented as the MOP document specifies + (deviations from the specification should be considered bugs; + please report them as such.) Note that some CLOS implementations + in widespread use (e.g., PCL) implement some things + (ENSURE-CLASS-USING-CLASS comes to mind) a bit differently from + what the MOP specifies. + + + + Concurrency issues + The entire CLOS class and generic function hierarchy is + effectively a (large, complicated) shared data structure; it's + not generally practical for a thread to request exclusive access + to all of CLOS, and the effects of volitional modification of + the CLOS hierarchy (via clas redefinition, CHANGE-CLASS, etc) in + a multithreaded environment aren't always tractable. + Native threads exacerbate this problem (in that they + increase the opportunities for concurrent modification and + access.) The implementation should try to ensure that a thread's + view of any subset of the CLOS hierarchy is consistent (to the + extent that that's possible) and should try to ensure that + incidental modifications of the hierarchy (cache updates, etc.) + happen atomically; it's not generally possible for the + implementation to guarantee that a thread's view of things is + correct and current. + If you are loading code and defining classes in the most + usual way, which is to say, via the compiler, using only a + single thread, these issues are probably not going to affect you + much. + If, however, you are making finicky changes to the class + hierarchy while you're running multiple threads which manipulate + objects related to each other, more care is required. Before + doing such a thing, you should know what you're doing and + already be aware of what precautions to take, without being + told. That said, if you do it, you should seriously consider + what your application's critical data is, and use locks for + critical code sections. + + + + + The Foreign-Function Interface + + + Specifying And Using Foreign Types + + + Overview + OpenMCL provides a fairly rich language for defining and + specifying foreign data types (this language is derived from + CMUCL's "alien type" system.) + In practice, most foreign type definitions are + introduced into OpenMCL via its interface database (see ), + though it's also possible to define foreign types + interactively and/or programmatically. + OpenMCL's foreign type system is "evolving" (a polite + word for not-quite-complete): there are some inconsistencies + involving package usage, for instance. Symbols used in foreign + type specifiers should be keywords, but + this convention isn't always enforced. + Foreign type, record, and field names are + case-sensitive; OpenMCL uses some escaping conventions (see ) + to allow keywords to be used to denote these names. + + + + Syntax of Foreign Type Specifiers + + + Some foreign types are builtin: keywords denote + primitive,builtin types such as the IEEE-double-float type + (denoted:DOUBLE-FLOAT), in much the same way as certain + symbols(CONS, FIXNUM,etc.) define primitive CL + types. + + + Constructors such as :SIGNED and :UNSIGNED can be + used to denotesigned and unsigned integer subtypes + (analogous to the CL typespecifiers SIGNED-BYTE and + UNSIGNED-BYTE.) :SIGNED is shorthand for(:SIGNED 32) and + :UNSIGNED is shorthand for (:UNSIGNED 32). + + + Aliases for other (perhaps more complicated) types + can bedefined via CCL:DEF-FOREIGN-TYPE (sort of like + CL:DEFTYPE or the Ctypedef facility). The type :CHAR is + defined as an alias for (:SIGNED8) on some platforms, as + (:UNSIGNED 8) on others. + + + The construct (:STRUCT name) + can be used torefer to a named structure type; (:UNION + name)can be used to refer to a named + union type. It isn't necessary toenumerate a structure or + union type's fields in order to refer tothe type. + + + If X is a valid foreign type + reference,then (:* X) denotes the + foreign type "pointerto X". By + convention, (:* T) denotes ananonymous pointer type, + vaguely equivalent to "void*" in C. + + + If a fieldlist is a list of lists, each of whose CAR + is a foreign field name (keyword) and whose CADR is a + foreign type specifier, then (:STRUCT + name ,@fieldlist) is adefinition of + the structure type name, + and (:UNION name ,@fieldlist) is a + definition of theunion type + name. Note that it's necessary + todefine a structure or union type in order to include + that type in a structure, union, or array, but only + necessary to "refer to" a strucure or union type in order + to define a type alias or a pointer type. + + + If X is a defined foreign type + , then (:array X &rest dims) + denotes the foreigntype "array of + X". Although multiplearray dimensions + are allowed by the :array constructor, + only single-dimensioned arrays are (at all) well-supported + in OpenMCL. + + + + + + + Foreign Function Calls + + + Overview + OpenMCL provides a number of constructs for calling + foreign functions from Lisp code (all of them based on the + function CCL:%FF-CALL). In many cases, OpenMCL's interface + translator (see ) provides information about the foreign + function's entrypoint name and argument and return types; this + enables the use of the #_ reader macro (described below), + which may be more concise and/or more readable than other + constructs. + OpenMCL also provides a mechanism for defining + callbacks: lisp functions which can be + called from foreign code. + There's no supported way to directly pass lisp data to + foreign functions: scalar lisp data must be coerced to an + equivalent foreign representatation, and lisp arrays (notably + strings) must be copied to non-GCed memory. + + + Type Designators for Arguments and Return Values + The types of foreign argument and return values in foreign + function calls and callbacks can be specified by any of the following +keywords: + + + :UNSIGNED-BYTE + + + The argument/return value is of type (UNSIGNED-BYTE 8) + + + + + :SIGNED-BYTE + + + The argument/return value is of type (SIGNED-BYTE 8) + + + + + :UNSIGNED-HALFWORD + + + The argument/return value is of type (UNSIGNED-BYTE 16) + + + + + :SIGNED-HALFWORD + + + The argument/return value is of type (SIGNED-BYTE 16) + + + + + :UNSIGNED-FULLWORD + + + The argument/return value is of type (UNSIGNED-BYTE 32) + + + + + :SIGNED-FULLWORD + + + The argument/return value is of type (SIGNED-BYTE 32) + + + + + :UNSIGNED-DOUBLEWORD + + + The argument/return value is of type (UNSIGNED-BYTE 64) + + + + + :SIGNED-DOUBLEWORD + + + The argument/return value is of type (SIGNED-BYTE 64) + + + + + :SINGLE-FLOAT + + + The argument/return value is of type SINGLE-FLOAT + + + + + :DOUBLE-FLOAT + + + The argument/return value is of type DOUBLE-FLOAT + + + + + :ADDRESS + + + The argument/return values + is a MACPTR. + + + + + :VOID + + + or NIL Not valid as an argument type specifier; specifies + that there is no meaningful return value + + + + + On some platforms, a small positive integer + N can also be used as an argument + specifier; it indicates that the corresponding argument is a + pointer to an N-word structure or union + which should be passed by value to the foreign + function. Exactly which foreign structures are passed + by value and how is very dependent on the Application + Binary Interface (ABI) of the platform; unless you're + very familar with ABI detatils (some of which are quite + baroque), it's often easier to let higher-level constructs + deal with these details. + + + + External Entrypoints and Named External Entrypoints + PowerPC machine instructions are always aligned on + 32-bit boundaries, so the two least significant bits of the + first instruction ("entrypoint") of a foreign function are + always 0. OpenMCL often represents an entrypoint address as + a fixnum that's binary-equivalent to the entrypoint address: + if E is an entrypoint address expressed + as a signed 32-bit integer, then (ash E + -2) is an equivalent fixnum representation of that + address. An entrypoint address can also be encapsulated in a + MACPTR (see FIXTHIS), but that's somewhat less efficient. + Although it's possible to use fixnums or macptrs to + represent entrypoint addresses, it's somewhat cumbersome to + do so. OpenMCL can cache the addresses of named external + functions in structure-like objects of type + CCL:EXTERNAL-ENTRY-POINT (sometimes abbreviated as EEP). + Through the use of LOAD-TIME-VALUE, compiled lisp functions + are able to reference EEPs as constants; the use of an + indirection allows OpenMCL runtime system to ensure that the + EEP's address is current and correct. + + + + + Return Conventions for C Structures + On some platforms, C functions that are defined to + return structures do so by reference: they actually + accept a first parameter of type "pointer to returned + struct/union" - which must be allocated by the caller - and + don't return a meaningful value. + Exactly how a C function that's + defined to return a foreign structure does so is dependent on + the ABI (and on the size ad composition of the structure/union + in many cases.) + + + + + Referencing and Using Foreign Memory Addresses + + + Overview + + + Basics + For a variety of technical reasons, it isn't generally + possible to directly reference arbitrary absolute addresses + (such as those returned by the C library function malloc(), + for instance) in OpenMCL. In OpenMCL (and in MCL), such + addresses need to be encapsulated in + objects of type CCL:MACPTR; one can think of a MACPTR as + being a specialized type of structure whose sole purpose is + to provide a way of referring to an underlying "raw" + address. + It's sometimes convenient to blur the distinction + between a MACPTR and the address it represents; it's + sometimes necessary to maintain that distiction. It's + important to remember that a MACPTR is (generally) a + first-class Lisp object in the same sense that a CONS cell + is: it'll get GCed when it's no longer possible to reference + it. The "lifetime" of a MACPTR doesn't generally have + anything to do with the lifetime of the block of memory its + address points to. + It might be tempting to ask "How does one obtain the + address encapsulated by a MACPTR ?". The answer to that + question is that one doesn't do that (and there's no way to + do that): addresses aren't first-class objects, and there's + no way to refer to one. + Two MACPTRs that encapsulate the same address are EQL + to each other. + There are a small number of ways to directly create a + MACPTR (and there's a fair amount of syntactic sugar built + on top of of those primitives.) These primitives will be + discussed in greater detail below, but they include: + + + + Creating a MACPTR with a specified address, usually + via thefunction CCL:%INT-TO-PTR. + + + Referencing the return valueof a foreign function + call (see )that's specified to return an address. + + + Referencing a memory location that's specified to + contain an address. + + + + All of these primitive MACPTR-creating operations are + usually open-coded by the compiler; it has a fairly good + notion of what low-level operations "produce" MACPTRs and + which operations "consume" the addresses that the + encapsulate, and will usually optimize out the introduction + of intermediate MACPTRs in a simple expression. + One consequence of the use of MACPTR objects to + encapsulate foreign addresses is that (naively) + every reference to a foreign address causes a + MACPTR to be allocated. + Consider a code fragment like the following: + +(defun get-next-event () + "get the next event from a hypothetical window system" + (loop + (let* ((event (#_get_next_window_system_event))) ; via an FF-CALL + (unless (null-event-p event) + (handle-event event))))) + + As this is written, each call to the (hypothetical) + foreign function #_get_next_window_system_event will return + a new MACPTR object. Ignoring for the sake of argument the + question of whether this code fragment exhibits a good way + to poll for external events (it doesn't), it's not hard to + imagine that this loop could execute several millon times + per second (producing several million MACPTRs per second.) + Clearly, the "naive" approach is impractical in many + cases. + + + + Stack allocation of - and destructive operations on - MACPTRs. + If certain conditions held in the environment in which + GET-NEXT-EVENT ran - namely, if it was guaranteed that + neither NULL-EVENT-P nor HANDLE-EVENT cached or otherwise + retained their arguments (the "event" pointer) - there'd be + a few alternatives to the naive approach. One of those + approaches would be to use the primitive function + %SETF-MACPTR (described in greater detail below) to + destructively modify a MACPTR (to change the value of the + address it encapsulates.) The GET-NEXT-EVENT example could + be re-written as: + +(defun get-next-event () + (let* ((event (%int-to-ptr 0))) ; create a MACPTR with address 0 + (loop + (%setf-macptr event (#_get_next_window_system_event)) ; re-use it + (unless (null-event-p event) + (handle-event event))))) + + That version's a bit more realistic: it allocates a + single MACPTR outside if the loop, then changes its address + to point to the current address of the hypothetical event + structure on each loop iteration. If there are a million + loop iterations per call to GET-NEXT-EVENT, we're allocating + a million times fewer MACPTRs per call; that sounds like a + Good Thing. + An Even Better Thing would be to advise the compiler + that the initial value (the null MACPTR) bound to the + variable event has dynamic extent (that value won't be + referenced once control leaves the extent of the binding of + that variable.) Common Lisp allows us to make such an + assertion via a DYNAMIC-EXTENT declaration; OpenMCL's + compiler can recognize the "primitive MACPTR-creating + operation" involved and can replace it with an equivalent + operation that stack-allocates the MACPTR object. If we're + not worried about the cost of allocating that MACPTR on + every iteration (the cost is small and there's no hidden GC + cost), we could move the binding back inside the + loop: + +(defun get-next-event () + (loop + (let* ((event (%null-ptr))) ; (%NULL-PTR) is shorthand for (%INT-TO-PTR 0) + (declare (dynamic-extent event)) + (%setf-macptr event (#_get_next_window_system_event)) + (unless (null-event-p event) + (handle-event event))))) + + The idiom of binding one or more variables to + stack-allocated MACPTRs, then destructively modifying those + MACPTRs before executing a body of code is common enough + that OpenMCL provides a macro (WITH-MACPTRS) that handles + all of the gory details. The following version of + GET-NEXT-EVENT is semantically equivalent to the previous + version, but hopefully a bit more concise: + +(defun get-next-event () + (loop + (with-macptrs ((event (#_get_next_window_system_event))) + (unless (null-event-p event) + (handle-event event))))) + + + + + Stack-allocated memory (and stack-allocated pointers to it.) + Fairly often, the blocks of foreign memory (obtained + by malloc or something similar) have well-defined lifetimes + (they can safely be freed at some point when it's known that + they're no longer needed and it's known that they're no + longer referenced.) A common idiom might be: + +(with-macptrs (p (#_allocate_foreign_memory size)) + (unwind-protect + (use-foreign-memory p) + (#_deallocate_foreign_memory p))) + + That's not unreasonable code, but it's fairly + expensive for a number of reasons: foreign functions calls + are themselves fairly expensive (as is UNWIND-PROTECT), and + most library routines for allocating and deallocating + foreign memory (things like malloc and free) can be fairly + expensive in their own right. + In the idiomatic code above, both the MACPTR P and the + block of memory that's being allocated and freed have + dynamic extent and are therefore good candidates for stack + allocation. OpenMCL provides the %STACK-BLOCK macro, which + executes a body of code with one or more variables bound to + stack-allocated MACPTRs which encapsulate the addresses of + stack-allocated blocks of foreign memory. Using + %STACK-BLOCK, the idiomatic code is: + +(%stack-block ((p size)) + (use-foreign-memory p)) + + which is a bit more efficient and a bit more concise + than the version presented earlier. + %STACK-BLOCK is used as the basis for slightly + higher-level things like RLET. (See FIXTHIS for more information + about RLET.) + + + + Caveats. + Reading from, writing to, allocating, and freeing + foreign memory are all potentially dangerous operations; + this is no less true when these operations are performed in + OpenMCL than when they're done in C or some other + lower-level language. In addition, destructive operations on + Lisp objects be dangerous, as can stack allocation if it's + abused (if DYNAMIC-EXTENT declarations are violated.) + Correct use of the constructs and primitives described here + is reliable and safe; slightly incorrect use of these + constructs and primitives can crash OpenMCL. + + + + + Foreign-Memory-Addresses Dictionary + Unless otherwise noted, all of the symbols mentioned + below are exported from the CCL package. + + + Scalar memory reference + + + Syntax + + + %get-signed-byte ptr &optional (offset 0) + + %get-unsigned-byte ptr &optional (offset 0) + + %get-signed-word ptr &optional (offset 0) + + %get-unsigned-word ptr &optional (offset 0) + + %get-signed-long ptr &optional (offset 0) + + %get-unsigned-long ptr &optional (offset 0) + + %%get-signed-longlong ptr &optional (offset 0) + + %%get-unsigned-longlong ptr &optional (offset 0) + + %get-ptr ptr &optional (offset 0) + + %get-single-float ptr &optional (offset 0) + + %get-double-float ptr &optional (offset 0) + + + + Description + + + References and returns the signed or unsigned 8-bit byte, + signed or unsigned 16-bit word, signed or unsigned 32-bit long + word, signed or unsigned 64-bit long long word, 32-bit address, + 32-bit single-float, or 64-bit double-float at the effective byte + address formed by adding offset to the address encapsulated by + ptr. + + + + + Arguments + + + + + + ptr + + + A MACPTR + + + + + offset + + + A fixnum + + + + + + + + All of the memory reference primitives described above can be + used with SETF. + + + + %get-bit [Function] + Syntax + %get-bit ptr bit-offset + Description + References and returns the bit-offsetth bit at the addressencapsulated by ptr. (Bit 0 at a given address is the mostsignificant bit of the byte at that address.) Can be used withSETF. + Arguments + + ptr + A MACPTR + bit-offset + A fixnum + + + + + + + + %get-bitfield [Function] + Syntax + %get-bitfield ptr bit-offset width + Description + References and returns an unsigned integer composed from thewidth bits found bit-offsetbits from the address encapsulated byptr. (The least significant bit of the result is the value of(%get-bit ptr (1- (+ bit-offset width)))). Can be used with SETF. + Arguments + + ptr + A MACPTR + bit-offset + A fixnum + width + A positive fixnum + + + + + + + + %int-to-ptr [Function] + Syntax + %int-to-ptr int + Description + Creates and returns a MACPTR whose address matches int. + Arguments + + int + An (unsigned-byte 32) + + + + + + + + %inc-ptr [Function] + Syntax + %inc-ptr ptr &optional (delta 1) + Description + Creates and returns a MACPTR whose address is the address ofptr plus delta. The idiom (%inc-ptr ptr 0) is sometimes used tocopy a MACPTR, e.g., to create a new MACPTR encapsulating the sameaddress as ptr. + Arguments + + ptr + A MACPTR + delta + A fixnum + + + + + + + + %ptr-to-int [Function] + Syntax + %ptr-to-int ptr + Description + Returns the address encapsulated by ptr, as an(unsigned-byte 32). + Arguments + + ptr + A MACPTR + + + + + + + + %null-ptr [Macro] + Syntax + %null-ptr + Description + Equivalent to (%ptr-to-int 0). + + + + + + %null-ptr-p [Function] + Syntax + %null-ptr-p ptr + Description + Returns T If ptr is a MACPTR encapsulating the address 0,NIL if ptr encapsulates some other address. + Arguments + + ptr + A MACPTR + + + + + + + + %setf-macptr [Function] + Syntax + %setf-macptr dest-ptr src-ptr + Description + Causes dest-ptr to encapsulate the same address that src-ptrdoes, then returns dest-ptr. + Arguments + + dest-ptr + A MACPTR + src-ptr + A MACPTR + + + + + + + + %incf-ptr [Macro] + Syntax + %incf-ptr ptr &optional (delta 1) + Description + Destructively modifies ptr, by adding delta to the addressit encapsulates. Returns ptr. + Arguments + + ptr + A MACPTR + delta + A fixnum + + + + + + + + with-macptrs [Macro] + Syntax + with-macptrs (var expr)* &body body + Description + Executes body in an environment in which each var is boundto a stack-allocated macptr which encapsulates the foreign addressyielded by the corresponding expr. Returns whatever value(s) bodyreturns. + Arguments + + var + A symbol (variable name) + expr + A MACPTR-valued expression + + + + + + + + %stack-block [Macro] + Syntax + %stack-block (var expr)* &body body + Description + Executes body in an environment in which each var is boundto a stack-allocated macptr which encapsulates the address of astack-allocated region of size expr bytes. Returns whatevervalue(s) body returns. + Arguments + + var + A symbol (variable name) + expr + An expression which should evaluate to a non-negativefixnum + + + + + + + + make-cstring [Function] + Syntax + make-cstring string + Description + Allocates a block of memory (via malloc) of length (1+(length string)). Copies the string to this block and appends atrailing NUL byte; returns a MACPTR to the block. + Arguments + + string + A lisp string + + + + + + + + with-cstrs [Macro] + Syntax + with-cstrs (var string)* &body body + Description + Executes body in an environment in which each var is boundto a stack-allocated macptr which encapsulates the %address of astack-allocated region of into which each string (and a trailingNUL byte) has been copied. Returns whatever value(s) body returns. + Arguments + + var + A symbol (variable name) + string + An expression which should evaluate to a lisp string + + + + + + + + %get-cstring [Function] + Syntax + %get-cstring ptr + Description + Interprets ptr as a pointer to a (NUL -terminated) C string;returns an equivalent lisp string. + Arguments + + ptr + A MACPTR + + + + + + + + %str-from-ptr [Function] + Syntax + %str-from-ptr ptr length + Description + Returns a lisp string of length length,whose contents are initialized from the bytes at ptr. + Arguments + + ptr + AMACPTR + length + anon-negative fixnum + + + + + + + + + + The Interface Database + + + Overview + OpenMCL uses a set of database files which contain + foreign type, record, constant, and function definitions + derived from the operating system's header files, be that + Linux or Darwin. An archive containing these database files + (and the shell scripts which were used in their creation) is + available; see the Distributions page for information about + obtaining current interface database files. + Not surprisingly, different platforms use different database files. + OpenMCL defines reader macros that consult these databases: + + #$foo looks up the value of the constant definition of FIXTHIS + #_foo looks up the foreign function definition for FIXTHIS + + + In both cases, the symbol foo is interned in the "OS" + package. The #$ reader macro has the side-effect of defining + foo as a constant (as if via DEFCONSTANT); the #_ reader macro + has the side effect of defining foo as a macro which will + expand into an (EXTERNAL-CALL form.) + It's important to remember that the side-effect happens + when the form containing the reader macro is + read. Macroexpansion functions that expand into forms which + contain instances of those reader macros don't do what one + might think that they do, unless the macros are expanded in + the same lisp session as the reader macro was read in. + In addition, references to foreign type, + structure/union, and field names (when used in the RREF/PREF + and RLET macros) will cause these database files to be + consulted. + Since the OpenMCL sources contain instances of these + reader macros (and references to foreign record types and + fields), compiling OpenMCL from those sources depends on the + ability to find and use (see ). + + + + Other issues: + + + OpenMCL now preserves the case of external symbols + in itsdatabase files. See for information about case in + foreign symbol names. + + + The Linux databases are derived from a somewhat + arbitrary set of Linux header files. Linux is enough of a + moving target that it may be difficult to define a standard, + reference set of interfaces from which to derive a standard, + reference set of database files.This seems to be less of + an issue with Darwin and FreeBSD. + + + For information about building the database files, + see . + + + + + Using Interface Directories + + + Overview + As distributed, the "ccl:headers;" (for LinuxPPC) + directory is organized like: + +headers/ +headers/gl/ +headers/gl/C/ +headers/gl/C/populate.sh +headers/gl/constants.cdb +headers/gl/functions.cdb +headers/gl/records.cdb +headers/gl/objc-classes.cdb +headers/gl/objc-methods.cdb +headers/gl/types.cdb +headers/gnome/ +headers/gnome/C/ +headers/gnome/C/populate.sh +headers/gnome/constants.cdb +headers/gnome/functions.cb +headers/gnome/records.cdb +headers/gnome/objc-classes.cdb +headers/gnome/objc-methods.cdb +headers/gnome/types.cdb +headers/gtk/ +headers/gtk/C/ +headers/gtk/C/populate.sh +headers/gtk/constants.cdb +headers/gtk/functions.cdb +headers/gtk/records.cdb +headers/gtk/objc-classes.cdb +headers/gtk/objc-methods.cdb +headers/gtk/types.cdb +headers/libc/ +headers/libc/C/ +headers/libc/C/populate.sh +headers/libc/constants.cdb +headers/libc/functions.cdb +headers/libc/records.cdb +headers/libc/objc-classes.cdb +headers/libc/objc-methods.cdb +headers/libc/types.cdb + + e.g, as a set of parallel subdirectories, each with a + lowercase name and each of which contains a set of 6 database + files and a "C" subdirectory which contains a shell script + used in the database creation process. + As one might assume, the database files in each of these + subdirectories contain foreign type, constant, and function + definitions - as well as ObjC class and method info -that + correspond (roughly) to the information contained in the + header files associated with a "-dev" package in a Linux + distribution. "libc" corresponds pretty closely to the + interfaces associated with "glibc/libc6" header files, "gl" + corresponds to an "openGL+GLUT" developmnent package, "gtk" + and "gnome" contain interface information from the GTK+1.2 and + GNOME libraries, respectively. + For Darwin, the "ccl:darwin-headers" directory contains + a "libc" subdirectory, whose contents roughly correspond to + those of "/usr/include" under Darwin, as well as + subdirectories corresponding to the MacOSX Carbon and Cocoa + frameworks. + To see the precise set of .h files used to generate the + database files in a given interface directory, consult the + corresponding "populate.sh" shell script (in the interface + directory's "C" subdirectory.) + The intent is that this initial set can be augmented to + meet local needs, and that this can be done in a fairly + incremental fashion: one needn't have unrelated header files + installed in order to generate interface databases for a + package of interest. + Hopefully, this scheme will also make it easier to + distribute patches and bug fixes. + OpenMCL maintains a list of directories; when looking + for a foreign type, constant, function, or record definition, + it'll consult the database files in each directory on that + list. Initially, the list contains an entry for the "libc" + interface directory. OpenMCL needs to be explicitly told to + look in other interface directories should it need to do + so. + + + + Creating new interface directories + This example refers to "ccl:headers;", which is + appropriate for LinuxPPC. The procedure's analogous under + Darwin, where the "ccl:darwin-headers;" directory would be + used instead. + To create a new interface directory, "foo", and a set of + database files in that directory: + + Create a subdirectory of "ccl:headers;" named"foo". + Create a subdirectory of "ccl:headers;foo;" named"C". + Create a file in "ccl:headers;foo;C;" named"populate.sh".One way of accomplishing the above steps is: + +? (close (open "ccl:headers;foo;C;populate.sh" :direction :output : + if-does-not-exist :create :if-exists :overwrite)) + + Edit the file created above, using the "populate.sh"files in the distribution as guidelines.The file might wind up looking something like: + +#/bin/sh +h-to-ffi.sh `foo-config -cflags` /usr/include/foo/foo.h + + + Refer to +for information about running the +interface translator and .ffi parser. + Assuming that all went well, there should now be .cdb files in +"ccl:headers;foo;". You can then do +(use-interface-dir :foo) whenever you need to +access the foreign type information in those database files. + + + + + Using Shared Libraries + + + Overview +OpenMCL provides facilities to open and close shared libraries. + "Opening" a shared library, which is done with +, maps the library's +code and +data into OpenMCL's address space and makes its exported symbols +accessible to OpenMCL. + "Closing" a shared library, which is done with +, unmaps the +library's code and +data and removes the library's symbols from the global namespace. + A small number of shared libraries (including libc, libm, libdl +under Linux, and the "system" library under Darwin) are opened by +the lisp kernel and can't be closed. + OpenMCL uses data structures of type EXTERNAL-ENTRY-POINT to map a +foreign function name (string) to that foreign function's +current< address. (A function's address may +vary from session to session as different versions of shared libraries may +load at different addresses; it may vary within a session for similar +reasons.) + An EXTERNAL-ENTRY-POINT whose address is known is said to be +resolved. When an external entry point is resolved, +the shared library which defines that entry point is noted; when a shared +library is closed, the entry points that it defines are made unresolved. +An EXTERNAL-ENTRY-POINT must be in the resolved state in order to be +FF-CALLed; calling an unresolved entry point causes a "last +chance" attempt to resolve it. Attempting to resolve an entrypoint +that was defined in a closed library will cause an attempt to reopen that +library. + OpenMCL keeps track of all libraries that have been opened in a lisp +session. When a saved application is first started, an attempt is made to +reopen all libraries that were open when the image was saved, and an +attempt is made to resolve all entrypoints that had been referenced when +the image was saved. Either of these attempts can fail "quietly", +leaving some entry points in an unresolved state. + Linux shared libraries can be referred to either by a string which +describes their full pathname or by their soname, a +shorter string that can be defined when the library is created. The +dynamic linker mechanisms used in Linux make it possible (through a series +of filesystem links and other means) to refer to a library via several +names; the library's soname is often the most appropriate identifier. + sonames are often less version-specific than other names for +libraries; a program that refers to a library by the name +"libc.so.6" is more portable than one which refers to +"libc-2.1.3.so" or to "libc-2.2.3.so", even though the +latter two names might each be platform-specific aliases of the first. + All of the global symbols described below are exported from the CCL +package. + + + + Limitations and known bugs + + Don't get me started. + The underlying functionality has a poor notion of dependency;it's not always possible to open libraries that depend on unopenedlibraries, but it's possible to close libraries on which otherlibraries depend.It may be possible to generate moreexplicit dependency information by parsing the output of the Linux lddand ldconfig programs. + + + + + + Darwin Notes +Darwin shared libraries come in two (basic) flavors: + + "dylibs" (which often have the extension".dylib") are primarily intended to be linked against atcompile/link time. They can be loaded dynamically,but can't be unloaded. Accordingly,OPEN-SHARED-LIBRARY can be used to open a .dylib-style library;calling CLOSE-SHARED-LIBRARY on the result of such a callproduces a warning, and has no other effect.It appears that (due to an OS bug) attempts to open .dylibshared-libraries that are already open can cause memory corruptionunless the full pathname of the .dylib file is specified on thefirst and all subsequent calls. + "bundles" are intended to serve as applicationextensions; they can be opened multiple times (creating multipleinstances of the library!) and closed properly. + + + Thanks to Michael Klingbeil for getting both kinds of Darwin shared +libraries working in OpenMCL. + + + + + The Interface Translator + + + Overview +OpenMCL uses an interface translation system based on the FFIGEN +system, which is described at +http://www.ccs.neu.edu/home/lth/ffigen/. +The interface translator makes +the constant, type, structure, and function definitions in a set of +C-language header files available to lisp code. + The basic idea of the FFIGEN scheme is to use the C compiler's +frontend and parser to translate .h files into semantically equivalent +.ffi files, which represent the definitions from the headers using a +syntax based on S-expressions. +Lisp code can then concentrate on +the .ffi representation, without having to concern itself with the +semantics of header file inclusion or the arcana of C parsing. + The original FFIGEN system used a modified version of the LCC C +compiler to produce .ffi files. Since many LinuxPPC header files contain +GCC-specific constructs, OpenMCL's translation system uses a modified +version of GCC (called, somewhat confusingly, ffigen.) + A version of ffigen based on GCC-4.0 was developed during the spring +and summer of 2005. Sources (diffs relative to the GCC-4.0 release) +are available here, and +binaries are available for +DarwinPPC +and for +LinuxPPC. +These versions should be insensitive to to the version of GCC (and its +preprocessor) installed on the system. + An older version was developed in 2001-2002; it depended on the installed +version of GCC being 2.95. It may still be of interest for people unable +to run the GCC-4.0-based version for whatever reason. + A LinuxPPC binary of this older version is available at ftp://clozure.com/pub/ffigen-0.1.tar.gz, +and LinuxPPC source differences are at ftp://clozure.com/pub/ffigen-src.tar.gz. + For Darwin, the binary of the older FFIGEN is available at ftp://clozure.com/pub/ffigen-darwin.tar.gz, +and the source differences are at ftp://clozure.com/pub/ffigen-darwin-src.tar.gz. + A shell script (distributed with the source and binary packages) +called h-to-ffi.sh reads a specified .h file (and optional +preprocessor arguments) and writes a (hopefully) equivalent .ffi file to +standard output, calling the installed C preprocessor and the ffigen +program with appropriate arguments. + For each interface directory (see ) +subdir distributed with OpenMCL, a shell script +(distributed with OpenMCL as "ccl:headers;subdir;C;populate.sh" +("ccl:darwin-headers;subdir;C;populate.sh" +for Darwin)) calls h-to-ffi.sh on a large number of the header files in +/usr/include (or some other system header path) and +creates a parallel directory tree in "ccl:headers;subdir;C;system;header;path;" +(or "ccl:darwin-headers;subdir;C;system;header;path;"), +populating that directory with .ffi files. + A lisp function defined in "ccl:library;parse-ffi.lisp" +reads the .ffi files in a specified interface directory +subdir and generates new versions of the databases +(files with the extension .cdb). + The CDB databases are used by the #$ and #_ reader macros and are +used in the expansion of RREF, RLET, and related macros. + + + + Details: rebuilding the CDB databases, step by step + + Ensure that the FFIGEN program is installed. See the"README" file in the source or binary archive for specificinstallation instructions.This example assumes LinuxPPC; for 32-bit DarwinPPC, substitute"ccl:darwin-headers;" for "ccl:headers;". For 64-bit DarwinPPC,substitute "ccl:darwin-headers64;". + Edit the "ccl:headers;subdir;C;populate.sh"shell script. When you're confident that the files andpreprocessor options match your environment, cd to the"ccl:headers;subdir;C;" directory andinvoke ./populate.sh. Repeat this step until you're able tocleanly translate all files refrenced in the shell script. + Run OpenMCL: + +? (require "PARSE-FFI") +PARSE-FFI + +? (parse-standard-ffi-files :SUBDIR) +;;; lots of output ... after a while, shiny new .cdb files should +;;; appear in "ccl:headers;subdir;" +;;; (or "ccl:darwin-headers;subdir;" under Darwin) + + + PARSE-STANDARD-FFI-FILES accepts a :PREPEND-UNDERSCORES keyword +argument. Darwin (and some other platforms) use a convention wherein the +symbols associated with C-visible external function and variables have +underscore characters prepended to their names. When this argument is +true, PARSE-STANDARD-FFI-FILES will prepend underscores to all foreign +function names written to the database, so that (#_foo ...) expands into +an EXTERNAL-CALL to "_foo". + + + + + Case-sensitivity of foreign names in OpenMCL + + + Overview +As of release 0.11, OpenMCL addresses the fact that foreign type, +constant, record, field, and function nams are case-sensitive and provides +mechanisms to refer to these names via lisp symbols. + Previous versions of OpenMCL have tried to ignore that fact, under +the belief that case conflicts were rare and that many users (and +implementors) would prefer not to deal with case-related issues. The fact +that some information in the interface databases was incomplete or +inaccessable because of this policy made it clearer that the policy was +untenable. I can't claim that the approach described here is +aesthetically pleasing, but I can honestly say that it's less +unpleasant than other approaches that I'd thought of. I'd be +interested to hear alternate proposals. + The issues described here have to do with how lisp symbols are used +to denote foreign functions, constants, types, records, and fields. It +doesn't affect how other lisp objects are sometimes used to denote +foreign objects. For instance, the first argument to the EXTERNAL-CALL +macros is now and has always been a case-sensitive string. + + + + Foreign constant and function names +The primary way of referring to foreign constant and function names +in OpenMCL is via the #$ and #_ reader macros. These reader macro +functions each read a symbol into the "OS" package, look up its +constant or function definition in the interface database, and assign the +value of the constant to the symbol or install a macroexpansion function +on the symbol. + In order to observe case-sensitivity, the reader-macros now read the +symbol with (READTABLE-CASE :PRESERVE) in effect. + This means that it's necessary to type the foreign constant or +function name in correct case, but it isn't necessary to use any +special escaping constructs when writing the variable name. For instance: + +(#_read fd buf n) ; refers to foreign symbol "read" +(#_READ fd buf n) ; refers to foreign symbol "READ", which may +; not exist ... +#$o_rdonly ; Probably doesn't exist +#$O_RDONLY ; Exists on most platforms + + + + + Foreign type, record, and field names +Constructs like RLET expect a foreign type or record name to be +denoted by a symbol (typically a keyword); RREF (and PREF) expect an +"accessor" form, typically a keyword formed by concatenating a +foreign type or record name with a sequence of one or more foreign field +names, separated by dots. These names are interned by the reader as other +lisp symbols are, with an arbitrary value of READTABLE-CASE in effect +(typically :UPCASE.) It seems like it would be very tedious to force users +to manually escape (via vertical bar or backslash syntax) all lowercase +characters in symbols used to specify foreign type, record, and field +names (especially given that many traditional POSIX structure, type, and +field names are entirely lowercase.) + The approach taken by OpenMCL is to allow the symbols (keywords) +used to denote foreign type, record, and field names to contain angle +brackets (< and >). Such symbols are translated to foreign names +via the following set of conventions: + + All instances of < and > in the symbol's pname arebalanced and don't nest. + Any alphabetic characters in the symbol's pname thataren't enclosed in angle brackets are treated as lower-case,regardless of the value of READTABLE-CASE and regardless of the casein which they were written. + Alphabetic characters that appear within angle brackets aremapped to upper-case, again regardless of how they were written orinterned. + + + There may be many ways of "escaping" (with angle brackets) +sequences of upper-case and non-lower-case characters in a symbol used to +denote a foreign name. When translating in the other direction, OpenMCL +always escapes the longest sequence that starts with an upper-case +character and doesn't contain a lower-case character. + It's often preferable to use this canonical form of a foreign +type name. + The accessor forms used by PREF/RREF should be viewed as a series of +foreign type/record and field names; upper-case sequences in the component +names should be escaped with angle brackets, but those sequences +shouldn't span components. (More simply, the separating dots +shouldn't be enclosed, even if both surrounding characters need to +be.) + Older POSIX code tends to use lower-case exclusively for type, +record, and field names; there are only a few cases in the OpenMCL sources +where mixed-case names need to be escaped. + + + + Examples + +;;; Allocate a record of type "window". +(rlet ((w :window)) ...) +;;; Allocate a record of type "Window", which is probably a +;;; different type +(rlet ((w :indow)) ...) +;;; This is equivalent to the last example +(rlet ((w :INDOW))) + + + + + + Tutorial: Using Basic Calls and Types +This tutorial is meant to cover the basics of OpenMCL for calling +external C functions and passing data back and forth. These basics +will provide the foundation for more advanced techniques which will +allow access to the various external libraries and toolkits. + The first step is to start with a simple C dynamic library in order to +actually observe what is actually passing between OpenMCL and C. So, +some C code is in order: + Create the file typetest.c, and put the following code into it: + + +#include + +void +void_void_test(void) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Exited %s:\n", __FUNCTION__); +} + +signed char +sc_sc_test(signed char data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %d\n", (signed int)data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + +unsigned char +uc_uc_test(unsigned char data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %d\n", (signed int)data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + + This defines three functions. If you're familiar with C, notice +that there's no main(), because we're just +building a library, not an executable. + The function void_void_test() doesn't take +any parameters, and doesn't return anything, but it prints two +lines to let us know it was called. +sc_sc_test() takes a signed char as a +parameter, prints it, and returns it. +uc_uc_test() does the same thing, but with +an unsigned char. Their purpose is just to prove to us +that we really can call C functions, pass them values, and get +values back from them. + + This code is compiled into a dynamic library on OS X 10.3.4 with the +command: + + +gcc -dynamiclib -Wall -o libtypetest.dylib typetest.c \ + -install_name ./libtypetest.dylib + + The -dynamiclib tells gcc that we will be compiling this into a +dynamic library and not an executable binary program. The output +filename is "libtypetest.dylib". Notice that we chose a name which +follows the normal OS X convention, being in the form +"libXXXXX.dylib", so that other programs can link to the library. +OpenMCL doesn't need it to be this way, but it is a good idea to +adhere to existing conventions. + The -install_name flag is primarily used when +building OS X "bundles". In this case, we are not using it, so we +put a placeholder into it, "./libtypetest.dylib". If we wanted +to use typetest in a bundle, the -install_name argument would +be a relative path from some "current" directory. + After creating this library, the first step is to tell OpenMCL to open +the dynamic library. This is done by calling +. + + +Welcome to OpenMCL Version (Beta: Darwin) 0.14.2-040506! + +? (open-shared-library "/Users/andewl/openmcl/libtypetest.dylib") +# + + You should use an absolute path here; using a relative one, such +as just "libtypetest.dylib", would appear to work, but there are +subtle problems which occur after reloading it. See the Darwin +notes on + for details. It would be +a bad idea anyway, because software should never rely on its +starting directory being anything in particular. + This command returns a reference to the opened shared library, and +OpenMCL also adds one to the global variable +ccl::*shared-libraries*: + + +? ccl::*shared-libraries* +(# + #) + + Before we call anything, let's check that the individual functions can +actually be found by the system. We don't have to do this, but +it helps to know how to find out whether this is the problem, +when something goes wrong. +We use : + + +? (external "_void_void_test") +# + +? (external "_sc_sc_test") +# + +? (external "_uc_uc_test") +# + + Notice that the actual function names have +been "mangled" by the C linker. The first function was named +"void_void_test" in typetest.c, but in +libtypetest.dylib, it has an underscore (a "_" symbol) before +it: "_void_void_test". So, this is the name which you have to use. +The mangling - the way the name is changed - may be different for +other operating systems or other versions, so you need to "just know" +how it's done... + Also, pay particular attention to the fact +that a hexadecimal value appears in the EXTERNAL-ENTRY-POINT. +(#x000CFDF8, for example - but what it is doesn't matter.) +These hex numbers mean that the +function can be dereferenced. Functions which aren't found will +not have a hex number. For example: + + +? (external "functiondoesnotexist") +# + + The "unresolved" tells us that OpenMCL wasn't able to find this +function, which means you would get an error, "Can't resolve foreign +symbol," if you tried to call it. + These +external function references also are stored in a hash table which is +accessible through a global variable, ccl::*eeps*. + At this point, we are ready to try our first external function +call: + + +? (external-call "_void_void_test" :void) +Entered void_void_test: +Exited void_void_test: +NIL + + We used , which is +is the normal mechanism for +accessing externally linked code. The "_void_void_test" +is the mangled name of the external function. +The :void +refers to the return type of the function. + If you're using ILISP to run OpenMCL inside of Emacs, you won't +see the "Entered" and "Exited" lines until you quit (as of +July 2004). It's not +clear why this is, but it's a pity. If you want to see them, run +OpenMCL from Terminal.app or in some other way. + The next step is to try passing a value to C, and getting one +back: + + +? (external-call "_sc_sc_test" :signed-byte -128 :signed-byte) +Entered sc_sc_test: +Data In: -128 +Exited sc_sc_test: +-128 + + The first :signed-byte gives the type of the first argument, +and then -128 gives the value to pass for it. The second +:signed-byte +gives the return type. The return type is always given by the +last argument to . + Everything looks good. Now, let's try a number outside +the range which fits in one byte: + + +? (external-call "_sc_sc_test" :signed-byte -567 :signed-byte) +Entered sc_sc_test: +Data In: -55 +Exited sc_sc_test: +-55 + + Hmmmm. A little odd. Let's look at the unsigned stuff +to see how it reacts: + + +? (external-call "_uc_uc_test" :unsigned-byte 255 :unsigned-byte) +Entered uc_uc_test: +Data In: 255 +Exited uc_uc_test: +255 + + That looks okay. Now, let's go outside the valid range again: + + +? (external-call "_uc_uc_test" :unsigned-byte 567 :unsigned-byte) +Entered uc_uc_test: +Data In: 55 +Exited uc_uc_test: +55 + +? (external-call "_uc_uc_test" :unsigned-byte -567 :unsigned-byte) +Entered uc_uc_test: +Data In: 201 +Exited uc_uc_test: +201 + + Since a signed byte can only hold values from -128 through 127, and +an unsigned one can only hold values from 0 through 255, any number +outside that range gets "clipped": only the low eight bits of it +are used. + What is important to remember is that external +function calls have +very few safety checks. +Data outside the valid range for its type will silently do +very strange things; pointers outside the valid range can very well +crash the system. + That's it for our first example library. If you're still following +along, let's add some more C code to look at the rest of the +primitive types. Then we'll need to recompile the dynamic library, +load it again, and then we can see what happens. + Add the following code to typetest.c: + + +int +si_si_test(int data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %d\n", data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + +long +sl_sl_test(long data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %ld\n", data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + +long long +sll_sll_test(long long data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %lld\n", data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + +float +f_f_test(float data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %e\n", data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + +double +d_d_test(double data) +{ + printf("Entered %s:\n", __FUNCTION__); + printf("Data In: %e\n", data); + printf("Exited %s:\n", __FUNCTION__); + return data; +} + + The command line to compile the dynamic library is the same as before: + + +gcc -dynamiclib -Wall -o libtypetest.dylib typetest.c \ + -install_name ./libtypetest.dylib + + Now, restart OpenMCL. This step is required because OpenMCL cannot +close and reload a dynamic library on OS X. + Have you restarted? Okay, try out the new code: + + +Welcome to OpenMCL Version (Beta: Darwin) 0.14.2-040506! + +? (open-shared-library "/Users/andewl/openmcl/libtypetest.dylib") +# + +? (external-call "_si_si_test" :signed-fullword -178965 :signed-fullword) +Entered si_si_test: +Data In: -178965 +Exited si_si_test: +-178965 + +? ;; long is the same size as int +(external-call "_sl_sl_test" :signed-fullword -178965 :signed-fullword) +Entered sl_sl_test: +Data In: -178965 +Exited sl_sl_test: +-178965 + +? (external-call "_sll_sll_test" + :signed-doubleword -973891578912 :signed-doubleword) +Entered sll_sll_test: +Data In: -973891578912 +Exited sll_sll_test: +-973891578912 + + Okay, everything seems to be acting as expected. However, just to +remind you that most of this stuff has no safety net, here's +what happens if somebody mistakes sl_sl_test() +for sll_sll_test(), thinking that a long is +actually a doubleword: + + +? (external-call "_sl_sl_test" + :signed-doubleword -973891578912 :signed-doubleword) +Entered sl_sl_test: +Data In: -227 +Exited sl_sl_test: +-974957576192 + + Ouch. The C function changes the value with no warning that something +is wrong. Even worse, it manages to pass the original value back to +OpenMCL, which hides the fact that something is wrong. + Finally, let's take a look at doing this with floating-point numbers. + + +Welcome to OpenMCL Version (Beta: Darwin) 0.14.2-040506! + +? (open-shared-library "/Users/andewl/openmcl/libtypetest.dylib") +# + +? (external-call "_f_f_test" :single-float -1.256791e+11 :single-float) +Entered f_f_test: +Data In: -1.256791e+11 +Exited f_f_test: +-1.256791E+11 + +? (external-call "_d_d_test" :double-float -1.256791d+290 :double-float) +Entered d_d_test: +Data In: -1.256791e+290 +Exited d_d_test: +-1.256791D+290 + + Notice that the number ends with "...e+11" for the single-float, +and "...d+290" for the +double-float. Lisp has both of these float types itself, and the +d instead of the e is how you specify which to create. If +you tried to pass :double-float 1.0e2 to external-call, Lisp would +be nice enough to notice and give you a type error. Don't get the +:double-float wrong, though, because then there's no protection. + Congratulations! You now know how to call external C functions from +within OpenMCL, and pass numbers back and forth. Now that the basic +mechanics of calling and passing work, the next step is to examine how +to pass more complex data structures around. + + + Acknowledgement +This chapter was generously contributed by +Andrew P. Lentvorski Jr. + + + + + Tutorial: Allocating Foreign Data on the Lisp Heap +Not every foreign function is so marvelously easy to use as the +ones we saw in the last section. Some of them require you to +allocate a C struct, fill it in with your own information, and +pass it a pointer to the struct. Some of them require you to +allocate an empty struct so they can fill it in, and then you can +read the information out of it. + Also, some of them have their own structs and return a pointer to +that same struct every time you call them, but those are easier to +deal with, +so they won't +be covered in this section. + You might know that Lisp (and, indeed, most programming languages) +has two separate regions of memory. There's the stack, which is +where variable bindings are kept. Memory on the stack is allocated +every time any function is called, and deallocated when it returns, +so it's useful for anything that doesn't need to last longer than +one function call, when there's only one thread. If that's all +you need, you can do it with +. + Then, there's the heap, which holds everything else, and is our +subject here. +There are two advantages and one big disadvantage to putting things on +the heap rather than the stack. First, data allocated on the heap can +be passed outside of the scope in which it was created. This is +useful for data which may need to be passed between multiple C calls +or multiple threads. Also, some data may be too large to copy multiple +times or may be too large to allocate on the stack. + The second advantage is security. If incoming data is being placed +directly onto the stack, the input data can cause stack overflows and +underflows. This is not something which Lisp users generally worry +about since garbage collection generally handles memory management. +However, "stack smashing" is one of the classic exploits in C which +malicious hackers can use to gain control of a machine. Not checking +external +data is always a bad idea; however, allocating it into the heap at +least offers more protection than direct stack allocation. + The big disadvantage to allocating data on the heap is that it must be +explicitly deallocated - you need to "free" it when you're done with +it. Ordinarily, in Lisp, you wouldn't allocate memory yourself, and +the garbage collector would know about it, so you wouldn't have to +think about it again. When you're doing it manually, it's very +different. +Memory management becomes a manual process, just like in C and +C++. + What that means is that, if you allocate something and then lose +track of the pointer to it, +there's no way to ever free that +memory. That's what's called a memory leak, and if your program +leaks enough memory it will eventually use up all of it! So, you +need to be careful to not lose your pointers. + That disadvantage, though, is also an advantage for using +foreign functions. Since the garbage collector doesn't know about +this memory, it will never move it around. External C code +needs this, because it doesn't know how to follow it to where it +moved, the way that Lisp code does. If you allocate data manually, +you can pass it to foreign code and know that no matter what that +code needs to do with it, it will be able to, until you +deallocated it. Of course, you'd better be sure it's done before +you do. Otherwise, your program will be unstable and might crash +sometime in the future, and you'll have trouble figuring out what +caused +the trouble, because there won't be anything pointing +back and saying "you deallocated this too soon." + And, so, on to the code... + As in the last tutorial, our first step +is to create a local dynamic library in order to help show +what is actually going on between OpenMCL and C. So, create the file +ptrtest.c, with the following code: + +#include + +void reverse_int_array(int * data, unsigned int dataobjs) +{ + int i, t; + + for(i=0; i + This defines three functions. reverse_int_array +takes a pointer to an array of ints, and a count +telling how many items are in the array, and loops through it +putting the elements in reverse. +reverse_int_ptr_array does the same thing, +but with an array of pointers to ints. It only +reverses the order the pointers are in; each pointer still points to +the same thing. +reverse_int_ptr_ptrtest +takes an array of pointers to arrays of ints. (With me?) +It doesn't need to be told their sizes; +it just assumes that the array of pointers has two items, +and that both of those are arrays which have four items. It +reverses the array of pointers, then it reverses each of the two +arrays of ints. + Now, compile ptrtest.c into a dynamic library using the command: + +gcc -dynamiclib -Wall -o libptrtest.dylib ptrtest.c -install_name ./libptrtest.dylib + + If that command doesn't make sense to you, feel free to go back +and read about it at . + Now, start OpenMCL and enter: + +? ;; make-heap-ivector courtesy of Gary Byers +(defun make-heap-ivector (element-count element-type) + (let* ((subtag (ccl::element-type-subtype element-type))) + (unless (= (logand subtag target::fulltagmask) + target::fulltag-immheader) + (error "~s is not an ivector subtype." element-type)) + (let* ((size-in-bytes (ccl::subtag-bytes subtag element-count))) + (ccl::%make-heap-ivector subtag size-in-bytes element-count)))) +MAKE-HEAP-IVECTOR + +? ;; dispose-heap-ivector created for symmetry +(defmacro dispose-heap-ivector (a mp) + `(progn + (ccl::%dispose-heap-ivector ,a) + ;; Demolish the arguments for safety + (setf ,a nil) + (setf ,mp nil))) +DISPOSE-HEAP-IVECTOR + + You don't understand how those functions do what they do. That's +okay; it gets into very fine detail which really doesn't matter, +because you don't need to change them. + The function make-heap-ivector is the primary +tool for +allocating objects in heap memory. It allocates a fixed-size OpenMCL +object in heap memory. It returns both an array reference, which can +be used directly from OpenMCL, and a macptr, which can +be used to access the underlying memory directly. For example: + +? ;; Create an array of 3 4-byte-long integers +(multiple-value-bind (la lap) + (make-heap-ivector 3 '(unsigned-byte 32)) + (setq a la) + (setq ap lap)) +;Compiler warnings : +; Undeclared free variable A, in an anonymous lambda form. +; Undeclared free variable AP, in an anonymous lambda form. +# + +? a +#(1396 2578 97862649) + +? ap +# + + It's important to realize that the contents of the +ivector we've just created +haven't been initialized, +so their values are unpredictable, +and you should be sure not to +read from them before you set them, to avoid confusing results. + At this point, a references an object which +works just +like a normal array. You can refer to any item of it with +the standard aref function, and set them +by combining that with setf. +As noted +above, the ivector's +contents haven't been initialized, +so that's the next order of business: + +? a +#(1396 2578 97862649) + +? (aref a 2) +97862649 + +? (setf (aref a 0) 3) +3 + +? (setf (aref a 1) 4) +4 + +? (setf (aref a 2) 5) +5 + +? a +#(3 4 5) + + In addition, the macptr allows direct access to the same +memory: + +? (setq *byte-length-of-long* 4) +4 + +? (%get-signed-long ap (* 2 *byte-length-of-long*)) +5 + +? (%get-signed-long ap (* 0 *byte-length-of-long*)) +3 + +? (setf (%get-signed-long ap (* 0 *byte-length-of-long*)) 6) +6 + +? (setf (%get-signed-long ap (* 2 *byte-length-of-long*)) 7) +7 + +? ;; Show that a actually got changed through ap +a +#(6 4 7) + + So far, there is nothing about this object that could not be done much +better with standard Lisp. However, the macptr can be +used to pass this chunk of memory off to a C function. Let's use +the C +code to reverse the elements in the array: + +? ;; Insert the full path to your copy of libptrtest.dylib +(open-shared-library "/Users/andrewl/openmcl/openmcl/gtk/libptrtest.dylib") +# + +? a +#(6 4 7) + +? ap +# + +? (external-call "_reverse_int_array" :address ap :unsigned-int (length a) :address) +# + +? a +#(7 4 6) + +? ap +# + + The array gets passed correctly to the C function, +reverse_int_array. The C function +reverses the contents of the array in-place; that is, it doesn't make +a new array, just keeps the same one and reverses what's in it. +Finally, the C function +passes control back to OpenMCL. Since the allocated array memory has +been directly modifed, OpenMCL reflects those changes directly in the +array as well. + There is one final bit of housekeeping to deal with. Before moving +on, the memory needs to be deallocated: + +? ;; dispose-heap-ivector created for symmetry +;; Macro repeated here for pedagogy +(defmacro dispose-heap-ivector (a mp) + `(progn + (ccl::%dispose-heap-ivector ,a) + ;; Demolish the arguments for safety + (setf ,a nil) + (setf ,mp nil))) +DISPOSE-HEAP-IVECTOR + +? (dispose-heap-ivector a ap) +NIL + +? a +NIL + +? ap +NIL + + The dispose-heap-ivector macro actually +deallocates the +ivector, releasing its memory into the heap for something else to +use. In addition, it makes sure +that the +variables which it was called with are set to nil, because otherwise +they would still be referencing the memory of the ivector - which is +no longer allocated, so that would be a bug. Making sure there are +no other variables set to it is up to you. + When do you call dispose-heap-ivector? +Anytime after you know the ivector will never be used again, but no +sooner. If you have a lot of ivectors, say, in a hash table, +you need to make sure that when whatever you were doing with the +hash table is done, those ivectors all get freed. Unless there's +still something somewhere else which refers to them, of course! +Exactly what strategy to take depends on the situation, +so just try to keep things simple unless you know better. + The simplest situation is when you have things set up so that a Lisp +object "encapsulates" a pointer to foreign data, taking care of all +the +details of using it. In this case, you don't want those two things +to have different lifetimes: You want to make sure your Lisp object +exists as long as the foreign data does, and no longer; and you want +to make sure the foreign data doesn't get deallocated while your +Lisp object still refers to it. + If you're willing to accept a few limitations, you can make this +easy. First, you can't let foreign code keep a permanent pointer +to the +memory; it has to always finish what it's doing, then return, and +not refer to that memory again. Second, you can't let any Lisp +code that isn't part of your encapsulating "wrapper" refer to the +pointer directly. Third, nothing, either foreign code or Lisp +code, should explicitly deallocate the memory. + If you can make sure all of these are true, you can at least +ensure that the foreign +pointer is deallocated when the encapsulating object is about to +become +garbage, by using OpenMCL's nonstandard "termination" mechanism, +which is essentially the same as what Java and other languages +call "finialization". + Termination is a way of asking the garbage collector to let you know +when it's about to destroy an object which isn't used anymore. +Before destroying the object, it calls a function which you write, +called a terminator. + So, you can use termination to find out when a particular +macptr is about to become garbage. +That's not quite +as helpful as it might seem: It's not exactly the same thing as +knowing that the block of memory it points to is unreferenced. +For example, there could be another macptr somewhere +to the +same block; or, if it's a struct, there could be a +macptr to one of its fields. Most problematically, +if the address of that memory has been passed to foreign code, +it's sometimes hard to know whether that code has kept the +pointer. Most foreign functions don't, but it's not hard to think of +exceptions. + You can use code such as this to make all this happen: + +(defclass wrapper (whatever) + ((element-type :initarg :element-type) + (element-count :initarg :element-count) + (ivector) + (macptr))) + +(defmethod initialize-instance ((wrapper wrapper) &rest initargs) + (declare (ignore initargs)) + (call-next-method) + (ccl:terminate-when-unreachable wrapper) + (with-slots (ivector macptr element-type element-count) wrapper + (multiple-value-bind (new-ivector new-macptr) + (make-heap-ivector element-count element-type) + (setq ivector new-ivector + macptr new-macptr)))) + +(defmethod ccl:terminate ((wrapper wrapper)) + (with-slots (ivector macptr) wrapper + (when ivector + (dispose-heap-ivector ivector macptr) + (setq ivector nil + macptr nil)))) + + The ccl:terminate method will be called on +some arbitrary thread +sometime (hopefully soon) after the GC has decided that there are no +strong references to an object which has been the argument of a +ccl:terminate-when-unreachable call. + If it makes sense to say that the foreign object should live as long +as there's Lisp code that references it (through the encapsulating +obect) and no longer, this is one way of doing that. + Now we've covered passing basic types back and forth with C, and +we've done the same with pointers. You may think this is all... +but we've only done pointers to basic types. Join us next time +for pointers... to pointers. + + + Acknowledgement +Much of this chapter was generously contributed by +Andrew P. LentvorskiJr. + + + + + The Foreign-Function-Interface Dictionary + + + DEF-FOREIGN-TYPE + def-foreign-type + Name + DEF-FOREIGN-TYPE — + Macro + Synopsis + + + def-foreign-type name foreign-type-spec + + + Values + name + NIL or a keyword; the keyword may containescaping constructs (see ). + foreign-type-spec + A foreign type specifier, whose syntax is (loosely)defined above. + + + Description + If name is non-NIL, defines name to be an alias for the +foreign type specified by foreign-type-spec. If foreign-type-spec +is a named structure or union type, additionally defines that +structure or union type. + If name is NIL, foreign-type-spec must be a named foreign +struct or union definition, in which case the foreign structure +or +union definition is put in effect. + Note that there are two separate namespaces for foreign +type names, one for the names of ordinary types and one for +the names of structs and unions. Which one +name refers to depends on +foreign-type-spec in the obvious manner. + + + + MAKE-RECORD + make-record + Name + MAKE-RECORD — + Macro + Synopsis + + + make-record typespec + &rest initforms => result + + + Values + typespec + A foreign type specifier, or a keyword which is usedas the name of a foreign struct or union. + initforms + If the type denoted by typespecis scalar, a single value appropriate for that type;otherwise, a list of alternating field names andvalues appropriate for the types of those fields. + result + A macptr which encapsulates the address of anewly-allocated record on the foreign heap. + + + Description + Expands into code which allocates and initalizes +an instance of the type +denoted by typespec, on the foreign +heap. The record is allocated using the C function +malloc, and the user of +make-record must explicitly call +the C function free to deallocate the +record, when it is no longer needed. + If initforms is provided, its value +or values are used in the initialization. When the type +is a scalar, initforms is either a single +value which can be coerced to that type, or no value, in which +case binary 0 is used. When the type is a struct, +initforms is a list, giving field names +and the values for each. Each field is treated in the same way +as a scalar is: If a value for it is given, it must be +coerceable to the field's type; if not, binary 0 is used. + When the type is an array, initforms may +not be provided, because make-record +cannot initialize its values. make-record +is also unable to initialize fields of a struct +which are themselves +structs. The user of +make-record should set these values +by another means. + A possibly-significant limitation is that it must be possible to +find the foreign type at the time the macro is expanded; +make-record signals an error if this is +not the case. + Notes + It is inconvenient that make-record is a +macro, because this means that typespec +cannot be a variable; it must be an immediate value. + If it weren't for this requirement, +make-record could be a function. However, +that would mean that any stand-alone application using it would +have to include a copy of the interface database +(see The Interface Database), which is undesireable +because it's large. + + + + RLET + rlet + Name + RLET — + Macro + Synopsis + + + rlet (var typespec &rest initforms)* + &body body + + + Values + var + A symbol (a lisp variable) + typespec + A foreign type specifier or foreign record name. + initforms + As described above, for + + + Description + Executes body +in an environment in which each var is bound +to a MACPTR (see ) encapsulating the +address of a stack-allocated foreign memory block, allocated and +initialized from typespec and initforms as per +. +Returns whatever value(s) body +returns. + Record fields that aren't explicitly initialized have +unspecified contents. + + + + RLETZ + rletz + Name + RLETZ — + Macro + Synopsis + + + rletz (var typespec &rest initforms)* + &body body + + + Values + var + A symbol (a lisp variable) + typespec + A foreign type specifier or foreign record name. + initforms + As described above, for ccl:make-record + + + Description + Executes body in an environment in which each var is bound +to a MACPTR (see ) encapuslating the +address of a stack-allocated foreign memory block, allocated and +initialized from typespec and initforms as +ccl:make-record. + Returns whatever value(s) body returns. + Unlike rlet, record fields that aren't explicitly +initialized are set to binary 0. + + + + PREF + pref + Name + PREF — + Macro + Synopsis + + + pref ptr accessor-form + + + Values + ptr + a MACPTR (see ). + accessor-form + a keyword which names a foreign type or record, asdescribed in . + + + Description + References an instance of a foreign type (or a component of +a foreign type) accessible via ptr. + Expands into code which references the indicated scalar type +or component, or returns a pointer to a composite type. + PREF can be used with SETF. + RREF is a deprecated alternative to PREF. It accepts a +:STORAGE keyword and rather loudly ignores it. + + + + OPEN-SHARED-LIBRARY + open-shared-library + Name + OPEN-SHARED-LIBRARY — Asks the operating system to load a shared library +for OpenMCL to use. + Function + Synopsis + + + open-shared-library name => library + + + Values + name + A SIMPLE-STRING which is presumed to be the so-name ofor a filesystem path to the library. + library + An object of type SHLIB which describes thelibrary denoted by name. + + + Description + If the library denoted by name can +be loaded by the +operating system, returns an object of type SHLIB that describes +the library; if the library is already open, increments a +reference count. If the library can't be loaded, signals a +SIMPLE-ERROR which contains an often-cryptic message from the +operating system. + Examples + +;;; Try to do something simple. +? (open-shared-library "libgtk.so") +> Error: Error opening shared library "libgtk.so": /usr/lib/libgtk.so: undefined symbol: gdk_threads_mutex +> While executing: OPEN-SHARED-LIBRARY + +;;; Grovel around, curse, and try to find out where "gdk_threads_mutex" +;;; might be defined. Then try again: + +? (open-shared-library "libgdk.so") +# + +? (open-shared-library "libgtk.so") +# + +;;; Reference an external symbol defined in one of those libraries. + +? (external "gtk_main") +# + +;;; Close those libraries. + +? (close-shared-library "libgtk.so") +T + +? (close-shared-library "libgdk.so") +T + +;;; Reference the external symbol again. + +? (external "gtk_main") +# + + Notes + It would be helpful to describe what an soname is and give +examples of one. + Does the SHLIB still get returned if the library is +already open? + + + + CLOSE-SHARED-LIBRARY + close-shared-library + Name + CLOSE-SHARED-LIBRARY — Stops using a shared library, informing the operating +system that it can be unloaded if appropriate. + Function + Synopsis + + + close-shared-library library &key + completely + + Values + library + either an object of type SHLIB, or a string whichdesignates one by its so-name. + completely + a boolean. The default is T. + + + Description + If completely is T, sets the +reference count of library to 0. Otherwise, +decrements it by 1. In either case, if the reference count +becomes 0, close-shared-library +frees all memory resources consumed library +and +causes any EXTERNAL-ENTRY-POINTs known to be defined by it to +become unresolved. + + + + EXTERNAL + external + Name + EXTERNAL — Resolves a reference to an external symbol which +is defined in a shared library. + Macro + Synopsis + + + external name => entry + + + Values + name + a simple-string which names an external symbol.Case-sensitive. + entry + an object of type EXTERNAL-ENTRY-POINT which maintainsthe address of the foreign symbol named byname. + + + Description + If there is already an EXTERNAL-ENTRY-POINT for +the symbol named by name, finds it and +returns it. If not, creates one and returns it. + Tries to resolve the entry point to a memory address, +and identify the containing library. + Be aware that under Darwin, external functions which +are callable from C have underscores prepended to their names, +as in "_fopen". + + + + %FF-CALL + %ff-call + Name + %FF-CALL — + Function + Synopsis + + + %ff-call entrypoint + {arg-type-keyword arg}* &optional result-type-keyword + + + Values + entrypoint + A fixnum or MACPTR + arg-type-keyword + One of the foreign argument-type keywords, describedabove + arg + A lisp value of type indicated by the correspondingarg-type-keyword + result-type-keyword + One of the foreign argument-type keywords, describedabove + + + Description + Calls the foreign function at address entrypoint passing the +values of each arg as a foreign argument of type indicated by the +corresponding arg-type-keyword. Returns the foreign function +result (coerced to a Lisp object of type indicated by +result-type-keyword), or NIL if result-type-keyword is :VOID or +NIL + + + + FF-CALL + ff-call + Name + FF-CALL — + Macro + Synopsis + + + ff-call entrypoint + {arg-type-specifier arg}* &optional result-type-specifier + + + Values + entrypoint + A fixnum or MACPTR + arg-type-specifer + One of the foreign argument-type keywords, describedabove, or an equivalent foreigntype specifier (see ). + arg + A lisp value of type indicated by the correspondingarg-type-specifier + result-type-specifier + One of the foreign argument-type keywords, describedabove, or an equivalent foreigntype specifier (see ). + + + Description + Calls the foreign function at address entrypoint passing the +values of each arg as a foreign argument of type indicated by the +corresponding arg-type-specifier. Returns the foreign function +result (coerced to a Lisp object of type indicated by +result-type-specifier), or NIL if result-type-specifer is :VOID or +NIL + + + + %REFERENCE-EXTERNAL-ENTRY-POINT + %reference-external-entry-point + Name + %REFERENCE-EXTERNAL-ENTRY-POINT — + Function + Synopsis + + + %reference-external-entry-point eep + + + Values + eep + An EXTERNAL-ENTRY-POINT, as obtained by the EXTERNALmacro. + + + Description + Tries to resolve the address of the EXTERNAL-ENTRY-POINT +eep; returns a fixnum representation of that address if +successful, else signals an error. + + + + EXTERNAL-CALL + external-call + Name + EXTERNAL-CALL — + Macro + Synopsis + + + external-call name + {arg-type-specifier arg}* &optional result-type-specifier + + + Values + name + A lisp string. See external, above. + arg-type-specifer + One of the foreign argument-type keywords, describedabove, or an equivalent foreigntype specifier (see ). + arg + A lisp value of type indicated by the correspondingarg-type-specifier + result-type-specifier + One of the foreign argument-type keywords, describedabove, or an equivalent foreigntype specifier (see ). + + + Description + Calls the foreign function at the address obtained by +resolving the external-entry-point associated with name, passing +the values of each arg as a foreign argument of type indicated by +the corresponding arg-type-specifier. Returns the foreign function +result (coerced to a Lisp object of type indicated by +result-type-specifier), or NIL if result-type-specifer is :VOID or +NIL + + + + FOREIGN-SYMBOL-ENTRY + foreign-symbol-entry + Name + FOREIGN-SYMBOL-ENTRY — + Function + Synopsis + + + foreign-symbol-entry name + + + Values + name + A lisp string. + + + Description + Tries to resolve the address of the foreign symbol name. If +successful, returns a fixnum representation of that address, else +returns NIL. + + + + FOREIGN-SYMBOL-ADDRESS + foreign-symbol-address + Name + FOREIGN-SYMBOL-ADDRESS — + Function + Synopsis + + + foreign-symbol-address name + + + Values + name + A lisp string. + + + Description + Tries to resolve the address of the foreign symbol name. If +successful, returns that address encapsulated +in a MACPTR (see ), else returns +NIL. + + + + DEFCALLBACK + defcallback + Name + DEFCALLBACK — + Macro + Synopsis + + + defcallback name + ({arg-type-specifier var}* &optional result-type-specifier) + &body body + + + Values + name + A symbol which can be made into a special variable + arg-type-specifer + One of the foreign argument-type keywords, describedabove, or an equivalent foreigntype specifier (see ).In addition, if the keyword:WITHOUT-INTERRUPTS is specified, the callback will beexecuted with lisp interrupts disabled if the correspondingvar is non-NIL. If :WITHOUT-INTERRUPTS is specified morethan once, the rightmost instance wins. + var + A symbol (lisp variable), which will be bound to avalue of the specified type. + body + A sequence of lisp forms, which should return a valuewhich can be coerced to the specified result-type. + + + Description + Proclaims name +to be a special variable; sets its value to a +MACPTR which, when called by foreign code, calls a lisp function +which expects foreign arguments of the specified types and which +returns a foreign value of the specified result type. Any argument +variables which correspond to foreign arguments of type :ADDRESS +are bound to stack-allocated MACPTRs. + If name +is already a callback function pointer, its value is +not changed; instead, it's arranged +that an +updated version of the lisp callback function will be called. +This feature allows for callback functions to be redefined +incrementally, just like Lisp functions are. + defcallback +returns the callback pointer, e.g., the +value of name. + + + + SHARP-AMPERSAND + #_ + Name + #_ — + Reader Macro + Description + Reads a symbol from the current input stream, with *PACKAGE* +bound to the "OS" package and with readtable-case preserved. + Does a lookup on that symbol +in the OpenMCL interface database (see ), +signalling +an error if no foreign function information can be found for the +symbol in any +active interface directory (see ). + Notes the foreign function information, including the foreign +function's return type, the number and type of the foreign +function's required arguments, and an indication of whether or +not the function accepts additional arguments (via e.g., the +"varargs" mechanism in C). + Defines a macroexpansion function on the symbol, which expand +macro calls involving the symbol into EXTERNAL-CALL forms where +foreign argument type specifiers for required arguments and the +return value specifer are provided from the information ind the +database. + Returns the symbol. + The effect of these steps is that it's possible to call +foreign functions that take fixed numbers of arguments by simply +providing argument values, as in: + +(#_isatty fd) +(#_read fd buf n) + + and to call foreign functions that take variable numbers of +arguments by specifying the types of non-required args, as in: + +(with-cstrs ((format-string "the answer is: %d")) + (#_printf format-string :int answer)) + + + + + SHARP-AMPERSAND + SHARP-AMPERSAND + Name + #& — + Reader Macro + Description + In OpenMCL 1.0 and later, the #& reader macro This +functionality was introduced as #? in 0.14.2, but ANSI CL requires +that #? be reserved for the user. #? is still supported as an alias +to #&, but that will change in the next release. can be used to +access foreign variables; this functionality depends on the presence +of "vars.cdb" files in the interface database. The current behavior of +the #& reader macro is to: + Read a symbol from the current input stream, with *PACKAGE* +bound to the "OS" package and with readtable-case preserved. + Use that symbol's pname to access the OpenMCL interface +database, signalling an error if no appropriate foreign variable +information can be found with that name in any active interface +directory. + Use type information recorded in the database to construct a +form which can be used to access the foreign variable, and return +that form. + Please note that the set of foreign variables declared in header files +may or may not match the set of foreign variables exported from +libraries (we're generally talking about C and Unix here ...). When +they do match, the form constructed by the #& reader macro manages the +details of resolving and tracking changes to the foreign variable's +address. + Future extensions (via prefix arguments to the reader macro) may +offer additional behavior; it might be convenient (for instance) to be +able to access the address of a foreign variable without dereferencing +that address. + Foreign variables in C code tend to be platform- and +package-specific (the canonical example - "errno" - is typically +not a variable when threads are involved. ) + In LinuxPPC, + +? #&stderr + + returns a pointer to the stdio error stream ("stderr" is a +macro under OSX/Darwin). + On both LinuxPPC and DarwinPPC, + +? #&sys_errlist + + returns a pointer to a C array of C error message strings. + + + + USE-INTERFACE-DIR + use-interface-dir + Name + USE-INTERFACE-DIR — + Function + Synopsis + + + use-interface-dir dir-id + + + Values + dir-id + A keyword whose pname, mapped to lower case, names asubdirectory of "ccl:headers;" (or"ccl:darwin-headers;") + + + Description + Tells OpenMCL to add the interface directory denoted by +dir-id to the list of interface directories which it consults for +foreign type and function information. Arranges that that +directory is searched before any others. + Note that use-interface-dir +merely adds an entry +to a search list. +If the named directory doesn't exist in the file system +or doesn't +contain a set of database files, a runtime error may occur +when OpenMCL +tries to open some database file in that directory, and it +will try to +open such a database file whenever it needs to find any +foreign type or +function information. +may come in +handy in that case. + Examples + One typically wants interface information to be +available at compile-time (or, in many cases, at read-time). +A typical idiom would be: + +(eval-when (:compile-toplevel :execute) + (use-interface-dir :GTK)) + + Using the :GTK interface directory makes available +information on +foreign types, functions, and constants. It's generally +necessary to +load foreign libraries before actually calling the +foreign code, which for GTK can be done like this: + +(load-gtk-libraries) + + It should now be possible to do things like: + +(#_gtk_widget_destroy w) + + + + + UNUSE-INTERFACE-DIR + unuse-interface-dir + Name + UNUSE-INTERFACE-DIR — + Function + Synopsis + + + unuse-interface-dir dir-id + + + Values + dir-id + A keyword whose pname, mapped to lower case, names asubdirectory of "ccl:headers;" (or"ccl:darwin-headers;") + + + Description + Tells OpenMCL to remove the interface directory denoted by +dir-id from the list of interface directories which are +consulted for +foreign type and function information. Returns T if the directory +was on the search list, NIL otherwise. + + + + TERMINATE-WHEN-UNREACHABLE + terminate-when-unreachable + Name + TERMINATE-WHEN-UNREACHABLE — + Function + Synopsis + + + terminate-when-unreachable object + + + Values + object + A CLOS object of a class for which there existsa method of the generic functionccl:terminate. + + + Description + The "termination" mechanism is a way to have the garbage +collector run a function right before an object is about to +become garbage. It is very similar to the "finalization" +mechanism which Java has. It is not standard Common Lisp, +although other Lisp implementations have similar features. +It is useful when there is some sort of special cleanup, +deallocation, or releasing of resources which needs to happen +when a certain object is no longer being used. + When the garbage collector discovers that an object is no +longer referred to anywhere in the program, it deallocates +that object, freeing its memory. However, if +ccl:terminate-when-unreachable has been +called on the object at any time, the garbage collector first +invokes the generic function ccl:terminate, +passing it the object as a parameter. + Therefore, to make termination do something useful, you need to +define a method on ccl:terminate. + Because calling +ccl:terminate-when-unreachable only +affects a single object, rather than all objects of its +class, you +may wish to put a call to it in the +initialize-instance method of a +class. Of course, this is only appropriate if you do in fact +want to use termination for all objects of a given class. + Example + +(defclass resource-wrapper () + ((resource :accessor resource))) + +(defmethod initialize-instance :after ((x resource-wrapper) &rest initargs) + (ccl:terminate-when-unreachable x)) + +(defmethod ccl:terminate ((x resource-wrapper)) + (when (resource x) + (deallocate (resource x)))) + + See Also + Tutorial: Allocating Foreign Data on the Lisp Heap + + + + + + The Objective-C Bridge +OS X APIs use a language called "Objective C", which is built on C. +The Objective-C bridge makes it possible to work with ObjC +objects and classes from Lisp, and to define classes in Lisp which +can be used by ObjC. + The ultimate +purpose of the ObjC and Cocoa bridges is to make Cocoa as +easy as possible to use from OpenMCL, in order to support the +development of GUI applications and IDEs. The eventual goal, +which is much closer than it used to be, is complete integration of +Cocoa into CLOS (whatever that means). + The current release provides Lisp-like syntax and naming +conventions for the basic ObjC operations, with automatic +type processing and messages checked for validity at +compile-time. It also provides some convenience facilities +for working with Cocoa. + + + Using Objective-C Classes +The class of most "standard" CLOS classes is the +class named STANDARD-CLASS. In the Objective-C object model, each +class is an instance of a (usually unique) metaclass, which is +itself an instance of a "base" metaclass (often the +metaclass of the class named "NSObject".) So, the +Objective-C class named "NSWindow" and the ObjC class +"NSArray" are (sole) instances of their distinct +metaclasses whose names are also "NSWindow" and +"NSArray", respectively. (In the Objective-C world, +it's much more common and useful to specialize class behavior +such as instance allocation.) + When foreign libraries containing Objective-C classes are first +loaded, the classes they contain are identified. The foreign class +name, such as "NSWindow", is mapped to an external symbol in the +"NS" package via the bridge's translation rules, such as NS:NS-WINDOW. +A similar transformation happens to the +metaclass name, with a "+" prepended, yielding something like +NS:+NS-WINDOW. + These classes are integrated into CLOS such that the +metaclass is an instance of the class OBJC:OBJC-METACLASS and +the class +is an instance of the metaclass. SLOT-DESCRIPTION metaobjects are +created for each instance variable, and the class and metaclass go +through something very similar to the "standard" CLOS class +initialization protocol (with a difference being that these classes +have already been allocated.) + Performing all this initialization, which is done when you +(require "COCOA"), currently takes several +seconds; it could conceivably be sped up some, but it's never likely +to be fast. + When the process is complete, CLOS is aware of several hundred +new ObjC classes and their metaclasses. OpenMCL's runtime system can +reliably recognize MACPTRs to ObjC classes as being CLASS objects, and +can (fairly reliably but heuristically) recognize instances of those +classes (though there are complicating factors here; see below.) +SLOT-VALUE can be used to access (and, with care, set) instance +variables in ObjC instances. To see this, do: + +? (require "COCOA") + + and, after waiting a bit longer for a Cocoa listener window to +appear, activate that Cocoa listener and do: + ? (describe (ccl::send ccl::*NSApp* 'key-window)) + + This sends a message asking for the key window, which is the window +that has the input focus (often the frontmost), and then describes +it. As we can see, NS:NS-WINDOWs have lots of interesting slots. + + + + Instantiating Objective-C Objects +Making an instance of an ObjC class (whether the class in question +is predefined or defined by the application) involves calling +MAKE-INSTANCE with the class and a set of initargs as arguments. +As with STANDARD-CLASS, making an instance involves initializing +(with INITIALIZE-INSTANCE) an object allocated with +ALLOCATE-INSTANCE. + For example, you can create an ns:ns-number like this: + +? (make-instance 'ns:ns-number :init-with-int 42) +# + + It's worth looking at how this would be done if you were +writing in Objective C: + +[[NSNumber alloc] initWithInt: 42] + + Allocating an instance of an ObjC class involves sending the +class an "alloc" message, and then using those initargs that +don't correspond to slot initags as the +"init" message to be sent to the newly-allocated instance. So, the +example above could have been done more verbosely as: + +? (defvar *n* (ccl::send (find-class 'ns:ns-number) 'alloc)) +*N* + +? (setq *n* (ccl::send *n* :init-with-int 42)) +# + + That setq is important; this is a case where init +decides to replace the object and return the new one, instead +of modifying the existing one. +In fact, if you leave out the setq and +then try to view the value of *N*, OpenMCL will freeze. There's +little reason to ever do it this way; this is just to show +what's going on. + You've seen that an ObjC initialization method doesn't have to +return the same object it was passed. In fact, it doesn't have +to return any object at all; in this case, the initialization fails +and make-instance returns nil. + In some special cases, such as loading an ns:ns-window-controller +from a .nib file, it may be necessary for you to pass the +instance itself as one of the parameters to the initialization +method. It goes like this: + +? (defvar *controller* + (make-instance 'ns:ns-window-controller)) +*CONTROLLER* + +? (setq *controller* + (ccl::send *controller* + :init-with-window-nib-name #@"DataWindow" + :owner *controller*)) +# (#x1FB520)> + + This example calls (make-instance) with no initargs. When you +do this, the object is only allocated, and not initialized. It +then sends the "init" message to do the initialization by hand. + + + + Calling Objective-C Methods +In Objective-C, methods are called "messages", and there's +a special syntax to send a message to an object: + +[w alphaValue] +[w setAlphaValue: 0.5] +[v mouse: p inRect: r] + + The first line sends the method "alphaValue" to the object +w, with no parameters. The second line +sends the method "setAlphaValue", with the parameter 0.5. +The third line sends the method "mouse:inRect:" - yes, all +one long word - with the +parameters p and r. + In Lisp, these same three lines are: + +(send w 'alpha-value) +(send w :set-alpha-value 0.5) +(send v :mouse p :in-rect r) + + Notice that when a method has no parameters, its name is an ordinary +symbol (it doesn't matter what package the symbol is in, as +only its name is checked). When a method has parameters, +each part of its name is a keyword, and the keywords alternate +with the values. + These two lines break those rules, and both will +result in error messages: + +(send w :alpha-value) +(send w 'set-alpha-value 0.5) + + Instead of (send), you can also invoke (send-super), with the +same interface. It has roughly the same purpose as CLOS's +(call-next-method); when you use (send-super), the message is +handled by the superclass. This can be used to get at the +original implementation of a method when it is shadowed by a +method in your subclass. + + + Type Coercion for ObjC Method Calls +OpenMCL's FFI handles many common conversions between Lisp +and foreign data, such as unboxing floating-point args and +boxing floating-point results. The bridge adds a few more +automatic conversions: + NIL is equivalent to (%NULL-PTR) for any message argument that +requires a pointer. + T/NIL are equivalent to #$YES/#$NO for any boolean argument. + A #$YES/#$NO returned by any method that returns BOOL will be +automatically converted to T/NIL. + To make this last conversion work, the bridge has to engage +in a bit of hackery. The bridge uses ObjC run-time type +info. Unfortunately, BOOL is typed as CHAR by ObjC. Thus, +a method that returns CHAR might actually return only BOOL, +or it might return any CHAR. + The bridge currently assumes +that any method that returns CHAR actually returns BOOL. +But it provides a facility for defining exceptions to this +assumption: (DEFINE-RETURNS-BOOLEAN-EXCEPTION "charValue"). +Eventually, the best way to handle issues like this is +probably to get our method type info directly from the +header files rather than using ObjC's runtime type system. + Note that no automatic conversion is currently performed between +Lisp strings and NSStrings. However, there is a convenient +reader macro for creating constant NSStrings: + +(SEND W :SET-TITLE #@"My Window") + + Note that #@"Hello" is a full ObjC object, so +messages can be sent to it: (SEND #@"Hello" 'LENGTH). + To go in the other direction, use the function +(ccl::lisp-string-from-nsstring). + To create variable NSStrings (which you will later +want to change the +values of), consider using CCL::NS-LISP-STRING. + + + + Methods which Return Structures +Some Cocoa methods return small structures, such as those used +to represent points, rects, sizes and ranges. When writing in +Objective C, the compiler hides the implementation details. +Unfortunately, in Lisp we must be slightly more aware of them. + Methods which return structures are called in a special way; +the caller allocates space for the result, and passes a pointer +to it as an extra argument to the method. This is called a +Structure Return, or STRET. Don't look at me; I don't name +these things. + Here's a simple use of this in Objective C. The first line +sends the "bounds" message to v1, which returns a rectangle. +The second line sends the "setBounds" message to v2, passing +that same rectangle as a parameter. + +NSRect r = [v1 bounds]; +[v2 setBounds r]; + + In Lisp, we must explicitly allocate the memory, which is +done most easily and safely with . +We do it like this: + +(rlet ((r :ect)) + (send/stret r v1 'bounds) + (send v2 :set-bounds r)) + + The rlet allocates the storage (but doesn't initialize it), +and makes sure that it will be deallocated when we're done. +It binds the variable r to refer to it. +The call to send/stret is just like +an ordinary call to +send, except that r is passed as an +extra, first parameter. The third line, which calls +send, does not need to do anything +special, because there's nothing complicated about passing +a structure as a parameter. + In order to make STRETs easier to use, the bridge provides two +conveniences. + First, you can use the macros slet +and slet* to allocate and +initialize local variables to foreign structures in one +step. The example above could have been written more tersely +as: + +(slet ((r (send v1 'bounds))) + (send v2 :set-bounds r)) + + Second, when one call to send is made +inside another, the inner one has an implicit +slet around it. So, one could in fact +just write: + +(send v1 :set-bounds (send v2 'bounds)) + + There are also several psuedo-functions provided for convenience +by the ObjC compiler, to make objects of specific types. The +following are currently supported by the bridge: NS-MAKE-POINT, +NS-MAKE-RANGE, NS-MAKE-RECT, and NS-MAKE-SIZE. + These pseudo-functions can be used within an SLET initform: + +(slet ((p (ns-make-point 100.0 200.0))) + (send w :set-frame-origin p)) + + Or within a call to send: + +(send w :set-origin (ns-make-point 100.0 200.0)) + + However, since these aren't real functions, a call like the +following won't work: + +(setq p (ns-make-point 100.0 200.0)) + + To extract fields from these objects, there are also some +convenience macros: NS-MAX-RANGE, NS-MIN-X, +NS-MIN-Y, NS-MAX-X, NS-MAX-Y, NS-MID-X, NS-MID-Y, +NS-HEIGHT, and NS-WIDTH. + Note that there is also a send-super/stret +for use within methods. Like send-super, +it ignores any shadowing methods in a subclass, and calls the +version of a method which belongs to its superclass. + + + + Variable-Arity Messages +There are a few messages in Cocoa which take variable numbers +of arguments. Perhaps the most common examples involve +formatted strings: + +[NSClass stringWithFormat: "%f %f" x y] + + In Lisp, this would be written: + +(send (find-class 'ns:ns-string) + :string-with-format #@"%f %f" + (:double-float x :double-float y)) + + Note that it's necessary to specify the foreign types of the +variables (in this example, :double-float), because the +compiler has no general way of knowing these types. (You +might think that it could parse the format string, but this +would only work for format strings which are not determined +at runtime.) + Because the ObjC runtime system does not provide any information +on which messages are variable arity, they must be explicitly +declared. The standard variable arity messages in Cocoa are +predeclared by the bridge. If you need to declare a new +variable arity message, use +(DEFINE-VARIABLE-ARITY-MESSAGE "myVariableArityMessage:"). + + + + Optimization +The bridge works fairly hard to optimize message sends, when +it has enough information to do so. There are two cases when +it does. In either, a message send should be nearly as +efficient as when writing in Objective C. + The first case is when both the message +and the receiver's class are known at +compile-time. In general, the only way the receiver's class +is known is if you declare it, which you can do with +either a DECLARE or a THE form. For example: + +(send (the ns:ns-window w) 'center) + + Note that there is no way in ObjC to name the class of a +class. Thus the bridge provides a declaration, @METACLASS. +The +type of an instance of "NSColor" is ns:ns-color. The type of +the class +"NSColor" is (@metaclass ns:ns-color): + +(let ((c (find-class 'ns:ns-color))) + (declare ((ccl::@metaclass ns:ns-color) c)) + (send c 'white-color)) + + The other case that alllows optimization is when only the +message is known at compile-time, but its type signature +is unique. Of the more-than-6000 messages currently provided by +Cocoa, only about 50 of them have nonunique type signatures. + An example of a message with a type signature that is not +unique is SET. It returns VOID for NSColor, but ID for NSSet. +In order to optimize sends of messages with nonunique type +signatures, the class of the receiver must be declared at +compile-time. + If the type signature is nonunique or the message is unknown +at compile-time, then a slower runtime call must be used. + When the receiver's class is unknown, the bridge's ability to +optimize relies on a type-signature table which it maintains. +When first loaded, the bridge initializes this table +by scanning every method of every ObjC class. When new methods +are defined later, the table must be updated. This happens +automatically when you define methods in Lisp. After any +other major change, such as loading an external framework, +you should rebuild the table: + +? (update-type-signatures) + + Because send and its relatives +send-super, +send/stret, and +send-super/stret are macros, they cannot +be funcalled, applyed, +or passed as arguments to functions. + To work around this, +there are function equivalents to them: +%send, +%send-super, +%send/stret, and +%send-super/stret. However, these +functions should be used only when the macros will not do, +because they are unable to optimize. + + + + + Defining Objective-C Classes +You can define your own foreign classes, which can then be +passed to foreign functions; the methods which you implement +in Lisp will be made available to the foreign code as +callbacks. + You can also define subclasses of existing classes, implementing +your subclass in Lisp even though the parent class was in +Objective C. One such subclass is +CCL::NS-LISP-STRING. +It is also particularly useful to make subclasses of +NS-WINDOW-CONTROLLER. + We can use the MOP to define new Objective-C classes, +but we have to do something a little +funny: the :METACLASS that we'd want to use in a DEFCLASS option +generally doesn't exist until we've created the class (recall +that ObjC classes have, for the sake of argument, unique and private +metaclasses.) We can sort of sleaze our way around this by specifying +a known ObjC metaclass object name as the value of +the DEFCLASS :METACLASS object; the metaclass of the root class +NS:NS-OBJECT, NS:+NS-OBJECT, makes a good choice. To make a subclass +of NS:NS-WINDOW (that, for simplicity's sake, doesn't define any +new slots), we could do: + +(defclass example-window (ns:ns-window) + () + (:metaclass ns:+ns-object)) + + That'll create a new ObjC class named EXAMPLE-WINDOW whose +metaclass is the class named +EXAMPLE-WINDOW. The class will be an +object of type OBJC:OBJC-CLASS, and the metaclass will be of type +OBJC:OBJC-METACLASS. EXAMPLE-WINDOW will be a subclass of +NS-WINDOW. + + + Defining classes with foreign slots +If a slot specification in an Objective-C class definition +contains the keyword :FOREIGN-TYPE, the slot will be a "foreign +slot" (i.e. an ObjC instance variable). Be aware that it is an +error to redefine an ObjC class so that its foreign slots +change in any way, and OpenMCL doesn't do anything consistent +when you try to. + The value of the :FOREIGN-TYPE initarg should be a foreign type +specifier. For example, if we wanted (for some reason) to define a +subclass of NS:NS-WINDOW that kept track of the number of key events +it had received (and needed an instance variable to keep that +information in), we could say: + +(defclass key-event-counting-window (ns:ns-window) + ((key-event-count :foreign-type :int + :initform 0 + :accessor window-key-event-count)) + (:metaclass ns:+ns-object)) + + Foreign slots are always SLOT-BOUNDP, and the initform +above is redundant: foreign slots are initialized to binary 0. + + + + Defining classes with Lisp slots +A slot specification in an ObjC class definition that doesn't +contain the :FOREIGN-TYPE initarg defines a pretty-much normal lisp +slot that'll happen to be associated with "an instance of a +foreign class". For instance: + +(defclass hemlock-buffer-string (ns:ns-string) + ((hemlock-buffer :type hi::hemlock-buffer + :initform hi::%make-hemlock-buffer + :accessor string-hemlock-buffer)) + (:metaclass ns:+ns-object)) + + As one might expect, this has memory-management +implications: we +have to maintain an association between a MACPTR and a set of lisp +objects (its slots) as long as the ObjC instance exists, and we have +to ensure that the ObjC instance exists (does not have its -dealloc +method called) while lisp is trying to think of it as a first-class +object that can't be "deallocated" while it's still +possible to reference it. Associating one or more lisp objects with a +foreign instance is something that's often very useful; if you +were to do this "by hand", you'd have to face many of the +same memory-management issues. + + + + + Defining Objective-C Methods +In ObjC, unlike in CLOS, every method belongs to some +particular class. This is probably not a strange +concept to you, because C++ and Java do the same thing. +When you use Lisp to define ObjC methods, it is only +possible to define methods belonging to ObjC classes which +have been defined in Lisp. + The macro define-objc-method is used +for this. As described in , the names +of ObjC methods are broken into pieces, each piece followed +by a parameter. The types of all parameters must be explicitly +declared. + Right now, I'm not sure how to formally describe the usage +of define-objc-method, so I'm going to do it with some short +examples. Let us define a class to use in them: + +(defclass data-window-controller (ns:ns-window-controller) + ((window :foreign-type :id :accessor window) + (data :initform nil :accessor data)) + (:metaclass ns:+ns-object)) + + There's nothing special about this class. +It inherits from ns:ns-window-controller. +It has two slots: window is a +foreign slot, stored in the ObjC world; and +data is an ordinary slot, stored in +the Lisp world. + Here is an example of how to define a method which takes +no arguments. It happens to be an initialization method, +but that's not important: + + (define-objc-method ((:id get-window) + data-window-controller) + (window self)) + + The return type of this method is the foreign type :id, which +is used for all ObjC objects. +The name of the method is +get-window. The body of the method is +the single line (window self). The variable +self is bound, within the body, to the +instance which is receiving the message. The call to +window +uses the CLOS accessor to get the value of the window field. + Here's an example which takes a parameter. Notice that +the name of the method without a parameter was an ordinary +symbol, but with a parameter, it's a keyword: + +(define-objc-method ((:id :init-with-multiplier (:int multiplier)) + data-window-controller) + (setf (data self) (make-array 100)) + (dotimes (i 100) + (setf (aref (data self) i) + (* i multiplier))) + self) + + To Objective-C code which uses the class, the name of this +method is "initWithMultiplier:". The name of the parameter +is multiplier, and its type is :int. +The body of the method +does some meaningless things. Then it returns +self, because +this is an initialization method. + Here's an example with more than one parameter: + +(define-objc-method ((:id :init-with-multiplier (:int multiplier) + :and-addend (:int addend)) + data-window-controller) + (setf (data self) (make-array size)) + (dotimes (i 100) + (setf (aref (data self) i) + (+ (* i multiplier) + addend))) + self) + + To Objective-C, the name of this method is +"initWithMultiplier:andAddend:". Both parameters are +of type :int; the first is named multiplier, +and the second is addend. Again, the +method returns self. + Here is a method which does not return any value, a so-called +"void method". Where our other methods said :id, this one says +:void for the return type: + +(define-objc-method ((:void :take-action (:id sender)) + data-window-controller) + (declare (ignore sender)) + (dotimes (i 100) + (setf (aref (data self) i) + (- (aref (data self) i))))) + + This method would be called "takeAction:" in ObjC. +The convention for methods that are going to be used as Cocoa +actions is that they take one parameter, which is the object +responsible for triggering the action. However, this method +doesn't actually need to use that parameter, so it explicitly +ignores it to avoid a compiler warning. As promised, the +method doesn't return any value. + There is also an alternate syntax, illustrated here. The following two method definitions are equivalent: + +(define-objc-method ("applicationShouldTerminate:" + "LispApplicationDelegate") + (:id sender :) + (declare (ignore sender)) + nil) + +(define-objc-method ((: + :application-should-terminate sender) + lisp-application-delegate) + (declare (ignore sender)) + nil) + + + Method Redefinition Constraints +Objective C was not designed, as Lisp was, with runtime +redefinition in mind. So, there are a few constraints about +how and when you can replace the definition of an Objective C +method. Currently, if you break these rules, nothing will +collapse, but the behaviour will be confusing; so don't. + Objective C methods can be redefined at runtime, but their +signatures shouldn't change. That is, the types of the arguments +and the return type have to stay the same. The reason for this +is that changing the signature changes the selector which is used +to call the method. + When a method has already been defined in one class, and you +define it in a subclass, shadowing the original method, they +must both have the same type signature. There is no such +constraint, though, if the two classes aren't related and the +methods just happen to have the same name. + + + + + How Objective-C Names are Mapped to Lisp Symbols +There is a standard set of naming conventions for Cocoa classes, +messages, etc. As long as they are followed, the bridge is fairly +good at automaticallly translating between ObjC and Lisp names. + For example, "NSOpenGLView" becomes ns:ns-opengl-view; +"NSURLHandleClient" becomes ns:ns-url-handle-client; and +"nextEventMatchingMask:untilDate:inMode:dequeue:" becomes +(:next-event-matching-mask :until-date :in-mode :dequeue). +What a mouthful. + To see how a given ObjC or Lisp name will be translated by the +bridge, you can use the following functions: + + + + + + (ccl::objc-to-lisp-classname string) + + + + (ccl::lisp-to-objc-classname symbol) + + + + (ccl::objc-to-lisp-message string) + + + + (ccl::lisp-to-objc-message string) + + + + (ccl::objc-to-lisp-init string) + + + + (ccl::lisp-to-objc-init keyword-list) + + + + + + Of course, there will always be exceptions to any naming convention. +Please tell us on the mailing lists if you come across any +name translation problems +that seem to be bugs. Otherwise, the bridge provides two ways of +dealing with exceptions: + First, you can pass a string as the class name of +MAKE-OBJC-INSTANCE and as the message to SEND. These strings will +be directly interpreted as ObjC names, with no translation. This +is useful for a one-time exception. For example: + +(ccl::make-objc-instance "WiErDclass") +(ccl::send o "WiErDmEsSaGe:WithARG:" x y) + + Alternatively, you can define a special translation rule for your +exception. This is useful for an exceptional name that you need +to use througout your code. Some examples: + +(ccl::define-classname-translation "WiErDclass" wierd-class) +(ccl::define-message-translation "WiErDmEsSaGe:WithARG:" (:weird-message :with-arg)) +(ccl::define-init-translation "WiErDiNiT:WITHOPTION:" (:weird-init :option)) + + The normal rule in ObjC names is that each word begins with a +capital letter (except possibly the first). Using this rule +literally, "NSWindow" would be translated as N-S-WINDOW, which +seems wrong. "NS" is a special word in ObjC that should not be +broken at each capital letter. Likewise "URL", "PDF", "OpenGL", +etc. Most common special words used in Cocoa are already defined +in the bridge, but you can define new ones as follows: + +(ccl::define-special-objc-word "QuickDraw") + + Note that message keywords in a SEND such as +(SEND V :MOUSE P :IN-RECT R) may +look like the keyword arguments in a Lisp function call, but they +really aren't. All keywords must be +present and the order is significant. Neither (:IN-RECT :MOUSE) nor +(:MOUSE) translate to "mouse:inRect:" + Also, as a special exception, an "init" prefix is optional in the +initializer keywords, so +(MAKE-OBJC-INSTANCE 'NS-NUMBER :INIT-WITH-FLOAT 2.7) can also +be expressed as +(MAKE-OBJC-INSTANCE 'NS-NUMBER :WITH-FLOAT 2.7) + + + + + Using OpenMCL with Mac OS X, the Darwin Kernel, and the Cocoa GUI API + + + Differences in OpenMCL between Darwin and MacOS X +Most of the documentation and whatever experience you may have in using OpenMCL under LinuxPPC should apply to using it under Darwin/MacOS X. There are some differences between the platforms, and these differences are sometimes exposed in the implementation. + + + FASL file segregation. +99% of the compiled lisp code in the two implementations is +identical. Since both implementations target the PowerPC, this +shouldn't be too surprising. + The other 1% - code which deals with foreign functions or data or +which deals directly with the operating system - is very different but +superficially similar. The effects of running such code compiled for +the wrong platform range from harmless to catastrophic. + During bootstrapping, the 1% proved troublesome enough that it seemed +wise to strictly separate the two implementations. + FASL files have different file types (.dfsl for Darwin, .pfsl for +LinuxPPC) and contain different identifying information; it's not +possible to successfully load FASL files compiled for one +implementation into the other. + I think that it's wise to maintain this distinction; I spent a lot +less time recompiling the 99% of "non-troublesome" code than I'd been +spending tracking down bugs caused by the other 1%. + + + + File-system case +Darwin and MacOS X use HFS+ file systems by default; HFS+ file systems +are case-insensitive. Most of OpenMCL's filesystem and pathname code +assumes that the underlying filesystem is case-sensitive; this +assumption extends to functions like EQUAL, which assumes that #p"FIXTHIS" +and #p"foo" denote different, un-EQUAL filenames. Since Darwin/MacOS X +can also use UFS and NFS filesystems, the opposite assumption would be +no more correct than the one that's currently made. + Whatever the best solution to this problem turns out to be, there are +some practical considerations. Doing: + +? (save-application "DPPCCL") + + has the unfortunate side-effect of trying to overwrite the Darwin +OpenMCL kernel, "dppccl", on a case-insensitive filesystem. + To work around this, the Darwin OpenMCL kernel expects the default +heap image file name to be the kernel's own filename with the string +".image" appended, so the idiom would be: + ? (save-application "dppccl.image") + + + + + Line Termination Characters +MacOSX effectively supports two distinct line-termination +conventions. Programs in its Darwin substrate follow the Unix +convention of recognizing #\LineFeed as a line terminator; traditional +MacOS programs use #\Return for this purpose. + OpenMCL follows the Unix convention on both Darwin and LinuxPPC, +but offers (as of version 0.11) some support for reading and writing +files that use the MacOS convention as well. + This support (and anything like it) is by nature heuristic: it +can successfully hide the distinction between newline conventions +much of the time, but could mistakenly change the meaning of +otherwise correct programs (typically when files contain both +#\Return and #\Linefeed characters or when files contain mixtures of +text and binary data.) Because of this concern, the default settings +of some of the variables that control newline translation and +interpretation are somewhat conservative. + Although the issue of multiple newline conventions primarily +affects MacOSX users, the functionality described here is available +under LinuxPPC as well (and may occasionally be useful there.) + None of this addresses issues related +to the third newline convention ("CRLF") in widespread use +(since that convention isn't native to any platform on which +OpenMCL currently runs). If OpenMCL is ever ported to such a +platform, that issue might be revisited. + Note that some MacOS programs (including some versions of +commercial MCL) may use HFS file type information to recognize TEXT +and other file types and so may fail to recognize files created with +OpenMCL or other Darwin applications (regardless of line termination +issues.) + Unless otherwise noted, the symbols mentioned in this +documentation are exported from the CCL package. + + + + Single-precision trig & transcendental functions +Despite what Darwin's man pages say, early versions of its math library +(up to and including OSX 10.2 (Jaguar) don't implement +single-precision variants of the transcendental and trig functions +(#_sinf, #_atanf, etc.) OpenMCL works around this by coercing +single-precision args to double-precision, calling the +double-precision version of the math library function, and coercing +the result back to a SINGLE-FLOAT. These steps can introduce rounding +errors (and potentially overflow conditions) that might not be present +or as severe if true 32-bit variants were available. + + + + Shared libraries +Darwin/MacOS X distinguishes between "shared libraries" and "bundles" +or "extensions"; Linux doesn't. In Darwin, "shared libraries" have the +file type "dylib" : the expectation is that this class of file is +linked against when executable files are created and loaded by the OS +when the executable is launched. The latter class - +"bundles/extensions" - are expected to be loaded into and unloaded +from a running application, via a mechanism like the one used by +OpenMCL's OPEN-SHARED-LIBRARY function. + OpenMCL wants to treat all shared libs as if they were "bundles"; this +is presumably possible, but none of it's implemented under OpenMCL +0.10. Ideally, whatever gets implemented will hide the distinction as +much as possible, but it may not be possible to hide it completely. + + + + + Unix/Posix/Darwin Features +OpenMCL has several convenience functions which allow you to +make Posix (portable Unix) calls without having to use the +foreign-function interface. Each of these corresponds directly +to a single Posix function call, as it might be made in C. There +is no attempt to make these calls correspond to Lisp idioms, such as +setf. This means that their behaviour is +simple and predictable. + For working with environment variables, there are +CCL::GETENV and CCL::SETENV. + For working with user and group IDs, there are +CCL::GETUID, CCL::SETUID, +and CCL::SETGID. To find the home directory +of an arbitrary user, as set in the user database (/etc/passwd), +there is CCL::GET-USER-HOME-DIR. + For process IDs, there +is CCL::GETPID. + For the system() function, there is +CCL::OS-COMMAND. Ordinarily, it is better - +both more efficient and more predictable - to use the features +described in . However, +sometimes you may want to specifically ask the shell to invoke a +command for you. + + + + Cocoa Programming in OpenMCL +Cocoa is one of Apple's APIs for GUI programming; for most purposes, +development is considerably faster with Cocoa than with the +alternatives. You should have a little familiarity with it, to +better understand this section. + A small sample Cocoa program can be invoked by evaluating (REQUIRE +'TINY) and then (CCL::TINY-SETUP). This program provides a simple +example of using several of the bridge's capabilities. + The Tiny demo creates Cocoa objects dynamically, at runtime, which is +always an option. However, for large applications, it is usually more +convenient to create your objects with Apple Interface Builder, and +store them in .nib files to be loaded when needed. Both approaches +can be freely mixed in a single program. + + + The Command Line and the Window System +OpenMCL is ordinarily a command-line application (it doesn't have a +connection to the OSX Window server, doesn't have its own menubar or +dock icon, etc.) By opening some libraries and jumping through some +hoops, it's able to sort of transform itself into a full-fledged GUI +application (while retaining its original TTY-based listener.) The +general idea is that this hybrid environment can be used to test and +protoype UI ideas and the resulting application can eventually be +fully transformed into a bundled, double-clickable application. This +is to some degree possible, but there needs to be a bit more +infrastructure in place before many people would find it easy. + Cocoa applications use the NSLog function to write +informational/warning/error messages to the application's standard +output stream. When launched by the Finder, a GUI application's +standard output is diverted to a logging facility that can be +monitored with the Console application (found in +/Applications/Utilities/Console.app). In the hybrid environment, the +application's standard output stream is usually the initial listener's +standard output stream. With two different buffered stream mechanisms +trying to write to the same underlying Unix file descriptor, it's not +uncommon to see NSLog output mixed with lisp output on the initial +listener. + + + + Writing (and reading) Cocoa code +The syntax of the constructs used to define Cocoa classes and methods +has changed a bit (it was never documented outside of the source code +and never too well documented at all), largely as the result of +functionality offered by Randall Beer's bridge; the “standard +name-mapping conventions” referenced below are described in his +CocoaBridgeDoc.txt file, as are the constructs used to invoke (“send +messages to”) Cocoa methods. + All of the symbols described below are currently internal to +the CCL package. + + + + + + CCL::@CLASS + + + + CCL::@SELECTOR + + + + CCL::DEFINE-OBJC-METHOD + + + + CCL::DEFINE-OBJC-CLASS-METHOD + + + + + + + + + The Application Kit and Multiple Threads +The Cocoa API is broken into several pieces. The Application Kit, +affectionately called AppKit, is the one which deals with window +management, drawing, and handling events. AppKit really wants all +these things to be done by a "distinguished thread". creation, and +drawing to take place on a distinguished thread. + Apple has published some guidelines which discuss these issues in some +detail; see +the Apple Multithreading Documentation, and in particular the guidelines +on +Using the Application Kit from Multiple Threads. The upshot is that there +can sometimes be unexpected behavior when objects are created in +threads other than the distinguished event thread; eg, the event +thread sometimes starts performing operations on objects that haven't +been fully initialized. + It's certainly more convenient to do certain types of exploratory +programming by typing things into a listener or evaluating a “defun” +in an Emacs buffer; it may sometimes be necessary to be aware of this +issue while doing so. + Each thread in the Cocoa runtime system is expected to maintain a +current “autorelease pool” (an instance of the NSAutoreleasePool +class); newly created objects are often added to the current +autorelease pool (via the -autorelease method), and periodically the +current autorelease pool is sent a “-release” message, which causes +it to send “-release” messages to all of the objects that've been +added to it. + If the current thread doesn't have a current autorelease pool, the +attempt to autorelease any object will result in a severe-looking +warning being written via NSLog. The event thread maintains an +autorelease pool (it releases the current pool after each event is +processed and creates a new one for the next event), so code that only +runs in that thread should never provoke any of these severe-looking +NSLog messages. + To try to suppress these messages (and still participate in the Cocoa +memory management scheme), each listener thread (the initial listener +and any created via the “New Listener” command in the IDE) is given +a default autorelease pool; there are REPL colon-commands for +manipulating the current listener's “toplevel auturelease pool”. + In the current scheme, every time that Cocoa calls lisp code, a lisp +error handler is established which maps any lisp conditions to ObjC +exceptions and arranges that this exception is raised when the +callback to lisp returns. Whenever lisp code invokes a Cocoa method, +it does so with an ObjC exception handler in place; this handler maps +ObjC exceptions to lisp conditions and signals those conditions. + Any unhandled lisp error or ObjC exception that occurs during the +execution of the distinguished event thread's event loop causes a +message to be NSLog'ed and the event loop to (try to) continue +execution. Any error that occurs in other threads is handled at the +point of the outermost Cocoa method invocation. (Note that the error +is not necessarily “handled” in the dynamic context in which it +occurs.) + Both of these behaviors could possibly be improved; both of them +seem to be substantial improvements over previous behaviors (where, +for instance, a misspelled message name typically terminated the +application.) + + + + Acknowledgement +The Cocoa bridge was originally developed, and generously contributed +by, Randal Beer. + + + + + Building an Application Bundle +You may have noticed that (require "COCOA") takes a long time to load. +It is possible to avoid this by saving a Lisp heap image which has +everything already loaded. There is an example file which allows you +to do this, "ccl/examples/cocoa-application.lisp", by producing a +double-clickable application which runs your program. First, load +your own program. Then, do: + +? (require "COCOA-APPLICATION") + + When it finishes, you should be able to double-click the OpenMCL icon +in the ccl directory, to quickly start your program. + The OS may have already decided that OpenMCL.app isn't a valid +executable bundle, and therefore won't let you double-click it. +If this happens to you, to force it to reconsider, just update the +last-modified time of the bundle. In Terminal: + > touch OpenMCL.app + + There is one important caveat. + Because of the way that the ObjC bridge currently works, a saved +image is dependent upon the exact versions of +the Cocoa libraries which were present when it was saved. +Specifically, the interface database is. So, for example, an +application produced under OS X 10.3.5 will not work under +OS X 10.3.6. This is inconvenient when you wish to distribute an +application you have built this way. + Work in this direction is ongoing. It is worth looking at the project +"Bosco", by Mikel Evins, which is a template that can be used to build +application bundles in a different way. It is +available here, as part +of the "Clotho" project, and there is +here. + When an image which had contained ObjC classes (which are also +CLOS classes) is re-launched, those classes are "revived": all +preexisting classes have their addresses updated destructively, so that +existing subclass/superclass/metaclass relationships are maintained. +It's not possible (and may never be) to preserve foreign +instances across SAVE-APPLICATION. (It may be the case that NSArchiver +and NSCoder and related classes offer some approximation of that.) + + + + Recommended Reading + Cocoa Documentation + This is the top page for all of Apple's documentation onCocoa. If you are unfamiliar with Cocoa, it is a goodplace to start. + Foundation Reference for Objective-C + This is one of the two most important Cocoa references; itcovers all of the basics, except for GUI programming. This isa reference, not a tutorial. + Application Kit Reference for Objective-C + This is the other; it covers GUI programming with Cocoain considerable depth. This is a reference, not a tutorial. + Apple Developer Documentation + This is the site which the above two documents are found on;go here to find the documentation on any other Apple API.Also go here if you need general guidance about OS X, Carbon,Cocoa, Core Foundation, or Objective C. + + + + + + Operating-System Dictionary + + + CCL::GETENV + getenv + Name + CCL::GETENV — + Function + Synopsis + +getenv name => value + + Arguments and Values + name + a string which is the name of an existingenvironment variable;case-sensitive + value + if there is an environment variable namedname, its value, as a string; if thereis not, NIL + + + Description + Looks up the value of the environment variable named by +name, in the OS environment. + + + + CCL::SETENV + setenv + Name + CCL::SETENV — + Function + Synopsis + +setenv name value => errno + + Arguments and Values + name + a string which is the name of a new or existingenvironment variable;case-sensitive + value + a string, to be the new value of theenvironment variablenamed by name + errno + zero if the function call completes successfully;otherwise, a platform-dependent integer which describesthe problem + + + Description + Sets the value of the environment variable named by +name, in the OS environment. If there is +no such environment +variable, creates it. + + + + CCL::CURRENT-DIRECTORY-NAME + current-directory-name + Name + CCL::CURRENT-DIRECTORY-NAME — + Function + Synopsis + +current-directory-name + => path + + Values + path + a string, an absolute pathname in Posix format - withdirectory components separated by slashes + + + Description + Looks up the current working directory of the OpenMCL process; +unless it has been changed, this is the directory OpenMCL was +started in. + + + + CCL::GETUID + getuid + Name + CCL::GETUID — + Function + Synopsis + +getuid => uid + + Values + uid + a non-negative integer, identifying a specific useraccount as defined in the OS user database + + + Description + Returns the ("real") user ID of the current user. + + + + CCL::SETUID + setuid + Name + CCL::SETUID — + Function + Synopsis + +setuid uid => errno + + Arguments and Values + uid + a non-negative integer, identifying a specific useraccount as defined in the OS user database + errno + zero if the function call completes successfully;otherwise, a platform-dependent integer which describesthe problem + + + Description + Attempts to change the current user ID (both "real" and +"effective"); fails unless +the OpenMCL process has super-user privileges or the ID +given is that of the current user. + + + + CCL::SETGID + setgid + Name + CCL::SETGID — + Function + Synopsis + +setgid gid => errno + + Arguments and Values + gid + a non-negative integer, identifying a specificgroup as defined in the OS user database + errno + zero if the function call completes successfully;otherwise, a platform-dependent integer which describesthe problem + + + Description + Attempts to change the current group ID (both "real" and +"effective"); fails unless +the OpenMCL process has super-user privileges or the ID +given is that of a group to which the current user belongs. + + + + CCL::GETPID + getpid + Name + CCL::GETPID — + Function + Synopsis + +getpid => pid + + Values + pid + a non-negative integer, identifying an OS process + + + Description + Returns the ID of the OpenMCL OS process. + + + + CCL::GET-USER-HOME-DIR + get-user-home-dir + Name + CCL::GET-USER-HOME-DIR — + Function + Synopsis + +get-user-home-dir + uid => path + + Values + uid + a non-negative integer, identifying a specific useraccount as defined in the OS user database + path + a string, an absolute pathname in Posix format - withdirectory components separated by slashes; or NIL + + + Description + Looks up and returns the defined home directory of the user +identified by uid. This value comes from the +OS user database, not from the $HOME +environment variable. Returns NIL if there is no user with +the ID uid. + + + + CCL::OS-COMMAND + os-command + Name + CCL::OS-COMMAND — + Function + Synopsis + +os-command command-line + => exit-code + + Values + command-line + a string, obeying all the whitespace andescapingconventions required by the user's default system shell + + + exit-code + a non-negative integer, returned as the exitcode of a subprocess; zero indicates success + + + Description + Invokes the Posix function system(), which +invokes the user's default system shell (such as +sh or tcsh) as a new process, and has that shell execute +command-line. + If the shell was able to find the command specified in +command-line, then exit-code +is the exit code of that command. If not, it is the exit +code of the shell itself. + Notes + By convention, an exit code of 0 indicates success. There are +also other conventions; unfortunately, they are OS-specific, and +the portable macros to decode their meaning are implemented +by the system headers as C preprocessor macros. This means +that there is no good, automated way to make them available +to Lisp. + + + + CCL::@CLASS + @class + Name + CCL::@CLASS — + Macro + Synopsis + +@class class-name + + Arguments and Values + class-name + a string which denotes an existing class name, or asymbol which can be mapped to such a string via the standardname-mapping conventions for class names + + + Description + Used to refer to a known ObjC class by name. (Via the use +LOAD-TIME-VALUE, the results of a class-name -> class lookup +are cached.) + @class is obsolete as of late 2004, because +find-class now works on ObjC classes. It is described here +only because some old code still uses it. + + + + CCL::@SELECTOR + @selector + Name + CCL::@SELECTOR — + Macro + Synopsis + +@selector string + + Arguments and Values + string + a string constant, used to canonically refer to anObjC method selector + + + Description + Used to refer to an ObjC method selector (method name). Uses +LOAD-TIME-VALUE to cache the result of a string -> selector +lookup. + + + + CCL::DEFINE-OBJC-METHOD + define-objc-method + Name + CCL::DEFINE-OBJC-METHOD — + Macro + Synopsis + +define-objc-method + (selector class-name) &body body + + Arguments and Values + selector + either a string which represents the name of theselector or a list which describ+es the method's returntype, selector components, and argument types (see below.)If the first form is used, then the first form in the bodymust be a list which describes the selector's argumenttypes and return value type, as per DEFCALLBACK. + class-name + either a string which names an existing ObjC classname or a list symbol which can map to such a string via thestandard name-mapping conventions for class names. (Notethat the "canonical" lisp class name is such asymbol) + + + Description + Defines an ObjC-callable method which implements the +specified message selector for instances of the existing ObjC +class class-name. + + + + CCL::DEFINE-OBJC-CLASS-METHOD + define-objc-class-method + Name + CCL::DEFINE-OBJC-CLASS-METHOD — + Macro + Synopsis + +define-objc-class-method + (selector class-name) &body body + + Arguments and Values + As per DEFINE-OBJC-METHOD + Description + Like DEFINE-OBJC-METHOD, only used to define methods on the +class named by class-name and on its +subclasses. + For both DEFINE-OBJC-METHOD and DEFINE-OBJC-CLASS-METHOD, the +"selector" argument can be a list whose first element is a +foreign type specifier for the method's return value type and whose +subsequent elements are either: + + a non-keyword symbol, which can be mapped to a selector stringfor a parameterless method according to the standard name-mappingconventions for method selectors. + a list of alternating keywords and variable/type specifiers,where the set of keywords can be mapped to a selector string for aparameteriezed method according to the standard name-mappingconventions for method selectors and each variable/type-specifier iseither a variable name (denoting a value of type :ID) or a list whoseCAR is a variable name and whose CADR is the correspondingargument's foreign type specifier. + + + + + + CCL:*ALTERNATE-LINE-TERMINATOR* + *alternate-line-terminator* + Name + CCL:*ALTERNATE-LINE-TERMINATOR* — + Variable + Description + This variable is currently only used by the standard reader macro +function for #\; (single-line comments); that function reads successive +characters until EOF, a #\NewLine is read, or a character EQL to the +value of *alternate-line-terminator* is read. In OpenMCL for Darwin, the +value of this variable is initially #\Return ; in OpenMCL for LinuxPPC, +it's initially NIL. + Their default treatment by the #\; reader macro is the primary way +in which #\Return and #\Linefeed differ syntactally; by extending the +#\; reader macro to (conditionally) treat #\Return as a +comment-terminator, that distinction is eliminated. This seems to make +LOAD and COMPILE-FILE insensitive to line-termination issues in many +cases. It could fail in the (hopefully rare) case where a LF-terminated +(Unix) text file contains embedded #\Return characters, and this +mechanism isn't adequate to handle cases where newlines are embedded +in string constants or other tokens (and presumably should be translated +from an external convention to the external one) : it doesn't change +what READ-CHAR or READ-LINE "see", and that may be necessary to +handle some more complicated cases. + + + + :EXTERNAL-FORMAT + :external-format + Name + :EXTERNAL-FORMAT — + Keyword Argument + Description + Per ANSI CL, OpenMCL supports the :EXTERNAL-FORMAT keyword +argument to the functions OPEN, LOAD, and COMPILE-FILE. This argument is +intended to provide a standard way of providing implementation-dependent +information about the format of files opened with an element-type of +CHARACTER. This argument can meaningfully take on the values :DEFAULT +(the default), :MACOS, :UNIX, or :INFERRED in OpenMCL. + When defaulted to or specified as :DEFAULT, the format of the file +stream is determined by the value of the variable +CCL:*DEFAULT-EXTERNAL-FORMAT*. See below. + When specified as :UNIX, all characters are read from and written +to files verbatim. + When specified as :MACOS, all #\Return characters read from the +file are immediately translated to #\Linefeed (#\Newline); all #\Newline +(#\Linefeed) characters are written externally as #\Return characters. + When specified as :INFERRED and the file is open for input, the +first bufferful of input data is examined; if a #\Return character +appears in the buffer before the first #\Linefeed, the file stream's +external-format is set to :MACOS; otherwise, it is set to :UNIX. + All other values of :EXTERNAL-FORMAT - and any combinations that +don't make sense, such as trying to infer the format of a +newly-created output file stream - are treated as if :UNIX was +specified. As mentioned above, the :EXTERNAL-FORMAT argument doesn't +apply to binary file streams. + The translation performed when :MACOS is specified or inferred has +a somewhat greater chance of doing the right thing than the +*alternate-line-terminator* mechanism does; it probably has a somewhat +greater chance of doing the wrong thing, as well. + + + + CCL:*DEFAULT-EXTERNAL-FORMAT* + *default-external-format* + Name + CCL:*DEFAULT-EXTERNAL-FORMAT* — + Variable + Description + The value of this variable is used when :EXTERNAL-FORMAT is +unspecified or specified as :DEFAULT. It can meaningfully be given any +of the values :UNIX, :MACOS, or :INFERRED, each of which is interpreted +as described above. + Because there's some risk that unsolicited newline translation +could have undesirable consequences, the initial value of this variable +in OpenMCL is :UNIX. + + + + CCL::NS-LISP-STRING + ns-lisp-string + Name + CCL::NS-LISP-STRING — + Class + Superclasses + NS:NS-STRING + Initargs + :string + a Lisp string which is to be the content ofthe newly-created ns-lisp-string. + + + Description + This class +implements the interface of an NSString, which means that it can +be passed to any Cocoa or Core Foundation function which expects +one. + The string itself is stored on the Lisp heap, which +means that its memory management is automatic. However, the +ns-lisp-string object itself is a foreign +object (that is, it has an objc metaclass), and resides on the +foreign heap. Therefore, it is necessary to explicitly free +it, by sending a dealloc message. + Examples + You can create an ns-lisp-string with +make-instance, just like +any normal Lisp class: + +? (defvar *the-string* + (make-instance 'ccl::ns-lisp-string + :string "Hello, Cocoa.")) + + When you are done with the string, you must explicitly +deallocate it: + +? (ccl::send *the-string* 'dealloc) + + You may wish to use an unwind-protect +form to ensure that this happens: + +(let (*the-string*) + (unwind-protect (progn (setq *the-string* + (make-instance 'ccl::ns-lisp-string + :string "Hello, Cocoa.")) + (format t "~&The string is ~D characters long.~%" + (ccl::send *the-string* 'length))) + (when *the-string* + (ccl::send *the-string* 'dealloc)))) + + Notes + Currently, ns-lisp-string is defined in +the file ccl/examples/cocoa-backtrace.lisp, which is a +rather awkward place. It was probably not originally meant +as a public utility at all. It would be good if it were +moved someplace else. Use at your own risk. + + + + + + Understanding and Configuring the Garbage Collector + + + Heap space allocation +Release 0.10 or later of OpenMCL uses a different memory management +scheme than previous versions did. Those earlier versions would allocate a +block of memory (of specified size) at startup and would allocate lisp +objects within that block. When that block filled with live (non-GCed) +objects, the lisp would signal a "heap full" condition. The heap +size imposed a limit on the size of the largest object that could be +allocated. + The new strategy involves reserving a very large (2GB on DarwinPPC32, +1GB on LinuxPPC, "very large" on 64-bit implementations) block at +startup and consuming (and relinquishing) its contents as the size of +the live lisp heap data grows and shrinks. After the initial heap image +loads and after each full GC, the lisp kernel will try to ensure that a +specified amount (the "lisp-heap-gc-threshold") of free memory is +available. The inital value of this kernel variable is 16MB on 32-bit +implementations and 32MB on 64-bit implementations ; it can be +manipulated from Lisp (see below.) + The large reserved memory block consumes very little in the way of +system resources; memory that's actually committed to the lisp heap +(live data and the "threshold" area where allocation takes place) +consumes finite resources (physical memory and swap space). The lisp's +consumption of those resources is proportional to its actual memory usage, +which is generally a good thing. + This scheme is much more flexible than the old one, but it may also +increase the possibility that those resources can become exhausted. +Neither the new scheme nor the old handles that situation gracefully; +under the old scheme, a program that consumes lots of memory may have run +into an artificial limit on heap size before exhausting virtual memory. + The -R or –heap-reserve command-line option can be use to limit the +size of the reserved block and therefore bound heap expansion. Running + +> openmcl --heap-reserve 8M + + would provide an execution environment that's very similar to +that provided by earlier OpenMCL versions. + + + + The Ephemeral GC +For many programs, the following observations are true to a very +large degree: + + Most heap-allocated objects have very short lifetimes ("areephemeral"): they become inaccessible soon after they're created. + Most non-ephemeral objects have very long lifetimes: it'srarely productive for the GC to consider reclaiming them, sinceit's rarely able to do so. (An object that's survived a largenumber of GCs is likely to survive the next one. That's not alwaystrue of course, but it's a reasonable heuristic.) + It's relatively rare for an old object to be destructivelymodified (via SETF) so that it points to a new one, therefore mostreferences to newly-created objects can be found in the stacks andregisters of active threads. It's not generally necessary to scanthe entire heap to find references to new objects (or to prove thatsuch references don't exists), though it is necessary to keeptrack of the (hopefully exceptional) cases where old objects aremodified to point at new ones. + + "Ephemeral" (or "generational") garbage collectors try to exploit +these observations: by concentrating on frequently reclaiming +newly-created objects quickly, it's less often necessary to do more +expensive GCs of the entire heap in order to reclaim unreferenced memory. +In some environments, the pauses associated with such full GCs can be +noticable and disruptive, and minimizing the frequency (and sometimes the +duration) of these pauses is probably the EGC's primary goal (though +there may be other benefits, such as increased locality of reference and +better paging behavior.) The EGC generally leads to slightly longer +execution times (and slightly higher, amortized GC time), but there are +cases where it can improve overall performance as well; the nature and +degree of its impact on performance is highly application-dependant. + Most EGC strategies (including the one employed by OpenMCL) +logically or physically divide memory into one or more areas of relatively +young objects ("generations") and one or more areas of old objects. +Objects that have survived one or more GCs as members of a young +generation are promoted (or "tenured") into an older generation, where +they may or may not survive long enough to be promoted to the next +generation and eventually may become "old" objects that can only be +reclaimed if a full GC proves that there are no live references to them. +This filtering process isn't perfect - a certain amount of premature +tenuring may take place - but it usually works very well in practive. + It's important to note that a GC of the youngest generation is +typically very fast (perhaps a few milliseconds on a modern CPU, depending +on various factors), OpenMCL's EGC is not concurrent and doesn't +offer realtime guarantees. + OpenMCL's EGC maintains three ephemeral +generations; all newly created objects are created as members of the +youngest generation. Each generation has an associated +threshold, which indicates the number of bytes in it +and all younger generations that can be allocated before a GC is +triggered. These GCs will involve the target generation and all younger +ones (and may therefore cause some premature tenuring); since the older +generations have larger thresholds, they're GCed less frequently and +most short-lived objects that make it into an older generation tend not to +survive there very long. + The EGC can be enabled or disabled +under program control; under some circumstances, it may be enabled but +inactive (because a full GC is imminent.) Since it +may be hard to know or predict the consing behavior of other threads, the +distinction between the "active" and "inactive" state isn't very +meaningful, especially when native threads are involved. + + + + GC Page reclamation policy + After a full GC finishes, it'll try to ensure that at least +(LISP-HEAP-GC-THRESHOLD) of virtual memory are available; objects will be +allocated in this block of memory until it fills up, the GC is triggered, +and the process repeats itself. + Many programs reach near stasis in terms of the amount of logical +memory that's in use after full GC (or run for long periods of time in +a nearly static state), so the logical address range used for consing +after the Nth full GC is likely to be nearly or entirely identical to the +address range used by the N+1th full GC. + By default (and traditionally in OpenMCL), the GC's policy is to +"release" the pages in this address range: to advise the virtual memory +system that the pages contain garbage and any physical pages associated +with them don't need to be swapped out to disk before being reused and +to (re-)map the logical address range so that the pages will be +zero-filled by the virtual memory system when they're next accessed. +This policy is intended to reduce the load on the VM system and keep +OpenMCL's working set to a minimum. + For some programs (especially those that cons at a very high rate), +the default policy may be less than ideal: releasing pages that're +going to be needed almost immediately - and zero-fill-faulting them back +in, lazily - incurs unnecessary overhead. (There's a false economy +associated with minimizing the size of the working set if it's just +going to shoot back up again until the next GC.) A policy of "retaining" +pages between GCs might work better in such an environment. + Functions described below give the user some control over this +behavior. An adaptive, feedback-mediated approach might yield a better +solution. + + + + "Pure" areas are read-only, paged from image file +SAVE-APPLICATION identifies code vectors and the pnames of interned +symbols and copies these objects to a "pure" area of the image +file it creates. (The "pure" area accounts for most of what the +ROOM function reports as "static" space.) + When the resulting image file is loaded, the pure area of the file +is now memory-mapped with read-only access. Code and pure data are paged +in from the image file as needed (and don't compete for global virtual +memory resources with other memory areas.) + Code-vectors and interned symbol pnames are immutable : it is an +error to try to change the contents of such an object. Previously, that +error would have manifested itself in some random way. In the new scheme, +it'll manifest itself as an "unhandled exception" error in the +Lisp kernel. The kernel could probably be made to detect a spurious, +accidental write to read-only space and signal a lisp error in that case, +but it doesn't yet do so. + The image file should be opened and/or mapped in some mode which +disallows writing to the memory-mapped regions of the file from other +processes. I'm not sure of how to do that; writing to the file when +it's mapped by OpenMCL can have unpredictable and unpleasant results. +SAVE-APPLICATION will delete its output file's directory entry and +create a new file; one may need to exercise care when using file system +utilities (like tar, for instance) that might overwrite an existing image +file. + + + + Weak Hash Tables +In general, a "weak reference" is a reference to an object which +will not prevent the object from being garbage-collected. For +example, suppose that you want to keep a list of all the objects +of a certain type. If you don't take special steps, the fact that +you have a list of them will mean that the objects are always +"live", because you can always reference them through the list. +Therefore, they will never be garbage-collected, and their memory +will never be reclaimed, even if they are referenced nowhere else +in the program. You may want this behaviour. If you don't, you +need weak references. + OpenMCL supports weak references with "weak hash tables". +Hash tables may be weak with respect to either their keys or +their values. To make a hash table with weak keys, invoke +make-hash-table with the option :weak t, +or, equivalently, :weak :key. To make one with weak values, +use :weak :value. When the key is weak, the equality test +must be #'eq (because it wouldn't make sense otherwise). + When garbage-collection occurs, key-value pairs are removed +from the hash table if there are no other references to the +weak element of the pair (key or value). + In general, weak-key hash tables are useful when you want to +use the hash to store some extra information about the objects +you look up in it, while weak-value hash tables are useful when you +want to use the hash as an index for looking up objects. + If you are experimenting with weak hash tables interactively, remember +that an object is not dead if it was returned by one of the last +three interactively-evaluated expressions, because of the variables +*, **, and +***. The easy workaround is to evaluate some +meaningless expression before invoking gc, +to get the object out of the repl variables. + + + + Garbage-Collection Dictionary + + + LISP-HEAP-GC-THRESHOLD + lisp-heap-gc-threshold + Name + LISP-HEAP-GC-THRESHOLD — + Function + Synopsis + +lisp-heap-gc-threshold + + Description + Returns the value of the kernel variable that specifies the +amount of free space to leave in the heap after full GC. + + + + SET-LISP-HEAP-GC-THRESHOLD + set-lisp-heap-gc-threshold + Name + SET-LISP-HEAP-GC-THRESHOLD — + Function + Synopsis + + + lisp-heap-gc-threshold new-threshold + + + Arguments and Values + new-value + The requested new lisp-heap-gc-threshold. + + + Description + Sets the value of the kernel variable that specifies the +amount of free space to leave in the heap after full GC to +new-value, which should be a non-negative fixnum. Returns the +value of that kernel variable (which may be somewhat larger than +what was specified). + + + + USE-LISP-HEAP-GC-THRESHOLD + use-lisp-heap-gc-threshold + Name + USE-LISP-HEAP-GC-THRESHOLD — + Function + Synopsis + + + use-lisp-heap-gc-threshold + + + Description + Tries to grow or shrink lisp's heap space, so that the +free space is (approximately) equal to the current heap threshold. +Returns NIL + + + + EGC + egc + Name + EGC — + Function + Synopsis + +egc arg + + Arguments and Values + arg + a generalized boolean + + + Description + Enables the EGC if arg is non-nil, disables the EGC +otherwise. Returns the previous enabled status. Although this +function is thread-safe (in the sense that calls to it are +serialized), it doesn't make a whole lot of sense to be +turning the EGC on and off from multiple threads ... + + + + EGC-ENABLED-P + egc-enabled-p + Name + EGC-ENABLED-P — + Function + Synopsis + +egc-enabled-p + + Description + Returns T if the EGC was enabled at the time of the call, +NIL otherwise. + + + + EGC-ACTIVE-P + egc-active-p + Name + EGC-ACTIVE-P — + Function + Synopsis + +egc-active-p + + Description + Returns T if the EGC was active at the time of the call, NIL +otherwise. Since this is generally a volatile piece of +information, it's not clear whether this function serves a +useful purpose when native threads are involved. + + + + EGC-CONFIGURATION + egc-configuration + Name + EGC-CONFIGURATION — + Function + Synopsis + +egc-configuration + + Description + Returns, as multiple values, the sizes in kilobytes of the +thresholds associated with the youngest ephemeral generation, the +middle ephemeral generation, and the oldest ephemeral generation + + + + CONFIGURE-EGC + configure-egc + Name + CONFIGURE-EGC — + Function + Synopsis + +configure-egc + generation-0-size generation-1-size + generation-2-size + + Arguments and Values + generation-0-size + the requested threshold size of the youngestgeneration, in kilobytes + generation-1-size + the requested threshold size of the middle generation,in kilobytes + generation-2-size + the requested threshold size of the oldest generation,in kilobytes + + + Description + If the EGC is currently disabled, puts the indicated +threshold sizes in effect and returns T, otherwise, returns NIL. +(The provided threshold sizes are rounded up to a multiple of +64Kbytes in OpenMCL 0.14 and later, and to a multiple of 32KBytes in earlier +versions.) + + + + GC-RETAIN-PAGES + gc-retain-pages + Name + GC-RETAIN-PAGES — + Function + Synopsis + +gc-retain-pages arg + + Arguments and Values + arg + a generalized boolean + + + Description + Tries to influence the GC to retain/recycle the pages +allocated between GCs if arg is true, and to release them +otherwise. This is generally a tradeoff between paging and other +VM considerations. + + + + GC-RETAINING-PAGES + gc-retaining-pages + Name + GC-RETAINING-PAGES — + Function + Synopsis + +gc-retaining-pages + + Description + Returns T if the GC tries to retain pages between full GCs +and NIL if it's trying to release them to improve VM paging +performance. + + + + + + Implementation Details of OpenMCL +This chapter describes many aspects of OpenMCL's implementation as of +(roughly) version 1.1. Details vary a bit between the three archutectures +(PPC32, PPC64, and X86-64) currently supported and those details change +over time, so the definitive reference is the source code (especially +some files in the ccl/compiler/ directory whose names contain the string +"arch" and some files in the ccl/lisp-kernel/ directory whose namee +contain the string "constants".) Hopefully, this chapter will make +it easier for someone who's interested to read and understand the +contents of those files. + + + Threads and exceptions +OpenMCL's threads are "native" (meaning that they're scheduled and +controlled by the operating system.) Most of the implications of +this are discussed elsewhere; this section tries to describe how +threads look from the lisp kernel's perspective (and especailly +from the GC's point of view.) + OpenMCL's runtime system tries to use machine-level exception +mechanisms (conditional traps when available, illegal instructions, +memory access protection in some cases) to detect and handle ... +exceptional situations. These situations include some TYPE-ERRORs +and PROGRAM-ERRORS (notably wrong-number-of-args errors), and also +include cases like "not being able to allocate memory without GCing +or obtaining more memory from the OS." The general idea is that +it's usually faster to pay (very occasional) exception-processing +overhead and figure out what's going on in an exception handler than +it is to maintain enough state and context to handle an exceptional +case via a lighter-weight mechanism when that exceptional case +(by definition) rarely occurs. + Some emulated execution environments (the Rosetta PPC emulator on +x86 versions of OSX) don't provide accurate exception information +to exception handling functions. OpenMCL can't run in such environments. + + + The Thread Context Record +When a lisp thread is first created (or when a thread created by +foreign code first calls back to lisp), a data structure called +a Thread Context Record (or TCR) is allocated and initialized. On +modern versions of Linux and FreeBSD, the allocation actually happens +via a set of thread-local-storage ABI extensions, so a thread's +TCR is created when the thread is created and dies when the thread +dies. (The World's Most Advanced Operating System - as Apple's +marketing literature refers to Darwin - is not very advanced in this +regard, and I know of no reason to assume that advances will be made +in this area anytime soon.) + A TCR contains a few dozen fields (and is therefore a few hundred +bytes in size.) The fields are mostly thread-specific information +about the thread's stacks' locations and sizes, information about +the underlying (POSIX) thread, and information about the thread's +dynamic binding history and pending CATCH/UNWIND-PROTECTs. Some +of this information could be kept in individual machine registers +while the thread is running (and the PPC - which has more registers +available - keeps a few things in registers that the X86-64 has +to access via the TCR), but it's important to remember that the +information is thread-specific and can't (for instance) be kept +in a fixed global memory location. + When lisp code is running, the current thread's TCR is kept in +a register. On PPC platforms, a general purpose register is used; +on x86-64, an (otherwise nearly useless) segment register works +well (prevents the expenditure of a more generally useful general- +purpose register for this purpose.) + The address of a TCR is aligned in memory in such a way that a +FIXNUM can be used to represent it. The lisp function CCL::%CURRENT-TCR +returns the calling thread's TCR as a fixnum; actual value of the +TCR's address is 4 or 8 times the value of this fixnum. + When the lisp kernel initializes a new TCR, it's added to a global +list maintained by the kernel; when a thread exits, its TCR is removed +from this list. + When a thread calls foreign code, lisp stack pointers are saved in its +TCR, lisp registers (at least those whose value should be preserved +across the call) are saved on the thread's value stack, and (on x86-64) +RSP is switched to the control stack. A field in the TCR +(tcr.valence) is then set to indicate that the thread is running foreigm +code, foreign argument registers are loaded from a frame on the foreign +stack, and the foreign function is called. (That's a little oversimplified +and possibly inaccurate, but the important things to note are that the +thread "stops following lisp stack and register usage conventions" and +that it advertises the fact that it's done so. Similar transitions in +a thread's state ("valence") occur when it enters of exits an exception +handler (which is sort of an OS/hardware-mandated foreign function call +where the OS thoughtfully saves the thread's register state for it +beforehand.) + + + + Exception contexts, and exception-handling in general +Unix-like OSes tend to refer to exceptions as "signals"; the same +general mechanism ("signal handling") is used to process both +asynchronous OS-level events (such as the result of the keyboard driver +noticing that ^C or ^Z has been pressed) and synchronous hardware-level +events (like trying to execute and illegal instruction or access +protected memory.) It makes some sense to defer ("block") handling +of aysnchronous signals so that some critical code sequences complete +without interruption; since it's generally not possible for a thread +to proceed after a synchronous exception unless and until its state +is modified by an exception handler, it makes no sense to talk about +blocking synchronous signals (though some OSes will let you do so +and doing so can have mysterious effects.) + On OSX/Darwin, the POSIX signal handling facilities coexist with +lower-level Mach-based exception handling facilities. Unfortunately, +the way that this is implemented interacts poorly with debugging +tools: GDB will generally stop whenever the target program +encounters a Mach-level exception and offers no way to proceed from +that point (and let the program's POSIX signal handler try to handle +the exception); Apple's CrashReporter program has had a similar issue +and, depending on how it's configured, may bombard the user with +alert dialogs which falsely claim that an application has crashed +(when in fact the application in question has routinely handled +a routine exception.) On Darwin/OSX, OpenMCL uses Mach thread-level +exception handling facilities which run before GDB or CrashReporter +get a chance to confuse themeselves; OpenMCL's Mach exception handling +tries to force the thread which received a synchronous exception to +invoke a signal handling function ("as if" signal handling worked +more usefully under Darwin.) Mach exception handlers run in a dedicated +thread (which basically does nothing but wait for exception messages +from the lisp kernel, obtain and modify information about the state +of threads in which exceptions have occurred, and reply to the +exception messages with an indication that the exception has been +handled. The reply from a thread-level exception handler keeps the +exception from being reported to GDB or CrashReporter and avoids +the problems related to those programs. Since OpenMCL's Mach exception +handler doesn't claim to handle debugging-related exceptions (from +breakpoints or single-step operations), it's possible to use GDB to +debug OpenMCL. + On platforms where signal handling and debugging don't get in each +other's way, a signal handler is entered with all signals blocked. +(This behavior is specified in the call to the sigaction() function +which established the signal handler.) The signal handler recieves +three arguments from the OS kernel; the first is an intger which +identifies the signal, the second is a pointer to an object of +type "siginfo_t", which may or may not contain a few fields that +would help to identify the cause of the exception, and the third +argument is a pointer to a data structure (called a "ucontext" +or something similar) which contains machine-dependent information +about the state of the tread at the time that the exception/signal +occurred. While asynchronous signals are blocked, the signal handler +stores the pointer to its third argument (the "signal context") in +a field in the current thread's TCR, sets some bits in another TCR +field to indicate that the thread is now waiting to handle an +exception, unblocks asynchronous signals, and waits for a global +exception lock which serializes exception processing. + On Darwin, the Mach exception thread creates a signal context (and +maybe a siginfo_t structure), stores the signal context in the +thread's TCR, sets the TCR field wich describes the thread's state, +and arranges that the thread resume execution at its signal handling +function (with a signal handler, possibly NULL siginfo_t, and signal +context as arguments. When the thread resumes, it waits for the +global exception lock. + On x86-64 platforms where signal handing can be used to handle +synchronous exceptions, there's an additional complication: the +OS kernel ordinarily allocates the signal context and siginfo +structures on the stack of the thread which received the signal; +in practice, that means "wherever RSP is pointing." OpenMCL's + require that the +thread's value stack - where RSP is usually pointing while lisp +code is running - contain only "nodes" (properly tagged lisp +objects), and scribbling a signal context all over the value +stack would violate this requirement. To maintain consistency, +the sigaltstack() mechanism is used to cause the signal to be +delivered on (and the signal context and siginfo to be allocated +on) a special stack area (the last few pages of the thread's +cntrol stack, in practice. When the signal handler runs, it +(carefully) copies the signal context and siginfo to the thread's +control stack and makes RSP point into that stack before invoking +the "real" signal handler. (The effect of this hack is that the +"real" signal handler always runs on the thread's control stack.) + Once the exception handler has obtained the global exception lock, +it uses the values of the signal number, siginfo_t, and signal context +arguments to determine the (logical) cause of the exception. Some +exceptions may be caused by factors that should generate lisp errors +or other serious conditions (stack overflow); if this is the case, +the kernel code may release the global exception lock and call out +to lisp code. (The lisp code in question may need to repeat some +of the exception decoding process; in particular, it needs to be +able to interpret register values in the signal context that it +receives as an argument.) + In some cases, the lisp kernel exception handler may not be +able to recover from the exception (this is currently true of +some types of memory-access fault and is also true of traps +or illegal instructions that occur during foreign code execution. +In such cases, the kernel exception handler reports the +exception as "unhandled", and the kernel debugger is invoked. + If the kernel exception handler identifies the exception' cause +as being a transient out-of-memory condition (indicating that +the current thread needs more memory to cons in), it tries to +make that memory available. In some cases, doing so involves +invoking the GC. + + + + Threads, exceptions, and the GC +OpenMCL's GC is not concurrent: when the GC is invoked in response to an +exception in a particular thread, all other lisp threads must stop until +the GC's work is done. The thread that triggered the GC iterates over +the global TCR list, sending each other thread a distinguished "suspend" +signal, then iterates over the list again, waiting for a per-thread +semaphore that indicates that the thread has received the "suspend" +signal and responded appropriatedly. Once all other threads have +acknowledged the request to suspend themselves, the GC thread can +run the GC proper (after doing any necessary .) Once +the GC's completed its work, the thread that invoked the GC iterates +over the global TCR list, raising a per-thread "resume" semaphore +for each other thread. + The signal handler for the asynchronous "suspend" signal is entered +with all asynchronous signals blocked. It saves its signal-context +argument in a TCR slot, raises the tcr's "suspend" semaphore, then +waits on the TCR's "resume" semaphore. + The GC thread has access to the signal contexts of all TCRs (including +its own) at the time when the thread received an exception or +acknowledged a request to suspend itself. This information (and +information about stack areas in the TCR itself) allows the GC to +identify the "stack locations and register contents" that are elements +of the GC's root set. + + + + PC-luseringIt's not quite accurate to say that OpenMCL's compiler and runtime +follow precise stack and register usage conventions at all times; there +are a few exceptions: + + On both PPC and x86-64 platforms, consing isn't fully atomic.It takes at least a few instructions to allocate an object in memory(and slap a header on it if necesssary); if a thread is interrupted inthe middle of that instruction sequence, the new object may or may nothave been created or fully initialized at the point in time that theinterrupt occurred. (There are actually a few different states ofpartial initialization) + On the PPC, the common act of building a lisp control stack frameinvolves allocating a four-word frame and storing three register valuesinto that frame. (The fourth word - the back pointer to the previousframe - is automatically set when the frame is allocated.) The previouscontents of those three words are unknown (there might have been aforeign stack frame at the same address a few instructions earlier),so interrupting a thread that's in the process of initializing aPPC control stack frame isn't GC-safe. + There are similar problems with the initialization of temp stackframes on the PPC. (Allocation and initialization doesn't happenatomically, and the newly allocated stack memory may have undefinedcontents.) + 's write barrier has to be implemented atomically (i.e.,both an intergenerational store and the update of a correspondingreference bit has to happen without interruption, or neither of theseevents can happen.) + There are a few more similar cases. + + + Fortunately, the number of these non-atomic instruction sequences is +small, and fortunately it's fairly easy for the interrupting thread +to recognize when the interrupted thread is in the middle of such +a sequence. When this is detected, the interrupting thread modfies +the state of the interrupted thread (modifying its PC and other +registers) so that it is no longer in the middle of such a sequenece +(it's either backed out of it or the remaining instructions are +emulated.) + This works because (a) many of the troublesome instruction sequences +are PPC-specific and it's relatively easy to partially disassemble the +instructions surrounding the interrupted thread's PC on the PPC and +(b) those instruction sequences are heavily stylized and intended to +be easily recognized. + + + + + Register usage and tagging + + + Register usage and tagging overview +Regardless of other details of its implementation, a garbage +collector's job is to partition the set of all heap-allocated +lisp objects (CONSes, STRINGs, INSTANCEs, etc.) into two subsets. +The first subset contains all objects that are transitively referenced +from a small set of "root" objects (the contents of the stacks +and registers of all active threads at the time the GC occurs +and the values of some global variables.) The second subset contains +everything else: those lisp objects that are not transitively +reachable from the roots are garbage, and the memory occupied +by garbage objects can be reclaimed (since the GC has just proven +that it's impossible to reference them.) + The set of live, reachable lisp objects basically form the nodes +of a (usually large) graph, with edges from each node A to any other +objects (nodes) that object A references. + Some nodes in this graph can never have outgoing edges: an array with a +specialized numeric or character type usually represents its elements in +some (possibly more compact) specialized way. Some nodes may refer +to lisp objects that are never allocated in memory (FIXNUMs, CHARACTERs, +SINGLE-FLOATs on 64-bit platforms ..) This latter class of objects +are sometimes called "immediates", but that's a little confusing +because the term "immediate" is sometimes used to refer to things +that can never be part of the big connectivity graph (e.g., the "raw" +bits that make up a floating-point value, foreign address, or +numeric value that needs to be used - at least fleetingly - in +compiled code.) + For the GC to be able to build the connectivity graph reliably, it's +necessary for it to be able to reliably tell (a) whether or not a +"potential root" - the contents of a machine register or stack location +- is in fact a node and (b) for any node, whether it may have components +that refer to other nodes. + There's no reliable way to answer the first question on stock hardware. +(If everything was a node, as might be the case on specially microcoded +"lisp machine" hardware, it wouldn't even need to be asked.) Since +there's no way to just look at a machine word (the contents of a machine +register or stack location) and tell whether or not it's a node or just +some random non-node value, we have to either adopt and enforce strict +conventions on register and stack usage or tolerate ambiguity. + "Tolerating ambiguity" is an approach taken by some ("conservative") GC +schemes; by contrast, OpenMCL's GC is "precise", which in this case +means that it believes that the contents of certain machine registers +and stack locations are always nodes and that other registers and stack +locations are never nodes and that these conventions are never violated +by the compiler or runtime system. The fact that threads are +preemptively scheduled means that a GC could occur (because of activity +in some other thread) on any instruction boundary, which in turn means +that the compiler and runtime system must follow precise at all times. + Once we've decided that a given machine word is a node, a describes how the node's value and type are encoded in that +machine word. + Most of this - so far - has discussed thigs from the GC's very low-level +perspective. From a much higher point of view, lisp functions accept +nodes as arguments, return nodes as values, and (usually) perform +some operations on those arguments in order to produce those results. +(In many cases, the operations in question involve raw non-node values.) +Higher-level parts of the lisp type system (functions like TYPE-OF +and CLASS-OF, etc.) depend on the . + + + + pc-locatives on the PPC +On the PPC, there's a third case (besides "node" and "immediate" values). +As discussed below, a node that denotes a memory-allocated lisp object +is a biased (tagged) pointer -to- that object; it's not generally possible +to point -into- some composite (multi-element) object (such a pointer +would not be a node, and the GC would have no way to update the pointer +if it were to move the underlying object.) + Such a pointer ("into" the interior of a heap-allocated object) is +often called a locative; the cases where locatives are allowed +in OpenMCL mostly involve the behavior of function call and return +instructions. (To be technicaly accurate, the other case also arises +on x86-64, but that case isn't as user-visible.) + On the PowerPC (both PPC32 and PPC64), all machine instructions are +32 bits wide and all instruction words are allocated on 32-bit boundaries. +In PPC OpenMCL, a CODE-VECTOR is a specialized type of vector-like object; +its elements are 32-bit PPC machine instructions. A CODE-VECTOR is +an attribute of FUNCTION object; a function call involves accessing the +function's code-vector and jumping to the address of its first instruction. + As each instruction in the code vector sequentially executes, the +hardware program counter (PC) register advances to the address of the +next instruction (a locative into the code vector); since PPC +instructions are always 32 bits wide and aligned on 32-bit boundaries, +the low two bits of the PC are always 0. If the function executes a +call (simple call instrucions have the mnemonic "bl" on the PPC, which +stands for "branch and link"), the address of the next instruction (also +a word-aligned locative into a code-vector) is copied into the special- +purpose PPC "link register" (lr); a function returns to its caller +via a "branch to link register" (blr) instruction. Some cases of +function call and return might also use the PPC's "count register" +(ctr), and if either the lr or ctr needs to be stored in memory it +needs to first be copied to a general-purpose registers. + OpenMCL's GC understands that certain registers contain these special +"pc-locatives" (locatives that point into CODE-VECTOR objects); it +contains specal support for finding the containing CODE-VECTOR object +and for adjusting all of these "pc-locatives" if the containing object +is moved in memory. The first part of that - finding the containing +object - is possible and practical on the PPC because of architectural +artifcacts (fixed-width instructions and arcana of instruction encoding.) +It's not possible on x86-64, but fortunately not necessary either (though +the second part - adjusting the PC/RIP when the containing object moves) +is both necessary and simple. + + + + Register and stack usage conventions + + + Stack conventions +On both PPC and X86 platforms, each lisp thread uses 3 stacks; the ways +in which these stacks are used differs between the PPC and X86. + Each thread has: + + A "control stack".On both platforms, this is "the stack" used by foreign code.On the PPC, it consists of a linked list of frameswhere the first word in each frame points to the first word in theprevious frame (and the outermost frame points to 0.) Some frameson a PPC control stack are lisp frames; lisp frames are always 4 wordsin size and contain (in addition to the back pointer to the previousframe) the calling function (a node), the return address (a "locative"into the calling function's code-vector), and the value to which thevalue-stack pointer (see below) should be restored on function exit.On the PPC, the GC has to look at control-stack frames, identifywhich of those frames are lisp frames, and treat the contents ofthe saved function slot as a node (and handle the return addresslocative specially.)On x86-64, the control stack is used for dynamic-extent allocationof immediate objects. Since the control stack never contains nodeson x86-64, the GC ignores it on that platform.Alignment of the control stack follows the ABI conventions of theplatform (at least at any point in time where foreign code could run.)On PPC, the r1 register always points to the top of the current thread'scontrol stack; on x86-64, the RSP register points to the top of thecurrent thread's control stack when the thread is running foreigncode and the address of the top of the control stack is kept in thethread's TCR see when not running foreigncode.The control stack "grows down." + A "value stack".On both platforms, all values on the value stack are nodes (including"tagged return addresses" on x86-64.) The value stack is always alignedto the native word size; objects are always pushed on the value stackusing atomic instructions ("stwu"/"stdu" on PPC, "push" on x86-64), sothe contents of the value stack between its bottom and top are alwaysunambiguously nodes; the compiler usually tries to pop or discardnodes from the value stack as soon as possible after their last use(as soon as they may have become garbage.)On x86-64, the RSP register addresses the top of the value stackwhen running lisp code; that address is saved in the TCR whenrunning foreign code.On the PPC, a dedicated regiter (VSP, currently r15) is used toaddress the top of the value stack when running lisp code, and theVSP value is saved in the TCR when running foreign code.The value stack grows down. + A "temp stack".The temp stack consists of a linked list of frames, each of which pointsto the previous temp stack frame. The number of native machine wordsin each temp stack frame is always even, so the temp stack is alignedon a two-word (64- or 128-bit) boundary.The temp stack is used for dynamic-extent objects on both platforms;on the PPC, it's used for essentially all such objects (regardlessof whether or not the objects contain nodes); on the x86-64, immediatedynamic-extent objects (strings, foreign pointers, etc.) are allocatedon the control stack and only node-containing dynamic-extent objectsare allocated on the temp stack.Data structures used to implement CATCH and UNWIND-PROTECT are stored onthe temp stack on both ppc and x86-64.Temp stack frames are always doublenode aligned and objects withina temp stack frame are aligned on doublenode boundaries. The firstword in each frame contains a back pointer to the previous frame; onthe PPC, the second word is used to indicate to the GC whethe theremaining objects are nodes (if the second word is 0) or immediate(otherwise.) On x86-64, where temp stack frames always contain nodes,the second word is always 0.The temp stack grows down. It usually takes several instuctions toallocate and safely initialize a temp stack frame that's intended tocontain nodes, and the GC has to recognize the case where a threadis in the process of allocating and initializing a temp stack frameand take care not to interpret any uninitialized words in the frameas nodes. See .The PPC keeps the current top of the temp stack in a dedicated register(TSP, currently r12) when running lisp code and saves this register'svalue in the TCR when running foreign code. The x86-64 keeps theaddress of the top of each thread's temp stack in the thread's TCR. + + + + + + Register conventions +If there are a "reasonable" (for some value of "reasonable") number or +general-purpose registers and the instruction set is "reasonably" +orthogonal (most instructions that operate on GPRs can operate on any +GPR), then it's possible to statically partition the GPRs into at least +two sets: "immediate registers" never contain nodes, and "node registers" +always contain nodes. (On the PPC, a few registers are members of a +third set of "PC locatives", and on both platforms some registers may +have dedicated roles as stack or heap pointers; the latter class is +treated as immediates by the GC proper but may be used to help determine +the bounds of stack and heap memory areas.) + The ultimate definition of register partitioning is hardwired +into the GC in functions like "mark_xp()" and "forward_xp()", which +process the values of some of the registers in an exception frame +as nodes and may give some sort of special treatment to other register +values they encounter there.) + On x86-64, the static register partitioning scheme involves: + + (only) two "immediate" registers.The RAX and RDX registers are used as the implicit operands andresults of some extended-precision multiply and divide instructionswhich generally involve non-node values; since their use in theseinstructions means that they can't be guaranteed to contain nodevalues at all times, it's natural to put these registers in the"immediate" set. RAX is generally given the symbolic name "imm0",and RDX is given the symbolic name "imm1"; you may see these namesin disassembled code, usually in operations involving type checking,array indexing, and foreign memory and function access. + (only) two "dedicated" registers.RSP and RBP have dedicated functionality dictated by the hardwareand calling conventions. (There are a few places where RBP istemporarily used as an extra immediate register.) + 12 "node" registers.All other registers (RBX, RCX, RSI, RDI, and R8-R15) are asserted tocontain node values at (almost) all times; legacy "string" operationsthat implicitly use RSI and/or RDI are not used. Shift and rotateintructions which shift/rotate by a variable number of bits arerequired by the architecture to use the low byte of RCX (the traditionalCL register) as the implicit shift count; when it's necessary to keepa non-node shift count in the low byte of RCX, the upper 7 bytes ofthe register are zeroed (so that misinterpetation of the immediatevalue in RCX as a node will not have negative GC affects. (The GCmight briefly treate it as a node, but since it's not pointing anywherenear the lisp heap it'll soon lose interest in it.)Legacy instructions that use RCX (or some portions of it) as a loopcounter can not be used (since such instructions might introducenon-node values into RCX.) + + + On the PPC, the static register partitioning scheme involves: + + 6 "immediate" registersRegisters r3-r8 are given the symbolic names imm0-imm5. As a RISCarchitecture with simpler addressing modes, the PPC probably usesimmediate registers a bit more often than the CISC x86-64 does, butthey're generally used for the same sort of things (type checking,array indexing, FFI, etc.) + 9 dedicated registers + + r0 (symbolic name rzero) always contains the value 0 when runninglisp code. Its value is sometimes read as 0 when it's used as thebase register in a memory address; keeping the value 0 there issometimes convenient and avoids asymmetry. + r1 (symbolic name sp) is the control stack pointer, by PPC convention. + r2 is used to hold the current thread's TCR on ppc64 systems; it'snot used on ppc32. + r9 and r10 (symbolic names allocptr and allocbase) are used to do per-thread memory allocation + r11 (symbolic name nargs) contains the number of function arguments on entry and the number of return values in multiple-value returning constructs. It's not used more generally as either a node or immediate register because of the way that certain trap instruction encodings are interpreted. + r12 (symbolic name tsp) holds the top of the current thread's temp stack. + r13 is used to hold the TCR on PPC32 sytems; it's not used on PPC64. + r14 (symbolic name loc-pc) is used to copy "pc-locative" values between main memory and special-purpose PPC registers (LR and CTR) used in function-call and return instructions. + r15 (symbolic name vsp) addresses the top of the current thread's value stack. + lr and ctr are PPC branch-unit registers used in function call and return instructions; they're always treated as "pc-locatives", which precludes the use of the ctr in some PPC looping constructs. + + + + 17 "node" registersr15-r31 are always treated as node registers + + + + + + + Tagging schemeOpenMCL always allocates lisp objects on double-node (64-bit for 32-bit +platforms, 128-bit for 64-bit platforms) boundaries; this mean that the +low 3 bits (32-bit lisp) or 4 bits (64-bit lisp) are always 0 and are +therefore redundant (we only really need to know the upper 29 or 60 bits +in order to identify the aligned object address.) The extra bits in a +lisp node can be used to encode at least some information about the +node's type, and the other 29/60 bits represent either an immediate +value or a doublenode-aligned memory address. The low 3 or 4 bits of a +node are called the node's "tag bits", and the conventions used to +encode type information in those tag bits are called a "tagging scheme." + It might be possible to use the same tagging scheme on all platforms (at +least on all platforms with the same word size and/or the same number of +available tag bits), but there are often some strong reasons for not +doing so. These arguments tend to be very machine-specific: sometimes, +there are fairly obvious machine-dependent tricks that can be exploited +to make common operations on some types of tagged objects faster; other +times, there are architectural restrictions that make it impractical to +use certain tags for certain types. (On PPC64, the "ld" (load +doubleword) and "std" (store doubleword) instructions - which load and +store a GPR operand at the effective address formed by adding the value +of another GPR operand and a 16-bit constant operand - require that the +low two bits of that constant operand be 0. Since such instructions +would typically be used to access the fields of things like CONS cells +and structures, it's desirable that that the tags chosen for CONS cells +and structures allow the use of these intructions as opposed to more +expensive alternatives.) + One architecture-dependent tagging trick that works well on all +architectures is to use a tag of 0 for FIXNUMs: a fixnum basically +encodes its value shifted left a few bits and keeps those low bits +clear. FIXNUM addition, subtraction, and binary logical operations can +operate directly on the node operands, addition and subtraction can +exploit hardware-based overflow detection, and (in the absence of +overflow) the hardware result of those operations is a node (fixnum). +Some other slightly-less-common operations may require a few extra +instructions, but arithmetic operations on FIXNUMs should be as +cheap as possible and using a tag of zero for FIXNUMs helps to +ensure that it will be. + If we have N available tag bits (N = 3 for 32-bit OpenMCL and N = 4 +for 64-bit OpenMCL), this way of representing fixnums with the low +M bits forced to 0 works as long as M <= N. The smaller we make +M, the larger the values of MOST-POSITIVE-FIXNUM and MOST-NEGATIVE +become; the larger we make N, the more distinct non-FIXNUM tags +become available. A reasonable compromise is to choose M = N-1; +this basically yields two distinct FIXNUM tags (one for even +fixnums, one for odd fixnums), gives 30-bit fixnums on 32-bit +platforms and 61-bit fixnums on 64-bit platforms, and leaves us +with 6 or 14 tags to encoded other types. + Once we get past the assignment of FIXNUM tags, things quickly devolve +into machine-dependencies. We can fairly easily see that we can't +directly all other primitive lisp object types with only 6 or 14 +available tag values; the details of how types are encoded vary between +the ppc32, ppc64, and x86-64 implementations, but there are some general +common principles: + + CONS cells always contain exactly 2 elements and are usually fairly common.It therefore makes sense to give CONS cells their own tag. Unlike thefixnum case - where a tag value of 0 had positive implications - theredoesn't seem to be any advantage to using any particular value. (A longtime ago - in the case of 68K MCL - the CONS tag and the order of CARand CDR in memory were chosen to allow smaller, cheaper addressing modesto be used to "cdr down a list." That's not a factor on ppc or x86-64,but all versions of OpenMCL still store the CDR of a CONS cell first inmemory. It doesn't matter, but doing it the way that the host systemdid made boostrapping to a new target system a little easier.) + Any way you look at it, NIL is a bit ... unusual.NIL is both a SYMBOL and a LIST (as well as being a canonical truthvalue and probably a few other things.) Its role as a LIST is probablymuch more important to most programs than its role as a SYMBOL is:LISTP has to be true of NIL and primitives like CAR and CDR do LISTPimplicitly when safe and want that operation to be fast.There are several possible approaches to this; OpenMCL uses two of them.On PPC32 and X86-64, NIL is basically a weird CONS cell that straddlestwo doublenodes; the tag of NIL is unique and congruent modulo 4 (modulo8 on 64-bit) with the tag used for CONS cells. LISTP is thereforetrue of any node whose low 2 (or 3) bits contain the appropriate tagvalue (it's not otherwise necessary to special-case NIL.) SYMBOLaccessors (SYMBOL-NAME, SYMBOL-VALUE, SYMBOL-PLIST ..) -do- haveto special-case NIL (and access the components of an internal proxysymbol.)On PPC64 (where architectural restrictions dictate the set of tagsthat can be used to access fixed components of an object), thatapproach wasn't practical. NIL is just a distinguished SYMBOL,and it just happens to be the case that its pname slot and valueslots are at the same offsets from a tagged pointer as a CONS cellsCDR and CAR would be. NIL's pname is set to NIL (SYMBOL-NAMEchecks for this and returns the string "NIL"), and LISTP (andtherefore safe CAR and CDR) have to check for (OR NULL CONSP).At least in the case of CAR and CDR, the fact that the PPC hasmultiple condition-code fields keeps that extra test from beingprohibitively expensive. + Some objects are immediate.(but not FIXNUMs).This is true of CHARACTERs and, on 64-bit platforms, SINGLE-FLOATs.It's also true of some nodes used in the runtime system (specialvalues used to indicate unbound variables and slots, for instance.)On 64-bit platforms, SINGLE-FLOATs have their own unique tag (makingthem a little easier to recognize; on all platforms, CHARACTERs sharea tag with other immediate objects (unbound markers) but are easyto recognize (by looking at several of their low bits.) The GCtreats any node with an immediate tag (and any node with a fixnumtag) as a leaf. + There are some advantages to treating everything else - memory-allocated objects that aren't CONS cells - uniformly.There are some disadvantages to that uniform treatment as well, and thetreatment of "memory-allocated non-CONS objects" isn't entirely uniformaccross all OpenMCL implementations. Let's first pretend that thetreatment is uniform, then discuss the ways in which it isn't.The "uniform approach" is to treat all memory-allocated non-CONS objectsas if they were vectors; this use of the term is a little looser thanwhat's implied by the CL VECTOR type. OpenMCL actually uses the term"uvector" to mean "a memory-allocated lisp object other than a CONS cell,whose first word is a header which describes the object's type andthe number of elements that it contains." In this view, a SYMBOL isa UVECTOR, as is a STRING, a STANDARD-INSTANCE, a CL array or vector,a FUNCTION, and even a DOUBLE-FLOAT.In the PPC implementations (where things are a little more ... uniform),a single tag value is used to denote any uvector; in order to determinesomething more specific about the type of the object in question, it'snecessary to fetch the low byte of the header word from memory. On thex86-64 platform, certain types of uvectors - SYMBOLs and FUNCTIONs -are given their own unique tags. The good news about the x86-64 approachis that SYMBOLs and FUNCTIONs can be recognized without referencingmemory; the slightly bad news is that primitive operations that workon UVECTOR-tagged objects - like the function CCL:UVREF - don't workon SYMBOLs or FUNCTIONs on x86-64 (but -do- work on those types of objectsin the PPC ports.)The header word which precedes a UVECTOR's data in memory contains 8bits of type information in the low byte and either 24 or 56 bits of"element-count" information in the rest of the word. (This is wherethe sometimes-limiting value of 2^24 for ARRAY-TOTAL-SIZE-LIMIT onPPC32 platforms comes from.) The low byte of the header - sometimescalled the uvector's subtag - is itself tagged (which means thatthe header is tagged.) The (3 or 4) tag bits in the subtag are usedto determine whether the uvector's elements are nodes or immediates.(A UVECTOR whose elements are nodes is called a GVECTOR; a UVECTORwhose elements are immediates is called an IVECTOR. This terminologycame from Spice Lisp, which was a predecessor of CMUCL.)Even though a uvector header is tagged, a header is not a node. There'sno (supported) way to get your hands on one in lisp and doing so couldbe dangerous. (If the value of a header wound up in a lisp noderegister and that register wound up getting pushed on a thread's valuestack, the GC might misinterpret that situation to mean that therewas a stack-allocated UVECTOR on the value stack.) + + + + + + + Heap Allocation +When the OpenMCL kernel first starts up, a large contiguous chunk of the +process's address space is mapped as "anonymous, no access" memory. ("Large" +means different things in different contexts; on LinuxPPC32, it means "about +1 gigabyte", on DarwinPPC32, it means "about 2 gigabytes", and on current +64-bit platforms it ranges from 128 to 512 gigabytes, depending on OS. These +values are both defaults and upper limits; the –heap-reserve argument can +be used to try to reserve less than the default.) + Reserving address space that can't (yet) be read or written to doesn't +cost much; in particular, it doesn't require that correspinding swap +space or physical memory be available. Marking the address range as being +"mapped" helps to ensure that other things (random calls to malloc(), +dynamically loaded shared libraries) won't be allocated in this region +that lisp has reserved for its own heap growth. + A small portion (around 1/32 on 32-bit platforms and 1/64 on 64-bit +platforms) of that large chunk of address space is reserved for GC +data structures. Memory pages reserved for these data structures are +mapped read-write as pages made writable in the main portion of the +heap. + The initial heap image is mapped into this reserved address space and +an additional (LISP-HEAP-GC-THRESHOLD) bytes are mapped read-write. +GC data structures grow to match the amount of GC-able memory in the +initial image + the gc threshold, and control is transferred to lisp +code. Inevitably, that code spoils everything and starts consing; +there are basically three layers of memory allocation that can go on. + + + Per-thread object allocationEach lisp thread has a private "reserved memory segment"; when a thread +starts up, its reserved memory segment is empty. PPC ports maintain the +highest unallocated addres and he lowest allocated address in the +current segment in registers when running lisp code; on x86-664, these +values are maintained in the current threads's TCR. (An "empty" heap +segment is one whose high pointer and low pointer are equal.) When +a thread is not in the midde of allocating something, the low 3 or +4 bits of the high and low pointers are clear (the pointers are +doublenode-aligned.) + A thread tries to allocate an object whose physical size in bytes is X +and whose tag is Y by: + + decrementing the "high" pointer by (- X Y) + trapping if the high pointer is less than the low pointer + using the (tagged) high pointer to initialize the object, if necessary + clearing the low bits of the high pointer + + On PPC32, where the size of a CONS cell is 8 bytes and the tag of a CONS +cell is 1, machine code which sets the arg_z register to the result of +doing (CONS arg_y arg_z) looks like: + + (SUBI ALLOCPTR ALLOCPTR 7) ; decrement the high pointer by (- 8 1) + (TWLLT ALLOCPTR ALLOCBASE) ; trap if the high pointer is below the base + (STW ARG_Z -1 ALLOCPTR) ; set the CDR of the tagged high pointer + (STW ARG_Y 3 ALLOCPTR) ; set the CAR + (MR ARG_Z ALLOCPTR) ; arg_z is the new CONS cell + (RLWINM ALLOCPTR ALLOCPTR 0 0 28) ; clear tag bits + + On x86-64, the idea's similar but the implementation is different. The +high and low pointers to the current thread's reserved segment are kept +in the TCR, which is addressed by the gs segment register. An x86-64 +CONS cell is 16 bytes wide and has a tag of 3; we canonically use the +temp0 register to initialize the object + + (subq ($ 13) ( (% gs) 216)) ; decrement allocptr + (movq ( (% gs) 216) (% temp0)); load allocptr into temp0 + (cmpq ( (% gs) 224) (% temp0)) ; compare to allocabase + (jg L1) ; skip trap + (uuo-alloc) ; uh, don't skip trap +L1 + (andb ($ 240) ( (% gs) 216)) ; untag allocptr in the tcr + (movq (% arg_y) ( 5 (% temp0))) ; set the car + (movq (% arg_z) ( -3 (% temp0))); set the cdr + (movq (% temp0) (% arg_z)) ; return the cons + + If we don't take the trap (if allocating 8-16 bytes doesn't exhaust +the thread's reserved memory segment), that's a fairly short and +simple instruction sequence. If we do take the trap, we'll have to +do some additional work in order to get a new segment for the current +thread. + + + + Allocation of reserved heap segmentsAfter lisp image is first mapped into memory - and after each full GC - the +lisp kernel ensures that (LISP-HEAP-GC-TRESHOLD) additional bytes beyond +the current end of the heap are mapped read-write. + If a thread traps while trying to allocate memory, the thread goes +through the usual exception-handling protocol (to ensure that any oher +thread that GCs "sees" the state of the trapping thread and to serialize +exception handling.) When the exception handler runs, it determines +the nature and size of the failed allocation and tries to complete the +allocation on the thread's behalf (and leave it with a reasonably large +thread-specific memory segment so that the next small allocation is +unlikely to trap. + Depending on the size of the requested segment allocation, the number of +segment allocations that have occurred since the last GC, and the EGC +and GC thresholds, the segment allocation trap handler may invoke a full +or ephemeral GC before returning a new segment. It's worth noting that +the [E]GC is triggered based on the number of and size of these segments +that've been allocated since the last GC; it doesn't have much to do +with how "full" each of those per-thread segments are. It's possible +for a large number of threads to do fairly incidental memory allocation +and trigger the GC as a result; avoiding this involves tuning the +per-thread allocation quantum and the GC/EGC thresholds appropriately. + + + + Heap growthAll OSes on which OpenMCL currently runs use an "overcommit" memory +allocation strategy by default (though some of them provide ways of +overriding that default.) What this means in general is that the OS +doesn't necessarily ensure that backing store is available when asked +to map pages as read-write; it'll often return a success indicator from +the mapping attempt (mapping the pages as "zero-fill, copy-on-write"), +and only try to allocate the backing store (swap space and/or physical +memory) when non-zero contents are written to the pages. + It -sounds- like it'd be better to have the mmap() call fail +immediately, but it's actually a complicated issue. (It's possible that +other applications will stop using some backing store before lisp code +actually touches the pages that need it, for instance.) It's also +not guaranteed that lisp code would be able to "cleanly" signal an +out-of-memory condition if lisp is ... out of memory + I don't know that I've ever seen an abrupt out-of-memory failure that +wasn't preceeded by several minutes of excessive paging activity. The +most expedient course in cases like this is to either (a) use less memory +or (b) get more memory; it's generally hard to use memory that you don't +have. + + + + + GC details +The GC uses a Mark/Compact algorithm; its execution time is +essentially a factor of the amount of live data in the heap. (The somewhat +better-known Mark/Sweep algorithms don't compact the live data but instead +traverse the garbage to rebuild free-lists; their execution time is +therefore a factor of the total heap size.) + As mentioned in , two auxiliary data structures +(proportional to the size of the lisp heap) are maintained. These are + + the markbits bitvector, which contains a bit for everydoublenode in the dynamic heap (plus a few extra words for alignmentand so that sub-bitvectors can start on word boundaries.) + the relocation table, which contains a native word for every 32 or 64doublenodes in the dynamic heap, plus an extra word used to keep trackof the end of the heap. + + The total GC space overhead is therefore on the order of 3% (2/64 or +1/32). + The general algorithm proceeds as follows: + + + Mark phase +Each doublenode in the dynamic heap has a corresponding bit in the +markbits vector. (For any doublenode in the heap, the index of its mark +bit is determined by subtracing the address of the start of the heap +from the address of the object and dividing the result by 8 or 16.) The GC +knows the markbit index of the free pointer, so determining that the +markbit index of a doubleword address is between the start of the heap +and the free pointer can be done with a single unsigned +comparison. + The markbits of all doublenodes in the dynamic heap are zeroed +before the mark phase begins. An object is marked +if the markbits of all of its constituent doublewords are set and +unmarked otherwise; setting an object's markbits involves setting the +corrsponding markbits of all constituent doublenodes in the +object. + The mark phase traverses each root. If the tag of the value of +the root indicates that it's a non-immediate node whose address lies +in the lisp heap, then: + + If the object is already marked, do nothing. + Set the object's markbit(s). + If the object is an ivector, do nothing further. + If the object is a cons cell, recursively mark its car andcdr. + Otherwise, the object is a gvector. Recursively mark itselements. + + Marking an object thus involves ensuring that its mark bits are set and +then recursively marking any pointers contained within the object if the +object was originally unmarked. If this recursive step was implemented +in the obvious manner, marking an object would take stack space +proportional to the length of the pointer chain from some root to that +object. Rather than storing that pointer chain implicitly on the stack +(in a series of recursive calls to the mark subroutine), the OpenMCL +marker uses mixture of recursion and a technique called link +inversion to store the pointer chain in the objects themselves. +(Recursion tends to be simpler and faster; if a recursive step notes +that stack space is becoming limited, the link-inversion technique is +used.) + Certain types of objects are treated a little specially: + + To support a feature called GCTWA + +I believe that theacronym comes from MACLISP, where it stood for "Garbage Collection ofTruly Worthless Atoms". + +, the vector which contains the internalsymbols of the current package is marked on entry to the mark phasebut the symbols themselves are not marked at this time. Near the endof the mark phase, symbols referenced from this vector which are nototherwise marked are marked if and only if they're somehowdistinguishable from newly created symbols (by virtue of their havingfunction bindings, value bindings, plists, or other attributes.) + Pools have their first element set to NIL before any otherelements are marked. + All hash tables have certain fields (used to cache previousresults) invalidated. + Weak Hash Tables and other weak objects are put on a linkedlist as they're encountered; their contents are only retained ifthere are other (non-weak) references to them. + + At the end of the mark phase, the markbits of all objects which +are transitively reachable from the roots are set and all other markbits +are clear. + + + + Relocation phase +The forwarding address of a doublenode in the +dynamic heap is ( - (size_of_doublenode * )) or alternately ( + (size_of_doublenode * )). Rather than count the number of preceding markbits each time, +the relocation table is used to precompute an approximation of the +forwarding addresses for all doublewords. Given this approximate address +and a pointer into the markbits vector, it's relatively easy to compute +the exact forwarding address. + The relocation table contains the forwarding addresses of each +pagelet, where a pagelet is 256 bytes (or 32 +doublenodes). The forwarding address of the first pagelet is the base of +the heap. The forwarding address of the second pagelet is the sum of the +forwarding address of the first and 8 bytes for each mark bit set in the +first 32-bit word in the markbits table. The last entry in the +relocation table contains the forwarding address that the freepointer +would have, e.g., the new value of the freepointer after +compaction. + In many programs, old objects rarely become garbage and new +objects often do. When building the relocation table, the relocation +phase notes the address of the first unmarked object in the dynamic +heap. Only the area of the heap between the first unmarked object and +the freepointer needs to be compacted; only pointers to this area will +need to be forwarded (the forwarding address of all other pointers to +the dynamic heap is the address of that pointer.) Often, the first +unmarked object is much nearer the free pointer than it is to the base +of the heap. + + + + Forwarding phase +The forwarding phase traverses all roots and the +"old" part of the dynamic heap (the part between the base +of the heap and the first unmarked object.) All references to objects +whose address is between the first unmarked object and the free pointer +are updated to point to the address the object will have after +compaction by using the relocation table and the markbits vector and +interpolating. + The relocation table entry for the pagelet nearest the object is +found. If the pagelet's address is less than the object's address, the +number of set markbits that precede the object on the pagelet is used to +determine the object's address; otherwise, the number of set markbits +the follow the object on the pagelet is used. + Since forwarding views the heap as a set of doublewords, locatives +are (mostly) treated like any other pointers. (The basic difference is +that locatives may appear to be tagged as fixnums, in which case they're +treated as word-aligned pointers into the object.) + If the forward phase changes the address of any hash table key in +a hash table that hashes by address (e.g., an EQ hash table), it sets a +bit in the hash table's header. The hash table code will rehash the hash +table's contents if it tries to do a lookup on a key in such a +table. + Profiling reveals that about half of the total time spent in the +GC is spent in the subroutine which determines a pointer's forwarding +address. Exploiting GCC-specific idioms, hand-coding the routine, and +inlining calls to it could all be expected to improve GC +performance. + + + + Compact phase +The compact phase compacts the area between the first unmarked +object and the freepointer so that it contains only marked objects. +While doing so, it forwards any pointers it finds in the objects it +copies. + When the compact phase is finished, so is the GC (more or less): +the free pointer and some other data structures are updated and control +returns to the exception handler that invoked the GC. If sufficient +memory has been freed to satisfy any allocation request that may have +triggered the GC, the exception handler returns; otherwise, a +"seriously low on memory" condition is signalled, possibly +after releasing a small emergency pool of memory. + + + + + The ephemeral GC +In the OpenMCL memory management scheme, the relative age of two +objects in the dynamic heap can be determined by their addresses: if +addresses X and Y are both addresses in the dynamic heap, X is younger +than Y (X was created more recently than Y) if it is nearer to the free +pointer (and farther from the base of the heap) than Y. + Ephemeral (or generational) garbage collectors attempt to exploit +the following assumptions: + + most newly created objects become garbage soon after they'recreated. + most objects that have already survived several GCs are unlikelyto ever become garbage. + old objects can only point to newer objects as the result of adestructive modification (e.g., via SETF.) + + + By concentrating its efforts on (frequently and quickly) reclaiming +newly created garbage, an ephemeral collector hopes to postpone the more +costly full GC as long as possible. It's important to note that most +programs create some long-lived garbage, so an EGC can't typically +eliminate the need for full GC. + An EGC views each object in the heap as belonging to exactly one +generation; generations are sets of objects that are +related to each other by age: some generation is the youngest, some the +oldest, and there's an age relationship between any intervening +generations. Objects are typically assigned to the youngest generation +when first allocated; any object that has survived some number of GCs in +its current generation is promoted (or tenured) into +an older generation. + When a generation is GCed, the roots consist of the stacks, +registers, and global variables as always and also of any pointers to +objects in that generation from other generations. To avoid the need to +scan those (often large) other generations looking for such +intergenerational references, the runtime system must note all such +intergenerational references at the point where they're created (via +Setf).This is sometimes called "The Write Barrier": all +assignments which might result in intergenerational references must be +noted, as if the other generations were write-protected. The set of pointers that may contain intergenerational +references is sometimes called the remembered +set. + In OpenMCL's EGC, the heap is organized exactly the same as otherwise; +"generations" are merely structures which contain pointers to +regions of the heap (which is already ordered by age.) When a generation +needs to be GCed, any younger generation is incorporated into it; all +objects which survive a GC of a given generation are promoted into the +next older generation. The only intergenerational references that can +exist are therefore those where an old object is modified to contain a +pointer to a new object. + The EGC uses exactly the same code as the full GC. When a given GC +is "ephemeral", + + the "base of the heap" used to determine anobject's markbit address is the base of the generation beingcollected; + the markbits vector is actually a pointer into the middle of theglobal markbits table; preceding entries in this table are used tonote doubleword addresses in older generations that (may) containintergenerational references; + some steps (notably GCTWA and the handling of weak objects) arenot performed; + the intergenerational references table is used to findadditional roots for the mark and forward phases. If a bit is set inthe intergenerational references table, that means that thecorresponding doubleword (in some "old" generation, insome "earlier" part of the heap) may have had a pointerto an object in a younger generation stored into it. + + + The intergenerational references table is maintained indirectly: +whenever a setf operation that may introduce an intergenerational +reference occurs, a pointer to the doubleword being stored into is pushed +onto the memo buffer, which is a stack whos top is +addressed by the memo register. Whenever the memo buffer +overflowsA guard page at the end of the memo buffer simplifies overflow +detection. when the EGC is active, the handler scans the buffer and +sets bits in the intergenerational references table for each doubleword +address it finds in the buffer that belongs to some generation other than +the youngest; the same scan is performed on entry to any ephemeral GC. +After (possibly) performing this scan, the handler resets the memo +register to point to the bottom of the memo stack; this means that when +the EGC is inactive, the memo buffer is constantly being filled and +emptied for no apparent reason. + With one exception (the implicit setfs that occur on entry to and +exit from the binding of a special variable), all setfs that might +introduce an intergenerational reference must be memoized.Note that the implicit setfs that occur when initializing an +object - as in the case of a call to cons or vector - can't introduce +intergenerational references, since the newly created object is always +younger than the objects used to initialize it. It's always safe to push any cons cell or gvector locative +onto the memo stack; it's never safe to push anything else. + Typically, the intergenerational references bitvector is sparse: a +relatively small number of old locations are stored into, although some of +them may have been stored into many times. The routine that scans the +memoization buffer does a lot of work and usually does it fairly often; it +uses a simple, brute-force method but might run faster if it was smarter +about recognizing addresses that it'd already seen. + When the EGC mark and forward phases scan the intergenerational +reference bits, they can clear any bits that denote doublewords that +definitely do not contain intergenerational references. + + + + Fasl files +The information in this section was current in November 2004. +Saving and loading of Fasl files is implemented in +xdump/faslenv.lisp, level-0/nfasload.lisp, and lib/nfcomp.lisp. +The information here is only an overview, which might help when +reading the source. + The OpenMCL Fasl format is forked from the old MCL Fasl format; there +are a few differences, but they are minor. The name "nfasload" +comes from the fact that this is the so-called "new" Fasl system, +which was true in 1986 or so. The format has held up well, although +it would certainly need extensions to deal with 64-bit data, and +some other modernization might be possible. + A Fasl file begins with a "file header", which contains version +information and a count of the following "blocks". There's typically +only one "block" per Fasl file. The blocks are part of a mechanism +for +combining multiple logical files into a single physical file, in order +to simplify the distribution of precompiled programs. (Nobody seems +to be doing anything interesting with this feature, at the moment, +probably because it isn't documented.) + Each block begins with a header for itself, which just describes the +size of the data that follows. + The data in each block is treated as a simple stream of bytes, which +define a bytecode program. The actual bytecodes, "fasl operators", +are defined in xdump/faslenv.lisp. The descriptions in the source +file are terse, but, according to Gary, "probably accurate". + Some of the operators are used to create a per-block "object table", +which +is a vector used to keep track of previously-loaded objects and +simplify references to them. When the table is created, an index +associated with it is set to zero; this is analogous to an array +fill-pointer, and allows the table to be treated like a stack. + The low seven bits of each bytecode are used to specify the +fasl operator; +currently, about fifty operators are defined. The high byte, when +set, indicates that the result of the operation should be pushed +onto the object table. + Most bytecodes are followed by operands; the operand data is +byte-aligned. +How many operands there are, and their type, depend on the bytecode. +Operands can be indices into the object table, immediate values, or +some combination of these. + An exception is the bytecode #xFF, which has the symbolic name +ccl::$faslend; it is used to mark the end of the block. + + + + Heap images +Needs complete rewrite. + + + + Operating system dependencies + + + + The Objective-C Bridge +Unlike the rest of this chapter, this section was written in late +2004. + + + How OpenMCL Recognizes Objective-C Objects +In most cases, pointers to instances of Objective-C classes are +recognized as such; the recognition is (and probably always will be) +slightly heuristic. Basically, any pointer that passes basic sanity +checks and whose first word is a pointer to a known ObjC class is +considered to be an instance of that class; the Objective-C runtime +system would reach the same conclusion. + It's certainly possible that a random pointer to an arbitrary +memory address could look enough like an ObjC instance to fool the lisp +runtime system, and it's possible that pointers could have their +contents change so that something that had either been a true ObjC +instance (or had looked a lot like one) is changed (possibly by virtue +of having been deallocated.) + In the first case, we can improve the heuristics substantially: we +can make stronger assertions that a particular pointer is really "of +type :ID" when it's a parameter to a function declared to take +such a pointer as an argument or a similarly declared function result; we +can be more confident of something we obtained via SLOT-VALUE of a slot +defined to be of type :ID than if we just dug a pointer out of memory +somewhere. + The second case is a little more subtle: ObjC memory management is +based on a reference-counting scheme, and it's possible for an +object to ... cease to be an object while lisp is still referencing it. +If we don't want to deal with this possibility (and we don't), +we'll basically have to ensure that the object is not deallocated +while lisp is still thinking of it as a first-class object. There's +some support for this in the case of objects created with MAKE-INSTANCE, +but we may need to give similar treatment to foreign objects that are +introduced to the lisp runtime in other ways (as function arguments, +return values, SLOT-VALUE results, etc. as well as those instances +that're created under lisp control.) + This doesn't all work yet (in fact, not much of it works yet); +in practice, this has not yet been as much of a problem as anticipated, +but that may be because existing Cocoa code deals primarily with +relatively long-lived objects such as windows, views, menus, etc. + + + + Recommended Reading + The Apple Objective-C Runtime Reference + This describes the internal data structures and programminginterface of Objective C as it is implemented on OS X. + + + + + + + + Modifying OpenMCL + + + Contributing Code Back to the OpenMCL Project +This section is a placeholder, added as of August 2004. The +full text is being written, and will be added as soon as it is +available. + + + + Using OpenMCL in "development" and in "user" mode +As it's distributed, OpenMCL starts up with *PACKAGE* set to the CL-USER +package and with most predefined functions and methods protected against +accidental redefinition. The package setting is of course a requirement +of ANSI CL, while the protection protection is intended to catch certain +types of programming errors (accidentally redefining a CL or CCL function) +before those errors have a chance to do much damage. + These settings may make using OpenMCL to develop OpenMCL a bit more +awkward, since much of that process assumes that the CCL package is +current (and a primary purpose of that process is to redefine some +"predefined, builtin functions".) The standard, "routine" +ways of building OpenMCL from sources (see ) +- COMPILE-CCL, XCOMPILE-CCL, and XLOAD-LEVEL-0 - bind *PACKAGE* to the +"CCL" package and enable the redefinition of predefined functions; +the symbols COMPILE-CCL, XCOMPILE-CCL, and XLOAD-LEVEL-0 are additionally +now exported from the "CCL" package. + Some other (more ad-hoc) ways of doing development on OpenMCL - +compiling and/or loading individual files, incrementally redefining +individual functions - may be awkward unless one reverts to the mode of +operation which was traditionally offered in OpenMCL. (Some OpenMCL source +files - especially those that comprise the bootstrapping image sources and +the first few files in the "cold load" sequence - are compiled and +loaded in the "CCL" package but don't contain (IN-PACKAGE +"CCL") forms, since IN-PACKAGE doesn't work until later in the +cold load sequence.) + The somewhat bizarre behavior of both SET-USER-ENVIRONMENT and +SET-DEVELOPMENT-ENVIRONMENT with respect to the special variables they +affect is intended to allow those constructs to take effect when the +read-eval-print loop next returns to a top-level '? ' prompt; +the constructs can meaningfully be used inside LOAD, for instance +(recall that LOAD binds *PACKAGE*), though using both constructs within +the same LOAD call would likely be pretty confusing. + "user" and "development" are otherwise very +generic terms; here they're intended to enforce the distinction +between "using" OpenMCL and "developing" it. + The initial environment from which OpenMCL images are saved is one +where (SET-USER-ENVIRONMENT T) has just been called; in previous +versions, it was effectively as if (SET-DEVELOPMENT-ENVIRONMENT T) had +just been called. + Hopefully, most users of OpenMCL can safely ignore these issues +most of the time. Note that doing (SET-USER-ENVIRONMENT T) after loading +one's own code (or 3rd-party code) into OpenMCL would protect that +code (as well as OpenMCL's) from accidental redefinition; that may +be useful in some cases. + + + + Debugging facilities in the lisp kernel +In a perfect world, something like this couldn't happen: + +Welcome to OpenMCL Version x.y! +? (defun foo (x) + (declare (cons x)) + (cdr x)) +FOO + +? (foo -1) ;Oops. Too late ... +Unhandled exception 11 at 0x300e90c8, context->regs at #x7ffff6b8 +Continue/Debugger/eXit ? + + As you may have noticed, it's not a perfect world; it's rare +that the cause (attempting to reference the CDR of -1, and therefore +accessing unmapped memory near location 0) of this effect (an +"Unhandled exception ..." message) is so obvious. + The addresses printed in the message above aren't very useful +unless you're debugging the kernel with GDB (and they're often +very useful if you are.) + Aside from causing an exception that the lisp kernel doesn't +know how to handle, one can also enter the kernel debugger (more) +deliberately: + +? (defun classify (n) + (cond ((> n 0) "Greater") + ((< n 0) "Less") + (t + ;;; Sheesh ! What else could it be ? + (ccl::bug "I give up. How could this happen ?")))) +CLASSIFY + +? (classify 0) +Bug in OpenMCL system code: +I give up. How could this happen ? +? for help +[12345] OpenMCL kernel debugger: + + CCL::BUG isn't quite the right tool for this example (a call to +BREAK or PRINT might do a better job of clearing up the mystery), but +it's sometimes helpful when those other tools can't be used. +The +lisp error system notices, for instance, if attempts to signal errors +themselves cause errors to be signaled; this sort of thing can happen if +CLOS or the I/O system are broken or missing. After some small number of +recursive errors, the error system gives up and calls CCL::BUG. + If one enters a '?' at the kernel debugger prompt, one will see output +like: + (S) Find and describe symbol matching specified name +(B) Show backtrace +(X) Exit from this debugger, asserting that any exception was handled +(K) Kill OpenMCL process +(?) Show this help + + CCL::BUG just does an FF-CALL into the lisp kernel. If the kernel debugger +was invoked because of an unhandled exception (such as an illegal memory +reference) the OS kernel saves the machine state +("context") in a data structure for us, and in that case some additional +options can be used to display the contents of the registers at the point +of the exception. Another function - CCL::DBG - causes a special exception +to be generated and enters the lisp kernel debugger with a non-null +"context": + ? (defun classify2 (n) + (cond ((> n 0) "Greater") + ((< n 0) "Less") + (t (dbg n)))) +CLASSIFY2 + +? (classify2 0) +Lisp Breakpoint + While executing: # +? for help +[12345] OpenMCL kernel debugger: ? +(G) Set specified GPR to new value +(A) Advance the program counter by one instruction (use with caution!) +(D) Describe the current exception in greater detail +(R) Show raw GPR/SPR register values +(L) Show Lisp values of tagged registers +(F) Show FPU registers +(S) Find and describe symbol matching specified name +(B) Show backtrace +(X) Exit from this debugger, asserting that any exception was handled +(P) Propagate the exception to another handler (debugger or OS) +(K) Kill OpenMCL process +(?) Show this help + + CCL::DBG takes an argument, whose value is copied into the register +that OpenMCL uses to return a function's primary value (arg_z, which +is r23 on the PowerPC). If we were to choose the (L) option at this point, +we'd see a dislay like: + rnil = 0x01836015 +nargs = 0 +r16 (fn) = # +r23 (arg_z) = 0 +r22 (arg_y) = 0 +r21 (arg_x) = 0 +r20 (temp0) = #<26-element vector subtag = 2F @#x303793ee> +r19 (temp1/next_method_context) = 6393788 +r18 (temp2/nfn) = # +r17 (temp3/fname) = CLASSIFY2 +r31 (save0) = 0 +r30 (save1) = *TERMINAL-IO* +r29 (save2) = 0 +r28 (save3) = (# #) +r27 (save4) = () +r26 (save5) = () +r25 (save6) = () +r24 (save7) = () + + From this we can conclude that the problematic argument to CLASSIFY2 +was 0 (see r23/arg_z), and that I need to work on a better example. + The R option shows the values of the ALU (and PPC branch unit) +registers in hex; the F option shows the values of the FPU registers. + The (B) option shows a raw stack backtrace; it'll try to +identify foreign functions as well as lisp functions. (Foreign function +names are guesses based on the nearest preceding exported symbol.) + If you ever unexpectedly find yourself in the "lisp kernel +debugger", the output of the (L) and (B) options are often the most +helpful things to include in a bug report. + + + + Using AltiVec in OpenMCL LAP functions + + + Overview +It's now possible to use AltiVec instructions in PPC LAP +(assembler) functions. + The lisp kernel detects the presence or absence of AltiVec and +preserves AltiVec state on lisp thread switch and in response to +exceptions, but the implementation doesn't otherwise use vector +operations. + This document doesn't document PPC LAP programming in general. +Ideally, there would be some document that did. + This document does explain AltiVec register-usage conventions in +OpenMCL and explains the use of some lap macros that help to enforce those +conventions. + All of the global symbols described below are exported from the CCL +package. Note that lap macro names, ppc instruction names, and (in most +cases) register names are treated as strings, so this only applies to +functions and global variable names. + Much of the OpenMCL support for AltiVec LAP programming is based on +work contributed to MCL by Shannon Spires. + + + + Register usage conventions +OpenMCL LAP functions that use AltiVec instructions must +interoperate with each other and with C functions; that suggests that they +follow C AltiVec register usage conventions. (vr0-vr1 scratch, vr2-vr13 +parameters/return value, vr14-vr19 temporaries, vr20-vr31 callee-save +non-volatile registers.) + The EABI (Embedded Application Binary Interface) used in LinuxPPC +doesn't ascribe particular significance to the vrsave special-purpose +register; on other platforms (notably MacOS), it's used as a bitmap +which indicates to system-level code which vector registers contain +meaningful values. + The WITH-ALTIVEC-REGISTERS lapmacro generates code which which +saves, updates, and restores VRSAVE on platforms where this is required +(as indicated by the value of the special variable which controls this) +and ignores VRSAVE on platforms that don't require it to be +maintained. + On all PPC platforms, it's necessary to save any non-volatile +vector registers (vr20 .. vr31) before assigning to them and to restore +such registers before returning to the caller. + On platforms that require that VRSAVE be maintained, it's not +necessary to mention the "use" of vector registers that're +used as incoming parameters. It's not incorrect to mention their use +in a WITH-ALTIVEC-REGISTERS form, but it may be unneccessary in many +interesting cases. One can likewise assume that the caller of any function +that returns a vector value (in vr2 has already set the apropriate bit in +VRSAVE to indicate that this register is live. One could therefore write a +leaf function that added the bytes in vr3 and vr2 and returned the result +in vr2 as: + +(defppclapfunction vaddubs ((y vr3) (z vr2)) + (vaddubs z y z) + (blr)) + + When vector registers that aren't incoming parameters are used +in a LAP function, WITH-ALTIVEC-REGISTERS takes care of maintaining VRSAVE +and of saving/restoring any non-volatile vector registers: + +(defppclapfunction load-array ((n arg_z)) + (check-nargs 1) + (with-altivec-registers (vr1 vr2 vr3 vr27) ; Clobbers imm0 + (li imm0 arch::misc-data-offset) + (lvx vr1 arg_z imm0) ; load MSQ + (lvsl vr27 arg_z imm0) ; set the permute vector + (addi imm0 imm0 16) ; address of LSQ + (lvx vr2 arg_z imm0) ; load LSQ + (vperm vr3 vr1 vr2 vr27) ; aligned result appears in VR3 + (dbg t)) ; Look at result in some debugger + (blr)) + + AltiVec registers are not preserved by CATCH and UNWIND-PROTECT. +Since AltiVec is only accessible from LAP in OpenMCL and since LAP +functions rarely use high- level control structures, this should rarely be +a problem in practice. + LAP functions which use non-volatile vector registers and which call +(Lisp ?) code which may use CATCH or UNWIND-PROTECT should save those +vector registers before such a call and restore them on return. This is +one of the intended uses of the WITH-VECTOR-BUFFER lap macro. + + + + + Development-Mode Dictionary + + + *WARN-IF-REDEFINE-KERNEL* + *warn-if-redefine-kernel + Name + *WARN-IF-REDEFINE-KERNEL* — + Variable + Description + When true, attempts to redefine (via DEFUN or DEFMETHOD) +functions and methods that are marked as being +"predefined" signal continuable errors. + Note that these are CERRORs, not warnings, and that +no lisp functions or methods have been defined in the kernel +in MCL or OpenMCL since 1987 or so. + + + + SET-DEVELOPMENT-ENVIRONMENT + set-development-environment + Name + SET-DEVELOPMENT-ENVIRONMENT — + Function + Synopsis + +set-development-environment + &optional + unmark-builtin-functions + + Description + Arranges that the outermost special bindings of *PACKAGE* +and *WARN-IF-REDEFINE-KERNEL* restore values of the "CCL" +package and NIL to these variables, respectively. If the optional +argument is true, marks all globally defined functions and methods +as being "not predefined" (this is a fairly expensive +operation.) + + + + SET-USER-ENVIRONMENT + set-user-environment + Name + SET-USER-ENVIRONMENT — + Function + Synopsis + +set-user-environment + &optional mark-builtin-functions + + Description + Arranges that the outermost special bindings of *PACKAGE* +and *WARN-IF-REDEFINE-KERNEL* restore values of the +"CL-USER" package and T to these variables, respectively. +If the optional argument is true, marks all globally defined +functions and methods as being "predefined" (this is a +fairly expensive operation.) + + + + *ALTIVEC-AVAILABLE* + *altivec-available* + Name + *ALTIVEC-AVAILABLE* — + Variable + Description + This variable is intitialized each time an OpenMCL session +starts based on information provided by the lisp kernel. Its value +is true if AltiVec is present and false otherwise. This variable +shouldn't be set by user code. + + + + ALTIVEC-AVAILABLE-P + altivec-available-p + Name + ALTIVEC-AVAILABLE-P — + Function + Synopsis + +altivec-available-p + + Description + Returns non-NIL if AltiVec is available. + + + + *ALTIVEC-LAPMACROS-MAINTAIN-VRSAVE-P* + *altivec-lapmacros-maintain-vrsave-p* + Name + *ALTIVEC-LAPMACROS-MAINTAIN-VRSAVE-P* — + Variable + Description + Intended to control the expansion of certain lap macros. +Initialized to NIL on LinuxPPC; initialized to T on platforms +(such as MacOS X/Darwin) that require that the VRSAVE SPR contain +a bitmask of active vector registers at all times. + + + + WITH-ALTIVEC-REGISTERS + with-altivec-registers + Name + WITH-ALTIVEC-REGISTERS — + LAP Macro + Synopsis + +with-altivec-registers + reglist &body body + + Arguments and Values + reglist + A list of vector register names (vr0 .. vr31). + body + A sequence of PPC LAP instructions. + + + Description + Specifies the set of AltiVec registers used in body. If +*altivec-lapmacros-maintain-vrsave-p* is true when the macro is +expanded, generates code to save the VRSAVE SPR and updates VRSAVE +to incude a bitmask generated from the specified register list. +Generates code which saves any non-volatile vector registers which +appear in the register list, executes body, and restores the saved +non-volatile vector registers (and, if +*altivec-lapmacros-maintain-vrsave-p* is true, restores VRSAVE as +well. Uses the IMM0 register (r3) as a temporary. + + + + WITH-VECTOR-BUFFER + with-vector-buffer + Name + WITH-VECTOR-BUFFER — + LAP Macro + Synopsis + +with-vector-buffer base n &body body + + Arguments and Values + base + Any available general-purpose register. + n + An integer between 1 and 254, inclusive. (Shouldtypically be much, much closer to 1.) Specifies the size ofthe buffer, in 16-byte units. + body + A sequence of PPC LAP instructions. + + + Description + Generates code which allocates a 16-byte aligned buffer +large enough to contain N vector registers; the GPR base points to +the lowest address of this buffer. After processing body, the +buffer will be deallocated. The body should preserve the value of +base as long as it needs to reference the buffer. It's +intended that base be used as a base register in stvx and lvx +instructions within the body. + + + Symbol Index + + + #_, see + + %ff-call, see + + %reference-external-entry-point, see + + *alternate-line-terminator*, see + + *altivec-available*, see + + *altivec-lapmacros-maintain-vrsave-p*, see + + *current-process*, see + + *default-external-format*, see + + *ticks-per-second*, see + + *warn-if-redefine-kernel, see + + :external-format, see + + :y, see + + @class, see + + @selector, see +A + + accept-connection, see + + all-processes, see + + altivec-available-p, see +C + + close, see + + close-shared-library, see + + configure-egc, see + + current-directory-name, see +D + + def-foreign-type, see + + defcallback, see + + define-objc-class-method, see + + define-objc-method, see + + dotted-to-ipaddr, see +E + + egc, see + + egc-active-p, see + + egc-configuration, see + + egc-enabled-p, see + + external, see + + external-call, see + + external-process-error-stream, see + + external-process-id, see + + external-process-input-stream, see + + external-process-output-stream, see + + external-process-status, see +F + + ff-call, see + + foreign-symbol-address, see + + foreign-symbol-entry, see +G + + gc-retain-pages, see + + gc-retaining-pages, see + + get-user-home-dir, see + + getenv, see + + getpid, see + + getuid, see + + grab-lock, see +I + + ipaddr-to-dotted, see + + ipaddr-to-hostname, see +L + + lisp-heap-gc-threshold, see + + local-host, see + + local-port, see + + lookup-hostname, see + + lookup-port, see +M + + make-lock, see + + make-process, see + + make-read-write-lock, see + + make-record, see + + make-semaphore, see + + make-socket, see +N + + ns-lisp-string, see +O + + open-shared-library, see + + os-command, see +P + + pref, see + + process-abort, see + + process-allow-schedule, see + + process-enable, see + + process-input-wait, see + + process-interrupt, see + + process-kill, see + + process-output-wait, see + + process-preset, see + + process-reset, see + + process-resume, see + + process-run-function, see + + process-suspend, see + + process-suspend-count, see + + process-wait, see + + process-wait-with-timeout, see + + process-whostate, see +R + + receive-from, see + + release-lock, see + + remote-host, see + + remote-port, see + + request-terminal-input-via-break, see + + rlet, see + + rletz, see + + run-program, see +S + + send-to, see + + set-development-environment, see + + set-lisp-heap-gc-threshold, see + + set-user-environment, see + + setenv, see + + setgid, see + + setuid, see + + SHARP-AMPERSAND, see + + shutdown, see + + signal-external-process, see + + signal-semaphore, see + + socket-address-family, see + + socket-connect, see + + socket-error, see + + socket-error-code, see + + socket-error-identifier, see + + socket-error-situation, see + + socket-format, see + + socket-os-fd, see + + socket-type, see + + stream-device, see + + stream-read-ivector, see + + stream-read-list, see + + stream-read-vector, see + + stream-write-ivector, see + + stream-write-list, see + + stream-write-vector, see +T + + terminate-when-unreachable, see + + timed-wait-on-semaphore, see + + try-lock, see +U + + unuse-interface-dir, see + + use-interface-dir, see + + use-lisp-heap-gc-threshold, see +W + + wait-on-semaphore, see + + with-altivec-registers, see + + with-lock-grabbed, see + + with-open-socket, see + + with-read-lock, see + + with-terminal-input, see + + with-vector-buffer, see + + with-write-lock, see + + without-interrupts, see + +