Changes between Version 5 and Version 6 of ReleaseNotes


Ignore:
Timestamp:
11/09/07 02:12:51 (6 years ago)
Author:
gb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ReleaseNotes

    v5 v6  
    107107 
    108108- The break-loop colon commands for showing the contents of 
    109   stack frames try to present the frame's contents in a way that's 
    110   (hopefully) more meaningful and useful.  For each stack frame 
    111   shown in detail, the corresponding function's argument list 
    112   is printed, followed by the current values of the function's 
    113   arguments (indented slightly), a blank line, and the current 
    114   values of the function's local variables (outdented slightly.) 
    115   The old method of showing a stack frame's "raw" contents is 
    116   still available as the :RAW break loop command. 
    117  
    118   The new style of presenting a stack-frame's contents is also 
    119   used in the Cocoa IDE. 
     109stack frames try to present the frame's contents in a way that's 
     110(hopefully) more meaningful and useful.  For each stack frame 
     111shown in detail, the corresponding function's argument list 
     112is printed, followed by the current values of the function's 
     113arguments (indented slightly), a blank line, and the current 
     114values of the function's local variables (outdented slightly.) 
     115The old method of showing a stack frame's "raw" contents is 
     116still available as the :RAW break loop command. 
     117 
     118The new style of presenting a stack-frame's contents is also 
     119used in the Cocoa IDE. 
    120120 
    121121- It's historically been possible to create stacks (for threads 
    122   other than the original one) whose size exceeds the nominal 
    123   OS resource limits for a stack's size.  (OpenMCL's threads 
    124   use multiple stacks; the stack in question is the one that 
    125   OpenMCL generally refers to as the "control" or "C" stack.) 
    126   It's not entirely clear what (if anything) the consequences 
    127   of exceeding these limits have been, but OpenMCL's GC can 
    128   use all of the available (C) stack space that it thinks it 
    129   has under some conditions, and, under OSX/Mach/Darwin, there 
    130   have been reports of excessive page file creation and paging 
    131   activity that don't seem related to heap behavior in environments 
    132   where the GC is running on (and possibly using much of) a stack 
    133   whose size greatly exceeds the hard resource limit on stack 
    134   size. 
    135  
    136   Trying to determine exactly what was causing the excessive 
    137   pages got me trapped in a twisty maze of Mach kernel sources, 
    138   all alike.  I tried to pin C stack size to the hard resource 
    139   limit on stack size and have not been able to provoke the 
    140   excessive paging problems since, but am not confident in 
    141   concluding (yet) that the problems had to do with resource 
    142   limits being exceeded. 
    143  
    144   The hard resource limits on stack size for the OS versions 
    145   that I have readily available (in bash, do "ulimit -s -H"; 
    146   in tcsh, it's "limit -h s", don't know offhand about other 
    147   shells) are: 
    148  
    149   unlimited on Linux 
    150   ~512M on FreeBSD 
    151   ~64M on Darwin 
    152  
    153   The effect of observing (rather than exceeding) this limit 
    154   on the maximum depth of lisp recursion in OpenMCL is: 
    155  
    156   * nothing, on x86-64 (the C stack is not used by lisp code 
    157     on x86-64) 
    158  
    159   * visible on ppc32, which uses 4 32-bit words on the control 
    160     stack for each lisp function invocation 
    161  
    162   * more visible on ppc64, which uses 4 64-bit words of control 
    163     stack for each lisp function invocation. 
    164  
    165   That seems to suggest that (given that the actual stack resource 
    166   limit is a bit under 64M and that OpenMCL signals stack overflow 
    167   when the stack pointer gets within a few hundred KB of the actual 
    168   limit) that ppc64 threads are now limited to a maximum of about 
    169   2000000 function calls. 
    170  
    171   (All of this only matters if attempts are made to create threads 
    172   with large stacks; the default stack sizes in OpenMCL are usually 
    173   1-2 MB.) 
     122other than the original one) whose size exceeds the nominal 
     123OS resource limits for a stack's size.  (OpenMCL's threads 
     124use multiple stacks; the stack in question is the one that 
     125OpenMCL generally refers to as the "control" or "C" stack.) 
     126It's not entirely clear what (if anything) the consequences 
     127of exceeding these limits have been, but OpenMCL's GC can 
     128use all of the available (C) stack space that it thinks it 
     129has under some conditions, and, under OSX/Mach/Darwin, there 
     130have been reports of excessive page file creation and paging 
     131activity that don't seem related to heap behavior in environments 
     132where the GC is running on (and possibly using much of) a stack 
     133whose size greatly exceeds the hard resource limit on stack 
     134size. 
     135 
     136Trying to determine exactly what was causing the excessive 
     137pages got me trapped in a twisty maze of Mach kernel sources, 
     138all alike.  I tried to pin C stack size to the hard resource 
     139limit on stack size and have not been able to provoke the 
     140excessive paging problems since, but am not confident in 
     141concluding (yet) that the problems had to do with resource 
     142limits being exceeded. 
     143 
     144The hard resource limits on stack size for the OS versions 
     145that I have readily available (in bash, do "ulimit -s -H"; 
     146in tcsh, it's "limit -h s", don't know offhand about other 
     147shells) are: 
     148 
     149  * unlimited on Linux 
     150  * ~512M on FreeBSD 
     151  * ~64M on Darwin 
     152 
     153The effect of observing (rather than exceeding) this limit 
     154on the maximum depth of lisp recursion in OpenMCL is: 
     155 
     156  * nothing, on x86-64 (the C stack is not used by lisp code on x86-64) 
     157 
     158  * visible on ppc32, which uses 4 32-bit words on the control stack for each lisp function invocation 
     159 
     160  * more visible on ppc64, which uses 4 64-bit words of control  stack for each lisp function invocation. 
     161 
     162That seems to suggest that (given that the actual stack resource 
     163limit is a bit under 64M and that OpenMCL signals stack overflow 
     164when the stack pointer gets within a few hundred KB of the actual 
     165limit) that ppc64 threads are now limited to a maximum of about 
     1662000000 function calls. 
     167 
     168(All of this only matters if attempts are made to create threads 
     169with large stacks; the default stack sizes in OpenMCL are usually 
     1701-2 MB.) 
    174171 
    175172- On a cheerier (and certainly less confusing) note: for the last few 
    176   years, OpenMCL has shipped with an extended example which provides an 
    177   integrated development environment (IDE) based on Cocoa; that's often 
    178   been described as "the demo IDE" and could also be fairly described as 
    179   "slow", "buggy", "incomplete", and "little more than a proof of 
    180   concept." 
    181  
    182   I think that it's fair to describe the current state of the IDE as 
    183   being "less slow", "less buggy", "less incomplete", and "much more 
    184   than a proof of concept" than it has been (e.g., there's been some 
    185   actual progress over the last few months and there are plans to 
    186   try to continue working on the IDE and related tools.)  It'd probably 
    187   be optimistic to call it "usable" in its current state (that may 
    188   depend on how low one's threshold of usability is), but I hope that 
    189   people who've been discouraged by the lack of IDE progress over the 
    190   last few years will see reason to be encouraged (and that anyone 
    191   interested will submit bug reports, patches, feature requests, code ...) 
     173years, OpenMCL has shipped with an extended example which provides an 
     174integrated development environment (IDE) based on Cocoa; that's often 
     175been described as "the demo IDE" and could also be fairly described as 
     176"slow", "buggy", "incomplete", and "little more than a proof of 
     177concept." 
     178 
     179I think that it's fair to describe the current state of the IDE as 
     180being "less slow", "less buggy", "less incomplete", and "much more 
     181than a proof of concept" than it has been (e.g., there's been some 
     182actual progress over the last few months and there are plans to 
     183try to continue working on the IDE and related tools.)  It'd probably 
     184be optimistic to call it "usable" in its current state (that may 
     185depend on how low one's threshold of usability is), but I hope that 
     186people who've been discouraged by the lack of IDE progress over the 
     187last few years will see reason to be encouraged (and that anyone 
     188interested will submit bug reports, patches, feature requests, code ...) 
    192189 
    193190- There are now "objc-bridge" and "cocoa-ide" subdirectories; by default, 
    194   REQUIRE will look in these directories for files whose name matches 
    195   a module name.  Several files were moved from the "examples" directory 
    196   to "objc-bridge"; other example files, the "OpenMCL.app" skeleton 
    197   bundle, and the "hemlock" directory were moved to "cocoa-ide". 
     191REQUIRE will look in these directories for files whose name matches 
     192a module name.  Several files were moved from the "examples" directory 
     193to "objc-bridge"; other example files, the "OpenMCL.app" skeleton 
     194bundle, and the "hemlock" directory were moved to "cocoa-ide". 
    198195 
    199196 
    200197= OpenMCL 1.1-pre-070512 = 
    201198- The FASL version changed (old FASL files won't work with this 
    202   lisp version), as did the version information which tries to 
    203   keep the kernel in sync with heap images.  Note that it's generally 
    204   a lot easier to recompile recent sources with recent images, e.g., 
    205   trying to compile 070512 sources with an 070407 image is unlikely 
    206   to work without tricky bootstrapping. 
     199lisp version), as did the version information which tries to 
     200keep the kernel in sync with heap images.  Note that it's generally 
     201a lot easier to recompile recent sources with recent images, e.g., 
     202trying to compile 070512 sources with an 070407 image is unlikely 
     203to work without tricky bootstrapping. 
     204 
    207205- Most of the changes in this release involve the calling sequence 
    208   used on x86-64.  In very general terms, some kinds of function-call 
    209   intensive code may see a significant performance boost, most code 
    210   should see a slight improvement, some code might see a (hopefully 
    211   very slight) degradation, and anything significantly slower than 
    212   previous releases should be reported as a bug. 
    213   It is -possible- that some of these changes may cause errors to 
    214   be reported differently (the function reported as the function 
    215   executing when the error ocurred might be different/wrong).  I 
    216   have not seen as many cases of this as I expected to when making 
    217   the change, but am also not sure that I fixed all possible cases. 
     206used on x86-64.  In very general terms, some kinds of function-call 
     207intensive code may see a significant performance boost, most code 
     208should see a slight improvement, some code might see a (hopefully 
     209very slight) degradation, and anything significantly slower than 
     210previous releases should be reported as a bug. 
     211 
     212It is -possible- that some of these changes may cause errors to 
     213be reported differently (the function reported as the function 
     214executing when the error ocurred might be different/wrong).  I 
     215have not seen as many cases of this as I expected to when making 
     216the change, but am also not sure that I fixed all possible cases. 
     217 
    218218- The FFI-related reader macros #_, #$, and #& all read a case-sensitive 
    219   foreign function, constant, or variable name from the input stream 
    220   and try to find the corresponding definition in the interface files. 
    221   If the name is prefixed with a #\? - as in #_?foo - the macros 
    222   return true if the definition could be found and false otherwise. 
    223   (The general idea is that this might be useful for conditionalizing 
    224   code in some cases, and there should be -some- way of quietly testing  
    225   that something's defined.) 
     219foreign function, constant, or variable name from the input stream 
     220and try to find the corresponding definition in the interface files. 
     221If the name is prefixed with a #\? - as in #_?foo - the macros 
     222return true if the definition could be found and false otherwise. 
     223(The general idea is that this might be useful for conditionalizing 
     224code in some cases, and there should be -some- way of quietly testing  
     225that something's defined.) 
     226 
    226227- There is now support for making the contents of (possibly very large) 
    227   files accessible as lisp vectors.  (This may be many times faster 
    228   than something like  
     228files accessible as lisp vectors.  (This may be many times faster 
     229than something like  
    229230 
    230231{{{ 
     
    235236}}} 
    236237 
    237   but has the similar effect of making the contents of VECTOR match the 
    238   contents of the file.) 
     238but has the similar effect of making the contents of VECTOR match the 
     239contents of the file.) 
    239240 
    240241{{{CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]}}} 
    241242 
    242   "element-type" should be a type specifier such that 
    243   (UPGRADED-ARRAY-ELEMENT-TYPE element-type) is a subtype 
    244   of either SIGNED-BYTE or UNSIGNED-BYTE. 
    245  
    246   Tries to open the file named by "pathname" for reading and to 
    247   map its contents into the process's address space via #_mmap; 
    248   if successful, returns a lisp vector of element-type 
    249   (UPGRADED-ARRAY-ELEMENT-TYPE element-type) which is displaced 
    250   to an underlying (SIMPLE-ARRAY element-type (*)) whose contents 
    251   match the mapped file's. 
    252  
    253   Because of alignment issues, the mapped file's contents will 
    254   start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit 
    255   platforms) "into" the vector; the displaced array returned by 
    256   CCL:MAP-FILE-TO-IVECTOR hides this overhead, but its usually 
    257   more efficient to operate on the underlying simple 1-dimensional 
    258   array.  Given a displaced array (like the value returned by 
    259   CCL:MAP-FILE-TO-IVECTOR), the CL function ARRAY-DISPLACEMENT 
    260   returns the underlying array and the displacement index in elements. 
    261  
    262   Currently, only read-only file mapping is supported; the underlying 
    263   vector will be allocated in read-only memory, and attempts to use 
    264   (e.g.) (SETF (AREF ...) ...) to modify the mapped vector's contents 
    265   will result in memory faults. 
     243"element-type" should be a type specifier such that 
     244(UPGRADED-ARRAY-ELEMENT-TYPE element-type) is a subtype 
     245of either SIGNED-BYTE or UNSIGNED-BYTE. 
     246 
     247Tries to open the file named by "pathname" for reading and to 
     248map its contents into the process's address space via #_mmap; 
     249if successful, returns a lisp vector of element-type 
     250(UPGRADED-ARRAY-ELEMENT-TYPE element-type) which is displaced 
     251to an underlying (SIMPLE-ARRAY element-type (*)) whose contents 
     252match the mapped file's. 
     253 
     254Because of alignment issues, the mapped file's contents will 
     255start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit 
     256platforms) "into" the vector; the displaced array returned by 
     257CCL:MAP-FILE-TO-IVECTOR hides this overhead, but its usually 
     258more efficient to operate on the underlying simple 1-dimensional 
     259array.  Given a displaced array (like the value returned by 
     260CCL:MAP-FILE-TO-IVECTOR), the CL function ARRAY-DISPLACEMENT 
     261returns the underlying array and the displacement index in elements. 
     262 
     263Currently, only read-only file mapping is supported; the underlying 
     264vector will be allocated in read-only memory, and attempts to use 
     265(e.g.) (SETF (AREF ...) ...) to modify the mapped vector's contents 
     266will result in memory faults. 
    266267   
    267   {{{CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]}}} 
    268  
    269   Equivalent to (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)). 
    270  
    271   {{{CCL:UNMAP-IVECTOR displaced-vector}}} 
    272  
    273   If the argument is a mapped vector (as returned by 
    274   MAP-FILE-TO-IVECTOR) that has not yet been "unmapped" by this 
    275   function, undoes the memory mapping, closes the mapped file, and 
    276   adjusts its argument so that it's displaced to a 0-length vector. 
    277  
    278   CCL:UNMAP-OCTET-VECTOR is an alias for CCL:UNMAP-IVECTOR 
    279  
    280   Note that whether a vector's created by MAKE-ARRAY or by mapping 
    281   a file's contents, it can't have ARRAY-TOTAL-SIZE-LIMIT or more 
    282   elements.  (ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) in 32-bit OpenMCL 
    283   and (EXPT 2 56) in 64-bit versions. 
     268{{{CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]}}} 
     269 
     270Equivalent to (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)). 
     271 
     272{{{CCL:UNMAP-IVECTOR displaced-vector}}} 
     273 
     274If the argument is a mapped vector (as returned by 
     275MAP-FILE-TO-IVECTOR) that has not yet been "unmapped" by this 
     276function, undoes the memory mapping, closes the mapped file, and 
     277adjusts its argument so that it's displaced to a 0-length vector. 
     278 
     279CCL:UNMAP-OCTET-VECTOR is an alias for CCL:UNMAP-IVECTOR 
     280 
     281Note that whether a vector's created by MAKE-ARRAY or by mapping 
     282a file's contents, it can't have ARRAY-TOTAL-SIZE-LIMIT or more 
     283elements.  (ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) in 32-bit OpenMCL 
     284and (EXPT 2 56) in 64-bit versions. 
    284285 
    285286- The lisp kernel now tries to signal memory faults that occur when 
    286   running lisp code as lisp errors.  As a silly example: 
     287running lisp code as lisp errors.  As a silly example: 
    287288 
    288289{{{ 
     
    297298}}} 
    298299 
    299   The fact that things are handled this way (rather than going 
    300   into the kernel debugger with no easy way of recovering) makes 
    301   it possible to continue a session without losing work in many 
    302   cases.  In a trivial example like the one above, it's relatively 
    303   easy to see that no harm has been done and the error should 
    304   not be hard to recover from.  In some other cases, it may be 
    305   true that a buggy function has been scribbling ofer memory for 
    306   a while before that scribbling resulted in a machine exception. 
    307  
    308   Moral: if you get an unexpected "memory fault" error (the 
    309   condition type is actually CCL::INVALID-MEMORY-ACCESS) and 
    310   don't understand why the fault occurred and the consequences 
    311   of continuing in the lisp session where the fault occurred, 
    312   you should view the state of that session with some suspicion. 
    313  
    314   Faults in foreign code (should) still trap into the kernel 
    315   debugger.  (It'd be nice to be able to treat these as lisp 
    316   errors with the same caveats as described above, but that 
    317   is more complicated in some cases and isn't yet implemented.) 
     300The fact that things are handled this way (rather than going 
     301into the kernel debugger with no easy way of recovering) makes 
     302it possible to continue a session without losing work in many 
     303cases.  In a trivial example like the one above, it's relatively 
     304easy to see that no harm has been done and the error should 
     305not be hard to recover from.  In some other cases, it may be 
     306true that a buggy function has been scribbling ofer memory for 
     307a while before that scribbling resulted in a machine exception. 
     308 
     309Moral: if you get an unexpected "memory fault" error (the 
     310condition type is actually CCL::INVALID-MEMORY-ACCESS) and 
     311don't understand why the fault occurred and the consequences 
     312of continuing in the lisp session where the fault occurred, 
     313you should view the state of that session with some suspicion. 
     314 
     315Faults in foreign code (should) still trap into the kernel 
     316debugger.  (It'd be nice to be able to treat these as lisp 
     317errors with the same caveats as described above, but that 
     318is more complicated in some cases and isn't yet implemented.) 
    318319 
    319320- An obscure kernel debugger command - (A), which tries to advance 
    320   the program counter by one instruction - is now disabled on x86-64. 
    321   (On the PPC, "one instruction" always meant "4 bytes"; implementing 
    322   this correctly on x86-64 would require the ability to at least 
    323   partially disassemble arbitrary x86-64 instructions.) 
    324  
    325   On the other hand, the kernel debugger should be able to show 
    326   FPU registers on x86-64. 
     321the program counter by one instruction - is now disabled on x86-64. 
     322(On the PPC, "one instruction" always meant "4 bytes"; implementing 
     323this correctly on x86-64 would require the ability to at least 
     324partially disassemble arbitrary x86-64 instructions.) 
     325 
     326On the other hand, the kernel debugger should be able to show 
     327FPU registers on x86-64. 
    327328 
    328329 
    329330= OpenMCL 1.1-pre-070408 = 
    330331- The FASL version changed (old FASL files won't work with this 
    331   lisp version), as did the version information which tries to 
    332   keep the kernel in sync with heap images.  Note that it's generally 
    333   a lot easier to recompile recent sources with recent images, e.g., 
    334   trying to compile 070408 sources with an 070214 image is unlikely 
    335   to work without tricky bootstrapping. 
     332lisp version), as did the version information which tries to 
     333keep the kernel in sync with heap images.  Note that it's generally 
     334a lot easier to recompile recent sources with recent images, e.g., 
     335trying to compile 070408 sources with an 070214 image is unlikely 
     336to work without tricky bootstrapping. 
     337 
    336338- There's now a Trac bug-tracking/wiki site for OpenMCL at  
    337   <http://trac.clozure.com/openmcl>.  It needs bug reports; please 
    338   visit that site and use the features there to report any bugs 
    339   that you find. 
     339<http://trac.clozure.com/openmcl>.  It needs bug reports; please 
     340visit that site and use the features there to report any bugs 
     341that you find. 
     342 
    340343- DEFSTATIC (aka DEFGLOBAL) 
    341   (CCL:DEFSTATIC var value &optional doc-string) 
    342   is like DEFPARAMETER in that it proclaims the variable "var" to 
    343   be special, sets its value to "value", and sets the symbol's 
    344   VARIABLE documentation to the optional doc-string.  It differs 
    345   from DEFPARAMETER in that it further asserts that the variable 
    346   should never be bound dynamically in any thread (via LET/LAMBDA/etc.); 
    347   the compiler treats any attempts to bind a "static" variable as an 
    348   error. 
    349   It is legal to change the value of a "static" variable, but since 
    350   all threads see the same (static) binding of that variable it may 
    351   be necessary to synchronize assignments made from multiple threads. 
    352   (A "static" variable binding is effectively a shared, global resource; 
    353   a dynamic binding is thread-private.) 
    354   Access to the value of a static variable is typically faster than 
    355   is access to the value a special variable that's not proclaimed to 
    356   be "static". 
    357   This functionality has been in MCL/OpenMCL for a long time under 
    358   the name CCL:DEFGLOBAL; CCL:DEFGLOBAL is an alias for CCL:DEFSTATIC, 
    359   but the latter seemed to be a better name. 
     344{{{(CCL:DEFSTATIC var value &optional doc-string)}}} 
     345is like DEFPARAMETER in that it proclaims the variable "var" to 
     346be special, sets its value to "value", and sets the symbol's 
     347VARIABLE documentation to the optional doc-string.  It differs 
     348from DEFPARAMETER in that it further asserts that the variable 
     349should never be bound dynamically in any thread (via LET/LAMBDA/etc.); 
     350the compiler treats any attempts to bind a "static" variable as an 
     351error. 
     352 
     353It is legal to change the value of a "static" variable, but since 
     354all threads see the same (static) binding of that variable it may 
     355be necessary to synchronize assignments made from multiple threads. 
     356(A "static" variable binding is effectively a shared, global resource; 
     357a dynamic binding is thread-private.) 
     358 
     359Access to the value of a static variable is typically faster than 
     360is access to the value a special variable that's not proclaimed to 
     361be "static". 
     362 
     363This functionality has been in MCL/OpenMCL for a long time under 
     364the name CCL:DEFGLOBAL; CCL:DEFGLOBAL is an alias for CCL:DEFSTATIC, 
     365but the latter seemed to be a better name. 
     366 
    360367- The type of foreign object that a MACPTR points to can now be 
    361   asserted (this means that a MACPTR object can contain a small 
    362   integer which identifies the alleged FOREIGN-TYPE of the object that 
    363   the points to.  RLET, MAKE-RECORD, and MAKE-GCABLE-RECORD (see below) 
    364   assert the foreign type of the object that the MACPTR object they 
    365   create (as do some new features of the ObjC bridge, described further 
    366   below.) 
    367   PRINT-OBJECT on a MACPTR will try to print information about the 
    368   asserted type of that pointer, as well as information about where 
    369   the pointer was allocated (heap, stack) and whether it's scheduled 
    370   for automatic reclamation by the GC. 
    371   A few constructs that conceivable should assert the type of the 
    372   pointers they create (e.g., some flavor of PREF, SLOT-VALUE in 
    373   the ObjC bridge) don't yet do so. 
    374   A rather obvious way of exploiting typed pointers - namely, extending 
    375   DESCRIBE and INSPECT to show the contents of foreign records - is 
    376   not yet implemented. 
     368asserted (this means that a MACPTR object can contain a small 
     369integer which identifies the alleged FOREIGN-TYPE of the object that 
     370the points to.  RLET, MAKE-RECORD, and MAKE-GCABLE-RECORD (see below) 
     371assert the foreign type of the object that the MACPTR object they 
     372create (as do some new features of the ObjC bridge, described further 
     373below.) 
     374 
     375PRINT-OBJECT on a MACPTR will try to print information about the 
     376asserted type of that pointer, as well as information about where 
     377the pointer was allocated (heap, stack) and whether it's scheduled 
     378for automatic reclamation by the GC. 
     379 
     380A few constructs that conceivable should assert the type of the 
     381pointers they create (e.g., some flavor of PREF, SLOT-VALUE in 
     382the ObjC bridge) don't yet do so. 
     383 
     384A rather obvious way of exploiting typed pointers - namely, extending 
     385DESCRIBE and INSPECT to show the contents of foreign records - is 
     386not yet implemented. 
     387 
    377388- MAKE-GCABLE-RECORD is like MAKE-RECORD, in that it "makes an instance 
    378   of a foreign record type".  (Or, to be more banal about it, uses 
    379   #_malloc to allocate a block of foreign memory of the size of the 
    380   foreign record type named by its argument.)  MAKE-GCABLE-RECORD 
    381   additionally tells the lisp garbage collector that it should free 
    382   the foreign memory when the MACPTR object that describes it becomes 
    383   garbage. 
    384   When using "gcable pointers", it's important to remember the 
    385   distinction between a MACPTR object (which is a lisp object, more- 
    386   or-less like any other) and the block of foreign memory that the 
    387   MACPTR object points to.  If a gcable MACPTR is the only thing 
    388   in the world ("lisp world" or "foreign world") that references 
    389   the underlying block of foreign memory, then freeing the foreign 
    390   memory when it becomes impossible to reference it is convenient 
    391   and sane.  If other lisp MACPTRs reference the underlying block 
    392   of foreign memory or if the address of that foreign memory is 
    393   passed to and retained by foreign code, having the GC free the 
    394   memory may have unpleasant consequences if those other references 
    395   are used. 
     389of a foreign record type".  (Or, to be more banal about it, uses 
     390#_malloc to allocate a block of foreign memory of the size of the 
     391foreign record type named by its argument.)  MAKE-GCABLE-RECORD 
     392additionally tells the lisp garbage collector that it should free 
     393the foreign memory when the MACPTR object that describes it becomes 
     394garbage. 
     395 
     396When using "gcable pointers", it's important to remember the 
     397distinction between a MACPTR object (which is a lisp object, more- 
     398or-less like any other) and the block of foreign memory that the 
     399MACPTR object points to.  If a gcable MACPTR is the only thing 
     400in the world ("lisp world" or "foreign world") that references 
     401the underlying block of foreign memory, then freeing the foreign 
     402memory when it becomes impossible to reference it is convenient 
     403and sane.  If other lisp MACPTRs reference the underlying block 
     404of foreign memory or if the address of that foreign memory is 
     405passed to and retained by foreign code, having the GC free the 
     406memory may have unpleasant consequences if those other references 
     407are used. 
     408 
    396409- CCL:FREE (which is mostly just a wrapper around #_free that allows 
    397   #_free to be called early in the bootstrapping process) is now  
    398   exported; if its argument is a gcable pointer (e.g., created via 
    399   MAKE-GCABLE-POINTER), it will tell the GC that the pointer's 
    400   foreign memory has been freed "manually" before calling #_free. 
     410#_free to be called early in the bootstrapping process) is now  
     411exported; if its argument is a gcable pointer (e.g., created via 
     412MAKE-GCABLE-POINTER), it will tell the GC that the pointer's 
     413foreign memory has been freed "manually" before calling #_free. 
     414 
    401415- The mechanisms used to implement locks has changed (for the curious, 
    402   the changes involve the use of spinlocks rather than a sequence 
    403   of atomic additions.)  Someone demonstrated a case of deadlock 
    404   (where a thread was waiting for a lock that was available) under 
    405   the old implementation.  I'm not sure that I fully understand how 
    406   that could have happened, but the new implementation at least has 
    407   the advantage of being a little easier to understand and might be 
    408   a tiny bit faster.  Please let me know if either of these assumptions 
    409   was incorrect. 
     416the changes involve the use of spinlocks rather than a sequence 
     417of atomic additions.)  Someone demonstrated a case of deadlock 
     418(where a thread was waiting for a lock that was available) under 
     419the old implementation.  I'm not sure that I fully understand how 
     420that could have happened, but the new implementation at least has 
     421the advantage of being a little easier to understand and might be 
     422a tiny bit faster.  Please let me know if either of these assumptions 
     423was incorrect. 
     424 
    410425- An EOF (control-d) in the REPL (when standard input is a tty or pty 
    411   device) has traditionally caused an exit to the outer break loop 
    412   (or had no effect if the REPL was not in a break loop).  If 
    413   CCL:*QUIT-ON-EOF* is set, an EOF causes the lisp to quit.  (It 
    414   actually invokes a listener-specific method, so in a multi-listener 
    415   window system environemt, it might simply cause the listener which 
    416   receives the EOF to exit.) 
    417   None of this has any effect when running under environments like 
    418   SLIME, and (as mentioned above) only matters if the standard input 
    419   devices is a tty or pseudo-tty (where it's possible to continue 
    420   reading after an EOF has been read.)  If running under an xterm 
    421   or OSX Terminal.app, standard input is probably a pty; if running 
    422   in an Emacs shell buffer or under other means under emacs, different 
    423   types of IPC mechanisms (pipes, sockets) might be used. 
     426device) has traditionally caused an exit to the outer break loop 
     427(or had no effect if the REPL was not in a break loop).  If 
     428CCL:*QUIT-ON-EOF* is set, an EOF causes the lisp to quit.  (It 
     429actually invokes a listener-specific method, so in a multi-listener 
     430window system environemt, it might simply cause the listener which 
     431receives the EOF to exit.) 
     432 
     433None of this has any effect when running under environments like 
     434SLIME, and (as mentioned above) only matters if the standard input 
     435devices is a tty or pseudo-tty (where it's possible to continue 
     436reading after an EOF has been read.)  If running under an xterm 
     437or OSX Terminal.app, standard input is probably a pty; if running 
     438in an Emacs shell buffer or under other means under emacs, different 
     439types of IPC mechanisms (pipes, sockets) might be used. 
     440 
    424441- SAVE-APPLICATION has historically changed the type of all MACPTR 
    425   objects (turning them into "dead macptrs", since it's generally 
    426   meaningless to refer to a foreign pointer from a previous session 
    427   and generally better to get a type error than some more mysterious 
    428   or severe failure).  This no longer happens for null pointers (pointers 
    429   to address 0); COMPILE-FILE also now allows null pointers to be referenced 
    430   as constants in compiled code. 
     442objects (turning them into "dead macptrs", since it's generally 
     443meaningless to refer to a foreign pointer from a previous session 
     444and generally better to get a type error than some more mysterious 
     445or severe failure).  This no longer happens for null pointers (pointers 
     446to address 0); COMPILE-FILE also now allows null pointers to be referenced 
     447as constants in compiled code. 
     448 
    431449- Not entirely coincidentally, CCL:+NULL-PTR+ is now defined as a constant 
    432   (whose value is a null pointer.)  In some cases, it may be more 
    433   efficient or convenient to pass CCL:+NULL-PTR+ to foreign code than 
    434   it would be to call (CCL:%NULL-PTR) to "produce" one. 
     450(whose value is a null pointer.)  In some cases, it may be more 
     451efficient or convenient to pass CCL:+NULL-PTR+ to foreign code than 
     452it would be to call (CCL:%NULL-PTR) to "produce" one. 
     453 
    435454- Historically, OpenMCL (and MCL) have maintained a list of open file 
    436   streams in the value of CCL:*OPEN-FILE-STREAMS*; maintaining this 
    437   list helps to ensure that streams get closed in as orderly a manner 
    438   as possible when the lisp exits.  The code which accessed this list 
    439   didn't do so in a thread-safe manner. 
    440   The list is now maintained in a lexical variable; the function 
    441   CCL:OPEN-FILE-STREAMS returns a copy of that list,  
    442   CCL:NOTE-OPEN-FILE-STREAM adds its argument (a file stream) to the 
    443   list, and CCL:REMOVE-OPEN-FILE-STREAM removes its argument (a file stream) 
    444   from the list.  (All of these functions use appropriate locking.) 
     455streams in the value of CCL:*OPEN-FILE-STREAMS*; maintaining this 
     456list helps to ensure that streams get closed in as orderly a manner 
     457as possible when the lisp exits.  The code which accessed this list 
     458didn't do so in a thread-safe manner. 
     459 
     460The list is now maintained in a lexical variable; the function 
     461CCL:OPEN-FILE-STREAMS returns a copy of that list,  
     462CCL:NOTE-OPEN-FILE-STREAM adds its argument (a file stream) to the 
     463list, and CCL:REMOVE-OPEN-FILE-STREAM removes its argument (a file stream) 
     464from the list.  (All of these functions use appropriate locking.) 
     465 
    445466- There were a number of timing-related problems related to PROCESS-INTERRUPT 
    446   (usually involving rapidly and repeatedly interrupting a thread over 
    447   a long period of time.)  This should be a lot more reliable now 
    448   (exactly what could go wrong and why and how is all a little hard to 
    449   describe.)  
     467(usually involving rapidly and repeatedly interrupting a thread over 
     468a long period of time.)  This should be a lot more reliable now 
     469(exactly what could go wrong and why and how is all a little hard to 
     470describe.)  
     471 
    450472- Some Linux distributions may initialize the user's environment in 
    451   a way that imposes a soft limit on the amount of virtual memory that 
    452   a process is allowed to map.  OpenMCL now tries to raise this limit 
    453   before reserving what may be a very large amount of address space, 
    454   thanks to a patch from Andi Kleen. 
     473a way that imposes a soft limit on the amount of virtual memory that 
     474a process is allowed to map.  OpenMCL now tries to raise this limit 
     475before reserving what may be a very large amount of address space, 
     476thanks to a patch from Andi Kleen. 
     477 
    455478- There were a number of problems with UTF-16 streams, found and 
    456   fixed by Takehiko Abe. 
     479fixed by Takehiko Abe. 
     480 
    457481- Takehiko Abe also provided fixes for some code in "ccl:lib;xref.lisp" 
    458   and in source-file recording/reporting that (still) didn't understand 
    459   the concept of EQL-SPECIALIZER metaobjects. 
     482and in source-file recording/reporting that (still) didn't understand 
     483the concept of EQL-SPECIALIZER metaobjects. 
     484 
    460485- ObjC bridge and ObjC examples 
    461   The ObjC bridge provides a few new mechanisms for defining ObjC 
    462   methods, for calling ObjC "generic functions" (e.g., message sending), 
    463   and for dealing with frequently-used record types and with differences 
    464   between 32-bit and (forthcoming) 64-bit ObjC/Cocoa implementations. 
     486The ObjC bridge provides a few new mechanisms for defining ObjC 
     487methods, for calling ObjC "generic functions" (e.g., message sending), 
     488and for dealing with frequently-used record types and with differences 
     489between 32-bit and (forthcoming) 64-bit ObjC/Cocoa implementations. 
    465490   
    466   A lot of macros/functions/other things that really should have been 
    467   exported from some package for the last few years finally have been 
    468   exported from the OBJC or NS packages (and a lot of things that have 
    469   historically been internal to CCL are re-imported into CCL). 
    470  
    471   Cocoa (and the underlying Core Graphics libraries) have historically 
    472   used 32-bit floats and 32-bit integers in data structures that describe 
    473   geometry, font sizes and metrics, and elsewhere.  64-bit Cocoa will 
    474   use 64-bit floats and 64-bit integers in many cases. 
    475  
    476   The bridge defines the type NS:CGFLOAT as the lisp type of the  
    477   preferred float type on the platform, and the constant NS:+CGFLOAT+. 
    478   On DarwinPPC32, the foreign types :cgfloat, :<NSUI>nteger, and 
    479   :<NSI>nteger are defined by the bridge (as 32-bit float, 32-bit 
    480   unsigned integer, and 32-bit signed integer, respectively.); these  
    481   types are defined (as 64-bit variants) in the 64-bit interfaces. 
    482  
    483   All ObjC classes are properly named, either with a name exported 
    484   from the NS package (in the case of a predefined class declared in 
    485   the interface files) or with the name provided in the DEFCLASS 
    486   form (with :METACLASS NS:+NS-OBJECT) which defines the class from 
    487   lisp.  The class's lisp name is now proclaimed to be a "static" 
    488   variable (as if by DEFSTATIC, as described above) and given the 
    489   class object as its value.  In other words: 
     491A lot of macros/functions/other things that really should have been 
     492exported from some package for the last few years finally have been 
     493exported from the OBJC or NS packages (and a lot of things that have 
     494historically been internal to CCL are re-imported into CCL). 
     495 
     496Cocoa (and the underlying Core Graphics libraries) have historically 
     497used 32-bit floats and 32-bit integers in data structures that describe 
     498geometry, font sizes and metrics, and elsewhere.  64-bit Cocoa will 
     499use 64-bit floats and 64-bit integers in many cases. 
     500 
     501The bridge defines the type NS:CGFLOAT as the lisp type of the  
     502preferred float type on the platform, and the constant NS:+CGFLOAT+. 
     503On DarwinPPC32, the foreign types :cgfloat, :<NSUI>nteger, and 
     504:<NSI>nteger are defined by the bridge (as 32-bit float, 32-bit 
     505unsigned integer, and 32-bit signed integer, respectively.); these  
     506types are defined (as 64-bit variants) in the 64-bit interfaces. 
     507 
     508All ObjC classes are properly named, either with a name exported 
     509from the NS package (in the case of a predefined class declared in 
     510the interface files) or with the name provided in the DEFCLASS 
     511form (with :METACLASS NS:+NS-OBJECT) which defines the class from 
     512lisp.  The class's lisp name is now proclaimed to be a "static" 
     513variable (as if by DEFSTATIC, as described above) and given the 
     514class object as its value.  In other words: 
    490515 
    491516{{{ 
     
    493518}}} 
    494519 
    495   and 
     520and 
    496521 
    497522{{{ 
     
    499524}}} 
    500525 
    501   are equivalent.  (Since it's not legal to bind a "static" variable, 
    502   it may be necessary to rename some things so that unrelated 
    503   variables whose names coincidentally conflict with ObjC class names 
    504   don't do so.) 
     526are equivalent.  (Since it's not legal to bind a "static" variable, 
     527it may be necessary to rename some things so that unrelated 
     528variables whose names coincidentally conflict with ObjC class names 
     529don't do so.) 
    505530 
    506531- A new reader macro - #/ - reads a sequence of "constituent" characters 
    507   (including colons) from the stream on which it appears and interns 
    508   that sequence - with case preserved and colons intact - in a new package 
    509   whose name is NEXTSTEP-FUNCTIONS, exporting the symbol from that package. 
    510   This means that the act of reading "#/alloc" returns the the symbol 
    511   NEXTSTEP-FUNCTIONS:|alloc|, and the act of reading "#/initWithFrame:" 
    512   returns the symbol NEXTSTEP-FUNCTIONS:|initWithFrame:|.  The intent 
    513   is that the reader macro can be used to construct symbols whose names 
    514   match ObjC message names; the reader macro tries to do some sanity 
    515   checks (such as insisting that a name that contains at least one 
    516   colon ends in a colon), but isn't totally rigourous about enforcing 
    517   ObjC message name conventions. 
    518  
    519   A symbol read using this macro can be used as an operand in 
    520   most places where an ObjC message name can be used, such as 
    521   in the (@SELECTOR ...) construct (which is now OBJC:@SELECTOR,  
    522   btw.) 
    523  
    524   Marco Baringer proposed the idea of using a reader macro to 
    525   construct lisp symbols which matched ObjC message names. 
     532(including colons) from the stream on which it appears and interns 
     533that sequence - with case preserved and colons intact - in a new package 
     534whose name is NEXTSTEP-FUNCTIONS, exporting the symbol from that package. 
     535This means that the act of reading "#/alloc" returns the the symbol 
     536NEXTSTEP-FUNCTIONS:|alloc|, and the act of reading "#/initWithFrame:" 
     537returns the symbol NEXTSTEP-FUNCTIONS:|initWithFrame:|.  The intent 
     538is that the reader macro can be used to construct symbols whose names 
     539match ObjC message names; the reader macro tries to do some sanity 
     540checks (such as insisting that a name that contains at least one 
     541colon ends in a colon), but isn't totally rigourous about enforcing 
     542ObjC message name conventions. 
     543 
     544A symbol read using this macro can be used as an operand in 
     545most places where an ObjC message name can be used, such as 
     546in the (@SELECTOR ...) construct (which is now OBJC:@SELECTOR,  
     547btw.) 
     548 
     549Marco Baringer proposed the idea of using a reader macro to 
     550construct lisp symbols which matched ObjC message names. 
    526551 
    527552- The act of interning a new symbol in the NEXTSTEP-FUNCTIONS 
    528   package triggers an interface database lookup of Objc methods 
    529   with the corresponding message name.  If any such information 
    530   is found, a special type of dispatching function is created 
    531   and initialized and the weird-looking symbol is given that 
    532   dispatching function as its function definition. 
    533  
    534   The dispatching knows how to call declared ObjC methods defined on 
    535   the message.  In many cases, all methods have the same foreign type 
    536   signature, and the dispatching function merely passes any arguments 
    537   that it receives to a function that does an ObjC message send with 
    538   the indicated foreign argument and return types.  In other cases, 
    539   where different ObjC messages have different type signatures, the 
    540   dispatching function tries to choose a function that handles the 
    541   right type signature based on the class of the dispatching function's 
    542   first argument. 
    543  
    544   If new information about ObjC methods is introduced (e.g., by 
    545   using additional interface files or as ObjC methods are defined 
    546   from lisp), the dispatch function is reinitialized to recognize 
    547   newly-introduced foreign type signatures. 
    548  
    549   The argument and result coercion that the bridge has tradionally 
    550   supported is supported by the new mechanism (e.g., :<BOOL> arguments 
    551   can be specified as lisp booleans and :<BOOL> results are returned 
    552   as lisp boolean values, and an argument value of NIL is coerced to 
    553   a null pointer if the corresponding argument type is :ID. 
    554  
    555   Some ObjC methods accept variable numbers of arguments; the 
    556   foreign types of non-required arguments are determined by the 
    557   lisp types of those arguments (e.g., integers are passed as 
    558   integers, floats as floats, pointers as pointers, record types 
    559   by reference.) 
    560  
    561   Some examples: 
     553package triggers an interface database lookup of Objc methods 
     554with the corresponding message name.  If any such information 
     555is found, a special type of dispatching function is created 
     556and initialized and the weird-looking symbol is given that 
     557dispatching function as its function definition. 
     558 
     559The dispatching function knows how to call declared ObjC methods defined on 
     560the message.  In many cases, all methods have the same foreign type 
     561signature, and the dispatching function merely passes any arguments 
     562that it receives to a function that does an ObjC message send with 
     563the indicated foreign argument and return types.  In other cases, 
     564where different ObjC messages have different type signatures, the 
     565dispatching function tries to choose a function that handles the 
     566right type signature based on the class of the dispatching function's 
     567first argument. 
     568 
     569If new information about ObjC methods is introduced (e.g., by 
     570using additional interface files or as ObjC methods are defined 
     571from lisp), the dispatch function is reinitialized to recognize 
     572newly-introduced foreign type signatures. 
     573 
     574The argument and result coercion that the bridge has tradionally 
     575supported is supported by the new mechanism (e.g., :<BOOL> arguments 
     576can be specified as lisp booleans and :<BOOL> results are returned 
     577as lisp boolean values, and an argument value of NIL is coerced to 
     578a null pointer if the corresponding argument type is :ID. 
     579 
     580Some ObjC methods accept variable numbers of arguments; the 
     581foreign types of non-required arguments are determined by the 
     582lisp types of those arguments (e.g., integers are passed as 
     583integers, floats as floats, pointers as pointers, record types 
     584by reference.) 
     585 
     586Some examples: 
    562587 
    563588{{{ 
     
    579604}}} 
    580605 
    581   ObjC methods that "return" structures return them as gcable pointers 
    582   when called via dispatch functions.  E.g., if "my-window" is an 
    583   NS:NS-WINDOW instance, then 
     606ObjC methods that "return" structures return them as gcable pointers 
     607when called via dispatch functions.  E.g., if "my-window" is an 
     608NS:NS-WINDOW instance, then 
    584609 
    585610{{{ 
     
    587612}}} 
    588613 
    589   will return a gcable pointer to a structure that describes that window's 
    590   frame rectangle.  (The good news is that there's no need to use SLET 
    591   or special structure-returning message send syntax; the bad news is 
    592   that #_malloc, #_free, and the GC are all involved in the creation 
    593   and eventual destruction of structure-typed return values.  Unless 
    594   and until those factors negatively affect performance, the advantages 
    595   seem to outweigh the disadvantages.) 
     614will return a gcable pointer to a structure that describes that window's 
     615frame rectangle.  (The good news is that there's no need to use SLET 
     616or special structure-returning message send syntax; the bad news is 
     617that #_malloc, #_free, and the GC are all involved in the creation 
     618and eventual destruction of structure-typed return values.  Unless 
     619and until those factors negatively affect performance, the advantages 
     620seem to outweigh the disadvantages.) 
    596621 
    597622- Since foreign pointers are now (sometimes, somewhat) typed, it's 
    598   possible to treat pointers to some foreign types as "instances of 
    599   built-in classes."  Specifically, a pointer to an :<NSR>ect is 
    600   recognized as an instance of the built-in class NS:NS-RECT, a 
    601   pointer to an <NSS>ize is treated as an instance of NS:NS-SIZE, 
    602   <NSP>oint is recognized as NS:NS-POINT, and <NSR>ange maps to 
    603   NS:NS-RANGE.  (There are a few other more obscure structure 
    604   types that get this treatment, and with a little more work the 
    605   mechanism could be made extensible.) 
    606  
    607   For each of these built-in classes: 
    608  
    609   - a PRINT-OBJECT method is defined 
    610  
    611   - a foreign type name derived from the class name (e.g., :NS-RECT 
     623possible to treat pointers to some foreign types as "instances of 
     624built-in classes."  Specifically, a pointer to an :<NSR>ect is 
     625recognized as an instance of the built-in class NS:NS-RECT, a 
     626pointer to an <NSS>ize is treated as an instance of NS:NS-SIZE, 
     627<NSP>oint is recognized as NS:NS-POINT, and <NSR>ange maps to 
     628NS:NS-RANGE.  (There are a few other more obscure structure 
     629types that get this treatment, and with a little more work the 
     630mechanism could be made extensible.) 
     631 
     632For each of these built-in classes: 
     633 
     634  * a PRINT-OBJECT method is defined 
     635 
     636  * a foreign type name derived from the class name (e.g., :NS-RECT 
    612637    for NS:NS-RECT) is made an alias for the corresponding type 
    613638    (so it's possible to say (RLET ((R :NS-RECT)) ...)).