Changes between Version 5 and Version 6 of ReleaseNotes


Ignore:
Timestamp:
Nov 9, 2007, 2:12:51 AM (12 years ago)
Author:
gb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ReleaseNotes

    v5 v6  
    107107
    108108- The break-loop colon commands for showing the contents of
    109   stack frames try to present the frame's contents in a way that's
    110   (hopefully) more meaningful and useful.  For each stack frame
    111   shown in detail, the corresponding function's argument list
    112   is printed, followed by the current values of the function's
    113   arguments (indented slightly), a blank line, and the current
    114   values of the function's local variables (outdented slightly.)
    115   The old method of showing a stack frame's "raw" contents is
    116   still available as the :RAW break loop command.
    117 
    118   The new style of presenting a stack-frame's contents is also
    119   used in the Cocoa IDE.
     109stack frames try to present the frame's contents in a way that's
     110(hopefully) more meaningful and useful.  For each stack frame
     111shown in detail, the corresponding function's argument list
     112is printed, followed by the current values of the function's
     113arguments (indented slightly), a blank line, and the current
     114values of the function's local variables (outdented slightly.)
     115The old method of showing a stack frame's "raw" contents is
     116still available as the :RAW break loop command.
     117
     118The new style of presenting a stack-frame's contents is also
     119used in the Cocoa IDE.
    120120
    121121- It's historically been possible to create stacks (for threads
    122   other than the original one) whose size exceeds the nominal
    123   OS resource limits for a stack's size.  (OpenMCL's threads
    124   use multiple stacks; the stack in question is the one that
    125   OpenMCL generally refers to as the "control" or "C" stack.)
    126   It's not entirely clear what (if anything) the consequences
    127   of exceeding these limits have been, but OpenMCL's GC can
    128   use all of the available (C) stack space that it thinks it
    129   has under some conditions, and, under OSX/Mach/Darwin, there
    130   have been reports of excessive page file creation and paging
    131   activity that don't seem related to heap behavior in environments
    132   where the GC is running on (and possibly using much of) a stack
    133   whose size greatly exceeds the hard resource limit on stack
    134   size.
    135 
    136   Trying to determine exactly what was causing the excessive
    137   pages got me trapped in a twisty maze of Mach kernel sources,
    138   all alike.  I tried to pin C stack size to the hard resource
    139   limit on stack size and have not been able to provoke the
    140   excessive paging problems since, but am not confident in
    141   concluding (yet) that the problems had to do with resource
    142   limits being exceeded.
    143 
    144   The hard resource limits on stack size for the OS versions
    145   that I have readily available (in bash, do "ulimit -s -H";
    146   in tcsh, it's "limit -h s", don't know offhand about other
    147   shells) are:
    148 
    149   unlimited on Linux
    150   ~512M on FreeBSD
    151   ~64M on Darwin
    152 
    153   The effect of observing (rather than exceeding) this limit
    154   on the maximum depth of lisp recursion in OpenMCL is:
    155 
    156   * nothing, on x86-64 (the C stack is not used by lisp code
    157     on x86-64)
    158 
    159   * visible on ppc32, which uses 4 32-bit words on the control
    160     stack for each lisp function invocation
    161 
    162   * more visible on ppc64, which uses 4 64-bit words of control
    163     stack for each lisp function invocation.
    164 
    165   That seems to suggest that (given that the actual stack resource
    166   limit is a bit under 64M and that OpenMCL signals stack overflow
    167   when the stack pointer gets within a few hundred KB of the actual
    168   limit) that ppc64 threads are now limited to a maximum of about
    169   2000000 function calls.
    170 
    171   (All of this only matters if attempts are made to create threads
    172   with large stacks; the default stack sizes in OpenMCL are usually
    173   1-2 MB.)
     122other than the original one) whose size exceeds the nominal
     123OS resource limits for a stack's size.  (OpenMCL's threads
     124use multiple stacks; the stack in question is the one that
     125OpenMCL generally refers to as the "control" or "C" stack.)
     126It's not entirely clear what (if anything) the consequences
     127of exceeding these limits have been, but OpenMCL's GC can
     128use all of the available (C) stack space that it thinks it
     129has under some conditions, and, under OSX/Mach/Darwin, there
     130have been reports of excessive page file creation and paging
     131activity that don't seem related to heap behavior in environments
     132where the GC is running on (and possibly using much of) a stack
     133whose size greatly exceeds the hard resource limit on stack
     134size.
     135
     136Trying to determine exactly what was causing the excessive
     137pages got me trapped in a twisty maze of Mach kernel sources,
     138all alike.  I tried to pin C stack size to the hard resource
     139limit on stack size and have not been able to provoke the
     140excessive paging problems since, but am not confident in
     141concluding (yet) that the problems had to do with resource
     142limits being exceeded.
     143
     144The hard resource limits on stack size for the OS versions
     145that I have readily available (in bash, do "ulimit -s -H";
     146in tcsh, it's "limit -h s", don't know offhand about other
     147shells) are:
     148
     149  * unlimited on Linux
     150  * ~512M on FreeBSD
     151  * ~64M on Darwin
     152
     153The effect of observing (rather than exceeding) this limit
     154on the maximum depth of lisp recursion in OpenMCL is:
     155
     156  * nothing, on x86-64 (the C stack is not used by lisp code on x86-64)
     157
     158  * visible on ppc32, which uses 4 32-bit words on the control stack for each lisp function invocation
     159
     160  * more visible on ppc64, which uses 4 64-bit words of control  stack for each lisp function invocation.
     161
     162That seems to suggest that (given that the actual stack resource
     163limit is a bit under 64M and that OpenMCL signals stack overflow
     164when the stack pointer gets within a few hundred KB of the actual
     165limit) that ppc64 threads are now limited to a maximum of about
     1662000000 function calls.
     167
     168(All of this only matters if attempts are made to create threads
     169with large stacks; the default stack sizes in OpenMCL are usually
     1701-2 MB.)
    174171
    175172- On a cheerier (and certainly less confusing) note: for the last few
    176   years, OpenMCL has shipped with an extended example which provides an
    177   integrated development environment (IDE) based on Cocoa; that's often
    178   been described as "the demo IDE" and could also be fairly described as
    179   "slow", "buggy", "incomplete", and "little more than a proof of
    180   concept."
    181 
    182   I think that it's fair to describe the current state of the IDE as
    183   being "less slow", "less buggy", "less incomplete", and "much more
    184   than a proof of concept" than it has been (e.g., there's been some
    185   actual progress over the last few months and there are plans to
    186   try to continue working on the IDE and related tools.)  It'd probably
    187   be optimistic to call it "usable" in its current state (that may
    188   depend on how low one's threshold of usability is), but I hope that
    189   people who've been discouraged by the lack of IDE progress over the
    190   last few years will see reason to be encouraged (and that anyone
    191   interested will submit bug reports, patches, feature requests, code ...)
     173years, OpenMCL has shipped with an extended example which provides an
     174integrated development environment (IDE) based on Cocoa; that's often
     175been described as "the demo IDE" and could also be fairly described as
     176"slow", "buggy", "incomplete", and "little more than a proof of
     177concept."
     178
     179I think that it's fair to describe the current state of the IDE as
     180being "less slow", "less buggy", "less incomplete", and "much more
     181than a proof of concept" than it has been (e.g., there's been some
     182actual progress over the last few months and there are plans to
     183try to continue working on the IDE and related tools.)  It'd probably
     184be optimistic to call it "usable" in its current state (that may
     185depend on how low one's threshold of usability is), but I hope that
     186people who've been discouraged by the lack of IDE progress over the
     187last few years will see reason to be encouraged (and that anyone
     188interested will submit bug reports, patches, feature requests, code ...)
    192189
    193190- There are now "objc-bridge" and "cocoa-ide" subdirectories; by default,
    194   REQUIRE will look in these directories for files whose name matches
    195   a module name.  Several files were moved from the "examples" directory
    196   to "objc-bridge"; other example files, the "OpenMCL.app" skeleton
    197   bundle, and the "hemlock" directory were moved to "cocoa-ide".
     191REQUIRE will look in these directories for files whose name matches
     192a module name.  Several files were moved from the "examples" directory
     193to "objc-bridge"; other example files, the "OpenMCL.app" skeleton
     194bundle, and the "hemlock" directory were moved to "cocoa-ide".
    198195
    199196
    200197= OpenMCL 1.1-pre-070512 =
    201198- The FASL version changed (old FASL files won't work with this
    202   lisp version), as did the version information which tries to
    203   keep the kernel in sync with heap images.  Note that it's generally
    204   a lot easier to recompile recent sources with recent images, e.g.,
    205   trying to compile 070512 sources with an 070407 image is unlikely
    206   to work without tricky bootstrapping.
     199lisp version), as did the version information which tries to
     200keep the kernel in sync with heap images.  Note that it's generally
     201a lot easier to recompile recent sources with recent images, e.g.,
     202trying to compile 070512 sources with an 070407 image is unlikely
     203to work without tricky bootstrapping.
     204
    207205- Most of the changes in this release involve the calling sequence
    208   used on x86-64.  In very general terms, some kinds of function-call
    209   intensive code may see a significant performance boost, most code
    210   should see a slight improvement, some code might see a (hopefully
    211   very slight) degradation, and anything significantly slower than
    212   previous releases should be reported as a bug.
    213   It is -possible- that some of these changes may cause errors to
    214   be reported differently (the function reported as the function
    215   executing when the error ocurred might be different/wrong).  I
    216   have not seen as many cases of this as I expected to when making
    217   the change, but am also not sure that I fixed all possible cases.
     206used on x86-64.  In very general terms, some kinds of function-call
     207intensive code may see a significant performance boost, most code
     208should see a slight improvement, some code might see a (hopefully
     209very slight) degradation, and anything significantly slower than
     210previous releases should be reported as a bug.
     211
     212It is -possible- that some of these changes may cause errors to
     213be reported differently (the function reported as the function
     214executing when the error ocurred might be different/wrong).  I
     215have not seen as many cases of this as I expected to when making
     216the change, but am also not sure that I fixed all possible cases.
     217
    218218- The FFI-related reader macros #_, #$, and #& all read a case-sensitive
    219   foreign function, constant, or variable name from the input stream
    220   and try to find the corresponding definition in the interface files.
    221   If the name is prefixed with a #\? - as in #_?foo - the macros
    222   return true if the definition could be found and false otherwise.
    223   (The general idea is that this might be useful for conditionalizing
    224   code in some cases, and there should be -some- way of quietly testing
    225   that something's defined.)
     219foreign function, constant, or variable name from the input stream
     220and try to find the corresponding definition in the interface files.
     221If the name is prefixed with a #\? - as in #_?foo - the macros
     222return true if the definition could be found and false otherwise.
     223(The general idea is that this might be useful for conditionalizing
     224code in some cases, and there should be -some- way of quietly testing
     225that something's defined.)
     226
    226227- There is now support for making the contents of (possibly very large)
    227   files accessible as lisp vectors.  (This may be many times faster
    228   than something like
     228files accessible as lisp vectors.  (This may be many times faster
     229than something like
    229230
    230231{{{
     
    235236}}}
    236237
    237   but has the similar effect of making the contents of VECTOR match the
    238   contents of the file.)
     238but has the similar effect of making the contents of VECTOR match the
     239contents of the file.)
    239240
    240241{{{CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]}}}
    241242
    242   "element-type" should be a type specifier such that
    243   (UPGRADED-ARRAY-ELEMENT-TYPE element-type) is a subtype
    244   of either SIGNED-BYTE or UNSIGNED-BYTE.
    245 
    246   Tries to open the file named by "pathname" for reading and to
    247   map its contents into the process's address space via #_mmap;
    248   if successful, returns a lisp vector of element-type
    249   (UPGRADED-ARRAY-ELEMENT-TYPE element-type) which is displaced
    250   to an underlying (SIMPLE-ARRAY element-type (*)) whose contents
    251   match the mapped file's.
    252 
    253   Because of alignment issues, the mapped file's contents will
    254   start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit
    255   platforms) "into" the vector; the displaced array returned by
    256   CCL:MAP-FILE-TO-IVECTOR hides this overhead, but its usually
    257   more efficient to operate on the underlying simple 1-dimensional
    258   array.  Given a displaced array (like the value returned by
    259   CCL:MAP-FILE-TO-IVECTOR), the CL function ARRAY-DISPLACEMENT
    260   returns the underlying array and the displacement index in elements.
    261 
    262   Currently, only read-only file mapping is supported; the underlying
    263   vector will be allocated in read-only memory, and attempts to use
    264   (e.g.) (SETF (AREF ...) ...) to modify the mapped vector's contents
    265   will result in memory faults.
     243"element-type" should be a type specifier such that
     244(UPGRADED-ARRAY-ELEMENT-TYPE element-type) is a subtype
     245of either SIGNED-BYTE or UNSIGNED-BYTE.
     246
     247Tries to open the file named by "pathname" for reading and to
     248map its contents into the process's address space via #_mmap;
     249if successful, returns a lisp vector of element-type
     250(UPGRADED-ARRAY-ELEMENT-TYPE element-type) which is displaced
     251to an underlying (SIMPLE-ARRAY element-type (*)) whose contents
     252match the mapped file's.
     253
     254Because of alignment issues, the mapped file's contents will
     255start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit
     256platforms) "into" the vector; the displaced array returned by
     257CCL:MAP-FILE-TO-IVECTOR hides this overhead, but its usually
     258more efficient to operate on the underlying simple 1-dimensional
     259array.  Given a displaced array (like the value returned by
     260CCL:MAP-FILE-TO-IVECTOR), the CL function ARRAY-DISPLACEMENT
     261returns the underlying array and the displacement index in elements.
     262
     263Currently, only read-only file mapping is supported; the underlying
     264vector will be allocated in read-only memory, and attempts to use
     265(e.g.) (SETF (AREF ...) ...) to modify the mapped vector's contents
     266will result in memory faults.
    266267 
    267   {{{CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]}}}
    268 
    269   Equivalent to (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)).
    270 
    271   {{{CCL:UNMAP-IVECTOR displaced-vector}}}
    272 
    273   If the argument is a mapped vector (as returned by
    274   MAP-FILE-TO-IVECTOR) that has not yet been "unmapped" by this
    275   function, undoes the memory mapping, closes the mapped file, and
    276   adjusts its argument so that it's displaced to a 0-length vector.
    277 
    278   CCL:UNMAP-OCTET-VECTOR is an alias for CCL:UNMAP-IVECTOR
    279 
    280   Note that whether a vector's created by MAKE-ARRAY or by mapping
    281   a file's contents, it can't have ARRAY-TOTAL-SIZE-LIMIT or more
    282   elements.  (ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) in 32-bit OpenMCL
    283   and (EXPT 2 56) in 64-bit versions.
     268{{{CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]}}}
     269
     270Equivalent to (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)).
     271
     272{{{CCL:UNMAP-IVECTOR displaced-vector}}}
     273
     274If the argument is a mapped vector (as returned by
     275MAP-FILE-TO-IVECTOR) that has not yet been "unmapped" by this
     276function, undoes the memory mapping, closes the mapped file, and
     277adjusts its argument so that it's displaced to a 0-length vector.
     278
     279CCL:UNMAP-OCTET-VECTOR is an alias for CCL:UNMAP-IVECTOR
     280
     281Note that whether a vector's created by MAKE-ARRAY or by mapping
     282a file's contents, it can't have ARRAY-TOTAL-SIZE-LIMIT or more
     283elements.  (ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) in 32-bit OpenMCL
     284and (EXPT 2 56) in 64-bit versions.
    284285
    285286- The lisp kernel now tries to signal memory faults that occur when
    286   running lisp code as lisp errors.  As a silly example:
     287running lisp code as lisp errors.  As a silly example:
    287288
    288289{{{
     
    297298}}}
    298299
    299   The fact that things are handled this way (rather than going
    300   into the kernel debugger with no easy way of recovering) makes
    301   it possible to continue a session without losing work in many
    302   cases.  In a trivial example like the one above, it's relatively
    303   easy to see that no harm has been done and the error should
    304   not be hard to recover from.  In some other cases, it may be
    305   true that a buggy function has been scribbling ofer memory for
    306   a while before that scribbling resulted in a machine exception.
    307 
    308   Moral: if you get an unexpected "memory fault" error (the
    309   condition type is actually CCL::INVALID-MEMORY-ACCESS) and
    310   don't understand why the fault occurred and the consequences
    311   of continuing in the lisp session where the fault occurred,
    312   you should view the state of that session with some suspicion.
    313 
    314   Faults in foreign code (should) still trap into the kernel
    315   debugger.  (It'd be nice to be able to treat these as lisp
    316   errors with the same caveats as described above, but that
    317   is more complicated in some cases and isn't yet implemented.)
     300The fact that things are handled this way (rather than going
     301into the kernel debugger with no easy way of recovering) makes
     302it possible to continue a session without losing work in many
     303cases.  In a trivial example like the one above, it's relatively
     304easy to see that no harm has been done and the error should
     305not be hard to recover from.  In some other cases, it may be
     306true that a buggy function has been scribbling ofer memory for
     307a while before that scribbling resulted in a machine exception.
     308
     309Moral: if you get an unexpected "memory fault" error (the
     310condition type is actually CCL::INVALID-MEMORY-ACCESS) and
     311don't understand why the fault occurred and the consequences
     312of continuing in the lisp session where the fault occurred,
     313you should view the state of that session with some suspicion.
     314
     315Faults in foreign code (should) still trap into the kernel
     316debugger.  (It'd be nice to be able to treat these as lisp
     317errors with the same caveats as described above, but that
     318is more complicated in some cases and isn't yet implemented.)
    318319
    319320- An obscure kernel debugger command - (A), which tries to advance
    320   the program counter by one instruction - is now disabled on x86-64.
    321   (On the PPC, "one instruction" always meant "4 bytes"; implementing
    322   this correctly on x86-64 would require the ability to at least
    323   partially disassemble arbitrary x86-64 instructions.)
    324 
    325   On the other hand, the kernel debugger should be able to show
    326   FPU registers on x86-64.
     321the program counter by one instruction - is now disabled on x86-64.
     322(On the PPC, "one instruction" always meant "4 bytes"; implementing
     323this correctly on x86-64 would require the ability to at least
     324partially disassemble arbitrary x86-64 instructions.)
     325
     326On the other hand, the kernel debugger should be able to show
     327FPU registers on x86-64.
    327328
    328329
    329330= OpenMCL 1.1-pre-070408 =
    330331- The FASL version changed (old FASL files won't work with this
    331   lisp version), as did the version information which tries to
    332   keep the kernel in sync with heap images.  Note that it's generally
    333   a lot easier to recompile recent sources with recent images, e.g.,
    334   trying to compile 070408 sources with an 070214 image is unlikely
    335   to work without tricky bootstrapping.
     332lisp version), as did the version information which tries to
     333keep the kernel in sync with heap images.  Note that it's generally
     334a lot easier to recompile recent sources with recent images, e.g.,
     335trying to compile 070408 sources with an 070214 image is unlikely
     336to work without tricky bootstrapping.
     337
    336338- There's now a Trac bug-tracking/wiki site for OpenMCL at
    337   <http://trac.clozure.com/openmcl>.  It needs bug reports; please
    338   visit that site and use the features there to report any bugs
    339   that you find.
     339<http://trac.clozure.com/openmcl>.  It needs bug reports; please
     340visit that site and use the features there to report any bugs
     341that you find.
     342
    340343- DEFSTATIC (aka DEFGLOBAL)
    341   (CCL:DEFSTATIC var value &optional doc-string)
    342   is like DEFPARAMETER in that it proclaims the variable "var" to
    343   be special, sets its value to "value", and sets the symbol's
    344   VARIABLE documentation to the optional doc-string.  It differs
    345   from DEFPARAMETER in that it further asserts that the variable
    346   should never be bound dynamically in any thread (via LET/LAMBDA/etc.);
    347   the compiler treats any attempts to bind a "static" variable as an
    348   error.
    349   It is legal to change the value of a "static" variable, but since
    350   all threads see the same (static) binding of that variable it may
    351   be necessary to synchronize assignments made from multiple threads.
    352   (A "static" variable binding is effectively a shared, global resource;
    353   a dynamic binding is thread-private.)
    354   Access to the value of a static variable is typically faster than
    355   is access to the value a special variable that's not proclaimed to
    356   be "static".
    357   This functionality has been in MCL/OpenMCL for a long time under
    358   the name CCL:DEFGLOBAL; CCL:DEFGLOBAL is an alias for CCL:DEFSTATIC,
    359   but the latter seemed to be a better name.
     344{{{(CCL:DEFSTATIC var value &optional doc-string)}}}
     345is like DEFPARAMETER in that it proclaims the variable "var" to
     346be special, sets its value to "value", and sets the symbol's
     347VARIABLE documentation to the optional doc-string.  It differs
     348from DEFPARAMETER in that it further asserts that the variable
     349should never be bound dynamically in any thread (via LET/LAMBDA/etc.);
     350the compiler treats any attempts to bind a "static" variable as an
     351error.
     352
     353It is legal to change the value of a "static" variable, but since
     354all threads see the same (static) binding of that variable it may
     355be necessary to synchronize assignments made from multiple threads.
     356(A "static" variable binding is effectively a shared, global resource;
     357a dynamic binding is thread-private.)
     358
     359Access to the value of a static variable is typically faster than
     360is access to the value a special variable that's not proclaimed to
     361be "static".
     362
     363This functionality has been in MCL/OpenMCL for a long time under
     364the name CCL:DEFGLOBAL; CCL:DEFGLOBAL is an alias for CCL:DEFSTATIC,
     365but the latter seemed to be a better name.
     366
    360367- The type of foreign object that a MACPTR points to can now be
    361   asserted (this means that a MACPTR object can contain a small
    362   integer which identifies the alleged FOREIGN-TYPE of the object that
    363   the points to.  RLET, MAKE-RECORD, and MAKE-GCABLE-RECORD (see below)
    364   assert the foreign type of the object that the MACPTR object they
    365   create (as do some new features of the ObjC bridge, described further
    366   below.)
    367   PRINT-OBJECT on a MACPTR will try to print information about the
    368   asserted type of that pointer, as well as information about where
    369   the pointer was allocated (heap, stack) and whether it's scheduled
    370   for automatic reclamation by the GC.
    371   A few constructs that conceivable should assert the type of the
    372   pointers they create (e.g., some flavor of PREF, SLOT-VALUE in
    373   the ObjC bridge) don't yet do so.
    374   A rather obvious way of exploiting typed pointers - namely, extending
    375   DESCRIBE and INSPECT to show the contents of foreign records - is
    376   not yet implemented.
     368asserted (this means that a MACPTR object can contain a small
     369integer which identifies the alleged FOREIGN-TYPE of the object that
     370the points to.  RLET, MAKE-RECORD, and MAKE-GCABLE-RECORD (see below)
     371assert the foreign type of the object that the MACPTR object they
     372create (as do some new features of the ObjC bridge, described further
     373below.)
     374
     375PRINT-OBJECT on a MACPTR will try to print information about the
     376asserted type of that pointer, as well as information about where
     377the pointer was allocated (heap, stack) and whether it's scheduled
     378for automatic reclamation by the GC.
     379
     380A few constructs that conceivable should assert the type of the
     381pointers they create (e.g., some flavor of PREF, SLOT-VALUE in
     382the ObjC bridge) don't yet do so.
     383
     384A rather obvious way of exploiting typed pointers - namely, extending
     385DESCRIBE and INSPECT to show the contents of foreign records - is
     386not yet implemented.
     387
    377388- MAKE-GCABLE-RECORD is like MAKE-RECORD, in that it "makes an instance
    378   of a foreign record type".  (Or, to be more banal about it, uses
    379   #_malloc to allocate a block of foreign memory of the size of the
    380   foreign record type named by its argument.)  MAKE-GCABLE-RECORD
    381   additionally tells the lisp garbage collector that it should free
    382   the foreign memory when the MACPTR object that describes it becomes
    383   garbage.
    384   When using "gcable pointers", it's important to remember the
    385   distinction between a MACPTR object (which is a lisp object, more-
    386   or-less like any other) and the block of foreign memory that the
    387   MACPTR object points to.  If a gcable MACPTR is the only thing
    388   in the world ("lisp world" or "foreign world") that references
    389   the underlying block of foreign memory, then freeing the foreign
    390   memory when it becomes impossible to reference it is convenient
    391   and sane.  If other lisp MACPTRs reference the underlying block
    392   of foreign memory or if the address of that foreign memory is
    393   passed to and retained by foreign code, having the GC free the
    394   memory may have unpleasant consequences if those other references
    395   are used.
     389of a foreign record type".  (Or, to be more banal about it, uses
     390#_malloc to allocate a block of foreign memory of the size of the
     391foreign record type named by its argument.)  MAKE-GCABLE-RECORD
     392additionally tells the lisp garbage collector that it should free
     393the foreign memory when the MACPTR object that describes it becomes
     394garbage.
     395
     396When using "gcable pointers", it's important to remember the
     397distinction between a MACPTR object (which is a lisp object, more-
     398or-less like any other) and the block of foreign memory that the
     399MACPTR object points to.  If a gcable MACPTR is the only thing
     400in the world ("lisp world" or "foreign world") that references
     401the underlying block of foreign memory, then freeing the foreign
     402memory when it becomes impossible to reference it is convenient
     403and sane.  If other lisp MACPTRs reference the underlying block
     404of foreign memory or if the address of that foreign memory is
     405passed to and retained by foreign code, having the GC free the
     406memory may have unpleasant consequences if those other references
     407are used.
     408
    396409- CCL:FREE (which is mostly just a wrapper around #_free that allows
    397   #_free to be called early in the bootstrapping process) is now
    398   exported; if its argument is a gcable pointer (e.g., created via
    399   MAKE-GCABLE-POINTER), it will tell the GC that the pointer's
    400   foreign memory has been freed "manually" before calling #_free.
     410#_free to be called early in the bootstrapping process) is now
     411exported; if its argument is a gcable pointer (e.g., created via
     412MAKE-GCABLE-POINTER), it will tell the GC that the pointer's
     413foreign memory has been freed "manually" before calling #_free.
     414
    401415- The mechanisms used to implement locks has changed (for the curious,
    402   the changes involve the use of spinlocks rather than a sequence
    403   of atomic additions.)  Someone demonstrated a case of deadlock
    404   (where a thread was waiting for a lock that was available) under
    405   the old implementation.  I'm not sure that I fully understand how
    406   that could have happened, but the new implementation at least has
    407   the advantage of being a little easier to understand and might be
    408   a tiny bit faster.  Please let me know if either of these assumptions
    409   was incorrect.
     416the changes involve the use of spinlocks rather than a sequence
     417of atomic additions.)  Someone demonstrated a case of deadlock
     418(where a thread was waiting for a lock that was available) under
     419the old implementation.  I'm not sure that I fully understand how
     420that could have happened, but the new implementation at least has
     421the advantage of being a little easier to understand and might be
     422a tiny bit faster.  Please let me know if either of these assumptions
     423was incorrect.
     424
    410425- An EOF (control-d) in the REPL (when standard input is a tty or pty
    411   device) has traditionally caused an exit to the outer break loop
    412   (or had no effect if the REPL was not in a break loop).  If
    413   CCL:*QUIT-ON-EOF* is set, an EOF causes the lisp to quit.  (It
    414   actually invokes a listener-specific method, so in a multi-listener
    415   window system environemt, it might simply cause the listener which
    416   receives the EOF to exit.)
    417   None of this has any effect when running under environments like
    418   SLIME, and (as mentioned above) only matters if the standard input
    419   devices is a tty or pseudo-tty (where it's possible to continue
    420   reading after an EOF has been read.)  If running under an xterm
    421   or OSX Terminal.app, standard input is probably a pty; if running
    422   in an Emacs shell buffer or under other means under emacs, different
    423   types of IPC mechanisms (pipes, sockets) might be used.
     426device) has traditionally caused an exit to the outer break loop
     427(or had no effect if the REPL was not in a break loop).  If
     428CCL:*QUIT-ON-EOF* is set, an EOF causes the lisp to quit.  (It
     429actually invokes a listener-specific method, so in a multi-listener
     430window system environemt, it might simply cause the listener which
     431receives the EOF to exit.)
     432
     433None of this has any effect when running under environments like
     434SLIME, and (as mentioned above) only matters if the standard input
     435devices is a tty or pseudo-tty (where it's possible to continue
     436reading after an EOF has been read.)  If running under an xterm
     437or OSX Terminal.app, standard input is probably a pty; if running
     438in an Emacs shell buffer or under other means under emacs, different
     439types of IPC mechanisms (pipes, sockets) might be used.
     440
    424441- SAVE-APPLICATION has historically changed the type of all MACPTR
    425   objects (turning them into "dead macptrs", since it's generally
    426   meaningless to refer to a foreign pointer from a previous session
    427   and generally better to get a type error than some more mysterious
    428   or severe failure).  This no longer happens for null pointers (pointers
    429   to address 0); COMPILE-FILE also now allows null pointers to be referenced
    430   as constants in compiled code.
     442objects (turning them into "dead macptrs", since it's generally
     443meaningless to refer to a foreign pointer from a previous session
     444and generally better to get a type error than some more mysterious
     445or severe failure).  This no longer happens for null pointers (pointers
     446to address 0); COMPILE-FILE also now allows null pointers to be referenced
     447as constants in compiled code.
     448
    431449- Not entirely coincidentally, CCL:+NULL-PTR+ is now defined as a constant
    432   (whose value is a null pointer.)  In some cases, it may be more
    433   efficient or convenient to pass CCL:+NULL-PTR+ to foreign code than
    434   it would be to call (CCL:%NULL-PTR) to "produce" one.
     450(whose value is a null pointer.)  In some cases, it may be more
     451efficient or convenient to pass CCL:+NULL-PTR+ to foreign code than
     452it would be to call (CCL:%NULL-PTR) to "produce" one.
     453
    435454- Historically, OpenMCL (and MCL) have maintained a list of open file
    436   streams in the value of CCL:*OPEN-FILE-STREAMS*; maintaining this
    437   list helps to ensure that streams get closed in as orderly a manner
    438   as possible when the lisp exits.  The code which accessed this list
    439   didn't do so in a thread-safe manner.
    440   The list is now maintained in a lexical variable; the function
    441   CCL:OPEN-FILE-STREAMS returns a copy of that list,
    442   CCL:NOTE-OPEN-FILE-STREAM adds its argument (a file stream) to the
    443   list, and CCL:REMOVE-OPEN-FILE-STREAM removes its argument (a file stream)
    444   from the list.  (All of these functions use appropriate locking.)
     455streams in the value of CCL:*OPEN-FILE-STREAMS*; maintaining this
     456list helps to ensure that streams get closed in as orderly a manner
     457as possible when the lisp exits.  The code which accessed this list
     458didn't do so in a thread-safe manner.
     459
     460The list is now maintained in a lexical variable; the function
     461CCL:OPEN-FILE-STREAMS returns a copy of that list,
     462CCL:NOTE-OPEN-FILE-STREAM adds its argument (a file stream) to the
     463list, and CCL:REMOVE-OPEN-FILE-STREAM removes its argument (a file stream)
     464from the list.  (All of these functions use appropriate locking.)
     465
    445466- There were a number of timing-related problems related to PROCESS-INTERRUPT
    446   (usually involving rapidly and repeatedly interrupting a thread over
    447   a long period of time.)  This should be a lot more reliable now
    448   (exactly what could go wrong and why and how is all a little hard to
    449   describe.)
     467(usually involving rapidly and repeatedly interrupting a thread over
     468a long period of time.)  This should be a lot more reliable now
     469(exactly what could go wrong and why and how is all a little hard to
     470describe.)
     471
    450472- Some Linux distributions may initialize the user's environment in
    451   a way that imposes a soft limit on the amount of virtual memory that
    452   a process is allowed to map.  OpenMCL now tries to raise this limit
    453   before reserving what may be a very large amount of address space,
    454   thanks to a patch from Andi Kleen.
     473a way that imposes a soft limit on the amount of virtual memory that
     474a process is allowed to map.  OpenMCL now tries to raise this limit
     475before reserving what may be a very large amount of address space,
     476thanks to a patch from Andi Kleen.
     477
    455478- There were a number of problems with UTF-16 streams, found and
    456   fixed by Takehiko Abe.
     479fixed by Takehiko Abe.
     480
    457481- Takehiko Abe also provided fixes for some code in "ccl:lib;xref.lisp"
    458   and in source-file recording/reporting that (still) didn't understand
    459   the concept of EQL-SPECIALIZER metaobjects.
     482and in source-file recording/reporting that (still) didn't understand
     483the concept of EQL-SPECIALIZER metaobjects.
     484
    460485- ObjC bridge and ObjC examples
    461   The ObjC bridge provides a few new mechanisms for defining ObjC
    462   methods, for calling ObjC "generic functions" (e.g., message sending),
    463   and for dealing with frequently-used record types and with differences
    464   between 32-bit and (forthcoming) 64-bit ObjC/Cocoa implementations.
     486The ObjC bridge provides a few new mechanisms for defining ObjC
     487methods, for calling ObjC "generic functions" (e.g., message sending),
     488and for dealing with frequently-used record types and with differences
     489between 32-bit and (forthcoming) 64-bit ObjC/Cocoa implementations.
    465490 
    466   A lot of macros/functions/other things that really should have been
    467   exported from some package for the last few years finally have been
    468   exported from the OBJC or NS packages (and a lot of things that have
    469   historically been internal to CCL are re-imported into CCL).
    470 
    471   Cocoa (and the underlying Core Graphics libraries) have historically
    472   used 32-bit floats and 32-bit integers in data structures that describe
    473   geometry, font sizes and metrics, and elsewhere.  64-bit Cocoa will
    474   use 64-bit floats and 64-bit integers in many cases.
    475 
    476   The bridge defines the type NS:CGFLOAT as the lisp type of the
    477   preferred float type on the platform, and the constant NS:+CGFLOAT+.
    478   On DarwinPPC32, the foreign types :cgfloat, :<NSUI>nteger, and
    479   :<NSI>nteger are defined by the bridge (as 32-bit float, 32-bit
    480   unsigned integer, and 32-bit signed integer, respectively.); these
    481   types are defined (as 64-bit variants) in the 64-bit interfaces.
    482 
    483   All ObjC classes are properly named, either with a name exported
    484   from the NS package (in the case of a predefined class declared in
    485   the interface files) or with the name provided in the DEFCLASS
    486   form (with :METACLASS NS:+NS-OBJECT) which defines the class from
    487   lisp.  The class's lisp name is now proclaimed to be a "static"
    488   variable (as if by DEFSTATIC, as described above) and given the
    489   class object as its value.  In other words:
     491A lot of macros/functions/other things that really should have been
     492exported from some package for the last few years finally have been
     493exported from the OBJC or NS packages (and a lot of things that have
     494historically been internal to CCL are re-imported into CCL).
     495
     496Cocoa (and the underlying Core Graphics libraries) have historically
     497used 32-bit floats and 32-bit integers in data structures that describe
     498geometry, font sizes and metrics, and elsewhere.  64-bit Cocoa will
     499use 64-bit floats and 64-bit integers in many cases.
     500
     501The bridge defines the type NS:CGFLOAT as the lisp type of the
     502preferred float type on the platform, and the constant NS:+CGFLOAT+.
     503On DarwinPPC32, the foreign types :cgfloat, :<NSUI>nteger, and
     504:<NSI>nteger are defined by the bridge (as 32-bit float, 32-bit
     505unsigned integer, and 32-bit signed integer, respectively.); these
     506types are defined (as 64-bit variants) in the 64-bit interfaces.
     507
     508All ObjC classes are properly named, either with a name exported
     509from the NS package (in the case of a predefined class declared in
     510the interface files) or with the name provided in the DEFCLASS
     511form (with :METACLASS NS:+NS-OBJECT) which defines the class from
     512lisp.  The class's lisp name is now proclaimed to be a "static"
     513variable (as if by DEFSTATIC, as described above) and given the
     514class object as its value.  In other words:
    490515
    491516{{{
     
    493518}}}
    494519
    495   and
     520and
    496521
    497522{{{
     
    499524}}}
    500525
    501   are equivalent.  (Since it's not legal to bind a "static" variable,
    502   it may be necessary to rename some things so that unrelated
    503   variables whose names coincidentally conflict with ObjC class names
    504   don't do so.)
     526are equivalent.  (Since it's not legal to bind a "static" variable,
     527it may be necessary to rename some things so that unrelated
     528variables whose names coincidentally conflict with ObjC class names
     529don't do so.)
    505530
    506531- A new reader macro - #/ - reads a sequence of "constituent" characters
    507   (including colons) from the stream on which it appears and interns
    508   that sequence - with case preserved and colons intact - in a new package
    509   whose name is NEXTSTEP-FUNCTIONS, exporting the symbol from that package.
    510   This means that the act of reading "#/alloc" returns the the symbol
    511   NEXTSTEP-FUNCTIONS:|alloc|, and the act of reading "#/initWithFrame:"
    512   returns the symbol NEXTSTEP-FUNCTIONS:|initWithFrame:|.  The intent
    513   is that the reader macro can be used to construct symbols whose names
    514   match ObjC message names; the reader macro tries to do some sanity
    515   checks (such as insisting that a name that contains at least one
    516   colon ends in a colon), but isn't totally rigourous about enforcing
    517   ObjC message name conventions.
    518 
    519   A symbol read using this macro can be used as an operand in
    520   most places where an ObjC message name can be used, such as
    521   in the (@SELECTOR ...) construct (which is now OBJC:@SELECTOR,
    522   btw.)
    523 
    524   Marco Baringer proposed the idea of using a reader macro to
    525   construct lisp symbols which matched ObjC message names.
     532(including colons) from the stream on which it appears and interns
     533that sequence - with case preserved and colons intact - in a new package
     534whose name is NEXTSTEP-FUNCTIONS, exporting the symbol from that package.
     535This means that the act of reading "#/alloc" returns the the symbol
     536NEXTSTEP-FUNCTIONS:|alloc|, and the act of reading "#/initWithFrame:"
     537returns the symbol NEXTSTEP-FUNCTIONS:|initWithFrame:|.  The intent
     538is that the reader macro can be used to construct symbols whose names
     539match ObjC message names; the reader macro tries to do some sanity
     540checks (such as insisting that a name that contains at least one
     541colon ends in a colon), but isn't totally rigourous about enforcing
     542ObjC message name conventions.
     543
     544A symbol read using this macro can be used as an operand in
     545most places where an ObjC message name can be used, such as
     546in the (@SELECTOR ...) construct (which is now OBJC:@SELECTOR,
     547btw.)
     548
     549Marco Baringer proposed the idea of using a reader macro to
     550construct lisp symbols which matched ObjC message names.
    526551
    527552- The act of interning a new symbol in the NEXTSTEP-FUNCTIONS
    528   package triggers an interface database lookup of Objc methods
    529   with the corresponding message name.  If any such information
    530   is found, a special type of dispatching function is created
    531   and initialized and the weird-looking symbol is given that
    532   dispatching function as its function definition.
    533 
    534   The dispatching knows how to call declared ObjC methods defined on
    535   the message.  In many cases, all methods have the same foreign type
    536   signature, and the dispatching function merely passes any arguments
    537   that it receives to a function that does an ObjC message send with
    538   the indicated foreign argument and return types.  In other cases,
    539   where different ObjC messages have different type signatures, the
    540   dispatching function tries to choose a function that handles the
    541   right type signature based on the class of the dispatching function's
    542   first argument.
    543 
    544   If new information about ObjC methods is introduced (e.g., by
    545   using additional interface files or as ObjC methods are defined
    546   from lisp), the dispatch function is reinitialized to recognize
    547   newly-introduced foreign type signatures.
    548 
    549   The argument and result coercion that the bridge has tradionally
    550   supported is supported by the new mechanism (e.g., :<BOOL> arguments
    551   can be specified as lisp booleans and :<BOOL> results are returned
    552   as lisp boolean values, and an argument value of NIL is coerced to
    553   a null pointer if the corresponding argument type is :ID.
    554 
    555   Some ObjC methods accept variable numbers of arguments; the
    556   foreign types of non-required arguments are determined by the
    557   lisp types of those arguments (e.g., integers are passed as
    558   integers, floats as floats, pointers as pointers, record types
    559   by reference.)
    560 
    561   Some examples:
     553package triggers an interface database lookup of Objc methods
     554with the corresponding message name.  If any such information
     555is found, a special type of dispatching function is created
     556and initialized and the weird-looking symbol is given that
     557dispatching function as its function definition.
     558
     559The dispatching function knows how to call declared ObjC methods defined on
     560the message.  In many cases, all methods have the same foreign type
     561signature, and the dispatching function merely passes any arguments
     562that it receives to a function that does an ObjC message send with
     563the indicated foreign argument and return types.  In other cases,
     564where different ObjC messages have different type signatures, the
     565dispatching function tries to choose a function that handles the
     566right type signature based on the class of the dispatching function's
     567first argument.
     568
     569If new information about ObjC methods is introduced (e.g., by
     570using additional interface files or as ObjC methods are defined
     571from lisp), the dispatch function is reinitialized to recognize
     572newly-introduced foreign type signatures.
     573
     574The argument and result coercion that the bridge has tradionally
     575supported is supported by the new mechanism (e.g., :<BOOL> arguments
     576can be specified as lisp booleans and :<BOOL> results are returned
     577as lisp boolean values, and an argument value of NIL is coerced to
     578a null pointer if the corresponding argument type is :ID.
     579
     580Some ObjC methods accept variable numbers of arguments; the
     581foreign types of non-required arguments are determined by the
     582lisp types of those arguments (e.g., integers are passed as
     583integers, floats as floats, pointers as pointers, record types
     584by reference.)
     585
     586Some examples:
    562587
    563588{{{
     
    579604}}}
    580605
    581   ObjC methods that "return" structures return them as gcable pointers
    582   when called via dispatch functions.  E.g., if "my-window" is an
    583   NS:NS-WINDOW instance, then
     606ObjC methods that "return" structures return them as gcable pointers
     607when called via dispatch functions.  E.g., if "my-window" is an
     608NS:NS-WINDOW instance, then
    584609
    585610{{{
     
    587612}}}
    588613
    589   will return a gcable pointer to a structure that describes that window's
    590   frame rectangle.  (The good news is that there's no need to use SLET
    591   or special structure-returning message send syntax; the bad news is
    592   that #_malloc, #_free, and the GC are all involved in the creation
    593   and eventual destruction of structure-typed return values.  Unless
    594   and until those factors negatively affect performance, the advantages
    595   seem to outweigh the disadvantages.)
     614will return a gcable pointer to a structure that describes that window's
     615frame rectangle.  (The good news is that there's no need to use SLET
     616or special structure-returning message send syntax; the bad news is
     617that #_malloc, #_free, and the GC are all involved in the creation
     618and eventual destruction of structure-typed return values.  Unless
     619and until those factors negatively affect performance, the advantages
     620seem to outweigh the disadvantages.)
    596621
    597622- Since foreign pointers are now (sometimes, somewhat) typed, it's
    598   possible to treat pointers to some foreign types as "instances of
    599   built-in classes."  Specifically, a pointer to an :<NSR>ect is
    600   recognized as an instance of the built-in class NS:NS-RECT, a
    601   pointer to an <NSS>ize is treated as an instance of NS:NS-SIZE,
    602   <NSP>oint is recognized as NS:NS-POINT, and <NSR>ange maps to
    603   NS:NS-RANGE.  (There are a few other more obscure structure
    604   types that get this treatment, and with a little more work the
    605   mechanism could be made extensible.)
    606 
    607   For each of these built-in classes:
    608 
    609   - a PRINT-OBJECT method is defined
    610 
    611   - a foreign type name derived from the class name (e.g., :NS-RECT
     623possible to treat pointers to some foreign types as "instances of
     624built-in classes."  Specifically, a pointer to an :<NSR>ect is
     625recognized as an instance of the built-in class NS:NS-RECT, a
     626pointer to an <NSS>ize is treated as an instance of NS:NS-SIZE,
     627<NSP>oint is recognized as NS:NS-POINT, and <NSR>ange maps to
     628NS:NS-RANGE.  (There are a few other more obscure structure
     629types that get this treatment, and with a little more work the
     630mechanism could be made extensible.)
     631
     632For each of these built-in classes:
     633
     634  * a PRINT-OBJECT method is defined
     635
     636  * a foreign type name derived from the class name (e.g., :NS-RECT
    612637    for NS:NS-RECT) is made an alias for the corresponding type
    613638    (so it's possible to say (RLET ((R :NS-RECT)) ...)).