source: trunk/ccl/release-notes.txt @ 6547

Last change on this file since 6547 was 6547, checked in by gb, 15 years ago

Update for 070512.

  • Property svn:eol-style set to native
  • Property svn:keywords set to Author Date Id Revision
File size: 92.8 KB
Line 
1OpenMCL 1.1-pre-070512
2- The FASL version changed (old FASL files won't work with this
3  lisp version), as did the version information which tries to
4  keep the kernel in sync with heap images.  Note that it's generally
5  a lot easier to recompile recent sources with recent images, e.g.,
6  trying to compile 070512 sources with an 070407 image is unlikely
7  to work without tricky bootstrapping.
8- Most of the changes in this release involve the calling sequence
9  used on x86-64.  In very general terms, some kinds of function-call
10  intensive code may see a significant performance boost, most code
11  should see a slight improvement, some code might see a (hopefully
12  very slight) degradation, and anything significantly slower than
13  previous releases should be reported as a bug.
14  It is -possible- that some of these changes may cause errors to
15  be reported differently (the function reported as the function
16  executing when the error ocurred might be different/wrong).  I
17  have not seen as many cases of this as I expected to when making
18  the change, but am also not sure that I fixed all possible cases.
19- The FFI-related reader macros #_, #$, and #& all read a case-sensitive
20  foreign function, constant, or variable name from the input stream
21  and try to find the corresponding definition in the interface files.
22  If the name is prefixed with a #\? - as in #_?foo - the macros
23  return true if the definition could be found and false otherwise.
24  (The general idea is that this might be useful for conditionalizing
25  code in some cases, and there should be -some- way of quietly testing
26  that something's defined.)
27- There is now support for making the contents of (possibly very large)
28  files accessible as lisp vectors.  (This may be many times faster
29  than something like
30
31  (let* ((stream (open pathname :direction :input :element-type 'whatever))
32         (vector (make-array (file-size-to-vector-size stream)
33                             :element-type 'whatever)))
34    (read-sequence vector stream))
35
36  but has the similar effect of making the contents of VECTOR match the
37  contents of the file.)
38
39  CCL:MAP-FILE-TO-IVECTOR pathname element-type [Function]
40
41  "element-type" should be a type specifier such that
42  (UPGRADED-ARRAY-ELEMENT-TYPE element-type) is a subtype
43  of either SIGNED-BYTE or UNSIGNED-BYTE.
44
45  Tries to open the file named by "pathname" for reading and to
46  map its contents into the process's address space via #_mmap;
47  if successful, returns a lisp vector of element-type
48  (UPGRADED-ARRAY-ELEMENT-TYPE element-type) which is displaced
49  to an underlying (SIMPLE-ARRAY element-type (*)) whose contents
50  match the mapped file's.
51
52  Because of alignment issues, the mapped file's contents will
53  start a few bytes (4 bytes on 32-bit platforms, 8 bytes on 64-bit
54  platforms) "into" the vector; the displaced array returned by
55  CCL:MAP-FILE-TO-IVECTOR hides this overhead, but its usually
56  more efficient to operate on the underlying simple 1-dimensional
57  array.  Given a displaced array (like the value returned by
58  CCL:MAP-FILE-TO-IVECTOR), the CL function ARRAY-DISPLACEMENT
59  returns the underlying array and the displacement index in elements.
60
61  Currently, only read-only file mapping is supported; the underlying
62  vector will be allocated in read-only memory, and attempts to use
63  (e.g.) (SETF (AREF ...) ...) to modify the mapped vector's contents
64  will result in memory faults.
65 
66  CCL:MAP-FILE-TO-OCTET-VECTOR pathname [Function]
67
68  Equivalent to (CCL:MAP-FILE-TO-IVECTOR pathname '(UNSIGNED-BYTE 8)).
69
70  CCL:UNMAP-IVECTOR displaced-vector
71
72  If the argument is a mapped vector (as returned by
73  MAP-FILE-TO-IVECTOR) that has not yet been "unmapped" by this
74  function, undoes the memory mapping, closes the mapped file, and
75  adjusts its argument so that it's displaced to a 0-length vector.
76
77  CCL:UNMAP-OCTET-VECTOR is an alias for CCL:UNMAP-IVECTOR
78
79  Note that whether a vector's created by MAKE-ARRAY or by mapping
80  a file's contents, it can't have ARRAY-TOTAL-SIZE-LIMIT or more
81  elements.  (ARRAY-TOTAL-SIZE-LIMIT is (EXPT 2 24) in 32-bit OpenMCL
82  and (EXPT 2 56) in 64-bit versions.
83
84- The lisp kernel now tries to signal memory faults that occur when
85  running lisp code as lisp errors.  As a silly example:
86
87  ? (defun foo (x)
88     "Crash and burn if X is not a list"
89     (declare (optimize (speed 3) (safety 0)) (list x))
90     (car x))
91  FOO
92  ? (foo 0)
93  > Error: Fault during read of memory address #x4
94  > While executing: FOO, in process listener(1).
95
96  The fact that things are handled this way (rather than going
97  into the kernel debugger with no easy way of recovering) makes
98  it possible to continue a session without losing work in many
99  cases.  In a trivial example like the one above, it's relatively
100  easy to see that no harm has been done and the error should
101  not be hard to recover from.  In some other cases, it may be
102  true that a buggy function has been scribbling ofer memory for
103  a while before that scribbling resulted in a machine exception.
104
105  Moral: if you get an unexpected "memory fault" error (the
106  condition type is actually CCL::INVALID-MEMORY-ACCESS) and
107  don't understand why the fault occurred and the consequences
108  of continuing in the lisp session where the fault occurred,
109  you should view the state of that session with some suspicion.
110
111  Faults in foreign code (should) still trap into the kernel
112  debugger.  (It'd be nice to be able to treat these as lisp
113  errors with the same caveats as described above, but that
114  is more complicated in some cases and isn't yet implemented.)
115
116- An obscure kernel debugger command - (A), which tries to advance
117  the program counter by one instruction - is now disabled on x86-64.
118  (On the PPC, "one instruction" always meant "4 bytes"; implementing
119  this correctly on x86-64 would require the ability to at least
120  partially disassemble arbitrary x86-64 instructions.)
121
122  On the other hand, the kernel debugger should be able to show
123  FPU registers on x86-64.
124
125
126OpenMCL 1.1-pre-070408
127- The FASL version changed (old FASL files won't work with this
128  lisp version), as did the version information which tries to
129  keep the kernel in sync with heap images.  Note that it's generally
130  a lot easier to recompile recent sources with recent images, e.g.,
131  trying to compile 070408 sources with an 070214 image is unlikely
132  to work without tricky bootstrapping.
133- There's now a Trac bug-tracking/wiki site for OpenMCL at
134  <http://trac.clozure.com/openmcl>.  It needs bug reports; please
135  visit that site and use the features there to report any bugs
136  that you find.
137- DEFSTATIC (aka DEFGLOBAL)
138  (CCL:DEFSTATIC var value &optional doc-string)
139  is like DEFPARAMETER in that it proclaims the variable "var" to
140  be special, sets its value to "value", and sets the symbol's
141  VARIABLE documentation to the optional doc-string.  It differs
142  from DEFPARAMETER in that it further asserts that the variable
143  should never be bound dynamically in any thread (via LET/LAMBDA/etc.);
144  the compiler treats any attempts to bind a "static" variable as an
145  error.
146  It is legal to change the value of a "static" variable, but since
147  all threads see the same (static) binding of that variable it may
148  be necessary to synchronize assignments made from multiple threads.
149  (A "static" variable binding is effectively a shared, global resource;
150  a dynamic binding is thread-private.)
151  Access to the value of a static variable is typically faster than
152  is access to the value a special variable that's not proclaimed to
153  be "static".
154  This functionality has been in MCL/OpenMCL for a long time under
155  the name CCL:DEFGLOBAL; CCL:DEFGLOBAL is an alias for CCL:DEFSTATIC,
156  but the latter seemed to be a better name.
157- The type of foreign object that a MACPTR points to can now be
158  asserted (this means that a MACPTR object can contain a small
159  integer which identifies the alleged FOREIGN-TYPE of the object that
160  the points to.  RLET, MAKE-RECORD, and MAKE-GCABLE-RECORD (see below)
161  assert the foreign type of the object that the MACPTR object they
162  create (as do some new features of the ObjC bridge, described further
163  below.)
164  PRINT-OBJECT on a MACPTR will try to print information about the
165  asserted type of that pointer, as well as information about where
166  the pointer was allocated (heap, stack) and whether it's scheduled
167  for automatic reclamation by the GC.
168  A few constructs that conceivable should assert the type of the
169  pointers they create (e.g., some flavor of PREF, SLOT-VALUE in
170  the ObjC bridge) don't yet do so.
171  A rather obvious way of exploiting typed pointers - namely, extending
172  DESCRIBE and INSPECT to show the contents of foreign records - is
173  not yet implemented.
174- MAKE-GCABLE-RECORD is like MAKE-RECORD, in that it "makes an instance
175  of a foreign record type".  (Or, to be more banal about it, uses
176  #_malloc to allocate a block of foreign memory of the size of the
177  foreign record type named by its argument.)  MAKE-GCABLE-RECORD
178  additionally tells the lisp garbage collector that it should free
179  the foreign memory when the MACPTR object that describes it becomes
180  garbage.
181  When using "gcable pointers", it's important to remember the
182  distinction between a MACPTR object (which is a lisp object, more-
183  or-less like any other) and the block of foreign memory that the
184  MACPTR object points to.  If a gcable MACPTR is the only thing
185  in the world ("lisp world" or "foreign world") that references
186  the underlying block of foreign memory, then freeing the foreign
187  memory when it becomes impossible to reference it is convenient
188  and sane.  If other lisp MACPTRs reference the underlying block
189  of foreign memory or if the address of that foreign memory is
190  passed to and retained by foreign code, having the GC free the
191  memory may have unpleasant consequences if those other references
192  are used.
193- CCL:FREE (which is mostly just a wrapper around #_free that allows
194  #_free to be called early in the bootstrapping process) is now
195  exported; if its argument is a gcable pointer (e.g., created via
196  MAKE-GCABLE-POINTER), it will tell the GC that the pointer's
197  foreign memory has been freed "manually" before calling #_free.
198- The mechanisms used to implement locks has changed (for the curious,
199  the changes involve the use of spinlocks rather than a sequence
200  of atomic additions.)  Someone demonstrated a case of deadlock
201  (where a thread was waiting for a lock that was available) under
202  the old implementation.  I'm not sure that I fully understand how
203  that could have happened, but the new implementation at least has
204  the advantage of being a little easier to understand and might be
205  a tiny bit faster.  Please let me know if either of these assumptions
206  was incorrect.
207- An EOF (control-d) in the REPL (when standard input is a tty or pty
208  device) has traditionally caused an exit to the outer break loop
209  (or had no effect if the REPL was not in a break loop).  If
210  CCL:*QUIT-ON-EOF* is set, an EOF causes the lisp to quit.  (It
211  actually invokes a listener-specific method, so in a multi-listener
212  window system environemt, it might simply cause the listener which
213  receives the EOF to exit.)
214  None of this has any effect when running under environments like
215  SLIME, and (as mentioned above) only matters if the standard input
216  devices is a tty or pseudo-tty (where it's possible to continue
217  reading after an EOF has been read.)  If running under an xterm
218  or OSX Terminal.app, standard input is probably a pty; if running
219  in an Emacs shell buffer or under other means under emacs, different
220  types of IPC mechanisms (pipes, sockets) might be used.
221- SAVE-APPLICATION has historically changed the type of all MACPTR
222  objects (turning them into "dead macptrs", since it's generally
223  meaningless to refer to a foreign pointer from a previous session
224  and generally better to get a type error than some more mysterious
225  or severe failure).  This no longer happens for null pointers (pointers
226  to address 0); COMPILE-FILE also now allows null pointers to be referenced
227  as constants in compiled code.
228- Not entirely coincidentally, CCL:+NULL-PTR+ is now defined as a constant
229  (whose value is a null pointer.)  In some cases, it may be more
230  efficient or convenient to pass CCL:+NULL-PTR+ to foreign code than
231  it would be to call (CCL:%NULL-PTR) to "produce" one.
232- Historically, OpenMCL (and MCL) have maintained a list of open file
233  streams in the value of CCL:*OPEN-FILE-STREAMS*; maintaining this
234  list helps to ensure that streams get closed in as orderly a manner
235  as possible when the lisp exits.  The code which accessed this list
236  didn't do so in a thread-safe manner.
237  The list is now maintained in a lexical variable; the function
238  CCL:OPEN-FILE-STREAMS returns a copy of that list,
239  CCL:NOTE-OPEN-FILE-STREAM adds its argument (a file stream) to the
240  list, and CCL:REMOVE-OPEN-FILE-STREAM removes its argument (a file stream)
241  from the list.  (All of these functions use appropriate locking.)
242- There were a number of timing-related problems related to PROCESS-INTERRUPT
243  (usually involving rapidly and repeatedly interrupting a thread over
244  a long period of time.)  This should be a lot more reliable now
245  (exactly what could go wrong and why and how is all a little hard to
246  describe.)
247- Some Linux distributions may initialize the user's environment in
248  a way that imposes a soft limit on the amount of virtual memory that
249  a process is allowed to map.  OpenMCL now tries to raise this limit
250  before reserving what may be a very large amount of address space,
251  thanks to a patch from Andi Kleen.
252- There were a number of problems with UTF-16 streams, found and
253  fixed by Takehiko Abe.
254- Takehiko Abe also provided fixes for some code in "ccl:lib;xref.lisp"
255  and in source-file recording/reporting that (still) didn't understand
256  the concept of EQL-SPECIALIZER metaobjects.
257- ObjC bridge and ObjC examples
258  The ObjC bridge provides a few new mechanisms for defining ObjC
259  methods, for calling ObjC "generic functions" (e.g., message sending),
260  and for dealing with frequently-used record types and with differences
261  between 32-bit and (forthcoming) 64-bit ObjC/Cocoa implementations.
262 
263  A lot of macros/functions/other things that really should have been
264  exported from some package for the last few years finally have been
265  exported from the OBJC or NS packages (and a lot of things that have
266  historically been internal to CCL are re-imported into CCL).
267
268  Cocoa (and the underlying Core Graphics libraries) have historically
269  used 32-bit floats and 32-bit integers in data structures that describe
270  geometry, font sizes and metrics, and elsewhere.  64-bit Cocoa will
271  use 64-bit floats and 64-bit integers in many cases.
272
273  The bridge defines the type NS:CGFLOAT as the lisp type of the
274  preferred float type on the platform, and the constant NS:+CGFLOAT+.
275  On DarwinPPC32, the foreign types :cgfloat, :<NSUI>nteger, and
276  :<NSI>nteger are defined by the bridge (as 32-bit float, 32-bit
277  unsigned integer, and 32-bit signed integer, respectively.); these
278  types are defined (as 64-bit variants) in the 64-bit interfaces.
279
280  All ObjC classes are properly named, either with a name exported
281  from the NS package (in the case of a predefined class declared in
282  the interface files) or with the name provided in the DEFCLASS
283  form (with :METACLASS NS:+NS-OBJECT) which defines the class from
284  lisp.  The class's lisp name is now proclaimed to be a "static"
285  variable (as if by DEFSTATIC, as described above) and given the
286  class object as its value.  In other words:
287
288(send (find-class 'ns:ns-application) 'shared-application)
289
290  and
291
292(send ns:ns-application 'shared-application)
293
294  are equivalent.  (Since it's not legal to bind a "static" variable,
295  it may be necessary to rename some things so that unrelated
296  variables whose names coincidentally conflict with ObjC class names
297  don't do so.)
298
299- A new reader macro - #/ - reads a sequence of "constituent" characters
300  (including colons) from the stream on which it appears and interns
301  that sequence - with case preserved and colons intact - in a new package
302  whose name is NEXTSTEP-FUNCTIONS, exporting the symbol from that package.
303  This means that the act of reading "#/alloc" returns the the symbol
304  NEXTSTEP-FUNCTIONS:|alloc|, and the act of reading "#/initWithFrame:"
305  returns the symbol NEXTSTEP-FUNCTIONS:|initWithFrame:|.  The intent
306  is that the reader macro can be used to construct symbols whose names
307  match ObjC message names; the reader macro tries to do some sanity
308  checks (such as insisting that a name that contains at least one
309  colon ends in a colon), but isn't totally rigourous about enforcing
310  ObjC message name conventions.
311
312  A symbol read using this macro can be used as an operand in
313  most places where an ObjC message name can be used, such as
314  in the (@SELECTOR ...) construct (which is now OBJC:@SELECTOR,
315  btw.)
316
317  Marco Baringer proposed the idea of using a reader macro to
318  construct lisp symbols which matched ObjC message names.
319
320- The act of interning a new symbol in the NEXTSTEP-FUNCTIONS
321  package triggers an interface database lookup of Objc methods
322  with the corresponding message name.  If any such information
323  is found, a special type of dispatching function is created
324  and initialized and the weird-looking symbol is given that
325  dispatching function as its function definition.
326
327  The dispatching knows how to call declared ObjC methods defined on
328  the message.  In many cases, all methods have the same foreign type
329  signature, and the dispatching function merely passes any arguments
330  that it receives to a function that does an ObjC message send with
331  the indicated foreign argument and return types.  In other cases,
332  where different ObjC messages have different type signatures, the
333  dispatching function tries to choose a function that handles the
334  right type signature based on the class of the dispatching function's
335  first argument.
336
337  If new information about ObjC methods is introduced (e.g., by
338  using additional interface files or as ObjC methods are defined
339  from lisp), the dispatch function is reinitialized to recognize
340  newly-introduced foreign type signatures.
341
342  The argument and result coercion that the bridge has tradionally
343  supported is supported by the new mechanism (e.g., :<BOOL> arguments
344  can be specified as lisp booleans and :<BOOL> results are returned
345  as lisp boolean values, and an argument value of NIL is coerced to
346  a null pointer if the corresponding argument type is :ID.
347
348  Some ObjC methods accept variable numbers of arguments; the
349  foreign types of non-required arguments are determined by the
350  lisp types of those arguments (e.g., integers are passed as
351  integers, floats as floats, pointers as pointers, record types
352  by reference.)
353
354  Some examples:
355
356;;; #/alloc is a known message.
357? #'#/alloc
358#<OBJC-DISPATCH-FUNCTION NEXTSTEP-FUNCTIONS:|alloc| #x300040E94EBF>
359;;; Sadly, #/foo is not ...
360? #'#/foo
361> Error: Undefined function: NEXTSTEP-FUNCTIONS:|foo|
362
363;;; We can send an "init" message to a newly-allocated instance of
364;;; "NSObject" by:
365
366(send (send ns:ns-object 'alloc) 'init)
367
368;;; or by
369
370(#/init (#/alloc ns:ns-object))
371
372  ObjC methods that "return" structures return them as gcable pointers
373  when called via dispatch functions.  E.g., if "my-window" is an
374  NS:NS-WINDOW instance, then
375
376(#/frame my-window)
377
378  will return a gcable pointer to a structure that describes that window's
379  frame rectangle.  (The good news is that there's no need to use SLET
380  or special structure-returning message send syntax; the bad news is
381  that #_malloc, #_free, and the GC are all involved in the creation
382  and eventual destruction of structure-typed return values.  Unless
383  and until those factors negatively affect performance, the advantages
384  seem to outweigh the disadvantages.)
385
386- Since foreign pointers are now (sometimes, somewhat) typed, it's
387  possible to treat pointers to some foreign types as "instances of
388  built-in classes."  Specifically, a pointer to an :<NSR>ect is
389  recognized as an instance of the built-in class NS:NS-RECT, a
390  pointer to an <NSS>ize is treated as an instance of NS:NS-SIZE,
391  <NSP>oint is recognized as NS:NS-POINT, and <NSR>ange maps to
392  NS:NS-RANGE.  (There are a few other more obscure structure
393  types that get this treatment, and with a little more work the
394  mechanism could be made extensible.)
395
396  For each of these built-in classes:
397
398  - a PRINT-OBJECT method is defined
399
400  - a foreign type name derived from the class name (e.g., :NS-RECT
401    for NS:NS-RECT) is made an alias for the corresponding type
402    (so it's possible to say (RLET ((R :NS-RECT)) ...)).
403
404  - the class is is integrated into the type system (so that
405    (TYPEP R 'NS:NS-RECT) is fairly efficently implemented.)
406
407  - inlined accessor and setf inverses are defined for the structure
408    type's fields.  In the case of an :<NSR>ect, the fields in question
409    are the fields of the embedded point and size, so NS:NS-RECT-X,
410    NS:NS-RECT-Y, NS:NS-RECT-WIDTH, NS-RECT-HEIGHT and SETF inverses
411    are defined.  The accessors and setter functions typecheck their
412    arguments and the setters handle coercion to the approprate type
413    of CGFLOAT where applicable.
414
415  - an initialization function is defined; (NS:INIT-NS-SIZE s w h) is
416    roughly equivalent to (SETF (NS:NS-SIZE-WIDTH s) w
417    (NS:NS-SIZE-HEIGHT s) h), but might be a little more efficient.
418
419  - a creation function is defined: (NS:NS-MAKE-POINT x y) is basically
420    equivalent to:
421    (LET ((P (MAKE-GCABLE-RECORD :NS-POINT)))
422      (NS:INIT-NS-POINT P X Y)
423      p)
424
425  - a macro is defined which (much like RLET) stack-allocates an
426    instance of the foreign record type, optionally iniitializes
427    that instance, and executes a body of code with a variable
428    bound to that instance.  E.g.
429
430    (ns:with-ns-range (r loc len)
431      (format t "~& range has location ~s, length ~s"
432         (ns:ns-range-location r) (ns:ns-range-length r)))
433
434    which is probably not the world's most realistic example.
435
436   Note that it's possible to construct a record
437   instance that has a very short useful lifetime:
438
439   (#/initWithFrame: new-view (ns:ns-make-rect 100 100 200 200))
440
441   The rectangle above will -eventually- get reclaimed by the GC;
442   if you don't want to give the GC so much work to do, you might
443   prefer to do:
444
445   (ns:with-ns-rect (r 100 100 200 200)
446     (#/initWithFrame: new-view r))
447
448
449 - The macro OBJC:DEFMETHOD can be used to define ObjC methods.
450   It looks superficially like CL:DEFMETHOD in some respects.
451   The syntax is:
452
453   (OBC:DEFMETHOD name-and-result-type ((receiver-arg-and-class) &rest other-args) &body body)
454
455   where:
456
457   "name-and-result-type" is either an ObjC message name (use #/ !)
458   for methods that return a value of type :ID, or a list of an ObjC
459   message name and a foreign type specifier for methods with a different
460   foreign result type
461
462   "receiver-type-and-class" is a two-element list whose CAR is
463   a variable name and whose CADR is the lisp name of an ObjC class
464   or metaclass.  The receiver variable name can be any bindable
465   lisp variable name, but SELF (in some package) might be a reasonable
466   choice.  The receiver variable is declared to be "unsettable", i.e.,
467   it is an error to try to change the value of the receiver in the
468   body of the method definition.
469
470   "other-args" are either variable names (denoting parameters of type
471   :ID) or 2-element lists whose first element is a variable name and
472    whose second element is a foreign type specifier.
473
474  For example:
475
476(objc:defmethod (#/characterAtIndex: :unichar)
477    ((self hemlock-buffer-string) (index :<NSUI>nteger))
478  ...)
479 
480  The method "characterAtIndex:", when invoked on an object of class
481  HEMLOCK-BUFFER-STRING with an additional argument of type :<NSU>integer
482  returns a value of type :unichar.)
483
484  Arguments that wind up as some non-:ID pointer type (pointers,
485  records passed by value) are represented as typed foreign pointers
486  (so the higher-level, type-checking accessors can be used on
487  arguments of type :ns-rect, :ns-pointe, etc.)
488
489  Within the body of methods defined via OBJC:DEFMETHOD, the local
490  function CL:CALL-NEXT-METHOD is defined.  It isn't quite as
491  general as CL:CALL-NEXT-METHOD is when used in a CLOS method,
492  but it has some of the same semantics.  It accepts as many arguments
493  as are present in the containing method's "other args" list and
494  invokes version of the containing method that would have been
495  invoked on instances of the receiver's class's superclass with
496  the receiver and other provided arguments.  (The idiom of passing
497  the current method's arguments to the next method is common enough
498  that the CALL-NEXT-METHOD in OBJC:DEFMETHODs should probably do
499  this if it receives no arguments.)
500
501  A method defined via OBJC:DEFMETHOD that returns a structure "by value"
502  can do so by returning a record created via MAKE-GCABLE-RECORD, by
503  returning the value returned via CALL-NEXT-METHOD, or by other similar
504  means.  Behind the scenes, there may be a pre-allocated instance of
505  the record type (used to support native structure-return conventions),
506  and any value returned by the method body will be copied to this
507  internal record instance.  Within the body of a method defined with
508  OBJC:DEFMETHOD that's declared to return a structure type, the local
509  macro OBJC:RETURNING-FOREIGN-STRUCT can be used to access the internal
510  structure:
511
512  (objc:defmethod (#/reallyTinyRectangleAtPoint: :ns-rect)
513    ((self really-tiny-view) (where :ns-point))
514    (objc:returning-foreign-struct (r)
515      (ns:init-ns-rect r (ns:ns-point-x where) (ns:ns-point-y where)
516                          single-float-epsilon single-float-epsilon)
517      r))
518
519 - If OBJC:DEFMETHOD introduces a new ObjC message, a ... message
520   to that effect.  Sometimes, these messages are merely informative
521   (and barely informative, at that ...), but they may also indicate
522   that a message name is misspelled (or possibly missing a trailing
523   colon.)  If a method is redefined in such a way that it's type
524   signature changes, a continuable error is signaled.
525
526 - there used to be some fairly obscure reasons that led to
527   MAKE-OBJC-INSTANCE being a bit more efficient than MAKE-INSTANCE
528   in some cases (some of the methods invoked by MAKE-INSTANCE did
529   some extra work to handle Lisp slots even if the class didn't
530   define any Lisp slots.  This work isn't done anymore, and consequently
531   there's less reason to prefer MAKE-OBJC-INSTANCE.  (MAKE-OBJC-INSTANCE
532   is still defined and exported from the OBJC:PACKAGE).
533
534 - the preferred means of loading an add-on framework and processing
535   the declarations in its interfaces has changed several times over
536   the last several months.  The currently preferred (new) way to
537   do that is via the new function OBJC:LOAD-FRAMEWORK
538
539   (OBJC:LOAD-FRAMEWORK framework-name interface-dir)
540
541   where "framework-name" is a string which names the framework and
542   "interface-dir" is a keyword that names the associated set of
543   interfaces.  OBJC:LOAD-FRAMEWORK should find and initialize the
544   framework bundle (looking in standard framework search paths),
545   introduce new ObjC classes to CLOS, update information about
546   declared messages and their methods' type signatures, adjust
547   affected dispatch functions, and make the interfaces other
548   definitions available.  The order in which it does these
549   things isn't specified, and may change in the future.
550
551 - Most Cocoa-related examples (the demo IDE, the Rubix and Webkit
552   examples) have been rewritten to use the new bridge features.
553   (I may have missed some contributed examples; if people want
554   to convert these, that'd be great.)  It's too early to say
555   whether the new approach is better or worse than the old, but
556   I've (so far) found some of the code easier to read and maintain.
557   We might find that some things that (for instance) SEND does
558   more efficiently could and should be done via SEND (I'm thinking
559   mostly of struct-return stuff), but so far I haven't seen the
560   new stuff keel over.
561
562   The converted code looks like "lisp code with strange-looking
563   function names" at first glance, and that seems about right.
564   The function names might get to look more familiar as the
565   reader becomes more familiar with Cocoa; as someone here pointed
566   out, it's arguably good that the function names are distinctive
567   in that that helps to remind the reader that these are likely
568   lower-level functions that are less tolerant of type- and other
569   errors than the typical lisp function would be.
570
571
572
573OpenMCL 1.1-pre-070214
574- The FASL version changed (old FASL files won't work with this
575  lisp version), as did the version information which tries to
576  keep the kernel in sync with heap images.
577- There are new interface files for all platforms.  These files
578  encode some foreign type information a little differently
579  than older ones did (notably information about foreign functions
580  that return structures or accept structure args by value.)  The
581  new .cdb files can't be used by older versions of OpenMCL; using
582  older .cdb files with this version is "allowed, but not supported
583  or recommended."
584- Almost all of the changes in functionality since the last (061231)
585  snapshots and since the CVS freeze on 070117 have to do with
586  relatively obscure issues having to do with passing structures
587  to foreign functions by value and/or returning structures from foreign
588  function calls.
589
590  These idioms are fairly rare in traditional C code (though it's
591  fairly common to pass -pointers- to structures by reference
592  and sometimes to return pointers to structures.  There are
593  a few C compiler runtime routines that perform some flavor
594  of integer division and return a two-element structure that
595  contains "quotient" and "remainder" fields, but that's typically
596  about the extent of the use of this idiom.)  The idioms are used
597  much more often in Apple's Carbon and Cooca libraries and in
598  some of the frameworks (CoreGraphics, CoreFoundation) that those
599  libraries are based on.
600
601  OpenMCL's FFI has provided some support for this in the past;
602  notably, it's provided support for (most of the) structure-returning
603  and struct-by-value conventions used on 32-bit PPC Darwin.  In these
604  conventions, a foreign function that returned a structure received
605  a pointer to an instance of that structure type as a first argument,
606  and a function that received a structure argument by value received
607  the structure's contents in 32-bit word-size integer chunks (regardless
608  of the types or sizes of the structure's fields.)  Knowledge of these
609  conventions was hardwired into various parts of the system (e.g.,
610  the interface database), so that it was not generally possible to
611  tell whether a given foreign function returned a structure type
612  (or just happened to take an extra pointer argument.)
613
614  Unfortunately, there are at least 4 other sets of conventions for
615  dealing with structure arguments/return values on the platforms
616  that OpenMCL runs on (and even the DarwinPPC32 conventions weren't
617  fully/correctly implemented.)  OpenMCL's FFI is generally pretty
618  low-level, but to the extent that it's reasonable to talk about
619  "higher level" constructs (EXTERNAL-CALL, SEND, FF-CALL, #_), those
620  higher-level constructs try to enforce uniform syntax and try
621  to hide the platform-specific details in backend-specific functions.
622
623  The impact of these changes should generally be pretty minimal.
624  In a "higher-level" construct used to call a foreign function that
625  returns a structure type, the first parameter in the call should
626  be a pointer to an instance of that structure type.
627
628  For example, if a :rect structure is defined as:
629
630  (def-foreign-type nil
631    (:struct :rect
632      (:width :int)
633      (:height :int)
634      (:x :int)  ; x coordinate of origin
635      (:y :int)))
636
637  and a foreign function named "inset_rect" takes a rect and an integer
638  delta and returns a new :rect "inset" by that delta, a call to that
639  foreign function might look like:
640
641  (rlet ((result :rect))
642    (ff-call *address-of-inset-rect* result (:struct :rect) r :int delta :(:struct rect))
643    ;; or, if "inset_rect" was declared in the interface database:
644    (#_inset_rect result r delta))
645
646
647  A callback that returns a :rect likewise should accept a pointer
648  to an instance of the :rect type as a first (unqualified) argument
649  and explicitly declare that it returns a (:STRUCT :RECT).
650
651  (defcallback *address-of-inset-rect (result (:struct :rect) r :int delta (:struct :rect))
652    (setf (pref result :rect.x) (+ (pref r :rect.x) delta)
653          (pref result :rect.y) (+ (pref r :rect.y) delta)
654          (pref result :rect.width) (- (pref r :rect.width) (* 2 delta))
655          (pref result :rect.height) (- (pref r :rect.height) (* 2 delta))))
656
657  Note that this is very similar to what's been (implicitly) supported
658  on DarwinPPC32; the basic difference is that the return type
659  ("(:STRUCT :RECT)") is explicitly specified (or, in the case of #_,
660  specified in the interface database).  Whether the "result" pointer
661  is actually passed as an argument or not is platform-dependent (on
662  DarwinPPC64, the :rect structure would be "returned" by returning
663  4 :int values in 4 different machine registers), but the same syntax
664  can be used (and hides those details) on all platforms.
665
666  In the examples above, we said that the (presumed source) rectangle
667  was passed by value as a value of type (:struct :rect), and we let
668  the FFI deal with the details.  Historically, this parameter could
669  have been specified as a small unsigned integer N (denoting the
670  DarwinPPC32 convention of passing the structure value a N
671  native-word-size integer arguments.)  Again, there are several
672  different conventions for passing and receiving structure values,
673  and it's best to let the FFI decide how to follow those conventions.
674  (Some of those conventions are quite complicated, and depend on
675  the size of the structure as well as the types of its fields.)
676
677  In all cases, a callback which declares a parameter to be of a
678  structure type can treat that parameter as a pointer an instance of
679  that structure type with fields initialized by the caller (as in
680  the case of "r" in the example above.)
681
682  In the ObjC bridge, the DEFINE-OBJC-METHOD macro has always provided
683  syntax for specifiying that the method "returns" a structure. (That
684  syntax is (:struct <struct-type> <parameter-name>). That continues
685  to be supported.
686
687  Apple's ObjC runtime provides different functions (#_objc_msgSend and
688  #_objc_msgSend_stret) to handle the cases of sending messages which
689  return non-structure and structure results.  These low-level functions
690  are very sensitive to whether the structure is actually returned via
691  an "invisible" first argument or not (this is only one of a few different
692  conventions on some platforms.)  OpenMCL's ObjC bridge makes similar
693  distinctions, but uses simple, consistent rules: a message that returns
694  a structure should always be sent via SEND/STRET (or some variant of
695  SEND/STRET) and should have a first parameter of type "pointer to
696  returned structure type", regardless of whether or not that pointer
697  is actually passed to the method implementation or just used as by
698  some platform-specific code to transfer register values.)
699
700  The end result of all of this (several weeks of bootstrapping) is
701  that most things are pretty much the same, at least on DarwinPPC32;
702  only foreign function calls/callbacks that involve passing structures
703  by value or returning structures need change at all, and the changes
704  generally involve being more explicit/declarative about what's going
705  on.  These changes -do- allow these idioms to be used on other
706  (64-bit) platforms, and since they're heavily used in Apple GUI
707  libraries and since 64-bit versions of Carbon and Cocoa are announced
708  features of Leopard, it seemed appropriate to get support for this
709  stuff into the FFI on those platforms and to try to do it in a way
710  that hid the platform-dependent details.  (I didn't expect all of
711  this to take so long.)
712
713- The initial listener PROCESS now persists across SAVE-APPLICATION.
714  This means that (for instance):
715
716  ? (defvar *listener-process* (current-process))
717  *LISTENER-PROCESS*
718  ? (save-application "new.image")
719  shell> openmcl new.image
720  ? (eq (current-process) *listener-process*)
721  T
722  ;; though of course the underlying OS thread, stacks, etc are unlikely
723  ;; to be "equal" in any sense.
724
725  The current process is sometimes used to mark "ownership" of thread-private
726  hash-tables and streams.  (Even though it doesn't make much sense for
727  STREAMs to persist across SAVE-APPLICATION, it does make sense for
728  HASH-TABLEs to do so; HASH-TABLES created with the :PRIVATE T option
729  and "owned" by the initial listener process continue to be owned by
730  that the current listener process in the new image.)
731
732- All of the FFI changes above do seem to allow the Cocoa IDE example
733  to run on ppc64/x86-64 (as well as ppc32) under Leopard, and
734  hopefully that'll soon be true of applications generated via Mikel
735  Evins' Bosco system as well.  The bridge and demo code have been
736  conditionalized to support ObjC 2.0 on 64-bit systems, to avoid
737  deprecated functions and methods, and to support 64-bit Cocoa
738  changes.  Hopefully, this has been done in a way that doesn't break
739  PPC32 Cocoa under Tiger (he said, quickly rushing to the nearest
740  PPC32 Tiger machine and breathing a sigh of relief when the Cocoa
741  listener appeared ..)  64-bit Cocoa sometimes used 64-bit signed and
742  unsigned integers in place of 32-bit integers; accordingly, the
743  foreign types :<NSI>nteger and :<NSUI>nteger are defined (as 32-bit
744  signed/unsigned integers) on 32-bit platforms, and these types are
745  used in some method and type definitions.  (Those integer types are
746  predefined in Objc 2.0, and are 64 bits wide on 64-bit platforms.)
747
748  More pervasively (and a little more problematically), CoreGraphics
749  (and things built on top of it, including Cocoa) uses double-floats
750  instead of single-floats for many things on 64-bit hardware; the
751  difference is abstracted (a little) via the new CGFloat type.
752  This means that (for instance) code which initializes a constant-sized
753  NSRect on a 32-bit machines and has traditionally done so via
754  something like:
755
756  (ns-make-rect 0.0 0.0 500.0 200.0)
757
758  now needs to do something like:
759
760  (ns-make-rect (float 0.0 ccl::+cgfloat-zero+) ..)
761
762  in order to compile and run on both 32-bit and 64-bit platforms.
763
764  where ccl::+cgfloat-zero+ is defined as 1.0f0 on 32-bit platforms
765  and as 1.0d0 on 64-bit machines.  Cases involving constants won't
766  incur any runtime overhead and the occasional runtime overhead in
767  other cases -probably- isn't that great in context (compared to
768  initializing a view hierarchy ...)  but it's certainly ugly to
769  look at.  It's possible that some of this ugliness could be
770  hidden in the bridge/FFI (by making them do the necessary coercions
771  for you), but there are tradeoffs there.
772
773- The ObjC bridge has had a long-standing bug whereby a standalone
774  Cocoa application may have needed to find the interface databases
775  at runtime in order for MAKE-OBJC-INSTANCE and MAKE-INSTANCE of
776  an ObjC class to work.  (These functions needed to be able to
777  send an "init" message to the newly-allocated instance, and needed
778  to know the type signature of that init message in order to do that.)
779  The current scheme tries to avoid this by pre-compiling helper
780  functions to enable calling all known "init" message signatures.
781  (More accurately, all fixed-argument "init" message signatures.)
782  This scheme avoids the need to send messages whose argument
783  and result types are computed at runtime (via %SEND), and %SEND
784  (a) was known to be inefficient and (b) would have a lot of
785  difficulty handling all known structure return/passing conventions
786  on supported platforms.  Accordingly, %SEND has been deprecated
787  (with extreme prejudice, e.g., removed.)
788
789- a couple of little functions are defined (but their names are
790  not yet exported) on x86-64: ccl::rdtsc and ccl::rdtsc64 provide
791  access to the values returned by on-chip cycle counting instructions.
792  For instance:
793
794? (let* ((start (ccl::rdtsc)))
795    (sleep 1)
796    (- (ccl::rdtsc) start))
7971995065244
798
799  Hmm.  Apparently, the 2.0GHz MacBook I tried that on is actually
800  a 1.995GHz MacBook.
801
802  There are all kinds of ways for rdtsc to lose (and return
803  inaccurate or misleading results): the cycle counters for
804  each CPU core in a multi-core system aren't necessarily
805  kept in sync, and many modern systems allow CPU clock rates
806  to vary (for power-management reasons) and/or allow the CPU
807  to sleep/hibernate.  OSes seem to offer some support for
808  compensating for these effects, and it seems like ccl::rdtsc
809  and ccl::rdtsc64 can be used to obtain interesting results.
810
811  The RDTSC instruction actually returns an unsigned 64-bit
812  result; apparently, some Intel documentation claims that this
813  value will not "wrap around" to 0 at contemporary clock rates
814  for at least 10 years after the system was booted.  (If you can
815  keep an Intel system running for 9 years between reboots, you
816  might consider telling Intel that the RDTSC counter wrapped around
817  a year early; they might give you a refund.  Or maybe not.)
818  A non-negative OpenMCL64 fixnum is limited to 60 bits; the
819  ccl::rdtsc function truncates the 64-bit counter value so
820  that it fits in a non-negative fixnum; if the 10 year limit
821  for the 64-bit value is accurate, the 60-bit value would
822  wrap around after about 223 days of uptime.
823
824  ccl::rdtsc64 returns the full 64-bit counter value, but
825  may return a bignum after 223 days of uptime.
826 
827- lots of bug fixes (not all of which involved the FFI or ObjC
828  bridge.) 
829
830 
831
832openmcl 1.1-pre-061231
833- The FASL version changed (old FASL files won't work with this
834  lisp version), as did the version information which tries to
835  keep the kernel in sync with heap images.
836  The binary incompatibility has to do with how a few pages of
837  low memory in the lisp kernel's address space are mapped and
838  used.  OpenMCL was generally assuming that these pages were
839  otherwise unused and could be used to map a small static data
840  area from the heap image file; on some platforms, the dynamic
841  linker may have already allocated data in those unused pages
842  before the lisp kernel even starts to run.  Fixing this involved
843  changing the address of that small static data area slightly,
844  and this caused the addresses of some objects contained within
845  that static data area - notably NIL - to change, as well.
846- This snapshot is otherwise just a set of bug fixes/work-in-progress
847  changes.
848- Even though no supported filesystem actually supports versioned files,
849  OpenMCL now tries to retain PATHNAME-VERSION informaton for physical
850  pathnames.  (The fact that it didn't caused several ANSI test failures.)
851  This change introduced/exposed a few other bugs; I think that I've
852  caught at least the most obvious ones, but it's possible that some
853  new pathname-related bugs have been introduced.
854- The cron job that runs on clozure.com and updates the ChangeLog from
855  CVS commit info stopped running as of a system upgrade in late November.
856  The problem was fixed a couple of weeks ago, so it's once again meaningful
857  to refer to the ChangeLog for details of bug fixes.
858- FSQRT and FSQRTS instructions are "optional" on the PPC.  In practice,
859  that often meant that they are implemented on chips made by IBM and
860  not on chips made by Motorola/FreeScale.  This version of OpenMCL
861  assumes that they're implemented and emulates them if they aren't.
862- OSX 10.2 (Jaguar) and earlier versions are officially no longer
863  supported.  (I honestly don't know if things have actually worked
864  on Jaguar in a while, but some recent changes are known not to
865  work on Jaguar and the kernel now inists on at least Panther on
866  startup.
867OpenMCL 1.1-pre-061205
868- This release is intended to package up the bug fixes since
869  the 061110 tarballs.  There aren't too many changes in
870  functionality or any deep architectural changes since 061110,
871  and it should be easy to bootstrap from current sources with
872  061110 images.
873  (It'd still be a good idea to recompile your code with
874  up-to-date images, whether you download those images or
875  build them yourself from CVS.)
876- The one (barely) notable change in functionality has to do
877  with how the lisp sets up pathname translations for the
878  "ccl" logical host when the "CCL_DEFAULT_DIRECTORY" environment
879  variable isn't set (e.g., when a shell script isn't used to
880  invoke the lisp.)  Previous versions just used the current
881  directory; this version tries to use the directory containing
882  the current heap image.  The new scheme might get fooled by
883  symbolic links (either following them or not following them
884  could be wrong), but it's more likely to work for people
885  who don't read or understand the discussion of the shell script
886  in the documentation.
887- All (knock wood) bugs that have been reported since the 061110
888  images were released should be fixed.  Well, almost all.  The
889  fixes include:
890
891  - a typo (wrong register) in the "generic" version of the
892    code which implements (SETF AREF) on 2-dimensional arrays
893    on x86-64
894  - incorrect bounds checking on vector references on x86-64,
895    which caused some invalid indices to be treated as valid
896    (usually leading to a segfault).  IIRC, the invalid indices
897    that were erroneously accepted were fixnums whose absolute
898    value was > (expt 2 56).  (More or less.).
899  - Missing stream methods (especially involving string streams)
900    affecting all platforms.
901  - Several bugs involving GCD, some of which were specific to
902    64-bit platforms and some of which affected all platforms.
903    (These bugs sometimes affected results returned by #'/,
904    LCM, and other funtions.)
905
906  - OpenMCL has only ever supported an ELEMENT-TYPE argument of
907   ([signed,unsigned]-byte 8|16|32|64) on binary file streams (with
908   64-bit types supported only on 64-bit platforms.)  It has not
909   previously tried to upgrade a supplied element-type to a supported
910   one (it does now) and any errors that resulted from supplying an
911   element-type that was not supported (and could not be upgraded) were
912   either obscure side-effects or quiet misbehavior; an error (a
913   SIMPLE-ERROR complaining about the unsupported element type) is now
914   signaled as soon as attempts to upgrade to a supported element type
915   fail.  I believe that the current behavior is both compliant and
916   reasonable; it's probably better to discuss that issue on
917   openmcl-devel than to do so here.
918
919
920OpenMCL 1.1-pre-061110
921- The FASL version changed (old FASL files won't work with this
922  lisp version), as did the version information which tries to
923  keep the kernel in sync with heap images.
924- Several bug fixes (see ChangeLog), and modest-to-moderate
925  performance improvements.  Notably, AREF and (SETF AREF)
926  of 2- and 3-dimensional arrays are open-coded in more cases
927  and are usually at least 5x faster than in previous versions.
928  If the compiler knows that the array in question is a
929  SIMPLE-ARRAY of appropiate dimensionality and knows the
930  array's element-type, the speedup can be much greater.
931  There are certainly opportunities for further improvements
932  here, both in breadth (handling more cases) and depth
933  (eliminating some type-and-bounds checking in safe code,
934  doing parts of index calculations at compile-time when
935  bounds and indices are constants ...), but things are
936  generally improved.
937- QUIT and SAVE-APPLICATION work a little differently; in
938  particular, SAVE-APPLICATION sometimes runs after #_exit
939  is called (via the #_atexit mechanism).
940  The motivation for this change has to do with how some
941  environments (Cocoa, to name one) conflate the ideas of
942  "shutting down the GUI" with "exiting the application".
943  Previous versions of OpenMCL tried to work around this
944  by overriding some internal Cocoa methods; that approach
945  was never particularly attractive and (predictably) it'll
946  break in future OSX releases.
947  The new scheme (which involves letting code run after #_exit
948  has been called) certainly offers other ways to lose; so
949  far, I haven't seen evidence of such lossage.
950- For historical reasons (forgotten historical reasons, in fact)
951  the PPC versions of OpenMCL run with floating-point underflow
952  exceptions disabled (the x86-64 versions enable these exceptions
953  by default.)  This should change soon (as soon as I remember
954  to change it ...); it's unlikely that this will affect much
955  user code, but it's possible that it'll do so.
956OpenMCL 1.1-pre-061024
957- The FASL version changed (old FASL files won't work with this
958  lisp version), as did the version information which tries to
959  keep the kernel in sync with heap images.
960- Linux users: it's possible (depending on the distribution that
961  you use) that the lisp kernel will claim to depend on newer
962  versions of some shared libraries than the versions that you
963  have installed.  This is mostly just an artifact of the GNU
964  linker, which adds version information to dependent library
965  references even though no strong dependency exists.  If you
966  run into this, you should be able to simply cd to the appropriate
967  build directory under ccl/lisp-kernel and do a "make".
968- There's now a port of OpenMCL to FreeBSD/amd64; it claims to be
969  of beta quality.  (The problems that made it too unstable
970  to release as of a few months ago have been fixed;  I stil run
971  into occasional FreeBSD-specific issues, and some such issues
972  may remain.)
973- The Darwin X8664 port is a bit more stable (no longer generates
974  obscure "Trace/BKPT trap" exits or spurious-looking FP exceptions.)
975  I'd never want to pass up a chance to speak ill of Mach, but both
976  of these bugs seemed to be OpenMCL problems rather than Mach kernel
977  problems, as I'd previously more-or-less assumed.
978- I generally don't use SLIME with OpenMCL, but limited testing
979  with the 2006-04-20 verson of SLIME seems to indicate that no
980  changes to SLIME are necessary to work with this version.
981- CHAR-CODE-LIMIT is now #x110000, which means that all Unicode
982  characters can be directly represented.  There is one CHARACTER
983  type (all CHARACTERs are BASE-CHARs) and one string type (all
984  STRINGs are BASE-STRINGs.)  This change (and some other changes
985  in the compiler and runtime) made the heap images a few MB larger
986  than in previous versions.
987- As of Unicode 5.0, only about 100,000 of 1114112./#x110000 CHAR-CODEs
988  are actually defined; the function CODE-CHAR knows that certain
989  ranges of code values (notably #xd800-#xddff) will never be valid
990  character codes and will return NIL for arguments in that range,
991  but may return a non-NIL value (an undefined/non-standard CHARACTER
992  object) for other unassigned code values.
993- The :EXTERNAL-FORMAT argument to OPEN/LOAD/COMPILE-FILE has been
994  extended to allow the stream's character encoding scheme (as well
995  as line-termination conventions) to be specified; see more
996  details below.  MAKE-SOCKET has been extended to allow an
997  :EXTERNAL-FORMAT argument with similar semantics.
998- Strings of the form "u+xxxx" - where "x" is a sequence of one
999  or more hex digits- can be used as as character names to denote
1000  the character whose code is the value of the string of hex digits.
1001  (The +  character is actually optional, so  #\u+0020, #\U0020, and
1002  #\U+20 all refer to the #\Space character.)  Characters with codes
1003  in the range #xa0-#x7ff (IIRC) also have symbolic names (the
1004  names from the Unicode standard with spaces replaced with underscores),
1005  so #\Greek_Capital_Letter_Epsilon can be used to refer to the character
1006  whose CHAR-CODE is #x395.
1007- The line-termination convention popularized with the CP/M operating
1008  system (and used in its descendants) - e.g., CRLF - is now supported,
1009  as is the use of Unicode #\Line_Separator (#\u+2028).
1010- About 15-20 character encoding schemes are defined (so far); these
1011  include UTF-8/16/32 and the big-endian/little-endian variants of
1012  the latter two and ISO-8859-* 8-bit encodings.  (There is not
1013  yet any support for traditional (non-Unicode) ways of externally
1014  encoding characters used in Asian languages, support for legacy
1015  MacOS encodings, legacy Windows/DOS/IBM encodings, ...)  It's hoped
1016  that the existing infrastructure will handle most (if not all) of
1017  what's missing; that may not be the case for "stateful" encodings
1018  (where the way that a given character is encoded/decoded depend
1019  on context, like the value of the preceding/following character.)
1020- There isn't yet any support for Unicode-aware collation (CHAR>
1021  and related CL functions just compare character codes, which
1022  can give meaningless results for non-STANDARD-CHARs), case-inversion,
1023  or normalization/denormalization.  There's generally good support
1024  for this sort of thing in OS-provided libraries (e.g., CoreFoundation
1025  on MacOSX), and it's not yet clear whether it'd be best to duplicate
1026  that in lisp or leverage library support.
1027- Unicode-aware FFI functions and macros are still in a sort of
1028  embryonic state if they're there at all; things like WITH-CSTRs
1029  continue to exist (and continue to assume an 8-bit character
1030  encoding.)
1031- Characters that can't be represented in a fixed-width 8-bit
1032  character encoding are replaced with #\Sub (= (code-char 26) =
1033  ^Z) on output, so if you do something like:
1034
1035? (format t "~a" #\u+20a0)
1036
1037  you might see a #\Sub character (however that's displayed on
1038  the terminal device/Emacs buffer) or a Euro currency sign or
1039  practically anything else (depending on how lisp is configured
1040  to encode output to *TERMINAL-IO* and on how the terminal/Emacs
1041  is configured to decode its input.
1042
1043  On output to streams with character encodings that can encode
1044  the full range of Unicode - and on input from any stream -
1045  "unencodable characters" are represented using the Unicode
1046  #\Replacement_Character (= #\U+fffd); the presence of such a
1047  character usually indicates that something got lost in translation
1048  (data wasn't encoded properly or there was a bug in the decoding
1049  process.)
1050- Streams encoded in schemes which use more than one octet per code unit
1051  (UTF-16, UTF-32, ...) and whose endianness is not explicit will be
1052  written with a leading byte-order-mark character on (new) output and
1053  will expect a BOM on input; if a BOM is missing from input data,
1054  that data will be assumed to have been serialized in big-endian order.
1055  Streams encoded in variants of these schemes whose endianness is
1056  explicit (UTF-16BE, UCS-4LE, ...) will not have byte-order-marks written
1057  on output or expected on input.  (UTF-8 streams might also contain
1058  encoded byte-order-marks; even though UTF-8 uses a single octet per
1059  code unit - and possibly more than one code unit per character - this
1060  convention is sometimes used to advertise that the stream is UTF-8-
1061  encoded.  The current implementation doesn't skip over/ignore leading
1062  BOMs on UTF8-encoded input, but it probably should.)
1063
1064  If the preceding paragraph made little sense, a shorter version is
1065  that sometimes the endianness of encoded data matters and there
1066  are conventions for expressing the endianness of encoded data; I
1067  think that OpenMCL gets it mostly right, but (even if that's true)
1068  the real world may be messier.
1069- By default, OpenMCL uses ISO-8859-1 encoding for *TERMINAL-IO*
1070  and for all streams whose EXTERNAL-FORMAT isn't explicitly specified.
1071  (ISO-8859-1 just covers the first 256 Unicode code points, where
1072  the first 128 code points are equivalent to US-ASCII.)  That should
1073  be pretty much equivalent to what previous versions (that only
1074  supported 8-bit characters) did, but it may not be optimal for
1075  users working in a particular locale.  The default for *TERMINAL-IO*
1076  can be set via a command-line argument (see below) and this setting
1077  persists across calls to SAVE-APPLICATION, but it's not clear that
1078  there's a good way of setting it automatically (e.g., by checking
1079  the POSIX "locale" settings on startup.)  Thing like POSIX locales
1080  aren't always set correctly (even if they're set correctly for
1081  the shell/terminal, they may not be set correctly when running
1082  under Emacs ...) and in general, *TERMINAL-IO*'s notion of the
1083  character encoding it's using and the "terminal device"/Emacs subprocess's
1084  notion need to agree (and fonts need to contain glyphs for the
1085  right set of characters) in order for everything to "work".  Using
1086  ISO-8859-1 as the default seemed to increase the likelyhood that
1087  most things would work even if things aren't quite set up ideally
1088  (since no character translation occurs for 8-bit characters in
1089  ISO-8859-1.)
1090- In non-Unicode-related news: the rewrite of OpenMCL's stream code
1091  that was started a few months ago should now be complete (no more
1092  "missing method for BASIC-STREAM" errors, or at least there shouldn't
1093  be any.)
1094- I haven't done anything with the Cocoa bridge/demos lately, besides
1095  a little bit of smoke-testing.
1096
1097Some implementation/usage details:
1098
1099Character encodings.
1100
1101CHARACTER-ENCODINGs are objects (structures) that're named by keywords
1102(:ISO-8859-1, :UTF-8, etc.).  The structures contain attributes of
1103the encoding and functions used to encode/decode external data, but
1104unless you're trying to define or debug an encoding there's little
1105reason to know much about the CHARACTER-ENCODING objects and it's
1106generally desirable (and sometimes necessary) to refer to the encoding
1107via its name.
1108
1109Most encodings have "aliases"; the encoding named :ISO-8859-1 can
1110also be referred to by the names :LATIN1 and :IBM819, among others.
1111Where possible, the keywordized name of an encoding is equivalent
1112to the preferred MIME charset name (and the aliases are all registered
1113IANA charset names.)
1114
1115NIL is an alias for the :ISO-8859-1 encoding; it's treated a little
1116specially by the I/O system.
1117
1118The function CCL:DESCRIBE-CHARACTER-ENCODINGS will write descriptions
1119of all defined character encodings to *terminal-io*; these descriptions
1120include the names of the encoding's aliases and a doc string which
1121briefly describes each encoding's properties and intended use.
1122
1123Line-termination conventions.
1124
1125As noted in the <=1.0 documentation, the keywords :UNIX, :MACOS, and
1126:INFERRED can be used to denote a stream's line-termination conventions.
1127(:INFERRED is only useful for FILE-STREAMs that're open for :INPUT or
1128:IO.)  In this release, the keyword :CR can also be used to indicate
1129that a stream uses #\Return characters for line-termination (equivalent
1130to :MACOS), the keyword :UNICODE denotes that the stream uses Unicode
1131#\Line_Separator characters to terminate lines, and the keywords :CRLF,
1132:CP/M, :MSDOS, :DOS, and :WINDOWS all indicate that lines are terminated
1133via a #\Return #\Linefeed sequence.
1134
1135In some contexts (when specifying EXTERNAL-FORMATs), the keyword :DEFAULT
1136can also be used; in this case, it's equivalent to specifying the value
1137of the variable CCL:*DEFAULT-LINE-TERMINATION*.  The initial value of
1138this variable is :UNIX.
1139
1140Note that the set of keywords used to denote CHARACTER-ENCODINGs and
1141the set of keywords used to denote line-termination conventions is
1142disjoint: a keyword denotes at most a character encoding or a line
1143termination convention, but never both.
1144
1145External-formats.
1146
1147EXTERNAL-FORMATs are also objects (structures) with two read-only
1148fields that can be accessed via the functions EXTERNAL-FORMAT-LINE-TERMINATION
1149and EXTERNAL-FORMAT-CHARACTER-ENCODING; the values of these fields are
1150line-termination-convention-names and character-encoding names as described
1151above.
1152
1153An EXTERNAL-FORMAT object via the function MAKE-EXTERNAL-FORMAT:
1154
1155MAKE-EXTERNAL-FORMAT &key domain character-encoding line-termination
1156
1157(Despite the function's name, it doesn't necessarily create a new,
1158unique EXTERNAL-FORMAT object: two calls to MAKE-EXTERNAL-FORMAT
1159with the same arguments made in the same dynamic environment will
1160return the same (eq) object.)
1161
1162Both the :LINE-TERMINATION and :CHARACTER-ENCODING arguments default
1163to :DEFAULT; if :LINE-TERMINATION is specified as or defaults to
1164:DEFAULT, the value of CCL:*DEFAULT-LINE-TERMINATION* is used to
1165provide a concrete value.
1166
1167When the :CHARACTER-ENCODING argument is specifed as/defaults to
1168:DEFAULT, the concrete character encoding name that's actually used
1169depends on the value of the :DOMAIN argument to MAKE-EXTERNAL-FORMAT.
1170The :DOMAIN-ARGUMENT's value can be practically anything; when it's
1171the keyword :FILE and the :CHARACTER-ENCODING argument's value is
1172:DEFAULT, the concrete character encoding name that's used will be
1173the value of the variable CCL:*DEFAULT-FILE-CHARACTER-ENCODING*; the
1174initial value of this variable is NIL (which is an alias for :ISO-8859-1).
1175If the value of the :DOMAIN argument is :SOCKET and the :CHARACTER-ENCODING
1176argument's value is :DEFAULT, the value of
1177CCL:*DEFAULT-SOCKET-CHARACTER-ENCODING* is used as a concrete character
1178encoding name.  The initial value of CCL:*DEFAULT-SOCKET-CHARACTER-ENCODING*
1179is NIL, again denoting the :ISO-8859-1 encoding.
1180If the value of the :DOMAIN argument is anything else, :ISO-8859-1 is
1181also used (but there's no way to override this.) 
1182
1183The result of a call to MAKE-EXTERNAL-FORMAT can be used as the value
1184of the :EXTERNAL-FORMAT argument to OPEN, LOAD, COMPILE-FILE, and
1185MAKE-SOCKET; it's also possible to use a few shorthand constructs
1186in these contexts:
1187
1188* if ARG is unspecified or specified as :DEFAULT, the value of the
1189  variable CCL:*DEFAULT-EXTERNAL-FORMAT* is used.  Since the value
1190  of this variable has historically been used to name a default
1191  line-termination convention, this case effectively falls into
1192  the next one:
1193* if ARG is a keyword which names a concrete line-termination convention,
1194  an EXTERNAL-FORMAT equivalent to the result of calling
1195  (MAKE-EXTERNAL-FORMAT :line-termination ARG)
1196   will be used
1197* if ARG is a keyword which names a character encoding, an EXTERNAL-FORMAT
1198  equvalent to the result of calling
1199  (MAKE-EXTERNAL-FORMAT :character-encoding ARG)
1200  will be used
1201* if ARG is a list, the result of (APPLY #'MAKE-EXTERNAL-FORMAT ARG)
1202  will be used
1203
1204(When MAKE-EXTERNAL-FORMAT is called to create an EXTERNAL-FORMAT
1205object from one of these shorthand designators, the value of the
1206:DOMAIN keyword argument is :FILE for OPEN,LOAD, and COMPILE-FILE
1207and :SOCKET for MAKE-SOCKET.)
1208
1209STREAM-EXTERNAL-FORMAT.
1210The CL function STREAM-EXTERNAL-FORMAT - which is portably defined
1211on FILE-STREAMs - can be applied to any open stream in this release
1212and will return an EXTERNAL-FORMAT object when applied to an open
1213CHARACTER-STREAM. For open CHARACTER-STREAMs (other than STRING-STREAMs),
1214SETF can be used with STREAM-EXTERNAL-FORMAT to change the stream's
1215character encoding, line-termination, or both.
1216
1217If a "shorthand" external-format designator is used in a call to
1218(SETF STREAM-EXTERNAL-FORMAT), the "domain" used to construct an
1219EXTERNAL-FORMAT is derived from the class of the stream in the
1220obvious way (:FILE for FILE-STREAMs, :SOCKET for ... well, for
1221sockets ...)
1222
1223Note that the effect or doing something like:
1224
1225(let* ((s (open "foo" ... :external-format :utf-8)))
1226  ...
1227  (unread-char ch s)
1228  (eetf (stream-external-format s) :us-ascii)
1229  (read-char s))
1230
1231might or might not be what was intended.  The current behavior is
1232that the call to READ-CHAR will return the previously unread character
1233CH, which might surprise any code which assumes that the READ-CHAR
1234will return something encodable in 7 or 8 bits.  Since functions
1235like READ may call UNREAD-CHAR "behind your back", it may or may
1236not be obvious that this has even occurred; the best approach to
1237dealing with this issue might be to avoid using READ or explicit
1238calls to UNREAD-CHAR when processing content encoded in multiple
1239external formats.
1240
1241There's a similar issue with "bivalent" streams (sockets) which
1242can do both character and binary I/O with an :ELEMENT-TYPE of
1243(UNSIGNED-BYTE 8).  Historically, the sequence:
1244
1245   (unread-char ch s)
1246   (read-byte s)
1247
1248caused the READ-BYTE to return (CHAR-CODE CH); that made sense
1249when everything was implicitly encoded as :ISO-8859-1, but may not
1250make any sense anymore.  (The only thing that seems to make sense
1251in that case is to clear the unread character and read the next
1252octet; that's implemented in some cases but I don't think that
1253things are always handled consistently.)
1254
1255Command-line argument for specifying the character encoding to
1256be used for *TERMINAL-IO*.
1257
1258Shortly after a saved lisp image starts up, it creates the standard
1259CL streams (like *STANDARD-OUTPUT*, *TERMINAL-IO*, *QUERY-IO*, etc.);
1260most of these streams are usually SYNONYM-STREAMS which reference
1261the TWO-WAY-STREAM *TERMINAL-IO*, which is itself comprised of
1262a pair of CHARACTER-STREAMs.  The character encoding used for
1263any CHARACTER-STREAMs created during this process is the one
1264named by the value of the variable CCL:*TERMINAL-CHARACTER-ENCODING-NAME*;
1265this value is initially NIL.
1266
1267The -K or --terminal-encoding command-line argument can be used to
1268set the value of this variable (the argument is processed before the
1269standard streams are created.)  The string which is the value of
1270the -K/--terminal-encoding argument is uppercased and interned in
1271the KEYWORD package; if an encoding named by that keyword exists,
1272CCL:*TERMINAL-CHARACTER-ENCODING-NAME* is set to the name of that
1273encoding.  For example:
1274
1275shell> openmcl -K utf-8
1276
1277will have the effect of making the standard CL streams use :UTF-8
1278as their character encoding.
1279
1280(It's probably possible - but a bit awkward - to use (SETF EXTERNAL-FORMAT)
1281from one's init file or --eval arguments or similar to change existing
1282streams' character encodings; the hard/awkward parts of doing so include
1283the difficulty of determining which standard streams are "real" character
1284streams and which are aliases/composite streams.)
1285
1286OpenMCL 1.1-pre-069826
1287- There's an (alpha-quality, maybe) port to x86-64 Darwin (e.g., the
1288  Mac Pro.)  Some known problems include:
1289 
1290  * infrequently (but not infrequently enough) the lisp dies on
1291    startup with a spurious "Trace/BKPT trap" error message.  This
1292    seems to be timing-dependent and (very generally) seems to
1293    involve the Mach exception thread not recognizing an exception
1294    used to effect exception return.  Sometimes, this shows up
1295    as a (:SIGNALED 5) error when REBUILD-CCL runs the lisp to
1296    create a new image.
1297
1298  * some math library primitives (#_asin, for one) generate
1299    spurious incidental FP exceptions that have nothing to
1300    do with the validity of the arguments or result.  To work around
1301    this, the lisp ignores FP exceptions which might have occurred
1302    during a call into the math library; that means that it doesn't
1303    detect -real- FP exceptions when they're signaled.  (This bug
1304    only affects things that call into the system math library;
1305    lisp arithmetic operations that're done inline are not affected.)
1306
1307  * The version of OSX/Darwin that shipped with the Mac Pro is missing
1308    some functionality that from OpenMCL's point of view is highly
1309    desirable (namely, the ability to keep application-level thread-
1310    specific data in a per-thread block of memory addressed by an
1311    otherwise unused segment register.)  To get things working (as
1312    well as they are), the lisp "shares" the segment register that
1313    the pthreads library uses to access thread data.  This scheme
1314    isn't intended to be long-lived (and negatively affects
1315    performance of things like foreign-function calls, callbacks,
1316    and exception handling).
1317 
1318  * The .cdb files (libc only for Tiger) in ccl:darwin-x86-headers64;
1319    were cross-developed on a Linux x86-64 system, since Apple
1320    has not yet released the sources to their x86-64 enabled gcc.
1321
1322- On all platforms, stream code has been rewritten and often offers
1323  better (sometimes substantially better) performance.  OPEN and
1324  MAKE-SOCKET have each been extended to take additional keyword
1325  arguments.
1326
1327  :SHARING, which can have the values :PRIVATE (the default), :LOCK,
1328  or :EXTERNAL (NIL is also accepted as synonym for :EXTERNAL)
1329
1330   :PRIVATE specifies that the stream can only be accessed by
1331   the thread that created it.  (There was some discussion on openmcl-devel
1332   about the idea of "transferring ownership" of a stream; this has
1333   not yet been implemented.)  Attempts to do I/O on a stream with
1334   :PRIVATE sharing from a thread other than the stream's owner yield
1335   an error.
1336
1337   :LOCK specifies that all access to the stream require the calling
1338   thread to obtain a lock; there are separate "read" and "write"
1339   locks for IO streams (so it's possible for one thread to read
1340   from such a stream while another thread writes to it, for instance.)
1341   :LOCK was the implicit default for all streams prior to this change.
1342   (See below - under the discussion of the AUTO-FLUSH mechanism -
1343   for a discussion of one of the implications of this change that
1344   affects SLIME users.)
1345
1346   :EXTERNAL (or NIL) specifies that I/O primitives enforce no
1347   access protocol.  This may be appropriate for some types of application
1348   which can control stream access via application-level protocols.  Note
1349   that since even the act of reading from a stream changes its internal
1350   state (and simultaneous access from multiple threads can therefore
1351   lead to corruption of that state), some care must be taken in the
1352   design of such protocols.
1353
1354  The :BASIC keyword argument influences whether or not the stream
1355  will be an instance of the class FUNDAMENTAL-STREAM (the superclass
1356  from which all Gray stream classes inherit) or a subclass of the
1357  built-in class CCL::BASIC-STREAM.  The default value of :BASIC
1358  is T and this has effect for FILE-STREAMs created via OPEN;
1359  SOCKETs are still always implemented as FUNDAMENTAL (Gray) streams,
1360  though this should change soon.
1361
1362   The tradeoff between FUNDAMENTAL and BASIC streams is entirely
1363   between flexibility and (potential or actual) performance.  I/O
1364   primitives can recognize BASIC-STREAMs and exploit knowledge of
1365   implementation details; FUNDAMENTAL stream classes can be
1366   subclassed in a semi-standard way (the Gray streams protocol.)
1367
1368   For existing stream classes (FILE-STREAMs, SOCKETs, and the
1369   internal CCL::FD-STREAM classes used to implement file streams
1370   and sockets), a lot of code can be shared between the
1371   FUNDAMENTAL and BASIC implementations.  The biggest difference
1372   should be that that code can be reached from I/O primitives
1373   like READ-CHAR without going through some steps that're there
1374   to support generality and extensibility, and skipping those
1375   steps when that support isn't needed can improve I/O performance.
1376
1377   Gray stream methods (STREAM-READ-CHAR) should work on
1378   appropriate BASIC-STREAMs.  (There may still be cases where
1379   such methods are undefined; such cases should be considered
1380   bugs.)  It is not guaranteed that Gray stream methods would
1381   ever be called by I/O primitives to read a character from
1382   a BASIC-STREAM (though there are still cases where this happens.)
1383
1384   A simple loop reading 2M characters from a text file runs about
1385   10X faster when the file is opened the new defaults (:SHARING :PRIVATE
1386   :BASIC T) than it had before these changes were made.  That sounds
1387   good, until one realizes that the "equivalent" C loop can be about
1388   10X faster still ...
1389
1390 - Forcing output to interactive streams.
1391
1392   OpenMCL has long had a (mostly undocumented) mechanism whereby
1393   a mostly idle thread wakes up a few (~3) times per second and
1394   calls FORCE-OUTPUT on specified OUTPUT-STREAMS; this helps to
1395   ensure that streams with which a user would be expected to
1396   interact (the output side of *TERMINAL-IO*, listener windows
1397   in a GUI, etc.) have all buffered output flushed without
1398   requiring application or I/O library code to be concerned about
1399   that.
1400
1401   The SLIME lisp interaction mode for Emacs uses this mechanism,
1402   but the changes described above interfere with SLIMEs use of
1403   it:  in order to be safely accessed from multiple threads (the
1404   SLIME REPL thread and the thread which does the background
1405   periodic flushing of buffered output), a stream must have
1406   been created with :SHARING :LOCK in effect.  This is no longer
1407   the effective default; the code which does the periodic
1408   output flushing ignores streams which do not use locks as an
1409   access/sharing mechanism.  THIS MEANS THAT BUFFERRED OUTPUT
1410   TO SLIME REPLs WILL NOT BE AUTOMATICALLY FLUSHED TO THE SCREEN.
1411   A small change to SLIME's "swank-openmcl.lisp" is required
1412   to restore this functionality.  First,  a brief description of
1413   a couple of new primitives:
1414
1415   (CCL:ADD-AUTO-FLUSH-STREAM s)
1416
1417    Adds "s", which should be a "simple" OUTPUT-STREAM as returned
1418    by OPEN or MAKE-SOCKET, to a list of streams whose buffered
1419    output should be periodically flushed.  If S was not created
1420    with :SHARING :LOCK in effect, the stream will have its
1421    :SHARING mode changed to put :SHARING :LOCK into effect.
1422
1423   (CCL:REMOVE-AUTO-FLUSH-STREAM s)
1424   
1425    Removes S from the internal list of automatically flushed
1426    streams.  Does not restore the stream's :SHARING mode, which
1427    may have been changed by a previous call to ADD-AUTO-FLUSH-STREAM.
1428
1429 - SLIME changes
1430   In slime:swank-openmcl.lisp, around line 182, the method
1431
1432(defmethod make-stream-interactive ((stream ccl:fundamental-output-stream))
1433  (push stream ccl::*auto-flush-streams*))
1434
1435   should be changed to use CCL:ADD-AUTOFLUSH-STREAM if it's defined:
1436
1437(defmethod make-stream-interactive ((stream ccl:fundamental-output-stream))
1438  (if (fboundp 'ccl::add-auto-flush-stream)
1439    (ccl::add-auto-flush-stream stream)
1440    (push stream ccl::*auto-flush-streams*)))
1441
1442   That's adequate for the moment, since sockets are still
1443   FUNDAMENTAL-STREAMs.  When that changes, some more extensive changes
1444   to swank-openmcl.lisp may become necessary.
1445
1446- on x86-64, floating-point-underflow exceptions are now enabled
1447  by default.  (They really should be on ppc as well.)  Again,
1448  this affects FP operations that are done in lisp code and
1449  the results of FP operations that are reported in response
1450  to calls to reasonable (non-Darwin) math libraries.  This
1451  can affect whether or not some "potential number"  reader
1452  tokens are representable as numbers, e.g., whether or not
1453  attempts to read something like "1.0f-50" signal underflow
1454  or are quietly mapped to 0.0f0.
1455
1456- examples: Phil (from the mailing list) has added code which
1457  supports some of the ffi examples from the documentation.
1458
1459- Bug fixes: see ChangeLog
1460
1461
1462
1463OpenMCL 1.1-pre-060705
1464- Bug fixes again.  Some internal changes to support a FreeBSD/AMD64
1465  port that's not quite ready.
1466
1467- :MCL is back on *features*; there seem to be too many packages out
1468  there that expect it to be, and there hasn't been enough advance
1469  notice of its pending removal.
1470   
1471OpenMCL 1.1-pre-060623
1472- Mostly bug fixes (a CLOS bug that prevented the ObjC bridge from
1473  working, FIXNUM arrays weren't quite finished on PPC)
1474
1475- Use Tiger inferfaces (from XCode 10.4u SDK) on DarwinPPC32
1476
1477- Add gl, gtk2, gnome2 interfaces for x86-64.  Add a tiny
1478  "gtk2-clock" example, tweak the opengl-ffi (GLUT) example
1479  so that it works on x86-64.
1480
1481- Some changes to the ObjC bridge to support loading additional
1482  frameworks; update the WebKit example to use these new features.
1483
1484- Still an outstanding issue where things like MAKE-OBJC-INSTANCE
1485  need access to the interfaces at runtime (and can crash if they
1486  aren't available.)
1487
1488- Build snapshots for LinuxPPC{32,64}.
1489
1490OpenMCL 1.1-pre-060608
1491- The FASL version changed, as did the version number which pairs
1492  the lisp kernel with heap images.  Images saved with older kernels
1493  can't be loaded on this one; the images/kernels in the 060608
1494  snapshot tarballs should match.
1495
1496  Most of the ABI changes that caused these version changes were
1497  x86-64 specific; some auxiliary stack pointers that had been
1498  kept in MMX registers are now kept in per-thread memory. (Signal/
1499  exception handlers generally need to be able to access these
1500  stack pointers, but at least some versions of the Linux kernel
1501  don't reliably pass correct values of the MMX registers in the
1502  signal contexts passed to signal handlers.  Moral: some kinds
1503  of stack-allocation and foreign-function operations may (or may not)
1504  be a few cycles slower, but OpenMCL should be a bit less prone
1505  to fatal segfault exceptions.)
1506
1507  Other than that, most changes since the 060530 snapshots are
1508  bugfixes (see the ChangeLog for details).  The x86-64 port has been
1509  exercised fairly heavily (if somewhat narrowly) and its welcome
1510  banner now claims that it's a beta release.  I think that that's
1511  probably fair, and hope that anyone who may have been reluctant to
1512  test an alpha release will agree and be less reluctant.
1513
1514- There's still much more to be done, but some preliminary 1.1 documentation
1515  is now online at:
1516
1517<http://newsite.openmcl.clozure.com/Doc>
1518
1519  Note that some relative links on "newsite" may be invalid, but the
1520  internal links in the Doc directory should work.
1521
1522  As noted above, it still needs a lot of work; feedback, criticism,
1523  and help would all be appreciated.
1524
1525OpenMCL 1.1-pre-060530
1526
1527- These release notes have gotten woefully out of date.
1528
1529- OpenMCL now runs on x86-64 (AMD64, Intel EM64T) systems under Linux.
1530  It announces itself as an alpha release in the Welcome banner; it should
1531  in fact be very nearly feature-complete (but possibly still buggy.)
1532  There's a chicken-and-egg issue in that it needs more testing before
1533  it can be formally released and some people may be waiting for a more
1534  stable version.
1535
1536  The build process and most user-visible things should behave the same
1537  way as on PPC; using REBUILD-CCL (described below) is generally the
1538  simplest way to rebuild from sources.  A few (intentional) differences:
1539 
1540  * the lisp kernel is named "lx86cl64", the default heap image is
1541    named "LX86CL64" (e.g., the kernel name, case-inverted) and the
1542    bootstrapping image is conventionally named "x86-boot64".
1543
1544  * FASL files have the extension "lx64fsl"
1545
1546  * the kernel build directory is "ccl/lisp-kernel/linuxx8664"
1547
1548  * the "openmcl64" shell script can be used to invoke the
1549    lisp, as on 64-bit PPC platforms.
1550
1551Other changes tend to be a little more modest:
1552
1553- there is now a specialized FIXNUM array element type on all platforms.
1554  (distinct from T or (SIGNED-BYTE <machine-word-size>)).  Access to
1555  such vectors is a little cheaper than the SIGNED-BYTE case (since
1556  elements are known to be fixnums) and a little easier on the GC
1557  than the T case (the GC can avoid looking at their contents and
1558  there are no associated EGC write-barrier issues.)
1559
1560- "colon" commands entered into the REPL/break loops don't need to
1561  be parenthesized if the command and all operands are on the same
1562  line. E.g.
1563
15641> :f 0
1565
1566  and
1567
15681> (:f 0)
1569
1570  are equivalent (and have the effect of examining the raw contents of
1571  the 0th stack frame)
1572
1573- the syntax of the :B (backtrace) break-loop has changed; rather
1574  than taking an optional argument which specifies whether or not
1575  frame details should be shown, it now accepts keyword arguments
1576  for specifying:
1577   
1578  :start        ; unsigned integer: the index of the first frame to show
1579  :count        ; unsigned integer: the maximum number of frames to show
1580  :detailed-p   ; boolean: whether or not to show frame detail
1581
1582- a new break-loop command :NFRAMES returns the number of stack frames
1583  accessible to backtrace.  (Both this change and the previous
1584  are intended to help deal with deep recursion/stack overflow cases.)
1585
1586- any command-line arguments that follow a "--" pseudo-argument
1587  are not processed by the lisp startup code and are stored
1588  (as a list of strings) in CCL:*UNPROCESSED-COMMAND-LINE-ARGUMENTS*.
1589  E.g.:
1590
1591shell> openmcl -- -foo 17
1592[...]
1593? ccl:*UNPROCESSED-COMMAND-LINE-ARGUMENTS*
1594=> ("-foo" "17")
1595
1596OpenMCL 1.1-pre-060226
1597
1598- The --thread-stack-size (or -Z)  command-line argument changes the values
1599  of the variables used to determine the sizes of the listener thread.
1600  The values of these variables will persist accross SAVE-APPLICATION;
1601  these values have no effect on the sizes of stacks in threads created
1602  under explicit user control.
1603
1604- New functions:
1605
1606  (CCL:GC-VERBOSE on-full-gc &optional (on-egc on-full-gc))
1607
1608  Causes the GC to print (or stop printing ...) informational messages
1609  on entry and exit.  The ON-FULL-GC argument controls whether or
1610  not these messages are printed on ... a full GC, and the ON-EGC
1611  argument (which defaults to the value of the ON-FULL-GC argument)
1612  controls whether messages are printed on ephemeral GCs.
1613
1614  (CCL:GC-VERBOSE-P)
1615
1616  Returns two values (corresponding to the arguments of the last call
1617  to CCL:GC-VERBOSE.)
1618
1619  (CCL:REBUILD-CCL &key :FULL :CLEAN :KERNEL :FORCE :RELOAD :EXIT
1620                         :RELOAD-ARGUMENTS)
1621
1622  Depending on the values of its arguments, recompiles lisp and/or
1623  kernel sources and optionallly re-generates ("reloads") a heap
1624  image.
1625
1626  Arguments:
1627
1628  clean   deletes FASL and .o files before performing other steps
1629  kernel  rebuilds the lisp kernel
1630  force   forces recompilation, even if binary is newer than source
1631  reload  tries to rebuild a full heap image after other build steps
1632  exit    quits after all other steps
1633  full    equivalent to :CLEAN T :KERNEL T :RELOAD T
1634  reload-arguments a list of strings, passed as additional arguments
1635                   to the reload step.  E.g. '("--thread-stack-size" "128M").
1636
1637  Output from the :KERNEL and :RELOAD steps is ordinarily only displayed
1638  if an error occurs.
1639 
1640
1641- Changes
1642
1643  TRACE now prints an integer (corresponding to the level of indentation)
1644  on each line of output.
1645
1646  Tracing callbacks is currently broken (it may be reimplemented; if so,
1647  it'd be implemented somewhat differently ...)
1648
1649- Bugs
1650
1651  Several bugs involving interactions between the GC and (many) active
1652  threads have been fixed; at least one such bug remains (the symptom
1653  involves a recently allocated  array somehow getting trashed or GCed
1654  incorrectly; the  cause has been under investigation for weeks but is
1655  still not known.)
1656
1657OpenMCL 1.1-pre-060125
1658
1659- FASL version changed; delete old FASL (.dfsl, .pfsl, .dfsl64, .pfsl64) files
1660 
1661- "kernel ABI version" changed; build a new lisp kernel before trying to load/use
1662   060125 images.
1663
1664-  Changes: (see also ChangeLog)
1665
1666   New variable:
1667
1668   CCL:*HOST-PAGE-SIZE*
1669
1670   Initialized on application startup to contain the MMU/OS page size in bytes.
1671   This is 4K on PPC platforms (and likely on most? all? x86 platforms).
1672
1673   New functions:
1674
1675   CCL:DEFAULT-ALLOCATION-QUANTUM
1676
1677   Returns an integer, the value of which is used by the lisp kernel when
1678   mapping heap memory from the OS.  Mapping requests are usually made in
1679   multiples of this value. 
1680
1681   This value is read-only; currently, it's 64KB on 32-bit platforms and
1682   128KB on 64-bit platforms.
1683
1684
1685   CCL:PROCESS-ALLOCATION-QUANTUM p
1686
1687   Returns the (per-thread) allocation quantum of the process P.  By default,
1688   this is the same value as that returned by CCL:DEFAULT-ALLOCATION-QUANTUM,
1689   but lower values can be specified on a per-process basis (see below.)
1690
1691   This value is read-only.
1692
1693   CCL:CURRENT-PROCESS-ALLOCATION-QUANTUM
1694
1695   Equivalent to (CCL:PROCESS-ALLOCATION-QUANTUM *CURRENT-PROCESS*),
1696   but can be used with SETF to change the current processes's
1697   allocation quantum to a value which is between *HOST-PAGE-SIZE* and
1698   (DEFAULT-ALLOCATION-QUANTUM), inclusive, and which is a power of 2.
1699
1700
1701   Changes to existing functions:
1702
1703   Both PROCESS-RUN-FUNCTION and MAKE-PROCESS accept an :ALLOCATION-QUANTUM
1704   &key argument, which defaults to the value returned by (DEFAULT-ALLOCATION-QUANTUM).
1705   If provided, the value of the argument should should satisfy the same
1706   constraints that (SETF (CURRENT-PROCESS-ALLOCATION-QUANTUM) is subject to.
1707
1708Discussion
1709
1710In general, larger per-thread allocation quanta are appropriate for programs
1711where a relatively small number of threads need to allocate memory frequently
1712and small per-thread quanta are appropriate for larger numbers of threads
1713that are expected to do small, infrequent memory allocations.
1714
1715The worst-case scenarios would involve a large number of threads doing
1716incidental memory allocation with large quanta (that wastes memory and may
1717trigger the GC too frequently) or a small number of threads doing frequent
1718memory allocation with small quanta (since such threads could be expected
1719to fill up their small per-thread memory allocations quickly and frequently
1720and would waste time frequently allocating more small chunks.)
1721
1722All of these values interact with the GC and EGC thresholds; the ability
1723to exercise some control over how much per-threads memory is allocated
1724at a time can help to ensure that those interactions are appropriate.
1725When these mechanisms are insufficient, applications should consider the
1726use of available mechanisms for adjusting GC and EGC thresholds.
1727
1728
1729
1730
1731OpenMCL 1.1-pre-051027
1732
1733- A lot of internal changes in the way that special bindings, UNWIND-PROTECT,
1734  and WITHOUT-INTERRUPTS are implemented (and in how they interact with
1735  each other.
1736
1737  One user-visible aspect of this is that UNWIND-PROTECT cleanup forms
1738  are run with interrupts disabled (the protected form is run with
1739  interrupts enabled if they were enabled on entry to the UNWIND-PROTECT.)
1740  This means that something like:
1741
1742  (unwind-protect
1743      nil
1744    (loop))
1745
1746  will loop uninterruptibly.
1747
1748- CCL:WITH-INTERRUPTS-ENABLED &body body executes the body with interrupts
1749  enabled.  The example above could be rewritten as:
1750
1751  (unwind-protect
1752      nil
1753    (with-interrupts-enabled (loop)))
1754
1755  and the loop would be interruptible.
1756
1757  These changes introduce binary incompatibility (the FASL version changed,
1758  as did an internal version number that tries to keep the kernel and
1759  heap image in synch.)
1760
1761  Things basically work, but there may be lingering bugs (e.g., as of
1762  a little while ago, QUIT didn't work because the initial process
1763  was running with interrupts disabled.)
1764
1765- PROCESS-TERMINATION-SEMAPHORE
1766  MAKE-PROCESS and PROCESS-RUN-FUNCTION accept a :TERMINATION-SEMAPHORE
1767  argument; processes have a PROCESS-TERMINATION-SEMAPHORE accessor
1768  method.  If the argument is specified and non-null, its value should
1769  of type SEMAPHORE.
1770
1771  If a process dies by any means after it's been successfully enabled
1772  and it has a non-null termination semaphore "at the time of its death",
1773  that semaphore will be signaled just before the underlying OS thread
1774  is destroyed.
1775
1776  SETF can be used with PROCESS-TERMINATION-SEMAPHORE to change or
1777  clear a the termination semaphore of a process.  If the target
1778  process is not the current process when this happens, it's possible
1779  that the process could die before the SETF takes effect; this
1780  possibility must be addressed at the application level (i.e., the
1781  implementation doesn't try to synchronize the calling thread and
1782  the target in any way.
1783
1784  A simple example:
1785
1786  (let* ((s (make-semaphore)))
1787    (process-run-function `(:name "sleepy" :termination-semaphore ,s)
1788                           #'(lambda () (sleep 10)))
1789    (wait-on-semaphore s))
1790
1791  The calling thread will wait for (roughly) 10 seconds (until the
1792  "sleepy" thread has had its nap and signals its termination semaphore.)
1793
1794- A change that was introduced prior to 0.14.3 led to strange, usually
1795  fatal crashes (usually an unhandled bus error, occasionally a cryptic
1796  "can't find active area" message and a trip to the kernel debugger)
1797  under Darwin.  This was caused by an attempt to use certain Mach
1798  primitives to suspend and resume threads (the way that those
1799  primitives were used, Mach exception messages were sometimes sent
1800  twice if the first send was interrupted, and the second send occurred
1801  after the exception had already been handled (because the first send
1802  was recieved but not replied to ...)
1803
1804  1.0 backed out of this change, and used signal handling primitives
1805  (instead of Mach primitives) to suspend and resume threads.  I -think-
1806  that I understand the issue with the Mach primitives
1807  (#_thread_abort_safely isn't necessary and caused the duplicate
1808  exception messages to be sent) and have tried to revert to using
1809  the Mach thread suspension mechanisms.  (If unhandled bus errors -
1810  that exit to the shell - or cryptic "can't find active area" messages
1811  reappear, this experiment will be shown to be a failure.)
1812
1813  There are some obscure but good reasons for favoring the Mach
1814  primiitves, so it'd be good to know if the problem with using them
1815  has indeed been identified.
1816
1817  (The test case involves bad luck and bad timing: two or more
1818  threads having pending exceptions at the same time and the thread
1819  whose exception is handled first tries to suspend the others, typically
1820  on behalf of the GC.  It was possible to run stress tests for many
1821  hours in 0.14.3 without encountering the bug, and possible to
1822  encounter it under seemingly light loads.)
1823
1824- INCF and DECF argument order and fixnum arithmetic.
1825
1826  Bryan fixed some ANSI test failures related to the order in which INCF
1827  and DECF evaluate their args.  (One example is:
1828
1829  (let* ((x 3))
1830    (incf x (setq x 5)))
1831
1832  where the correct answer is 10, not 8.)  We both found that fixing
1833  some cases involving INCF caused some OpenMCL code to compile
1834  incorrectly and were nervous about introducing these changes fairly
1835  late in the development cycle, so we backed out of them prior to
1836  the 1.0 code freeze.
1837
1838  The reasons for the miscompiled code have to do with how the
1839  compiler interprets fixnum declarations under typical optimization
1840  settings.  If A and B are both declared to be FIXNUMS, then
1841  the expression
1842
1843  (setq a (+ a b))
1844
1845  will usually compile to a simple ADD instruction (with no overflow
1846  checking); if A and B are fixnums, the result will be a fixnum,
1847  though if an undetected overflow occurred in the addition, the
1848  result might be missing a significant bit.
1849
1850  There was code in OpenMCL that assumed that
1851
1852  (incf a b)
1853
1854  was exactly the same as
1855
1856  (setq a (+ a b))
1857
1858  and in fact that was true under the old (incorrect) definition of
1859  INCF.  The new definition introduced some temporary bindings:
1860
1861  (let* ((...)
1862         (#:temp (+ a b))
1863         (...))
1864     (setq a #:temp))
1865
1866  In this case, the addition was allowed to generate an overflow
1867  (no type declaration on #:temp), and the SETQ quietly violated
1868  a type declaration (assigning a non-FIXNUM value to A), leading
1869  to further problems.
1870
1871  So far, I found a couple of cases of this in the OpenMCL sources.
1872  (FWIW, both functions were originally transliterated from C code
1873  and were trying to mimic C's silent overflow behavior.)
1874
1875  Moral: if you have code that assumes that INCF or DECF expand
1876  into simple assignments and are trying to exploit the ways that
1877  those assignments interact with type declarations, you may
1878  want to review those assumptions.  If you write code that has
1879  side effects in the DELTA arguments of INCF or DECF rorms,
1880  you'll (hopefully) be pleased to see that Bryan's changes
1881  allow these side-effects to be handled correctly (at the
1882  right time.)  If you don't fall into either of these categories,
1883  you probably won't notice any difference ...
1884
1885- 64-bit Linux support
1886
1887  There's a 64-bit LinuxPPC heap image and some rudimentary (libc-only)
1888  64-bit Linux interfaces in the testing directory.
1889
1890  (Unlike 64-bit Darwin, 64-bit Linux distributions typically provide
1891  64-bit versions of "all" standard libraries; I haven't gotten around
1892  to building 64-bit gnome/gtk/X11/... interfaces yet, but wouldn't
1893  expect there to be a problem.)
1894
1895  The 64-bit Linux OpenMCL seems to basically work, but ... OpenMCL
1896  likes to map its kernel into low addresses (around #x5000); this
1897  allows compiled lisp code to use conditional branches to "short"
1898  (16-bit) absolute addresses.  Newer Linux kernels provide a
1899  "vdso" shared library that's intended to simply communication
1900  between the OS kernel and userspace libraries and programs; when
1901  a program is mapped at "non-standard" addresses, the vdso gets
1902  mapped at address 0.
1903
1904  I don't fully understand the imlications of this (beyond the fact that
1905  indirecting through a NULL pointer will access bits and pieces
1906  of the vdso instead of segfaulting.)  As far as I know, this is
1907  seen as a minor bug in the Linux kernel, and I -think- that I've
1908  seen kernel ChangeLog entries that indicate that the problem's been
1909  fixed in the relatively recent past (and will likely start to
1910  make it into Linux distributions in the near future.)
1911
1912  That said - and seeing a library at address 0 certainly makes me a
1913  little nervous - the LinuxPPC64 port seems to work at least as
1914  well as the DarwinPPC64 port does (i.e., there may be word-size
1915  or other bugs lurking around or hiding in plain sight, but it's
1916  not usually easy to encounter them.)
1917
1918- As documented (and as hasn't been true in a long time), EOF
1919  from *STANDARD-INPUT* terminates the REPL when the --batch argument
1920  is in effect (even if *STANDARD-INPUT* is a tty.)
1921
1922- QUIT does a FRESH-LINE on and FORCE-OUTPUT to the standard output
1923  stream (people had reported that output wasn't always flushed
1924  when --batch or --eval was used; 1.0 was better about this than
1925  previous versions were, but it still wasn't reliable.)
1926
1927OpenMCL 1.1-pre-051028
1928I had been doing development on G5s, and hadn't noticed that the
192932-bit lisp had been using a 64-bit instruction.  (I'm a little
1930confused about how that could have worked; perhaps the 64-bit
1931instruction gets emulated by the OS, or perhaps my model of
1932whether 64-bit instructions can be executed in 32-bit mode
1933is simply incorrect.)
1934
1935In any case, the 32-bit images produced yesterday don't run on
1936G4s (or presumably G3s or older systems.)  Ooops.  New images.
1937
1938OpenMCL 1.1-pre-051029
1939 A function used by both SET-USER-ENVIRONMENT
1940and SET-DEVELOPMENT-ENVIRONMENT wasn't properly changing saved bindings
1941of *PACKAGE*; the last few 1.1-pre releases have come up in the CCL
1942package, as a result.  Ooops again; new images, again.
1943
1944
1945OpenMCL 1.1-pre-051204
1946Not a lot of user-visible changes, but the changes that're there
1947are a little hard to bootstrap.
1948Note that new kernel build directories (darwinppc, darwinppc64,
1949linuxppc, linuxppc64, ...) repace the old versions that don't
1950have "ppc" in their names.  CVS may not prune the old directories,
1951especially if they contain files (.o, random junk).
Note: See TracBrowser for help on using the repository browser.