Opened 12 years ago

Last modified 12 years ago

#201 new defect

Loading [CR-terminated] file creates weirdly bogus code

Reported by: alms Owned by: gb
Priority: minor Milestone:
Component: IDE Version:
Keywords: Streams Cc:


The attached file demonstrates a strange bug in file loading. To duplicate the bug:

  • Unzip attachment
  • Start CCL, open tab-bug.lisp, and choose the "Load Buffer" menu item.
  • Go to the listener and type (foo) - you'll see that the ~nT format directive does not insert the right number of spaces.
  • Click back on the editor file, select and re-evaluate foo by Enter or Command-E.
  • retype (foo) in the listener, hey presto formatting is ok!
  • Go back tab-bug.lisp and choose "Load Buffer" again and you will once again have the bogosity.

If you copy/paste the contents of tab-bug.lisp into a new window and save that window, the contents of that window can be loaded and the formatting is correct.

Attachments (1) (640 bytes) - added by alms 12 years ago.

Download all attachments as: .zip

Change History (6)

Changed 12 years ago by alms

comment:1 Changed 12 years ago by gz

  • Priority changed from major to minor

Diagnosis (summarizing off-list discussion):

The file uses #\Return to terminate lines, which is the old mac line-end convention. FOO uses a format string in the file with an embedded newline, which in the case of this file is an embedded Return. When the format string is processed, the Return is written to the output stream, which uses the default #\Linefeed line termination, the new mac convention. Hence it doesn't recognize the Return from the format string as the start of a new line, so it doesn't reset the column position for ~T.

The Load Buffer command loads the raw file, and has the problem described above. The editor converts Return-terminated files into Linefeed-terminated files for the duration of editing. Hence Command-E/Enter, which read from the editor, do not have the problem, ditto with saving a copy of the text copied from the editor.

The easiest way to deal with this issue is to just convert your files to be Linefeed-terminated, as this is the wave of the future. This can be done manually by the copy-and-save method above, or en-masse with a simple lisp program.

The drawback of this approach is that MCL doesn't deal well with the resulting files, which it calls "Unix" files. This can be improved by putting the following in your mcl-init.lisp file:

(setq ccl::*do-unix-hack* t)
(setq ccl::*linefeed-equals-newline* t)

I'll leave this bug open in case we decide to try harder to support Return-terminated files, but I'm downgrading the priority.

comment:2 Changed 12 years ago by gz

  • Component changed from IDE to Runtime (threads, GC)
  • Keywords Streams added
  • Summary changed from Loading file creates weirdly bogus code to Loading [CR-terminated] file creates weirdly bogus code

comment:3 Changed 12 years ago by gb

The lisp does support CR-terminated files, but (a) part of that support (the function CCL::INFER-LINE-TERMINATION) was broken and (b) the IDE doesn't yet do anything which would make it easy to use this support. (And (c), some documentation describing the support seems to have been lost.)

It seems obvious to me that the IDE should expose this support (or at least be more aware of it.) The "Load Buffer" menu item can certainly tell how lines in the file were encoded; other things can ask you. Other people may disagree (and possibly suggest that we have a vote on what the default line termination should be ...), but this seems pretty clear to me.


(load "/path/to/tab-bug.lisp" :external-format :macos)

would have "worked" (and would work for the vast majority of cases.)

I'd agree that the convention of using carriage-returns to terminate lines (as MCL does) is not in very widespread use anymore, and that converting your files to use linefeed as a line terminator is probably advisable. Windows uses yet another line-termination convention, so it may be the case that similar issues would arise when dealing with code that originated on/has to be used on Windows.

It's difficult for automated tools to handle line-termination issues reliably in all cases; it is possible and simple for tools (including the IDE) to handle a very high percentage of all cases reliably and to help you, and I think that it should do so.

comment:4 Changed 12 years ago by gb

  • Component changed from Runtime (threads, GC) to IDE

I think that this is an IDE issue.

comment:5 Changed 12 years ago by ds

FWIW, from my side I'm happy with the status quo and copy/paste solution now I know it's just old MCL files doing this. Thanks, Dimitri

Note: See TracTickets for help on using tickets.