Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#555 closed defect (fixed)

Two top-level command system bugs

Reported by: rongarret Owned by:
Priority: minor Milestone:
Component: other Version: trunk
Keywords: Cc:

Description

1) TLC'c interact badly with symbol macros. For example:

? (define-symbol-macro pwd 1)
PWD
? pwd
#P"/"
? 

This wouldn't be so bad except that 2) there also seems to be a bug in the top-level command processor for TLCs that take arguments, e.g.:

? :cd "/"
#P"/"
? cd "/"
Too few arguments in call to #<Compiled-function CCL::CD (Non-Global)  #x3000405443CF>:
0 arguments provided, at least 1 required. 
? 
"/"
? 

TLC could use some documentation as well.

Change History (13)

comment:1 Changed 10 years ago by gb

In their original incarnation, toplevel commands were named by keywords. A parenthesized form whose CAR is a keyword which names a toplevel command was treated by the REPL as a call to the function associated with that command (with the rest of the form's elements provided as arguments.)

Parentheses became optional in the common case where the command name (keyword) and any arguments were provided on the same line (with the keyword the first non-blank token on the line.)

These conventions meant that:

  • it wasn't possible to call functions whose names were keywords from the REPL (though one could do something like '(funcall :funny-function 1 2 3)')
  • keywords that would otherwise just be self-evaluating when typed into the REPL might instead invoke some command (e.g., :POP in a break loop exited the break loop, while :FOO returned itself.)

Neither of these things seem to conflict with useful functionality.

At some point, a control variable - CCL::*TOPEVEL-COMMANDS-DWIM* was introduced and defaulted to T. When this variable is true, it (among other things) removes the restriction that a symbol has to be a keyword in order for it to be interpreted as the name of a toplevel command, and this has the unfortunate consequence of introducing ambiguity and also (as you note) doesn't handle arguments correctly.

I don't remember when or why this was introduced, but the fact that the variable name contains the well-known acronym for "Do What Is Moronic" suggests that it's a fairly clear example of why customers shouldn't be in the lisp design business. (To be fair, I don't know the whole story, and this could be something where the design is sounder than it appears to be and the problems are just on the implementation side; it's possible that something was merged incorrectly or incompletely, or something like that.)

I'd recommend setting CCL::*TOPLEVEL-COMMANDS-DWIM* to NIL until that can be resolved. With that variable set to NIL,

? :cd "/"
and
? (:cd "/")

are equivalent ways of invoking the toplevel command that tries to change the current directory, and:

? cd "/"           ; A
and
? (cd "/")         ; B

are (A) a reference to a variable/symbol-macro named "cd" followed by a self-evaluating form and (B) a call to a function or macro named CD.

comment:2 Changed 10 years ago by gz

  • Resolution set to fixed
  • Status changed from new to closed

1) is fixed in r12403. It was just a bug, not some deep design flaw.

2) is a little more complicated. Basically the kludge that interprets sequences of toplevel expressions on one line as a single command only works when the line starts with a colon. So

? cd "/"

is read as two separate expressions, a variable/symbol-macro named "cd" followed by a self-evaluating form "/". Since cd is unbound, this wasn't going to do anything useful regardless of the setting of *toplevel-commands-dwim*. With dwim turned on, it interprets cd as a command (which is how you meant for it to be interpreted) but the command doesn't see the form you meant to be the argument.

The bottom line is that for multi-argument commands, you either have to type the colon or you have to type the parentheses, i.e. either of these will work:

? :cd "/"
or
? (cd "/")

I'm going to close this bug, claiming this is as designed. If you'd like to add a request to make

? cd "/"

work as well, please enter a separate ticket for that.

comment:3 Changed 10 years ago by gb

I respectfully suggest - with no sniping intended - that we not have CCL::*TOPLEVEL-COMMANDS-DWIM* enabled by default.

comment:4 Changed 10 years ago by gz

Why?

comment:5 Changed 10 years ago by gb

I can't easily explain (and can barely understand) its behavior, and I can't easily justify its existence: as far as I can see, it's just additional clutter that seems to make things unnecessarily harder to understand.

Why should we keep it enabled (i.e., what good does it do anyone) ?

I honestly don't know.

comment:6 Changed 10 years ago by gz

Here's how I'd explain it: There are commands, they have names you type at the toplevel prompt (sort of like unix). If there is a global definition of the same name as a built-in command, that definition shadows its use as a command. In that case, you can still access the command by typing colon in front of the name. Some people prefer to always type their commands using a colon (sort of like ITS). That's allowed too.

I use this all the time, i.e. I don't generally type a colon in front of commands any more. Customers have requested not having to type the colon, and there are at least a couple dozen people who use it every day. Apparently Ron discovered it and has been using it it too. It causes no confusion except for the case of the (undocumented) single-line kludge. If that's the big problem, I'll just fix it.

comment:7 Changed 10 years ago by rongarret

For the record, I was unaware of the entire existence of the top-level-command system (I thought it was only active in break loops) until I tried to do something like:

(define-symbol-macro cd () (change-directory (if (listen) (read) "~")))

I like not having to type the colon. I want to move all my commonly used unix shell commands into Lisp so I don't have to have a separate terminal window open to do greps and whatnot. But symbol macros should properly shadow TLCs (I presume this is the bug that has been fixed).

And not that I want to spin this discussion off on a wild tangent, but it should be noted that something very like TLCs can be implemented in portable ANSI CL thusly:

(let ( (r (copy-readtable nil)) )
  (defun read-symbol (stream)
    (let ( (*readtable* r) )
      (read-preserving-whitespace stream))))

(defun prefix-funcall-reader (stream char)
  (unread-char char stream)
  (let ( (s (read-symbol stream)) )
    (if (eql (peek-char nil stream nil nil) #\()
        (cons s (read stream))
      s)))

(defun symbol-reader-macro-reader (stream char)
  (unread-char char stream)
  (let ( (s (read-symbol stream)) )
    (let ( (f (get s 'symbol-reader-macro)) )
      (cond ( f (funcall f stream s) )
            ( (eql (peek-char nil stream nil nil) #\()
              (cons s (read stream)) )
            (t s)))))

(map nil (lambda (c) (set-macro-character c 'symbol-reader-macro-reader t))
     "ABCDEFGHIJKLMNOPQRSTUVWXYZ")

(map nil (lambda (c) (set-macro-character c 'prefix-funcall-reader t))
     "abcdefghijklmnopqrstuvwxyz")

(defun set-macro-symbol (symbol readfn)
  (setf (get symbol 'symbol-reader-macro) readfn)
  t)

(defun make-special-form-reader (nargs)
  (lambda (stream symbol)
    (cons symbol (loop for i from 1 upto nargs collect (read stream)))))

#|
Examples:

(set-macro-symbol 'while (make-special-form-reader 2))
(set-macro-symbol 'setf (make-special-form-reader 2))

; Note that symbol reader macros are currently case-senstitive
Setf x 1
While (< x 10) print(incf(x))

; For C fans:
list(1 2 3)
cons('car 'cdr)
|#

comment:8 follow-up: Changed 10 years ago by gb

I guess kludges are in the eye of the beholder; I find that yet another set of undocumented context-sensitive rules (enabled by an undocumented unexported special variable) fits my definition of "kludge" as well as anything that I can think of.

If I look at (or provide) a listener transcript that contains:

1 > (:cd "/")

then presumably all parties involved know that the :CD toplevel command is being invoked. If we instead have:

1 > (cd "/")

then we have to know whether or not there's a CD function in order to know whether that function or the built-in command is being invoked. (I don't think that it's necessarily that unlikely that command names would match function names - I think that it's probable, in fact; there seem to be 7 commands whose name match the names of functions in CL and CCL - but which additional conflicts exist will vary from user to user and likely change over time (e.g., "I wasn't even thinking about the REPL command when I named some function KILL, but now that I did that I call it by accident every time I try to kill a process ...")

I can't imagine why anyone would want the additional overhead of having to think about whether or not they need to type a colon (just so that they can - if they remembered correctly - avoid whatever horrors are supposedly associated with "having" to type that colon.)

I can't quite wrap my head around the idea of "the convenience of not having to type a colon", and I'm a long ways from understanding "the convenience of not having to type a colon, except when you have to." This makes a lot less sense to me - especially as default behavior - than what had been the status quo ("commands are named by keywords; since functions are rarely if ever named by keywords, that's pretty damned easy to understand. Period.")

Just so that I'm sure that I understand, is

1 > (POP)

a malformed call to the CL:POP macro, or a colonless invocation of the nullary :POP break-loop command ? How would anyone know/guess what was intended here ?

I suspect that our sensibilities about this sort of thing are very different.

It's possible that there are users who are used to different systems (which have different issues) and who never do anything that exposes or introduces ambiguity. What the default setting of CCL::*TOPLEVEL-COMMANDS-DWIM* might offer some fuzzy/blurry approximation of what some people are used to/expect, but there seem to be enough problems with it that I think that the current default is inappropriate.

FWIW: the kludge which makes parens optional when keywords are used has been present for 4-5 years. Like many things, it was only documented in the release notes, but I don't know of anyone who's aware of it who didn't quickly get out of the habit of typing unnecessary parens. Whether one likes it or not, it has simple, predictable behavior that's not dependent on the set of functions/variables/symbol-macros/declared specials/whatever else we currently think or someday will think affect the colon-optional kludge.

comment:9 follow-up: Changed 10 years ago by rongarret

I guess kludges are in the eye of the beholder;

Indeed.

I find that yet another set of undocumented context-sensitive rules (enabled by an undocumented unexported special variable) fits my definition of "kludge" as well as anything that I can think of.

And I would agree. But what does that have to do with the topic at hand? *readtable* is both exported and documented, so I'm not sure what you're referring to.

I suspect that our sensibilities about this sort of thing are very different.

I suspect so. Maybe I don't understand what the point of TLCs is even supposed to be. I always thought (seriously) that the whole point was so that you could do things that acted like function calls but without having to type parens. If typing parens is not a concern, why even bother with TLCs? Why not just have utility functions/macros that do what TLCs do? You'd have to come up with a different name for the POP command, but so what? Call it BPOP or STACK-POP or something like that. The QUIT function is a good example. It's a function, not a TLC, but it does something that strongly resembles the kind of things that TLCs do. Why not implement all TLC functionality that way?

(Should we maybe move this discussion out of Trac and onto the mailing list?)

comment:10 in reply to: ↑ 8 Changed 10 years ago by gz

Replying to gb:

What can I say, I like this feature of not having to type a colon. It's been in the trunk for over a year, and aside from this now-fixed bug with symbol macros, as best we can tell it hasn't bothered or confused you or anybody else in all that time. It's there for people who use it, and it's as close to being invisible to people who don't use it as you can get.

Your arguments don't make a case against allowing the feature as much as against liking it. They don't make me want to stop using it, except that I do see your point about the confusion in function call syntax. In fact, having

 ? (cd "/")

invoke the cd command plays no part in what's useful about the dwim, so I'll go ahead and remove support for that.

comment:11 in reply to: ↑ 9 Changed 10 years ago by gb

Replying to rongarret:

I find that yet another set of undocumented context-sensitive rules (enabled by an undocumented unexported special variable) fits my definition of "kludge" as well as anything that I can think of.

And I would agree. But what does that have to do with the topic at hand? *readtable* is both exported and documented, so I'm not sure what you're referring to.

You're right that this isn't the place for it, but gz and I were discussing whether or not CCL::*TOPLEVEL-COMMAND-DWIM* should continue to default to T or not.

comment:12 Changed 10 years ago by gb

Beating a dead horse in reply to gz:

Replying to gb:

What can I say, I like this feature of not having to type a colon. It's been in the trunk for over a year, and aside from this now-fixed bug with symbol macros, as best we can tell it hasn't bothered or confused you or anybody else in all that time. It's there for people who use it, and it's as close to being invisible to people who don't use it as you can get.

The fact that we haven't heard of people getting screwed by this doesn't make me think that this is a good idea. (It's certainly true that I just don't like this, but I think that that dislike is based on thinking that this is a bad idea; I don't think that I have any aesthetic attachment to typing colons, beyond the fact that doing so eliminates ambiguity.)

I agree that it isn't too likely that someone would have a global variable or global symbol macro named SOME-PACKAGE:Q, but if they do and they type

1 > q

in a break loop and aren't in SOME-PACKAGE, I don't think that we can guess whether they were trying to evaluate that variable/symbol-macro or exit to toplevel. Guessing wrong (guessing that they meant to exit the break loop when they didn't, invoking a symbol-macro that has side-effects when that wasn't intended) seems to be pretty clearly wrong (and the fact that this scenario is unlikely doesn't make it seem less wrong.)

There's obviously a similar case involving ... oh, let's say a function name SOME-PACKAGE:KILL being called from a break loop (when *PACKAGE* isn't SOME-PACKAGE but the user might think it is.)

We don't know whether:

1 > (kill 3)

is an attempt to kill Dwarf #3 in some game or Thread #3, and we seem to have a 50% chance of guessing wrong. (You may have said that guessing is undesirable in apparent function calls; whether or not we believe that for the same reason, I'd agree.)

If we're really only concerned about the cases that don't look like function calls, then I'd find this easier to swallow if we made it less heuristic and said something like:

  1. if you type a symbol into the repl and its name matches the name of a command that's in effect, it's always interpreted as a command when *TOPLEVEL-COMMANDS-DWIM* is true.
  2. if you want to evaluate a variable or symbol macro whose name matches the name of a command in that case, you have to wrap a PROGN around the reference.
1 > (progn q)

is certainly a bit awkward, but at least we know that means. (We could explain this requirement in the :? help text.)

comment:13 Changed 10 years ago by rongarret

FWIW, now that I can shadow TLCs with symbol macros, I am happy with the status quo. I suggest we table this discussion until we all run out of other things to do. :-)

Note: See TracTickets for help on using tickets.