Opened 10 years ago

Closed 10 years ago

#597 closed enhancement (fixed)

Add nx1-combination-hook

Reported by: rongarret Owned by: gb
Priority: normal Milestone:
Component: Compiler Version: trunk
Keywords: Cc:

Description

Per the discussion on the mailing list I request that support for a user hook for ((...) ...) syntax be added to CCL. A reference implementation can be found here:

http://www.flownet.com/ron/lisp/nx1-combination-hook.lisp

Change History (10)

comment:1 Changed 10 years ago by alms

How about just adding it to the contrib directory. I don't think we're going to get consensus to add this to the CCL trunk.

comment:2 Changed 10 years ago by rongarret

Actually, so far the comments on the mailing list have been 100% positive.

The reason I want it in the code base rather than in the contrib directory is that that violates the DRY principle. The contrib code now has to be manually maintained to track the CCL source. And because changes this deep in the system are rarely advertised, doing this tracking involves a non-trivial level of vigilance.

Note also that the change I'm asking for doesn't actually change the behavior of CCL in any way. It just breaks out the fall-through case into a separate function so that its behavior can be changed by user code without having to duplicate code in nx1-combination and cheap-eval-in-environment.

comment:3 Changed 10 years ago by gb

I have no idea how I got into the habit, but I often erroneously use '%'-directives in FORMAT control strings and use '~' in C printf calls (e.g., I get it completely backwards.) I've been doing this for a long time and aware that I've been doing it for (probably) almost as long, but I still catch myself doing it from time to time.

If I was more ambitious, I'd write compiler macros on FORMAT (and ERROR ...) that at least tried to warn about this and possibly did what they could to fix it (and if I was really ambitios, I'd write a C preprocessor/frontend that did the equivalent, but I'm not that ambitious.)

Suppose that I'd at least done the first part (written some compiler macros and put them in my init file.) That might have worked at one point, but there are currently compiler macros on FORMAT (at least) that do more general and more broadly useful things. Should those system-provided compiler-macros provide some sort of *GB-INIT-FILE-HOOK* for my benefit (and promise to never overlap in functionality with what's in my init file), so that I can extend some cases of FORMAT's control-string syntax to treat % like ~ ?

There's probably a small (non-zero) chance that doing something like that would make it hard to maintain and extend the FORMAT compiler macro or increase the risk of bugs or other unintended behavior, but I think that there are other reasons that I'd answer my own question in the negative. It's quite simply wrong to use (e.g.) %s instead of ~s in a FORMAT directive, and it'd be doubly wrong to "fix" this (since there's no general way of knowing whether the % was used by mistake or just intended as a literal character. I think that it'd be even more wrong (3x, 4x ?) for the system to take steps to make this easy and wouldn't want to have to document or encourage or even justify something like this.

My bad habit of conflating ~ and % is sadly real, but the hypothetical solution - having a hook to allow "fixing" that in some cases by rewriting the control string in a compiler macro - is just that. I could maybe convince myself that having that compiler macro in my init file (along with redefinitions of everything I need to hook into to support it), since that bad habit often leads to bad behavior (getting an obscure FORMAT error when trying to describe some other condition that may be much more important), but (as I said above) I don't think that the lisp should go out of its way to ultimately make it easier for me to maintain my init file to work around a bad habit in a questionable way.

This isn't a perfect analogy with what you're proposing, but I have much the same reaction to your proposal as I think I'd have to my hypothetical solution to my FORMAT/printf confusion. I don't know if a % in a FORMAT control string is intended to be a ~, so I don't want the implementation to go out of it's way to make it possible to (misguidedly) "fix" that, and though I can't keep anyone who wants to try to "fix" that from doing so in their init file I don't think that the implementation should have to go to any lengths to provide hooks to make that easy. I don't know whether a non-SYMBOL/LAMBDA in operator position is an attempt to use the Scheme convention or is a symptom of a syntax error elsewhere (misplaced parens, etc.); I certainly get the error often enough and am not trying to follow the Scheme convention when I do so. From my point of view, your wanting to treat that as an implicit funcall isn't much different from my wanting to interpret

  (format t "The total is %d" total)

as if I hadn't done it yet again. I should clearly type "~d" when I mean "~d", and I think that people should type FUNCALL when they mean FUNCALL, and of all the places where the system should allow customization and extensibility these cases seem to be far enough down the list that it's difficult for me to see a real difference.

All that said: if you want to be able to customize the behavior of NX1-COMBINATION in your init file and do so in a way that doesn't require that CCL compromise its suddenly lofty principles by providing a hook for you to do so, you can use ADVISE. (I think that one reason that ADVISE is undocumented is that no one remembers or particularly likes its syntax, but it's been around for a long time, isn't likely to go away, and works well enough and is somewhat widely used, despite being a secret ...) As long as the signature (arglist) of an internal function like CCL::NX1-COMBINATION doesn't change, you don't have to know any details of the implementation in order to customize its behavior. (And you aren't redefining that internal function, just wrapping some code around it.) An incantation that seems to work in this case is:

(ccl:advise ccl::nx1-combination 
  (destructuring-bind (callform &rest rest) ccl::arglist
    (if (and (consp callform)
             (let* ((operator (car callform)))
               (not (or (typep operator 'symbol)
                         (ccl::lambda-expression-p operator)))))
      (apply #'ccl::nx1-combination `(funcall ,@callform) rest)
      (:do-it)))
   :when :around
   :name implicit-funcall)

As that's written, it assumes that CCL::NX1-COMBINATION is globally defined, that it takes at least one argument, and that that first argument is a "combination" (a function application or something that looks like one.) It tries to recognize and handle the case of concern to you and does something like CALL-NEXT-METHOD via the strangely-named :DO-IT macro. It's not sensitive to the implementation of NX1-COMBINATION (and in fact, the advice will remain in place if NX1-COMBINATION is redefined, IIRC; it used to get removed if the redefinition was made by loading a fasl file, but I think that that behavior changed.)

At some point in the future, CCL::NX1-COMBINATION may disappear or have its signature or semantics change. It's an internal function, and that's the nature of internal functions. It's somewhat more likely that there'll be changes to its implementation, and if you use ADVISE you don't have to worry about those changes and I don't have to worry about maintaining an explicit extension mechanism for something that I sincerely believe shouldn't be extended; that seems like a reasonable compromise.

comment:4 Changed 10 years ago by rongarret

At some point in the future, CCL::NX1-COMBINATION may disappear

Yes, I know. That is exactly why I am requesting this.

Using ADVISE is a good idea (I hadn't thought of that) but it misses the point. The point is not to get it to work with fewer lines of code. The point is to get it to work in a way that I can rely on to continue working in future releases, and through an interface that I can then lean on other implementors to provide as well.

I think that people should type FUNCALL when they mean FUNCALL

Even if that means losing customers? I don't want to get into a debate about the merits of Scheme syntax here because that will get us nowhere. But the (possibly sad) fact is that newcomers often see FUNCALL and go "ugh!" I don't have any data on how many of those "ughs" ultimately result in failures to convert, but I think it's not unreasonable to suppose that it could be a significant number relative to the size of the current user base. Is it really worth taking that risk to maintain technical purity?

That's a serious question by the way. I don't actually know what Clozure's long-term goals are, but I presume that all else being equal you'd rather have more users than less. But I could be wrong. Not everyone in the CL community wants to increase the size of the user base. But *my* long-term goals *depend* on increasing the size of the user base. I hope that will influence your decision.

comment:5 follow-up: Changed 10 years ago by rongarret

BTW, the advise solution doesn't actually work because (apparently) recursive calls from nx1-combination and cheap-eval-in-environment call the unwrapped function. So it only works for one level of nesting.

comment:6 Changed 10 years ago by alms

I certainly understand the appeal of a Lisp-1 approach. I spent several years of my life designing a Lisp-1 that I thought at the time could displace Common Lisp. That said, I don't think you get much mileage or many converts by pretending that Common Lisp is a Lisp-1. It's not, and I believe it just furthers the confusion of Lisp-1 versus Lisp-2 by trying to gloss over that fact.

CCL is appealing because it is small, fast, efficient, and encumbered by as few hacks and bells and whistles as possible. This hook is certainly interesting, and there are clearly people who will want to use it. But that doesn't mean that it should be baked into the CCL compiler. The Common Lisp standard already presents enough complex constraints on our compiler, thank you. We don't need to give Gary and Matt more things to think about and remember and maintain while they're working on CCL. Yes, putting something in the contrib directory means that the people who like something may need to update it occasionally to keep it working. Frankly, that seems more appropriate to me than asking people who don't find it appealing to maintain it. The beauty of open source software is that you can do this yourself. The price of open source software is that you may have to.

comment:7 in reply to: ↑ 5 Changed 10 years ago by gb

Replying to rongarret:

BTW, the advise solution doesn't actually work because (apparently) recursive calls from nx1-combination and cheap-eval-in-environment call the unwrapped function. So it only works for one level of nesting.

Ron:

I'd appreciate it if you can remember/provide a concrete example where the advised version of NX1-COMBINATION doesn't work and the hook-based version does.

With the advised version above in effect,

(defun foo (x)
  (let* ((fns #(1+ 1-)))
    ((aref fns 0) ((aref fns 1) x))))

behaves as I'd expect it to.

Note that in general calls to a function named NAME from inside advice on NAME simply call the global definition of NAME (which, in a case like that above, is the advised version. (In other words, it's certainly possible that the ADVISE-based definition above fails in some case(s), but it seems more likely that that'd be because that definition's incorrect than because of the behavior of ADVISE.)

comment:8 Changed 10 years ago by rongarret

This is the example I'm using that doesn't work. It's a factorial using an inline Y-combinator. If you don't want to deal with this mess I can boil it down to something simpler for you.

(defmacro λ (args &body body)
  `(lambda ,args
     (flet ,(mapcar (lambda (arg) `(,arg (&rest args) (apply ,arg args))) args)
       ,@body)))

(((λ (f) ((λ (g) (g g)) (λ (h) (λ (x) ((f (h h)) x)))))
  (λ (f) (λ (n) (if (zerop n) 1 (* n (f (1- n)))))))
 15)

Here's a transcript demonstrating the problem:

Welcome to Clozure Common Lisp Version 1.5-dev-r13340M-trunk  (DarwinX8664)!
? (defun combination-hook (form) (cons 'funcall form))
COMBINATION-HOOK
? (advise ccl::nx1-combination 
  (destructuring-bind (form &rest rest) ccl::arglist
    (if (and (consp form)
             (let* ((operator (car form)))
               (not (or (typep operator 'symbol)
                        (ccl::lambda-expression-p operator)))))
      (apply 'ccl::nx1-combination (combination-hook form) rest)
      (:do-it)))
  :when :around
  :name combination-hook)
#<Compiled-function (CCL::ADVISED 'CCL::NX1-COMBINATION) (Non-Global)  #x3020015D49FF>
? (advise ccl::cheap-eval-in-environment 
  (destructuring-bind (form &rest rest) ccl::arglist
    (if (and (consp form)
             (let* ((operator (car form)))
               (not (or (typep operator 'symbol)
                        (ccl::lambda-expression-p operator)))))
      (apply 'ccl::cheap-eval-in-environment (combination-hook form) rest)
      (:do-it)))
  :when :around
  :name implicit-funcall)
#<Compiled-function (CCL::ADVISED 'CCL::CHEAP-EVAL-IN-ENVIRONMENT) (Non-Global)  #x30200163677F>
? (defmacro λ (args &body body)
  `(lambda ,args
     (flet ,(mapcar (lambda (arg) `(,arg (&rest args) (apply ,arg args))) args)
       ,@body)))
Λ
? (((λ (f) ((λ (g) (g g)) (λ (h) (λ (x) ((f (h h)) x)))))
  (λ (f) (λ (n) (if (zerop n) 1 (* n (f (1- n)))))))
 15)
> Error: Car of ((Λ (F) ((Λ (G) (G G)) (Λ (H) (Λ (X) ((F (H H)) X))))) (Λ (F) (Λ (N) (IF (ZEROP N) 1 (* N (F (1- N))))))) is not a function name or lambda-expression.
> While executing: CCL::CHEAP-EVAL-IN-ENVIRONMENT, in process Listener(7).
> Type cmd-. to abort, cmd-\ for a list of available restarts.
> Type :? for other options.
1 > 
? (unadvise ccl::nx1-combination)
((CCL::NX1-COMBINATION :AROUND COMBINATION-HOOK))
? (unadvise ccl::cheap-eval-in-environment)
((CCL::CHEAP-EVAL-IN-ENVIRONMENT :AROUND IMPLICIT-FUNCALL))
? (require :nx1-combination-hook)
;Loading #P"/Users/ron/devel/Lisp code/nx1-combination-hook.lisp"...
#<Package "CCL">
*NX1-TARGET-INHIBIT*
NX1-COMBINATION
NX1-COMBINATION-HOOK
NX1-COMBINATION-HOOK
CHEAP-EVAL-IN-ENVIRONMENT
CHEAP-EVAL-COMBINATION-HOOK
CHEAP-EVAL-COMBINATION-HOOK
:NX1-COMBINATION-HOOK
("NX1-COMBINATION-HOOK")
? (((λ (f) ((λ (g) (g g)) (λ (h) (λ (x) ((f (h h)) x)))))
  (λ (f) (λ (n) (if (zerop n) 1 (* n (f (1- n)))))))
 15)
1307674368000
? 

comment:9 Changed 10 years ago by gb

Ah. Using ADVISE to redefine NX1-COMBINATION works fine. In compiled code:

? (defun example ()
    (((λ (f) ((λ (g) (g g)) (λ (h) (λ (x) ((f (h h)) x)))))
      (λ (f) (λ (n) (if (zerop n) 1 (* n (f (1- n)))))))
     15)
EXAMPLE
? (example)
1307674368000

The same approach fails for CHEAP-EVAL-IN-ENVIRONMENT, not because of ADVISE's behavior per se, but because the unwrapped CHEAP-EVAL-IN-ENVIRONMENT calls itself recursively and those self-calls ordinarily compile to simple CALL or JMP instructions. (The same optimization prevents TRACE from working as people expect when self-calls are involved.)

Whatever one thinks of those tradeoffs in general, it seems pretty clear that the benefits of being able to fully TRACE/ADVISE CHEAP-EVAL-IN-ENVIRONMENT outweigh the cost of doing an indirection through the symbol's function cell on those self-calls. I changed this in r13343, and your example works as expected in evaluated code with that change in effect.

comment:10 Changed 10 years ago by rongarret

  • Resolution set to fixed
  • Status changed from new to closed

That works. Thanks!

Note: See TracTickets for help on using tickets.