Ticket #1117 (closed defect: invalid)

Opened 10 months ago

Last modified 9 months ago

Missing external-process-wait on Windows

Reported by: fare Owned by:
Priority: normal Milestone: Clozure CL 1.10
Component: Runtime (threads, GC) Version: trunk
Keywords: Cc:

Description

The function ccl::external-process-wait is missing on #+windows-target.

Looking at the source for run-program, it looks like this might work:

(in-package :ccl)

(defun external-process-wait (proc)
  (when (external-process-pid proc))
    (with-interrupts-enabled
        (wait-on-semaphore (external-process-completed proc)))))))

Change History

comment:1 Changed 10 months ago by fare

Actually, there better be a test in that function that the process hasn't finished yet before you wait for it to finish.

comment:2 Changed 10 months ago by gb

There are probably lots of other functions named by unexported symbols that you shouldn't be referencing either. What's interesting about this symbol that apparently makes you think that its definition won't change or disappear in the future ?

It's possible that you could argue that CCL::EXTERNAL-PROCESS-WAIT should be exported. We'll never know unless you try to make that argument; if you just note that some random internal symbol isn't defined in CCL on Windows, why should anyone care whether it is or not ?

comment:3 Changed 10 months ago by fare

Well, whether this particular function is exported or not, *some* function must be exported that allows to synchronize on a process being completed so we can be confident that we've got all the output and extract its exit code.

Otherwise, the entire functionality of run-program without :wait t is useless.

comment:4 Changed 9 months ago by fare

Of course, my expression above is misparenthesized. It should be:

(defun external-process-wait (proc)
  (when (external-process-pid proc)
    (with-interrupts-enabled
        (wait-on-semaphore (external-process-completed proc)))))))

And the when seems precisely designed to protect against the process having already stopped. Or not.

comment:5 Changed 9 months ago by gb

The intended use of (PROCESS-RUN-FUNCTION ... :WAIT NIL) is something like:

(defvar *bc* (run-program "bc" () :input :stream :output :stream :error :output :wait nil))
*BC*
? (external-process-status *bc*)
:RUNNING
NIL
? (let* ((in (external-process-input-stream *bc*)))
  (write-line "3 + 4" in)
  (force-output in))
NIL
? (values (read-line (external-process-output-stream *bc*)))
"7"
? (let* ((in (external-process-input-stream *bc*)))
  (write-line "halt" in)
  (force-output in))
NIL
? (external-process-status *bc*)
:EXITED
0
? (let* ((out (external-process-output-stream *bc*)))
  (read-line out))
> Error: Unexpected end of file on #<BASIC-CHARACTER-INPUT-STREAM UTF-8 (PIPE/8) #x3020005A273D>

You're certainly welcome to see that as "useless", but I'm equally welcome to dismiss that opinion as being (at least) uninformed.

Since we now have evidence that characterizing something as useless without understanding it is ... stupid, I'll just say that I haven't been able to understand how or when running a process without waiting for it to terminate and then waiting for it to terminate is generally useful.

If one runs a process "interactively" (without waiting for it to terminate) and wants to read its output in a straightforward way, then the stream by which one reads that output pretty much has to be an "interactive stream" (one which doesn't confuse "no data available at the moment" with EOF); STRING-STREAMs and FILE-STREAMs generally aren't "interactive streams". This is the only context in which I can understand the claim that it's somehow necessary to do a blocking wait for process termination in order to reliably read its output. My example seems to demonstrate otherwise.

comment:6 Changed 9 months ago by fare

There are *some* useful cases for interactive use without synchronizing on termination -- or, in the case you demonstrated, by synchronizing on the incidental closing of a pipe. However, a pipe is not always available, nor is the closing necessarily the sign that the process died. In the case that the output is to a file, if you can't detect that the process died, you'll never know if it's still going to write more stuff to the file or not. And you'll never know if it returned a success exit-code or not, which is crucial information.

comment:7 Changed 9 months ago by gb

Showing that reading from the stream returns EOF was in response to the claim that one had to wait for process termination in order to know whether or not output was available. I'm sorry if that wasn't clear; I have no idea how to make that point any more clear or any simpler.

If the output is being directed to a file ... well, read the last paragraph of my previous entry.

The only way that I can interpret your comments and claims that makes any sense to me is that you think that ":wait nil" is intended to be something like "run in background and let me synchronize later". It isn't. If I tell you that "the indended use of :wait nil is ...." and that elicits the grudging response that "that could be useful, but ..." then it seems that we Just Aren't Communicating Here.

I'm now guessing that you believe that the intended use of ":wait nil" is something else; I wish that I didn't have to guess here, because that risks wasting even more of your time and mine. I'm further guessing that what you really want is to run the process without waiting for it to complete but to reliably know when it's completed (so that non-interactive output can be read and for whatever other reasons); that can be done reasonably using documented functionality.

(run-program "sleep" '("5") :wait nil)

will obviously sleep for a little more than 5 seconds and then exit. If I cared when it completed, I'd likely need to busy-wait by calling EXTERNAL-PROCESS-STATUS repeatedly (which may interfere with whatever I was doing instead of waiting.)

Rather than busy-waiting/polling for the process status to change (or saying "just kidding when I said :wait nil, I changed my mind and want to wait after all"), you can ask to be notified whenever the external-process's status changes:

(defun status-changed (p)
  (format t "~& status of ~s changed" p))

(run-program "sleep" '("5") :wait-nil :status-hook #'status-changed)

Unlike the previous version, that'll print a message after sleeping for ~5 seconds (and after the process's status changed from :running to :exited). The hook function can do arbitrary things (raising a semaphore, interrupting a thread, ...) to notify interested parties of that status change. The hook function runs in an arbitrary thread (a thread used to monitor the external-process's status and manage its I/O) and the function is expected to do whatever it does in bounded time (not to sit around waiting to enter a break loop in response to an error.)

If this scenario (running an external process without waiting for it to complete but knowing when it does complete) isn't what you're talking about, then I have to confess that I'm unable to guess. If not and you want to clarify that, please feel free to do so; please explain that rather than some artifact (like "random internal function isn't implemented on windows", ":wait nil is useless", "well, the intended use of :wait nil isn't useless, but the secret use that I have in mind is") and feel free to assume that your audience has no way of knowing what you're talking about unless you try to explain that.

comment:8 Changed 9 months ago by fare

When I run a process in the background, I most of the time eventually want to get its termination code, so I know whether I can trust its results, or have to restart and try again or otherwise warn the user that something went wrong and where.

All other Lisps with the ability to spawn a background process provide this functionality, and for good reason. I am hoping to be able to tell people to "just use common lisp" as a replacement for shell interaction, but obviously, this isn't going to be possible if every single Lisp implementation has issues, and authors fail to acknowledge a defect. (SBCL gets argument quoting wrong on Windows; Allegro gets argument parsing wrong, I don't know about quoting; many implementations won't let you do it in the background, so I have to work around that somehow.)

comment:9 Changed 9 months ago by fare

And yes, looking at the source, it seems that indeed a convoluted use of status-hook and a semaphore can probably reimplement the feature on top of the API. My apologies for missing this hook.

(Oh, my, sleeping and polling — Googling for a better way, shouldn't you be using WaitForSingleObject? to wait for process termination? It is also necessary to survive the confusing case when the process returns with status 259.)

comment:10 Changed 9 months ago by gb

  • Status changed from new to closed
  • Resolution set to invalid
Note: See TracTickets for help on using tickets.