Luke Gorrie luke@bluetail.com writes:

Maybe a better idea would have been to go for chaos? We send N requests to Lisp at a time, and it replies in any order (with an ID tag to identify the result), we can get multiple unknown threads hitting the debugger at the same time, etc.

I tried to remove the state machine some time ago, but wasn't very successful. The id argument in eval-string stems from that attempt.

Here some problems I encountered:

The current state stack contains (state-name . variables) pairs. It wasn't obvious to me where I should store the variables instead. Including the variables in the request itself doesn't work, because the variables contain things like window configurations. Storing them in a buffer -- e.g., the variables of the debugging state in the sldb buffer -- doesn't work because the user can delete the buffer. So I stored them on a stack again :-)

Another problem was that there where no 'activate' events anymore. When you exited from the debugger at level 3 to level 2, Emacs didn't notice it. My idea was to generate the activate events at the Lisp side. So I changed the Lisp sldb-loop and sent an 'activate-debugger' event every time before reading the next request. The sldb-loop looked like this:

(send-to-emacs '(:initialize-debugger ...)) (loop (send-to-emacs '(:activate-debugger ...)) (read-from-emacs))

This too was problematic, because the :initialize-debugger event could cause a RPC from Emacs to Lisp (e.g., fetching the backtrace). But the result of the RPC arrived after the :activate-event. I gave at this point.

I think we cannot blindly accept every event at any time. Maybe we need something like a mailbox with Erlang-style selective receive :-)

Helmut.

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

The sldb-loop looked like this:

(send-to-emacs '(:initialize-debugger ...)) (loop (send-to-emacs '(:activate-debugger ...)) (read-from-emacs))

This too was problematic, because the :initialize-debugger event could cause a RPC from Emacs to Lisp (e.g., fetching the backtrace). But the result of the RPC arrived after the :activate-event. I gave at this point.

How about making it look like:

(loop (send-to-emacs `(:debug ...)) (read-from-emacs))

i.e. just like today except with the :DEBUG message moved into the loop. This feels okay so far (after 5 minutes..)

I think we cannot blindly accept every event at any time. Maybe we need something like a mailbox with Erlang-style selective receive :-)

Do you have more bad cases in mind already? I wanna wanna wanna make this work, it could solve all our interrupt/thread/etc woes.

I've attached my currently hacked patch just for your info / trouble spotting :-)

Luke Gorrie luke@bluetail.com writes:

Oops.. now I see why you split activate and initialize.

Next incarnation. Pushes the "do we need to reinit *sldb*?" logic from Emacs into Lisp:

(defvar *sldb-level-in-emacs* nil)

(defun sldb-loop (level) (flet ((activate-emacs () (send-to-emacs (list* :debug *sldb-level* (debugger-info-for-emacs 0 *sldb-initial-frames*))) (setq *sldb-level-in-emacs* level))) (activate-emacs) (unwind-protect (loop (catch 'sldb-loop-catcher (with-simple-restart (abort "Return to sldb level ~D." level) (unless (eql level *sldb-level-in-emacs*) (activate-emacs)) (handler-bind ((sldb-condition #'handle-sldb-condition)) (read-from-emacs))))) (send-to-emacs `(:debug-return ,level)))))

I'll continue trying to beat it into submission, and try to properly handle e.g. user killing *sldb* at an inconvenient time, tomorrow.

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

A single variable for slime-read-string-tag could cause some trouble. Reading from the REPL can be invoked recursively, because the user can evaluate arbitrary code, e.g, with C-c C-:.

True, thanks.

I've been working on getting it through the test suite. I'm down to two failing tests, but there are other missing things (like this read-string bit).

For read-string maybe the solution is for Emacs to just send (repl-input <string>) and let Lisp decide what to do with it.

I also found an exciting new class of dastardly hard bug :-) example:

Make async request Make sync request A Async reply arrives, in continuation: Make sync request B Result of A arrives and is thrown up the stack past B's handler

Can involve a lot of "what happened to A?" head-scratching :-)

With the old protocol we didn't have this problem since it was illegal to make two requests in a row. On the other hand, this wasn't as hard to debug as an `activate'-related bug in the state machine once, and I _think_ it can be solved by the rule "don't make synchronous requests from asynchronous continuations".

Currently the "interesting" bit is figuring out what slime-sync, slime-busy-p, etc are supposed to do, or how to change their callers. This is the chance to make it compatible with out-of-order (e.g. thread-per-request) operation.

Lots of code deleted! That's good :-)

Let's hope it can stay deleted :-) will see how far I can get tonight.

Current patch attached. It contains "more than a couple" of debug-printfs, mostly as a relic of chasing the aforementioned bug :-)

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

How about this rule: "RPCs should be properly nested"? So a sequence

request A request B reply A reply B

would be illegal. This would be easy to check (with a stack) and is probably a requirement for synchronous RPCs. The downside of this rule is, is that it looks a lot like what we had before :-(

Some comforting observations:

Synchronous RPCs mostly take care of this themselves by blocking Emacs. The only way to make two is if the outer one breaks into a recursive edit, which will probably happen due to needing to be debugged, where it's probably safe to make inner synchronous RPCs.

Synchronous commands always have to be able to cope with abort because of C-g. So if we just abort them when the stack returns out of order, nothing really bad should happen, the command just gets aborted.

I'm guessing we can program around the problems pretty easily. But you probably noticed the ol' PowerPoint trick of using a bullet-list for an argument that I can't express convincingly in prose :-)

Allowing "pipelining" multiple requests to Lisp and accepting results out-of-order would open a lot of flexibility for threading. For example with thread-per-request you could do `C-c C-k' and still use M-. and so on while you wait for the compilation. I *totally* missed that idea in my previous multithreading attempts but now I suspect it's the most important bit.

Have you any plans for how to debug in the thread-per-request world?

Some ideas at least. We extend the (:DEBUG ..) message with a THREAD-ID so that we can give threads their own *sldb* buffers. We also extend RPCs from Emacs to include a THREAD-ID so that the *sldb* buffers can talk to the right thread (and so the REPL can always talk to the same thread, and maybe commands in buffers could use T to mean "in a new thread" perhaps).

I think that's actually enough for the machinery, since now the debugger state only exists in Lisp stacks (per thread) and the *sldb* buffer-locals (per thread). We could then build a user interface on top that decides when debug buffers should actually be popped up etc.

Or something :-)

Luke Gorrie luke@bluetail.com writes:

Ahoy,

I have checked in my majorly hacked version on a new branch called `stateless-emacs'. If anyone wants to take it for a spin that would be helpful for flushing out problems not covered by the test suite. There is a STATELESS-EMACS tag set at the point where I branched.

I'm working on this version and changed the CMUCL backend to use signal driven IO. This means that M-. works now even during a compilation. It's even possible to compile when CMUCL is already compiling :-), but it shouldn't be too difficult to disable unwanted nested evaluations. The advantage of signal driven IO is, that the Lisp side can respond to requests at any moment. For instance, interrupting is now more or less a normal message and we don't need Unix signals for IPC. This should also work on remote hosts.

I'm now adding similar functionality for mutlithreaded Lisps. My plan looks like this: we have 3 communicating threads, a "reader", a "control" thread and a "worker" thread. The "reader" reads request from the socket and sends them to the "control" thread. The control thread dispatches requests to worker threads and handles interrupt requests. Currently I create a new worker thread for most requests, but interrupt and debugging messages are sent to the appropriate worker thread. A worker thread executes the request and sends the result back to the controller and the controller sends the result to Emacs. A worker only communicates with the controller and doesn't write directly to the socket. Many messages include now a thread field, so that Emacs knows which thread to talk to.

The thread-per-request support is implemented for SBCL, Lispworks, and Allegro. The SBCL version doesn't work very well, apparently some deadlock problem. Will probably switch to signal driven IO for SBCL. An OpenMCL port should be easy. Haven't decided yet what to do with CLISP. I tried to add signal driven IO for CLISP, but SIGIO is blocked in the SIGIO handler and I get segfaults when I set the SA_NODEFER flag.

I'm also thinking about a possibility to query the state of the Lisp side. The idea is that Emacs receives a list describing the interesting bits of Lisp's current call stack. That would be very useful for writing test cases and perhaps something like a resync command. It would be nice if Lisp could answer such 'get-current-state' requests immediately; it should be treated like an interrupt request.

My question now is: can we assume that Lisp can answer such requests immediately?

It's probably difficult to implement this for CLISP, but it would reduce the complexity of the Emacs side.

Helmut.

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

I've merged this branch into the main trunk. The major user visible change is that M-. remains usable, even when Lisp performs a lengthy computation. This is a major change in the protocol, so be prepared for some new bugs :-)

Very very very cool :-)

Michael Livshin usenet@cmm.kakpryg.net writes:

aye. Ctrl-C in CMUCL no longer lands me in the sldb.

but you've probably already seen this. :)

No I didn't. Thanks for the note. Should be fixed in CVS.

Helmut.

The issue that triggered the note was when I macroexpanded a defstruct form. In that case the accessors were being defined in the wrong package. So it looks like I misdiagnosed the issue - I think it must be that *package* isn't bound. Since I was also evaluating these defstructs I was also confronted with the rather confusing(amusing only in retrospect) situation of no longer having defstruct accessors after I did so. I'll make the change.

What's your experience that makes you worry about binding these kinds of variables?

-Alan

On Jan 24, 2004, at 3:18 AM, Helmut Eller wrote:

Alan Ruttenberg alanralanr@comcast.net writes:

...

Looks like macroexpand and other uses of eval-string don't read in the right package.

eval-string and our macroexpand functions use from-string for reading. from-string reads in the *buffer-package*. Looks right to me.

Or are you suggesting that eval-string should bind *package* to *buffer-package* before evaluating ? That's probably a similar issue as with *load-pathname*. I think it is better not to mess around with user visible dynamic variables. But again, if you think otherwise, just change the code and let us see how that works.

Helmut.