On 12 Dec 2003, at 15:57, Sean O'Rourke wrote:

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

...

Sven Van Caekenberghe sven@beta9.be writes:

...

However, there is a problem in the debugger connection in cvs head (as compared to the version we checked out 2 days ago): an undefined function like (foo) gives an (presumably emacs) error: error in process filter: Assertion failed: (<= (length frames) maximum-length)

I couldn't reproduce this. Are you using XEmacs or Emacs? Until we know what the problem is, you can disable the SLIME debugger

Also, commenting out the assertion seems to "fix" the problem -- nothing else seems to break when you do so.

I don't know who commited what, but I just did an update and the problem that I reported went away, Thanks!

In the general discussion about SLIME and debugging, I think that a way to recover better from 'problems' (i.e. SLIME losing control and an error appearing in the inferior lisp) so that one can continue to work. The greatest showstopper is that you have to relaunch the inferior lisp (or worse, that emacs hangs waiting for something).

Is there some release tagging for SLIME ? I think it would be useful for users to be able to report bugs against version numbers - just a thought.

Sven

-- Sven Van Caekenberghe - mailto:sven@beta9.be Beta Nine - software engineering - http://www.beta9.be .Mac - svc@mac.com - http://homepage.mac.com/svc

"Lisp isn't a language, it's a building material." - Alan Kay

Helmut Eller e9626484@stud3.tuwien.ac.at writes:

...

One of the reasons that we like SLIME so much is the fact that it doesn't choke on asynchroneous output (as ILISP does): when working with a multithreaded server that makes all the difference in the world. The output that non-evaluator threads write to stdout goes to the inferior lisp buffer, we can live with that, but maybe it would be nicer if (optionally) that output was inserted in the normal SLIME REPL buffer (maybe in some other color) - is that possible, maybe the behavior is different on other platforms ?

That's not yet implemented :-) SLIME is currently only able to talk to a single thread. We hope that we can support this in the future, but progress is a bit slow in this direction.

I can do some multiprocessing hacking this weekend.

Here is how I envision "phase 1":

There is just one thread that SLIME uses for normal requests (fetching arglists, evaluation with C-c : or in *slime-repl*, etc etc etc). This is just like today, and in a single-threaded system this would be the only thread.

Printed output from other threads can be sent to Emacs. It will be inserted into e.g. *slime-repl* in a different colour.

Other threads can ask to be debugged by SLIME. A small signalling protocol would be used to tell Emacs "thread <x> has entered the debugger - please debug it when you're ready." The user could then see which threads are waiting to be debugged and attach the SLIME debugger to them.

Only the debugger would interact with these other threads.

I'm thinking of a user interface similar to the one in Distel (a "SLIME for Erlang"). There you have a buffer listing all the threads (erlang "processes") that are waiting to be debugged, like this: http://www.bluetail.com/~luke/misc/erlang/dbg-monitor.png

This "monitor" buffer automatically pops up when a new thread becomes debugable.

If you select a thread with RET or a mouse click, the debugger gets attached to it. You can either debug it to completion, or just partly debug it and pop back into the monitor to do something else while it waits. The attached debugger is like the *sldb* buffer, in Distel it looks like this: http://www.bluetail.com/~luke/misc/erlang/dbg-attach.png (The red lines indicate breakpoints.)

How does that sound?

There are more possible features that I think of as "phase 2", such as multiple *slime-repl* buffers each talking to different threads. I'm sure we can learn a lot from ELI when we come to that point. Are there also some ELI features that we should get familiar with for "phase 1"?

Cheers, Luke

On Fri, Dec 12, 2003 at 07:18:50PM +0100, Luke Gorrie wrote:

It would be great if the folks already using SLIME with multithreaded programs can post a note about what features you want the most (if you haven't already), and which Lisp you're using.

(This is more of a general debugging thing, with SBCL/CMUCL)

Is there a way to, once you're in the SLIME debugger, to throw out to the tty-mode debugger? Sometimes I'm stuck wanting to do something that sldb cannot (like list-locations, setting breakpoints, etc.), and there's no way to get there and save the debugger state. You have to abort completely and get the debugger to show up again in *inferior-lisp*. If threads are going to automatically trigger sldb, this is a bigger problem as even that escape hatch is gone.

(wishful-thinking "Of course, raising SLIME's debugging integration to the level of one of the commercial lisps (I've used Xanlysis's debugger, and think it's great) would be an acceptible substitute. Setting breakpoints in the source would be far, far cooler than supporting list-locations. :)")

-bcd

Luke Gorrie luke@bluetail.com writes:

This "monitor" buffer automatically pops up when a new thread becomes debugable.

If you select a thread with RET or a mouse click, the debugger gets attached to it. You can either debug it to completion, or just partly debug it and pop back into the monitor to do something else while it waits. The attached debugger is like the *sldb* buffer, in Distel it looks like this: http://www.bluetail.com/~luke/misc/erlang/dbg-attach.png (The red lines indicate breakpoints.)

How does that sound?

Are you planning to make it possible to debug multiple threads simultaneously--I.e. it might be nice to set break points in two places that will be hit be different threads and then debug them both at the same time (in two different debugging buffers); for example step a little way in one thread, then switch to the other; etc.

-Peter

Luke Gorrie luke@bluetail.com writes:

Peter Seibel peter@javamonkey.com writes:

...

Are you planning to make it possible to debug multiple threads simultaneously--I.e. it might be nice to set break points in two places that will be hit be different threads and then debug them both at the same time (in two different debugging buffers); for example step a little way in one thread, then switch to the other; etc.

Not initially. Is that something you need?

Not initially. ;-)

This is slightly tricky in SLIME because we support a synchronous evaluation interface in Emacs Lisp. In Elisp you can write:

(slime-eval '(swank:some-function))

And it will evaluate the form in Lisp and then return the result. However, it might enter the debugger before it finishes, in which case Emacs enters a "recursive edit" to do the debugging without unwinding the Elisp stack (so that when you finish debugging it can return the final result to the `slime-eval' call). If during debugging you switched over to another thread and did another `slime-eval' there, the stack would end up rather tangled -- you have two `slime-eval's on the stack and have to return to one before the other.

A few possible solutions come to mind, but I think it will be better to burn this bridge when we come to it and do some experimentation. Ideas are always welcome, of course.

Hmmm. I'm pretty ignorant of the details of recursive edits. Which gives me the advantage of not knowing what's hard. ;-) Here's how I was envisioning it: when I pick a thread to debug from the list of debuggable therads SLIME creates a new buffer for interacting with that thread. I switch between threads by switching between buffers. Thus I can C-x 2 to split my window and put one thread-buffer in each window and switch back and forth with C-x o. Peeking at the info page for recursive edits I see this:

In general, we try to minimize the use of recursive editing levels in GNU Emacs. This is because they constrain you to "go back" in a particular order--from the innermost level toward the top level. When possible, we present different activities in separate buffers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ so that you can switch between them as you please.Some commands ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ switch to a new major mode which provides a command to switch back. These approaches give you more flexibility to go back to unfinished tasks in the order you choose.

I'm not sure I grok all the implications of that underlined portion but it sort of sounds like if you were following that advice instead of debugging in a recursive edit you'd debug in a new buffer. Which you might switch to automatically if you're interacting with the Lisp synchronously.

-Peter

Luke Gorrie luke@bluetail.com writes:

See the problem?

NB: By "the problem" I didn't mean my using "up the stack" and "down the stack" interchangably. :-)

Luke Gorrie luke@bluetail.com writes:

Peter Seibel peter@javamonkey.com writes:

...

Here's how I was envisioning it: when I pick a thread to debug from the list of debuggable therads SLIME creates a new buffer for interacting with that thread. I switch between threads by switching between buffers. Thus I can C-x 2 to split my window and put one thread-buffer in each window and switch back and forth with C-x o.

The main issue is for people writing SLIME-based Elisp code. Anyone not interested in how that works can safely skip this explanation :-)

The tricky part is when Emacs has made RPCs to Lisp, and the Lisp threads being debugged are actually computing results to return to Emacs functions.

Initially the Emacs stack looks something like:

((emacs-event-loop))

Then we call an Emacs function that wants to RPC to Lisp to evaluate the expression FOO in thread T1 and do-something-with the result (the top of the stack comes first):

((receive-reply-from T1) (do-something-with (eval-in-thread FOO T1)) (emacs-event-loop))

But FOO causes an error and invokes the debugger, so Emacs enters a recursive edit (i.e. calls the event loop recursively) to do the debugging, hoping to debug the problem and then get its result:

Just to make sure I understand this, it can do that because it actually got back a response from the Common Lisp which it grokked is an event that told it to invoke the debugger. Otherwise it would still be blocked, right. Does that mean if I call (eval-in-thread (sleep 300) t1), my emacs is going to hang for 5 minutes?

((emacs-event-loop) (debug-thread T1) (receive-reply-from T1) (do-something-with (eval-in-thread FOO T1)) (emacs-event-loop))

Now we have one thread available for debugging. But suppose we let it wait a while, and now ask another thread T2 to evaluate an expression BAR and do-something-else with that result, and T2 also enters the debugger:

((emacs-event-loop) (debug-thread T2) (do-something-else-with (eval-in-thread BAR T2)) (emacs-event-loop) (debug-thread T1) (receive-reply-from T1) (do-something-with (eval-in-thread FOO T1)) (emacs-event-loop))

Now we have two Emacs functions on the stack that are expecting a value - `do-something-with' and `do-something-else-with'.

See the problem?

Yeah. But what I don't understand (because, I think, I don't know enough about what a recursive edit really implies in emacs, let alone SLIME) is what other options there are. If there's only one emacs stack, regardless of how many buffers you have--which is what I'm gathering--then I'd say eventually the way to go is to make all interactions with Common Lisp be asynchronous--when you pass an expression to Lisp to be evaluated you have four choices what happens with the result--it gets stuck into a buffer, it gets passed to MESSAGE, it goes to the bit bucket in the sky, or it gets passed to a user-supplied function.

To deliver the result of FOO to do-something-with, we need to have popped everything below off the stack, including do-something-else-with. So either we have to debug BAR first, so that it can deliver its result and unwind the stack for us, or we get the result of FOO and 'throw' up the stack, so that do-something-else-with is discarded and there won't be anything left to use the result of BAR.

Okay, so maybe this is not so exciting to you guys doing pure CL, but it's a big deal for those of us using Emacs as a UI toolkit, and indirectly for people using the UIs we write :-)

There are some solutions available. One is to just do the throw -- we have to be able to handle that anyway (and we can) because the same thing happens if the user calls `abort-recursive-edit' (`C-]').

That seems bad because that means you lose all that state that was on the stack. That's okay for aborts but seems an impovrished environment for writing more sophisticated features.

Alternatively, we could ditch our synchronous-eval elisp funtion and use a purely asynchronous interface based on continuation-passing-style -- then we don't need the Elisp stack or recursive edits, and can deliver results in any order.

That seems like the way to go.

But I don't think the time is ripe yet :-)

Well, you're the one hacking on it, not me. But I'd be worried that if you build an synchronous API and everybody builds on it, you'll soon have a code base that prevents you from switching to an asynch API. And then you'll never be able to provide the good debug-two-threads-in-interleaved-fashion UI.

BTW, Distel has exactly the debugging interface you describe. All its Elisp code is totally asynchronous, based on (macro-assisted) CPS with a trampolined-style scheduler providing a multiprocessing abstraction in Elisp. Make me happy, read all about it :-)

Will do, it just finished printing out.

Distel: Distributed Emacs Lisp (for Erlang) http://www.bluetail.com/~luke/distel/distel-euc.pdf

-Peter

Luke Gorrie luke@bluetail.com writes:

How does that sound?

Well, I feel rather unqualified to comment about threading issues. You Erlang guys probably know how to these things.

I think multi-threading support should "feel" like job-control in a shell. Usually you only interact with the foreground thread. There should be the possibility to interrupt the foreground thread if it is busy, to place it in the background, and to switch to another thread. Only the foreground thread should be allowed to pop up buffers and to read from the REPL. The output can probably be mixed and just be inserted in the output buffer.

But, that's probably similar to your ideas.

Helmut.

On Sat, 13 Dec 2003, Helmut Eller wrote:

I think multi-threading support should "feel" like job-control in a shell. Usually you only interact with the foreground thread. There should be the possibility to interrupt the foreground thread if it is busy, to place it in the background, and to switch to another thread. Only the foreground thread should be allowed to pop up buffers and to read from the REPL. The output can probably be mixed and just be inserted in the output buffer.

If all threads are sharing the same value of *TERMINAL-IO* etc in some way shape or form, that's probably what it does feel like. (In OpenMCL's case, it feels like "job control that doesn't always work right and can't be as hard to get right as it seems"; in some other multithreaded lisps that I've used, it feels like "job control that may act predictably and reliably but which some people find totally unintuitive", and in other multithreaded lisps it may feel like "job control that works reliably and intuitively".)

SLIME does seem to offer other ways of presenting interactive output from, interactive input to, and debugging support for lisp threads. Naively (I confess to not having done my SLIME homework), it doesn't -seem- that starting a second (or third or Nth) SLIME REPL (with associated debugging and auxiliary buffers) needs to be a whole lot different than starting the frst session was, and whether the lisp thread on the other end of the connection is one of many in a multithreaded lisp or the only thread in another instance of a single-threaded lisp or ... probably needn't matter too much from SLIME's point of view. If SLIME already support the notion of multiple sessions (within a single running [X]Emacs), then I think that the hard work of supporting this style of interaction's already been done; if it doesn't currently support multiple sessions, then some of that hard work might involve deciding on policy matters (e.g., which REPL do editor commands interact with ? how is this changed ?).

To what extent can/should a SLIME REPL be decoupled from "an Emacs stream connected to a remote Lisp stream on one end and to an Emacs buffer on the other" ? (Pretty obviously, these things are already decoupled, but it might be nice to mix and match: I could imagine wanting to use SLIME's debugger, even if a given session/thread wasn't running a SLIME REPL (or any other REPL, for that matter.)

There are lots of cases where I think it'd be handy to create an IPC connection to [X]Emacs from the lisp side: to be able to create from lisp a stream whose output appeared in and/or input came from an Emacs buffer. One could almost imagine as many uses for this capability in single-threaded lisps as in multithreaded ones. (Those uses are maybe a little more obvious in the latter case.)

Sorry if I'm raisng questions that could be easily answered by looking at the SLIME sources more carefully than I have. I think that it's worth noting that a lot of the issues related to some kinds of "SLIME-interaction-with-multithreaded-lisps" are common to "SLIME-multiple-concurrent=session-support". Whatever that is, exactly ....

Helmut.

Gary Byers gb@clozure.com