Ahoy,
I checked in some multiprocessing support under CMUCL.
[While writing this mail, I realised that it's seriously broken -- you'll notice my tone changes towards the bottom of the mail :-)]
With respect to what I said a "version one" should be, this is about half finished. It might at least be a major improvement for people already living on the edge and using threads :-)
Obligatory screenshot, debugging three McCLIM Listener processes here: http://www.bluetail.com/~luke/misc/lisp/slime-threads.png
To use it, add this to your .emacs:
(setq slime-multiprocessing t) (setq slime-global-debugger-hook t)
(That second line installs the slime debugger hook globally -- this is a convenience to automatically setup the hook that Dan made for Aranaida use.)
And this to your ~/.swank.lisp:
(setq swank::*start-swank-in-background* t) ;; Use SERVE-EVENT
The good parts:
Threads that "asynchronously" hit an error -- i.e. not while performing an RPC for Emacs -- will suspend until Emacs decides to debug them.
The Lisp side uses locks so that only one thread will enter a conversation with Emacs at a time. If several threads hit the debugger at once, they will always be debugged one-at-a-time.
The Emacs side has a new "thread control" buffer (top right of the screenshot) listing all suspended threads. You press RET to "give the goahead" to one of them so that it enters the debugger.
The sharp edges and bare wires:
I haven't done anything about input and output streams, aside from using a lock to atomic'ify all reads and writes to the Emacs socket.
There an obscure-ish race condition: if you "gohead" a thread into the debugger, but send an RPC before it actually hits the debugger, the protocol will reach an inconsistent state and bail out.
Also, something really horrible that just occured while writing this mail: there is no coherent scheme to say which thread is supposed to be reading from the Emacs socket. There is a 'conversation-lock' that prevents two threads from actually entering into talks with Emacs at the same time, but nothing to tell the SERVE-EVENT loop that "hey, this other thread is being debugged just now, so don't try to take the I/O-lock and read from Emacs just because the file descriptor becomes readable."
It seems to work anyway, but I can't say why. Shades of Java.
This is likely to be a problem to port to OpenMCL just now. OpenMCL doesn't use SERVE-EVENT (which mysteriously seems to work) but instead has a dedicated thread calling READ-FROM-EMACS to get and evaluate requests. That thread should typically be blocking in a read with the I/O-lock held, preventing any other thread (e.g. one trying to do debugging) from reading from Emacs. At the moment the only workaround I can think of is a bit.. unspeakable.
HELP, HELP! SAVE ME! I don't know how to write correct multithreaded programs!
How do people do synchronization in Lisp? Do you use higher-level abstractions than locks and shared variables, or is there some book I can read to actually understand how to program with those things?
The mind boggles. Please someone stop me before I write a homebrew message-passing system.
Cheers, Luke
Luke Gorrie luke@bluetail.com writes:
HELP, HELP! SAVE ME! I don't know how to write correct multithreaded programs!
How do people do synchronization in Lisp? Do you use higher-level abstractions than locks and shared variables, or is there some book I can read to actually understand how to program with those things?
Okay, so your claim of ignorance that is probably largely tongue-in-cheek, but just in case, Doug Lea's, _Concurrent Programming in Java_ is pretty good despite the obvious Java focus. And you definitely want to check out Hoare's paper, _Monitors: An Operating System Structuring Concept_[1] if you haven't already.
The mind boggles. Please someone stop me before I write a homebrew message-passing system.
It seems to me that you want to make your own event queue--have the single thread doing the IO and putting work requests on queues for other threads to pick up and actually do the work. (To the extent I understand Erlang, this should be quite familiar--you just have to roll it yourself. Though there may be certain data structures other than the queues that need to be accessed concurrently which makes it less like Erlang. Again, as I understand Erlang which is not very much at all.) Of course that may be what you meant by a homebrew message-passing system. In which case, go for it.
-Peter
[1] http://www.acm.org/classics/feb96/
Peter Seibel peter@javamonkey.com writes:
... Doug Lea's, _Concurrent Programming in Java_ is pretty good despite the obvious Java focus. And you definitely want to check out Hoare's paper, _Monitors: An Operating System Structuring Concept_[1] if you haven't already.
Have you read Brinch-Hansen's _Architecture of Concurrent Programs_? If not you would love it! And a bargain at $6.70 on bookfinder.com :-)
-Luke
Luke Gorrie luke@bluetail.com writes:
HELP, HELP! SAVE ME! I don't know how to write correct multithreaded programs!
Let me be slightly more specific. :-)
Here is a problem statement for the current implementation:
Only one thread may read and evaluate requests from Emacs at a time.
Initially a dedicated "Swank" thread does this.
Emacs can ask the Swank thread to temporarily pass control of the socket to another thread. That thread then gets exclusive read access for the duration of a function call, and then returns control of the socket to the Swank thread.
The challenge is to implement this protocol using only the threading primitives available. Those primitives are loosely defined as the intersection of threading primitives in all Lisps that we want to support -- CMUCL, SBCL, OpenMCL, LispWorks, ACL, and ideally keeping an open mind towards others.
Can someone with experience comment on what that intersection is? So far I have assumed it is "mutexes", but that doesn't seem satisfactory. I haven't done thread programming in Lisp before.
Peter Siebel suggests using monitors. Now that I think about it, I have already defined (SUSPEND-THIS-THREAD) and (RESUME THREAD) primitives, which would be enough to build monitors with. Maybe that is the way to go.
But before more hacking, I'd appreciate some words of advice on what thread primitives we can reasonably assume the Lisps will support. Any takers?
Cheers, Luke
Luke Gorrie luke@bluetail.com writes:
But before more hacking, I'd appreciate some words of advice on what thread primitives we can reasonably assume the Lisps will support. Any takers?
Most Lisps will, one way or another, support the stuff in CLIM-SYS. You should note, however, that some of them support some of it badly - specifically, when your Lisp threads are scheduled by the OS kernel it's difficult to implement PROCESS-WAIT (and other functions that wait for an arbitrary predicate) except by doing something rather akin to busy-waiting.
Mutexes should be pretty safe. Alternatively, specify a higher-level interface (a message queue, or something) and let each backend implement it in whatever fashion makes sense. SBCL has condition variables, OpenMCL has counting semaphores (I think), etc etc.
-dan
Luke Gorrie luke@bluetail.com writes:
Peter Siebel suggests using monitors. Now that I think about it, I have already defined (SUSPEND-THIS-THREAD) and (RESUME THREAD) primitives, which would be enough to build monitors with. Maybe that is the way to go.
I asked some guys at work how to do this. It turns out that mutexes are enough, and it can be done without those SUSPEND/RESUME primitives or monitors. "Bread and butter POSIX threads stuff."
The algorithm is this:
We define a global lock L, and a global variable WHO that is bound to the thread currently allowed to talk to Emacs.
Every thread that wants to talk to Emacs loops: Acquire lock L: If I am WHO, do my processing. Release L
The Swank thread's "processing" is to read and evaluate a command from Emacs, and then loop. That command may have the side-effect of reassigning WHO to another thread.
The "processing" of all other threads is to conduct a conversation with Emacs and then reassign WHO back to the Swank thread.
When a thread wants to be debugged, it goes until a loop waiting until it becomes WHO. When Emacs chooses to debug such a thread, it tells the Swank thread to assign WHO accordingly. When that thread finishes debugging, it reassigns WHO to the Swank thread.
I'll implement this tomorrow unless a better way comes along.
Cheers, Luke
Luke Gorrie luke@bluetail.com writes:
I asked some guys at work how to do this. It turns out that mutexes are enough, and it can be done without those SUSPEND/RESUME primitives or monitors. "Bread and butter POSIX threads stuff."
The algorithm is this:
... *ahem*, the algorithm I described is just an implementation of a monitor using a mutex. Oops :-)
I'll shush up and do some more experimentation, and also look into what it would take to do it the way Gary Byers described instead.
Cheers, Luke
-
Daniel Barlow -
Luke Gorrie -
Peter Seibel