I wonder if anybody has yet to tread this path.
I have been using CL at on my school's cluster. I usually develop within Slime locally, then move it to the cluster's master node and have a trial run over a remote Slime session, then once everything works, I ditch Slime and invoke my programs on the worker nodes using a bash or Lisp script (yes, we are running without a queue; crazy, huh?). I would very much like to use dispatch the jobs to the worker nodes from within Slime. More to the point, I would like to be able to start jobs on the worker nodes from my Slime connection to the master node. Now, of course I can connect to each node using a tunnel and start the jobs that way, but I want an automated interface as sketched below.
I was thinking about developing something that would work for me and perhaps others. I just wanted to put this out there first to make sure it hasn't already been done and see if there are any suggests or comments on an undertaking like this. Basically all I would do is define a CL client for swank and use that to connect to worker Lisps running on each worker node from the one instance on the master. Then, if I want to send a job to the first node, all I have to do is send some code over the swank connection for the worker to evaluate and return the value. I also thought I'd write a kind of forwarder for the Swank connection so I can jump to a worker node from the master by setting the master to forward all data to the worker, and return anything coming from the worker down its connect to Slime (although, I suppose this could be arbitrarily deep, this is analogous to jumping from server to server using ssh but without the secure-ness). I am fairly convinced that this will also work (sans persistent worker Lisps) for any cluster with a queuing system that allows for interactive jobs (like the Torque/Moab combo that we use on another cluster here) and probably in a EC2 Cloud like environment as well. I'm just saying that it is possible that this could be generally useful for other people as well.
Almost all of this would be on the Swank side (and thank goodness because I feel like a fish out of water in ELisp). However, I have already had trouble understanding the Slime message protocol. I'll figure it out eventually, but are there any good internals guides for Slime/Swank?
Any advice is appreciated,
Zach KS