I forgot to add the following:

4- By managing OS processes, I mean make sure that around our event loop we promptly waitpid(2) for any process we fork(2) that may die, so as to a- avoid zombie processes b- be able to handle process death the Erlang way, by killing the rest of the process tree and/or otherwise sending the process manager a death notification message (if enabled).

5- We WILL need an efficient event loop internally to solve the "C10K" problem. http://www.kegel.com/c10k.html a- that's why I want the project to use and/or enhance IOlib, that tries to provide a useful abstraction for an event-loop. b- on linux, we'll probably want to use epoll and sigfd's (through IOlib). c- on bsd/osx, we'll probably want to use kqueue (also through IOlib). d- on windows - we'll figure it out later (unless you're a windows expert yourself or recruit one to this group) e- BTW, an "interesting" bug I once hit while using IOlib was trying to persist an event loop in a saved lisp image -- oops, it contains process-specific file descriptors, and you need to reinitialize it when you restore the lisp image.

6- Note that you'll have to become familiar with the semantics of many system calls as described in section 2 of the unix man(1) pages, even though you may or may not call them directly but hopefully through IOlib (thin or thick) abstractions. a- a short list of interesting man pages that you'll read before this project is completely started: fork(2), wait(2), pipe(2), socketpair(2), socket(7), send(2), recv(2), cmsg(3), bind(2), listen(2), accept(2), shutdown(2), socket(2), ip(7), tcp(7), udp(7), unix(7), select(2), epoll(2)(linux), kqueue(2)(bsd)... b- you don't have to know all the content of these manpages by heart. However, a quick read or two to know what exists can be good. Then you'll come back to the man pages as you need them.

7- Do not hesitate to ask questions. We're here to help. a- We're experienced, and we might be right when you are wrong b- We're not infallible, and you might be right when we are wrong c- The ultimate test of who's right is running code. Don't accept any other standard. d- Existing Erlang implementations contain a lot of such running code. I don't have much knowledge about that body of code, but Robert does, and you may want to dig into it at times.

8- When in doubt, write a simple program and test your hypotheses. Ideally, the Lisp REPL should help make such testing relatively easy.

9- This data, too, might be usefully added to documentation.

10- Apart from that, I'm glad you can start working on this as soon as wednesday. It would be very nice to receive a daily report on this list and/or a blog (did you start one? or just add a category to your existing blog?)

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Invent a clever saying, and your name shall live forever. -- Anonymous

2008/5/20 Faré fahree@gmail.com:

(oops, I initially sent this answer only to Matt)

1- There are several semi-portable posix libraries for CL. Osicat and IOlib are two of them. I happen to like IOlib, or at least think it's a good platform to start from and hack further.

2- as for fork, yes, you should start with just a simple CL-based wrapper to posix fork, and build around it. a- see however how in philip-jose I provide (at least in SBCL) a mechanism to proactively GC before you fork to avoid an early GC in the child.

3- the purpose of my heavily-frobbing fork was to provide a portable way to fork-and-exec even when you can't control the GC. a- If a GC happens in the child in between the fork and the exec, the result may be serious lossage, especially if the parent had multiple system threads running. It is important then to frob and exec "atomically" with respect to the Lisp heap. b- This isn't necessary for Erlang-like programming, only for CLSH shell-like activity.

4- writing a portable "run-program" however is a good way to test the (OS)process-tracking and fd-frobbing mechanisms. a- such a step I think is essential before we attempt to implement fancy communication protocols between processes. b- Having all the bells-and-whistles of my attempted frob-and-fork implementation is not necessary, and a first pure-lisp attempt is probably preferrable

In short, start simple. If you don't understand why I (or someone else) wrote some code in philip-jose or some other code, then it is good to a- start by trying without the complexity (maybe doing things better indeed, or maybe learning why and how the complexity has to be added) b- inquire (as you just did) why the complexity was added.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] When your hammer is C++, everything begins to look like a thumb. -- Steve Hoflich on comp.lang.c++

2008/5/18 Matt Bone thatmattbone@gmail.com:

...

Ok, so I've been getting up to speed on my systems programming in common lisp. I've actually been a bit surprised that there is no portable posix interface (though it shouldn't be too hard to make the things we're interested in work on various implementations).

As far as pulling in the fork stuff from philip-jose, I've been screwing around with a lot of different things. If it's alright with everyone I think I'd prefer to abandon the C file and just use implementation specific posix calls (like sb-posix:fork) when available and CFFI when not.

To be honest I'm still not understanding how far we need to take the fork mechanism. Are we trying to match, for example, the features in SCSH so this can be a useful standalone component? Or is this just the concurrency mechanism we're looking to use as a starting point (if so, I have a few other questions/ideas that I'll need to ask/consider).

I'll be away at a conference to present a paper tomorrow and tuesday. After that, all my non-SoC obligations are complete until the fall.

--matt