May 2008 - erlang-in-lisp-devel@common-lisp.net

[erlang-in-lisp-devel] fare-match question
by Matt Bone 01 Jun '08

01 Jun '08

This is not a huge deal, and probably a bit out of the order we discussed. I started thinking about pattern matching yesterday, though, and I was trying to do the simple example in fare-matcher (a simple modification of what's in the docs): (defun test-match (param) (fare-matcher:match param (:ping (print :pong)) (:quit (print :bye)) ((:add a b) (print :add)))) Matching with the simple symbols ping and quit works just fine, but things go south when trying to match with add and bind a and b. I was attempting to do: (test-match '(:add 1 2)) But something is wrong in the macroexpansion. Am I even using my test-match function properly? Does match handle quoted sexpr properly or was I just masking this problem with :ping and :quit by handing it symbols that evaluate to themselves? Also, since I believe both authors are on the list, should I consider using cl-match instead? Or are our pattern matching requirements so minimal that a) it does not matter and b) we could even roll our own ala some of the Norvig examples? --matt

5 4

[erlang-in-lisp-devel] event loop
by Faré 20 May '08

20 May '08

I forgot to add the following: 4- By managing OS processes, I mean make sure that around our event loop we promptly waitpid(2) for any process we fork(2) that may die, so as to a- avoid zombie processes b- be able to handle process death the Erlang way, by killing the rest of the process tree and/or otherwise sending the process manager a death notification message (if enabled). 5- We WILL need an efficient event loop internally to solve the "C10K" problem. http://www.kegel.com/c10k.html a- that's why I want the project to use and/or enhance IOlib, that tries to provide a useful abstraction for an event-loop. b- on linux, we'll probably want to use epoll and sigfd's (through IOlib). c- on bsd/osx, we'll probably want to use kqueue (also through IOlib). d- on windows - we'll figure it out later (unless you're a windows expert yourself or recruit one to this group) e- BTW, an "interesting" bug I once hit while using IOlib was trying to persist an event loop in a saved lisp image -- oops, it contains process-specific file descriptors, and you need to reinitialize it when you restore the lisp image. 6- Note that you'll have to become familiar with the semantics of many system calls as described in section 2 of the unix man(1) pages, even though you may or may not call them directly but hopefully through IOlib (thin or thick) abstractions. a- a short list of interesting man pages that you'll read before this project is completely started: fork(2), wait(2), pipe(2), socketpair(2), socket(7), send(2), recv(2), cmsg(3), bind(2), listen(2), accept(2), shutdown(2), socket(2), ip(7), tcp(7), udp(7), unix(7), select(2), epoll(2)(linux), kqueue(2)(bsd)... b- you don't have to know all the content of these manpages by heart. However, a quick read or two to know what exists can be good. Then you'll come back to the man pages as you need them. 7- Do not hesitate to ask questions. We're here to help. a- We're experienced, and we might be right when you are wrong b- We're not infallible, and you might be right when we are wrong c- The ultimate test of who's right is running code. Don't accept any other standard. d- Existing Erlang implementations contain a lot of such running code. I don't have much knowledge about that body of code, but Robert does, and you may want to dig into it at times. 8- When in doubt, write a simple program and test your hypotheses. Ideally, the Lisp REPL should help make such testing relatively easy. 9- This data, too, might be usefully added to documentation. 10- Apart from that, I'm glad you can start working on this as soon as wednesday. It would be very nice to receive a daily report on this list and/or a blog (did you start one? or just add a category to your existing blog?) [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Invent a clever saying, and your name shall live forever. -- Anonymous 2008/5/20 Faré <fahree(a)gmail.com>: > (oops, I initially sent this answer only to Matt) > > 1- There are several semi-portable posix libraries for CL. Osicat and > IOlib are two of them. I happen to like IOlib, or at least think it's > a good platform to start from and hack further. > > 2- as for fork, yes, you should start with just a simple CL-based > wrapper to posix fork, and build around it. > a- see however how in philip-jose I provide (at least in SBCL) a > mechanism to proactively GC before you fork to avoid an early GC in > the child. > > 3- the purpose of my heavily-frobbing fork was to provide a portable > way to fork-and-exec even when you can't control the GC. > a- If a GC happens in the child in between the fork and the exec, the > result may be serious lossage, especially if the parent had multiple > system threads running. It is important then to frob and exec > "atomically" with respect to the Lisp heap. > b- This isn't necessary for Erlang-like programming, only for CLSH > shell-like activity. > > 4- writing a portable "run-program" however is a good way to test the > (OS)process-tracking and fd-frobbing mechanisms. > a- such a step I think is essential before we attempt to implement > fancy communication protocols between processes. > b- Having all the bells-and-whistles of my attempted frob-and-fork > implementation is not necessary, and a first pure-lisp attempt is > probably preferrable > > In short, start simple. If you don't understand why I (or someone > else) wrote some code in philip-jose or some other code, then it is > good to > a- start by trying without the complexity (maybe doing things better > indeed, or maybe learning why and how the complexity has to be added) > b- inquire (as you just did) why the complexity was added. > > [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] > When your hammer is C++, everything begins to look like a thumb. > -- Steve Hoflich on comp.lang.c++ > > > 2008/5/18 Matt Bone <thatmattbone(a)gmail.com>: >> Ok, so I've been getting up to speed on my systems programming in >> common lisp. I've actually been a bit surprised that there is no >> portable posix interface (though it shouldn't be too hard to make the >> things we're interested in work on various implementations). >> >> As far as pulling in the fork stuff from philip-jose, I've been >> screwing around with a lot of different things. If it's alright with >> everyone I think I'd prefer to abandon the C file and just use >> implementation specific posix calls (like sb-posix:fork) when >> available and CFFI when not. >> >> To be honest I'm still not understanding how far we need to take the >> fork mechanism. Are we trying to match, for example, the features in >> SCSH so this can be a useful standalone component? Or is this just >> the concurrency mechanism we're looking to use as a starting point (if >> so, I have a few other questions/ideas that I'll need to >> ask/consider). >> >> I'll be away at a conference to present a paper tomorrow and tuesday. >> After that, all my non-SoC obligations are complete until the fall. >> >> --matt

1 0

[erlang-in-lisp-devel] Re: Erlang in Lisp
by Faré 20 May '08

20 May '08

(oops, I initially sent this answer only to Matt) 1- There are several semi-portable posix libraries for CL. Osicat and IOlib are two of them. I happen to like IOlib, or at least think it's a good platform to start from and hack further. 2- as for fork, yes, you should start with just a simple CL-based wrapper to posix fork, and build around it. a- see however how in philip-jose I provide (at least in SBCL) a mechanism to proactively GC before you fork to avoid an early GC in the child. 3- the purpose of my heavily-frobbing fork was to provide a portable way to fork-and-exec even when you can't control the GC. a- If a GC happens in the child in between the fork and the exec, the result may be serious lossage, especially if the parent had multiple system threads running. It is important then to frob and exec "atomically" with respect to the Lisp heap. b- This isn't necessary for Erlang-like programming, only for CLSH shell-like activity. 4- writing a portable "run-program" however is a good way to test the (OS)process-tracking and fd-frobbing mechanisms. a- such a step I think is essential before we attempt to implement fancy communication protocols between processes. b- Having all the bells-and-whistles of my attempted frob-and-fork implementation is not necessary, and a first pure-lisp attempt is probably preferrable In short, start simple. If you don't understand why I (or someone else) wrote some code in philip-jose or some other code, then it is good to a- start by trying without the complexity (maybe doing things better indeed, or maybe learning why and how the complexity has to be added) b- inquire (as you just did) why the complexity was added. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] When your hammer is C++, everything begins to look like a thumb. -- Steve Hoflich on comp.lang.c++ 2008/5/18 Matt Bone <thatmattbone(a)gmail.com>: > Ok, so I've been getting up to speed on my systems programming in > common lisp. I've actually been a bit surprised that there is no > portable posix interface (though it shouldn't be too hard to make the > things we're interested in work on various implementations). > > As far as pulling in the fork stuff from philip-jose, I've been > screwing around with a lot of different things. If it's alright with > everyone I think I'd prefer to abandon the C file and just use > implementation specific posix calls (like sb-posix:fork) when > available and CFFI when not. > > To be honest I'm still not understanding how far we need to take the > fork mechanism. Are we trying to match, for example, the features in > SCSH so this can be a useful standalone component? Or is this just > the concurrency mechanism we're looking to use as a starting point (if > so, I have a few other questions/ideas that I'll need to > ask/consider). > > I'll be away at a conference to present a paper tomorrow and tuesday. > After that, all my non-SoC obligations are complete until the fall. > > --matt

1 0

[erlang-in-lisp-devel] Using fork
by Faré 20 May '08

20 May '08

My plan for Erlang-in-Lisp was to start with fork as the primary method for spawning language-level processes using a "one OS process per Erlang process" model indeed. The reasons follow. 1- Yes, the performance of fork doesn't scale and thus severely limits the applicability of the actor paradigm, but scaling can be achieved later by other means: portable lisp-in-lisp green threads (as in the Tube or philip-jose), or implementation-specific green threads. 2- fork is portable (to any unix/cygwin), and provides desirable isolation properties that would require a large number of hours of debugging to enforce using other methods (especially when Lisp implementations and libraries were not designed to be thread-safe). Yes, these other methods will have to be implemented eventually, but fork-based concurrency will allow us to have a reference implementation by then that can help discriminate between implementation bugs vs application bugs. 3- Fork-based concurrency also provides for very desirable isolation from bugs in third-party libraries, and for native compilation of lisp code without an intermediate translation layer - two properties that we will certainly want to preserve even when we have other implementation strategies available for multiprogramming. 4- last but not least, fork-based concurrency seems like it's the easiest, cheapest and fastest way to get an initial working system. So no, fork-based concurrency not the be-all end-all of Erlang-in-Lisp, but I really think it's the best way to start. Once we have a simple working implementation that doesn't scale, we can add new strategies: * lisp threads as in cl-muproc, whether the underlying lisp uses system threads (as SBCL does) or green threads (as CMUCL does), * a portable but very slow meta-programming green threads as in philip-jose (uses arnesi's slow ((lisp with continuations) in lisp) interpreter), that should allow migration of processes (within some restrictions). * a similar portable but less slow method using a lisp-in-lisp compiler instead like the one in Screamer, which might not allow for migration (or might, if we use the same tricks as the Tube). * pushing the code to a remote Erlang implementation (whether Erlang-in-Lisp or plain Erlang), if it doesn't use any of our Lisp-specific "extensions", but only things available with say LFE. * any other method du jour. Hopefully, we can make it easy to select what strategy we use to spawn any particular language-level process. (Hum, such design rationale should probably be pushed into a design document included with the code.) [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] I don't program enough to be called a "programmer", and I don't non-program enough to be called a "non-programmer". So I'm a non-non-programmer, which, in intuitionistic logic, is weaker than a programmer. -- Faré 2008/5/18 Robert Virding <rvirding(a)gmail.com>: > I saw the reference to fork in the earlier message. Seeing I have come in a > little late you will have to excuse my asking. What do you intend to use > fork for in erlang-in-lisp? Erlang processes? Many Erlang processes can > generate many processes. > > Robert

1 0

[erlang-in-lisp-devel] Re: Erlang in Lisp
by Matt Bone 18 May '08

18 May '08

Ok, so I've been getting up to speed on my systems programming in common lisp. I've actually been a bit surprised that there is no portable posix interface (though it shouldn't be too hard to make the things we're interested in work on various implementations). As far as pulling in the fork stuff from philip-jose, I've been screwing around with a lot of different things. If it's alright with everyone I think I'd prefer to abandon the C file and just use implementation specific posix calls (like sb-posix:fork) when available and CFFI when not. To be honest I'm still not understanding how far we need to take the fork mechanism. Are we trying to match, for example, the features in SCSH so this can be a useful standalone component? Or is this just the concurrency mechanism we're looking to use as a starting point (if so, I have a few other questions/ideas that I'll need to ask/consider). I'll be away at a conference to present a paper tomorrow and tuesday. After that, all my non-SoC obligations are complete until the fall. --matt

2 1

[erlang-in-lisp-devel] Erlang in Lisp
by Faré 08 May '08

08 May '08

Dear Matt, dear Robert, LIST I propose we continue these conversations on the public mailing-list erlang-in-lisp-devel at common-lisp.net: http://common-lisp.net/cgi-bin/mailman/listinfo/erlang-in-lisp-devel REPO OK, I pulled a copy of the git repo, and see you've been active importing stuff. I think that a shell script to checkout and/or update the existing code would have been better, but let's say what's there is good where it is. You may want to try the other approach for pulling the erlang sources, or the sources of other CL libraries we'll need: iolib and its dependencies, etc. PLANNING as far as google is concerned, we're still in a planning phase, not the mad coding phase yet, but "learn about what exists, learn about each other, draw a plan". FORK Since the first step is forking, you can see in philip-jose what I was thinking of regarding the fork thing -- grep for fork there, except that the code isn't very developed. The idea would be that there should be a DSL to express all the parameters that you may want to preserve or tweak when you fork: shuffling fd's, closing some, changing directory, dropping privileges, changing process group, and all kind of weird things. Then, either you exec as part of the same C call (so as to not risk any lisp runtime confusion because of GC or other interruptions), or you immediately do a lot of adjustments before/after you get back to the lisp side in the child: frob the fd streams, reset the thread/lock structures and/or if any (hopefully none), etc. See also how the lisp-based fork tries to GC before to fork if tight in memory. But for the very first step, I think that a portable run-program implementation based on a simple C fork with a trivial DSL for shuffling fd's would be a good first goal. We should be using iolib and synchronous wait for the process management, not signal handlers. Luigi said he had such an implementation - we may ask him about it, or start from the run-program present in an existing CL implementation (not SBCL or CMUCL - it's broken with broken signal handling race conditions). YOUR TURN sorry for my long silence. I was busy at work. I'll try to be on track now. Matt: what about posting regular updates on your blog and/or on this list? [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The Constitution may not be perfect, but it's a lot better than what we've got! 2008/4/26 Matt Bone <thatmattbone(a)gmail.com>: > I have my cl.net account and I created a git repository (empty at the > moment). I'll write this up, but for now we can: > git clone http://common-lisp.net/project/erlang-in-lisp/git > > and to push a patch: > git push /project/erlang-in-lisp/public_html/git master > > --matt > > > > On Fri, Apr 25, 2008 at 6:01 PM, Faré <fahree(a)gmail.com> wrote: > > Robert Virding was making lots of interesting suggestions as to which > > level of Erlang implementation we want so as to leverage as much as > > possible of the Erlang front-end and support. It looks like Core > > Erlang or Kernel Erlang would be most fit. > > > > Drew: can you give an ETA for creating those common-lisp.net accounts, > > GIT repository and mailing-list? I suppose Robert should get an > > account, too. > > > > Matt: can you download all relevant software/documents and make a list > > of URLs/checkout commands so everyone else can do it, too? > > > > [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] > > Most people think they need a ruler. Perhaps we should give them a fake > > one that doesn't actually do anything, and then they won't think about > > it. It is sort of like giving an infant a pacifier. -- Perry Metzger

1 0