Hi guys.
First of all, thanks for your input. Second, I owe you an apology for wasting your time. A lot of what you have supplied me with in this thread was already known to me, but I failed to make that evident on my site.
Faré fahree@gmail.com writes:
: "Hoehle, Joerg-Cyril" Joerg-Cyril.Hoehle@t-systems.com
: Dirk Gerrits dirk@dirkgerrits.com
Therefore I was disappointed by your roadmap which talks a lot about OS threads and special (thread-local) variables.
Sorry to hear that, care to elaborate?
After a few tens of thousands of threads (may hundreds of thousands, on modern machines with several gibibytes of RAM?), the performance of most (all) Unix kernels becomes very bad. If only because each thread eats at the very least a few kilobytes of kernel memory, and ultimately the machine runs out of memory. Green thread have a potentially much lower overhead.
I was more interested in the special variables comment, and about my roadmap talking "a lot" about OS threads. I realize now that I did not make that clear *at all* in my reply. Sorry.
- Using continuations as green threads. This would limit the points at which processes can be preempted, but that can be partially alleviated by assigning an operating system thread to every few green threads. Also, less preemption is not necessarily a problem.
With sufficient engineering and code-walking, preemption can remain possible.
I do not see how a process calling a Common Lisp sequence function on a large array can be preempted by sufficient engineering and code-walking. But please enlighten me, I'm just a newbie.
If you meant that preemption in user functions is possible, then I think I know what you mean. In Erlang implementation, it is customary (as far as I have been able to tell so far) to have a "reduction counter" that gets decremented on every function call. When it reaches zero, the process is preempted and another process gets a chance to run, with a reset reduction counter. This shouldn't be too hard to implement in a CPS code walker, but I worry about its performance. Nevertheless, it is something I'm planning to try, as only trying it out can give me any real data on its performance.
However, preemption necessitates proper handling of invariants, with locking, roll-back and/or roll-forward. As a cheaper way to do things, cooperative multithreading can do, and can be used automatically or semi-automatically. (Only potentially long-running routines need have (yield) statements inserted, and with proper code-walking, this can be done automatically.)
What I have considered doing is to have cooperative multithreading that yields /only/ on Erlang's receive statements. I'm curious as to whether that is really as bad as it sounds.
I could indeed have extra yields be added automatically, but that may be more tricky than you make it sound (or maybe I'm misinterpreting). For example, putting a yield in a tight loop might kill performance, and not putting it there might kill cooperation. Trying to solve this would bring us pretty close to the above reduction counter, wouldn't you agree?
and a CPS code walker is more work than using an OS thread library.
There already are CPS code walkers, in Screamer and in UCW.
I've looked at those in Arnesi, UCW, and Screamer in the past and know that I can't use them out-of-the-box. But I could "borrow" some code and adapt it to my purposes, is that what you suggest? I am not opposed to that, althoug I think that may deprive me of a valuable Common Lisp learning experience.
Cf. http://c2.com/cgi/wiki?MessagingAsAlternativeToMultiThreading
http://groups.google.com/groups?selm=87acurudww.fsf%40dirkgerrits.com&rn...
I agree that a better place for discussion is needed. How about a common-lisp.net mailing list?
If you create it, please subscribe me :)
After Edi's suggestion, I subscribed myself to CLUMP (http://www.caddr.com/clump/). Do you think that is a good place to continue this discussion (and future Erlisp discussions?).
Kind regards,
Dirk Gerrits