September 2005 - erlisp-devel@lists.common-lisp.net

[erlisp-devel] Fwd: [Sbcl-devel] async signals
by Faré 20 Sep '05

20 Sep '05

I found this very interesting discussion on sbcl-devel. It shows the difficulty of doing things right if we are to kill processes asynchronously, yet they want to do things properly wrt catching them, ensuring atomicity of memory operations, freeing resources (notably alien data), etc. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Perhaps those of us who care about quality programs have not spoken up often enough -- `for bad programs to triumph requires only that good programmers remain silent.' I call this passivity the `Silence of the Lambdas.' -- hbaker ---------- Forwarded message ---------- From: Gábor Melis <mega(a)hotpop.com> Date: 30-Aug-2005 06:15 Subject: Re: [Sbcl-devel] async signals To: Alexander Kjeldaas <astor(a)fast.no> Cc: sbcl-devel(a)lists.sourceforge.net On Monday 22 August 2005 22:23, Alexander Kjeldaas wrote: > OPEN, CLOSE and other functions that need cleanup could assert that > *INTERRUPTS-ENABLED* is NIL. Thus unclean code could be weeded out. > > Maybe something like this: > > (let (x) > (unwind-protect-without-interrupts-in-cleanup > (progn > (setq-without-interrupts x (open ...)) > ...) > (when x (close x))))) > > would be easier to read? > > astor Yes, it's easier to read and within the bowels of sbcl it might even be utilized. Since async signals are not standard, people just don't expect arbitrary things to happen at any time. It's just hard. Something must be done to bend async signals into lisp. I thought about making serve-event check for pending interrupts. When a possibly blocking I/O call is made the user should be well prepared to handle arbitrary conditions anyway, right? Yeah, it is a bit of stretch. Especially considering that the caller needs to know exactly which calls can do I/O or call serve-event. Not likely, around methods and such make it even less so. At this point I got disheartened and come to think that abandoning a side-effecting computation (with-timeout, terminate-thread, C-c + abort) is practically unsafe wrt unwind-protect. With the exception of with-timeout they should not "normally" (not during development or as last resort) be used. Ordinary code is just full of assumptions of what can and cannot fail. That leaves us with some documentation to write and with-timeout. A quick audit on the paserve code base revealed that its need could be satisfied with: - deadlines on streams Currently, timeouts are per-operation. Sending one byte every timeout seconds is enough to keep a reader on the other end of the stream busy. For a web server, that has a stream for each request but performs multiple operations on the stream an upper limit for the sum of the duration of those operations is needed. (setf (stream-deadline stream) (+ 180 (get-universal-time))) (loop repeat 100 do (read-char stream)) A slight problem is clobbering deadlines already set. The following macro temporarily overwrites the deadline if the new deadline is earlier than the old one. (with-stream-deadline (stream (+ 5 (get-universal-time))) (read stream)) - a timeout on socket-connect, open (?), close The current implementation of with-timeout should work for aborting the system calls (connect, open). That would mean adding a :timeout parameter to socket-connect, and open on the lisp side. close with :abort t should not block, unless SO_LINGER is set. The plan: 1) implement a with-timeout that works with threads 2) add timeout parameters to socket-connect, open, ... (?) 3) implement stream deadlines Unfortunately, 1) needs a scheduler (like Zach's timer http://www.cliki.net/TIMER, 500 lines), but - on the bright side - it's still possible to implement with setitimer and does not require a dedicated thread. Please, flame this approach properly. Cheers, Gábor ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf _______________________________________________ Sbcl-devel mailing list Sbcl-devel(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sbcl-devel

3 9

[erlisp-devel] broken process linking code
by Eric Lavigne 16 Sep '05

16 Sep '05

This is the process-linking code I wrote over the summer. It doesn't work, and classes keep me too busy to continue working on it right now. Hopefully someone will have time to fix it. Otherwise I will probably get back to it in a few months. The problem, most likely, is in messaging.lisp. I added calls to a new function at the beginning of send, receive, and receive-with-matcher, so that if a process were marked for destruction it would die before affecting other processes (thus the hook in send) and as soon as possible (thus the hooks in a couple other places). I didn't understand the macros that I was trying to modify, so I was not so surprised to get a bad result. Keeping the hook only in send might fix the problem, but that could be horribly inefficient depending on how often the process tries to send messages. There is also a need for a new test for process-linking. I vaguely remember writing at least a basic test, but there doesn't seem to be one in my working directory. manager.lisp most new code messaging.lisp hooks into old code created in send/receive/receive-with-matcher process.lisp "time-to-die" flag added to process class erlisp.asd new entry for manager.lisp

1 0

[erlisp-devel] Fwd: [Sbcl-devel] async signals
by Faré 15 Sep '05

15 Sep '05

Next message by Gabor on same thread... [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] In reviewing the history of the English Government, its wars and its taxes, a bystander, not blinded by prejudice nor warped by interest, would declare that taxes were not raised to carry on wars, but that wars were raised to carry on taxes. -- Thomas Paine, "Rights of Man" ---------- Forwarded message ---------- From: Gábor Melis <mega(a)hotpop.com> Date: 15-Sep-2005 11:08 Subject: Re: [Sbcl-devel] async signals To: sbcl-devel(a)lists.sourceforge.net Time flies. With #lisp becoming immune to async unwind, it's now sbcl-devil's time to think. The following might become part of the internals manual if the patch goes in, hence the repetition from earlier posts. * Issues with asynchronous unwinds (AUs) Consider the following example: (with-timeout 1 (let (fd) (unwind-protect (progn (setq fd (open ...)) ...) (when fd (close fd))))) There are several ways things can go wrong: ** (SETQ FD (OPEN ...)) can be interrupted and unwound just after OPEN returns but before SETQ is done ** foreign code may not like being aborted: imagine a Berkeley db update call in the second ..., it will not like being unwound in the middle ** cleanups may not be run: if the cleanup is interrupted it can be unwound without closing the fd ** AUs can clobber each other Suppose we have a thread that can be terminated cleanly: (defun greet () (unwind-protect (loop (write-line "Hello, world") (sleep 1)) (write-line "Good bye"))) (let ((thread (make-thread #'greet))) (sleep (random 6)) (terminate-thread thread)) So far so good. But what if two other threads try to terminate it at the same time? The second terminate can hit while the first unwinding is in still progress. This is not a problem since the target of the two throws is the same. Now, what happens if there is another AU with a different target? (defun greet-impatiently () (handler-case (sb-ext:with-timeout 3 (greet)) (sb-ext:timeout ()))) Let's try terminating it: (let ((thread (make-thread #'greet-impatiently))) (sleep (random 6)) (terminate-thread thread)) There are several possible outcomes, but the most interesting is when TERMINATE-THREAD starts unwinding then the timeout hits does another NLX to the HANDLER-CASE and the thread termination request is lost. Note that the UNWIND-PROTECT in GREET is not strictly needed for this scenario to occur. In general it is very hard to write reliable programs when multiple AUs play and can cancel or steer away (see http://www.lisp.org/HyperSpec/Issues/iss152-writeup.html) an ongoing unwind. * Implementation ** Interruption By definition an interruption is a function that is invoked asynchronously. ** AU unsafe zone A thread is said to be in AU unsafe zone iff execution is within a WITHOUT-ASYNCHRONOUS-UNWIND form, an UNWIND-PROTECT cleanup or it's unwinding. Note that there can be multiple simultaneous unwinds: (catch 'aaa (unwind-protect (throw 'aaa 'a) (catch 'bbb (throw 'bbb 'b)))) The implementation (on x86 only for the time being) keeps track of the outermost unwind-protect block that's being unwound to. If an AU occurs in an unsafe zone then the current AU handlers get to decide what happens. ** AU handlers Interruptions are run in a pretty normal dynamic environment with all the condition handlers that were setup in the thread. When an NLX occurs that leaves the interruption UNSAFE-ASYNCHRONOUS-UNWIND (a SERIOUS-CONDITION) is signalled if the interrupted thread is in an unsafe zone. Now if this condition was handled by the normal condition handlers a simple HANDLER-CASE for SERIOUS-CONDITION could unknowingly unwind the stack when it's unsafe. Hence, UNSAFE-ASYNCHRONOUS-UNWIND is signalled with a different set of condition handlers active. These handlers can be established by ASYNCHRONOUS-UNWIND-HANDLER-BIND or from within the interruption with PUSH-INTERRUPTION-UNWIND-HANDLERS, POP-INTERRUPTION-UNWIND-HANDLERS, the setfable INTERRUPTION-UNWIND-HANDLERS. (Bleh. I can see no way around this.) These handlers are run, protected by a WITHOUT-INTERRUPTS, when the interruption is about to be left (from an UNWIND-PROTECT cleanup around the interruption, in fact). The *only* restarts available are RETRY-LATER, ABORT (aborts the unwind) and CONTINUE (forces the unsafe unwind to continue). The AU handler may do an NLX by any of the usual suspects (THROW, RETURN-FROM, GO), but if it signals a condition only the AU handlers are there to help. Behind the scenes interruptions are run by INVOKE-INTERRUPTION that on a NLX from the interruption signals an UNSAFE-ASYNCHRONOUS-UNWIND (a SERIOUS-CONDITION). ** The RETRY-LATER restart The default AU handler invokes this restart. This is similar to the ABORT restart, but it sets up a timer to run the whole interruption again in a short time (~0.1s, randomized). Alternatively the implementation could poll for interruptions when the unsafe zone is potentially exited. This is not done since detecting unsafe zones is currently expensive, because the stack is searched for cleanup frames. ** Foreign code Foreign code is to be wrapped in WITHOUT-ASYNCHRONOUS-UNWIND by default. System calls are a different: they can be interrupted without ill effects. ** Examples Consider the slightly modified example: (with-timeout 1 (let (fd) (unwind-protect (progn (without-asynchronous-unwind () (setq fd (open ...))) ...) (when fd (close fd))))) This is bullet-proof wrt to open/close. The problems outlined above are gone: the fd cannot be lost by unwinding just after OPEN returns but before the SETQ is completed; the cleanup and CLOSE within it is guaranteed to run without much disturbance. But an unwind due to timeout cannot happen in the AU unsafe zones, and that means OPEN and maybe CLOSE should take a timeout argument. Also note that WITH-TIMEOUT doesn't work in cleanup forms (!), so this will not return in one second if CLOSE runs into problems: (unwind-protect ... (with-timeout 1 (close fd))) One can work around that by: (unwind-protect ... (asynchronous-unwind-handler-bind ((unsafe-asynchronous-unwind #'continue)) (with-timeout 1 (close fd)))) but more care is needed as it allows an ongoing unwind to be lost. * An alternative: double cross implementation It can be argued that an AU becomes a problem only if it would cross a WITHOUT-ASYNCHRONOUS-UNWIND, a cleanup, or cancel/steer away an ongoing unwind (i.e. it crosses an border of an interruption and an unsafe zone delimiter). Ultimately, it is about user expectations about what can go wrong and when. It's unclear if such a weakening of semantic the guarantees is worthwhile. In the above example if CLOSE has a HANDLER-CASE for ERROR then an AU can still make it fail without closing the fd. * TODO ** detection of cleanups is slow (walks the stack) ** detection of cleanups is racy And here is the patch: http://retes.hu/~mega/au.patch It has just reached a state where one can experiment with C-c-ing: (without-asynchronous-unwind () (sleep 5)) (unwind-protect t (sleep 5)) and get reasonable behaviour, so it must be ready for comments. Cheers, Gábor ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php _______________________________________________ Sbcl-devel mailing list Sbcl-devel(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sbcl-devel

1 0

[erlisp-devel] Erlisp file loading order
by Eric Lavigne 02 Sep '05

02 Sep '05

I am running into another issue involving file loading order. node.lisp contains the following line: (defvar *current-node* (make-instance 'local-node)) Now that nodes are more than just an empty class (each node has a system-manager), there is a need to call functions while creating node instances (such as spawn). Is it okay to have circular dependencies in asdf? (:file "node" :depends-on ("package" "process")) (:file "process" :depends-on ("package" "mailbox" "node" "compatibility"))

3 2

[erlisp-devel] undefined-symbol
by Eric Lavigne 02 Sep '05

02 Sep '05

I am confused by this error message. ; ; File: /root/erlisp/erlisp-cmucl/erlisp/src/manager.lisp ; In: DEFUN SYSTEM-MANAGER-PROCESS ; (RECEIVE-WITH-MATCHER (COND-MATCHER M) (T M)) ; Error: (during macroexpansion) ; Error in KERNEL::UNDEFINED-SYMBOL-ERROR-HANDLER: the function LISTEN-TO-SYSTEM-MANAGER is undefined. ; ; Error: (during macroexpansion) ; Error in KERNEL::UNDEFINED-SYMBOL-ERROR-HANDLER: the function LISTEN-TO-SYSTEM-MANAGER is undefined. ; ; Converted SYSTEM-MANAGER-PROCESS. It complains that function LISTEN-TO-SYSTEM-MANAGER is undefined, but you can see here that this undefined function is defined immediately before the function that complains about it being missing. It seems like everything I've tried over the last two days lead to some sort of missing function error, and this still doesn't make sense to me. When I write small Lisp codes on my own, it doesn't seem to matter in which order I define functions. As long as all the functions are defined before I start actually running the functions, everything just works. What is different here? Is symbol scope related to Excerpt from manager.lisp: (defun listen-to-system-manager () (when (slot-value (current-process) 'time-to-die) (suicide "System manager requested termination."))) (defun system-manager-process () (do ((msg (receive-with-matcher (cond-matcher m) (t m)) (receive-with-matcher (cond-matcher m) (t m)))) (nil) (cond ((eq 'link (first msg)) (setf (gethash (list (second msg) (third msg)) *link-table*) (fourth msg))) ((eq 'exit (first msg)) (maphash #'(lambda (k v) (let ((src (second msg)) (reason (third msg))) (when (member src k) (when (eq src (first k)) (cond ((or (eq v 'notify) (and (eq v 'default) (eq reason 'normal))) (send (second k) msg)) (t (kill-process (second k))))) (remhash k *link-table*)))) *link-table*))))) For reference, I have enclosed manager.lisp (new) and process.lisp (modified). Eric

3 3

[erlisp-devel] receive
by Eric Lavigne 02 Sep '05

02 Sep '05

(setf msg (receive t)) Would this receive any available message and assign that message to msg? Eric

2 2

[erlisp-devel] Description of pushed patches.
by Dirk Gerrits 01 Sep '05

01 Sep '05

sbcl no longer supports (sb-thread:current-thread-id). bryan d. o'connor <bryan(a)lunch.org>**20050830023327] openmcl-threads using semaphores. bryan d. o'connor <bryan(a)lunch.org>**20050901181244] allegro-threads using gates. bryan d. o'connor <bryan(a)lunch.org>**20050901235948 patch written by Dirk Gerrits.

1 0

[erlisp-devel] receiving, pattern-matching
by Eric Lavigne 01 Sep '05

01 Sep '05

How does receive work? I see that one of the arguments is a pattern for indicating what sort of message I am interested in, but I don't understand how to write/interpret such a pattern. Also, will I receive only the first message that matches this pattern? What happens if no messages match the pattern? Returns nil immediately? Block until such a message arrives? Specifically, I would like to send a list as a message and recognize the message type based on the first element. Example (recognize that this starts with 'system): (list 'system 'link-request 'notify) I would like to receive the first message from the mailbox in *current-process* whose first element is 'system, returning immediately in the case where no such message is found. How could I set up a pattern-matcher for this? Can I use receive? Or will I need to use receive-with-timeout and a short time limit? Better still, is there a way to set up a default handler that would perform predefined action whenever a message matching its pattern shows up? If so, I would like to use this. If not, I think it would be worth adding later. I can imagine writing an entire program made up entirely of default handlers for various message patterns. I think that I'm very close to finishing process-linking, but my confusion over the messaging system is holding me back. Eric

6 10

[erlisp-devel] openmcl patches
by bryan o'connor 01 Sep '05

01 Sep '05

i've filled in the missing pieces in compatibility.lisp to support openmcl. all tests pass reliably except the test receive-with-timeout occassionally runs a few milliseconds too long. my darcs repo is here: http://monday-monkey.com/repos/erlisp/ ...bryan

3 5

[erlisp-devel] Description of pushed patches.
by Dirk Gerrits 01 Sep '05

01 Sep '05

openmcl-threads bryan d. o'connor <bryan(a)lunch.org>**20050830025955 Modified threading procedures in compatibility.lisp to support OpenMCL.

1 0