February 2010 - bordeaux-threads-devel - mailman3.common-lisp.net

[Bordeaux-threads-devel] license verification for inclusion in de.setf.amqp
by james anderson 15 Feb '10

15 Feb '10

good morning; i have released a library, de.setf.amqp[1], which incorporates work of yours. the combined license is the gnu affero. please see the readme[2] for more information. this is to verify the contact information for your library's license. yours is the last respective contact. in some cases, there was none and i have substituted a mailing list as an immediate place-holder, even though, as subscription-only lists, it is not well suited to this purpose. please advise me, should an alternative contact be available. [1] : http://github.com/lisp/de.setf.amqp [2] : http://github.com/lisp/de.setf.amqp/blob/master/README.md

1 0

[Bordeaux-threads-devel] changes for mcl@5.2 (3)
by james anderson 14 Feb '10

14 Feb '10

2 1

[Bordeaux-threads-devel] changes for mcl@5.2
by james anderson 14 Feb '10

14 Feb '10

good afternoon; please find enclosed below a diff relative to patch 162 to improve support for mcl-5.2 + define acquire-lock and release-lock * add dynamic-extent declarations for mcl and ccl ? why repeat the deftype per implementation rather than use satisfies? --- diff -rN old-bordeaux-threads/src/bordeaux-threads.lisp new-bordeaux- threads/src/bordeaux-threads.lisp 110a111,113 > > > (deftype bt:thread () '(satisfies bt:threadp)) diff -rN old-bordeaux-threads/src/impl-clozure.lisp new-bordeaux- threads/src/impl-clozure.lisp 85a86 > (declare (dynamic-extent args)) diff -rN old-bordeaux-threads/src/impl-mcl.lisp new-bordeaux-threads/ src/impl-mcl.lisp 32a33,43 > (defun acquire-lock (lock &optional (wait-p t)) > (if wait-p > (ccl:process-lock lock ccl:*current-process*) > ;; this is broken, but it's better than a no-op > (ccl:without-interrupts > (when (null (ccl::lock.value lock)) > (ccl:process-lock lock ccl:*current-process*))))) > > (defun release-lock (lock) > (ccl:process-unlock lock)) > 44a56 > (declare (dynamic-extent args))

2 1

[Bordeaux-threads-devel] changes for mcl@5.2 (2)
by james anderson 14 Feb '10

14 Feb '10

good afternoon; please find enclosed below a diff relative to patch 162 to improve support for mcl-5.2. apologies for missing the symbol problem in the 0.6 -> 0.7 change + define acquire-lock and release-lock * add dynamic-extent declarations for mcl and ccl * ccl::process is not external ? why repeat the deftype per implementation rather than use satisfies? --- diff -rN old-bordeaux-threads/src/bordeaux-threads.lisp new-bordeaux- threads/src/bordeaux-threads.lisp 110a111,113 > > > (deftype bt:thread () '(satisfies bt:threadp)) diff -rN old-bordeaux-threads/src/impl-clozure.lisp new-bordeaux- threads/src/impl-clozure.lisp 85a86 > (declare (dynamic-extent args)) diff -rN old-bordeaux-threads/src/impl-mcl.lisp new-bordeaux-threads/ src/impl-mcl.lisp 12c12 < 'ccl:process) --- > 'ccl::process) 32a33,43 > (defun acquire-lock (lock &optional (wait-p t)) > (if wait-p > (ccl:process-lock lock ccl:*current-process*) > ;; this is broken, but it's better than a no-op > (ccl:without-interrupts > (when (null (ccl::lock.value lock)) > (ccl:process-lock lock ccl:*current-process*))))) > > (defun release-lock (lock) > (ccl:process-unlock lock)) > 44a56 > (declare (dynamic-extent args))

1 0

[Bordeaux-threads-devel] Bordeaux-threads 0.7->Lispworks support (4, 5 and 6) patch
by Matt Lamari 10 Feb '10

10 Feb '10

Diff is from published 0.7 to a working copy. Lispworks pre-6.0 and 6.0 tested (I tested on 5.1 and 6.0) Added the rest of the API for lispworks. make-recursive-lock and make-lock are now differentiated in LW6 The real differences come in condition support. Pre-6.0 (4 and 5) Lispworks has condition variables emulated via lispworks' old polling lock. Lispworks 6+ condition code just calls through to the new 6.0 condition constructs. Both pass the unit-test, and both notify only a single thread. Please add this to the bordeaux-threads tip and publish. If the format or contents are an impediment please advise so I can correct. Thanks, Matt --- ..\bordeaux-threads_new\bordeaux-threads.asd 2009-12-24 19:35:44.000000000 -0600 +++ bordeaux-threads.asd 2009-12-29 23:21:40.637166200 -0600 @@ -53,7 +53,8 @@ #-thread-support "unsupported") (:file "default-implementations") #+(and thread-support - (or armedbear digitool ecl lispworks)) + (or armedbear digitool ecl) + (not lispworks)) (:file "condition-variables")))) :in-order-to ((test-op (load-op bordeaux-threads-test))) :perform (test-op :after (op c) --- c:\t\impl-lispworks.lisp 2009-12-24 19:35:44.000000000 -0600 +++ impl-lispworks.lisp 2010-02-03 18:07:37.851838700 -0600 @@ -11,7 +11,7 @@ ;;; documentation on the LispWorks Multiprocessing interface can be found at ;;; http://www.lispworks.com/documentation/lw445/LWUG/html/lwuser-156.htm -(mp:initialize-multiprocessing) +; (mp:initialize-multiprocessing) ; Had to comment this out as it would block deployment running ;;; Thread Creation @@ -29,8 +29,15 @@ ;;; Resource contention: locks and recursive locks +(defun make-recursive-lock (&optional name) + (mp:make-lock :name (or name "Anonymous lock requested as recursive") + #-(or lispworks4 lispworks5) :recursivep + #-(or lispworks4 lispworks5) t)) + (defun make-lock (&optional name) - (mp:make-lock :name (or name "Anonymous lock"))) + (mp:make-lock :name (or name "Anonymous lock") + #-(or lispworks4 lispworks5) :recursivep + #-(or lispworks4 lispworks5) nil)) (defun acquire-lock (lock &optional (wait-p t)) (mp:process-lock lock nil @@ -38,12 +45,21 @@ ((numberp wait-p) wait-p) (t nil)))) +(defun acquire-recursive-lock (&rest rest) + (apply #'acquire-lock rest)) + (defun release-lock (lock) (mp:process-unlock lock)) +(defun release-recursive-lock (lock) + (release-lock lock)) + (defmacro with-lock-held ((place) &body body) `(mp:with-lock (,place) ,@body)) +(defmacro with-recursive-lock-held (&rest rest) + `(with-lock-held ,@rest)) + ;;; Resource contention: condition variables (defun thread-yield () @@ -68,4 +84,216 @@ (mp:process-wait (format nil "Waiting for thread ~A to complete" thread) (complement #'mp:process-alive-p) thread)) + + + +#+(or lispworks4 lispworks5) +(progn +; Lispworks 4 and 5 condition support is simulated, albeit via a lightweight wrapper over +; its own polling-based wait primitive. Waiters register with the condition variable, +; and use MP:process-wait which queries for permission to proceed at its own (usspecified) interval. +; http://www.lispworks.com/documentation/lw51/LWRM/html/lwref-445.htm +; A wakeup callback (on notify) is provided to lighten this query to not have to do a hash lookup +; on every poll (or have to serialize on the condition variable) and a mechanism is put +; in place to unregister any waiter that exits wait for other reasons, +; and to resend any (single) notification that may have been consumed before this (corner +; case). Much of the complexity present is to support single notification (as recommended in +; the spec); but a distinct condition-notify-all is provided for reference. +; Single-notification follows a first-in first-out ordering +; +; Performance: With 1000 threads waiting on one condition-variable, the steady-state hit (at least +; as tested on a 3GHz Win32 box) is noise - hovering at 0% on Task manager. +; While not true zero like a true native solution, the use of the Lispworks native checks appear +; fast enough to be an equivalent substitute (thread count will cause issue before the +; waiting overhead becomes significant) +(defstruct (condition-variable (:constructor make-lw-condition (name))) + name + (lock (mp:make-lock :name "For condition-variable") :type mp:lock :read-only t) + (wait-tlist (cons nil nil) :type cons :read-only t) + (wait-hash (make-hash-table :test 'eq) :type hash-table :read-only t) + ; unconsumed-notifications is to track :remove-from-consideration + ; for entries that may have exited prematurely - notification is sent through + ; to someone else, and offender is removed from hash and list + (unconsumed-notifications (make-hash-table :test 'eq) :type hash-table :read-only t)) + +(defun make-condition-variable (&key name) + (make-lw-condition name)) + +(defmacro with-cv-access (condition-variable &body body) + (let ((cv-sym (gensym)) + (slots '(lock wait-tlist wait-hash unconsumed-notifications))) + `(let ((,cv-sym ,condition-variable)) + (with-slots ,slots + ,cv-sym + (macrolet ((locked (&body body) `(mp:with-lock (lock) ,@body))) + (labels ((,(gensym) () ,@slots))) ; Trigger expansion of the symbol-macrolets to ignore + ,@body))))) + +(defmacro defcvfun (function-name (condition-variable &rest args) &body body) + `(defun ,function-name (,condition-variable ,@args) + (with-cv-access ,condition-variable + ,@body))) +#+lispworks (editor:setup-indent "defcvfun" 2 2 7) ; indent defcvfun + +; utility function thath assumes process is locked on condition-variable's lock. +(defcvfun do-notify-single (condition-variable) ; assumes already locked + (let ((id (caar wait-tlist))) + (when id + (pop (car wait-tlist)) + (unless (car wait-tlist) ; check for empty + (setf (cdr wait-tlist) nil)) + (funcall (gethash id wait-hash)) ; call waiter-wakeup + (remhash id wait-hash) ; absence of entry = permission to proceed + (setf (gethash id unconsumed-notifications) t)))) + +; Added for completeness/to show how it's done in this paradigm; but +; The symbol for this call is not exposed in the api +(defcvfun condition-notify-all (condition-variable) + (locked + (loop for waiter-wakeup being the hash-values in wait-hash do (funcall waiter-wakeup)) + (clrhash wait-hash) + (clrhash unconsumed-notifications) ; don't care as everyone just got notified + (setf (car wait-tlist) nil) + (setf (cdr wait-tlist) nil))) + +; Currently implemented so as to notify only one waiting thread +(defcvfun condition-notify (condition-variable) + (locked (do-notify-single condition-variable))) + + + +(defun delete-from-tlist (tlist element) + (let ((deleter + (lambda () + (setf (car tlist) (cdar tlist)) + (unless (car tlist) + (setf (cdr tlist) nil))))) + (loop for cons in (car tlist) do + (if (eq element (car cons)) + (progn + (funcall deleter) + (return nil)) + (let ((cons cons)) + (setq deleter + (lambda () + (setf (cdr cons) (cddr cons)) + (unless (cdr cons) + (setf (cdr tlist) cons))))))))) + +(defun add-to-tlist-tail (tlist element) + (let ((new-link (cons element nil))) + (cond + ((car tlist) + (setf (cddr tlist) new-link) + (setf (cdr tlist) new-link)) + (t + (setf (car tlist) new-link) + (setf (cdr tlist) new-link))))) + +(defcvfun condition-wait (condition-variable lock-) + (mp:process-unlock lock-) + (unwind-protect ; for the re-taking of the lock. Guarding all of the code + (let ((wakeup-allowed-to-proceed nil) + (wakeup-lock (mp:make-lock :name "wakeup lock for condition-wait"))) + ; wakeup-allowed-to-proceed is an optimisation to avoid having to serialize all waiters and + ; search the hashtable. That it is locked is for safety/completeness, although + ; as wakeup-allowed-to-proceed only transitions nil -> t, and that missing it once or twice is + ; moot in this situation, it would be redundant even if ever a Lispworks implementation ever became + ; non-atomic in its assigments + (let ((id (cons nil nil)) + (clean-exit nil)) + (locked + (add-to-tlist-tail wait-tlist id) + (setf (gethash id wait-hash) (lambda () (mp:with-lock (wakeup-lock) (setq wakeup-allowed-to-proceed t))))) + (unwind-protect + (progn + (mp:process-wait + "Waiting for notification" + (lambda () + (when (mp:with-lock (wakeup-lock) wakeup-allowed-to-proceed) + (locked (not (gethash id wait-hash)))))) + (locked (remhash id unconsumed-notifications)) + (setq clean-exit t)) ; Notification was consumed + ; Have to call remove-from-consideration just in case process was interrupted + ; rather than having condition met + (unless clean-exit ; clean-exit is just an optimization + (locked + (when (gethash id wait-hash) ; not notified - must have been interrupted + ; Have to unsubscribe + (remhash id wait-hash) + (delete-from-tlist wait-tlist id)) + ; note - it's possible to be removed from wait-hash/wait-tlist (in notify-single); but still have an unconsumed notification! + (when (gethash id unconsumed-notifications) ; Must have exited for reasons unrelated to notification + (remhash id unconsumed-notifications) ; Have to pass on the notification to an eligible waiter + (do-notify-single condition-variable))))))) + (mp:process-lock lock-))) +) + +#-(or lispworks4 lispworks5) +(progn ; LW6+ version + +(defun make-condition-variable (&key name) + (mp:make-condition-variable :name name)) + +(defun condition-wait (condition-variable lock) + (mp:condition-variable-wait condition-variable lock)) + +(defun condition-notify (condition-variable) ; notifies single + (mp:condition-variable-signal condition-variable)) + +) + +; Generally safe sanity check for the locks and single-notify +(defun unit-test-lw-conditions () + (let ((condition-variable (make-condition-variable :name "Test")) + (test-lock (make-lock)) + (completed nil)) + (loop for id from 0 to 5 do + (let ((id id)) + (make-thread (lambda () (with-lock-held (test-lock) (condition-wait condition-variable test-lock) (push id completed) (condition-notify condition-variable)))))) + (sleep 2) + (if completed + (print "Failed: Premature passage through condition-wait") + (print "Successfully waited on condition")) + (condition-notify condition-variable) + (sleep 2) + (if (and completed (eql (length completed) 6) (equal (sort completed #'<) (loop for id from 0 to 5 collect id))) + (print "Success: All elements notified") + (print (format nil "Failed: Of 6 expected elements, only ~A proceeded" completed))) + + #+(or lispworks4 lispworks5) + (with-cv-access condition-variable + (if (and + (not (or (car wait-tlist) (cdr wait-tlist))) + (zerop (hash-table-count wait-hash)) + (zerop (hash-table-count unconsumed-notifications))) + (print "Success: condition variable restored to initial state") + (print "Error: condition variable retains residue from completed waiters"))) + + (setq completed nil) + (loop for id from 0 to 5 do + (let ((id id)) + (make-thread (lambda () (with-lock-held (test-lock) (condition-wait condition-variable test-lock) (push id completed)))))) + (sleep 2) + (condition-notify condition-variable) + (sleep 2) + (if (= (length completed) 1) + (print "Success: Notify-single only notified a single waiter to restart") + (print (format nil "Failure: Notify-single restarted ~A items" (length completed)))) + (condition-notify condition-variable) + (sleep 2) + (if (= (length completed) 2) + (print "Success: second Notify-single only notified a single waiter to restart") + (print (format nil "Failure: Two Notify-singles restarted ~A items" (length completed)))) + (print "shutting down test") + (loop while (with-lock-held (test-lock) (not (= (length completed) 6))) do + (progn + (condition-notify condition-variable) + (sleep 0))) + (loop for i from 0 to 5 do (condition-notify condition-variable)) + (print "Note: In the case of any failures, assume there are outstanding waiting threads") + (values))) + + + (mark-supported)

3 4

[Bordeaux-threads-devel] the way sb-thread:get-mutex is called
by Attila Lendvai 08 Feb '10

08 Feb '10

...and while we are at acquire-lock, the docstring of sb-thread:get-mutex says that without-interrupts is necessary to be around the call to sb-thread:get-mutex. -- attila

1 0

[Bordeaux-threads-devel] bug around unimplemented functions
by Attila Lendvai 08 Feb '10

08 Feb '10

i've told this to Stelian on IRC, just making sure it gets the record: (bt:acquire-recursive-lock nil) simply returns T on sbcl. a quick glance suggests that unimplemented functions are simply noop (instead of an error i guess). -- attila

1 0