Eh. Upon reflection, not entirely sure that the increments are happening CV correctly. I will try to move the test to counting the total entries in the hash table, rather than only accessing one key. This should be closer to Alan's original problem.
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
However, since the combination of the two is a common pattern, it would certainly be useful to have an easy to use function that guaranteed the whole update was synchronized.
In any case, that is why I chose to use the keys. Each key for a result is unique so there shouldn't be any contention over key/value pair.
I'm going to so some more tests to see whether the problem is generic or related to the regex call.
I think all the variables I use in the processing lambda are thread local but I will have another look there. Um, just as a sanity check, allocation of space for the closures are thread safe, right?
-Alan
-Alan
On Fri, Sep 27, 2013 at 8:27 AM, Mark Evenson evenson@panix.com wrote:
Eh. Upon reflection, not entirely sure that the increments are happening CV correctly. I will try to move the test to counting the total entries in the hash table, rather than only accessing one key. This should be closer to Alan's original problem.
On Sep 27, 2013, at 8:13 PM, "Alan Ruttenberg" alanruttenberg@gmail.com wrote:
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
A version that only adds to unique keys in the hash table, relying on the HASH-TABLE-COUNT value to indicate successful update. The keys are the squares of the first eight primes, so the total threads spawned is eight. the parameter to RUN now indicates how many squares of the given prime basis. But still weirdness, in that the first thread which should increment the keys of the hashtable indexed by the powers of 2 doesn't seem to execute. Something in my LOOP clause?
On Sep 27, 2013, at 9:28 PM, Mark Evenson evenson@panix.com wrote:
On Sep 27, 2013, at 8:13 PM, "Alan Ruttenberg" alanruttenberg@gmail.com wrote:
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
A version that only adds to unique keys in the hash table, relying on the HASH-TABLE-COUNT value to indicate successful update. The keys are the squares of the first eight primes, so the total threads spawned is eight. the parameter to RUN now indicates how many squares of the given prime basis. But still weirdness, in that the first thread which should increment the keys of the hashtable indexed by the powers of 2 doesn't seem to execute. Something in my LOOP clause?
<threaded-hash.lisp>
Indeed a faulty understanding of LOOP:
CL-USER> (loop :for n :in '(2 3 5 7 11 13 15 17) :doing (threads:make-thread (lambda () (format t "~A " n)))) 3 5 7 11 13 15 17 17 NIL
The value of "n" has already been incremented past the first member of the list when the closure is created with LAMBDA, so the thread with "2" never gets executed.
SLIME users: one needs to place the form (setf swank:*globally-redirect-io* t) in ~/.swank.lisp to get all the FORMAT output in the REPL buffer. Otherwise look in the corresponding *inferior-lisp* buffer.
So, I guess one should loop over closures that have been correctly initialized with the right values in some other manner. I'd go for using DO over LOOP for unless someone can correct my understanding. Either that, or I need to follow macro expansions.
Mark Evenson evenson@panix.com writes:
On Sep 27, 2013, at 9:28 PM, Mark Evenson evenson@panix.com wrote:
On Sep 27, 2013, at 8:13 PM, "Alan Ruttenberg" alanruttenberg@gmail.com wrote:
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
A version that only adds to unique keys in the hash table, relying on the HASH-TABLE-COUNT value to indicate successful update. The keys are the squares of the first eight primes, so the total threads spawned is eight. the parameter to RUN now indicates how many squares of the given prime basis. But still weirdness, in that the first thread which should increment the keys of the hashtable indexed by the powers of 2 doesn't seem to execute. Something in my LOOP clause?
<threaded-hash.lisp>
Indeed a faulty understanding of LOOP:
CL-USER> (loop :for n :in '(2 3 5 7 11 13 15 17) :doing (threads:make-thread (lambda () (format t "~A " n))))
3 5 7 11 13 15 17 17 NIL
The value of "n" has already been incremented past the first member of the list when the closure is created with LAMBDA, so the thread with "2" never gets executed.
SLIME users: one needs to place the form (setf swank:*globally-redirect-io* t) in ~/.swank.lisp to get all the FORMAT output in the REPL buffer. Otherwise look in the corresponding *inferior-lisp* buffer.
So, I guess one should loop over closures that have been correctly initialized with the right values in some other manner. I'd go for using DO over LOOP for unless someone can correct my understanding. Either that, or I need to follow macro expansions.
DO won't help better. For LOOP, DOTIMES, DOLIST, it's unspecified whether the variables are new bindings or updated bindings. But for DO and DO*, it is specified they're updated. So while with the former you had a chance for it to work (not conformingly), you stand no such chance with DO.
What you must write is:
(ql:quickload :bordeaux-threads)
(loop :for n :in '(2 3 5 7 11 13 15 17) :do (let ((n n)) (bt:make-thread (lambda () (format t "~A " n)))))
(do ((ns '(2 3 5 7 11 13 15 17) (cdr ns))) ((null ns)) (let ((n (car ns))) (bt:make-thread (lambda () (format t "~A " n)))))
Ooh good catch! I have such an issue in my thread-per-jar function, the variable f which is bound in the loop, closed over in the thread run function but executed in a thread! That could explain both the missing values and the duplicates one. I'll report back.
Thanks!
Alan
-Alan
On Sat, Sep 28, 2013 at 11:44 AM, Pascal J. Bourguignon pjb@informatimago.com wrote:
Mark Evenson evenson@panix.com writes:
On Sep 27, 2013, at 9:28 PM, Mark Evenson evenson@panix.com wrote:
On Sep 27, 2013, at 8:13 PM, "Alan Ruttenberg" alanruttenberg@gmail.com wrote:
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
A version that only adds to unique keys in the hash table, relying on the HASH-TABLE-COUNT value to indicate successful update. The keys are the squares of the first eight primes, so the total threads spawned is eight. the parameter to RUN now indicates how many squares of the given prime basis. But still weirdness, in that the first thread which should increment the keys of the hashtable indexed by the powers of 2 doesn't seem to execute. Something in my LOOP clause?
<threaded-hash.lisp>
Indeed a faulty understanding of LOOP:
CL-USER> (loop :for n :in '(2 3 5 7 11 13 15 17) :doing (threads:make-thread (lambda () (format t "~A " n))))
3 5 7 11 13 15 17 17 NIL
The value of "n" has already been incremented past the first member of the list when the closure is created with LAMBDA, so the thread with "2" never gets executed.
SLIME users: one needs to place the form (setf swank:*globally-redirect-io* t) in ~/.swank.lisp to get all the FORMAT output in the REPL buffer. Otherwise look in the corresponding *inferior-lisp* buffer.
So, I guess one should loop over closures that have been correctly initialized with the right values in some other manner. I'd go for using DO over LOOP for unless someone can correct my understanding. Either that, or I need to follow macro expansions.
DO won't help better. For LOOP, DOTIMES, DOLIST, it's unspecified whether the variables are new bindings or updated bindings. But for DO and DO*, it is specified they're updated. So while with the former you had a chance for it to work (not conformingly), you stand no such chance with DO. What you must write is: (ql:quickload :bordeaux-threads) (loop :for n :in '(2 3 5 7 11 13 15 17) :do (let ((n n)) (bt:make-thread (lambda () (format t "~A " n))))) (do ((ns '(2 3 5 7 11 13 15 17) (cdr ns))) ((null ns)) (let ((n (car ns))) (bt:make-thread (lambda () (format t "~A " n))))) -- __Pascal Bourguignon__ http://www.informatimago.com/
Verified that that was the problem. Sneaky threads! Thanks for the help!
-Alan
-Alan
On Sat, Sep 28, 2013 at 11:48 AM, Alan Ruttenberg alanruttenberg@gmail.com wrote:
Ooh good catch! I have such an issue in my thread-per-jar function, the variable f which is bound in the loop, closed over in the thread run function but executed in a thread! That could explain both the missing values and the duplicates one. I'll report back. Thanks! Alan -Alan http://alan.ruttenbergs.com/ On Sat, Sep 28, 2013 at 11:44 AM, Pascal J. Bourguignon pjb@informatimago.com wrote:
Mark Evenson evenson@panix.com writes:
On Sep 27, 2013, at 9:28 PM, Mark Evenson evenson@panix.com wrote:
On Sep 27, 2013, at 8:13 PM, "Alan Ruttenberg" alanruttenberg@gmail.com wrote:
Agreed. While the gethash and puthash should independently work, the combination of the two isn't thread safe. For example two processes could get the value in the ht before either has a chance to write back the increment.
A version that only adds to unique keys in the hash table, relying on the HASH-TABLE-COUNT value to indicate successful update. The keys are the squares of the first eight primes, so the total threads spawned is eight. the parameter to RUN now indicates how many squares of the given prime basis. But still weirdness, in that the first thread which should increment the keys of the hashtable indexed by the powers of 2 doesn't seem to execute. Something in my LOOP clause?
<threaded-hash.lisp>
Indeed a faulty understanding of LOOP:
CL-USER> (loop :for n :in '(2 3 5 7 11 13 15 17) :doing (threads:make-thread (lambda () (format t "~A " n))))
3 5 7 11 13 15 17 17 NIL
The value of "n" has already been incremented past the first member of the list when the closure is created with LAMBDA, so the thread with "2" never gets executed.
SLIME users: one needs to place the form (setf swank:*globally-redirect-io* t) in ~/.swank.lisp to get all the FORMAT output in the REPL buffer. Otherwise look in the corresponding *inferior-lisp* buffer.
So, I guess one should loop over closures that have been correctly initialized with the right values in some other manner. I'd go for using DO over LOOP for unless someone can correct my understanding. Either that, or I need to follow macro expansions.
DO won't help better. For LOOP, DOTIMES, DOLIST, it's unspecified whether the variables are new bindings or updated bindings. But for DO and DO*, it is specified they're updated. So while with the former you had a chance for it to work (not conformingly), you stand no such chance with DO. What you must write is: (ql:quickload :bordeaux-threads) (loop :for n :in '(2 3 5 7 11 13 15 17) :do (let ((n n)) (bt:make-thread (lambda () (format t "~A " n))))) (do ((ns '(2 3 5 7 11 13 15 17) (cdr ns))) ((null ns)) (let ((n (car ns))) (bt:make-thread (lambda () (format t "~A " n))))) -- __Pascal Bourguignon__ http://www.informatimago.com/
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Mark Evenson -
Pascal J. Bourguignon