Duane Searsmith wrote:
Hi --
Quick question. I skimmed through the archives and didn't see any obvious answer to this question. Is cells thread safe? If not, is there a particular pattern of use that one can follow to avoid problems in a multiprocess scenario?
I have an application that is multithreaded and I would like to explore how to use cells to drive the dataflow. I've read through the examples and looked over the code a bit. Seems like I could get in trouble if multiple threads tried to change a cell with dependencies at the same time.
If this is a silly question and I have missed some obvious explanation somewhere, please just point me in the right direction.
Not at all, good question, I have been waiting for it. :) I am afraid I have never programmed a threaded application, so I do not know what to say. Can threads enqueue on some resource (in the case of Cells, that would be the "data pulse", just a sequential counter of state changes that drives data integrity)? That would do the trick (I think!), but then does that defeat the whole point of threads? If each thread operates on a discrete subset of the application universe, well, I /did/ start to mess with allowing such a beast to have its own pulse -- then we just have to figure out the boundary (assuming one wants at least /some/ dependency to reach thru the wormhole, if you will).
Hmmm, I guess if you are worried about two threads setting the same input cell then clearly they can enqueue on some resource. I guess the next problem is if you want to have the Cells engine itself running in multiple threads, which could be tough. And in /my/ Cells apps, Cells tend to ineluctably take over the whole application, so if Cells runs in one thread only then again -- well, like I said, I have never programmed with threads. I guess I will leave the implications to you.
Any help?
ken
I have an application that is multithreaded and I would like to explore how to use cells to drive the dataflow. I've read through the examples and looked over the code a bit. Seems like I could get in trouble if multiple threads tried to change a cell with dependencies at the same time.
i think a global read-write lock could work, which is acquired for read whenever a cell is checked for validity and upgraded to writing when the cell is invalid and needs recalculation. a write lock locks out all other readers and writers, while readers can operate paralel.
interesting questions arise when the calculations themselves have to acquire other locks in the application... then it's not trivial to ensure the proper locking order everywhere that avoids random deadlocks.
but at first i would just create a big-lock-of-the-world which is acquired whenever anything is used that uses Cells in your app. but this kills paralelism more and more as the Cells-using part of the app is bigger and bigger.
hope i said something useful,
Thank you Ken and Attila for your feedback. I think I have a fair understanding of the issues now. I'm going to do some brainstorming over exactly how I would like the threads to interact with the Cells engine and then try some experiments with different locking strategies and see what happens.
I'll report back on what I discover.
Best, -- Duane
Attila Lendvai wrote:
I have an application that is multithreaded and I would like to explore how to use cells to drive the dataflow. I've read through the examples and looked over the code a bit. Seems like I could get in trouble if multiple threads tried to change a cell with dependencies at the same time.
i think a global read-write lock could work, which is acquired for read whenever a cell is checked for validity and upgraded to writing when the cell is invalid and needs recalculation. a write lock locks out all other readers and writers, while readers can operate paralel.
interesting questions arise when the calculations themselves have to acquire other locks in the application... then it's not trivial to ensure the proper locking order everywhere that avoids random deadlocks.
but at first i would just create a big-lock-of-the-world which is acquired whenever anything is used that uses Cells in your app. but this kills paralelism more and more as the Cells-using part of the app is bigger and bigger.
hope i said something useful,
cells-devel site list cells-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/cells-devel
Duane Searsmith wrote:
Thank you Ken and Attila for your feedback. I think I have a fair understanding of the issues now. I'm going to do some brainstorming over exactly how I would like the threads to interact with the Cells engine and then try some experiments with different locking strategies and see what happens. I'll report back on what I discover.
OK, don't be afraid to ask. And don't forget about lazy cells. :) If you use those, that creates other issues. I don't much.
One thing that might help (or hurt) is that in Cells3 I started forcing the programmer to single-thread model perturbations:
(with-integrity (:change ...) (setf <some-cell> ...)
....and there are multiple queues for follow-up work and propagation arising from any change, including a "client" queue for application work, and one can specify either or both a client queue sort function and client queue handler. So there might be some place to wire in a little thread awareness and let things keep moving despite activity by other threads. And with all that information around, there may also be a way for the programmer to indicate they are taking responsibility for data integrity so Just Let Me Do This. ie, They may know X is against the rules but X is quite safe so Just Do It.
kt
-
Attila Lendvai -
Duane Searsmith -
Ken Tilton