Ken,
as discussed in the ECLM thread:
I look forward to the IR, maybe you have run into something I want to add to Cells to better handle things going away (a frequent problem). In my latest app I am running into a lot of problems with what is generally called referential integrity, specifically external references to kids that have been not-to-be'd. eg, my window's keep a reference to the "focus", which might be in a math problem the student decides to delete. uh-oh. Sprinkling (setf (focus w) nil) all over the place broke down when it ended up erasing a new value that had been set by code that tried to handle the problem by, say, moving the focus to the next problem when deleting the current problem.
What happens in my case is that I have one family observing another like
(defmodel observer (family) () (:default-initargs :kids (kids-list? (loop for kid in (kids (value self)) collecting (make-instance 'observer :value kid :fm-parent *parent*)))))
Then, when we remove a kid in the observed tree, its kids are declared md-dead *before* the observers on those are disposed of, and immediately cells complains about accessing a dead cell.
I am at a loss here, so any insight is greatly appreciated.
My second question sounds really simple: I want to define a list of kids in an input slot, like
(make-instance 'node :kids (c-in (list (make-instance 'node))))
Obviously I am missing the :fm-parent initarg here, so it won't work. But: How do I get the :fm-parent properly assigned for a (c-in initform)?
Thanks, Peter
Peter Hildebrandt wrote:
Ken,
as discussed in the ECLM thread:
I look forward to the IR, maybe you have run into something I want to add to Cells to better handle things going away (a frequent problem). In my latest app I am running into a lot of problems with what is generally called referential integrity, specifically external references to kids that have been not-to-be'd. eg, my window's keep a reference to the "focus", which might be in a math problem the student decides to delete. uh-oh. Sprinkling (setf (focus w) nil) all over the place broke down when it ended up erasing a new value that had been set by code that tried to handle the problem by, say, moving the focus to the next problem when deleting the current problem.
What happens in my case is that I have one family observing another like
(defmodel observer (family) () (:default-initargs :kids (kids-list? (loop for kid in (kids (value self)) collecting (make-instance 'observer :value kid :fm-parent *parent*)))))
Then, when we remove a kid in the observed tree, its kids are declared md-dead *before* the observers on those are disposed of, and immediately cells complains about accessing a dead cell.
I am at a loss here, so any insight is greatly appreciated.
I was thinking while doing the dishes. I can image the observer class being interesting in its own right, and slots over there ending up dependent on the same original kids-list in some way, as well of course as the value of the observer. Propagation would then try to update this slot and when it got to the value of the observer find a dead instance. Any rule that got to the value of the observer by accessing the list of observers (say something iterating over them) would not encounter such an observer/value, but rules lower down that get at the value directly will.
Now normally this is not a problem because such lower down rules would tend not to depend also on the original list, and indeed the reason I have left this unaddressed is that in most cases I have seen a simple way to rewrite my rules that was even better and which did not end up with these widespread dependencies (if you have followed me so far on that, and if I hasten to add all this guesswork is on the money). I like to hold out for Real Problems before whacking away at the code, I think that is a slippery slope.
btw, all that stuff in their that worries about dead instances is preemptive safeguard stuff -- I think if you disable that most things will just work. The rules that are failing now will run harmlessly and in a few cycles everything gets cleaned up anyway. Cells ran for /years/ with this happening to no ill effect (until RoboCup, of all things).
If you want to send me your whole project I will look to see how I would rewrite the rules if that is even possible, and if not take a look at solving this formally.
fyi, in the past I have done silly things like having Cells just return nil on slot-value access to dead cells, but we may want to find something more elegant. :)
kt
Ken,
thanks a lot for all the insight. First things first, using c?n for the kids works like a charm. setf on a c? cell still produces an error, suggesting to initialize the cell with c-in. Anyway, that's settled now.
As to the not-to-be issue, everything can be reproduced using test-gtk. I can share that in the cells cvs, just let me know whether it would be ok to restructure the cvs to match the old cells-gtk directory structure (I'd rather keep it the way it is in the old cells-gtk, so that I can commit it in there one day).
I did some narrowing it down and going through the back trace. So let us look at an example:
root node 1 ---- observer 1 node 1.1 ---- observer 1.1 node 2 ---- observer 2
The basic code for the observer is
(defmodel family-observer (family) ;; we'll use the "value" slot for the observed ((row :reader row :initarg :row) (:default-initargs :kids (kids-list? (bwhen (val (^value)) (mapcar #'(lambda (src) (mk-observer self src)) (kids val)))))))) :row (c? (when-bind* ((parent (upper self)) (pos (position self (kids parent)))) (let ((new-row (tree-row-create (row parent) (id parent)))) (when (tree-row-valid new-row) (tree-row-set-path new-row (row parent) pos) new-row)))))
Where mk-observer is a method specializing on both parameters, so that we can have different kinds of observers on the same type of targets.
The row is some gui object to be kept in sync.
Now we remove node1:
(with-integrity (:change 'tv-del-node) (setf (kids (upper node1)) (remove node1 (kids (upper node1)))))
Then first node2 and observer2 die, ok. Then node1 dies, and so does observer1.
The interesting part:
not-to-be :before on observer 1.1 is called -- and at this point observer 1.1 itself is already :eternal-rest, in other words, not-to-be is called on a dead object. Now not-to-be of the observer wishes to do something, so it accesses a (ruled) slot of the passed object:
(defmethod not-to-be :before ((self cells-tree-node)) (tree-row-destroy (row self))
The call to the accessor (row self) with the dead self triggers a bunch of cells calls:
Backtrace: 0: (CELLS::ENSURE-VALUE-IS-CURRENT NIL #<unused argument> #<unused argument>) 1: ((LABELS CELLS::CHECK-REVERSED) (NIL)) 2: (CELLS::ENSURE-VALUE-IS-CURRENT (NIL . <vld>)=492/OPTIMIZED-AWAY/ROW/DEAD!NODE-TREE-NODE3440] #<unused argument> #<unused argument>) 3: ((LAMBDA (CELLS::OPCODE CELLS::DEFER-INFO)) #<unused argument> #<unused argument>) 4: (CELLS::CALL-WITH-INTEGRITY NIL NIL #<CLOSURE (LAMBDA (CELLS::OPCODE CELLS::DEFER-INFO)) {BBBD3DD}>) 5: (CELLS::CELL-READ (NIL . <vld>)=492/OPTIMIZED-AWAY/ROW/DEAD!NODE-TREE-NODE3440]) 6: (CELLS::MD-SLOT-VALUE DEAD!NODE-TREE-NODE3440 CELLS-GTK::ROW)
The last call is in
(defun ensure-value-is-current (c debug-id ensurer) (declare (ignorable debug-id ensurer)) (count-it :ensure-value-is-current) (when (and (not (symbolp (c-model c)))(eq :eternal-rest (md-state (c-model c)))) (break "model ~a of cell ~a is dead" (c-model c) c))
.... in particular the form:
(c-model c)
which breaks with:
The value NIL is not of type CELL. [Condition of type TYPE-ERROR]
To sum up, I believe the problem is that at a change of the kids list - First the kids are declared dead - Then not-to-be is called recursively and thus not-to-be is passed a dead self.
However, I wish to do some cleanup work when kids are kicked out, and for this I need to access a few slots.
What I'd like to have is - not-to-be being called before the object is declared dead or - another method (last-will?) to be executed right before the kids die or - an interims state (:zombie?) in which cell slots are still accessible with their last cached value
Or is the solution to have an observer on the kids slot instead of a not-to-be-method, which does the cleanup work for the kids?
I was thinking while doing the dishes. I can image the observer class being interesting in its own right, and slots over there ending up dependent on the same original kids-list in some way, as well of course as the value of the observer. Propagation would then try to update this slot and when it got to the value of the observer find a dead instance. Any rule that got to the value of the observer by accessing the list of observers (say something iterating over them) would not encounter such an observer/value, but rules lower down that get at the value directly will.
This is an interesting idea, but I doubt that this is my problem here. It really seems to be the relation between ruled kids slots, declaring cells dead, and not-to-be.
Now normally this is not a problem because such lower down rules would tend not to depend also on the original list, and indeed the reason I have left this unaddressed is that in most cases I have seen a simple way to rewrite my rules that was even better and which did not end up with these widespread dependencies (if you have followed me so far on that, and if I hasten to add all this guesswork is on the money). I like to hold out for Real Problems before whacking away at the code, I think that is a slippery slope.
That is surely true, as this case proves. I feel what we're looking for here is something like family-finalizers which are specified to be called everytime a kid dies.
btw, all that stuff in their that worries about dead instances is preemptive safeguard stuff -- I think if you disable that most things will just work. The rules that are failing now will run harmlessly and in a few cycles everything gets cleaned up anyway. Cells ran for /years/ with this happening to no ill effect (until RoboCup, of all things).
Yep, which is why it broke after introducing cells3 :-)
OTOH, I see why it is good to have these safe guards. I hacked a solution today to the fm-other tree searches which were all over cells-gtk -- now we have with-widget and with-widget-value which do the right thing without kicking off tree searches (I introduced an automatically maintained hashtable of active instances hashing by md-name, like I did in cells-ode).
If you want to send me your whole project I will look to see how I would rewrite the rules if that is even possible, and if not take a look at solving this formally.
As I said, I like to try things out in test-gtk first, so that I can isolate the error (and create a nice demo on the way). I will work through cvs and commit it tomorrow, I hope. I just don't want to force you to have to deal with my whole project -- and all the other issues it has.
fyi, in the past I have done silly things like having Cells just return nil on slot-value access to dead cells, but we may want to find something more elegant. :)
I have such code in my project, too:
(defun deadp (cell) (eql (slot-value cell 'cells::.md-state) :eternal-rest))
However, since I need the slot-value in my case, this does not help ;-)
Thanks again, Peter
Peter Hildebrandt wrote:
[A great article! <g>]
Ken,
thanks a lot for all the insight. First things first, using c?n for the kids works like a charm. setf on a c? cell still produces an error, suggesting to initialize the cell with c-in.
OK, I thought that was only in debug mode, but I did not actually check.
Anyway, that's settled now.
As to the not-to-be issue,...
Ohhhh, it's in the not-to-be. Understood.
Meanwhile over here working on OpenAIR I ended up trying to read dead instances. Have not figured out why, but I have a guess. Still, this comes up so often I think we might want to make this less painful for developers.
... everything can be reproduced using test-gtk. I can share that in the cells cvs, just let me know whether it would be ok to restructure the cvs to match the old cells-gtk directory structure (I'd rather keep it the way it is in the old cells-gtk, so that I can commit it in there one day).
Sure, go for it.
I did some narrowing it down and going through the back trace. So let us look at an example:
root node 1 ---- observer 1 node 1.1 ---- observer 1.1 node 2 ---- observer 2
The basic code for the observer is
(defmodel family-observer (family) ;; we'll use the "value" slot for the observed ((row :reader row :initarg :row) (:default-initargs :kids (kids-list? (bwhen (val (^value)) (mapcar #'(lambda (src) (mk-observer self src)) (kids val)))))))) :row (c? (when-bind* ((parent (upper self)) (pos (position self (kids parent)))) (let ((new-row (tree-row-create (row parent) (id parent)))) (when (tree-row-valid new-row) (tree-row-set-path new-row (row parent) pos) new-row)))))
Where mk-observer is a method specializing on both parameters, so that we can have different kinds of observers on the same type of targets.
May I just observe at this juncture that I wish you had chosen a name other than observer for this class given that Cells already has the observer name in play. :)
The row is some gui object to be kept in sync.
Now we remove node1:
(with-integrity (:change 'tv-del-node) (setf (kids (upper node1)) (remove node1 (kids (upper node1)))))
Then first node2 and observer2 die, ok. Then node1 dies, and so does observer1.
The interesting part:
not-to-be :before on observer 1.1 is called -- and at this point observer 1.1 itself is already :eternal-rest, in other words, not-to-be is called on a dead object. Now not-to-be of the observer wishes to do something, so it accesses a (ruled) slot of the passed object:
(defmethod not-to-be :before ((self cells-tree-node)) (tree-row-destroy (row self))
The call to the accessor (row self) with the dead self triggers a bunch of cells calls:
Backtrace: 0: (CELLS::ENSURE-VALUE-IS-CURRENT NIL #<unused argument> #<unused argument>) 1: ((LABELS CELLS::CHECK-REVERSED) (NIL)) 2: (CELLS::ENSURE-VALUE-IS-CURRENT (NIL . <vld>)=492/OPTIMIZED-AWAY/ROW/DEAD!NODE-TREE-NODE3440] #<unused argument> #<unused argument>) 3: ((LAMBDA (CELLS::OPCODE CELLS::DEFER-INFO)) #<unused argument> #<unused argument>) 4: (CELLS::CALL-WITH-INTEGRITY NIL NIL #<CLOSURE (LAMBDA (CELLS::OPCODE CELLS::DEFER-INFO)) {BBBD3DD}>) 5: (CELLS::CELL-READ (NIL . <vld>)=492/OPTIMIZED-AWAY/ROW/DEAD!NODE-TREE-NODE3440]) 6: (CELLS::MD-SLOT-VALUE DEAD!NODE-TREE-NODE3440 CELLS-GTK::ROW)
The last call is in
(defun ensure-value-is-current (c debug-id ensurer) (declare (ignorable debug-id ensurer)) (count-it :ensure-value-is-current) (when (and (not (symbolp (c-model c)))(eq :eternal-rest (md-state (c-model c)))) (break "model ~a of cell ~a is dead" (c-model c) c))
.... in particular the form:
(c-model c)
which breaks with:
The value NIL is not of type CELL. [Condition of type TYPE-ERROR]
To sum up, I believe the problem is that at a change of the kids list
- First the kids are declared dead
- Then not-to-be is called recursively
and thus not-to-be is passed a dead self.
Yeah, I ran into this once in an observer.
However, I wish to do some cleanup work when kids are kicked out, and for this I need to access a few slots.
Yep.
What I'd like to have is
- not-to-be being called before the object is declared dead
or
- another method (last-will?) to be executed right before the kids die
or
- an interims state (:zombie?) in which cell slots are still
accessible with their last cached value
Or is the solution to have an observer on the kids slot instead of a not-to-be-method, which does the cleanup work for the kids?
I used to do that with kids, but not-to-be is a good place for it.
I might have just made a mistake declaring instances dead before not-to-be. And I am not even sure why I am so unpleasant about allowing access to slot-values of dead stuff. More on this below (you guessed right).
I was thinking while doing the dishes. I can image the observer class being interesting in its own right, and slots over there ending up dependent on
the same original kids-list in some way, as well of course as the value of
the observer. Propagation would then try to update this slot and when it got to the value of the observer find a dead instance. Any rule that got to the
value of the observer by accessing the list of observers (say something
iterating over them) would not encounter such an observer/value, but rules lower down that get at the value directly will.
This is an interesting idea, but I doubt that this is my problem here. It really seems to be the relation between ruled kids slots, declaring cells dead, and not-to-be.
Yeah, I did not know it was a not-to-be.
Now normally this is not a problem because such lower down rules would tend
not to depend also on the original list, and indeed the reason I have left this unaddressed is that in most cases I have seen a simple way to rewrite
my rules that was even better and which did not end up with these widespread
dependencies (if you have followed me so far on that, and if I hasten to add all this guesswork is on the money). I like to hold out for Real Problems
before whacking away at the code, I think that is a slippery slope.
That is surely true, as this case proves. I feel what we're looking for here is something like family-finalizers which are specified to be called everytime a kid dies.
btw, all that stuff in their that worries about dead instances is preemptive safeguard stuff -- I think if you disable that most things will
just work. The rules that are failing now will run harmlessly and in a few
cycles everything gets cleaned up anyway. Cells ran for /years/ with this happening to no ill effect (until RoboCup, of all things).
Yep, which is why it broke after introducing cells3 :-)
OTOH, I see why it is good to have these safe guards.
When this all happened I was days away from having to demo RoboCells at an ILC and I just did what I had to to deal with the many attempts to get to dead instances under Cells2. I think some consequent paranoia carried over to this code.
In 2001, Hal when challenged on his determination of a fault in some device suggests putting it back in and allowing it to fail. I suggest we do the same, but not use the opportunity to send an astronaut drifting off into space to eventually suffocate.
I hacked a solution today to the fm-other tree searches which were all over cells-gtk -- now we have with-widget and with-widget-value which do the right thing without kicking off tree searches (I introduced an automatically maintained hashtable of active instances hashing by md-name, like I did in cells-ode).
haha, almost every day now I think about implementing namespace search that way. :) It is kind of cool to have this automatic location-relative search that Just Works when we have repeating structures and lots of widgets with the same name, but so often a simple hash lookup would suffice.
Part of the magic of the existing scheme, btw, is that it does not matter that some widget might get awakened before the other widget it seeks has even been created. The navigation works by hitting kids slots, which get run JIT to spawn the thing being looked for.
When I saw that working without having been planned for I had a feeling I had stumbled onto something.
If you want to send me your whole project I will look to see how I would rewrite the rules if that is even possible, and if not take a look at solving this formally.
As I said, I like to try things out in test-gtk first, so that I can isolate the error (and create a nice demo on the way). I will work through cvs and commit it tomorrow, I hope. I just don't want to force you to have to deal with my whole project -- and all the other issues it has.
Oh, no need for a reproducible. Now that I know it was in not-to-be it is an understood issue.
fyi, in the past I have done silly things like having Cells just return nil
on slot-value access to dead cells, but we may want to find something more
elegant. :)
I have such code in my project, too:
(defun deadp (cell) (eql (slot-value cell 'cells::.md-state) :eternal-rest))
However, since I need the slot-value in my case, this does not help ;-)
Funny you should mention slot-value. :) That's what I used to get at the slots of dead instances in my not-to-be situation.
Well... I am hesitating. There /is/ an evil variant of corpse access to be blocked as an aid to the developer, a form of access wholly unjustifiable. I think we need to yell if these happen. As for deferring the death certificate, well, what if your not-to-be wants access to a sibling or parent or child who is also being interred? Perhaps they get not-to-be'd, declared dead, and now it is your turn and you try to read them? We ran but could not hide?
ie, I am not sure then where to put the certification of death, tho it occurs to be that the unfinished-business queue (if you have read that far) is exactly where this should go, ie we queue instances up for death certification to run after full propagation (including not-to-be processing) after which point it will be safe to say slot access is an absolute bug.
The neat thing is that we do have a distinct "awaken" stage in ufb processing, why not a distinct "ex-parrot" stage?
Another very quick fix would be simply to do something like with-integrity, which explicitly says "I know this may not run right away". We could wrap the not-to-be call in a with-post-mortem macro dynamically binding *dead-is-cool* to t and then any access anywhere to dead things would be allowed. This would avoid the runtime cost of queueing things up for annihilation (but then I do not think that would be a big runtime burden at all).
Anyway, let me implement *dead-is-cool* and have that bound to t before calling not-to-be so there is no need to wrap not-to-be code at all and see what happens. I'll commit something soon, lemme know if it works. :)
kt
fixes...
To sum up, I believe the problem is that at a change of the kids list
- First the kids are declared dead
- Then not-to-be is called recursively
and thus not-to-be is passed a dead self.
Yeah, I ran into this once in an observer.
Sorry, meant to say "in a not-to-be".
Anyway, let me implement *dead-is-cool* and have that bound to t before calling not-to-be so there is no need to wrap not-to-be code at all and see what happens. I'll commit something soon, lemme know if it works. :)
Done. The special is *not-to-be*.
Cells regression test passes but did not exercise any tests of mortality.
kt
Ken,
First of all, thanks for the quick fix. However, it does not behave quite as it should: cell slots now return t during not-to-be. I think the reason is in md-slot-value.lisp:ensure-value-is-current, more exactly:
(when *not-to-be* (return-from ensure-value-is-current t))
While the function returns otherwise:
(bwhen (v (c-value c)) (if (mdead v) (progn #+shhh (format t "~&on pulse ~a ensure-value still got and still not returning ~a dead value ~a" *data-pulse-id* c v) nil) v)))
That is, (c-value c).
I changed it accordingly:
(when *not-to-be* (return-from ensure-value-is-current (c-value c)))
Which works fine with my example code (that is, the cells-tree-view). I checked it into cvs.
I am not quite sure about the implications, though. I think what I am doing here is return the value of a cell without recalculating it -- That is, I get the value calculated the last time the cell was accessed alive. That means -- if I am right -- the value I work on might be quite old, and a more granular check of the *not-to-be* special might be appropriate.
However, I did a quick test:
;; make two nodes to create a cell dependency CTEST> (defparameter *val* (make-instance 'node :value (c-in 1))) *VAL* CTEST> (defparameter *root* (make-instance 'node :kids (c?n (list (make-kid 'node :value (c? (value *val*))))))) *ROOT*
;; check whether it works CTEST> (value (first (kids *root*))) 1 CTEST> (setf (value *val*) 2) 2 CTEST> (value (first (kids *root*))) 2 ;; yep
;; look at not-to-be, CTEST> (defmethod not-to-be :before ((self node)) (trc "not-to-be :before" (value self))) #<STANDARD-METHOD NOT-TO-BE :BEFORE (NODE) {CEFE609}>
;; make a change CTEST> (setf (value *val*) 3) 3
;; and see ... CTEST> (setf (kids *root*) nil) 0> not-to-be :before 3 0> not-to-be :before 3 NIL ;; wonderful!
So maybe that's it :-)
Cheers, Peter
On Sun, Apr 13, 2008 at 12:57 AM, Ken Tilton kennytilton@optonline.net wrote:
fixes...
To sum up, I believe the problem is that at a change of the kids list
- First the kids are declared dead
- Then not-to-be is called recursively
and thus not-to-be is passed a dead self.
Yeah, I ran into this once in an observer.
Sorry, meant to say "in a not-to-be".
Anyway, let me implement *dead-is-cool* and have that bound to t before
calling not-to-be so there is no need to wrap not-to-be code at all and see what happens. I'll commit something soon, lemme know if it works. :)
Done. The special is *not-to-be*.
Cells regression test passes but did not exercise any tests of mortality.
kt
Peter Hildebrandt wrote:
Ken,
First of all, thanks for the quick fix. However, it does not behave quite as it should: cell slots now return t during not-to-be. I think the reason is in md-slot-value.lisp:ensure-value-is-current, more exactly:
(when *not-to-be* (return-from ensure-value-is-current t))
While the function returns otherwise:
(bwhen (v (c-value c)) (if (mdead v) (progn #+shhh (format t "~&on pulse ~a ensure-value still got and still not returning ~a dead value ~a" *data-pulse-id* c v) nil) v)))
That is, (c-value c).
I changed it accordingly:
(when *not-to-be* (return-from ensure-value-is-current (c-value c)))
Right, thanks, sorry for the head fake. Eventually something should be added to the regression test.
Which works fine with my example code (that is, the cells-tree-view). I checked it into cvs.
I am not quite sure about the implications, though.
Yeah, it is not clear what to do. The instance is no longer part of the model, so some rules will outright fail and those that do not would possibly be circular or at least form crazy dependencies of things no longer extant on those that are.
I think what I am doing here is return the value of a cell without recalculating it -- That is, I get the value calculated the last time the cell was accessed alive. That means -- if I am right -- the value I work on might be quite old, and a more granular check of the *not-to-be* special might be appropriate.
However, I did a quick test:
;; make two nodes to create a cell dependency CTEST> (defparameter *val* (make-instance 'node :value (c-in 1))) *VAL* CTEST> (defparameter *root* (make-instance 'node :kids (c?n (list (make-kid 'node :value (c? (value *val*))))))) *ROOT*
;; check whether it works CTEST> (value (first (kids *root*))) 1 CTEST> (setf (value *val*) 2) 2 CTEST> (value (first (kids *root*))) 2 ;; yep
;; look at not-to-be, CTEST> (defmethod not-to-be :before ((self node)) (trc "not-to-be :before" (value self))) #<STANDARD-METHOD NOT-TO-BE :BEFORE (NODE) {CEFE609}>
;; make a change CTEST> (setf (value *val*) 3) 3
;; and see ... CTEST> (setf (kids *root*) nil) 0> not-to-be :before 3 0> not-to-be :before 3 NIL ;; wonderful!
Hmmm. I see 'not-to-be gets called pretty early, pretty much as soon as a slot value has changed and owned things are seen to be no longer in that slot, right during propagation. So where a dying instance reads slot X and slot X is still alive and would normally update when read (and still will later during this same propagation (I think <g>)) then the dying instance will see either an obsolete value or (I just realized) a new value if propagation got to X first by another path.
Nothing like a little non-determinism to start the day. :)
Well, The Lisp Way is to let people shoot themselves in the foot, we can probably just leave it as it is and see what happens.
If there was any documentation we could document not-to-be and point all this out and remind them that code in not-to-be should not be doing things in or with the model, it should be, eg, notifying C libraries that blocks they allocated can be scavenged.
Hmmm, those /could/ be the result of Cell rules, I guess it is good to return the old value.
kt
On Sun, Apr 13, 2008 at 5:21 PM, Ken Tilton kennytilton@optonline.net wrote:
Right, thanks, sorry for the head fake. Eventually something should be added to the regression test.
Well, we don't want to end up with more lines of test code than program code. So I'd leave that to the debugger for now ;-)
I am not quite sure about the implications, though.
Yeah, it is not clear what to do. The instance is no longer part of the model, so some rules will outright fail and those that do not would possibly be circular or at least form crazy dependencies of things no longer extant on those that are.
I assume that c-value is at least safe (that is, it is bound to return *something*) to do, so the only issue we're facing is that the value might be old.
So my question becomes: How old can the value in the c-value slot become? I do not understand cells well enough to know whether we can safely assume that every change (setf) from before the changing of the kids list has trickled down to every ruled cell *before* not-to-be gets called as a result of a (distinct) change to the kids slot. I assume it would, though.
It is trickier if the change to one input cell through a bunch of dependencies leads to a change in the kids list of some family member. Then not-to-be gets called, and we have no way of being sure which consequences of the original change have already been propagated and which haven't.
But then, again, the value in c-value should be a max of one change old, that is, it should reflect the correct value from before the change which affected the kid to get thrown out.
And now I realize I have just reinvented what you wrote. Damn, I should read the whole message *before* responding.
Hmmm. I see 'not-to-be gets called pretty early, pretty much as soon as a slot value has changed and owned things are seen to be no longer in that slot, right during propagation. So where a dying instance reads slot X and slot X is still alive and would normally update when read (and still will later during this same propagation (I think <g>)) then the dying instance will see either an obsolete value or (I just realized) a new value if propagation got to X first by another path.
Nothing like a little non-determinism to start the day. :)
Well, The Lisp Way is to let people shoot themselves in the foot, we can probably just leave it as it is and see what happens.
Agreed. Especially since I feel the situations in which that could cause a problem are academic in nature. Usually not-to-be will only be concerned with cleanup stuff, and the cell values which are needed therefore would normally be somewhat stable, I assume.
So if it will actually become an issue, we can still go back and invent the :death-bed stage in which not-to-be's are called after all changes have been propagated.
If there was any documentation we could document not-to-be and point all this out and remind them that code in not-to-be should not be doing things in or with the model, it should be, eg, notifying C libraries that blocks they allocated can be scavenged.
I guess the cells-devel archives serve as a doc in some way, so by that logic it is already part of the documentation.
Hmmm, those /could/ be the result of Cell rules, I guess it is good to return the old value.
This is hard now. Because it implies that *no* changes should be propagate dbefore not-to-be is called. But how to do that? Build a queue for *all* changes, filter out the changes to the kids, call not-to-be on the old, intact model, then propagate the changes from the queue? And what if not-to-be inflicts changes? My head is spinning ... I have to wait until after I have finished that bottle of wine.
Cheers, Peter
Peter Hildebrandt wrote:
On Sun, Apr 13, 2008 at 5:21 PM, Ken Tilton kennytilton@optonline.net wrote:
Right, thanks, sorry for the head fake. Eventually something should be added to the regression test.
Well, we don't want to end up with more lines of test code than program code. So I'd leave that to the debugger for now ;-)
Not an XP fan? :) Cells is pretty stable (but that will come unglued if people like you and Andy keep showing up) but I love taking things that go wrong and throwing them into a regression test suite precisely so I can continue hacking like a cowboy and then just re-running the test to see what broke, and if nothing breaks the code is declared perfect cuz if there is any behavior one does not like one has to get into the regression test.
I know the XP guys write the tests first -- that might be a good policy for RFEs: if someone wants something changed they have to specify it in the form of a test that will not pass now. Just a great exercise for clarifying what one wants, seeing if it might already be there, and doing so in a way that guarantees no future change will back it out.
(Sorry, just thinking out loud here.)
I am not quite sure about the implications, though.
Yeah, it is not clear what to do. The instance is no longer part of the model, so some rules will outright fail and those that do not would possibly be circular or at least form crazy dependencies of things no longer extant on those that are.
I assume that c-value is at least safe (that is, it is bound to return *something*) to do, so the only issue we're facing is that the value might be old.
So my question becomes: How old can the value in the c-value slot become? I do not understand cells well enough to know whether we can safely assume that every change (setf) from before the changing of the kids list has trickled down to every ruled cell *before* not-to-be gets called as a result of a (distinct) change to the kids slot. I assume it would, though.
Ah, I was wondering why you were surprised that 3 had gotten through. A naked SETF ends up wrapped in a within-integrity call that ensures data integrity as defined in cells-manifesto.txt (half-way in look for a list of a half-dozen things defining "integrity"). I think that will clarify things greatly.
It is trickier if the change to one input cell through a bunch of dependencies leads to a change in the kids list of some family member. Then not-to-be gets called, and we have no way of being sure which consequences of the original change have already been propagated and which haven't.
But then, again, the value in c-value should be a max of one change old, that is, it should reflect the correct value from before the change which affected the kid to get thrown out.
And now I realize I have just reinvented what you wrote. Damn, I should read the whole message *before* responding.
Hmmm. I see 'not-to-be gets called pretty early, pretty much as soon as a slot value has changed and owned things are seen to be no longer in that slot, right during propagation. So where a dying instance reads slot X and slot X is still alive and would normally update when read (and still will later during this same propagation (I think <g>)) then the dying instance will see either an obsolete value or (I just realized) a new value if propagation got to X first by another path.
Nothing like a little non-determinism to start the day. :)
Well, The Lisp Way is to let people shoot themselves in the foot, we can probably just leave it as it is and see what happens.
Agreed. Especially since I feel the situations in which that could cause a problem are academic in nature. Usually not-to-be will only be concerned with cleanup stuff, and the cell values which are needed therefore would normally be somewhat stable, I assume.
So if it will actually become an issue, we can still go back and invent the :death-bed stage in which not-to-be's are called after all changes have been propagated.
Right, a new :ex-parrot unfinished-business queue to be processed before (I suppose) looking for a new change perhaps kicked off by an observer.
If there was any documentation we could document not-to-be and point all this out and remind them that code in not-to-be should not be doing things in or with the model, it should be, eg, notifying C libraries that blocks they allocated can be scavenged.
I guess the cells-devel archives serve as a doc in some way, so by that logic it is already part of the documentation.
Hmmm, those /could/ be the result of Cell rules, I guess it is good to return the old value.
This is hard now. Because it implies that *no* changes should be propagate dbefore not-to-be is called. But how to do that? Build a queue for *all* changes, filter out the changes to the kids,... call not-to-be on the old, intact model, then propagate the changes from the queue? And what if not-to-be inflicts changes? My head is spinning ... I have to wait until after I have finished that bottle of wine.
Right, it's a lost cause. The Real Problem is not clearly dividing model-time from non-model-time by doing anything modelly during not-to-be. It was a bit extreme (as I once had it and did again by mistake) to force... ha, maybe that /is/ the way to go, force them to use slot-value. I like things like that that say to the user "I do not think you will forget you are in not-to-be as long as only slot-value works.
Anyway...
kt
Well, we don't want to end up with more lines of test code than program code. So I'd leave that to the debugger for now ;-)
Not an XP fan? :) Cells is pretty stable (but that will come unglued if people like you and Andy keep showing up) but I love taking things that go wrong and throwing them into a regression test suite precisely so I can continue hacking like a cowboy and then just re-running the test to see what broke, and if nothing breaks the code is declared perfect cuz if there is any behavior one does not like one has to get into the regression test.
You got a point here: It does make hacking safer. In general, XP always sounded kinda strange to me, though; I feel I trust a function more when it looks well-written than when it passes *some* test.
Maybe that's just because I am not good at writing solid tests. When I write test code it usually only reflects the angle I am looking from at the time of writing and thus becomes rather one-sided. Toss the code in a real world scenario, and the requirements are everything but what I wrote in the tests. And knowing my defencies at writing tests, I rather try to write sound code in the first place than to rely on my (tainted) tests.
It looks quite different in the case you point out, however:
I know the XP guys write the tests first -- that might be a good policy for RFEs: if someone wants something changed they have to specify it in the form of a test that will not pass now. Just a great exercise for clarifying what one wants, seeing if it might already be there, and doing so in a way that guarantees no future change will back it out.
i.e. when the person writing the test is the user of the functionality, not the one who implements it. That makes perfect sense to me, then.
I will try to adapt this idea and post future requests here in the form of a stand-alone piece of code which would work if cells worked the way I want it to.
(Sorry, just thinking out loud here.)
Nothing better than that on a rainy afternoon. :)
So my question becomes: How old can the value in the c-value slot
Ah, I was wondering why you were surprised that 3 had gotten through. A naked SETF ends up wrapped in a within-integrity call that ensures data integrity as defined in cells-manifesto.txt (half-way in look for a list of a half-dozen things defining "integrity"). I think that will clarify things greatly.
Thanks for the pointer! I have read the manifesto a while ago (when I got started with cells), but it makes a lot more sense now :)
So if it will actually become an issue, we can still go back and invent the :death-bed stage in which not-to-be's are called after all changes have been propagated.
Right, a new :ex-parrot unfinished-business queue to be processed before (I suppose) looking for a new change perhaps kicked off by an observer.
Yep, I guess the perfect order would be along these lines:
- a change occurs - build dependency tree - find one or more changes of kids slots in the tree - do a *dry run* of the propagation to figure out what the new kids would be - call all relevant not-to-be's (which need to defer calls to setf with with-integrity) - then do the actual propagation - call observers - run the defered stuff
i.e. one would need to figure out the changes to kids slots, but the n run not-to-be while the system is still in pre-propagation state, so that not-to-be code can use stuff "as if it was still alive".
As I said before, I doubt that the effort for this kind of modification is justified. Let's hack on until we really need that.
Right, it's a lost cause. The Real Problem is not clearly dividing model-time from non-model-time by doing anything modelly during not-to-be. It was a bit extreme (as I once had it and did again by mistake) to force... ha, maybe that /is/ the way to go, force them to use slot-value. I like things like that that say to the user "I do not think you will forget you are in not-to-be as long as only slot-value works.
While we are at it, maybe we should introduce mandatory type declarations for ruled and input cells? :->
Cheers Peter
One hallmark of a good fix is when it makes comments and kludges go away. Just spotted this in Cello:
(defmethod not-to-be :after ((self ogl-node)) (bwhen (dl (slot-value self 'dsp-list)) ;; don't trigger lazy cell (gl-delete-lists dl 1)))
:)
kt
Ken,
as I said, my cells-tree-view works now -- wonderful. Gotta get going with (+ cells-ode open-gl) right away :-)
Meanwhile over here working on OpenAIR I ended up trying to read dead instances. Have not figured out why, but I have a guess. Still, this comes up so often I think we might want to make this less painful for developers.
I see what you mean. It cost me quite a bit of time to track down those instances, especially since the problem is mostly transient, i.e the references to dead objects usually disappear quickly after the objects itself die. That's why I suggested the zombie state. You talk about it in more detail below, so I will hold my comments until then.
Sure, go for it.
Ok, I created a new module for cells-gtk3 in the cells repository, called cells-gtk3. It is up to date, the demo includes the cells-tree-view and two cairo demos, and it runs fairly stable (i.e. I did not manage to break it) both in threading and non-threading mode.
May I just observe at this juncture that I wish you had chosen a name other than observer for this class given that Cells already has the observer name in play. :)
Well, it was your suggestion :) I asked for something "like an observer on a whole family" a while ago on cells-devel, and you suggested the family-observer. In the code, I call it f-observer, because, well, it is something like an observer, only that it observer a whole family, not just a single cell.
What I'd like to have is
- not-to-be being called before the object is declared dead
or
- another method (last-will?) to be executed right before the kids die
or
- an interims state (:zombie?) in which cell slots are still
accessible with their last cached value
Or is the solution to have an observer on the kids slot instead of a not-to-be-method, which does the cleanup work for the kids?
I used to do that with kids, but not-to-be is a good place for it.
Agreed. It makes sense for the object to clean up after itself.
I hacked a solution today to the fm-other tree searches which were all over cells-gtk -- now we have with-widget and with-widget-value which do the right thing without kicking off tree searches (I introduced an automatically maintained hashtable of active instances hashing by md-name, like I did in cells-ode).
haha, almost every day now I think about implementing namespace search that way. :) It is kind of cool to have this automatic location-relative search that Just Works when we have repeating structures and lots of widgets with the same name, but so often a simple hash lookup would suffice.
So I guess we're attacking different problems here. Both in cells-ode and in the cells-gtk cases I was dealing with, I do not have equal names, and the dependencies might very well be wide spread -- in particular I needed that when I wanted to have primitves in the canvas slot of the cairo-drawing-area get their parameters from gtk-widgets. The primitves cannot reside in the kids of the drawing area since cells-gtk assumes that kids of widgets are widgets, and so the family structure breaks there. Maybe I should make the canvas slot :owning t. We'll see.
Part of the magic of the existing scheme, btw, is that it does not matter that some widget might get awakened before the other widget it seeks has even been created. The navigation works by hitting kids slots, which get run JIT to spawn the thing being looked for.
Yep, we talked about that when tracking down evil failures with (+ cells-gtk cells3). I figured that on average a hash table look up will suffice. My macro handles the case of the target nor exisitng (or not being initialized) by returning a specified default value. This way the dependent cell will first default to 0/nil/"" (or whatever specified), then switch to the "real" value once the initialization of the source widget is being propagated.
When I saw that working without having been planned for I had a feeling I had stumbled onto something.
:)
Well... I am hesitating. There /is/ an evil variant of corpse access to be blocked as an aid to the developer, a form of access wholly unjustifiable. I think we need to yell if these happen. As for deferring the death certificate, well, what if your not-to-be wants access to a sibling or parent or child who is also being interred? Perhaps they get not-to-be'd, declared dead, and now it is your turn and you try to read them? We ran but could not hide?
Maybe it is a reasonable limitation to limit access in not-to-be to slots of the passed self object. I see how one could need to update other things, but I also see how this could be solved more elegantly with cells rules (i.e. doing too much in not-to-be looks like imparative programming through the backdoor -- this is not Java, after all ;-)).
ie, I am not sure then where to put the certification of death, tho it occurs to be that the unfinished-business queue (if you have read that far) is exactly where this should go, ie we queue instances up for death certification to run after full propagation (including not-to-be processing) after which point it will be safe to say slot access is an absolute bug.
The neat thing is that we do have a distinct "awaken" stage in ufb processing, why not a distinct "ex-parrot" stage?
Sounds like the ultimate fix (somewhat like the zombie state I suggested), but I wonder whether it is overkill, given that your fix just works.
Another very quick fix would be simply to do something like with-integrity, which explicitly says "I know this may not run right away". We could wrap the not-to-be call in a with-post-mortem macro dynamically binding *dead-is-cool* to t and then any access anywhere to dead things would be allowed. This would avoid the runtime cost of queueing things up for annihilation (but then I do not think that would be a big runtime burden at all).
Anyway, let me implement *dead-is-cool* and have that bound to t before calling not-to-be so there is no need to wrap not-to-be code at all and see what happens. I'll commit something soon, lemme know if it works. :)
As I wrote before, I fixed one form in ensure-value-is-current, and it works beautifully :-)
Cheers, Peter
-
Ken Tilton -
Peter Hildebrandt