Hi All;
tl:dr - AFAICT post-version-2.26-ish versions of ASDF systems listed in :depends-on are unconditionally loaded rather than required - ditto for asdf:load-system. If this is correct (both in intent and my understanding), then how to create a GUI for quicklisp in McCLIM which can quickload any system which :depends-on McCLIM?
Apologies in advance in the likely event this is Pilot Error.
Am trying to write a guihttps://bitbucket.org/symbolicsimulation/com.symsim.oss.ql-guifor quicklisp. Testing on various platforms (Raspberry Pi, linux/SBCL, linux/CCL) with various vintages of ASDF produces differing behavior. In particular, quickloading a system which depends upon McCLIM from the McCLIM-based gui screws up (I haven't tracked down the exact problem, but I could understand why unconditionally reloading McCLIM while it's running could get weird).
After some digging (and rebuilding various versions of SBCL up to 1.1.9), I got as far as seeing asdf:load-system behavior diverging from 2.26 (SBCL 1.0.9) to both 2.32 (CCL 1.9) and 3.0.2 (both SBCL and CCL). AFAICT 2.26 load-system does not unconditionally load an already-loaded system (behaving like and perhaps redundant to asdf:require-system). 2.32 and 3.0.2 load-system unconditionally load systems, and require-system does not. I think I understand why the newer vintages do this.
The issue for me is loading systems that :depend-on McCLIM from inside a McCLIM frame command. In my naivety, I would think that the :depends-on systems should be required rather than loaded (if you've patched them, say, you wouldn't want the patch hammered). For my own stuff, I guess I could require prerequisite systems within the asd file rather than :depending-on them, but this seems wrong (breaks introspection?) - so wrong I feel I may be going about this entirely the wrong way.
Would anybody please be so kind as to throw some advice my way?
-jm
John Morrison wrote:
Hi All;
tl:dr - AFAICT post-version-2.26-ish versions of ASDF systems listed in :depends-on are unconditionally loaded rather than required - ditto for asdf:load-system. If this is correct (both in intent and my understanding), then how to create a GUI for quicklisp in McCLIM which can quickload any system which :depends-on McCLIM?
Apologies in advance in the likely event this is Pilot Error.
Am trying to write a gui https://bitbucket.org/symbolicsimulation/com.symsim.oss.ql-gui for quicklisp. Testing on various platforms (Raspberry Pi, linux/SBCL, linux/CCL) with various vintages of ASDF produces differing behavior. In particular, quickloading a system which depends upon McCLIM from the McCLIM-based gui screws up (I haven't tracked down the exact problem, but I could understand why unconditionally reloading McCLIM while it's running could get weird).
After some digging (and rebuilding various versions of SBCL up to 1.1.9), I got as far as seeing asdf:load-system behavior diverging from 2.26 (SBCL 1.0.9) to both 2.32 (CCL 1.9) and 3.0.2 (both SBCL and CCL). AFAICT 2.26 load-system does not unconditionally load an already-loaded system (behaving like and perhaps redundant to asdf:require-system). 2.32 and 3.0.2 load-system unconditionally load systems, and require-system does not. I think I understand why the newer vintages do this.
The issue for me is loading systems that :depend-on McCLIM from inside a McCLIM frame command. In my naivety, I would think that the :depends-on systems should be required rather than loaded (if you've patched them, say, you wouldn't want the patch hammered). For my own stuff, I guess I could require prerequisite systems within the asd file rather than :depending-on them, but this seems wrong (breaks introspection?) - so wrong I feel I may be going about this entirely the wrong way.
Would anybody please be so kind as to throw some advice my way?
-jm
ASDF:REQUIRE-SYSTEM uses :FORCE-NOT to prevent ASDF from reloading any of the parent systems that have changed.
But LOAD-SYSTEM should not *unconditionally* load parent systems, unless :FORCE is used, or those systems have changed.
Is it possible that something is happening in your interaction with QUICKLISP to cause ASDF to mistakenly believe some of these upstream parts of McCLIM have changed?
Another possibility: the McCLIM ASDF structure is *very* complicated because of the back-ends, which cannot be modeled happily in ASDF, because ASDF isn't great about modeling conditional structure in environmental dependencies. [IMO, really McCLIM should have used LOAD-SYSTEM keywords to allow the loader to specify backends. But that's water under the bridge now...] Perhaps some aspect of this complex structure causes ASDF to mistakenly believe something needs reloading.
best, r
Hi Robert;
Thanks for the advice and suggestions for where to start looking. I will get out the software shovel and start digging, and I shall report back.
Thanks again!
-jm
On Tuesday 03 September 2013 15:51:04 Robert Goldman wrote:
John Morrison wrote:
Hi All;
tl:dr - AFAICT post-version-2.26-ish versions of ASDF systems listed in :depends-on are unconditionally loaded rather than required - ditto for asdf:load-system. If this is correct (both in intent and my understanding), then how to create a GUI for quicklisp in McCLIM which can quickload any system which :depends-on McCLIM?
Apologies in advance in the likely event this is Pilot Error.
Am trying to write a gui https://bitbucket.org/symbolicsimulation/com.symsim.oss.ql-gui for quicklisp. Testing on various platforms (Raspberry Pi, linux/SBCL, linux/CCL) with various vintages of ASDF produces differing behavior. In particular, quickloading a system which depends upon McCLIM from the McCLIM-based gui screws up (I haven't tracked down the exact problem, but I could understand why unconditionally reloading McCLIM while it's running could get weird).
After some digging (and rebuilding various versions of SBCL up to 1.1.9), I got as far as seeing asdf:load-system behavior diverging from 2.26 (SBCL 1.0.9) to both 2.32 (CCL 1.9) and 3.0.2 (both SBCL and CCL). AFAICT 2.26 load-system does not unconditionally load an already-loaded system (behaving like and perhaps redundant to asdf:require-system). 2.32 and 3.0.2 load-system unconditionally load systems, and require-system does not. I think I understand why the newer vintages do this.
The issue for me is loading systems that :depend-on McCLIM from inside a McCLIM frame command. In my naivety, I would think that the
:depends-on systems should be required rather than loaded (if you've
patched them, say, you wouldn't want the patch hammered). For my own stuff, I guess I could require prerequisite systems within the asd file rather than :depending-on them, but this seems wrong (breaks introspection?) - so wrong I feel I may be going about this entirely the wrong way.
Would anybody please be so kind as to throw some advice my way?
-jm
ASDF:REQUIRE-SYSTEM uses :FORCE-NOT to prevent ASDF from reloading any of the parent systems that have changed.
But LOAD-SYSTEM should not *unconditionally* load parent systems, unless :FORCE is used, or those systems have changed.
Is it possible that something is happening in your interaction with QUICKLISP to cause ASDF to mistakenly believe some of these upstream parts of McCLIM have changed?
Another possibility: the McCLIM ASDF structure is *very* complicated because of the back-ends, which cannot be modeled happily in ASDF, because ASDF isn't great about modeling conditional structure in environmental dependencies. [IMO, really McCLIM should have used LOAD-SYSTEM keywords to allow the loader to specify backends. But that's water under the bridge now...] Perhaps some aspect of this complex structure causes ASDF to mistakenly believe something needs reloading.
best, r
A quick further thought: before you load the system (affecting the current state of the image), interrogate ASDF to get the plan it will execute for the load system. That may reveal something interesting, as would some tracing of the process to see what the :FORCE values are, and whether they are passed everywhere appropriately. Comparing traces done with REQUIRE-SYSTEM (to see the FORCE values it supplies) and LOAD-SYSTEM should be instructive. Check to see what files ASDF thinks have changed, to see why it thinks it must reload parts of McCLIM.
Honestly, I forget whether ASDF triggers rebuilds if it thinks the system definition itself has changed. But that might be a possibility (snooping on FIND-SYSTEM might help). I could imagine that some interaction with quicklisp might fool ASDF into thinking the system definition has changed.
Good luck! r
John, which version of McCLIM are you using on which platform? Give us the URL, etc.
I remember battling with mcclim.asd before I could release ASDF 3. It included a lot of cruft from the ASDF 1 days, and various bugs that I fixed together with rpgolman.
It is also possible that McCLIM or one of its dependencies has dependency bugs that used to be hidden by ASDF bugs: unlike ASDF3, ASDF1 and after it ASDF2 don't properly propagate timestamps transitively through dependencies. If somehow McCLIM has some kind of circularity or bootstrapping whereby a file that is depended upon earlier in the build is touched or re-created later in the build, or even more simply, a declared input that doesn't exist or a declared output that is never created, that will cause ASDF3 to reload a lot of things where earlier versions wouldn't.
Can you (1) check that you use the latest versions of McCLIM and its dependencies (2) find a minimal case where things go wrong, and give us a trace of perform-plan?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Ever wonder why the SAME PEOPLE make up ALL the conspiracy theories?
Well, I think I found the thing that triggers the problem - even if I didn't really find the problem itself. The :clim defsystem defines a dependency that reaches into one of its components - the Goatee/presentation-history file. (Maybe the "huh?" comment in the asd file should've clued me in a little before now, eh?) My guess (based on a later filesystem date on the file presentation-history as compared to the other file components of goatee-core) is that recompiling the file as a part of clim later than and as opposed to goatee-core made goatee appear to have changed, forcing recompiiling of parents... Then things got weird.
When I put the #+NIL in the obvious place (see below), things work as expected. (This with ASDF 3.0.2.4 - haven't check the older versions yet...)
Does anybody have any clue why this might have been this way? It seems to be wrong in spirit (recompiling files of another component from outside any operations on that specific component), and clearly raised the eyebrows of at least one McCLIM maintainer... In any event, thanks for the troubleshooting clues, guys, and sorry for any unintentional casting of aspersions...
-jm
(defsystem :clim :depends-on (:clim-core :goatee-core :clim-postscript :drei-mcclim) :components (#+NIL(:file "Goatee/presentation-history" ; XXX: this is loaded as part of the Goatee system. huh? :pathname #.(make-pathname :directory '(:relative "Goatee") :name "presentation-history" :type "lisp")) (:file "input-editing-goatee") (:file "input-editing-drei") (:file "text-editor-gadget") (:file "Extensions/tab-layout" :pathname #.(make-pathname :directory '(:relative "Extensions") :name "tab-layout" :type "lisp"))))
On Wed, Sep 4, 2013 at 1:17 PM, Faré fahree@gmail.com wrote:
John, which version of McCLIM are you using on which platform? Give us the URL, etc.
I remember battling with mcclim.asd before I could release ASDF 3. It included a lot of cruft from the ASDF 1 days, and various bugs that I fixed together with rpgolman.
It is also possible that McCLIM or one of its dependencies has dependency bugs that used to be hidden by ASDF bugs: unlike ASDF3, ASDF1 and after it ASDF2 don't properly propagate timestamps transitively through dependencies. If somehow McCLIM has some kind of circularity or bootstrapping whereby a file that is depended upon earlier in the build is touched or re-created later in the build, or even more simply, a declared input that doesn't exist or a declared output that is never created, that will cause ASDF3 to reload a lot of things where earlier versions wouldn't.
Can you (1) check that you use the latest versions of McCLIM and its dependencies (2) find a minimal case where things go wrong, and give us a trace of perform-plan?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Ever wonder why the SAME PEOPLE make up ALL the conspiracy theories?
On Wed, Sep 11, 2013 at 3:40 PM, John Morrison john.nmi.morrison@gmail.com wrote:
Well, I think I found the thing that triggers the problem - even if I didn't really find the problem itself. The :clim defsystem defines a dependency that reaches into one of its components - the Goatee/presentation-history file. (Maybe the "huh?" comment in the asd file should've clued me in a little before now, eh?) My guess (based on a later filesystem date on the file presentation-history as compared to the other file components of goatee-core) is that recompiling the file as a part of clim later than and as opposed to goatee-core made goatee appear to have changed, forcing recompiiling of parents... Then things got weird.
When I put the #+NIL in the obvious place (see below), things work as expected. (This with ASDF 3.0.2.4 - haven't check the older versions yet...)
Does anybody have any clue why this might have been this way? It seems to be wrong in spirit (recompiling files of another component from outside any operations on that specific component), and clearly raised the eyebrows of at least one McCLIM maintainer... In any event, thanks for the troubleshooting clues, guys, and sorry for any unintentional casting of aspersions...
ASDF 1 (and 2) failed to propagate timestamps from dependencies to dependents, and a file could appear twice, in two different systems, and not cause a dependency loop.
ASDF 3 correctly propagates timestamps from dependencies to dependents. If a file appears twice, the uncompiled or out-of-date first occurrence causes everything that follows to be recompiled, including, later, the second occurrence, which will have a later timestamp, that next time around, will mean the first occurrence has been compiled after everything in the dependency loop.
Conclusion: don't introduce dependency loops. They were always a bug, just one that was previously hidden by a deep bug in ASDF itself.
Don't #+NIL the bastard, just plain kill it.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Dost thou love life? Then do not squander time, for that's the stuff life is made of. — Benjamin Franklin
Understood.
And thanks again!
-jm
On Wed, Sep 11, 2013 at 3:55 PM, Faré fahree@gmail.com wrote:
On Wed, Sep 11, 2013 at 3:40 PM, John Morrison john.nmi.morrison@gmail.com wrote:
Well, I think I found the thing that triggers the problem - even if I
didn't
really find the problem itself. The :clim defsystem defines a dependency that reaches into one of its components - the Goatee/presentation-history file. (Maybe the "huh?" comment in the asd file should've clued me in a little before now, eh?) My guess (based on a later filesystem date on
the
file presentation-history as compared to the other file components of goatee-core) is that recompiling the file as a part of clim later than
and
as opposed to goatee-core made goatee appear to have changed, forcing recompiiling of parents... Then things got weird.
When I put the #+NIL in the obvious place (see below), things work as expected. (This with ASDF 3.0.2.4 - haven't check the older versions yet...)
Does anybody have any clue why this might have been this way? It seems
to
be wrong in spirit (recompiling files of another component from outside
any
operations on that specific component), and clearly raised the eyebrows
of
at least one McCLIM maintainer... In any event, thanks for the troubleshooting clues, guys, and sorry for any unintentional casting of aspersions...
ASDF 1 (and 2) failed to propagate timestamps from dependencies to dependents, and a file could appear twice, in two different systems, and not cause a dependency loop.
ASDF 3 correctly propagates timestamps from dependencies to dependents. If a file appears twice, the uncompiled or out-of-date first occurrence causes everything that follows to be recompiled, including, later, the second occurrence, which will have a later timestamp, that next time around, will mean the first occurrence has been compiled after everything in the dependency loop.
Conclusion: don't introduce dependency loops. They were always a bug, just one that was previously hidden by a deep bug in ASDF itself.
Don't #+NIL the bastard, just plain kill it.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Dost thou love life? Then do not squander time, for that's the stuff life is made of. — Benjamin Franklin
Faré wrote:
On Wed, Sep 11, 2013 at 3:40 PM, John Morrison john.nmi.morrison@gmail.com wrote:
Well, I think I found the thing that triggers the problem - even if I didn't really find the problem itself. The :clim defsystem defines a dependency that reaches into one of its components - the Goatee/presentation-history file. (Maybe the "huh?" comment in the asd file should've clued me in a little before now, eh?) My guess (based on a later filesystem date on the file presentation-history as compared to the other file components of goatee-core) is that recompiling the file as a part of clim later than and as opposed to goatee-core made goatee appear to have changed, forcing recompiiling of parents... Then things got weird.
When I put the #+NIL in the obvious place (see below), things work as expected. (This with ASDF 3.0.2.4 - haven't check the older versions yet...)
Does anybody have any clue why this might have been this way? It seems to be wrong in spirit (recompiling files of another component from outside any operations on that specific component), and clearly raised the eyebrows of at least one McCLIM maintainer... In any event, thanks for the troubleshooting clues, guys, and sorry for any unintentional casting of aspersions...
ASDF 1 (and 2) failed to propagate timestamps from dependencies to dependents, and a file could appear twice, in two different systems, and not cause a dependency loop.
Do you think that could be why the rogue duplicate reference is there? Because it defined something (a macro, perhaps?) such that any changes to Goatee's presentation-history.lisp *should* cause other components of CLIM to recompile, but did not because of the bug in ASDFs 1&2?
Killing this should be safe now, if so.
cheers, r
ASDF 1 (and 2) failed to propagate timestamps from dependencies to dependents, and a file could appear twice, in two different systems, and not cause a dependency loop.
Do you think that could be why the rogue duplicate reference is there? Because it defined something (a macro, perhaps?) such that any changes to Goatee's presentation-history.lisp *should* cause other components of CLIM to recompile, but did not because of the bug in ASDFs 1&2?
That's very possible. At ITA, such crocks had been used to, as workarounds for that deep bug in ASDF, and I had to remove them when upgrading to ASDF 3.
Killing this should be safe now, if so.
Indeed.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org No one can make you feel inferior without your consent — Eleanor Roosevelt But you're only fooling yourself if you can't recognize your superiors — #f
-
Faré -
John Morrison -
Robert Goldman -
Robert P. Goldman