dear list,
my use-case: i want to make sure that when a system is loaded (compiled) due to its test suite being invoked, then a full-blown optional dependency be loaded first, and only then the system itself (which checks for #+(cl:find-package :foo) at compile time).
to be more specific, it's about whether to load or not a full logging library with heavy dependencies. when the test suite is invoked, then debugging will most probably follow, so load logging first that affects the compilation of the system itself.
and to be even more specific, we have a custom develop-op, that does all kind of things, including the seeking and loading of all the optional swank integration systems we have, etc. (note: i was doing the experiments below using plain 'asdf:load-system invoked on :hu.dwim.reiterate+hu.dwim.logger, and AFAICS there are no customizations that should interfere, but hit me on the head with the ASDF manual if you believe otherwise, and i should go back and try to synthesize a stripped down test case.)
this is how i wanted to achieve it:
CL-USER> (asdf:asdf-version) "2.017"
try #1 (in any order, reiterate is compiled before the logger):
:in-order-to ((load-op (load-op :hu.dwim.logger :hu.dwim.reiterate)) (compile-op (compile-op :hu.dwim.logger :hu.dwim.reiterate)))
without any :depends-on entry, which as far as i understand compiles into :in-order-to entries.
try #2 (the before methods seem to get invoked too late when logger has fasls and reiterate doesn't?! but seems to work when neither have fasls?!):
(defsystem :hu.dwim.reiterate+hu.dwim.logger :class hu.dwim.system :description "Loads hu.dwim.logger first, so that proper logging is available when developing, but not making the extra dependency mandatory." :depends-on (:hu.dwim.reiterate :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate+hu.dwim.logger)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate.test)))) (load-system :hu.dwim.logger))
but to the point: what's the contract regarding the load/compile order of dependencies? and if there's any, then how can i tell ASDF to load hu.dwim.logger first? (i couldn't find anything in the manual, but i welcome a "try again in chapter 6", because asdf sources are not too easy on my parser...)
and if the order is not specified in the ASDF contract, then can we apply a randomizer to the otherwise equal dependencies? i've seen it many times before that unrecorded dependencies showed up as filesystem order changed (or something else, it's just an assumption, but the missing dependency often materialized when deploying the system on a different machine).
Dear Attila,
I'm not sure what you want exactly, but it smells like what you really need is
(defsystem hu.dwim.reiterate-test :defsystem-depends-on (:hu.dwim.asdf :hu.dwim.logger) :class :hu.dwim.system :depends-on (:hu.dwim.reiterate) ...)
So did you find a Lisp job in .kz ?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org So that there be reality, there must be an observer. "I am, therefore someone thinks." — Faré
On Thu, Nov 24, 2011 at 04:14, Attila Lendvai attila.lendvai@gmail.com wrote:
dear list,
my use-case: i want to make sure that when a system is loaded (compiled) due to its test suite being invoked, then a full-blown optional dependency be loaded first, and only then the system itself (which checks for #+(cl:find-package :foo) at compile time).
to be more specific, it's about whether to load or not a full logging library with heavy dependencies. when the test suite is invoked, then debugging will most probably follow, so load logging first that affects the compilation of the system itself.
and to be even more specific, we have a custom develop-op, that does all kind of things, including the seeking and loading of all the optional swank integration systems we have, etc. (note: i was doing the experiments below using plain 'asdf:load-system invoked on :hu.dwim.reiterate+hu.dwim.logger, and AFAICS there are no customizations that should interfere, but hit me on the head with the ASDF manual if you believe otherwise, and i should go back and try to synthesize a stripped down test case.)
this is how i wanted to achieve it:
CL-USER> (asdf:asdf-version) "2.017"
try #1 (in any order, reiterate is compiled before the logger):
:in-order-to ((load-op (load-op :hu.dwim.logger :hu.dwim.reiterate)) (compile-op (compile-op :hu.dwim.logger :hu.dwim.reiterate)))
without any :depends-on entry, which as far as i understand compiles into :in-order-to entries.
try #2 (the before methods seem to get invoked too late when logger has fasls and reiterate doesn't?! but seems to work when neither have fasls?!):
(defsystem :hu.dwim.reiterate+hu.dwim.logger :class hu.dwim.system :description "Loads hu.dwim.logger first, so that proper logging is available when developing, but not making the extra dependency mandatory." :depends-on (:hu.dwim.reiterate :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate+hu.dwim.logger)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate.test)))) (load-system :hu.dwim.logger))
but to the point: what's the contract regarding the load/compile order of dependencies? and if there's any, then how can i tell ASDF to load hu.dwim.logger first? (i couldn't find anything in the manual, but i welcome a "try again in chapter 6", because asdf sources are not too easy on my parser...)
and if the order is not specified in the ASDF contract, then can we apply a randomizer to the otherwise equal dependencies? i've seen it many times before that unrecorded dependencies showed up as filesystem order changed (or something else, it's just an assumption, but the missing dependency often materialized when deploying the system on a different machine).
-- attila
Notice the erosion of your (digital) freedom, and do something about it!
PGP: 2FA1 A9DC 9C1E BA25 A59C 963F 5D5F 45C7 DFCD 0A39 OTR XMPP: 8647EEAC EA30FEEF E1B55146 573E52EE 21B1FF06
asdf-devel mailing list asdf-devel@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel
I'm not sure what you want exactly, but it smells like what you really need is
(defsystem hu.dwim.reiterate-test :defsystem-depends-on (:hu.dwim.asdf :hu.dwim.logger) :class :hu.dwim.system :depends-on (:hu.dwim.reiterate) ...)
well, it's not exactly what i want, but it achieves my goal... :)
what i want is to load/compile :hu.dwim.logger only when the test system (not its definition as with the above solution) is loaded, and then load/compile it *before* the main system.
So did you find a Lisp job in .kz ?
i'm not looking for (local) work anymore. the primary reason was to help getting a visa, but there are other alternatives on my radar.
but there are some relatives (here everyone is everyone else's brothers and sisters :) who have a company that is looking for a java guy to program their embedded devices. seems like some evangelism is waiting for me... :)
but anyways, thanks for the solution!
On Fri, Nov 25, 2011 at 09:43, Attila Lendvai attila.lendvai@gmail.com wrote:
I'm not sure what you want exactly, but it smells like what you really need is
(defsystem hu.dwim.reiterate-test :defsystem-depends-on (:hu.dwim.asdf :hu.dwim.logger) :class :hu.dwim.system :depends-on (:hu.dwim.reiterate) ...)
well, it's not exactly what i want, but it achieves my goal... :)
what i want is to load/compile :hu.dwim.logger only when the test system (not its definition as with the above solution) is loaded, and then load/compile it *before* the main system.
There have been many attempts in the past to subvert ASDF into something that allows this kind of effects (see asdf-system-connections), but the semantics of such connections is not very clear (to me at least), and being familiar with ASDF itself, I don't see how such tricks can be made reliable at all.
I'm personally worried about reproducibility, and since ASDF has no way to express control over what gets loaded when beyond its dependencies mechanism, either it's in the dependencies and will always be loaded before, or it's not, and might be loaded before or after and you can't retroactively unload and reload the system to have logging magically take effect.
The only thing I can offer with ASDF is that if you control which outermost toplevel system is loaded (or loading script is run), then this system can control the order in which other systems are loaded. But even then, if you want to recompile in a different context, it's up to you to play with the output-translations and/or clear the fasl cache.
but there are some relatives (here everyone is everyone else's brothers and sisters :) who have a company that is looking for a java guy to program their embedded devices. seems like some evangelism is waiting for me... :)
A few years back, I introduced my cousin's husband to the Gambit Scheme guys in Montreal, and now that's what he's using in his next generation ERP, and he's so happy compared to his previous C++ mess. So — as long as you find intelligent people to discuss with, they may understand that using a better language will give them an edge.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The Law of Eristic Escalation is as follows: Imposition of Order = Escalation of Chaos. Fenderson's Amendment adds that the tighter the order in question is maintained, the longer the consequent chaos takes to escalate, but the more it does when it does.
On 11/24/11 Nov 24 -3:14 AM, Attila Lendvai wrote:
dear list,
my use-case: i want to make sure that when a system is loaded (compiled) due to its test suite being invoked, then a full-blown optional dependency be loaded first, and only then the system itself (which checks for #+(cl:find-package :foo) at compile time).
to be more specific, it's about whether to load or not a full logging library with heavy dependencies. when the test suite is invoked, then debugging will most probably follow, so load logging first that affects the compilation of the system itself.
and to be even more specific, we have a custom develop-op, that does all kind of things, including the seeking and loading of all the optional swank integration systems we have, etc. (note: i was doing the experiments below using plain 'asdf:load-system invoked on :hu.dwim.reiterate+hu.dwim.logger, and AFAICS there are no customizations that should interfere, but hit me on the head with the ASDF manual if you believe otherwise, and i should go back and try to synthesize a stripped down test case.)
this is how i wanted to achieve it:
CL-USER> (asdf:asdf-version) "2.017"
try #1 (in any order, reiterate is compiled before the logger):
:in-order-to ((load-op (load-op :hu.dwim.logger :hu.dwim.reiterate)) (compile-op (compile-op :hu.dwim.logger :hu.dwim.reiterate)))
without any :depends-on entry, which as far as i understand compiles into :in-order-to entries.
It would, I believe, compile into dependencies from LOAD-OP and COMPILE-OP onto LOAD-OP, so would be a little different from what you have here. I am having difficulty thinking about cases where you would really want a COMPILE-OP dependency onto COMPILE-OP instead of LOAD-OP, although I am sure that such cases exist.
try #2 (the before methods seem to get invoked too late when logger has fasls and reiterate doesn't?! but seems to work when neither have fasls?!):
(defsystem :hu.dwim.reiterate+hu.dwim.logger :class hu.dwim.system :description "Loads hu.dwim.logger first, so that proper logging is available when developing, but not making the extra dependency mandatory." :depends-on (:hu.dwim.reiterate :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate+hu.dwim.logger)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate)))) (load-system :hu.dwim.logger))
(defmethod perform :before ((op operation) (system (eql (find-system :hu.dwim.reiterate.test)))) (load-system :hu.dwim.logger))
The above probably won't do what you want. The problem is that, if you have a system X containing serial file components A, B, and C, this is the order of PERFORM calls:
(PERFORM LOAD-OP A) (PERFORM LOAD-OP B) (PERFORM LOAD-OP C) (PERFORM LOAD-OP X)
So that the PERFORM on the SYSTEM actually acts as a kind of AFTER method on the operations on the components, rather than a container, as you would intuitively expect. This is one of the least well-understood attributes of ASDF, but unfortunately would take a fair amount of work to fix.
This *appears* to work in the case you mentioned (no compilations necessary), I believe because you see this:
(PERFORM COMPILE-OP X) (PERFORM LOAD-OP A) (PERFORM LOAD-OP B) (PERFORM LOAD-OP C) (PERFORM LOAD-OP X)
(if there are fasls that need compiling, the COMPILE-OP calls on A, B, and C will come before the COMPILE-OP call on X, and your problem returns).
As a generalization, if you are writing a :BEFORE or :AROUND method on PERFORM for a SYSTEM, you are probably trying to do something that won't work.
I think, as Faré suggests, what you really want here are DEFSYSTEM-DEPENDS-ON entries for reiterate+logger, reiterate and reiterate.test.
but to the point: what's the contract regarding the load/compile order of dependencies? and if there's any, then how can i tell ASDF to load hu.dwim.logger first? (i couldn't find anything in the manual, but i welcome a "try again in chapter 6", because asdf sources are not too easy on my parser...)
and if the order is not specified in the ASDF contract, then can we apply a randomizer to the otherwise equal dependencies? i've seen it many times before that unrecorded dependencies showed up as filesystem order changed (or something else, it's just an assumption, but the missing dependency often materialized when deploying the system on a different machine).
It might be an interesting test discipline (typically of interest only in combination with :FORCE t) to allow people to run ASDF operations with random tie-breaking. This seems like a poor debugging technique, though, since the number of such orderings will explode in systems with weak RECORDED dependencies, so stumbling on a missing dependency might well not happen quickly....
Best, r
There have been many attempts in the past to subvert ASDF into something that allows this kind of effects (see asdf-system-connections), but the semantics of such connections is not very clear (to me at least), and being familiar with ASDF itself, I don't see how such tricks can be made reliable at all.
well, what i need is less than asdf-system-connections, because in my case the dependencies are explicit. i just want to have a word in the load order, also.
a-s-c wants to jump in automagically and load new systems if some other systems are already loaded. we used it for a while but after countless headaches we switched to explicit systems called project1+project2+project3 that depended on the 3 projects and listed what else to load.
before, or it's not, and might be loaded before or after and you can't retroactively unload and reload the system to have logging magically take effect.
i'm programming for long enough to have learnt to make compromises, and i'd be happy to rmfasl and reload when i'm about to debug something.
and re :system-depends-on, it bit me hard. took me hours of debugging why a file is being compiled/loaded twice while i was reinstalled http://dwim.hu and building the sbcl executable that runs on it (the symptom was a warning from defpackage about already exported symbols, it stopped the build).
i was suspecting all kind of weird issues until i've unpulled this patch from hu.dwim.reiterate.
for now, i'm ok. i'll just make sure i don't pull this patch to the live system...
thanks for the help!
It might be an interesting test discipline (typically of interest only in combination with :FORCE t) to allow people to run ASDF operations with random tie-breaking. This seems like a poor debugging technique, though, since the number of such orderings will explode in systems with weak RECORDED dependencies, so stumbling on a missing dependency might well not happen quickly....
i think that exponential blowup of the problem space is not so bad, because the load order issues also materialize in big subspaces of it.
but how about making then sorting the load order? (but then what about different lisps sorting unicode differently?)
i've learned to live with this, but i thought i bring this up here, maybe someone has some good idea that could go into ASDF2.
On Fri, Dec 9, 2011 at 03:23, Attila Lendvai attila.lendvai@gmail.com wrote:
It might be an interesting test discipline (typically of interest only in combination with :FORCE t) to allow people to run ASDF operations with random tie-breaking. This seems like a poor debugging technique, though, since the number of such orderings will explode in systems with weak RECORDED dependencies, so stumbling on a missing dependency might well not happen quickly....
i think that exponential blowup of the problem space is not so bad, because the load order issues also materialize in big subspaces of it.
but how about making then sorting the load order? (but then what about different lisps sorting unicode differently?)
i've learned to live with this, but i thought i bring this up here, maybe someone has some good idea that could go into ASDF2.
So far, ASDF2, after ASDF, has some kind of determinism in its behaviour. If you load the same source in the same order from the same image, you should always have the same result. Adding user-controlled randomization might be fine, but it's just not on my TODO list.
Actually, my TODO list for ASDF is: make a last release 2.20 this year, and try to touch it as little as possible. I do have projects for XCVB, though, and I encourage you to join me in my efforts with XCVB if you're interested in a better build system for CL.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Lambda calculus:
Casual call dumb, A bad CACM lull us, Ada club all scum.
Abacus clad mull, Lab calculus mad. A Dacca bull slum?
Bald caucus mall Balm ad calculus; Calamus call bud.
MBA calculus lad, A clad callus bum, CACM laud as bull.
UCLA all bad scum: A scald alum club, A baud clam scull.
Lambda calculus — Call us a mad club.
- - - Enjoy! 'james
-
Attila Lendvai -
Faré -
Robert Goldman