[asdf-devel] Fixing asdf-sytem-connections
I am using asdf-system-connections and notice a glitch in its design. As the original author no longer maintains it, I'm try to fix this myself. This system is designed to allow the loading of a connected system (say "A") automatically when two or more dependent systems (say "B" and "C") are loaded. The problem I'm encountering is that when another system (say "D") is defined to depend on B and C, then A is not loaded until after D is done loading, but D may need the definitions in A. asdf-system-connections defines a function load-connected-systems which does the work of loading the connected system definitions, but this is called from a new #'operate :after method which apparently doesn't get called until A is done loading. I am looking for a place from which to call load-connected-systems that will trigger its call as soon as it's able, i.e., immediately once both "B" and "C" are loaded. I'm not familiar with the internals of ASDF and wonder if someone can guide me to a good place to define this. Thanks, Liam
Dear Liam,
I am using asdf-system-connections and notice a glitch in its design. As the original author no longer maintains it, I'm try to fix this myself. This system is designed to allow the loading of a connected system (say "A") automatically when two or more dependent systems (say "B" and "C") are loaded. The problem I'm encountering is that when another system (say "D") is defined to depend on B and C, then A is not loaded until after D is done loading, but D may need the definitions in A. asdf-system-connections defines a function load-connected-systems which does the work of loading the connected system definitions, but this is called from a new #'operate :after method which apparently doesn't get called until A is done loading.
If you depend on A rather than merely on B and C, then the obvious solution is to depend on A. Actually, I would go so far as to discourage use of asdf-system-connections, and encourage only explicit dependency on connection systems. IIUC, the hu.dwim team changed their systems this way, at my suggestion.
I am looking for a place from which to call load-connected-systems that will trigger its call as soon as it's able, i.e., immediately once both "B" and "C" are loaded. I'm not familiar with the internals of ASDF and wonder if someone can guide me to a good place to define this.
There is no such place. Even what asdf-system-connections is a bit of a stretch. On @(ASDF3), you could try an after method on make-plan, or something like that, but it will be quite ugly, and will confuse the hell out of the methods that call traverse-action directly, etc. Frankly, the whole idea is bunk. A build system ought to be predictable, reproducible, and minimize surprise. asdf-system-connections, as it stands, is not great in this regards, but at least does not disrupt the internal dependency model of ASDF; it stands on top. Your proposed extension requires subverting basic invariants of ASDF internals. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Many people escape old fairy tales (creationism, homeopathy, monarchy) only to fall into new ones (democracy, socialism, global warmism)...
Actually, I would go so far as to discourage use of asdf-system-connections, and encourage only explicit dependency on connection systems. IIUC, the hu.dwim team changed their systems this way, at my suggestion.
and everything became much much better. we just add a new .asd with the name system1+system2.asd and explicitly list whatever dependencies it has, and whatever extra files need to be loaded, usually from under the dir called integration/ in system1. chosing which system/repo/project to put the integration files in is subjective, but usually the choice gives itself. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “There is only one basic human right, the right to do as you damn well please. And with it comes the only basic human duty, the duty to take the consequences.” — P.J. O'Rourke (1947–)
On Sat, Mar 8, 2014 at 7:15 PM, Faré <fahree@gmail.com> wrote:
Dear Liam,
I am using asdf-system-connections and notice a glitch in its design. As the original author no longer maintains it, I'm try to fix this myself. This system is designed to allow the loading of a connected system (say "A") automatically when two or more dependent systems (say "B" and "C") are loaded. The problem I'm encountering is that when another system (say "D") is defined to depend on B and C, then A is not loaded until after D is done loading, but D may need the definitions in A. asdf-system-connections defines a function load-connected-systems which does the work of loading the connected system definitions, but this is called from a new #'operate :after method which apparently doesn't get called until A is done loading.
If you depend on A rather than merely on B and C, then the obvious solution is to depend on A.
Thanks Faré, this is so obvious, I'm embarrassed I didn't think of it myself! Actually, I would go so far as to discourage use of asdf-system-connections,
and encourage only explicit dependency on connection systems. IIUC, the hu.dwim team changed their systems this way, at my suggestion.
I am looking for a place from which to call load-connected-systems that will trigger its call as soon as it's able, i.e., immediately once both "B" and "C" are loaded. I'm not familiar with the internals of ASDF and wonder if someone can guide me to a good place to define this.
There is no such place. Even what asdf-system-connections is a bit of a stretch. On @(ASDF3), you could try an after method on make-plan, or something like that, but it will be quite ugly, and will confuse the hell out of the methods that call traverse-action directly, etc.
Frankly, the whole idea is bunk. A build system ought to be predictable, reproducible, and minimize surprise. asdf-system-connections, as it stands, is not great in this regards, but at least does not disrupt the internal dependency model of ASDF; it stands on top. Your proposed extension requires subverting basic invariants of ASDF internals.
From Symbolics SCT, Volume 12, "Program Development Utilities," p. 24: :load-when-systems-loaded "The :load-when-systems-loaded option instructs SCT not to load some modules of a system when a set of required systems is not loaded. When all the required systems become loaded, SCT automatically loads
Well, I think it has some use, though now in the Quicklisp era it's less important, and that is to define an optional dependency for something that's incidental to the main system. For example, one of my projects (Antik) defines units as part of a large system for doing science/engineering math. As a convenience, it will use the degree symbol. In order to get the degree symbol, I use cl-unicode. This is an incidental use of one (big) system for the occasional (cosmetic) augmentation of another. Most people will not care about units at all, and so loading cl-unicode grows the system mostly unnecessarily, and adding a new user-visible system definition complicates users' mindspace for a mostly unused benefit (and imagine you have 3 optional definitions like this - then you would need to define 7 new systems to capture each combination). The alternative is to give up on the degree symbol, but it is nice to have it when you use degrees, and easy (one line of source code) to add if cl-unicode is present. The old Lisp machine (Symbolics) had an option in the defsystem form :load-when-systems-loaded, which allowed you to add on modules/files to be loaded when certain other specified system(s) were loaded, but would not force the load of those systems. This worked quite well and was very nice for a case like this. the unloaded modules. Including this option in a module has two effects: - If the required systems are not all loaded, that module is not loaded. - When all the required systems become loaded, SCT goes back and loads the module. - You cannot safely patch that module, as it might not be loaded yet at the time patches are loaded. :load-when-systems-loaded differs from the :required-systems option to defsystem in that :required-systems gives an error if the required systems are not present. :load-when-systems-loaded never gives an error. ... Example: (defsystem print-spooler (:default-pathname t) (:module unix-spooler ("ux-spool") (:load-when-systems-loaded :unix-support)) (:serial (:parallel "defs" "macros") "spooler" "unix-spooler")) Suppose the system UNIX-SUPPORT is _not_ loaded. When you load the PRINT-SPOOLER system, all the files in the system are loaded except for those files in UNIX-SPOOLER module (namely, ux-spool). If and when you load the UNIX-SUPPORT system, the files in the UNIX-SPOOLER [module] will get loaded." [Yes, I still have the full set of Symbolics manuals on my shelf.] Liam
Frankly, the whole idea is bunk. A build system ought to be predictable, reproducible, and minimize surprise. asdf-system-connections, as it stands, is not great in this regards, but at least does not disrupt the internal dependency model of ASDF; it stands on top. Your proposed extension requires subverting basic invariants of ASDF internals.
Well, I think it has some use, though now in the Quicklisp era it's less important, and that is to define an optional dependency for something that's incidental to the main system. For example, one of my projects (Antik) defines units as part of a large system for doing science/engineering math. As a convenience, it will use the degree symbol. In order to get the degree symbol, I use cl-unicode. This is an incidental use of one (big) system for the occasional (cosmetic) augmentation of another. Most people will not care about units at all, and so loading cl-unicode grows the system mostly unnecessarily, and adding a new user-visible system definition complicates users' mindspace for a mostly unused benefit (and imagine you have 3 optional definitions like this - then you would need to define 7 new systems to capture each combination). The alternative is to give up on the degree symbol, but it is nice to have it when you use degrees, and easy (one line of source code) to add if cl-unicode is present.
In such cases, I would bite the bullet and always use the other system. RAM is quite cheap, these days, as compared to Lisp program size, and more importantly, to human mindspace. Yes, the combinatorial explosion of dealing with all these options is awful. But having to write it down as combination systems only makes you explicitly aware of the complexity that is implicit in the whole scheme, and that you have to be aware of anyway to debug it. If that complexity is too much for you to deal with, hiding it under the carpet is only making it an attractive nuisance whereby you'll take more complexity than you're ready to deal with. Note that if the concerns are separable, you only need n combined systems, not -1+2^n — but even that might be too much.
The old Lisp machine (Symbolics) had an option in the defsystem form :load-when-systems-loaded, which allowed you to add on modules/files to be loaded when certain other specified system(s) were loaded, but would not force the load of those systems. This worked quite well and was very nice for a case like this.
Those were days of memory-constrained proprietary systems, when legal and technical reasons prevented you from loading all the code. Unless you're targetting handheld devices, I wouldn't bother. Or, if it DOES bother you, you might want to embrace the complexity. PS: in this particular case, I don't see why you need cl-unicode to support just one character. You can just add a constant in your program. "°" or #\° or (code-char 176) is not that hard to write.
[Yes, I still have the full set of Symbolics manuals on my shelf.]
I only had the .ps. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Blaming the prince of the fools should not blind anyone to the vast confederacy of fools that made him their prince.
Liam Healy wrote:
Well, I think it has some use, though now in the Quicklisp era it's less important, and that is to define an optional dependency for something that's incidental to the main system. For example, one of my projects (Antik) defines units as part of a large system for doing science/engineering math. As a convenience, it will use the degree symbol. In order to get the degree symbol, I use cl-unicode. This is an incidental use of one (big) system for the occasional (cosmetic) augmentation of another. Most people will not care about units at all, and so loading cl-unicode grows the system mostly unnecessarily, and adding a new user-visible system definition complicates users' mindspace for a mostly unused benefit (and imagine you have 3 optional definitions like this - then you would need to define 7 new systems to capture each combination). The alternative is to give up on the degree symbol, but it is nice to have it when you use degrees, and easy (one line of source code) to add if cl-unicode is present.
Wouldn't it be sufficient to: 1. Weakly depend upon the cl-unicode system. This will load the cl-unicode system if available, but ignore it (instead of failing the load) if cl-unicode is not present. 2. Detect the successful load or failure downstream in a file-component that adds the degree symbol. Something like (:component "turn-on-unicode-symbols" :if-feature :cl-unicode) I note that this conflicts with Faré's advice in the current manual, which is to use asdf-system-connections *in place of* weakly-depends-on. I am removing that advice, given that A-S-C is deprecated. Question for Faré: you say that weakly-depends-on is not deterministic, and that it's effects depend on load order. Off-hand, I don't see this. If I have a system A that weakly-depends-on B, and then a component, as above, that uses if-feature, where does the order-dependence arise? Indeed, w-d-o seems more deterministic than A-S-C, because the latter tries to load if B is already loaded -- and NOT try to load B if it's not there. That means that A-S-C is more procedural (depending on global state) and less declarative than W-D-O, not less. Or I'm missing something about the semantics. cheers, r -- Robert P. Goldman Staff Scientist Smart Information Flow Technologies (d/b/a SIFT, LLC) 319 1st Ave N., Suite 400 Minneapolis, MN 55401 Voice: (612) 326-3934 Email: rpgoldman@SIFT.net
On Sun, Mar 9, 2014 at 12:46 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Wouldn't it be sufficient to:
1. Weakly depend upon the cl-unicode system. This will load the cl-unicode system if available, but ignore it (instead of failing the load) if cl-unicode is not present.
That would work. I also would have liked to deprecate weakly-depends-on, for the same reasons, but didn't try hard. Not my problem anymore.
2. Detect the successful load or failure downstream in a file-component that adds the degree symbol. Something like
(:component "turn-on-unicode-symbols" :if-feature :cl-unicode)
Won't work, because the plan is computed before the system is loaded.
I note that this conflicts with Faré's advice in the current manual, which is to use asdf-system-connections *in place of* weakly-depends-on. I am removing that advice, given that A-S-C is deprecated.
I would deprecate both.
Question for Faré: you say that weakly-depends-on is not deterministic, and that it's effects depend on load order. Off-hand, I don't see this.
weakly-depends-on should be deterministic in a given installation, so there's that. Load order effects happen only if some crazy extension makes new systems appear. However, weakly-depends-on isn't deterministic from one machine to the next, which makes reproducing bugs harder. I never liked it. I much prefer explicit connection systems A+B.asd
If I have a system A that weakly-depends-on B, and then a component, as above, that uses if-feature, where does the order-dependence arise? Indeed, w-d-o seems more deterministic than A-S-C, because the latter tries to load if B is already loaded -- and NOT try to load B if it's not there. That means that A-S-C is more procedural (depending on global state) and less declarative than W-D-O, not less.
Or I'm missing something about the semantics.
if-feature shouldn't be used for features that are not present at the very beginning of the build. Otherwise w-d-o is cleaner than a-s-c, but still less nice than I like it. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The War on Terrorism is missing the point: what we need is a War on War! — Kennita Watson
Thanks for the clarifications, Faré. I will ponder on further changes to the manual accordingly. I hadn't thought about *when* that :if-feature's value would be computed. I will make a note accordingly. Cheers, r
One last thing occurred to me: as long as the conditionally loaded code is part of one system (e.g., the code for using the degree symbol is part of the units library), we can already do all that's needed using :if-feature. The problem with A-S-C comes if the code is not part of any single system, but is added on to whichever is loaded last. That symmetry means that A-S-C must be a kind of forward chaining system, that is hard to integrate into ASDF. I don't know about the internals of the Symbolics build system; maybe they didn't have the plan-then-execute structure that's in ASDF. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
participants (6)
-
Attila Lendvai -
Faré -
Liam Healy -
Liam Healy -
Robert Goldman -
Robert P. Goldman