A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
Zach
Zach Beane wrote:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
I feel like I'm missing something. Is there some reason you can't simply make
(defsystem foo-extra :depends-on (:foo) ....)
?
ASDF's process for constructing a build plan from partial-order dependencies is (unless Faré changed something when I wasn't looking) non-deterministic.
If you want it to be deterministic, you should add dependencies sufficient to force build order.
Even if you *could* get the behavior you wanted out of left-to-right ordering in the :depends-on slot, this isn't something you should rely upon.
Best, r
ASDF's process for constructing a build plan from partial-order dependencies is (unless Faré changed something when I wasn't looking) non-deterministic.
does it mean that it's actively randomized?
i remember in our busier days (well before ASDF3) we used to have load order anomalies that seemed like they depended on filesystem order or something external to lisp.
the symptom was that on my dev machine stuff compiled/loaded fine reproducibly, then when the recorded changes got pulled to the live server it suddenly failed to compile/load due to a dependency that was not explicitly added to the .asd file.
i don't really remember if it was just partial reloading, or clean recompile/reload, but i think it was the former, because it was rather baffling and the former wouldn't be too baffling.
if the ordering is actually not deterministic (as opposed to merely unspecified), then maybe it's a good idea to actively randomize lists before dependency constraints are applied?
On Wed, Jul 17, 2013 at 11:56 AM, Attila Lendvai attila.lendvai@gmail.com wrote:
ASDF's process for constructing a build plan from partial-order dependencies is (unless Faré changed something when I wasn't looking) non-deterministic.
does it mean that it's actively randomized?
No, there was never active randomization, but the infamous union-of-dependencies function used in practice to reverse the order in which depends-on is consulted as compared to the order in which it is specified.
i remember in our busier days (well before ASDF3) we used to have load order anomalies that seemed like they depended on filesystem order or something external to lisp.
There used to be many anomalies related to timestamps. ASDF3 should have that fixed, at least.
the symptom was that on my dev machine stuff compiled/loaded fine reproducibly, then when the recorded changes got pulled to the live server it suddenly failed to compile/load due to a dependency that was not explicitly added to the .asd file.
i don't really remember if it was just partial reloading, or clean recompile/reload, but i think it was the former, because it was rather baffling and the former wouldn't be too baffling.
This description is not enough for me to identify what bug (or "feature") you may have been hitting. Inter-system dependencies used to just not be done correctly; also, timestamps could be "interesting" when compiling in one image and loading in another. Finally, time skew between filesystem server and local kernel could be very "interesting". All things that should now be fixed with ASDF3.
if the ordering is actually not deterministic (as opposed to merely unspecified), then maybe it's a good idea to actively randomize lists before dependency constraints are applied?
It is merely unspecified, and happened to be reversed in ASDF1 and after it ASDF2, as compared to user order (however, components would still be traversed in user order).
(In ASDF2, I incrementally refactored the traversal algorithm until I understood it, and some bugs were fixed in corner cases, but the algorithm mainly stayed the same. In ASDF3, to fix a "last" couple of bugs, I had to completely rewrite the algorithm, twice, and all the traversal bugs and quirks should be gone now. See my ASDF3 talk at ELS 2013.)
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org An apple every eight hours will keep three doctors away.
I have seen the same phenomenon that Attila refers to, across different machines and different lisp implementations.
I had been chalking it up to differences in sorting and hash table behaviors.
It can be annoying because, as Attila points out, the nondeterminism/partial order (in the sense that the algorithm as coded in CL does not uniquely determine a build order) can mask bugs in system dependency specification -- system definitions with missing dependencies can seem to work.
I do not believe that this problem is acute enough to try to fix with explicit randomization, especially since the partial ordering is something ASDF exploits to minimize rebuilds. I think it would be better to provide groveling support that supports programmers who wish to check/improve their system definitions. The cost of introducing explicit randomization in code complexity, need for repeated test runs, etc., seems like a bad trade-off.
Actually, an ASDF variant that shuffles partially ordered components randomly, and repeatedly rebuilds, as a tool that one could run on a cloud service or something to check definitions might be a fun project. But it's not something I think should go into core ASDF.
cheers, r
Sent from my iPad
On Jul 17, 2013, at 9:24, Faré fahree@gmail.com wrote:
On Wed, Jul 17, 2013 at 11:56 AM, Attila Lendvai attila.lendvai@gmail.com wrote:
ASDF's process for constructing a build plan from partial-order dependencies is (unless Faré changed something when I wasn't looking) non-deterministic.
does it mean that it's actively randomized?
No, there was never active randomization, but the infamous union-of-dependencies function used in practice to reverse the order in which depends-on is consulted as compared to the order in which it is specified.
i remember in our busier days (well before ASDF3) we used to have load order anomalies that seemed like they depended on filesystem order or something external to lisp.
There used to be many anomalies related to timestamps. ASDF3 should have that fixed, at least.
the symptom was that on my dev machine stuff compiled/loaded fine reproducibly, then when the recorded changes got pulled to the live server it suddenly failed to compile/load due to a dependency that was not explicitly added to the .asd file.
i don't really remember if it was just partial reloading, or clean recompile/reload, but i think it was the former, because it was rather baffling and the former wouldn't be too baffling.
This description is not enough for me to identify what bug (or "feature") you may have been hitting. Inter-system dependencies used to just not be done correctly; also, timestamps could be "interesting" when compiling in one image and loading in another. Finally, time skew between filesystem server and local kernel could be very "interesting". All things that should now be fixed with ASDF3.
if the ordering is actually not deterministic (as opposed to merely unspecified), then maybe it's a good idea to actively randomize lists before dependency constraints are applied?
It is merely unspecified, and happened to be reversed in ASDF1 and after it ASDF2, as compared to user order (however, components would still be traversed in user order).
(In ASDF2, I incrementally refactored the traversal algorithm until I understood it, and some bugs were fixed in corner cases, but the algorithm mainly stayed the same. In ASDF3, to fix a "last" couple of bugs, I had to completely rewrite the algorithm, twice, and all the traversal bugs and quirks should be gone now. See my ASDF3 talk at ELS 2013.)
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org An apple every eight hours will keep three doctors away.
On Wed, Jul 17, 2013 at 12:37 PM, Robert P. Goldman rpgoldman@sift.info wrote:
I have seen the same phenomenon that Attila refers to, across different machines and different lisp implementations.
I remember having lots of issues with ASDF1 and ASDF2 in the past. Hopefully, they are fixed with ASDF3.
I had been chalking it up to differences in sorting and hash table behaviors.
I don't remember ASDF ever iterating over hash-tables, or using sort, for the matter. It seems to me that ASDF has always been deterministic in its traversal order, though its order hasn't always been quite intuitive.
Now if you want actual non-determinism, try POIU in non-deterministic mode (used to be the only option, now deterministic is the default and the somewhat faster non-deterministic mode can be enabled by the user). Of course, any determinism is dependent on the state of the filesystem with respect to what was already successfully compiled, and to your program not depending on the presence or absence of compile-time side-effects that weren't in a proper eval-when.
It can be annoying because, as Attila points out, the nondeterminism/partial order (in the sense that the algorithm as coded in CL does not uniquely determine a build order) can mask bugs in system dependency specification -- system definitions with missing dependencies can seem to work.
I do not believe that this problem is acute enough to try to fix with explicit randomization, especially since the partial ordering is something ASDF exploits to minimize rebuilds. I think it would be better to provide groveling support that supports programmers who wish to check/improve their system definitions. The cost of introducing explicit randomization in code complexity, need for repeated test runs, etc., seems like a bad trade-off.
Actually, an ASDF variant that shuffles partially ordered components randomly, and repeatedly rebuilds, as a tool that one could run on a cloud service or something to check definitions might be a fun project. But it's not something I think should go into core ASDF.
Of course, there is the solution provided by XCVB, of always building each component from its declared dependencies only. A variant of ASDF could use POIU-style forking to achieve the same effect. Or you could use XCVB itself.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org You can learn many things from children. How much patience you have, for instance. — Franklin P. Jones
Robert Goldman rpgoldman@sift.info writes:
Zach Beane wrote:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
I feel like I'm missing something. Is there some reason you can't simply make
(defsystem foo-extra :depends-on (:foo) ....)
?
If they were in separate files, yes. They aren't.
Even if you *could* get the behavior you wanted out of left-to-right ordering in the :depends-on slot, this isn't something you should rely upon.
This was relied upon by some projects, and now it doesn't work.
Zach
Zach Beane wrote:
Robert Goldman rpgoldman@sift.info writes:
Zach Beane wrote:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
I feel like I'm missing something. Is there some reason you can't simply make
(defsystem foo-extra :depends-on (:foo) ....)
?
If they were in separate files, yes. They aren't.
I misunderstood your original question: I thought that this was a dependency problem, but it's not. The problem is that the system authors depended on something contingent about the way FIND-SYSTEM was invoked to find depended-upon systems. I haven't gone back to check, but my guess is that something once iterated and pushed, and now collects.
This was relied upon by some projects, and now it doesn't work.
I'm sorry, but to the best of my knowledge that behavior was never offered up by ASDF to be relied upon. Programmers should not have depended on this behavior.
The only solution I know of is to break foo.asd into multiple files. I realize that this is inconvenient, but restoring backward compatibility involves restoring a state where programmers are relying on a reading that is not just contingent, but contingent in a way that encourages people to write system definitions in a way that will be actively misleading to the reader (since conventionally we read left to right).
Does this happen in enough systems in quicklisp that this is a serious problem? How many? Perhaps we can work out some solution. I realize that in many cases the one-system-per-file mapping can be a nuisance.
Best, r
Robert Goldman rpgoldman@sift.info writes:
Zach Beane wrote:
Robert Goldman rpgoldman@sift.info writes:
Zach Beane wrote:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
I feel like I'm missing something. Is there some reason you can't simply make
(defsystem foo-extra :depends-on (:foo) ....)
?
If they were in separate files, yes. They aren't.
I misunderstood your original question: I thought that this was a dependency problem, but it's not. The problem is that the system authors depended on something contingent about the way FIND-SYSTEM was invoked to find depended-upon systems. I haven't gone back to check, but my guess is that something once iterated and pushed, and now collects.
This was relied upon by some projects, and now it doesn't work.
I'm sorry, but to the best of my knowledge that behavior was never offered up by ASDF to be relied upon. Programmers should not have depended on this behavior.
The only solution I know of is to break foo.asd into multiple files. I realize that this is inconvenient, but restoring backward compatibility involves restoring a state where programmers are relying on a reading that is not just contingent, but contingent in a way that encourages people to write system definitions in a way that will be actively misleading to the reader (since conventionally we read left to right).
Does this happen in enough systems in quicklisp that this is a serious problem? How many? Perhaps we can work out some solution. I realize that in many cases the one-system-per-file mapping can be a nuisance.
It happens for mcclim-freetype and mcclim-truetype and mcclim-freetype-cffi and perhaps some others.
Zach
Yes, Zach's diagnosis is correct.
I consider that a bug in McCLIM, that depended upon unspecified behavior of ASDF 1&2: in ASDF1 and ASDF2, parse-component-form uses union-of-dependencies which somehow reverses the list of dependencies as compared to the specified order. I consider that a fail, and it's probably unintentional behavior in ASDF1, though it is possible that Dan Barlow noticed the order reversal and didn't bother to preserve order, because it is incorrect for the user to depend on the order used by ASDF — which it is, in any case.
A solution that is both backward-compatible and forward compatible is
1- to rename the systems so they follow the foo/bar convention for naming systems in foo.asd, so that ASDF3 can find them. This solves forward compatibility
2- to keep the same order as in the existing defsystem, so it continues to abuse ASDF1 & ASDF2 into magically finding the system before it is used.
Unhappily, McCLIM has to be fixed.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Language independence means you're proud of not calling your barking a language — Faré, about designers of DBMS, OS or other software who claim such
On Sun, Jul 7, 2013 at 8:15 PM, Robert Goldman rpgoldman@sift.info wrote:
Zach Beane wrote:
It happens for mcclim-freetype and mcclim-truetype and mcclim-freetype-cffi and perhaps some others.
OK. Let me have a look at McCLIM, and perhaps we can come up with a work-around.
Best, r
Faré fahree@gmail.com writes:
A solution that is both backward-compatible and forward compatible is
1- to rename the systems so they follow the foo/bar convention for naming systems in foo.asd, so that ASDF3 can find them. This solves forward compatibility
If anyone considering fixing mcclim is reading this, please don't use a slash-named system. It's incompatible with Quicklisp.
Zach
08.07.2013, 01:34, "Zach Beane" xach@xach.com:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
If we surround the non-findable system with the main system like this:
(defsystem bar :depends-on (:foo :foo-extra :foo))
will it work?
:)
It's a dirty workaround (worksurround), but the level of dirtiness is the same as the original: have a system not findable by asdf:find-system due to difference between the system name and it's .asd file name; solve it by relying on the order ASDF loads dependencies, having a findable system name first.
Anton Vodonosov avodonosov@yandex.ru writes:
08.07.2013, 01:34, "Zach Beane" xach@xach.com:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
If we surround the non-findable system with the main system like this:
(defsystem bar :depends-on (:foo :foo-extra :foo))
will it work?
:)
It's a dirty workaround (worksurround), but the level of dirtiness is the same as the original: have a system not findable by asdf:find-system due to difference between the system name and it's .asd file name; solve it by relying on the order ASDF loads dependencies, having a findable system name first.
Ooh, that is so delightfully dirty that even if it doesn't work, I admire the twisted kind of mind that would come up with it. Nice!
I'll give it a try.
Zach
09.07.2013, 06:17, "Anton Vodonosov" avodonosov@yandex.ru:
08.07.2013, 01:34, "Zach Beane" xach@xach.com:
A few projects in quicklisp work something like this:
;;; foo.asd
(defsystem foo ...)
(defsystem foo-extra ...)
;;; bar.asd
(defsystem bar :depends-on (:foo-extra :foo))
With asdf 2, (asdf:load-system "bar") seems to work fine, I guess because asdf 2 does the equivalent of find-system on the elements from right-to-left.
With asdf 3, it doesn't seem to work fine, I guess because asdf 3 does the equivalent of find-system on the elements from left-to-right.
Are those guesses correct?
What's the best way to have a system definition that works equally well in asdf2 and asdf3 in this kind of situation?
If we surround the non-findable system with the main system like this:
(defsystem bar :depends-on (:foo :foo-extra :foo))
will it work?
:)
It's a dirty workaround (worksurround), but the level of dirtiness is the same as the original: have a system not findable by asdf:find-system due to difference between the system name and it's .asd file name; solve it by relying on the order ASDF loads dependencies, having a findable system name first.
But the best solution is separation of these ASDF systems into different .asd files. Right now everyone seems to prefer dealing with systems written that way - separate .asd files per ASDF system.
On Tue, 2013-07-09 at 23:12 +0400, Anton Vodonosov wrote: [...]
But the best solution is separation of these ASDF systems into different .asd files. Right now everyone seems to prefer dealing with systems written that way - separate .asd files per ASDF system.
Not me: https://github.com/sionescu/iolib/blob/master/iolib.asd
-
Anton Vodonosov -
Attila Lendvai -
Faré -
Robert Goldman -
Robert P. Goldman
-
Stelian Ionescu -
Zach Beane