[asdf-devel] Unexpected behavior when loading different versions of a system
Say I download version 1 of the FOO library and put it in /tmp/foo-1/, so it looks like this: /tmp/foo-1/foo.asd /tmp/foo-1/a.lisp foo.asd has this: (asdf:defsystem #:foo :serial t :components ((:file "a"))) Then I add the path to the central registry and load it: (push #p"/tmp/foo-1/" asdf:*central-registry*) (asdf:load-system "foo") Everything loads fine. Oops, I need to upgrade, I better get version 2 and put it in /tmp/foo-2/, which looks like this: /tmp/foo-2/foo.asd /tmp/foo-2/a.lisp /tmp/foo-2/b.lisp foo.asd looks like this: (asdf:defsystem #:foo :serial t :components ((:file "a") (:file "b"))) Then I add the path to central registry: (push #p"/tmp/foo-2/" asdf:*central-registry*) It seems to find the new system: (asdf:system-source-directory "foo") => #p"/tmp/foo-2/" However, when I try this: (asdf:load-system "foo") I end up with this error, which I don't really understand: failed to find the TRUENAME of /tmp/foo-1/b.lisp: No such file or directory Why does that happen? What's the best way to work around it? I'm using ASDF 2.017 and a very recent SBCL from git. You can reproduce my scenario by unpacking http://xach.com/tmp/foofail.tgz and loading "foofail.lisp". Thanks, Zach
On 9/1/11 Sep 1 -6:58 AM, Zach Beane wrote:
Say I download version 1 of the FOO library and put it in /tmp/foo-1/, so it looks like this:
/tmp/foo-1/foo.asd /tmp/foo-1/a.lisp
foo.asd has this:
(asdf:defsystem #:foo :serial t :components ((:file "a")))
Then I add the path to the central registry and load it:
(push #p"/tmp/foo-1/" asdf:*central-registry*) (asdf:load-system "foo")
Everything loads fine.
Oops, I need to upgrade, I better get version 2 and put it in /tmp/foo-2/, which looks like this:
/tmp/foo-2/foo.asd /tmp/foo-2/a.lisp /tmp/foo-2/b.lisp
foo.asd looks like this:
(asdf:defsystem #:foo :serial t :components ((:file "a") (:file "b")))
Then I add the path to central registry:
(push #p"/tmp/foo-2/" asdf:*central-registry*)
It seems to find the new system:
(asdf:system-source-directory "foo") => #p"/tmp/foo-2/"
However, when I try this:
(asdf:load-system "foo")
I end up with this error, which I don't really understand:
failed to find the TRUENAME of /tmp/foo-1/b.lisp: No such file or directory
Why does that happen? What's the best way to work around it?
I'm using ASDF 2.017 and a very recent SBCL from git. You can reproduce my scenario by unpacking http://xach.com/tmp/foofail.tgz and loading "foofail.lisp".
FWIW, I get the same error on ACL, so I believe this is an ASDF issue, and not SBCL-specific.... I haven't solved the problem, but when I look into the "foo" system object I see an interesting mismatch between the relative and absolute pathname slots: A NEW ASDF-UTILITIES:SYSTEM @ #x100436e292 = #<ASDF-UTILITIES:SYSTEM "foo"> 0 Class --------> #<STANDARD-CLASS ASDF-UTILITIES:SYSTEM> 1 NAME ---------> A simple-string (3) "foo" 2 VERSION ------> symbol :--UNBOUND-- 3 LOAD-DEPENDENCIES -> symbol NIL 4 IN-ORDER-TO --> symbol NIL 5 DO-FIRST -----> ((ASDF-UTILITIES:COMPILE-OP (ASDF-UTILITIES:LOAD-OP))), a proper list with 1 element 6 INLINE-METHODS -> symbol NIL 7 PARENT -------> symbol NIL 8 RELATIVE-PATHNAME -> PATHNAME struct = #P"/Users/rpg/Downloads/foofail/foo-2/" 9 ABSOLUTE-PATHNAME -> PATHNAME struct = #P"/Users/rpg/Downloads/foofail/foo-1/" .... This is probably the right place to look for the bug. I'm afraid that I will be in meetings all day, and am unlikely to get around to this in the near future... Zach, would you mind blowing this into the launchpad? thanks, r
OK, I have investigated, and I see what's wrong. The ABSOLUTE-PATHNAME slot of COMPONENT acts like a kind of cache (it's filled by resolving the RELATIVE-PATHNAME), but it is not being cleared when the system is reloaded. I have a proposed patch that involves a REINITIALIZE-INSTANCE :AFTER method on COMPONENT, and it seems to fix the problem. I am just finishing up a test case (as usual, the bugs in the test script are harder to find than the bugs in ASDF!), and expect to push a fix tonight or tomorrow morning. cheers, r
On 9/1/11 Sep 1 -10:07 PM, Robert Goldman wrote:
OK, I have investigated, and I see what's wrong. The ABSOLUTE-PATHNAME slot of COMPONENT acts like a kind of cache (it's filled by resolving the RELATIVE-PATHNAME), but it is not being cleared when the system is reloaded.
I have a proposed patch that involves a REINITIALIZE-INSTANCE :AFTER method on COMPONENT, and it seems to fix the problem. I am just finishing up a test case (as usual, the bugs in the test script are harder to find than the bugs in ASDF!), and expect to push a fix tonight or tomorrow morning.
Hm. I spoke too soon --- that modification only kicks the problem down the road. Here is what it fixes: it fixes the fact that the ABSOLUTE-PATHNAME of the FOO system is not updated when you load a new system definition in a new file. UNFORTUNATELY, there are other problems with reusing the COMPONENT objects when you reload the definition of a system with the same name. Here's the new problem: when we reload Xach's foo system, we get a new definition with files "a" and "b." Unfortunately, when ASDF reads this new definition, it first says "Oh, file a in system foo --- I have seen that before, and it's already loaded." So when I tell ASDF to load foo (again), it says "a is already loaded, I need only load b." Unfortunately, that is not true --- the OLD "a" component is already loaded, but the NEW one has not been loaded. I am not entirely sure why we reuse the same COMPONENT objects when we reload the system definition. Arguably, the conservative thing to do would be to say "a new system definition has been loaded, all bets are off about whether the load-op (or any other operation) of existing components has been satisfactorily completed," and simply blow away the old child components of the parent. Yes, this would lead to some unnecessary recompiles in the more common cases of interactive development, where the system definition has only changed in a minor way, but it would fix these really bad cases where the system definition has really invalidated all dependency information. [Indeed, Faré and I had in the past discussed the possibility that when a new system definition for an existing system is loaded, we should simply regard the entire system as "dirty" and always rebuild everything in it.] This is a REALLY big change to the behavior of ASDF, so I would like to hear some comments before proposing a patch. cheers, r
On 9/1/11 Sep 1 -10:40 PM, Robert Goldman wrote: Here's the new problem: when we reload Xach's foo system, we get a new definition with files "a" and "b." Unfortunately, when ASDF reads this new definition, it first says "Oh, file a in system foo --- I have seen that before, and it's already loaded." So when I tell ASDF to load foo (again), it says "a is already loaded, I need only load b." Unfortunately, that is not true --- the OLD "a" component is already loaded, but the NEW one has not been loaded. ... [Indeed, Faré and I had in the past discussed the possibility that when a new system definition for an existing system is loaded, we should simply regard the entire system as "dirty" and always rebuild everything in it.]
This is a REALLY big change to the behavior of ASDF, so I would like to hear some comments before proposing a patch.
Makefiles handle this by checking timestamps between source and object files. Other build systems do this by checking hashes. However, both approaches rely on dependency tracking and assume that each output is compiled in isolation and everything is relinked at the end. In a serial CL process (all files loaded/compiled by the same image), the presence or absence of a file may greatly affect all files visited after it. Thus given this interactive image model, I don't think any simple option will work in all cases. The current behavior is reasonable (modulo where the files are found). Thus, I wouldn't force a rebuild when the system definition changes, but a rebuild might be deserved when the source location changes. Isn't there an option to force rebuilding of all files in a system? How does that interact with Zach's case? - Daniel
On 9/2/11 Sep 2 -12:25 PM, dherring@tentpost.com wrote:
On 9/1/11 Sep 1 -10:40 PM, Robert Goldman wrote: Here's the new problem: when we reload Xach's foo system, we get a new definition with files "a" and "b." Unfortunately, when ASDF reads this new definition, it first says "Oh, file a in system foo --- I have seen that before, and it's already loaded." So when I tell ASDF to load foo (again), it says "a is already loaded, I need only load b." Unfortunately, that is not true --- the OLD "a" component is already loaded, but the NEW one has not been loaded. ... [Indeed, Faré and I had in the past discussed the possibility that when a new system definition for an existing system is loaded, we should simply regard the entire system as "dirty" and always rebuild everything in it.]
This is a REALLY big change to the behavior of ASDF, so I would like to hear some comments before proposing a patch.
Makefiles handle this by checking timestamps between source and object files. Other build systems do this by checking hashes. However, both approaches rely on dependency tracking and assume that each output is compiled in isolation and everything is relinked at the end.
In a serial CL process (all files loaded/compiled by the same image), the presence or absence of a file may greatly affect all files visited after it. Thus given this interactive image model, I don't think any simple option will work in all cases. The current behavior is reasonable (modulo where the files are found).
I fundamentally disagree about this. For two reasons: 1. Correctness over convenience: It is more important to restore correct functioning after changes than to avoid excessive recompilation. CPU cycles are cheaper than programmer cycles. Introducing a cryptic, transient, buggy behavior and baffling the programmer is worse than recompiling a few extra files. 2. The model is fundamentally busted. If you reload a system file, the data structures in ASDF are in a deeply wrong state. The SYSTEM objects (and other components) will contain an ill-characterized moosh of data, some from one version of the system definition and some from the later version. Xach has found one bad behavior that comes from this. But there could be a bottomless well of other bad behaviors from this, since ASDF's internal state is corrupted.
Thus, I wouldn't force a rebuild when the system definition changes, but a rebuild might be deserved when the source location changes.
The bad behaviors are not limited to bad behaviors from location changes. If you change the membership of the system, but the system is in the same location, that can introduce corruption, as well. I don't see a way to keep correctness here and maintain the current behavior. This is not a step I take lightly. Best, Robert
For what it's worth, I agree with Robert Goldman that well-defined behavior is preferrable to a collection of patches to address quirks. We can't completely tighten the semantics of ASDF, because some of its fundamental assumptions regarding incremental compilation depend on programmers following unenforceable ill-specified restrictions (e.g. that any visible side-effect from compiling a file is included in loading the file and not negatively interacting with compilation of other files that may happen before said loading, that dependencies are properly declared, etc.). Still I think that modification of the .asd file, as detected by either timestamp or pathname change, should trigger the invalidation of the system. If you want to avoid unnecessary recompilation of objects, you should be using a system that more closely tracks dependencies between objects, such as XCVB. BTW, it is possible to load XCVB systems from ASDF and vice-versa. I really should re-package the whole thing. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org On Fri, Sep 2, 2011 at 14:03, Robert Goldman <rpgoldman@sift.info> wrote:
On 9/2/11 Sep 2 -12:25 PM, dherring@tentpost.com wrote:
On 9/1/11 Sep 1 -10:40 PM, Robert Goldman wrote: Here's the new problem: when we reload Xach's foo system, we get a new definition with files "a" and "b." Unfortunately, when ASDF reads this new definition, it first says "Oh, file a in system foo --- I have seen that before, and it's already loaded." So when I tell ASDF to load foo (again), it says "a is already loaded, I need only load b." Unfortunately, that is not true --- the OLD "a" component is already loaded, but the NEW one has not been loaded. ... [Indeed, Faré and I had in the past discussed the possibility that when a new system definition for an existing system is loaded, we should simply regard the entire system as "dirty" and always rebuild everything in it.]
This is a REALLY big change to the behavior of ASDF, so I would like to hear some comments before proposing a patch.
Makefiles handle this by checking timestamps between source and object files. Other build systems do this by checking hashes. However, both approaches rely on dependency tracking and assume that each output is compiled in isolation and everything is relinked at the end.
In a serial CL process (all files loaded/compiled by the same image), the presence or absence of a file may greatly affect all files visited after it. Thus given this interactive image model, I don't think any simple option will work in all cases. The current behavior is reasonable (modulo where the files are found).
I fundamentally disagree about this. For two reasons:
1. Correctness over convenience: It is more important to restore correct functioning after changes than to avoid excessive recompilation. CPU cycles are cheaper than programmer cycles. Introducing a cryptic, transient, buggy behavior and baffling the programmer is worse than recompiling a few extra files.
2. The model is fundamentally busted. If you reload a system file, the data structures in ASDF are in a deeply wrong state. The SYSTEM objects (and other components) will contain an ill-characterized moosh of data, some from one version of the system definition and some from the later version. Xach has found one bad behavior that comes from this. But there could be a bottomless well of other bad behaviors from this, since ASDF's internal state is corrupted.
Thus, I wouldn't force a rebuild when the system definition changes, but a rebuild might be deserved when the source location changes.
The bad behaviors are not limited to bad behaviors from location changes. If you change the membership of the system, but the system is in the same location, that can introduce corruption, as well.
I don't see a way to keep correctness here and maintain the current behavior.
This is not a step I take lightly.
Best, Robert
On Fri, 2 Sep 2011, Robert Goldman wrote:
On 9/2/11 Sep 2 -12:25 PM, dherring@tentpost.com wrote:
[Indeed, Faré and I had in the past discussed the possibility that when a new system definition for an existing system is loaded, we should simply regard the entire system as "dirty" and always rebuild everything in it.] ...
In a serial CL process (all files loaded/compiled by the same image), the presence or absence of a file may greatly affect all files visited after it. Thus given this interactive image model, I don't think any simple option will work in all cases. The current behavior is reasonable (modulo where the files are found).
I fundamentally disagree about this. For two reasons:
1. Correctness over convenience: It is more important to restore ... 2. The model is fundamentally busted. If you reload a system file, the data structures in ASDF are in a deeply wrong state. The SYSTEM objects ...
Thus, I wouldn't force a rebuild when the system definition changes, but a rebuild might be deserved when the source location changes.
The bad behaviors are not limited to bad behaviors from location changes. If you change the membership of the system, but the system is in the same location, that can introduce corruption, as well.
I don't see a way to keep correctness here and maintain the current behavior.
I'll try rephrasing my point. Under the interactive image model, recompiling the whole system provides only slightly stronger guarantees than not. State changes since the initial compilation may adversely affect the recompile, and downstream systems also face unknown issues. Thus I wouldn't force a recompile unless there was reason to believe the source files had changed (ASDF's default behavior). For example, consider the difficulty ASDF has in bootstrapping new versions of itself. Now, if your argument is that ASDF's internal state is completely unreliable after a system change, and the cleanest way to fix it is the nuclear option, then that is a strong argument for recompiling everything in the project. I agree with Fare that XCVB addresses some of these issues. Don't understand it enough to say by how much (still todo... sigh). Later, Daniel P.S. I tend to agree with the "recompile cycles are cheap" argument, but it would be nice to have an option for people to skip a full recompile caused by minor system definition changes. It would be very useful when doing minor tweaks on large systems.
participants (5)
-
Daniel Herring -
dherring＠tentpost.com
-
Faré -
Robert Goldman -
Zach Beane