I've saved an executable SBCL image that includes asdf and calls load-system. If the system it's loading depends-on sbcl contribs, e.g. sb-bsd-sockets, it is recompiling the contrib sources.
What causes contribs to get recompiled? Is there an easy way to inhibit it?
Thanks, Zach
On Thu, Oct 31, 2013 at 2:43 PM, Zach Beane xach@xach.com wrote:
I've saved an executable SBCL image that includes asdf and calls load-system. If the system it's loading depends-on sbcl contribs, e.g. sb-bsd-sockets, it is recompiling the contrib sources.
What causes contribs to get recompiled? Is there an easy way to inhibit it?
Which versions of SBCL and ASDF are you using? Are you using some ASDF options such as enabling deferred-warnings?
There are bugs in some old versions of SBCL and/or ASDF that can cause this behavior. At the other extreme, the latestest SBCL doesn't rely on ASDF for contribs at runtime anymore, and this bug becomes impossible.
Are you deploying on the same system?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The greatest lies always come in a thin coating of truth.
Faré fahree@gmail.com writes:
On Thu, Oct 31, 2013 at 2:43 PM, Zach Beane xach@xach.com wrote:
I've saved an executable SBCL image that includes asdf and calls load-system. If the system it's loading depends-on sbcl contribs, e.g. sb-bsd-sockets, it is recompiling the contrib sources.
What causes contribs to get recompiled? Is there an easy way to inhibit it?
Which versions of SBCL and ASDF are you using? Are you using some ASDF options such as enabling deferred-warnings?
I am using SBCL 1.1.8.30+, and ASDF 3.0.3. I am initializing output translations with something like:
(initialize-output-translations '(:output-translations (t "/path/to/fasls/") :inherit-configuration))
There are bugs in some old versions of SBCL and/or ASDF that can cause this behavior. At the other extreme, the latestest SBCL doesn't rely on ASDF for contribs at runtime anymore, and this bug becomes impossible.
Are you deploying on the same system?
Yes.
Zach
On Thu, Oct 31, 2013 at 5:32 PM, Zach Beane xach@xach.com wrote:
Faré fahree@gmail.com writes:
On Thu, Oct 31, 2013 at 2:43 PM, Zach Beane xach@xach.com wrote:
I've saved an executable SBCL image that includes asdf and calls load-system. If the system it's loading depends-on sbcl contribs, e.g. sb-bsd-sockets, it is recompiling the contrib sources.
What causes contribs to get recompiled? Is there an easy way to inhibit it?
Which versions of SBCL and ASDF are you using? Are you using some ASDF options such as enabling deferred-warnings?
I am using SBCL 1.1.8.30+, and ASDF 3.0.3. I am initializing output translations with something like:
(initialize-output-translations '(:output-translations (t "/path/to/fasls/") :inherit-configuration))
Weird. Can you do the following: (1) use (asdf:traverse 'asdf:load-op :your-system) to identify the first few things that get recompiled (2) use (asdf:input-files 'asdf:whichever-op '("sb-whatever" "component-name")) and (asdf:output-files 'asdf:whichever-op '("sb-whatever" "component-name")) to see what are the files involved, and (3) use ls -l to check the timestamps of all said files
One thing that could have gone wrong is that somehow during your SBCL installation, timestamps were not preserved, and some .lisp file got a timestamp later than the corresponding .fasl.
Another thing that could have gone wrong is that somehow your output-translations configuration fails to specially treat the sbcl contribs.
Or then again, it could be a bug in ASDF that I missed because I've been using for months the "no ASDF for contribs at runtime" patch that was just included in the latest SBCL release. But I'm pretty sure I faced that issue earlier and that it was working when I released ASDF 2.27.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The major advances in civilization are processes that all but wreck the societies in which they occur. — A.N. Whitehead
I recompiled an SBCL 1.1.9, which is close to your 1.1.8.30 and just before it upgraded to ASDF 3.0.2, and couldn't reproduce your issue; can you give me steps to reproduce, if possible minimal?
Also, is your sbcl install the fruit of a cp -r without timestamp preservation? Are there clock skew issues between your host and e.g. an NFS server?
In your TRAVERSE output, what we're looking for is the first compilation of a lisp file. It's also possible that some static-file with a high timestamp cause the next system over to think it must be recompiled.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If it's not worth doing right, it's not worth doing. — Scott McKay
On Thu, Oct 31, 2013 at 5:59 PM, Faré fahree@gmail.com wrote:
On Thu, Oct 31, 2013 at 5:32 PM, Zach Beane xach@xach.com wrote:
Faré fahree@gmail.com writes:
On Thu, Oct 31, 2013 at 2:43 PM, Zach Beane xach@xach.com wrote:
I've saved an executable SBCL image that includes asdf and calls load-system. If the system it's loading depends-on sbcl contribs, e.g. sb-bsd-sockets, it is recompiling the contrib sources.
What causes contribs to get recompiled? Is there an easy way to inhibit it?
Which versions of SBCL and ASDF are you using? Are you using some ASDF options such as enabling deferred-warnings?
I am using SBCL 1.1.8.30+, and ASDF 3.0.3. I am initializing output translations with something like:
(initialize-output-translations '(:output-translations (t "/path/to/fasls/") :inherit-configuration))
Weird. Can you do the following: (1) use (asdf:traverse 'asdf:load-op :your-system) to identify the first few things that get recompiled (2) use (asdf:input-files 'asdf:whichever-op '("sb-whatever" "component-name")) and (asdf:output-files 'asdf:whichever-op '("sb-whatever" "component-name")) to see what are the files involved, and (3) use ls -l to check the timestamps of all said files
One thing that could have gone wrong is that somehow during your SBCL installation, timestamps were not preserved, and some .lisp file got a timestamp later than the corresponding .fasl.
Another thing that could have gone wrong is that somehow your output-translations configuration fails to specially treat the sbcl contribs.
Or then again, it could be a bug in ASDF that I missed because I've been using for months the "no ASDF for contribs at runtime" patch that was just included in the latest SBCL release. But I'm pretty sure I faced that issue earlier and that it was working when I released ASDF 2.27.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The major advances in civilization are processes that all but wreck the societies in which they occur. — A.N. Whitehead
Faré fahree@gmail.com writes:
I recompiled an SBCL 1.1.9, which is close to your 1.1.8.30 and just before it upgraded to ASDF 3.0.2, and couldn't reproduce your issue; can you give me steps to reproduce, if possible minimal?
Also, is your sbcl install the fruit of a cp -r without timestamp preservation? Are there clock skew issues between your host and e.g. an NFS server?
In your TRAVERSE output, what we're looking for is the first compilation of a lisp file. It's also possible that some static-file with a high timestamp cause the next system over to think it must be recompiled.
I'll try to put together a small test case. In the meantime, here are my ultimate goals:
- Load internal (e.g. "ASDF") and special (e.g. SBCL contribs) systems normally
- Strictly and explicitly map where all other systems are found, with no fallback to the central registry, source registry, etc.
- Save all FASLs for non-internal, non-special systems in a fresh directory of my choosing
The idea is to isolate the system load and avoid introducing untracked implicit dependencies from outside sources.
Zach
On Fri, Nov 1, 2013 at 8:46 PM, Zach Beane xach@xach.com wrote:
I'll try to put together a small test case. In the meantime, here are my ultimate goals:
Load internal (e.g. "ASDF") and special (e.g. SBCL contribs) systems normally
Strictly and explicitly map where all other systems are found, with no fallback to the central registry, source registry, etc.
Save all FASLs for non-internal, non-special systems in a fresh directory of my choosing
The idea is to isolate the system load and avoid introducing untracked implicit dependencies from outside sources.
Interesting: as I'm integrating our Lisp build into the Google infrastructure, I find that I'm doing pretty much the same thing. The build is driven by a program called blaze that is exactly what you'd imagine if you were to design a massively scalable build system: http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-...
We build and store single fasls for our blaze packages with the equivalent of fasl-op or monolithic-fasl-op, after having loaded all fasl's for the dependencies, and (if using asdf at all) registering all the dependencies as preloaded. We configure empty source registries except for the tree of the package being built, so ASDF when we use it never looks for (or at) anything but the system at hand. They are empty when deploying. If we were not building at all, we could disable the central and source registries entirely by removing them from the *system-definition-search-functions*. This strategy works because blaze always knows in advance the dependencies that a lisp process will load during build. If you want to let the user dynamically choose which systems to load, you can "simply" replace the search function by one that does what you want, or register a directory in either registry that has a trivial .asd file that loads the corresponding fasl, in the style of the binary-op output.
Unhappily, my current work is all too tied to Google-specific infrastructure and unlikely to be untangled enough to be opensourced (unless someone publishes a blaze clone), but you get the idea.
Are you building binary distributions for the next version of quicklisp?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The kind of man who demands that government enforce his ideas is always the kind whose ideas are idiotic. — H. L. Mencken
Faré fahree@gmail.com writes:
Are you building binary distributions for the next version of quicklisp?
No, this is how Quicklisp dist building has always worked. The first version from 2010 used the hackiest thing that worked quickly, and now I'm trying to clean it up to make it easily parallelizable and use supported ASDF 3 features for configuration and system introspection.
Zach
Zach Beane xach@xach.com writes:
Faré fahree@gmail.com writes:
Are you building binary distributions for the next version of quicklisp?
No, this is how Quicklisp dist building has always worked. The first version from 2010 used the hackiest thing that worked quickly, and now I'm trying to clean it up to make it easily parallelizable and use supported ASDF 3 features for configuration and system introspection.
Speaking of which, I have a defsystem that looks like this:
(defsystem #:system :weakly-depends-on (#:weak-dep) :depends-on (#:dep) :defsystem-depends-on (#:defsystem-dep) ...)
If I have the system object in hand, what's the easiest, most direct way to get a list of the dependency names ("weak-dep" "dep" "defsystem-dep")?
Zach
On Sat, Nov 2, 2013 at 3:53 PM, Zach Beane xach@xach.com wrote:
Speaking of which, I have a defsystem that looks like this:
(defsystem #:system :weakly-depends-on (#:weak-dep) :depends-on (#:dep) :defsystem-depends-on (#:defsystem-dep) ...)
If I have the system object in hand, what's the easiest, most direct way to get a list of the dependency names ("weak-dep" "dep" "defsystem-dep")?
tl;dr: * (and (slot-boundp system 'asdf::defsystem-depends-on) (asdf::system-defsystem-depends-on system)) * (asdf:component-sideway-dependencies system) ;; or if compatible with asdf2, (asdf:component-sideway-dependencies system)
For defsystem-depends-on dependencies, there's asdf::system-defsystem-depends-on aka asdf/system:system-defsystem-depends-on in ASDF3. Unhappily, the slot is sometimes unbound (e.g. for preloaded systems), which is probably a bug (Robert, are you on it?), and you need to test for (slot-boundp system 'asdf::defsystem-depends-on) (exported from asdf/component — why isn't it in asdf/system? That's probably another bug. That said, it shouldn't be "just fixed" without running the upgrade test — I believe a :recycle asdf/component is then needed in asdf/system. At the very least, it should be flagged for fixing in case there's ever a backward-incompatible ASDF4).
For depends-on and weakly-depends-on, unhappily, there is only one reader for the resulting dependencies, called asdf:component-sideway-dependencies (also defined and exported in asdf/component), at least it's always defined. For compatibility with the later version of ASDF2, you can use asdf::component-load-dependencies instead. Problem is, I've never liked weakly-depends-on, and I never gave the feature much love. I even flagged it as ;; ASDF4: deprecate this feature and remove it. Of course, it's now up to the new maintainer. If the feature is here to stay, it probably needs some love. If the feature is going away, users need to be identified and provided with a way out. Personally, I much prefer explicitly loading a system foo+bar, such that foo has a hook, and foo+bar hooks the functionality of bar into foo. Failing that, I recommend using asdf-system-connections, that defines and automatically loads such a system foo+bar when both foo and bar are loaded. My point of view is that anything that makes system construction non-deterministic, conditional on the environment, sensitive to the order in which things are defined or otherwise surprising or hard to track is only going to make things harder in the end.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The last good thing written in C was Franz Schubert's Symphony number 9. — Erwin Dieterich erwin@cvt12.verfahrenstechnik.uni-stuttgart.de
My point of view is that anything that makes system construction non-deterministic, conditional on the environment, sensitive to the order in which things are defined or otherwise surprising or hard to track is only going to make things harder in the end.
my painful memories loudly add a big +1 for that.
we moved to the foo+bar explicit dependency solution after we spent way too much time debugging stupid things that e.g. depended on inode order in the filesystem...
Faré wrote:
For defsystem-depends-on dependencies, there's asdf::system-defsystem-depends-on aka asdf/system:system-defsystem-depends-on in ASDF3. Unhappily, the slot is sometimes unbound (e.g. for preloaded systems), which is probably a bug (Robert, are you on it?), and you need to test for (slot-boundp system 'asdf::defsystem-depends-on) (exported from asdf/component — why isn't it in asdf/system? That's probably another bug. That said, it shouldn't be "just fixed" without running the upgrade test — I believe a :recycle asdf/component is then needed in asdf/system. At the very least, it should be flagged for fixing in case there's ever a backward-incompatible ASDF4).
https://bugs.launchpad.net/asdf/+bug/1247662
Cheers, r
-
Attila Lendvai -
Faré -
Robert P. Goldman -
Zach Beane