"Solved" (was: Re: [asdf-devel] (asdf:system-source-directory :uiop) not working after upgrade to 3.0.2.25)
It's not crucial that I release with this latest 3.0.2.32, but I just
bring
up this issue in case it is actually a lurking asdf bug which needs to
be
shaken out before 3.0.3 release...
If something is broken, we definitely want to solve before release. But I fear the problem is with your configuration somehow.
I "solved" the issue by the following bootstrap procedure:
1. compile/load adsf3 2. (remhash "uiop" asdf::*preloaded-systems* 3. load quicklisp's setup.lisp
Without step (2), the preloaded system of uiop occurs first in *system-definition-search-functions*, so the separable project for uiop (in my case, in local-projects), never ends up being found, and therefore ends up missing from the monofasl (I do suspect that the fact that it silently fails to be included in the monofasl can be considered a bug -- this probably should throw an error when trying to build the monofasl and realizing that the uiop.fasl is not showing up to the party, shouldn't it?)
So is this a use case which should be more directly supported? Does this call for an unregister-preloaded-system or some such thing?
Regards,
Dave
On Tue, Oct 22, 2013 at 10:57 AM, Faré fahree@gmail.com wrote:
So I went ahead and tried
(load (compile-file "quicklisp/asdf.lisp"))
before loading quicklisp/setup.lisp.
If you already changed the quicklisp copy and *required-asdf-version* to something later than your implementation-provided ones, as I believe you are, this step is redundant.
And yes, I copied version.lisp.expr into quicklisp/local-projects/uiop/.
Did you use cp -aL with -L to dereference the symlink? It worked for me.
To test that it wasn't ignoring the uiop in local-projects because it is
the
same version as what is already loaded, I even temporarily bumped the version number in that version.lisp-expr to 3.0.2.33.
No, I don't think it's magically checking the version against downgrade, as it is doing for asdf itself. Maybe it should?
CL_SOURCE_REGISTRY=~/cl/poiu rlwrap sbcl --eval "'(#.(map () 'require '(:asdf :poiu)) #.(in-package :asdf) #.(load "~/quicklisp/setup"))"
- (system-source-directory :uiop)
#P"/home/tunes/quicklisp/local-projects/uiop/"
I didn't even have to (ql:register-local-projects) in that session — or maybe its effects were persistent across section? Looking at the source, it builds an index, so you need to run it once after you installed the copy of uiop, and thereafter its effects should persist.
Still no dice -- I still get
(asdf:system-source-directory :uiop) --> nil
after loading the quicklisp/setup.lisp and (ql:register-local-projects)
Quicklisp works in general after the upgrade. It just can't find the
source
directory (or any source directory) for :uiop.
It's not crucial that I release with this latest 3.0.2.32, but I just
bring
up this issue in case it is actually a lurking asdf bug which needs to be shaken out before 3.0.3 release...
If something is broken, we definitely want to solve before release. But I fear the problem is with your configuration somehow.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org What one person receives without working for, another person must work for without receiving. — Adrian Rogers
On Tue, Oct 22, 2013 at 2:08 PM, Dave Cooper david.cooper@genworks.com wrote:
I "solved" the issue by the following bootstrap procedure:
- compile/load adsf3
- (remhash "uiop" asdf::*preloaded-systems*
- load quicklisp's setup.lisp
Without step (2), the preloaded system of uiop occurs first in *system-definition-search-functions*, so the separable project for uiop (in my case, in local-projects), never ends up being found, and therefore ends up missing from the monofasl (I do suspect that the fact that it silently fails to be included in the monofasl can be considered a bug -- this probably should throw an error when trying to build the monofasl and realizing that the uiop.fasl is not showing up to the party, shouldn't it?)
So is this a use case which should be more directly supported? Does this call for an unregister-preloaded-system or some such thing?
OK, I understand both what is happening and why I previously failed to reproduce this failure:
1- I failed to reproduce because I was actually using 3.0.2, which registers asdf-driver but not uiop, so it wasn't interfering with your use.
2- The problem is that in *system-definition-search-functions*, quicklisp puts its things last, after sysdef-preloaded-system-search, and so the preloaded uiop preempts the one in your local-projects.
I suppose the solution is for search-for-system-definition to treat sysdef-preloaded-system-search specially and put it at the end of the search, just like it magically puts find-system-if-being-defined first. For backward compatibility, we can either remove sysdef-preloaded-system-search from the *system-definition-search-functions*, or have it become a no-op, and have its effect magically at the end of search-for-system-definition under a different name.
Even easy software is hard.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If this country is worth saving, it's worth saving at a profit. — H. L. Hunt This country can only be saved if it can be saved at a profit. — Patri Friedman
On Tue, Oct 22, 2013 at 3:27 PM, Faré fahree@gmail.com wrote:
I suppose the solution is for search-for-system-definition to treat sysdef-preloaded-system-search specially and put it at the end of the search, just like it magically puts find-system-if-being-defined first. For backward compatibility, we can either remove sysdef-preloaded-system-search from the *system-definition-search-functions*, or have it become a no-op, and have its effect magically at the end of search-for-system-definition under a different name.
But isn't uiop a bit of a special case? As I understand it, according to normal use, you would indeed want preloaded-systems to come first in the search, wouldn't you? Let's say I ship an image with cl-who built into it. Then the downstream user loads quicklisp and asdf (which I specifically do _not_ build into my image, but the current-version source code for which I provide with my distribution, with a convenient function for the user to load them).
I plan to arrange for all the built-in systems to get registered immediately after the user loads quicklisp and asdf (of course the user could circumvent that if she really wanted to, but by default I would think this should be the case).
Then let's say the user next loads some library which :depends-on (:cl-who). Since cl-who is built into the image, we really don't want quicklisp going out and fetching it again, do we? Isn't that the whole point of registering preloaded sytems?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
On Tue, Oct 22, 2013 at 3:57 PM, Dave Cooper david.cooper@genworks.com wrote:
I suppose the solution is for search-for-system-definition to treat sysdef-preloaded-system-search specially and put it at the end of the search, just like it magically puts find-system-if-being-defined first. For backward compatibility, we can either remove sysdef-preloaded-system-search from the *system-definition-search-functions*, or have it become a no-op, and have its effect magically at the end of search-for-system-definition under a different name.
But isn't uiop a bit of a special case? As I understand it, according to normal use, you would indeed want preloaded-systems to come first in the search, wouldn't you? Let's say I ship an image with cl-who built into it. Then the downstream user loads quicklisp and asdf (which I specifically do _not_ build into my image, but the current-version source code for which I provide with my distribution, with a convenient function for the user to load them).
I plan to arrange for all the built-in systems to get registered immediately after the user loads quicklisp and asdf (of course the user could circumvent that if she really wanted to, but by default I would think this should be the case).
Then let's say the user next loads some library which :depends-on (:cl-who). Since cl-who is built into the image, we really don't want quicklisp going out and fetching it again, do we? Isn't that the whole point of registering preloaded sytems?
Preloaded systems should come LAST, not first. If the user took pains to install some source code, he probably wants to use it.
If the user wants to black out some of the systems in quicklisp that would otherwise install some source, he may have to take some extra steps, but these won't be builtin to asdf. Quicklisp should offer a way to unregister some packages, and/or sysdef-preloaded-system-search or a variant thereof can be explicitly tucked in front of the quicklisp search functions in *system-definition-search-functions*. In any case, this should NOT be the default behavior of ASDF — but you're welcome to have gendl modify this default, if that makes things more stable for your users.
That the current ASDF 3.0.2 preempts quicklisp in finding preloaded systems is a bug in ASDF and/or quicklisp. (It's arguably a bug in quicklisp that it didn't work around that misdesign in ASDF, but there was still at least a distasteful design in ASDF, and it's now fixed in 3.0.2.34.) That bug wasn't too bad in 3.0.2 where the only systems registered as preloaded were asdf (that isn't in quicklisp as a distributed system anyway) and asdf-driver (which legacy name is only used by a few legacy libraries that will accept a newer backward compatible uiop) . But the bug just got worse in 3.0.2.x where uiop and asdf-defsystem are also preloaded; in the hopefully near future, asdf-package-system will be preloaded, too, as will any future asdf extensions that get moved into asdf itself (notionally, we could have included asdf-bundle, though no one depended on it by name, it seems; asdf-utils could also have been registered as preloaded, but all users have moved on).
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Guns & bullets don't kill people — blood loss and organ damage kills people.
Dear Dave,
please try again with 3.0.2.34 — and thanks a lot for finding subtle bugs like these! Your testing will have made 3.0.3 a much more robust release than would otherwise have been (as witnessed by the current patch number).
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He who refuses to do arithmetic is doomed to talk nonsense. — John McCarthy, in his webpage on Progress and Sustainability
Faré fahree@gmail.com writes:
I suppose the solution is for search-for-system-definition to treat sysdef-preloaded-system-search specially and put it at the end of the search, just like it magically puts find-system-if-being-defined first.
Does that mean this behavior can only be controlled by modifying the internals of ASDF? Once a system has been "preloaded", it must always be found in preference to any other system through any other mechanism? Is there a way to un-preload something?
Zach
On Tue, Oct 22, 2013 at 3:55 PM, Zach Beane xach@xach.com wrote:
Faré fahree@gmail.com writes:
I suppose the solution is for search-for-system-definition to treat sysdef-preloaded-system-search specially and put it at the end of the search, just like it magically puts find-system-if-being-defined first.
Does that mean this behavior can only be controlled by modifying the internals of ASDF? Once a system has been "preloaded", it must always be found in preference to any other system through any other mechanism? Is there a way to un-preload something?
No, that's the opposite: if it's preloaded, it should *never* be found in preference to anything else, it's just available in the image as a fallback in case no source code was found. And I don't want people to have to retroactively modify quicklisp so it bumps sysdef-preloaded-system-search in last position, either. Therefore, I'm having sysdef-preloaded-system-search be a magic case at the end of search-for-system-definition. Notice: not at the beginning. At the beginning, we magically have find-system-if-being-defined to avoid infinite recursion.
preloaded means: the code is already in the image, though the source might not be available.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "The direct use of force is such a poor solution to any problem, it is generally employed only by small children and large nations." – David Friedman
Faré fahree@gmail.com writes:
No, that's the opposite: if it's preloaded, it should *never* be found in preference to anything else, it's just available in the image as a fallback in case no source code was found. And I don't want people to have to retroactively modify quicklisp so it bumps sysdef-preloaded-system-search in last position, either. Therefore, I'm having sysdef-preloaded-system-search be a magic case at the end of search-for-system-definition. Notice: not at the beginning. At the beginning, we magically have find-system-if-being-defined to avoid infinite recursion.
preloaded means: the code is already in the image, though the source might not be available.
Thanks for clarifying. Can this behavior only be changed by modifying the internals of ASDF?
Zach
On Tue, Oct 22, 2013 at 4:36 PM, Zach Beane xach@xach.com wrote:
Faré fahree@gmail.com writes:
preloaded means: the code is already in the image, though the source might not be available.
Thanks for clarifying. Can this behavior only be changed by modifying the internals of ASDF?
I'm not sure what you call "this behavior", but yes, upgrade to a new ASDF (and getting it fixed if there's a bug) is probably the official solution as long as ASDF is being maintained. In case ASDF falls back into not being actively maintained anymore, you can always override functions, the way we used to do it in ASDF 1 days. Yay for dynamic binding in CL.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org On the subject of C program indentation:
"In My Egotistical Opinion, most people's C programs should be indented six feet downward and covered with dirt." — Blair P. Houghton
Faré fahree@gmail.com writes:
I'm not sure what you call "this behavior", but yes, upgrade to a new ASDF (and getting it fixed if there's a bug) is probably the official solution as long as ASDF is being maintained. In case ASDF falls back into not being actively maintained anymore, you can always override functions, the way we used to do it in ASDF 1 days. Yay for dynamic binding in CL.
Is there a supported way to enumerate preloaded systems?
Zach
On Tue, Oct 22, 2013 at 4:55 PM, Zach Beane xach@xach.com wrote:
Faré fahree@gmail.com writes:
I'm not sure what you call "this behavior", but yes, upgrade to a new ASDF (and getting it fixed if there's a bug) is probably the official solution as long as ASDF is being maintained. In case ASDF falls back into not being actively maintained anymore, you can always override functions, the way we used to do it in ASDF 1 days. Yay for dynamic binding in CL.
Is there a supported way to enumerate preloaded systems?
Preloaded system names are the keys to the hash-table asdf::*preloaded-systems*.
If you feel like there's a need to define a formal interface to them, we could export a function to list those names and/or those names and keys. That said, this part of ASDF is probably not going to move any time soon.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you're a quiet, law-abiding citizen most of the time but occasionally cut someone up and bury them in your backyard, you're a bad guy. — Paul Graham, "The Power of the Marginal"
-
Dave Cooper -
Faré -
Zach Beane