Following previous discussions on this list and on #lisp, I pushed yet another change to ASDF.
The patch includes better support for system-finding functions, and caching of source-registry data so we only hit the filesystem the first time around. I hope this will make Xach happy. I'll export any symbol he wants supported before the next release, if he asks me.
The patch also handles self-upgrades more correctly. It will only work if the *old* ASDF is 2.014.8 or later, though.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] I have not yet begun to procrastinate
Faré fahree@gmail.com writes:
Following previous discussions on this list and on #lisp, I pushed yet another change to ASDF.
The patch includes better support for system-finding functions, and caching of source-registry data so we only hit the filesystem the first time around. I hope this will make Xach happy. I'll export any symbol he wants supported before the next release, if he asks me.
The patch also handles self-upgrades more correctly. It will only work if the *old* ASDF is 2.014.8 or later, though.
FYI, the recent updates removing and reinstating ASDF:SYSTEM-DEFINITION-PATHNAME broke my dist construction machinery, which relied on the original behavior of that function. Switching to ASDF:SYSTEM-SOURCE-FILE caused subtle problems for me, because functions in ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* must now return actual pathname objects, and pathname designators like strings do not suffice. (They result in a component-not-found error.)
Zach
- Zach Beane knpu-RYTmBzmmF7b@choyvp.tznar.bet [2011-04-22 09:52:43 -0400]:
... my dist construction machinery ...
very interesting...
On 22 April 2011 09:52, Zach Beane xach@xach.com wrote:
FYI, the recent updates removing and reinstating ASDF:SYSTEM-DEFINITION-PATHNAME broke my dist construction machinery, which relied on the original behavior of that function.
How did you rely on that function? It doesn't always make sense, for instance with runtime-computed of "fallback" systems that are not backed by a .asd file.
If needed, I could possibly split find-system into subfunctions, but you'd have to deal with more than just pathnames.
Switching to ASDF:SYSTEM-SOURCE-FILE caused subtle problems for me, because functions in ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* must now return actual pathname objects, and pathname designators like strings do not suffice. (They result in a component-not-found error.)
What about returning a pathname instead of a string? Alternatively I could add a clause to that cond in find-system: ((stringp found) (parse-namestring found))
In other words, I'm eager to improve ASDF to make it more suitable to your purposes, but I require more precise feedback as to what to do.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Statism is the secular version of salvation through faith: it doesn't matter what bureaucrats do, only that they do it with good intentions.
Faré fahree@gmail.com writes:
On 22 April 2011 09:52, Zach Beane xach@xach.com wrote:
FYI, the recent updates removing and reinstating ASDF:SYSTEM-DEFINITION-PATHNAME broke my dist construction machinery, which relied on the original behavior of that function.
How did you rely on that function?
I wanted to find the system file that would be loaded by a (find-system "foo") without loading the file first.
It doesn't always make sense, for instance with runtime-computed of "fallback" systems that are not backed by a .asd file.
Is there more than one such system, i.e. is this just a single-use hack for loading ASDF as a system?
If needed, I could possibly split find-system into subfunctions, but you'd have to deal with more than just pathnames.
I wound up changing my approach entirely. My goal was to answer the question "What systems are defined by foo.asd?" Instead of determining the filename in advance, I got it after the fact and used map-systems to find related systems defined in the same file.
Switching to ASDF:SYSTEM-SOURCE-FILE caused subtle problems for me, because functions in ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* must now return actual pathname objects, and pathname designators like strings do not suffice. (They result in a component-not-found error.)
What about returning a pathname instead of a string?
Yes, that's what I did.
In other words, I'm eager to improve ASDF to make it more suitable to your purposes, but I require more precise feedback as to what to do.
It would be nice the behavior of exported functions didn't change in subtle ways between minor releases.
Zach
I wanted to find the system file that would be loaded by a (find-system "foo") without loading the file first.
I suppose I could reinstate some function to do that - but again, not just pathnames could be returned, but system objects, too.
It doesn't always make sense, for instance with runtime-computed of "fallback" systems that are not backed by a .asd file.
Is there more than one such system, i.e. is this just a single-use hack for loading ASDF as a system?
I admit that currently, I know only ASDF and cl-launch that use that functionality (that I know of), as fallbacks when source is not available.
I wound up changing my approach entirely. My goal was to answer the question "What systems are defined by foo.asd?" Instead of determining the filename in advance, I got it after the fact and used map-systems to find related systems defined in the same file.
It's probably a safer bet.
In other words, I'm eager to improve ASDF to make it more suitable to your purposes, but I require more precise feedback as to what to do.
It would be nice the behavior of exported functions didn't change in subtle ways between minor releases.
Well, I (obviously falsely) believed that this exported function was never really used, except by one client, as a bad way to achieve the functionality of system-source-file, from times before system-source-file was available. Also, same function didn't make much sense. Apologies.
The whole "execute code at system-definition time, which is in the same image that will be built" model makes ASDF a mess of a system.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] There are a thousand hacking at the branches of evil to one who is striking at the root. — Thoreau
Faré fahree@gmail.com writes:
It would be nice the behavior of exported functions didn't change in subtle ways between minor releases.
Well, I (obviously falsely) believed that this exported function was never really used, except by one client, ...
I was thinking more of the subtle change in requirements of functions in *system-definition-search-functions* to return a pathname rather than a string. It seems somewhat arbitrary.
Zach
Faré fahree@gmail.com writes:
On 22 April 2011 09:52, Zach Beane xach@xach.com wrote:
FYI, the recent updates removing and reinstating ASDF:SYSTEM-DEFINITION-PATHNAME broke my dist construction machinery, which relied on the original behavior of that function.
How did you rely on that function? It doesn't always make sense, for instance with runtime-computed of "fallback" systems that are not backed by a .asd file.
After building the Quicklisp universe again with 2.014.8, it's clear that many projects (over 30) are affected by the ASDF:SYSTEM-DEFINITION-PATHNAME changes. It seems like anything that uses cffi-grovel is broken, e.g. osicat, gsll, shuffletron, hemlock, and many more.
Zach
After building the Quicklisp universe again with 2.014.8, it's clear that many projects (over 30) are affected by the ASDF:SYSTEM-DEFINITION-PATHNAME changes. It seems like anything that uses cffi-grovel is broken, e.g. osicat, gsll, shuffletron, hemlock, and many more.
Oops. I'll remove that cerror for now, and just have a docstring.
Thanks for the testing...
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Absent, adj.: Exposed to the attacks of friends and acquaintances; defamed; slandered.
On 22 April 2011 13:59, Faré fahree@gmail.com wrote:
After building the Quicklisp universe again with 2.014.8, it's clear that many projects (over 30) are affected by the ASDF:SYSTEM-DEFINITION-PATHNAME changes. It seems like anything that uses cffi-grovel is broken, e.g. osicat, gsll, shuffletron, hemlock, and many more.
Oops. I'll remove that cerror for now, and just have a docstring.
Thanks for the testing...
Please retry with 2.014.9.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Every technique is first developed, then used, important, obsolete, normalized, and finally understood.
Faré fahree@gmail.com writes:
On 22 April 2011 13:59, Faré fahree@gmail.com wrote:
After building the Quicklisp universe again with 2.014.8, it's clear that many projects (over 30) are affected by the ASDF:SYSTEM-DEFINITION-PATHNAME changes. It seems like anything that uses cffi-grovel is broken, e.g. osicat, gsll, shuffletron, hemlock, and many more.
Oops. I'll remove that cerror for now, and just have a docstring.
Thanks for the testing...
Please retry with 2.014.9.
I've retried with 2.014.9, and the cffi-grovel-related problems are gone, but there's an interesting new problem.
2.014.9 includes this line:
(declaim (optimize (speed 2) (debug 3) (safety 3))) ; XXXXX debug only
At (safety 3), SBCL is more aggressive about type-checking CLOS slots. As a result of the new declaration of the VERSION slot of a component to be of type STRING, several projects have systems that fail to load due to type errors:
- cl-jpeg uses ":version 1.025"
- nekthuth uses ":version (format nil "~A" +nekthuth-version+)", apparently expecting it to be evaluated
- js-parser and jwacs use ":version *version*"
- meta-sexp uses ":version +meta-sexp-version+"
- rutils uses ":version '(0 3 1)"
Was the :version option evaluated in the past?
Zach
On 4/22/11 Apr 22 -3:05 PM, Zach Beane wrote:
Faré fahree@gmail.com writes:
On 22 April 2011 13:59, Faré fahree@gmail.com wrote:
After building the Quicklisp universe again with 2.014.8, it's clear that many projects (over 30) are affected by the ASDF:SYSTEM-DEFINITION-PATHNAME changes. It seems like anything that uses cffi-grovel is broken, e.g. osicat, gsll, shuffletron, hemlock, and many more.
Oops. I'll remove that cerror for now, and just have a docstring.
Thanks for the testing...
Please retry with 2.014.9.
I've retried with 2.014.9, and the cffi-grovel-related problems are gone, but there's an interesting new problem.
2.014.9 includes this line:
(declaim (optimize (speed 2) (debug 3) (safety 3))) ; XXXXX debug only
At (safety 3), SBCL is more aggressive about type-checking CLOS slots. As a result of the new declaration of the VERSION slot of a component to be of type STRING, several projects have systems that fail to load due to type errors:
cl-jpeg uses ":version 1.025"
nekthuth uses ":version (format nil "~A" +nekthuth-version+)", apparently expecting it to be evaluated
js-parser and jwacs use ":version *version*"
meta-sexp uses ":version +meta-sexp-version+"
rutils uses ":version '(0 3 1)"
Was the :version option evaluated in the past?
I believe not. I am looking at the definition of the now pretty old SHOP2 system and I see:
:version #.cl-user::+shop-version+
I suppose that would have worked even if the version /was/ evaluated, though, since strings would evaluate to themselves....
I find it relatively easy to believe that no one was looking seriously at these versions, since many of the versions in the wild, even the ones that are correctly string-valued, are semantically defective.
r
I've retried with 2.014.9, and the cffi-grovel-related problems are gone, but there's an interesting new problem.
2.014.9 includes this line:
(declaim (optimize (speed 2) (debug 3) (safety 3))) ; XXXXX debug only
Yes, my bad. I'll be gone in an upcoming 2.014.10 :-/
At (safety 3), SBCL is more aggressive about type-checking CLOS slots. As a result of the new declaration of the VERSION slot of a component to be of type STRING, several projects have systems that fail to load due to type errors:
- cl-jpeg uses ":version 1.025"
- nekthuth uses ":version (format nil "~A" +nekthuth-version+)", apparently expecting it to be evaluated
- js-parser and jwacs use ":version *version*"
- meta-sexp uses ":version +meta-sexp-version+"
- rutils uses ":version '(0 3 1)"
Was the :version option evaluated in the past?
The option is only evaluated when someone tries to enforce version with a dependency like (:version "cl-jpeg" "3.4.5"). Sigh.
That's an interesting problem. What's the correct solution?
Accept bad values and throw a style-warning? a warning? a cerror?
I've just committed 2.014.10 where I issue a warning when I detect it.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Arbitrary limits to programs are evil, particularly when they go either enforced or unenforced.
Faré fahree@gmail.com writes:
Was the :version option evaluated in the past?
The option is only evaluated when someone tries to enforce version with a dependency like (:version "cl-jpeg" "3.4.5"). Sigh.
That's an interesting problem. What's the correct solution?
Accept bad values and throw a style-warning? a warning? a cerror?
Is it feasible to evaluate the option, and give a style-warning like "Ignoring non-string version ~S" unless it does evaluate to a string?
Zach
On Fri, 22 Apr 2011, Zach Beane wrote:
As a result of the new declaration of the VERSION slot of a component to be of type STRING, several projects have systems that fail to load due to type errors:
cl-jpeg uses ":version 1.025"
nekthuth uses ":version (format nil "~A" +nekthuth-version+)", apparently expecting it to be evaluated
js-parser and jwacs use ":version *version*"
meta-sexp uses ":version +meta-sexp-version+"
rutils uses ":version '(0 3 1)"
This is one of the spots where I think it is appropriate for a distribution like Quicklisp to patch the sources locally, until the proper changes are accepted upstream...
Such portability and upgrade details need not be the concern of every library author for life.
- Daniel
Daniel Herring dherring@tentpost.com writes:
This is one of the spots where I think it is appropriate for a distribution like Quicklisp to patch the sources locally, until the proper changes are accepted upstream...
Such portability and upgrade details need not be the concern of every library author for life.
The problem doesn't actually manifest itself unless the user has e.g. (declaim (optimize (safety 3))) in their init file in SBCL, and even then, it's because of an incompatible change in an ASDF minor version that, I think, can be fixed in ASDF.
Zach
On Sat, 23 Apr 2011, Zach Beane wrote:
Daniel Herring dherring@tentpost.com writes:
This is one of the spots where I think it is appropriate for a distribution like Quicklisp to patch the sources locally, until the proper changes are accepted upstream...
Such portability and upgrade details need not be the concern of every library author for life.
The problem doesn't actually manifest itself unless the user has e.g. (declaim (optimize (safety 3))) in their init file in SBCL, and even then, it's because of an incompatible change in an ASDF minor version that, I think, can be fixed in ASDF.
Yes and no. As it stands now, version information in ASDF files is virtually worthless. If we nudge people towards using a single convention...
but there are other issues as well and it might not be worth the effort right now.
- Daniel
With all due respect, it's not a matter of pushing people to follow a convention. The code and documentation specify a data type and the interpretation of its values.
It's just that people don't follow this specification. I don't want to break code unnecessarily, but raising a CONTINUABLE type error on non-string values and even bad string values would be reasonable.
Maybe if we had that we wouldn't have so many uselessly versioned systems in the wild.
"Robert P. Goldman" rpgoldman@sift.info writes:
With all due respect, it's not a matter of pushing people to follow a convention. The code and documentation specify a data type and the interpretation of its values.
The ASDF code didn't actually mandate a string type for the slot until a couple minor versions ago, and that's what's causing the compile-time failure on SBCL (and only then at safety 3). Code that actually cared about the version would have failed at runtime. Clearly, at the moment, nobody cares - it would be nice to defer the problem until someone starts caring.
Maybe if we had that we wouldn't have so many uselessly versioned systems in the wild.
I don't see that mattering an awful lot to anyone any time soon.
Zach
On 4/23/11 Apr 23 -5:03 PM, Zach Beane wrote:
"Robert P. Goldman" rpgoldman@sift.info writes:
With all due respect, it's not a matter of pushing people to follow a convention. The code and documentation specify a data type and the interpretation of its values.
The ASDF code didn't actually mandate a string type for the slot until a couple minor versions ago, and that's what's causing the compile-time failure on SBCL (and only then at safety 3). Code that actually cared about the version would have failed at runtime. Clearly, at the moment, nobody cares - it would be nice to defer the problem until someone starts caring.
I don't actually agree with this claim. The correct functioning of version-satisfies has always required a string type (and a very specifically constructed string type) for this slot, as far as I can tell, certainly as far back as 2001.
We never /checked/ before, but that doesn't mean non-specially constructed string values were /ever/ valid. That would be like saying that we have never worried about using English versus metric units in our navigation software to date, so it's ok....
That said, I agree that it's unrealistic to refuse to load systems that have non-conforming version values, simply because this is so prevalent in the wild. But I don't agree that this is an OK state of affairs. Killing the version slot would even be better than offering it as supporting a version satisfaction check that it cannot support.
I think the best compromise between realism and ideals will be to evaluate any version specifier that's present (not evaluating it seems to me to be unproductive purism --- forcing #. makes ASDF undesirably less declarative --- and evaluating strings won't break anything not already broken), catch errors that arise when evaluating, then check the results for being well-formed period-separated strings. Bad values won't do anyone any good, so just ignore them and raise a warning.
The warning will encourage developers to provide well-formed version specifiers, without breaking the load of systems with bad ones.
We may need to handle the warnings (by printing them and continuing), or make them of some subtype of STYLE-WARNING if that is necessary to keep some implementations from aborting ASDF load upon warning. If a normal warning would break loading, I'd suggest we use a new subtype of STYLE-WARNING.
Ideally, this would evolve from a warning towards a TYPE-ERROR, but there's no hurry.
In sum, I believe the principle should be to try to help, or even enforce, having good values in one's data structures, but that the principle must come second to not breaking the community library infrastructure. I also believe that the above policy would be a reasonable compromise.
Best, R
On 23 April 2011 18:03, Zach Beane xach@xach.com wrote:
The ASDF code didn't actually mandate a string type for the slot until a couple minor versions ago, and that's what's causing the compile-time failure on SBCL (and only then at safety 3).
Following your failure report, I promptly commented out the :type string.
It is an error according to age-old spec that anything be used as version but a string that satisfies parse-version (a function I just added, but that only robustly implements the spec from a28bd5a2 and earlier in 2002).
Now, considering how many systems in the wild are in such error, they should be, if not grandfathered, at least given a reasonable chance to get fixed. And so I removed the enforcement and replaced it with a warning while parsing the defsystem form.
Hopefully, by next year (haha), we can either replace that with stricter enforcement (after getting those packages fixed) or agree on dropping the requirement entirely.
Code that actually cared about the version would have failed at runtime. Clearly, at the moment, nobody cares - it would be nice to defer the problem until someone starts caring.
Indeed. Sorry for breaking stuff, and thanks again for testing!
Please test again with 2.014.11.
PS: +1 on comments by rpgoldman
PPS: I just made asdf more verbose in verbose mode. However, verbose mode is the default, so this may or may not be annoying. Please tell me.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Fraud is the homage that force pays to reason. — Charles Curtis
Faré fahree@gmail.com writes:
PPS: I just made asdf more verbose in verbose mode. However, verbose mode is the default, so this may or may not be annoying. Please tell me.
Now that I've seen it, I do find it both annoying and confusing.
Before:
* (require 'sb-rotate-byte) ("SB-ROTATE-BYTE" "ASDF")
After:
* (require 'sb-rotate-byte)
; Loading #<CL-SOURCE-FILE "sb-rotate-byte" "package"> ; Loading #<CL-SOURCE-FILE "sb-rotate-byte" "compiler"> ; Loading #<CL-SOURCE-FILE "sb-rotate-byte" "vm" "x86-64-vm"> ; Loading #<MODULE "sb-rotate-byte" "vm"> ; Loading #<CL-SOURCE-FILE "sb-rotate-byte" "rotate-byte"> ; Loading #<SYSTEM "sb-rotate-byte"> ("SB-ROTATE-BYTE")
More than one project has something like this in the .asd:
#+sbcl (require 'some-sbcl-thing)
There isn't a good way to intercede to make these non-verbose, is there?
I would prefer to see nothing at all in that circumstance, except the return value.
Also, the display of the object as a "CL-SOURCE-FILE" is confusing. It's actually loading the FASL. (I know why it prints the way it does, but it's still confusing.)
Zach
-
Daniel Herring -
Faré -
Robert Goldman -
Robert P. Goldman
-
Sam Steingold -
Zach Beane