Per my recent post to launchpad, I don't think dependency updating is the fatal flaw in D-D-O. D-D-O's bad interaction with the package system is worse.
Specifically, D-D-O cannot be used to introduce ASDF extensions unless those ASDF extensions live in the ASDF package. Otherwise, by the time a DEFSYSTEM form is read, the enclosing package's imports are already established. This means that ASDF has no protection against namespace pollution, since extenders are practically forced to add symbols to ASDF.
For example, we have a FIVEAM-ASDF extension that introduces a new defsystem class that has features for using 5AM as an implementation for the TEST-OP. But we cannot put a DEFSYSTEM-DEPENDS-ON for FIVEAM-ASDF in a system and reference FIVEAM-ASDF:FIVEAM-TESTER-SYSTEM as the system class, because the FIVEAM-ASDF package does not (in general) exist when the enclosing DEFSYSTEM is read.
I can think of three ways of handling this:
1. Deprecate and remove D-D-O. This seems undesirable, since D-D-O is commonly used.
2. Replace the use of the lisp reader for processing DEFSYSTEM forms, and introduce some kind of parser. This seems like an enormous amount of work and difficult to get right.
3. In some way extend the keyword-processing features of DEFSYSTEM to allow it to reference symbols that are not only in the current package and ASDF, but also programmer-specified packages. Unfortunately, processing of keyword symbols in DEFSYSTEM is very case-specific -- we know particular locations where there use is permitted, and handle each one specifically. Without a cleaner way to handle this, we are likely to simply be backing ourselves into alternative #2. OTOH, if we *could* do this, it would be a great solution, because it would maintain the maximal amount of backwards-compatibility.
This is the problem I would most like to see solved in ASDF.
On Wed, Sep 21, 2016 at 9:22 AM, Robert Goldman rpgoldman@sift.net wrote:
Per my recent post to launchpad, I don't think dependency updating is the fatal flaw in D-D-O. D-D-O's bad interaction with the package system is worse.
Specifically, D-D-O cannot be used to introduce ASDF extensions unless those ASDF extensions live in the ASDF package. Otherwise, by the time a DEFSYSTEM form is read, the enclosing package's imports are already established. This means that ASDF has no protection against namespace pollution, since extenders are practically forced to add symbols to ASDF.
For example, we have a FIVEAM-ASDF extension that introduces a new defsystem class that has features for using 5AM as an implementation for the TEST-OP. But we cannot put a DEFSYSTEM-DEPENDS-ON for FIVEAM-ASDF in a system and reference FIVEAM-ASDF:FIVEAM-TESTER-SYSTEM as the system class, because the FIVEAM-ASDF package does not (in general) exist when the enclosing DEFSYSTEM is read.
I can think of three ways of handling this:
- Deprecate and remove D-D-O. This seems undesirable, since D-D-O is
commonly used.
- Replace the use of the lisp reader for processing DEFSYSTEM forms,
and introduce some kind of parser. This seems like an enormous amount of work and difficult to get right.
- In some way extend the keyword-processing features of DEFSYSTEM to
allow it to reference symbols that are not only in the current package and ASDF, but also programmer-specified packages. Unfortunately, processing of keyword symbols in DEFSYSTEM is very case-specific -- we know particular locations where there use is permitted, and handle each one specifically. Without a cleaner way to handle this, we are likely to simply be backing ourselves into alternative #2. OTOH, if we *could* do this, it would be a great solution, because it would maintain the maximal amount of backwards-compatibility.
This is the problem I would most like to see solved in ASDF.
(defsystem "foo-test" :defsystem-depends-on ("fiveam") :depends-on ("foo") :class "fiveam-asdf:fiveam-tester-system" :components (("some-system:its-component-class" "myfile" ...) ...) ...)
This has been working since 2.31.4.
IIRC, it was Stelian who prodded me to get D-D-O working properly.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Those who speak of "post-scarcity economy" thereby illustrate the irreducible scarcity of the single most important resource, human intelligence.
[...]
This is the problem I would most like to see solved in ASDF.
(defsystem "foo-test" :defsystem-depends-on ("fiveam") :depends-on ("foo") :class "fiveam-asdf:fiveam-tester-system" :components (("some-system:its-component-class" "myfile" ...) ...) ...)
This has been working since 2.31.4.
IIRC, it was Stelian who prodded me to get D-D-O working properly.
I try to use as many ASDF features as possible and useful, and I think that you can point to iolib.asd as the golden example of current best practices. And yes, I had some issues with D-D-O and Fare fixed them.
On 9/22/16 Sep 22 -9:25 AM, Stelian Ionescu wrote:
[...]
This is the problem I would most like to see solved in ASDF.
(defsystem "foo-test" :defsystem-depends-on ("fiveam") :depends-on ("foo") :class "fiveam-asdf:fiveam-tester-system" :components (("some-system:its-component-class" "myfile" ...) ...) ...)
This has been working since 2.31.4.
IIRC, it was Stelian who prodded me to get D-D-O working properly.
I try to use as many ASDF features as possible and useful, and I think that you can point to iolib.asd as the golden example of current best practices. And yes, I had some issues with D-D-O and Fare fixed them.
I'm boggled. I have been complaining about this issue in the past, and I attach an email from February of this year, in which Faré clearly implies that there is no solution to the D-D-O/package bad package interaction issues. See his point #3.
And now.... it's been fixed for years? It seems like if so, even Faré had forgotten....
[...]
On 9/22/16 Sep 22 -9:25 AM, Stelian Ionescu wrote:
[...]
This is the problem I would most like to see solved in ASDF.
(defsystem "foo-test" :defsystem-depends-on ("fiveam") :depends-on ("foo") :class "fiveam-asdf:fiveam-tester-system" :components (("some-system:its-component-class" "myfile" ...) ...) ...)
This has been working since 2.31.4.
IIRC, it was Stelian who prodded me to get D-D-O working properly.
I try to use as many ASDF features as possible and useful, and I think that you can point to iolib.asd as the golden example of current best practices. And yes, I had some issues with D-D-O and Fare fixed them.
I'm boggled. I have been complaining about this issue in the past, and I attach an email from February of this year, in which Faré clearly implies that there is no solution to the D-D-O/package bad package interaction issues. See his point #3.
And now.... it's been fixed for years? It seems like if so, even Faré had forgotten....
I think that there are sometimes misunderstandings on mailing lists and probably the best way to avoid them is to have better documentation. Faré :) ? Also, maybe I'll get to blogging about best practices in writing ASDF extensions.
:Faré
(defsystem "foo-test" :defsystem-depends-on ("fiveam") :depends-on ("foo") :class "fiveam-asdf:fiveam-tester-system" :components (("some-system:its-component-class" "myfile" ...) ...) ...)
This has been working since 2.31.4.
:Robert
I'm boggled. I have been complaining about this issue in the past, and I attach an email from February of this year, in which Faré clearly implies that there is no solution to the D-D-O/package bad package interaction issues. See his point #3.
And now.... it's been fixed for years? It seems like if so, even Faré had forgotten....
No, I thought that you of all people knew that *this* issue with D-D-O had been fixed, since it was fixed in part in response to your legitimate complaints, and you were maintainer, and I had discussed the topic on the mailing-list. I thought you were complaining about the many *remaining* issues, of which there are many.
But, yeah, D-D-O has been usable since 2.29 in February 2013, before the official 3.0.1 release of ASDF 3.
As for my point 3, it addressed using prefix as a namespace strategy, to avoid strings and use symbols or keywords. I believe this is a good enough strategy, at least for widely used free software libraries, like CFFI. If somehow you have large proprietary ASDF extensions, you might want to stick to using strings that include package prefixes.
:Stelian
I think that there are sometimes misunderstandings on mailing lists and probably the best way to avoid them is to have better documentation. Faré :) ? Also, maybe I'll get to blogging about best practices in writing ASDF extensions.
I obviously haven't been very good at documenting all the code I wrote. They say writing the docs is twice as long and hard as writing the code; I admit I didn't have the time and energy to write the docs for all the code I wrote, fixed, or rewrote. It wasn't motivating that nobody seemed to be listening anyway, beyond the very basics (and even there).
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The competitor to be feared is one who never bothers about you at all, but goes on making his own business better all the time. — Henry Ford
-
Faré -
Robert Goldman -
Stelian Ionescu