What's the right way to extend ASDF with new symbols?
Commonly, one wants to extend ASDF with new operation and component classes. But the support in ASDF for referencing such classes all involves automagically interpreting keyword symbols (and some unqualified symbols?) in the ASDF package. If I understand this correctly, this leaves the ASDF extender with the alternatives of either jamming their new symbols into the ASDF package late, and exporting them, or having them live in a new package, which is cleaner, but relegates these new components to second-class status. Consider the following (defpackage my-system-def (:use common-lisp asdf)) (in-package :my-system-def) (defsystem my-system :defsystem-depends-on ("my-asdf-extension") :components ((:my-new-file "file1"))) I don't seem to be able to define MY-NEW-FILE in my-asdf-extension and export it into my-system-def, because my-system-def is already defined by the time my-asdf-extension is loaded. That means I pretty much have to either muck around in the ASDF package (which doesn't scale to multiple different people extending ASDF) or use side effects: (asdf:load-system "my-asdf-extension") (defpackage my-system-def (:use common-lisp asdf my-asdf-extension)) (in-package :my-system-def) (defsystem my-system :components ((:my-new-file "file1"))) Am I right about this? If I am, is there some way to better handle this? Or do we need to rethink how :defsystem-depends-on works?
On Wed, Nov 18, 2015 at 8:59 AM, Robert Goldman <rpgoldman@sift.net> wrote:
Commonly, one wants to extend ASDF with new operation and component classes.
But the support in ASDF for referencing such classes all involves automagically interpreting keyword symbols (and some unqualified symbols?) in the ASDF package.
If I understand this correctly, this leaves the ASDF extender with the alternatives of either jamming their new symbols into the ASDF package late, and exporting them, or having them live in a new package, which is cleaner, but relegates these new components to second-class status.
Consider the following
(defpackage my-system-def (:use common-lisp asdf))
(in-package :my-system-def)
(defsystem my-system :defsystem-depends-on ("my-asdf-extension") :components ((:my-new-file "file1")))
I don't seem to be able to define MY-NEW-FILE in my-asdf-extension and export it into my-system-def, because my-system-def is already defined by the time my-asdf-extension is loaded. That means I pretty much have to either muck around in the ASDF package (which doesn't scale to multiple different people extending ASDF) or use side effects:
(asdf:load-system "my-asdf-extension")
(defpackage my-system-def (:use common-lisp asdf my-asdf-extension))
(in-package :my-system-def)
(defsystem my-system :components ((:my-new-file "file1")))
Am I right about this? If I am, is there some way to better handle this? Or do we need to rethink how :defsystem-depends-on works?
The problem is that the defsystem form is read before the :defsystem-depends-on clause can be processed, and hence any symbol used in the defsystem form MUST be in a package that is already defined by the time the defsystem form is read. If you stick to a purely declarative style of not having definitions in your .asd file, that pretty much means only the asdf and keyword packages are allowed. defsystem-depends-on is also broken in that ASDF assumes that it doesn't mark such extensions as needing to be reloaded and the .asd files that use them reprocessed should these extensions or their dependencies be modified. On the one hand, ASDF is very wrong. On the other hand, side effects everywhere mean that even if we ever fix that wrong (which will be a lot of work), there's still a lot of wrong in trusting extensions to not have persistent side-effects. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Be careful what you set your heart on — for it will surely be yours. — James Baldwin, "Nobody Knows My Name"
On 2/11/16 Feb 11 -8:12 PM, Faré wrote:
Commonly, one wants to extend ASDF with new operation and component classes.
But the support in ASDF for referencing such classes all involves automagically interpreting keyword symbols (and some unqualified symbols?) in the ASDF package.
If I understand this correctly, this leaves the ASDF extender with the alternatives of either jamming their new symbols into the ASDF package late, and exporting them, or having them live in a new package, which is cleaner, but relegates these new components to second-class status.
Consider the following [...snip....] The problem is that the defsystem form is read before the :defsystem-depends-on clause can be processed, and hence any symbol used in the defsystem form MUST be in a package that is already defined by the time the defsystem form is read. If you stick to a
On Wed, Nov 18, 2015 at 8:59 AM, Robert Goldman <rpgoldman@sift.net> wrote: purely declarative style of not having definitions in your .asd file, that pretty much means only the asdf and keyword packages are allowed.
defsystem-depends-on is also broken in that ASDF assumes that it doesn't mark such extensions as needing to be reloaded and the .asd files that use them reprocessed should these extensions or their dependencies be modified.
On the one hand, ASDF is very wrong. On the other hand, side effects everywhere mean that even if we ever fix that wrong (which will be a lot of work), there's still a lot of wrong in trusting extensions to not have persistent side-effects.
This is very frustrating to me. In the old days, we would just put (asdf:load-system "extension") at the top of the file. Everybody said "oh, noes! side effects!" So we added DEFSYSTEM-DEPENDS-ON. This adds a truckload of complex code to ASDF that we must maintain, but basically doesn't fix anything: * The extensions don't work like native-to-ASDF constructs, unless you put them in the ASDF package, so we are encouraging programmers to perpetrate name collisions. Note that the "bad" side-effecting construct is *BETTER* than this, because the load form can be put upstream of a defpackage in the asd file, so that the symbols will exist before the DEFSYSTEM is read. This means that ASDF extenders are *not* encouraged to mess up the ASDF package. * The dependencies, as you point out, are broken. Indeed, they may be *more* broken than when we put ASDF:LOAD-SYSTEM forms in the file. The only thing that DEFSYSTEM-DEPENDS-ON fixes, then, is introspection. But a *declaration*, instead of a semantic form would solve this problem, in a much simpler way. I suggest that DEFSYSTEM-DEPENDS-ON should turn into a declaration. I.e., it should provide introspection semantics for, e.g., quicklisp, but that's it. To keep stupid things from happening, it should error out if the depended-on system is not loaded at the time the follow-on system is loaded. That's it. End of story. Simplification ensues, as opposed to us having to maintain fussy propagation code. Also, we stop encouraging people to bork the ASDF package, which is A Bad Thing to do. Indeed, it's tempting to package lock ASDF on lisps that support that operation....
A quick follow-up: Perhaps this should be a lesson that trying to find a "pure" construct for a fundamentally side-effecting operation is a Bad Idea. Fundamentally, extending ASDF is a global change that affects the whole running image. Pretending it is something else doesn't help anyone. Now, if one wanted to have a build environment around a system build, and change only that local environment -- so that the build of the next ASDF system will occur in a pristine environment -- that might be possible. If anyone wants to try to develop such a build system -- perhaps YASDF -- more power to 'em! But it won't be ASDF. And maybe you should just be writing Haskell, instead ;-) Cheers
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But 1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so. 2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system. 3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you. 4- Yes, at some point, a better Lisp than CL will be needed. And Haskell isn't quite it. Neither is Scheme or Clojure, I fear. Maybe some future version or variant of Racket. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Who released the most slaves? The one who spent his wealth buying them back? Or the capitalist who found a way to power mills with water? — Paul Claudel On Fri, Feb 12, 2016 at 3:11 PM, Robert Goldman <rpgoldman@sift.net> wrote:
On 2/11/16 Feb 11 -8:12 PM, Faré wrote:
Commonly, one wants to extend ASDF with new operation and component classes.
But the support in ASDF for referencing such classes all involves automagically interpreting keyword symbols (and some unqualified symbols?) in the ASDF package.
If I understand this correctly, this leaves the ASDF extender with the alternatives of either jamming their new symbols into the ASDF package late, and exporting them, or having them live in a new package, which is cleaner, but relegates these new components to second-class status.
Consider the following [...snip....] The problem is that the defsystem form is read before the :defsystem-depends-on clause can be processed, and hence any symbol used in the defsystem form MUST be in a package that is already defined by the time the defsystem form is read. If you stick to a
On Wed, Nov 18, 2015 at 8:59 AM, Robert Goldman <rpgoldman@sift.net> wrote: purely declarative style of not having definitions in your .asd file, that pretty much means only the asdf and keyword packages are allowed.
defsystem-depends-on is also broken in that ASDF assumes that it doesn't mark such extensions as needing to be reloaded and the .asd files that use them reprocessed should these extensions or their dependencies be modified.
On the one hand, ASDF is very wrong. On the other hand, side effects everywhere mean that even if we ever fix that wrong (which will be a lot of work), there's still a lot of wrong in trusting extensions to not have persistent side-effects.
This is very frustrating to me.
In the old days, we would just put
(asdf:load-system "extension")
at the top of the file. Everybody said "oh, noes! side effects!"
So we added DEFSYSTEM-DEPENDS-ON.
This adds a truckload of complex code to ASDF that we must maintain, but basically doesn't fix anything:
* The extensions don't work like native-to-ASDF constructs, unless you put them in the ASDF package, so we are encouraging programmers to perpetrate name collisions.
Note that the "bad" side-effecting construct is *BETTER* than this, because the load form can be put upstream of a defpackage in the asd file, so that the symbols will exist before the DEFSYSTEM is read. This means that ASDF extenders are *not* encouraged to mess up the ASDF package.
* The dependencies, as you point out, are broken. Indeed, they may be *more* broken than when we put ASDF:LOAD-SYSTEM forms in the file.
The only thing that DEFSYSTEM-DEPENDS-ON fixes, then, is introspection. But a *declaration*, instead of a semantic form would solve this problem, in a much simpler way.
I suggest that DEFSYSTEM-DEPENDS-ON should turn into a declaration. I.e., it should provide introspection semantics for, e.g., quicklisp, but that's it. To keep stupid things from happening, it should error out if the depended-on system is not loaded at the time the follow-on system is loaded. That's it. End of story.
Simplification ensues, as opposed to us having to maintain fussy propagation code.
Also, we stop encouraging people to bork the ASDF package, which is A Bad Thing to do.
Indeed, it's tempting to package lock ASDF on lisps that support that operation....
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated: 1. It effectively forces you to stick new symbols into the ASDF namespace. I don't understand your proposed rebuttal, involving slash-named packages. I don't see any evidence of this being legal ASDF syntax, looking at FIND-CLASS*, and trying an experiment. If it works, it needs documentation. If it does not work, it will not be added -- ASDF must get simpler, not more complex. Please amplify, thanks! 2. If you are arguing that we can just solve this with a naming convention, I don't buy it. Consider different libraries, each of which hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes. This is not a far-fetched example; I have seen it. They both try to jam these symbols into ASDF. This behavior should be *strongly* discouraged, even to the extent of (as I said earlier) package-locking ASDF when possible. Currently, it is actively *ENCOURAGED* -- close to mandated, in fact. 3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of simplification ensues, including eliminating the complex double-parsing of DEFSYSTEM. ASDF is currently over-complicated and over-long. 4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed. 4. backwards compatibility involves nothing more than adding a LOAD-SYSTEM form to the top of a file. 5. DEFSYSTEM-DEPENDS-ON as introspection will persist, so that introspection continues to be supported. TL;DR: We have a "declarative" construct which is not declarative at all. Worse, it almost forces poisonous namespace pollution. Killing it would simplify the code. Minimal backwards-compatibility issues. Alternative: if you, or someone else, will take over ASDF maintainership, you can keep DEFSYSTEM-DEPENDS-ON. I will happily leave it to you! The bottom line: ASDF is way too big for me to wrap my head around, in very limited available time. It has to get simpler. Removing things that don't work is a good start.
Hey, I just want to say that I strongly support simplification of the ASDF. It's a great project and solves gazillion of problems, but having actual specification with some core features would make it way better for me. Even if not for this release, the next release might focus on stabilizing the API and freezing it, because constant evolving isn't the best thing I could imagine for the de-facto standard. Best regards, Daniel Kochmański Robert Goldman writes:
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated:
1. It effectively forces you to stick new symbols into the ASDF namespace.
I don't understand your proposed rebuttal, involving slash-named packages. I don't see any evidence of this being legal ASDF syntax, looking at FIND-CLASS*, and trying an experiment. If it works, it needs documentation. If it does not work, it will not be added -- ASDF must get simpler, not more complex. Please amplify, thanks!
2. If you are arguing that we can just solve this with a naming convention, I don't buy it. Consider different libraries, each of which hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes. This is not a far-fetched example; I have seen it. They both try to jam these symbols into ASDF.
This behavior should be *strongly* discouraged, even to the extent of (as I said earlier) package-locking ASDF when possible. Currently, it is actively *ENCOURAGED* -- close to mandated, in fact.
3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of simplification ensues, including eliminating the complex double-parsing of DEFSYSTEM. ASDF is currently over-complicated and over-long.
4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed.
4. backwards compatibility involves nothing more than adding a LOAD-SYSTEM form to the top of a file.
5. DEFSYSTEM-DEPENDS-ON as introspection will persist, so that introspection continues to be supported.
TL;DR: We have a "declarative" construct which is not declarative at all. Worse, it almost forces poisonous namespace pollution. Killing it would simplify the code. Minimal backwards-compatibility issues.
Alternative: if you, or someone else, will take over ASDF maintainership, you can keep DEFSYSTEM-DEPENDS-ON. I will happily leave it to you!
The bottom line: ASDF is way too big for me to wrap my head around, in very limited available time. It has to get simpler. Removing things that don't work is a good start.
-- Daniel Kochmański ;; aka jackdaniel | Poznań, Poland TurtleWare - Daniel Kochmański | www.turtleware.eu "Be the change that you wish to see in the world." - Mahatma Gandhi
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated:
1. It effectively forces you to stick new symbols into the ASDF namespace.
Technically yes, but that's not the essential requirement. Because the CL reader interns eagerly, ASDF extension classes need to be interned into a package that is owned by ASDF. Currently that's the ASDF package itself, but it would be a good idea to add a special package for extensions, and start encouraging people to use that by showing appropriate deprecation messages.
I don't understand your proposed rebuttal, involving slash-named packages. I don't see any evidence of this being legal ASDF syntax, looking at FIND-CLASS*, and trying an experiment. If it works, it needs documentation. If it does not work, it will not be added -- ASDF must get simpler, not more complex. Please amplify, thanks!
It's not "syntax", it's just manual namespacing. The symbol 'asdf::pkg/class is symply a legal symbol. And I don't agree that ASDF must get simpler at all costs, rather that it should have as simple an implementation as possible, while still allowing the use cases that its users require.
2. If you are arguing that we can just solve this with a naming convention, I don't buy it. Consider different libraries, each of which hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes. This is not a far-fetched example; I have seen it. They both try to jam these symbols into ASDF.
See my previous reply.
This behavior should be *strongly* discouraged, even to the extent of (as I said earlier) package-locking ASDF when possible. Currently, it is actively *ENCOURAGED* -- close to mandated, in fact.
By dedicating a package for naming extension classes, we can even lock ASDF while still making QA easier.
3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of simplification ensues, including eliminating the complex double-parsing of DEFSYSTEM. ASDF is currently over-complicated and over-long.
In what sense is it not currently a declaration ?
4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed.
I think this might be an issue with the current implementation of DEFSYSTEM-DEPENDS-ON, but that's not a necessity. E.g.: (defsystem foo :defsystem-depends-on (foo/asd) :components ((:foo/file "foo"))) This ought to mean that LOAD-SYSTEM of "foo" depends on LOAD-SYSTEM of "foo/asd", and that the exact class object named by "foo/file" should be fetched right after loading "foo/asd".
4. backwards compatibility involves nothing more than adding a LOAD-SYSTEM form to the top of a file.
This breaks one of the most important use cases: doing large scale static analysis of dependencies, like in the case of Quicklisp. At the moment, one cannot even be sure of parsing a .asd without compiling code, unless using a custom reader and/or regular expressions.
5. DEFSYSTEM-DEPENDS-ON as introspection will persist, so that introspection continues to be supported.
I'm not sure what you mean by this.
TL;DR: We have a "declarative" construct which is not declarative at all. Worse, it almost forces poisonous namespace pollution. Killing it would simplify the code. Minimal backwards-compatibility issues.
I think I showed how it can be improved without removing it. Killing it would be a disaster for those who rely on it.
Alternative: if you, or someone else, will take over ASDF maintainership, you can keep DEFSYSTEM-DEPENDS-ON. I will happily leave it to you!
I'm considering this. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 2/15/16 Feb 15 -10:26 AM, Stelian Ionescu wrote:
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated:
1. It effectively forces you to stick new symbols into the ASDF namespace.
Technically yes, but that's not the essential requirement. Because the CL reader interns eagerly, ASDF extension classes need to be interned into a package that is owned by ASDF. Currently that's the ASDF package itself, but it would be a good idea to add a special package for extensions, and start encouraging people to use that by showing appropriate deprecation messages.
This doesn't do anything to resolve the underlying problem, AFAICT. I.e., instead of getting name collisions in ASDF (I believe that's ASDF/INTERFACE), we get name collisions in ASDF/EXTENSIONS. Am I missing something about this suggestion?
I don't understand your proposed rebuttal, involving slash-named packages. I don't see any evidence of this being legal ASDF syntax, looking at FIND-CLASS*, and trying an experiment. If it works, it needs documentation. If it does not work, it will not be added -- ASDF must get simpler, not more complex. Please amplify, thanks!
It's not "syntax", it's just manual namespacing. The symbol 'asdf::pkg/class is symply a legal symbol. And I don't agree that ASDF must get simpler at all costs, rather that it should have as simple an implementation as possible, while still allowing the use cases that its users require.
With all due respect, I believe ASDF to be unmaintainable now, except for a Hail Mary Pass to Faré at regular intervals. Again, you are welcome to have a whack at it. I won't be upset if you prove me wrong!
2. If you are arguing that we can just solve this with a naming convention, I don't buy it. Consider different libraries, each of which hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes. This is not a far-fetched example; I have seen it. They both try to jam these symbols into ASDF.
See my previous reply.
This behavior should be *strongly* discouraged, even to the extent of (as I said earlier) package-locking ASDF when possible. Currently, it is actively *ENCOURAGED* -- close to mandated, in fact.
By dedicating a package for naming extension classes, we can even lock ASDF while still making QA easier.
Per earlier response, this seems to me to just kick the problem from one namespace/package to another.
3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of simplification ensues, including eliminating the complex double-parsing of DEFSYSTEM. ASDF is currently over-complicated and over-long.
In what sense is it not currently a declaration ?
What I meant is that it's not *only* a declaration. It has extra operational import. I would like to change D-D-O to be only declarative (except that we will also check it).
4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed.
I think this might be an issue with the current implementation of DEFSYSTEM-DEPENDS-ON, but that's not a necessity. E.g.:
(defsystem foo :defsystem-depends-on (foo/asd) :components ((:foo/file "foo")))
This ought to mean that LOAD-SYSTEM of "foo" depends on LOAD-SYSTEM of "foo/asd", and that the exact class object named by "foo/file" should be fetched right after loading "foo/asd".
Fetched from where? Do you want to further extend the syntax so that FOO/FILE is interpreted as FOO:FILE? That might be feasible (suggest you try). It would certainly be preferable to adding an ASDF/EXTENSIONS package and fighting over its contents. *HOWEVER* that doesn't address the operation-naming problem, which would require further syntactic extensions. This is what really concerns me -- doing DEFSYSTEM-DEPENDS-ON *properly* requires a lot more work, which has not been well thought out. And it's not something I want to do (I maintain ASDF because I rely on it, not because I have a fundamental interest in system-building systems -- Faré was more interested in this as a research problem). I'd rather remove a mostly-broken implementation than maintain it.
4. backwards compatibility involves nothing more than adding a LOAD-SYSTEM form to the top of a file.
This breaks one of the most important use cases: doing large scale static analysis of dependencies, like in the case of Quicklisp. At the moment, one cannot even be sure of parsing a .asd without compiling code, unless using a custom reader and/or regular expressions.
I get your point, but this won't break anything that isn't already broken. If you want to refer to new components as first class entities (i.e., without syntactic mess), the only thing you can do is the following: (1) (load-system <extension-system>) (defpackage my-system-asd (:use asdf extension-package common-lisp)) <system declaration> ... or (2) shovel stuff into the ASDF package. So if programmers do (1) already, your important use case is already broken. And there is no way to prevent them from doing so. My guess is that Quicklisp simply doesn't contain libraries with that construct (which is fine for QL). For example, CFFI takes method (2) and jams stuff into the ASDF package to make GROVEL-FILE work, rather than keeping its extensions in a separate package. If ASDF was a different system, and it had special files, instead of files that are lisp source, and can contain arbitrary forms, then this argument would be stronger. But it's not that way. Maybe it *should* be, but that's a different system. As an aside: I think ASDF has gone as far as it can go on gut, rather than spec. The discussion here shows that we have reached those limits, because we have features that are poorly understood, not well-documented anywhere, and that become critical while still only lightly baked. E.g., for me the namespace pollution is a critical pain-point, which doesn't seem to bother others. Contrariwise, I have no idea what requirements Quicklisp exerts on ASDF (especially now that Xach has fled this mailing list). I don't believe either of us well understands the current design intent.
5. DEFSYSTEM-DEPENDS-ON as introspection will persist, so that introspection continues to be supported.
I'm not sure what you mean by this.
You will still be able to load systems and find out which ones depend on which other ones. In the old days, before DEFSYSTEM-DEPENDS-ON, method (1) was mandatory for extending ASDF, and there was no way to interrogate the lisp image t o tell what extensions were required by what systems.
TL;DR: We have a "declarative" construct which is not declarative at all. Worse, it almost forces poisonous namespace pollution. Killing it would simplify the code. Minimal backwards-compatibility issues.
I think I showed how it can be improved without removing it. Killing it would be a disaster for those who rely on it.
I don't think we are there yet. Showing how it can be improved requires an approach that covers *both* component types *and* operations, for one thing. It should also support use of normal syntax (possibly extended, like your "slashy" syntax) to reference extensions *without* shoving new symbols into ASDF. If someone wants to provide this, I would take such in place of removing the current behavior. *BUT* "providing" does not mean a patch only. It would have to be accompanied by a list of requirements -- nothing too fancy, a bulleted list in Markdown or something like it would be fine -- and a paragraph or two explaining how the proposed approach meets those requirements. If you are interested in that, I'd be happy to help you assemble the requirements: I think they are hidden in this email exchange.
Alternative: if you, or someone else, will take over ASDF maintainership, you can keep DEFSYSTEM-DEPENDS-ON. I will happily leave it to you!
I'm considering this.
Offer is still open. Cheers, r
On Mon, 2016-02-15 at 11:13 -0600, Robert Goldman wrote:
On 2/15/16 Feb 15 -10:26 AM, Stelian Ionescu wrote:
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated:
1. It effectively forces you to stick new symbols into the ASDF namespace.
Technically yes, but that's not the essential requirement. Because the CL reader interns eagerly, ASDF extension classes need to be interned into a package that is owned by ASDF. Currently that's the ASDF package itself, but it would be a good idea to add a special package for extensions, and start encouraging people to use that by showing appropriate deprecation messages.
This doesn't do anything to resolve the underlying problem, AFAICT. I.e., instead of getting name collisions in ASDF (I believe that's ASDF/INTERFACE), we get name collisions in ASDF/EXTENSIONS.
I don't see this as a problem. A few times there were collisions in Quicklisp, so somebody had to rename the project/package, etc... If two different authors want to use asdf/extensions::foo/file then one will have to give up. This is a technical problem(the design of the CL reader) that is best solved socially. Also, it's not like people write new ASDF extensions everyday.
Am I missing something about this suggestion?
I don't understand your proposed rebuttal, involving slash-named packages. I don't see any evidence of this being legal ASDF syntax, looking at FIND-CLASS*, and trying an experiment. If it works, it needs documentation. If it does not work, it will not be added -- ASDF must get simpler, not more complex. Please amplify, thanks!
It's not "syntax", it's just manual namespacing. The symbol 'asdf::pkg/class is symply a legal symbol. And I don't agree that ASDF must get simpler at all costs, rather that it should have as simple an implementation as possible, while still allowing the use cases that its users require.
With all due respect, I believe ASDF to be unmaintainable now, except for a Hail Mary Pass to Faré at regular intervals. Again, you are welcome to have a whack at it. I won't be upset if you prove me wrong!
2. If you are arguing that we can just solve this with a naming convention, I don't buy it. Consider different libraries, each of which hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes. This is not a far-fetched example; I have seen it. They both try to jam these symbols into ASDF.
See my previous reply.
This behavior should be *strongly* discouraged, even to the extent of (as I said earlier) package-locking ASDF when possible. Currently, it is actively *ENCOURAGED* -- close to mandated, in fact.
By dedicating a package for naming extension classes, we can even lock ASDF while still making QA easier.
Per earlier response, this seems to me to just kick the problem from one namespace/package to another.
3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of simplification ensues, including eliminating the complex double-parsing of DEFSYSTEM. ASDF is currently over-complicated and over-long.
In what sense is it not currently a declaration ?
What I meant is that it's not *only* a declaration. It has extra operational import. I would like to change D-D-O to be only declarative (except that we will also check it).
4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed.
I think this might be an issue with the current implementation of DEFSYSTEM-DEPENDS-ON, but that's not a necessity. E.g.:
(defsystem foo :defsystem-depends-on (foo/asd) :components ((:foo/file "foo")))
This ought to mean that LOAD-SYSTEM of "foo" depends on LOAD-SYSTEM of "foo/asd", and that the exact class object named by "foo/file" should be fetched right after loading "foo/asd".
Fetched from where? Do you want to further extend the syntax so that FOO/FILE is interpreted as FOO:FILE? That might be feasible (suggest you try). It would certainly be preferable to adding an ASDF/EXTENSIONS package and fighting over its contents.
No, have it search for a symbol named (string :foo/file) first in ASDF/EXTENSIONS then ASDF, for backwards-compatibility, then one day only ASDF/EXTENSIONS. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Mon, Feb 15, 2016 at 3:19 PM, Stelian Ionescu <sionescu@cddr.org> wrote:
No, have it search for a symbol named (string :foo/file) first in ASDF/EXTENSIONS then ASDF, for backwards-compatibility, then one day only ASDF/EXTENSIONS.
1- Why not just the use existing ASDF/USER rather than a newfangled ASDF/EXTENSIONS ? 2- I understand both points of view, and don't have an opinion — except that whichever way the ASDF maintainer moves, it will take a lot of efforts and a lot of time before you can pull the plug on the current code. For reference, Clojure also has the convention of prefixing keywords with a "namespace" name (their namespaces are more or less the moral equivalent of CL packages), as in :foo/bar for a keyword bar that declares being owned by foo. I can also see the point of using the existing CL package conventions. I removed in ASDF3 the old ASDF 1.x trick of having temporary packages ASDF0, ASDF1, etc., for each .asd file being read, instead of a common ASDF-USER and letting people manage their own packages manually. That old didn't actually help with symbol hygiene, and only got in the way, while people still needed to defpackage their own packages for extensions and anything worth clashing. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Documentation is worth it just to be able to answer all your mail with 'RTFM'. — Alan Cox
On 2/15/16 Feb 15 -2:19 PM, Stelian Ionescu wrote:
On Mon, 2016-02-15 at 11:13 -0600, Robert Goldman wrote:
On 2/15/16 Feb 15 -10:26 AM, Stelian Ionescu wrote:
On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system (or load-systems) the official way to go. But
1- This of course requires heads up, updating all users before retiring the feature, etc. From my experience, if you start seriously deprecating today and sending patches to all authors who use it in quicklisp, you can expect to be removing that part of the code in two years or so.
2- To make these dependencies work properly still requires modifying ASDF to add explicit plan nodes for loading ASD files, that will contain the systems loaded by load-system. The same trick will also automatically make the :defsystem-depends-on work, since it itself calls load-system.
3- Yes, defining things in the ASDF package is ugly, but extensions are few enough, and using a prefix is a good enough namespace management strategy. Not the most horrible thing that working with CL does to you.
Please don't. It's a net improvement compared to the previous situation. It's easy to simply name your class with a keyword :package/foo-file.
With all due respect, no, it is not. It is a net *negative* with respect to the previous situation. Here are the reasons why, briefly reiterated:
1. It effectively forces you to stick new symbols into the ASDF namespace.
Technically yes, but that's not the essential requirement. Because the CL reader interns eagerly, ASDF extension classes need to be interned into a package that is owned by ASDF. Currently that's the ASDF package itself, but it would be a good idea to add a special package for extensions, and start encouraging people to use that by showing appropriate deprecation messages.
This doesn't do anything to resolve the underlying problem, AFAICT. I.e., instead of getting name collisions in ASDF (I believe that's ASDF/INTERFACE), we get name collisions in ASDF/EXTENSIONS.
I don't see this as a problem. A few times there were collisions in Quicklisp, so somebody had to rename the project/package, etc... If two different authors want to use asdf/extensions::foo/file then one will have to give up. This is a technical problem(the design of the CL reader) that is best solved socially. Also, it's not like people write new ASDF extensions everyday.
I'm sorry -- I *do* see this as a problem. And it's precisely the problem that CL packages were intended to solve. I do not agree that this is a problem of the design of the CL reader that must be solved socially. I am willing to see a different solution than my proposal, but I will not accept any solution -- including the status quo -- that requires people to worry about every other possible ASDF extension. If you want the maintainership, you can have it your way. Otherwise, this goes. [..snip...]
4. The double-parsing doesn't even work, because the packages don't exist at the right time. That's why, even with DEFSYSTEM-DEPENDS-ON, you must either mess with the ASDF package, or put in a LOAD-SYSTEM call to get the symbols created, and stuck into *PACKAGE* before the defsystem form is parsed.
I think this might be an issue with the current implementation of DEFSYSTEM-DEPENDS-ON, but that's not a necessity. E.g.:
(defsystem foo :defsystem-depends-on (foo/asd) :components ((:foo/file "foo")))
This ought to mean that LOAD-SYSTEM of "foo" depends on LOAD-SYSTEM of "foo/asd", and that the exact class object named by "foo/file" should be fetched right after loading "foo/asd".
Fetched from where? Do you want to further extend the syntax so that FOO/FILE is interpreted as FOO:FILE? That might be feasible (suggest you try). It would certainly be preferable to adding an ASDF/EXTENSIONS package and fighting over its contents.
No, have it search for a symbol named (string :foo/file) first in ASDF/EXTENSIONS then ASDF, for backwards-compatibility, then one day only ASDF/EXTENSIONS.
Sorry, I'm not trying to be difficult, but that solution is unacceptable to me. From my POV as maintainer, it seems like the worst of all worlds. We would be introducing yet more moving parts -- a new package, ASDF-EXTENSIONS, that we would have to keep track of -- without solving the problem of name collisions. This seems strictly worse than the status quo. If someone wants to *fix* the double-parsing solution, that's fine. But I'm not going to keep around the broken double-parsing solution. I'm willing to try to support what QL needs, but I need some help in return.
Trying again: Can someone please state what it is that Quicklisp needs? IIUC Quicklisp does *something* with .asd files that does not involve the defsystem-depends-on being resolved. Is this reading? Or loading? If it's loading, then the DEFSYSTEM-DEPENDS-ON entries are resolved by REGISTER-SYSTEM-DEFINITION, so the status quo doesn't solve the problem of large scale analysis without side-effects. If it's reading, then my solution is benign. At least as far as I can tell. Can someone give me a clue here? thanks! r
On Mon, 2016-02-15 at 14:48 -0600, Robert Goldman wrote:
Trying again:
Can someone please state what it is that Quicklisp needs?
IIUC Quicklisp does *something* with .asd files that does not involve the defsystem-depends-on being resolved.
Is this reading? Or loading?
If it's loading, then the DEFSYSTEM-DEPENDS-ON entries are resolved by REGISTER-SYSTEM-DEFINITION, so the status quo doesn't solve the problem of large scale analysis without side-effects.
If it's reading, then my solution is benign.
At least as far as I can tell.
Can someone give me a clue here?
Quicklisp needs to compute the complete dependency graph between systems, given a set of .asd files. I'd like to extend the current Slime integration to search for systems in all registered .asd files and autocomplete. All sorts of things become possible if one could treat a set of .asd files as a database, with the guarantee that querying would never trigger a compile-file operation, ever. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 2/15/16 Feb 15 -3:06 PM, Stelian Ionescu wrote:
On Mon, 2016-02-15 at 14:48 -0600, Robert Goldman wrote:
Trying again:
Can someone please state what it is that Quicklisp needs?
IIUC Quicklisp does *something* with .asd files that does not involve the defsystem-depends-on being resolved.
Is this reading? Or loading?
If it's loading, then the DEFSYSTEM-DEPENDS-ON entries are resolved by REGISTER-SYSTEM-DEFINITION, so the status quo doesn't solve the problem of large scale analysis without side-effects.
If it's reading, then my solution is benign.
At least as far as I can tell.
Can someone give me a clue here?
Quicklisp needs to compute the complete dependency graph between systems, given a set of .asd files. I'd like to extend the current Slime integration to search for systems in all registered .asd files and autocomplete. All sorts of things become possible if one could treat a set of .asd files as a database, with the guarantee that querying would never trigger a compile-file operation, ever.
If that's the case, then I believe that the existing implementation *already* fails your requirement. When you register an ASDF system, the DEFSYSTEM-DEPENDS-ON dependencies must be loaded (and therefore, possibly compiled). ASDF needs to do this to parse the defsystem form correctly. So I think that we may have been arguing over nothing. Thanks, r
On Mon, Feb 15, 2016 at 3:48 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Trying again:
Can someone please state what it is that Quicklisp needs?
Quicklisp needs some way to determine all the direct dependencies required to load a system and its definition. It currently does this by direct inspection of the defsystem form (with a *macroexpand-hook*) *and* by registering an :around method for asdf:operate to catch any direct calls to asdf:load-system. -Eric
On 2/16/16 Feb 16 -9:51 AM, Eric Timmons wrote:
On Mon, Feb 15, 2016 at 3:48 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Trying again:
Can someone please state what it is that Quicklisp needs?
Quicklisp needs some way to determine all the direct dependencies required to load a system and its definition. It currently does this by direct inspection of the defsystem form (with a *macroexpand-hook*) *and* by registering an :around method for asdf:operate to catch any direct calls to asdf:load-system.
-Eric
If that is the case, then my proposed change will not upset QL. I was pretty sure it wouldn't, after checking the current implementation of DEFSYSTEM-DEPENDS-ON, but this adds a lot of confidence. Thank you very much, Eric. r
On 2/16/16 Feb 16 -9:51 AM, Eric Timmons wrote:
On Mon, Feb 15, 2016 at 3:48 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Trying again:
Can someone please state what it is that Quicklisp needs?
Quicklisp needs some way to determine all the direct dependencies required to load a system and its definition. It currently does this by direct inspection of the defsystem form (with a *macroexpand-hook*) *and* by registering an :around method for asdf:operate to catch any direct calls to asdf:load-system.
-Eric
P.S. if we can work out some way of integrating the inspection into ASDF, I'd be happy to do that. That way we could put a test for it in the ASDF test suite, and would know if we were borking QL by ASDF changes going forward.
For some historical perspective, defsystem-depends-on was added at a time when we wanted a more declarative solution than using load-system in a .asd file, and wasn't initially thought through. It took over a year for it to become an actually useful feature, after realizing the package issue and allowing and recommending the use of symbols in the asdf and/or keyword packages as a workaround. At the time, the dependency graph of ASDF wasn't understood by anyone, and indeed was broken, and there was no hope of ever turning a load-system into a dependency. Now, the dependency graph is understood and fixed, and there's a way to handle such dependencies right, whether through load-system or through defsystem-depends-on — but it requires a new maintainer to do some non-trivial re-coding of load-asd and/or parse-defsystem. That would mean a major release of ASDF 3.2 and/or ASDF 4 as a result. Basically, you'll need to (a) add new dependency nodes for loading an asd file, (b) properly handle recursive calls to operate from find-system and/or make-plan, and (c) use special variables to track context and maintain dependency graph in presence of such calls. Will anyone do it? Neither Robert nor I has the will to do it, though I will gladly consult with whoever does it (if he cares from my opinions). Once this capability is added to ASDF, the rationale for defsystem-depends-on vanishes: just use load-system or load-systems from a .asd file. You may still want to restrict .asd file to a short whitelist of allowed forms to keep it "declarative". As for what Quicklisp needs, I suppose it is a way to reliably find *all* the transitive dependencies of a system, if possible without loading it, though perhaps after loading its defsystem-depends-on dependencies, which might be required in the general case. Right now, the best you can do is load into an empty image, and inspect all that's loaded afterwards. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Constitution, article 1: any person who claims that a document called "constitution" possesses any authority whatsoever on other people is an impostor guilty of high treason and liable to immediate capital punishment.
On Mon, Feb 15, 2016 at 3:34 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Sorry, I'm not trying to be difficult, but that solution is unacceptable to me. From my POV as maintainer, it seems like the worst of all worlds. We would be introducing yet more moving parts -- a new package, ASDF-EXTENSIONS, that we would have to keep track of -- without solving the problem of name collisions. This seems strictly worse than the status quo.
If someone wants to *fix* the double-parsing solution, that's fine. But I'm not going to keep around the broken double-parsing solution.
Would there be anything wrong with saying everyone should use strings to name symbols from ASDF extensions? Component types and :class already seem to support this. A quick glance through the code makes me think :in-order-to and inline methods would be fairly easy to modify to support it as well. I'd be happy to send in a patch for it. It seems this would solve the issue with namespace clashes and get rid of the need for load-systems before the defsystem. There's still a lot of work if someone wants a completely declarative version of defsystem, but at least it's a step in that direction. -Eric
On 2/16/16 Feb 16 -10:12 AM, Eric Timmons wrote:
On Mon, Feb 15, 2016 at 3:34 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Sorry, I'm not trying to be difficult, but that solution is unacceptable to me. From my POV as maintainer, it seems like the worst of all worlds. We would be introducing yet more moving parts -- a new package, ASDF-EXTENSIONS, that we would have to keep track of -- without solving the problem of name collisions. This seems strictly worse than the status quo.
If someone wants to *fix* the double-parsing solution, that's fine. But I'm not going to keep around the broken double-parsing solution.
Would there be anything wrong with saying everyone should use strings to name symbols from ASDF extensions? Component types and :class already seem to support this. A quick glance through the code makes me think :in-order-to and inline methods would be fairly easy to modify to support it as well. I'd be happy to send in a patch for it.
It seems this would solve the issue with namespace clashes and get rid of the need for load-systems before the defsystem. There's still a lot of work if someone wants a completely declarative version of defsystem, but at least it's a step in that direction.
I don't see that strings make an obvious improvement over the way we use keywords. Here's the situation: Keywords are interpreted as follows: i. names in current package (this is how we get in extensions, if we have imported the extension package into the current package) ii. names in ASDF (this is how we get in extensions if we have done the Bad Thing and imported our extension names there). I would like to avoid adding further complications. Note that strings would make processing the operation names more complicated than they are now. If you want to reference an extension operator, and you import that extension into the current package, then everything will work fine. There's no need to add anything -- unless you want to keep the current DEFSYSTEM-DEPENDS-ON. If you do, then complications ensue. What I am looking for is a way to leave as much as possible of the work to Common Lisp, and as little as possible to additional code in ASDF. That's why I want to have people use ASDF extensions the way they would use any other library: - Load it. - Import the symbols into the current package (either through selective import or use-package) - parse as normal. I appreciate your offer, but I would prefer not to take it because: 1. It adds complexity, instead of removing it 2. It still doesn't handle the namespace collision problem, because you would still need some magic to decide where to look for the referents of the strings. loading the required extension systems "by hand" makes all these complications -- and the namespace problems -- go away, by letting Common Lisp do its normal Common Lisp thing. Best, r
On Tue, Feb 16, 2016 at 3:14 PM, Robert Goldman <rpgoldman@sift.net> wrote:
On 2/16/16 Feb 16 -10:12 AM, Eric Timmons wrote:
On Mon, Feb 15, 2016 at 3:34 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Sorry, I'm not trying to be difficult, but that solution is unacceptable to me. From my POV as maintainer, it seems like the worst of all worlds. We would be introducing yet more moving parts -- a new package, ASDF-EXTENSIONS, that we would have to keep track of -- without solving the problem of name collisions. This seems strictly worse than the status quo.
If someone wants to *fix* the double-parsing solution, that's fine. But I'm not going to keep around the broken double-parsing solution.
Would there be anything wrong with saying everyone should use strings to name symbols from ASDF extensions? Component types and :class already seem to support this. A quick glance through the code makes me think :in-order-to and inline methods would be fairly easy to modify to support it as well. I'd be happy to send in a patch for it.
It seems this would solve the issue with namespace clashes and get rid of the need for load-systems before the defsystem. There's still a lot of work if someone wants a completely declarative version of defsystem, but at least it's a step in that direction.
I don't see that strings make an obvious improvement over the way we use keywords. Here's the situation:
Keywords are interpreted as follows: i. names in current package (this is how we get in extensions, if we have imported the extension package into the current package) ii. names in ASDF (this is how we get in extensions if we have done the Bad Thing and imported our extension names there).
Sorry, I think I may not have expressed what I meant clearly. When I said strings, I really meant "strings containing a fully qualified symbol name". To use CFFI as an example, this: ``` (cl:eval-when (:load-toplevel :execute) (asdf:operate 'asdf:load-op 'cffi-grovel)) (asdf:defsystem example-software :depends-on (cffi) :serial t :components ((:file "package") (cffi-grovel:grovel-file "example-grovelling") (:file "example"))) ``` Would become this: ``` (asdf:defsystem example-software :depends-on (cffi) :defsystem-depends-on (cffi-grovel) :serial t :components ((:file "package") ("cffi-grovel:grovel-file" "example-grovelling") (:file "example"))) ``` This is already supported for component types and the :class argument (using coerce-class). I can't find it in the documentation (but I could swear I saw it in there at some point). The logic in coerce-class does exactly what you're describing for symbols and keywords. For strings it uses uiop:safe-read-from-string. Since the reader never sees the symbol from the extension package before it's loaded, there's no need to load-system it beforehand. But if someone doesn't want to always prefix with a package name, they can continue loading the system beforehand and using the symbol. It looks like minor tweaks to the code to add the same support to operations in the defsystem as well. It's the same logic as coerce-class, just possibly without a find-class in there. Strings do have a certain ugliness to them that keywords don't, so maybe this isn't the best direction for ASDF. But I think making support for this uniform, locking the ASDF package, and encouraging people to use it would solve the namespacing problem. -Eric
On 2/16/16 Feb 16 -3:05 PM, Eric Timmons wrote:
On Tue, Feb 16, 2016 at 3:14 PM, Robert Goldman <rpgoldman@sift.net> wrote:
On 2/16/16 Feb 16 -10:12 AM, Eric Timmons wrote:
On Mon, Feb 15, 2016 at 3:34 PM, Robert Goldman <rpgoldman@sift.net> wrote:
Sorry, I'm not trying to be difficult, but that solution is unacceptable to me. From my POV as maintainer, it seems like the worst of all worlds. We would be introducing yet more moving parts -- a new package, ASDF-EXTENSIONS, that we would have to keep track of -- without solving the problem of name collisions. This seems strictly worse than the status quo.
If someone wants to *fix* the double-parsing solution, that's fine. But I'm not going to keep around the broken double-parsing solution.
Would there be anything wrong with saying everyone should use strings to name symbols from ASDF extensions? Component types and :class already seem to support this. A quick glance through the code makes me think :in-order-to and inline methods would be fairly easy to modify to support it as well. I'd be happy to send in a patch for it.
It seems this would solve the issue with namespace clashes and get rid of the need for load-systems before the defsystem. There's still a lot of work if someone wants a completely declarative version of defsystem, but at least it's a step in that direction.
I don't see that strings make an obvious improvement over the way we use keywords. Here's the situation:
Keywords are interpreted as follows: i. names in current package (this is how we get in extensions, if we have imported the extension package into the current package) ii. names in ASDF (this is how we get in extensions if we have done the Bad Thing and imported our extension names there).
Sorry, I think I may not have expressed what I meant clearly. When I said strings, I really meant "strings containing a fully qualified symbol name". To use CFFI as an example, this:
``` (cl:eval-when (:load-toplevel :execute) (asdf:operate 'asdf:load-op 'cffi-grovel))
(asdf:defsystem example-software :depends-on (cffi) :serial t :components ((:file "package") (cffi-grovel:grovel-file "example-grovelling") (:file "example"))) ```
Would become this:
``` (asdf:defsystem example-software :depends-on (cffi) :defsystem-depends-on (cffi-grovel) :serial t :components ((:file "package") ("cffi-grovel:grovel-file" "example-grovelling") (:file "example"))) ```
This is already supported for component types and the :class argument (using coerce-class). I can't find it in the documentation (but I could swear I saw it in there at some point).
The logic in coerce-class does exactly what you're describing for symbols and keywords. For strings it uses uiop:safe-read-from-string. Since the reader never sees the symbol from the extension package before it's loaded, there's no need to load-system it beforehand. But if someone doesn't want to always prefix with a package name, they can continue loading the system beforehand and using the symbol.
It looks like minor tweaks to the code to add the same support to operations in the defsystem as well. It's the same logic as coerce-class, just possibly without a find-class in there.
Strings do have a certain ugliness to them that keywords don't, so maybe this isn't the best direction for ASDF. But I think making support for this uniform, locking the ASDF package, and encouraging people to use it would solve the namespacing problem.
I see your point, and you're right: strings have the obvious advantage of postponing interaction with the package system, so that the first pass of parsing defsystem can slurp the whole form, without crashing on non-existent package errors. TBH, this seems really unpleasant to me, as a programmer, versus the solution I have proposed. Also, it requires substantial rewriting of existing system definitions. The proposal I have made simply requires introduction of (ASDF:LOAD-SYSTEM <ASDF extension>) in a small number of locations. Even if this wasn't the case, I want to make ASDF simpler, not more complicated. Your proposal will make things more complicated (since arbitrary parts of the defsystem form would have to be re-engineered to accept strings in place of symbols and Do The Right Thing). I think the "ultimate right" answer is to have a new system, YASDF, which has system definition forms that are not processed by READ. Instead, they would be read as files of strings by YASDF, and then parsed by a parser, instead of by READ. Then we wouldn't have these worries about the package system. Ideally, YASDF would also have an explicit design rationale. However, although I feel that YASDF would be more of The Right Thing, writing YASDF is not something I want to spend any hours of my life on. It's just not my thing. Thanks! r
Hi!
In the old days, we would just put (asdf:load-system "extension") That's good. This is declarative enough to be processed by tools like quicklisp.
What goes to side effects, they are just inevitable if we want i) asdf to be extensible ii) some dependency language is used to declarate which system-extension relationships The only way to avoid side effects at asd file processing time is to load all extensions together with asdf, but this is what we want to avoid when we talk about build system, isnt' it?
participants (7)
-
73budden . -
Daniel Kochmański -
Eric Timmons -
Faré -
Faré -
Robert Goldman -
Stelian Ionescu