ASDF components are brittle for backwards compatibility
ASDF checks to make sure all of the initargs are defined when parsing a `defsystem`. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible. I can think of two ways to fix this: 1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value. The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do. I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative. Note that for one's own `system` and `component` subclasses, the set of initargs can be extended without any monkeying around.
ASDF checks to make sure all of the initargs are defined when parsing a `defsystem`. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible.
I can think of two ways to fix this:
1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me.
2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do.
How can ASDF or a developer determine that the unknown property doesn't change the semantics of the DEFSYSTEM, and therefore discarding it would have an effect that's contrary to the author's intention ? ASDF has no syntactic distinction between properties that affect the compilation process and mere annotations like author, licence, etc...
I suppose that we could also add an initarg to tell ASDF to continue such errors silently.
I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative. Note that for one's own `system` and `component` subclasses, the set of initargs can be extended without any monkeying around.
This is a recipe for maintenance nightmares. Let's not go there. -- Stelian Ionescu
On 29 Apr 2021, at 11:42, Stelian Ionescu wrote:
How can ASDF or a developer determine that the unknown property doesn't change the semantics of the DEFSYSTEM, and therefore discarding it would have an effect that's contrary to the author's intention ?
ASDF has no syntactic distinction between properties that affect the compilation process and mere annotations like author, licence, etc...
You can safely assume that the unknown property doesn't change the semantics, because if it does, then the system must require a particular version of ASDF. In other words, this does not introduce a failure mode that isn't already present. If there's a `:use-apple-clang` property introduced in ASDF 29.7 and you didn't put some requirement for ASDF >= 29.7 then your system is busted whether the user discards and continues or not -- the library developer has erred, and nothing about this new extensibility makes the problem worse. But it *does* make it possible to extend ASDF *in safe ways* without compromising backwards compatibility. Note the "in safe ways" -- if we added new properties that change the semantics, we must do something to protect people using older versions. The proposed capability wouldn't change that one way or another. It just makes it possible to extend metadata without excessive pain.
On 29 Apr 2021, at 11:42, Stelian Ionescu wrote:
How can ASDF or a developer determine that the unknown property doesn't change the semantics of the DEFSYSTEM, and therefore discarding it would have an effect that's contrary to the author's intention ?
ASDF has no syntactic distinction between properties that affect the compilation process and mere annotations like author, licence, etc...
You can safely assume that the unknown property doesn't change the semantics, because if it does, then the system must require a particular version of ASDF.
Yes but you can't assume that the system author specified the minimum ASDF version correctly. Developers currently have no way of figuring out what is the actual minimum required version, and version guards are rare and the result of guesswork (including those I put in the libraries I maintain).
In other words, this does not introduce a failure mode that isn't already present. If there's a `:use-apple-clang` property introduced in ASDF 29.7 and you didn't put some requirement for ASDF >= 29.7 then your system is busted whether the user discards and continues or not -- the library developer has erred, and nothing about this new extensibility makes the problem worse.
But it *does* make it possible to extend ASDF *in safe ways* without compromising backwards compatibility.
Note the "in safe ways" -- if we added new properties that change the semantics, we must do something to protect people using older versions. The proposed capability wouldn't change that one way or another. It just makes it possible to extend metadata without excessive pain.
Instead of dedicating much work to ensuring that newer DEFSYSTEMs work with old ASDF versions, we should make it clear that one should just update ASDF frequently. -- Stelian Ionescu
On Thu, 29 Apr 2021, Stelian Ionescu wrote:
Instead of dedicating much work to ensuring that newer DEFSYSTEMs work with old ASDF versions, we should make it clear that one should just update ASDF frequently.
I have no horse in the present race, but I want to add my support to the general sentiment of not support old stuff excessively. /andreas -- "economics is a pseudoscience; the astrology of our time" Kim Stanley Robinson
"Stelian" == Stelian Ionescu <sionescu@cddr.org> writes:
Stelian> Instead of dedicating much work to ensuring that newer Stelian> DEFSYSTEMs work with old ASDF versions, we should make it Stelian> clear that one should just update ASDF frequently. While I don't follow too closely, I think most Lisps that include asdf update asdf fairly frequently. But isn't the biggest problem quicklisp which uses a fairly old version of asdf? (But maybe that's changed now.)
Hi Robert Isn't the :properties slot already there? I just used it (a month ago) to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got the hang of it and it seals the general ASDF machinery from random stuff. Marco On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman <rpgoldman@sift.info> wrote:
ASDF checks to make sure all of the initargs are defined when parsing a defsystem. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible.
I can think of two ways to fix this:
1.
Add a "garbage can" slot to component that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2.
Add a new defsystem parsing error class that is unknown-component-property, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do.
I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative.
Note that for one's own system and component subclasses, the set of initargs can be extended without any monkeying around.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
That slot *is* there, but it is specifically noted as being for ASDF 2 compatibility only, and not to be used. So you are using this at your own risk. Having a `:plist` slot might be a good addition, as well (i.e., both of my 2 proposed solutions) as something lighter weight than my second alternative which is aimed at gracefully being able to extend ASDF's canonical metadata properties. On 29 Apr 2021, at 13:00, Marco Antoniotti wrote:
Hi Robert
Isn't the :properties slot already there? I just used it (a month ago) to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got the hang of it and it seals the general ASDF machinery from random stuff.
Marco
On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman <rpgoldman@sift.info> wrote:
ASDF checks to make sure all of the initargs are defined when parsing a defsystem. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible.
I can think of two ways to fix this:
1.
Add a "garbage can" slot to component that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2.
Add a new defsystem parsing error class that is unknown-component-property, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do.
I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative.
Note that for one's own system and component subclasses, the set of initargs can be extended without any monkeying around.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
Robert, it was you who suggested to use it if I remember correctly. How would it be different from what you just proposed? Marco On Thu, Apr 29, 2021 at 8:07 PM Robert Goldman <rpgoldman@sift.info> wrote:
That slot *is* there, but it is specifically noted as being for ASDF 2 compatibility only, and not to be used. So you are using this at your own risk.
Having a :plist slot might be a good addition, as well (i.e., both of my 2 proposed solutions) as something lighter weight than my second alternative which is aimed at gracefully being able to extend ASDF's canonical metadata properties.
On 29 Apr 2021, at 13:00, Marco Antoniotti wrote:
Hi Robert
Isn't the :properties slot already there? I just used it (a month ago) to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got the hang of it and it seals the general ASDF machinery from random stuff.
Marco
On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman <rpgoldman@sift.info> wrote:
ASDF checks to make sure all of the initargs are defined when parsing a defsystem. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible.
I can think of two ways to fix this:
1.
Add a "garbage can" slot to component that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2.
Add a new defsystem parsing error class that is unknown-component-property, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do.
I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative.
Note that for one's own system and component subclasses, the set of initargs can be extended without any monkeying around.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
On 29 Apr 2021, at 13:38, Marco Antoniotti wrote:
Robert, it was you who suggested to use it if I remember correctly.
How would it be different from what you just proposed?
My suggestion to use the `:properties` slot was not a good one. Looking at the code, this slot is clearly marked as being only for ASDF 2 backwards compatibility (it's hard for me to believe that this is even an issue any more). Hence my suggestion of a new name, `:plist` for use instead. I should probably deprecate the `:properties` slot, since I don't know what it was originally used for... At least generate a warning.
Marco
On Thu, Apr 29, 2021 at 8:07 PM Robert Goldman <rpgoldman@sift.info> wrote:
That slot *is* there, but it is specifically noted as being for ASDF 2 compatibility only, and not to be used. So you are using this at your own risk.
Having a :plist slot might be a good addition, as well (i.e., both of my 2 proposed solutions) as something lighter weight than my second alternative which is aimed at gracefully being able to extend ASDF's canonical metadata properties.
On 29 Apr 2021, at 13:00, Marco Antoniotti wrote:
Hi Robert
Isn't the :properties slot already there? I just used it (a month ago) to add an ASDF DOCUMENT op to HELambdaP. Worked like a charm once I got the hang of it and it seals the general ASDF machinery from random stuff.
Marco
On Thu, Apr 29, 2021 at 6:13 PM Robert Goldman <rpgoldman@sift.info> wrote:
ASDF checks to make sure all of the initargs are defined when parsing a defsystem. This is good for catching errors, but is terrible for extensibility. This means that any attempt to add additional metadata will be backwards incompatible.
I can think of two ways to fix this:
1.
Add a "garbage can" slot to component that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me. 2.
Add a new defsystem parsing error class that is unknown-component-property, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
The second alternative is what I favor. It isn't great, because it will only maintain extensibility going forward, but I think it's the best we can do.
I suppose that we could also add an initarg to tell ASDF to continue such errors silently. I'd be inclined to suggest that this take an ASDF version expression as value, so that the error is quietly ignored only by ASDF versions below that. This means that the property will start to be checked when it has become authoritative.
Note that for one's own system and component subclasses, the set of initargs can be extended without any monkeying around.
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 http://cdac2021.lakecomoschool.org I-20126 Milan (MI) ITALY
On Thu, 2021-04-29 at 18:58 -0500, Robert Goldman wrote:
On 29 Apr 2021, at 13:38, Marco Antoniotti wrote:
Robert, it was you who suggested to use it if I remember correctly.
How would it be different from what you just proposed?
My suggestion to use the `:properties` slot was not a good one. Looking at the code, this slot is clearly marked as being only for ASDF 2 backwards compatibility (it's hard for me to believe that this is even an issue any more).
Then you can safely un-deprecate it. Unless there is a clear reason to think that using that slot would break existing systems, there's no reason to add another slot. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On 29 Apr 2021, at 19:42, Stelian Ionescu wrote:
On Thu, 2021-04-29 at 18:58 -0500, Robert Goldman wrote:
On 29 Apr 2021, at 13:38, Marco Antoniotti wrote:
Robert, it was you who suggested to use it if I remember correctly.
How would it be different from what you just proposed?
My suggestion to use the `:properties` slot was not a good one. Looking at the code, this slot is clearly marked as being only for ASDF 2 backwards compatibility (it's hard for me to believe that this is even an issue any more).
Then you can safely un-deprecate it. Unless there is a clear reason to think that using that slot would break existing systems, there's no reason to add another slot.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Yeah, I am being lazy. I need to see what is being done with the current `properties` slot and whether it can be taken over, or if it is doing some weird classic ASDF thing that I need to stay away from. I will post more once I have groveled over the code.
"Robert Goldman" <rpgoldman@sift.info> writes:
2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
How practical is this one actually? The initarg checking is done by reinitialize/make-instance. I just tested on SBCL and got an internal error (SB-PCL::INITARG-ERROR) that was then intercepted by ASDF and turned into the catch-all load-system-definition-error. I don't think the spec specifies the actual error signaled on invalid initargs, so we'd have to figure out what each implementation's internal error is (and hope they all include enough information to determine which initargs were invalid).
1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me.
For the same reasons as above I'm not sure this one is practical either. We just have no way of knowing which initargs are valid or not. I think the best we can do at the moment is a plist slot that stores all initargs, invalid or not. Then if the system writer really wants it to work on older ASDFs, they can add :allow-other-keys t to their system to disable the built-in initarg checking (completely at their own peril), but still have their extra metadata saved. It's definitely worth considering for ASDF 4, separating all the "metadata" that can't affect build semantics into a :metadata initarg and being lax by default on what's included there. -Eric
1. If you're going to add :plist, you might as well un-deprecate :properties indeed. 2. The "idea" for extension is that the system class from :class would tell help inform what the system does, and, possibly, how to parse the defsystem. Of course, that only helps you if you define a new class, not if you have a new version of the "same" class that is defined in a new version of ASDF... unless somehow a new version of ASDF has several system classes, such as system3.5 vs system 3.4. 3. Maybe indeed ASDF would usefully include some form or initarg that ensures a recent-enough ASDF is available, early enough in the build to provide a good error message... but of course, you'd have to guard usage of this form or initarg with #+asdf3.4 since it isn't defined yet... so defining such a form now will only lead to pain reduction after at least 2 years. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Sex without love is an empty experience, but, as empty experiences go, it's one of the best. — Woody Allen On Thu, Apr 29, 2021 at 9:53 PM Eric Timmons <etimmons@mit.edu> wrote:
"Robert Goldman" <rpgoldman@sift.info> writes:
2. Add a new `defsystem` parsing error class that is `unknown-component-property`, raise it when an unknown property is encountered and allow the user to continue, discarding the initarg and accompanying value.
How practical is this one actually? The initarg checking is done by reinitialize/make-instance. I just tested on SBCL and got an internal error (SB-PCL::INITARG-ERROR) that was then intercepted by ASDF and turned into the catch-all load-system-definition-error.
I don't think the spec specifies the actual error signaled on invalid initargs, so we'd have to figure out what each implementation's internal error is (and hope they all include enough information to determine which initargs were invalid).
1. Add a "garbage can" slot to `component` that will be filled with a property list, and allow programmers to do whatever they want here. This doesn't seem great to me.
For the same reasons as above I'm not sure this one is practical either. We just have no way of knowing which initargs are valid or not.
I think the best we can do at the moment is a plist slot that stores all initargs, invalid or not. Then if the system writer really wants it to work on older ASDFs, they can add :allow-other-keys t to their system to disable the built-in initarg checking (completely at their own peril), but still have their extra metadata saved.
It's definitely worth considering for ASDF 4, separating all the "metadata" that can't affect build semantics into a :metadata initarg and being lax by default on what's included there.
-Eric
participants (7)
-
Andreas Davour -
Eric Timmons -
Faré -
Marco Antoniotti -
Raymond Toy -
Robert Goldman -
Stelian Ionescu