Anton Vodonosov wrote:

22.01.2014, 02:47, "Robert P. Goldman" rpgoldman@sift.info:

...

Fixes for this error should look something like this:

(defclass my-operation (OPERATION) ...)

should turn into something like

(defclass my-operation (#-asdf3 OPERATION #+asdf3 DOWNWARD-OPERATION) ...)

[...]

...

FWIW, *all* the OPERATION-redefining systems will be broken. This is intentional. All such systems need reexamination, and potentially patching.

Have you considered leaving old asdf:operation as a deprecated backward compatibility stub defined as

(defclass operation (downward-operation) ())

And the hierarchy base class will be called for example base-operation:

(defclass base-operation () )

(defclass downward-operation (base-operation) ) (defclass upward-operation (base-operation) ) (defclass sideway-operation (base-operation) ) (defclass selfward-operation (base-operation) ) (defclass non-propagating-operation (base-operation) )

Best regards,

• Anton

I did consider this, and Fare indicated that this was not an acceptable alternative. I believe the upgrading might be quite awkward if we did this (although he can speak to this more clearly). We also considered trying to trap the definition of OPERATION subclasses, but Pascal Costanza, who I believe to be the leading expert on practical use of the MOP, indicated that would be impractical.

The current expedient is a way to "fail loud," since we have no way of reliably detecting only the cases that need fixing.

However, I can speak to the difficulty of identifying the need to revise these definitions based on personal experience at SIFT. The failures that you get from missing dependencies like this can be *very* difficult to trace back to the ASDF change. In our case, the only visible error was in a bash script that post-processed the results of a user-defined OPERATION! Indeed, the people directly encountering the bug didn't even know to look at TRAVERSE, much less could they figure out the need to revise an ASDF system definition.

Best, R

On Tue, Jan 21, 2014 at 8:45 PM, Robert P. Goldman rpgoldman@sift.info wrote:

...

...

FWIW, *all* the OPERATION-redefining systems will be broken. This is intentional. All such systems need reexamination, and potentially patching.

For the record, I still think this intentional breakage is a bad idea. In the short term, it causes a known inconvenience for users of existing software. In the long run, it introduces a new class of dubious value without MOP enforcement (and MOP enforcement might or might not be achievabe portably, even after importing a good chunk of closer-mop). All that for the sake of hypothetical software by people who have been wilfully out of the conversation for over a year.

...

Have you considered leaving old asdf:operation as a deprecated backward compatibility stub defined as

(defclass operation (downward-operation) ())

And the hierarchy base class will be called for example base-operation:

(defclass base-operation () )

I did consider this, and Fare indicated that this was not an acceptable alternative. I believe the upgrading might be quite awkward if we did this (although he can speak to this more clearly). We also considered trying to trap the definition of OPERATION subclasses, but Pascal Costanza, who I believe to be the leading expert on practical use of the MOP, indicated that would be impractical.

Well, that would break some non-trivial amount of code that defines methods specialized on OPERATION and assumes that's the base of the hierarchy so the method works on everything. And that code is somewhat harder to automatically detect at runtime.

But the current patch also breaks code, so, oh well.

The current expedient is a way to "fail loud," since we have no way of reliably detecting only the cases that need fixing.

However, I can speak to the difficulty of identifying the need to revise these definitions based on personal experience at SIFT. The failures that you get from missing dependencies like this can be *very* difficult to trace back to the ASDF change. In our case, the only visible error was in a bash script that post-processed the results of a user-defined OPERATION! Indeed, the people directly encountering the bug didn't even know to look at TRAVERSE, much less could they figure out the need to revise an ASDF system definition.

Would that patch have caught your issue at SIFT?

PS: I used to avoid committing spaces at end of line. I see that you have some.

PPS: Anton reports no error. Is that because the classes that are broken are never invoked during the loading of those libraries, only during the using of them, but no code in quicklisp actually uses them?

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Fraud is the homage that force pays to reason. — Charles Curtis

On Wed, Jan 22, 2014 at 1:19 AM, Anton Vodonosov avodonosov@yandex.ru wrote:

22.01.2014, 09:16, "Faré" fare@tunes.org:

...

PPS: Anton reports no error. Is that because the classes that are broken are never invoked during the loading of those libraries, only during the using of them, but no code in quicklisp actually uses them?

Maybe there are no such classes? Why would people extend operation?

There are many such classes. Extending operation is how you add, i.e. cffi grovel and cffi wrappers support, automatic documentation generation, dependency groveling, tarball generation, cross-compilation, and more.

As for signalling the problem, I agree with Robert in the sense that if such users exists, it is friendly to save them from debugging an issue like this.

If not error, then at least very very bold warning, or cerror

It's nice to notify users of problems... but what if this is done by introducing new problems for sure, whereas these users are hypothetical, especially since they are supposed to have extended ASDF yet been unaware of change introduced one year ago in ASDF?

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org As for poverty, no one need be ashamed to admit it: the real shame is in not taking practical measures to escape from it. — Perikles

...

It's nice to notify users of problems... but what if this is done by introducing new problems for sure, whereas these users are hypothetical, especially since they are supposed to have extended ASDF yet been unaware of change introduced one year ago in ASDF?

I understand your concern, but I wish you would stop using the term "hypothetical" here. The concern is not hypothetical, since I have encountered it myself. Handling the upgrade cost us at least four person days, only terminating when I thought to ask you a related question.

I'm sorry about that, and sad I didn't advertise the issue loud enough (I tried, but the signal probably got drowned into all the other issues that had to be addressed.)

Still, whether any *remaining* code afterwards remains is hypothetical. One (costly) alert after a year — how many other alerts in the years to come? What are the odds someone went into deep ASDF extension hacking, but never bothered to check the asdf-devel mailing-list or be subscribed to it, nor to upgrade his ASDF and test his software? Is such person worth some backward incompatible change? Fifty bucks here say this person doesn't exist and/or will never touch Lisp again to notice the breakage and complain about it.

While it is desirable that they do so, expecting people to track ASDF development is not a reasonable expectation, any more than it is reasonable to expect people to track developments in make.

If make had an extension mechanism, it would be reasonable for authors of extensions to touch base within a year after a major version change.

Furthermore (although I have an as yet uncommitted partial draft), the current manual provides no support for programmers grappling with such problems.

The manual is severly lacking. I improved it tremendously, with your help, but it's still quite incomplete and less useful than it could be. Compatibility with a moving target make it harder to write the manual in a simple understandable style. I believe references to compatibility with ASDF 1 can be removed; hopefully, after LispWorks and Quicklisp adopt ASDF 3, notes for compatibility with ASDF 2 can soon be retired, too.

I am willing to do what I can to minimize the dislocation by trying methods other than the alternative I have pushed to the repository.

The one thing I am not willing to do is to release another ASDF 3 without a programmatic warning to users of new OPERATION subclasses that the semantics of OPERATION has changed from "implicitly downward-propagating dependencies" to "not propagating any dependencies."

NB: not just downward, but also sideway, and even selfward (for the input-files method). All used to be implicit. None is now.

I would like us to focus on the issue of minimizing the number of these warnings that are false positives, rather than continuing to argue for their complete removal. E.g., in cases where we know that a new OPERATION subclass is safe (see Faré's earlier survey of quicklisp libraries), we could whitelist it. But I don't have time for further discussion of shipping a new ASDF without some warning.

I was trying to avoid whitelisting, especially since older variants of the same libraries might be in the wild, and incompatible.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If debugging is the process of removing bugs, then programming must be the process of putting them in. — Dijkstra

22.01.2014, 09:51, "Anton Vodonosov" avodonosov@yandex.ru:

22.01.2014, 09:16, "Faré" fare@tunes.org:

...

...

...

Have you considered leaving old asdf:operation as a deprecated backward compatibility stub defined as

(defclass operation (downward-operation) ())

And the hierarchy base class will be called for example base-operation:

(defclass base-operation ()    )

I did consider this, and Fare indicated that this was not an acceptable alternative.  I believe the upgrading might be quite awkward if we did this (although he can speak to this more clearly).  We also considered trying to trap the definition of OPERATION subclasses, but Pascal Costanza, who I believe to be the leading expert on practical use of the MOP, indicated that would be impractical.

Well, that would break some non-trivial amount of code  that defines methods specialized on OPERATION and assumes  that's the base of the hierarchy so the method works on everything.  And that code is somewhat harder to automatically detect at runtime.

You are right. Then how about this:

Old code knows only 5 classes

operation   compile-op -> operation   load-op -> operation   load-source-op -> operation   test-op -> operation

Here "->" means "inherits from"

If new ASDF will provide operation as a back compatibility stub   (defclass operation (downward-operation) ()) and ensure the compile-op, load-op, load-source-op, test-op are all inherited from operation, then the old code assumptions will be satisfied.

I see a problem.

Only new compile-op, load-op, load-source-op are inherided from downward-operation, so they would be compatible with the proposed stub (defclass operation (downward-operation) ())

But new test-op inherits only from selfward-operation, without downward-operation. So inheriting it from the stub operation will introduce downward-operation and change test-op semantics.

I don't know if that's a stopper the idea.

Attila Lendvai wrote:

...

For the record, I still think this intentional breakage is a bad idea. In the short term, it causes a known inconvenience

FTR, i pulled asdf when i saw the request for testing, tried to build our stuff, saw that iolib failed (a dependency), and then i decided to git reset and stay there until others struggle through this, even though we also have some operation subclasses.

there are situations when the least inconvenient and most productive way of breaking backwards compatibility is by starting with a clean slate.

Thank you. The PROCESS-OP in iolib will fail with the new test. There are two possible solutions:

1. If you want the old behavior, where PROCESS-OP on a system should cause PROCESS-OP to be applied to its children, make PROCESS-OP inherit from DOWNWARD-OPERATION:

(defclass process-op (#-asdf3 asdf:operation #+asdf3 asdf:downward-operation) ...)

2. If you want the new behavior, where PROCESS-OP to a system does not cause any dependent operations to occur, then make PROCESS-OP inherit from asdf:non-propagating-operation. Perhaps a solution will be found that does not require NON-PROPAGATING-OPERATION, but after prolonged discussion, I am pessimistic.

...

PS: I used to avoid committing spaces at end of line. I see that you have some.

emacs has nice tools to show lose whitespace (see the varaible whitespace-style and there are some faces, too).

it even has automatic cleanup, but that can be too intrusive when working on other people's code.

Yes, I turned these off in my editor. I ended up with too many distracting red blocks on my screen, and then too many spurious diffs when I would commit automatic whitespace cleanup.

I don't have a good solution to this problem. Maybe before we do the final release, I could do a whitespace cleanup pass over all the source, and make a single commit.

Best,

r

Attila Lendvai wrote:

...

Yes, I turned these off in my editor. I ended up with too many distracting red blocks on my screen, and then too many spurious diffs when I would commit automatic whitespace cleanup.

yep. i turned auto fixing off myself, and i have a very slight red tint on the whitespace mistakes.

thanks for the mini-doc! i'm doing one more trial.

Probably while we wait to see if anyone can come up with a less disruptive solution than mine, it will be best for you to park your fix on a topic branch.

Unfortunately, I don't see such a solution on the horizon: Pascal has demonstrated to my satisfaction that we cannot trap the *definition* of new OPERATION subclasses, only their instantiation. Faré has similarly ruled out deprecating OPERATION itself.

Best,

r

Anton Vodonosov wrote:

22.01.2014, 20:11, "Robert P. Goldman" rpgoldman@sift.info:

...

Unfortunately, I don't see such a solution on the horizon: Pascal has demonstrated to my satisfaction that we cannot trap the *definition* of new OPERATION subclasses, only their instantiation. Faré has similarly ruled out deprecating OPERATION itself.

I don't think that preserving OPERATION semantics is really ruled out. Lets consider it a little bit more?

I am willing to do so. Faré is really the one who needs to judge this. See my remarks below.

Is it true that old ASDF:OPERATION is semantically equivalent to the new DOWNWARD-OPERATION? If yes, the proposal I made earlier looks appropriate:

OPERATION inherit from DOWNWARD-OPERATION COMPILE-OP inherit from OPERATION LOAD-OP inherit from OPERATION LOAD-SOURCE-OP inherit from OPERATION

I think if we were to do this, we would need to add a BASIC-OPERATION that would sit above the other operations and that would serve half of the purpose of the current OPERATION (providing a common root for all of the classes).

I think the danger is that existing methods that dispatch on OPERATION, intending to affect *all* operations would now be broken. They would have to be moved to BASIC-OPERATION. Repairing that might involve modifying more code than the current solution.

If we make so, these operations are backward compatible and at the same time fit the new ASDF 3 design.

The only relatively small issue we have is with TEST-OP. ASDF 3 wants TEST-OP to be just SELFWARD-OPERATION, while previous semantics of TEST-OP was inherited from old OPERATION, i.e. equal to the new DOWNWARD-OPERATION.

I think it is a small issue and there are number of ways solving it:

1. Make TEST-OP just SELFWARD-OPERATION, thus breaking compatibility, but only for the code depending which rely on TEST-OP to have downward semantics. It is a smaller compatibility break than breaking any OPERATION-extending code. Probably there is no code at all which relies on TEST-OP being downward.

I believe that to be correct. The only examples of TEST-OP that I know of involves a test system, with files that define tests according to some library, and a PERFORM method on TEST-OP at the system level that invokes some function in the test framework (e.g., fiveam:run!) to run the tests that the component files have defined.

When I say "I believe that to be correct," credit is due to Faré for thinking this true and convincing me that SELFWARD is correct and DOWNWARD is not correct.

2. Accept that new TEST-OP is a DOWNWARD-OPERATION - maybe it compromises new ASDF 3 design a bit, but we are fully backward compatible.

I believe that the way the existing code for TEST-OP was made to work in the presence of downward dependency propagation was to define no-op methods something like this:

(defmethod perform ((op test-op) c) (values))

so that when dependencies were propagated down to individual files, they would do nothing.

Your solution 2 would preserve these.

3. Do with TEST-OP the same I propose for OPERATION - make it a backward compatibility stub, a DOWNWARD-OPERATION, but also introduce new ASDF3:TEST-OP which is a SELFWARD-OPERATION

I would prefer, in order, your solution 1, 2, 3.

On Wed, Jan 22, 2014 at 12:40 PM, Anton Vodonosov avodonosov@yandex.ru wrote:

I don't think that preserving OPERATION semantics is really ruled out. Lets consider it a little bit more?

Is it true that old ASDF:OPERATION is semantically equivalent to the new DOWNWARD-OPERATION? If yes, the proposal I made earlier looks appropriate:

OPERATION inherit from DOWNWARD-OPERATION COMPILE-OP inherit from OPERATION LOAD-OP inherit from OPERATION LOAD-SOURCE-OP inherit from OPERATION

If we make so, these operations are backward compatible and at the same time fit the new ASDF 3 design.

It's not backward compatible with systems that define methods on operation, and expect the method to be always calls for all operations. I admit I haven't kept track of how many of them there were while auditing quicklisp; a few, still, that will have to be updated — making the exercise self-defeating as a way to detect old code.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Two possibilities exist: Either we are alone in the Universe or we are not. Both are equally terrifying. — Arthur C. Clarke

Anton Vodonosov wrote:

...

...

It's not backward compatible with systems that define methods on operation, and expect the method to be always calls for all operations.

But does "all operations" for the old code means only: operation, compile-op, load-op, load-source-op, test-op and any descendants of those? Because old code knew only this hierarchy.

I think what Faré is pointing out is that one could build, for example, an introspection library by adding for example, PERFORM methods that would dispatch on OPERATION and that would catch *all* PERFORMS and write a log or something like that.

If we no longer make OPERATION the root of the hierarchy then such introspection libraries will no longer work.

While Faré and I differ on how active we should be about alerting (he might say "annoying" ;->) programmers, he has convinced me that the solution will have to involve leaving OPERATION as the root class.

In that case, library programmers will simply have to adjust to the new semantics, rather than ASDF fixing its semantics to be backward compatible.

I certainly hope I am wrong about this! But I fear that I am not.

Best, Robert

Faré wrote:

On Wed, Jan 22, 2014 at 2:54 PM, Robert P. Goldman rpgoldman@sift.info wrote:

...

I think what Faré is pointing out is that one could build, for example, an introspection library by adding for example, PERFORM methods that would dispatch on OPERATION and that would catch *all* PERFORMS and write a log or something like that.

If we no longer make OPERATION the root of the hierarchy then such introspection libraries will no longer work.

I was more thinking about people defining methods on operation.

I *think* we were trying to say the same thing. Maybe not? I was giving PERFORM as an example of a generic function on which we could define methods for the OPERATION type.

I don't know what sorts of code would do that: introspection seemed like a logical example.

Are we talking about something different?

less -p 'defmethod.* operation)' $(grep -il 'defmethod.* operation)' ~/quicklisp/dists/quicklisp/software/**/*.{asd,lisp}) Reveals 9 files beside copies of ASDF that define methods on operation. Some of them may or may not be affected in practice by making operation not the root of the hierarchy. POIU also does it, though if you change the base class incompatibly, I can update POIU.

ITA's QRes also defines methods on operation (as you can see in the parts that were open sourced); maybe other proprietary systems do, too. Once again, if anyone is extending ASDF in proprietary system (or just ones not on quicklisp), he'd be well-advised to be on this mailing-list.

Maybe that's true, but we have no way of enforcing it. Hell, we don't even have the means to *suggest* it -- the Catch-22 is that we can only suggest subscribing to ASDF-devel... on ASDF-devel! ;-)

cheers, r

22.01.2014, 23:54, "Robert P. Goldman" rpgoldman@sift.info:

I think what Faré is pointing out is that one could build, for example, an introspection library by adding for example, PERFORM methods that would dispatch on OPERATION and that would catch *all* PERFORMS and write a log or something like that.

If we no longer make OPERATION the root of the hierarchy then such introspection libraries will no longer work.

Ah, I see.

A formalist excuse for such cases may be "if you want to intercept all PERFORMS, you should specialize your OPERATE method on T. If you specialize on OPERATION, then you restrict the method to OPERATION" :)

If speak seriously, then I agree, the solution with OPERATION being backward comp stub inherited from DOWNWARDS-OPERATION, SELFWARD-OPERATION, etc is not absolute.

Another, very artificial example, is if someone examines class hierarchy programmatically, to ensure asdf:operation is the root:

(assert (equal (mapcar #'find-class '(asdf:operation standard-object t)) (class-precedence-list (find-class 'asdf:operation))))

There is some chance that clients will break when we introduce new root BASE-OPERATION and leave OPERATION as a back comp stub.

The current ASDF3 solution where OPERATION becomes higher level root, loosing part of its semantics also has some chance to break clients.

Then the solution with lower chance to break clients is better.

But I can't say how probable the "OPERATION is the root" solution to break clients, as far as I now understand some of OPERATION-extending systems may remain working, right? Would cffi-grovel and other use cases provided by Fare work?

Sorry if I am making too much noise on the list, I become curious why it is so problematic and it is an interesting design exercise. Also you could have rejected the "new BASE-OPERATION root" solution early, before the hard difficulties with other alternatives were found; that happens some time. So we now reevaluated it.

Best regards, - Anton

On Wed, 2014-01-22 at 21:27 +0600, Attila Lendvai wrote:

...

For the record, I still think this intentional breakage is a bad idea. In the short term, it causes a known inconvenience

FTR, i pulled asdf when i saw the request for testing, tried to build our stuff, saw that iolib failed (a dependency), and then i decided to git reset and stay there until others struggle through this, even though we also have some operation subclasses.

FWIW, on my machine IOlib HEAD compiles using ASDF HEAD.

Anton Vodonosov wrote:

22.01.2014, 04:25, "Faré" fahree@gmail.com:

...

I'm worried about this patch. Can you get it tested with cl-test-grid before release? I fear there are many currently working systems that will break due to this change.

Anton, can you help?

Started tests for 3.1.0.53.

SBCL will be ready in few hours, then CCL, then others. But I think the results for a single lisp will show if any OPERATION-redefining systems are broken.

If I be asleep when the results are ready then I will report tomorrow.

Best regards,

• Anton

FWIW, *all* the OPERATION-redefining systems will be broken. This is intentional. All such systems need reexamination, and potentially patching.