James Anderson has asked for a "concise statement of the issue at hand."
Faré has had to change the semantics of the OPERATION class. Previously, the OPERATION class would automatically trigger dependencies downward (notably, if you did an OPERATION on a system, then you had to do that OPERATION as well on that OPERATION's children), "leftward," and onto itself ("selfward").
The implementation of this left holes, and also caused problems for extensions (e.g., for TEST-OP, one typically had to define a default PERFORM that did nothing, since testing a system necessarily tested all of the files).
To fix this, Faré had to add new mixin classes, DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, and SELFWARD-OPERATION.
In order to preserve the integrity of the code, OPERATION remained as the superclass of all operation objects. But its semantics has changed.
This is not a problem for anyone with a "vanilla" ASDF system. The only programmers who are vulnerable to this change are those who have defined their own subclasses of OPERATION (or those who use libraries whose programmers have defined subclasses of OPERATION).
For those cases, I have pushed a change that checks to make sure that all operations that are instantiated are subclasses of either DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, SELFWARD-OPERATION, or a new NON-PROPAGATING-OPERATION. If this run-time test fails, an error will be signaled and the programmer will have to fix the offending OPERATION subclass's definition.
On Thu, 23 Jan 2014, Robert P. Goldman wrote:
Fare has had to change the semantics of the OPERATION class. Previously, the OPERATION class would automatically trigger dependencies downward, ... "leftward," and onto itself ("selfward").
...
In order to preserve the integrity of the code, OPERATION remained as the superclass of all operation objects. But its semantics has changed.
...
For those cases, I have pushed a change that checks to make sure that all operations that are instantiated are subclasses of either DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, SELFWARD-OPERATION, or a new NON-PROPAGATING-OPERATION. If this run-time test fails, an error will be signaled and the programmer will have to fix the offending OPERATION subclass's definition.
Backwards compatibility is a really important goal for low-level infrastructure. It provides the stability needed for others to trust building on a framework. While the proposed changes do affect a minority of the userbase, they do "punish" past efforts to use the advanced API features...
Would it be possible for your test to inject the old default behavior into such classes?
I'm not thinking of any clean CLOS or MOP tricks, but a direct implementation might not be too hard.
For example, add an API-VERSION field to the OPERATION class. By default, it will be uninitialized, signifying :asdf2 behavior. New subclasses could explicitly set it to :asdf3. The mixins or a variant of the subclass test might also set it to :asdf3. Then the dependency traversal code could maintain a check for this field along with the new mixins.
If we are unable to find a transparent solution, then a secondary goal is to find a solution that allows new libraries the option to support both the old and new ASDF semantics. Since ASDF can upgrade in-place, I believe this rules out reader conditionals. Does ASDF have a hook API that notifies libraries of the possible need to re-initialize their ASDF extensions when it reloads?
Later, Daniel
Daniel Herring wrote:
Would it be possible for your test to inject the old default behavior into such classes?
I'm not thinking of any clean CLOS or MOP tricks, but a direct implementation might not be too hard.
For example, add an API-VERSION field to the OPERATION class. By default, it will be uninitialized, signifying :asdf2 behavior. New subclasses could explicitly set it to :asdf3. The mixins or a variant of the subclass test might also set it to :asdf3. Then the dependency traversal code could maintain a check for this field along with the new mixins.
So the idea would be that instead of signaling an error, we would check for API-VERSION :asdf3 and if we don't apply it, we would do something like
(change-class obj 'BACKWARDS-COMPATIBLE-OPERATION)
?
where we have
(defclass backwards-compatible-operation (sideway-operation downward-operation selfward-operation) ())
I think that would *almost* work. We'd have to tweak the way SELFWARD-OPERATION is used, but that should be feasible.
We could even do that using the existing superclass check (backpatch unless we find one of the new OPERATION subclasses as parent), rather than having to introduce API-VERSION.
I am open to further discussion of this option.
best, r
Well, this should teach me to think more than 2.4 seconds before responding to emails.
I'm afraid my proposed CHANGE-CLASS solution won't work. The problem is that we would be required to change the parent classes of a new OPERATION to remove OPERATION and add BACKWARDS-COMPATIBLE-OPERATION, and this is precisely what Pascal has explained cannot be done portably, or at least not without a very big new body of compatibility code, because of the non-standard nature of the MOP.
So you are suggesting that we modify the traversal code to check for non-updated OPERATION classes, and then replicate a fixed version of the old logic?
I'm willing to entertain this as a suggestion, but it seems to me very likely to involve adding a large layer of cruft to ASDF. The error check involves only 10 lines. It also has the virtue of forcing people to fix their systems, instead of moving responsibility for their continued correct functioning to the shoulders of the not-very-large ASDF team.
Do you see a way around these issues?
Best, Robert
I have not been following every last detail of this conversation, so please forgive me if what I'm about to suggest is a terrible idea.
It appears that Robert is concerned about breaking ASDF files containing code that defines classes that inherit from OPERATION. I have written code like this and had to do some serious searching before I found an example to follow, so I personally believe that such code is rare and not worth worrying too much about breaking.
However, can't we find and fix 95% of the breakage by running "grep 'defclass.*operation' *.asd" on all the Quicklisp libraries? That would find my PROTO-TO-LISP class, which is presumably now broken. In addition to Quicklisp libraries, my Slurp code can check out several hundred other Lisp packages from web repositories. I could run the same check on those.
By the way, regarding my PROTO-TO-LISP class. I want to inherit from DOWNWARD-OPERATION, right?
Bob
On Fri, Jan 24, 2014 at 11:23 AM, Robert Goldman rpgoldman@sift.net wrote:
Well, this should teach me to think more than 2.4 seconds before responding to emails.
I'm afraid my proposed CHANGE-CLASS solution won't work. The problem is that we would be required to change the parent classes of a new OPERATION to remove OPERATION and add BACKWARDS-COMPATIBLE-OPERATION, and this is precisely what Pascal has explained cannot be done portably, or at least not without a very big new body of compatibility code, because of the non-standard nature of the MOP.
So you are suggesting that we modify the traversal code to check for non-updated OPERATION classes, and then replicate a fixed version of the old logic?
I'm willing to entertain this as a suggestion, but it seems to me very likely to involve adding a large layer of cruft to ASDF. The error check involves only 10 lines. It also has the virtue of forcing people to fix their systems, instead of moving responsibility for their continued correct functioning to the shoulders of the not-very-large ASDF team.
Do you see a way around these issues?
Best, Robert
Robert Brown wrote:
I have not been following every last detail of this conversation, so please forgive me if what I'm about to suggest is a terrible idea.
[..snip..]
All of the discussion I have snipped has been covered before, and fits under my declining to discuss further. I apologize, but I simply don't have any more time to discuss it further. Briefly: the extent of the systems I intend ASDF to support is not limited to the set of systems in quicklisp.
Please, no more posts saying "we could just fix all of the libraries in quicklisp." Faré has been saying this, if not until he's blue in the face, certainly until *I* am blue in the face.
By the way, regarding my PROTO-TO-LISP class. I want to inherit from DOWNWARD-OPERATION, right?
I'm not sure. Two possibilities:
1. You will OPERATE PROTO-TO-LISP on your system. In this case, yes, because you want all the components to undergo this operation. You may want SIDEWAY-OPERATION, as well, if PROTO-TO-LISP on one file requires PROTO-TO-LISP on :Depends-on files.
2. You will OPERATE PROTO-TO-LISP only as a predecessor to COMPILE-OP and LOAD-OP on the file. In this case a NON-PROPAGATING-OPERATION is sufficient.
Cheers, r
On Fri, Jan 24, 2014 at 12:21 AM, Daniel Herring dherring@tentpost.com wrote:
For those cases, I have pushed a change that checks to make sure that all operations that are instantiated are subclasses of either DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, SELFWARD-OPERATION, or a new NON-PROPAGATING-OPERATION. If this run-time test fails, an error will be signaled and the programmer will have to fix the offending OPERATION subclass's definition.
Backwards compatibility is a really important goal for low-level infrastructure. It provides the stability needed for others to trust building on a framework. While the proposed changes do affect a minority of the userbase, they do "punish" past efforts to use the advanced API features...
Would it be possible for your test to inject the old default behavior into such classes?
The issue at stake is that the old ASDF2 behavior was a bug. A deep, essential bug, that required massive refactoring and cleanup.
The downward and sideway propagation of operations was baked into ASDF1's TRAVERSE algorithm, which went essentially unchanged in ASDF2, though its implementation was gradually broken down from one giant spaghetti function to plenty of small ones, allowing rpgoldman and I to make sense of it and fix many bugs.
However, this propagation was a bug. Not only is this propagation an expensive nuisance for many operations, necessitating various workarounds (like perform methods that do nothing), some essential operations need to propagate UPWARD, not downward: PREPARE-OP, PREPARE-SOURCE-OP, etc. Because ASDF1 couldn't express this propagation, these operations were missing, and the required dependencies were expressed implicitly through the order of the TRAVERSE algorithm, which didn't quite work in incremental builds across systems, though this bug was largely hidden by the even more glaring bug that ASDF, after earlier DEFSYSTEMs, completely failed to propagate timestamps from one action to the next.
ASDF3 thus involved two complete rewrites of TRAVERSE, which had already been completely revamped since ASDF1 (though faithfully preserving its semantics). Now, to fix this design bug in ASDF2, from ASDF1, from MK-DEFSYSTEM, from DEFSYSTEM, it was just not possible to be 100% compatible. Being 100% compatible meant keeping the bug. Defeats the purpose. Whoever wrote extensions that relied on the bug would suffer, so that everyone else could be rid of the bug. That's not the first time this happens, though this time, it's the breakage will be subtle, and hard to debug for who isn't asking me for help. So I fixed everything in Quicklisp and at work. But I can't fix what I don't have access to.
Now comes Robert, who has been badly bitten, for the breakage was subtle enough that it took him a lot of time before he figured out I was the culprit. Oops. My bad. Then he wonders how to warn other people who like him haven't yet upgraded to ASDF3, and have proprietary code not on Quicklisp that includes ASDF extensions.
While I sympathize, I conjecture that the set is empty, yet that his fix will cause breakage of its own. I'm actually surprised that Anton didn't find more breakage when Robert introduced an error at instantiation time for operation classes that don't inherit from NON-PROPAGATING-OPERATION or at least one of the propagating classes. I strongly suspect that no one in quicklisp uses these ASDF extensions, and so that the patch does break software, just not the building of this software. But I'm a bit too lazy to try right now.
Robert or Anton: could you grep the archives for my previous audit, and try *using* some of the operations defined in Quicklisp? This definitely needs be done before the release.
I'm not thinking of any clean CLOS or MOP tricks, but a direct implementation might not be too hard.
For example, add an API-VERSION field to the OPERATION class. By default, it will be uninitialized, signifying :asdf2 behavior. New subclasses could explicitly set it to :asdf3. The mixins or a variant of the subclass test might also set it to :asdf3. Then the dependency traversal code could maintain a check for this field along with the new mixins.
If we are unable to find a transparent solution, then a secondary goal is to find a solution that allows new libraries the option to support both the old and new ASDF semantics. Since ASDF can upgrade in-place, I believe this rules out reader conditionals. Does ASDF have a hook API that notifies libraries of the possible need to re-initialize their ASDF extensions when it reloads?
Backward compatibility is key. If we don't care about backward compatibility, then there's no problem anyway. And if we do, then your solution has the issue that since existing client code doesn't use this versioning technique, we have to either assume it all working (in which case there is nothing to do and Robert can revert his patches), or all non-working (in which case you break all the code that does work, which makes the patch a regression rather than an advancement.
Regards,
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Don't worry about results. You can't wish them into existence. Focus on the process. You have decision power on it. And it will take care of results.
Faré wrote:
Robert or Anton: could you grep the archives for my previous audit, and try *using* some of the operations defined in Quicklisp? This definitely needs be done before the release.
Unfortunately, I am not a quicklisp user. It does not fit the development methods of my company.
Doubly unfortunately, I do not have the time to start using quicklisp, certainly not before the end of February.
So someone else will have to step up and take this on, if it is going to happen.
I am less convinced than you are that this is critical. It is not the job of ASDF to fix every program that uses ASDF, any more than the maintainers of make need to fix every C program.
The new ASDF will break systems that subclass OPERATION. Those libraries' maintainers will have to look at their code, based on the error message, and see what needs to be done. For the vast majority of them, five minutes work will suffice. For those that take more... well, as you point out, the bug fix had to be made, and I stand ready to help those who need assistance adjusting.
This will be my last word on the desirability of this change. ASDF will go out signaling this error; there will be a different mechanism provided to warn programmers who have subclassed OPERATION; or there will be a new lead maintainer.
I will be happy to discuss alternative mechanisms. I will not respond to any further correspondence discussing the need for warnings to programmers. I am happy to step aside if someone else wishes to take over.
Yours very sincerely, Robert
The new ASDF will break systems that subclass OPERATION. Those libraries' maintainers will have to look at their code, based on the error message, and see what needs to be done. For the vast majority of them, five minutes work will suffice. For those that take more... well, as you point out, the bug fix had to be made, and I stand ready to help those who need assistance adjusting.
It's good to signal an error or a warning for this case.
It would have been better if asdf:operation would have kept its old semantics, maybe with a deprecation warning when it's used, and that the new semantics would have been provided by, say, asdf3:operation. People who see the warning and care about keeping things up to date could react to the warning and move from asdf:operation to asdf3:operation, which is not much more work than what is necessary now. People who care less (maybe because other things are more important - asdf is not the center of the universe after all) would not need worry, because everything would work as before, even in a setting where both asdf2 and asdf3 semantics would be needed side by side.
Pascal
Pascal Costanza wrote:
The new ASDF will break systems that subclass OPERATION. Those libraries' maintainers will have to look at their code, based on the error message, and see what needs to be done. For the vast majority of them, five minutes work will suffice. For those that take more... well, as you point out, the bug fix had to be made, and I stand ready to help those who need assistance adjusting.
It's good to signal an error or a warning for this case.
It would have been better if asdf:operation would have kept its old semantics, maybe with a deprecation warning when it's used, and that the new semantics would have been provided by, say, asdf3:operation. People who see the warning and care about keeping things up to date could react to the warning and move from asdf:operation to asdf3:operation, which is not much more work than what is necessary now. People who care less (maybe because other things are more important - asdf is not the center of the universe after all) would not need worry, because everything would work as before, even in a setting where both asdf2 and asdf3 semantics would be needed side by side.
I believe that the issue was that Faré couldn't preserve both:
1. OPERATION is the universal superclass so that methods on it could do things like introspection, collect statistics, etc. and
2. OPERATION provides a certain expected set of behaviors to those who wish to provide their own subclasses.
He had to choose one or the other, so he chose 1, and sacrificed 2. He has successfully convinced me, at least, that he couldn't do both.
This suggests as a lesson for the future that one might wish to always distinguish between class FOO and class FOO-EXTENSION where the library user is to subclass only FOO-EXTENSION (but may, for special cases, define methods on FOO).
cheers, r
24.01.2014, 22:58, "Robert P. Goldman" rpgoldman@sift.info:
I believe that the issue was that Faré couldn't preserve both:
- OPERATION is the universal superclass so that methods on it could do
things like introspection, collect statistics, etc. and
- OPERATION provides a certain expected set of behaviors to those who
wish to provide their own subclasses.
He had to choose one or the other, so he chose 1, and sacrificed 2. He has successfully convinced me, at least, that he couldn't do both.
Idea.
Theoretically, OPERATION may be the root class and keep the old semantics (downward + selfward + other). And subclasses override this semantics as they do now: DOWNWARD-PERATION, SELFWARD-OPERATION, etc.
This may lead to some code duplication in ASDF.
On Sat, Jan 25, 2014 at 7:58 AM, Anton Vodonosov avodonosov@yandex.ru wrote:
Theoretically, OPERATION may be the root class and keep the old semantics (downward + selfward + other). And subclasses override this semantics as they do now: DOWNWARD-PERATION, SELFWARD-OPERATION, etc.
This may lead to some code duplication in ASDF.
This sounds like a great transition plan indeed. Then we could have a warning for now, that would be transformed in an error in a year, and migrate to this new setup that detects wrongful combinations, without forcefully breaking things in the meantime.
Robert, also, in the manual you should advertise #+asdf3.1 non-propagating-operation, or something, because it won't exist earlier, and we probably want code to keep running with asdf 3.0 until every implementation has upgraded to 3.1.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Gilb's Law: Anything you need to quantify can be measured in some way that is superior to not measuring at all.
Faré wrote:
On Sat, Jan 25, 2014 at 7:58 AM, Anton Vodonosov avodonosov@yandex.ru wrote:
Theoretically, OPERATION may be the root class and keep the old semantics (downward + selfward + other). And subclasses override this semantics as they do now: DOWNWARD-PERATION, SELFWARD-OPERATION, etc.
This may lead to some code duplication in ASDF.
This sounds like a great transition plan indeed. Then we could have a warning for now, that would be transformed in an error in a year, and migrate to this new setup that detects wrongful combinations, without forcefully breaking things in the meantime.
Robert, also, in the manual you should advertise #+asdf3.1 non-propagating-operation, or something, because it won't exist earlier, and we probably want code to keep running with asdf 3.0 until every implementation has upgraded to 3.1.
OK, will do. I will try to get all of the new operations written up, but I don't believe I will have time until after the 31st (conference deadlines, and end of contract dates mean that the ASDF manual has to wait...).
I do not yet the proposal. How can OPERATION keep the old semantics and still be the root? There are 11 direct subclasses of OPERATION now, and some fan out from there (e.g., a fair sized subgraph under BUNDLE-OP). How is the overriding to be managed?
Currently, unlike in the proposal ("subclasses override this semantics as they do now"0, subclasses do not override the semantics, do they? My understanding was that the behavior of the new dependency-propagating classes was strictly additive. Is that not true? To make this new proposal work, as I understand it, we have to add *subtractive*, or non-monotonic behavior inheritance.
Unless I'm missing something, it seems like this new proposal will be much more complex than the current approach, which just involves two checks, and no modifications to core ASDF code.
I'm happy to see an approach that would provide no breakage, but I'd like to see a more detailed proposal before we proceed.
thanks, r
26.01.14, 00:51, "Robert P. Goldman" rpgoldman@sift.info":
Faré wrote:
On Sat, Jan 25, 2014 at 7:58 AM, Anton Vodonosov avodonosov@yandex.ru wrote:
Theoretically, OPERATION may be the root class and keep the old semantics (downward + selfward + other). And subclasses override this semantics as they do now: DOWNWARD-PERATION, SELFWARD-OPERATION, etc.
This may lead to some code duplication in ASDF.
This sounds like a great transition plan indeed. Then we could have a warning for now, that would be transformed in an error in a year, and migrate to this new setup that detects wrongful combinations, without forcefully breaking things in the meantime.
Robert, also, in the manual you should advertise #+asdf3.1 non-propagating-operation, or something, because it won't exist earlier, and we probably want code to keep running with asdf 3.0 until every implementation has upgraded to 3.1.
OK, will do. I will try to get all of the new operations written up, but I don't believe I will have time until after the 31st (conference deadlines, and end of contract dates mean that the ASDF manual has to wait...).
I do not yet the proposal. How can OPERATION keep the old semantics and still be the root? There are 11 direct subclasses of OPERATION now, and some fan out from there (e.g., a fair sized subgraph under BUNDLE-OP). How is the overriding to be managed?
Currently, unlike in the proposal ("subclasses override this semantics as they do now"0, subclasses do not override the semantics, do they? My understanding was that the behavior of the new dependency-propagating classes was strictly additive. Is that not true? To make this new proposal work, as I understand it, we have to add *subtractive*, or non-monotonic behavior inheritance.
Unless I'm missing something, it seems like this new proposal will be much more complex than the current approach, which just involves two checks, and no modifications to core ASDF code.
I'm happy to see an approach that would provide no breakage, but I'd like to see a more detailed proposal before we proceed.
thanks, r
I think this may be done without too radical changes.
I suppose the operation behaviour is represented by some generic functions (probably returning the list of subcumponents to process by the operation, when invoked on a component). By overriding these functions, a subclass may tune its behaviour, no matter what superclasses it has. BTW, I am curious what exactly generic functions
Each direct subclass of OPERATION should have overriden all necessary methods to behave according to the subclass purpose (DOWNWARD-OPERATION, SELFWARD-OPERATION, ...)
This may be achieved introduction of an intermedate, unexporded, class between OPERATION and DOWNWARD-OPERATION, SELFWARD-OPERATION, ...
It may be called NO-OPERATION, for example. This class overrides all the generic functions in the operation behaviour protocol to work equally to the current OPERATION.
In this case, DOWNWARD-OPERATION and siblings change inheritance from OPERATION to NO-OPERATION; rest of their code remain unchanged.
Below this level all the inheritance works exactly as now.
What remains it to implemente the old behavious in OPERATION without inheritence from DOWNWARD-OPERATION and siblings. I don't know ASDF code, maybe some helper functions need to factored out from DOWNWARD-OPERATION and simblings implementation, to be reused in OPERATION, probably something will be duplicated, or maybe some ASDF mechanizm will help (I remember I saw in some .asd file function in-order-to mentioned; function named like that may probably be helpful).
Best regards, - Anton
OK, will do. I will try to get all of the new operations written up, but I don't believe I will have time until after the 31st (conference deadlines, and end of contract dates mean that the ASDF manual has to wait...).
Take your time. We're all impatient about it.
I do not yet the proposal. How can OPERATION keep the old semantics and still be the root? There are 11 direct subclasses of OPERATION now, and some fan out from there (e.g., a fair sized subgraph under BUNDLE-OP). How is the overriding to be managed?
I suppose the trick is as follows: 1- move the crucial behavior of downward-operation, sideway-operation, selfward-operation, in separate defun's that are called by the respective defmethods. 2- have methods on operation that check whether the operation is a subtype of any of the three above or non-propagating-operation. If not — then apply the methods for all of them. 3- at instantiation-time, call the same detection function, and issue a WARNING. In six months or a year, it will be replaced by a CERROR, then an ERROR, at which point the defuns can be merged back into the respective defmethods.
This way, people are warned now already, but ASDF keeps working, and we don't break either all the software in quicklisp until later.
Congratulations to Anton V. for this clever suggestion.
My understanding was that the behavior of the new dependency-propagating classes was strictly additive. Is that not true? To make this new proposal work, as I understand it, we have to add *subtractive*, or non-monotonic behavior inheritance.
Yes, the above is essentially emulating non-monotonic behavior inheritance, but it needs only rely on typep as introspection mechanism.
Unless I'm missing something, it seems like this new proposal will be much more complex than the current approach, which just involves two checks, and no modifications to core ASDF code.
This proposal adds a small bit of complexity, but only local changes, and maximally preserves compatibility.
I'm happy to see an approach that would provide no breakage, but I'd like to see a more detailed proposal before we proceed.
*If* we have time tomorrow, I could do that live during the walkthrough — that could be a good exercise.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The ultimate result of shielding men from the effects of folly is to fill the world with fools. — Herbert Spencer
Faré wrote:
OK, will do. I will try to get all of the new operations written up, but I don't believe I will have time until after the 31st (conference deadlines, and end of contract dates mean that the ASDF manual has to wait...).
Take your time. We're all impatient about it.
I do not yet the proposal. How can OPERATION keep the old semantics and still be the root? There are 11 direct subclasses of OPERATION now, and some fan out from there (e.g., a fair sized subgraph under BUNDLE-OP). How is the overriding to be managed?
I suppose the trick is as follows: 1- move the crucial behavior of downward-operation, sideway-operation, selfward-operation, in separate defun's that are called by the respective defmethods. 2- have methods on operation that check whether the operation is a subtype of any of the three above or non-propagating-operation. If not — then apply the methods for all of them. 3- at instantiation-time, call the same detection function, and issue a WARNING. In six months or a year, it will be replaced by a CERROR, then an ERROR, at which point the defuns can be merged back into the respective defmethods.
This way, people are warned now already, but ASDF keeps working, and we don't break either all the software in quicklisp until later.
Congratulations to Anton V. for this clever suggestion.
My understanding was that the behavior of the new dependency-propagating classes was strictly additive. Is that not true? To make this new proposal work, as I understand it, we have to add *subtractive*, or non-monotonic behavior inheritance.
Yes, the above is essentially emulating non-monotonic behavior inheritance, but it needs only rely on typep as introspection mechanism.
Unless I'm missing something, it seems like this new proposal will be much more complex than the current approach, which just involves two checks, and no modifications to core ASDF code.
This proposal adds a small bit of complexity, but only local changes, and maximally preserves compatibility.
I'm happy to see an approach that would provide no breakage, but I'd like to see a more detailed proposal before we proceed.
*If* we have time tomorrow, I could do that live during the walkthrough — that could be a good exercise.
If you are satisfied that this can be done cleanly, I am satisfied.
I agree -- congratulations to Anton for a clever solution to a knotty problem.
Best, Robert
Thanks to Anton's suggestion, the fix was relatively simple — see ASDF 3.1.0.56, commit eb2da723.
I also committed some improvements initiated during the walkthrough.
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Think you can, or think you can't — either way, you'll be right. — Henry Ford
27.01.2014, 11:04, "Faré" fahree@gmail.com:
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
Will do durint the next days
27.01.2014, 11:04, "Faré" fahree@gmail.com:
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
I have results for ASDF 3.1.0.63 for these lisps:
abcl-1.2.0-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
No regressions found so far.
The report: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-28.html
It compares the ASDF 3.1.0.63 results to the results with original ASDF versions coming with these lisps, on quicklisp 2013-12-13. All the failures we see here were discussed previously, they are not ASDF problems (quicklisp bug + the strange CCL:SLOT-VALUE-USING-CLASS on CCL when building cl-l10n).
I am running more tests, will follow up here when results are ready.
Best regards, - Anton
10.02.2014, 02:31, "Anton Vodonosov" avodonosov@yandex.ru:
27.01.2014, 11:04, "Faré" fahree@gmail.com:
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
I have results for ASDF 3.1.0.63 for these lisps:
abcl-1.2.0-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
No regressions found so far.
The report: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-28.html
It compares the ASDF 3.1.0.63 results to the results with original ASDF versions coming with these lisps, on quicklisp 2013-12-13. All the failures we see here were discussed previously, they are not ASDF problems (quicklisp bug + the strange CCL:SLOT-VALUE-USING-CLASS on CCL when building cl-l10n).
I am running more tests, will follow up here when results are ready.
I also tested on the same quicklisp version our previous baseline - asdf.2.32.35, for the following lisps:
abcl-1.2.0-fasl42-linux-x86 abcl-1.2.1-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
The same result - no regressions detected.
Next I will try the deferred-warnings enabled and :uiop in the use-list of :asdf-user.
On Tue, Feb 11, 2014 at 11:56 PM, Anton Vodonosov avodonosov@yandex.ru wrote:
Next I will try the deferred-warnings enabled and :uiop in the use-list of :asdf-user.
Thanks a lot!
I assume you're doing these tests separately: I expect ~25 failures in the first test, and hopefully 0 in the second.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Malthusianism: because growth may stop within centuries, we must use extreme violence to put an end to it now and make everyone poor today.
12.02.2014, 08:57, "Anton Vodonosov" avodonosov@yandex.ru:
10.02.2014, 02:31, "Anton Vodonosov" avodonosov@yandex.ru:
27.01.2014, 11:04, "Faré" fahree@gmail.com:
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
I have results for ASDF 3.1.0.63 for these lisps:
abcl-1.2.0-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
No regressions found so far.
The report: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-28.html
It compares the ASDF 3.1.0.63 results to the results with original ASDF versions coming with these lisps, on quicklisp 2013-12-13. All the failures we see here were discussed previously, they are not ASDF problems (quicklisp bug + the strange CCL:SLOT-VALUE-USING-CLASS on CCL when building cl-l10n).
I am running more tests, will follow up here when results are ready.
I also tested on the same quicklisp version our previous baseline - asdf.2.32.35, for the following lisps:
abcl-1.2.0-fasl42-linux-x86 abcl-1.2.1-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
The same result - no regressions detected.
Next I will try the deferred-warnings enabled and :uiop in the use-list of :asdf-user.
Probably I made some mistake in my setup and the latest ASDF was not in effect when tests are run.
We hare a regresssion
There is no applicable method for the generic function #<COMMON-LISP:STANDARD-GENERIC-FUNCTION ASDF/COMPONENT:COMPONENT-CHILDREN (1)> when called with arguments (#<CFFI-GROVEL:GROVEL-FILE "static-vectors" "ffi-types">)
The examples are
scriptl http://cl-test-grid.appspot.com/blob?key=1bc1pz0jwt http://cl-test-grid.appspot.com/blob?key=1lj39kniag
trivial-features-tests http://cl-test-grid.appspot.com/blob?key=1w15tfux4w http://cl-test-grid.appspot.com/blob?key=1s3h4rlltj
To reproduce it:
./lisps/sbcl-bin-1.1.11/run.sh --no-sysinit --no-userinit --load asdf/build/asdf.lisp --load quicklisp/setup.lisp (ql:quickload :scriptl)
Why does this error happen? Should ASDF provide default method for ASDF/COMPONENT:COMPONENT-CHILDREN ?
Best regards, - Anton
Should ASDF provide default method for ASDF/COMPONENT:COMPONENT-CHILDREN ?
That's another valid approach.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "Reality is that which, when you stop believing in it, doesn't go away". — Philip K. Dick
Thanks a lot Anton.
Robert, is there any blocker to release?
Anton, would you have time to run the tests with the deferred-warnings enabled? With :uiop in the use-list of :asdf-user?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Living your life is a task so difficult, it has never been attempted before.
On Sun, Feb 9, 2014 at 5:28 PM, Anton Vodonosov avodonosov@yandex.ru wrote:
27.01.2014, 11:04, "Faré" fahree@gmail.com:
Anton: can you run cl-test-grid against that? If you have time, also try with (uiop:enable-deferred-warnings-check) and also try with :uiop in the :use list of :asdf-user.
I have results for ASDF 3.1.0.63 for these lisps:
abcl-1.2.0-fasl42-linux-x86 ccl-1.9-f96-linux-x86 clisp-2.49-unix-x86 cmu-snapshot-2014-01__20e_unicode_-linux-x86 ecl-13.5.1-unknown-linux-i686-bytecode ecl-13.5.1-unknown-linux-i686-lisp-to-c sbcl-1.1.11-linux-x86
No regressions found so far.
The report: http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-28.html
It compares the ASDF 3.1.0.63 results to the results with original ASDF versions coming with these lisps, on quicklisp 2013-12-13. All the failures we see here were discussed previously, they are not ASDF problems (quicklisp bug + the strange CCL:SLOT-VALUE-USING-CLASS on CCL when building cl-l10n).
I am running more tests, will follow up here when results are ready.
Best regards,
- Anton
10.02.2014, 02:49, "Faré" fahree@gmail.com:
Anton, would you have time to run the tests with the deferred-warnings enabled? With :uiop in the use-list of :asdf-user?
I plan to do so, after the usual tests.
On 25 Jan 2014, at 23:29, Faré fahree@gmail.com wrote:
OK, will do. I will try to get all of the new operations written up, but I don't believe I will have time until after the 31st (conference deadlines, and end of contract dates mean that the ASDF manual has to wait...).
Take your time. We're all impatient about it.
I do not yet the proposal. How can OPERATION keep the old semantics and still be the root? There are 11 direct subclasses of OPERATION now, and some fan out from there (e.g., a fair sized subgraph under BUNDLE-OP). How is the overriding to be managed?
I suppose the trick is as follows: 1- move the crucial behavior of downward-operation, sideway-operation, selfward-operation, in separate defun's that are called by the respective defmethods. 2- have methods on operation that check whether the operation is a subtype of any of the three above or non-propagating-operation. If not — then apply the methods for all of them. 3- at instantiation-time, call the same detection function, and issue a WARNING. In six months or a year, it will be replaced by a CERROR, then an ERROR, at which point the defuns can be merged back into the respective defmethods.
This way, people are warned now already, but ASDF keeps working, and we don't break either all the software in quicklisp until later.
Congratulations to Anton V. for this clever suggestion.
You have to be careful: If anybody defines :around methods, you will get surprising results. This is why the CLOS MOP forbids user code to define methods on pre-defined metaclasses, where a very similar trick is used to bottom out the reflective tower - specified generic functions on standard-xyz classes test whether the class of the metaobject parameter is exactly standard-xyz or a subclass, and branch accordingly. You may want to specify similar restrictions on user code to prevent such surprises.
I also strongly advise against turning warnings into cerror or error. It just prevents working code from continuing to work, which is gratuitous IMHO.
Pascal
-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.
On 24 Jan 2014, at 19:56, Robert P. Goldman rpgoldman@sift.info wrote:
Pascal Costanza wrote:
The new ASDF will break systems that subclass OPERATION. Those libraries' maintainers will have to look at their code, based on the error message, and see what needs to be done. For the vast majority of them, five minutes work will suffice. For those that take more... well, as you point out, the bug fix had to be made, and I stand ready to help those who need assistance adjusting.
It's good to signal an error or a warning for this case.
It would have been better if asdf:operation would have kept its old semantics, maybe with a deprecation warning when it's used, and that the new semantics would have been provided by, say, asdf3:operation. People who see the warning and care about keeping things up to date could react to the warning and move from asdf:operation to asdf3:operation, which is not much more work than what is necessary now. People who care less (maybe because other things are more important - asdf is not the center of the universe after all) would not need worry, because everything would work as before, even in a setting where both asdf2 and asdf3 semantics would be needed side by side.
I believe that the issue was that Faré couldn't preserve both:
- OPERATION is the universal superclass so that methods on it could do
things like introspection, collect statistics, etc. and
- OPERATION provides a certain expected set of behaviors to those who
wish to provide their own subclasses.
He had to choose one or the other, so he chose 1, and sacrificed 2. He has successfully convinced me, at least, that he couldn't do both.
This suggests as a lesson for the future that one might wish to always distinguish between class FOO and class FOO-EXTENSION where the library user is to subclass only FOO-EXTENSION (but may, for special cases, define methods on FOO).
This is what the CLOS MOP already does (object -> standard-object, class -> standard-class, method -> standard-method, etc.).
Pascal
-- Pascal Costanza The views expressed in this email are my own, and not those of my employer.
good afternoon,
On 23 Jan 2014, at 6:18 PM, Robert P. Goldman wrote:
James Anderson has asked for a "concise statement of the issue at hand."
thank you for responding to my questions and for shifting the focus of the discussion to how to ensure that deployments of your library continue to work, rather than that they fail - even if evidently rather than nefariously.
i have understood from the ensuing discussion that,
1. the deprecated (2.*) version provided just one implementation for an effective operation - that is, for the behavior of perform given operation and system graph instances.
2. this effective operation incorporated depth-first graph traversal and post-order operator application.
3. that version is to be superseded with one (3.1.0.*) which binds the alternative traversal and application implementations to OPERATION specializations, where each specialization implements a specific combination of traversal and application. (the *-OPERATION classes)
4. the 3.1.0.* version provides no implementation for the OPERATION class.
5. the observed failure mode has been that in systems for which the definitions include an OPERATION specialization, the intended actions do not occur because no dependencies are observed.
6. in order to rectify these failures, it is proposed to reinstate the depth-first+post-order mechanism for the OPERATION class in addition to the alternative mechanisms for specializations of that class.
Faré has had to change the semantics of the OPERATION class. Previously, the OPERATION class would automatically trigger dependencies downward (notably, if you did an OPERATION on a system, then you had to do that OPERATION as well on that OPERATION's children), "leftward," and onto itself ("selfward").
The implementation of this left holes, and also caused problems for extensions (e.g., for TEST-OP, one typically had to define a default PERFORM that did nothing, since testing a system necessarily tested all of the files).
To fix this, Faré had to add new mixin classes, DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, and SELFWARD-OPERATION.
In order to preserve the integrity of the code, OPERATION remained as the superclass of all operation objects. But its semantics has changed.
if the above is accurate, the change was the methods specialized for OPERATION now do nothing.
This is not a problem for anyone with a "vanilla" ASDF system. The only programmers who are vulnerable to this change are those who have defined their own subclasses of OPERATION (or those who use libraries whose programmers have defined subclasses of OPERATION).
For those cases, I have pushed a change that checks to make sure that all operations that are instantiated are subclasses of either DOWNWARD-OPERATION, UPWARD-OPERATION, SIDEWAY-OPERATION, SELFWARD-OPERATION, or a new NON-PROPAGATING-OPERATION. If this run-time test fails, an error will be signaled and the programmer will have to fix the offending OPERATION subclass's definition.
several issues are under consideration
- how to integrate into PERFORM precedence relations other than "depends on component"? - how to retain the "depends on component" PERFORM protocol for those deployments which require it? - how to provide for future alternative PERFORM protocols?
the situation requires a representation for the graph traversal and operation execution mechanism other than the "depends on component" mechanism which was present in-line in the superseded version 2.* PERFORM and TRAVERSE methods specialized on OPERATION. there are several alternative ways to retain functional integrity despite the protocol changes, some of which have already been considered. the possible realizations manifest a alternative combinations of patterns:
- distinguish those classes which serve as protocol declarations from concrete classes which bind implementations. in the superseded asdf version, OPERATION played both roles. in version 3.1.0.* OPERATION is declared as a protocol class, with implementations bound to the concrete <direction>-OPERATION classes. the vodonosov version would be to ascribe to OPERATION the additional role as a concrete class by reiterating (aspect of) other class definitions for it.
- provide explicit declarations for alternative aspects of the PERFORM protocol and compose them to compute the effective operation. the 3.1.0.* approach is to bind aspects to *-OPERATION specializations and compose them according to class relations in the operations effective method. in this regard, one might also consider decomposing the aspects more explicitly in terms of the representation of the relation according to which the precedence is to be established. one way to express this would be to name the operation specializations according to the component slot which represents the respective relation. this would make it easier to comprehend their composition. another would be to delegate the graph access to other (singleton) instances bound to the operation. this would facilitate future extensions.
among possible approaches:
(defclass operation () ()) (defclass downward-operation (operation) ()) (defmethod apply ((op downward-operation)) (do-x) (do-y) (do-z)) (defmethod require ((op downward-operation)) (propagate-downward))
has been implemented by 3.1.0.*, but compromises deployments of the form
(defclass special-operation (operation) ()) (defmethod apply ((op special-operation)) (do-special))
as no dependency relation is implemented. it has been suggested (the vodonosov version), to reiterate (aspects of) the *-OPERATION methods for OPERATION. to wit (ignoring arguments):
(defclass operation () ()) (defclass downward-operation (operation) ()) (defmethod apply ((op downward-operation)) (do-downward)) (defmethod require ((op downward-operation)) (require-downward)) (defmethod apply ((op operation)) (do-downward)) (defmethod require ((op operation)) (require-downward)) (defun do-downward () (do-x) (do-y) (do-z)) (defun require-downward () (propagate-downward))
this has the benefit, that it could reduce failures among existing deployments. On the other hand, given the dual role which OPERATION would need to play, it is a "encumbered" realization of the "CLASS + SIMPLE-CLASS" protocol pattern, which would impair future extensions as use of call-next-method is constrained.
another alternative, is to realize the inverse, the "ABSTRACT-CLASS + CLASS" pattern , in which OPERATION serves just the role of CLASS and binds a concrete implementation while alternative inherit from ABSTRACT-CLASS.
(defclass abstract-operation () ()) (defclass operation (abstract-operation) ()) (defclass downward-operation (abstract-operation) ()) (defmethod apply ((op downward-operation)) (do-downward)) (defmethod require ((op downward-operation)) (require-downward)) (defmethod apply ((op operation)) (do-downward)) (defmethod require ((op operation)) (require-downward)) (defun do-downward () (do-x) (do-y) (do-z)) (defun require-downward () (propagate-downward))
this would require changes, to move methods from OPERATION to ABSTRACT-OPERATION, but would offer the benefit, that no class which specializes ABSTRACT-OPERATION would necessarily be encumbered by the OPERATION implementation.
as yet further alternatives, it would be possible to implement the combinations of propagation and application by delegating the implementation to a collection of singleton instances. in this case the alternatives could be declared as the default initialization value(s) for the respective (ABSTRACT-)OPERATION subclass. this would no doubt require broader changes to the asdf code base, but would offer the advantage, that the combination could have a degree of independence from the class hierarchy. for the case at hand, it would then be more natural to declare the default behavior for OPERATION than the approach of duplicating the methods of other specializations.
best regards, from berlin,
[Note to Robert: plenty of implementation notes in this email; you may want to save it to include the data in the TODO file and/or the manual.]
Dear James,
in case I didn't tell you, it's great to have you back in asdf-devel. You kind of left some time after I didn't take your suggestion to split ASDF into many files (because I felt the concatenation needed to be done by ASDF, or bust); years afterwards, I took your suggestion (after it became trivial for ASDF to handle the concatenation), and there you're back. I can't swear that it was a good idea to wait that long before to split ASDF, but in any case, the result of the splitting is immensely satisfying, when combined with the one-package-per-file approach of quick-build, that ensures some well-ordering of internal dependencies. Thanks for this and your many other suggestions and contributions (I admit I haven't fully grokked you asdf-pathname-test thing yet, despite massively refactoring it already — everytime I debug something, I have to try reunderstand what it's trying to do.)
i have understood from the ensuing discussion that,
the deprecated (2.*) version provided just one implementation for an effective operation - that is, for the behavior of perform given operation and system graph instances.
this effective operation incorporated depth-first graph traversal and post-order operator application.
ASDF is still traversing the graph depth-first for planning, then performing the planned actions in post-traversal order: that's just what it means to traverse a dependency graph.
The problem is that the downward and sideway dependencies were not implemented in an object-oriented way through COMPONENT-DEPENDS-ON, but baked into the infamous TRAVERSE algorithm inherited from ASDF 1. As we fixed bugs during the ASDF 2 days, Robert Goldman and I incrementally refactored the TRAVERSE algorithm into smaller functions until we finally understood what the thing actually does. Then trying to solve the one or two residual bugs that remained, I eventually figured what it instead *should have done* and implemented it, through many and many iterations, until it was just right. Now, with ASDF 3, the dependencies of an ACTION (operation + component) is wholly specified in an objected way through methods on COMPONENT-DEPENDS-ON (misnomer inherited from ASDF 1) — it should have been named ACTION-DEPENDS-ON instead, except that the concept of action was implicit in ASDF 1 and there was no name for it.
The problem was that if you traverse dependencies the ASDF 1 way, many of the actual dependencies end up missing: if module or system B depends on module or system A, then compiling and loading files in B should depend on loading and compiling files in A. People didn't usually notice in a build from scratch, because these dependencies were implicitly in the traversal order of the algorithm and serial nature of the build. But that could cause serious bugs in an incremental or parallel build: then, if files in A were recompiled, files in B wouldn't be. In practice, this means that incremental builds would sometimes fail in strange and mysterious ways, and people would then have to remove their build output files and restart from scratch. To get reliability in building large systems (e.g. at ITA), we needed to rebuilding things from scratch every time.
Robert Goldman, who had tried to fix the cross-system dependency problem, suggested there were such missing dependencies, but I didn't understand. Andreas Fuchs' parallel build (POIU) had dealt with it in an clever undocumented kludge, but I didn't understand either.
While implementing timestamp propagation (big bug #1), I had to massively rewrite the TRAVERSE algorithm, and rewrite POIU to fit the new ASDF internals, which is always a good test of ASDF extensibility. I tried to plainly remove the POIU kluge that I didn't understand (accumulating a huge list of dependencies from transitive parents), and of course the parallel build started failing. Then I realized that these dependencies were needed, and that the correct way to handle this situation without polynomial slowdown was to introduce new nodes: LOAD-OP and COMPILE-OP both need to depend on a node that corresponds to loading (via LOAD-OP) the parent's dependencies; I created a new operation called PARENT-OP and eventually renamed to PARENT-LOAD-OP then PREPARE-OP. Now, PREPARE-OP has to propagate upward, and neither downward nor sideway (or rather, sideway, it propagates to a LOAD-OP). And so I initially tried to kluge TRAVERSE to do that, until it became apparent that the TRAVERSE algorithm could be massively simplified by just moving all that stuff to COMPONENT-DEPENDS-ON, and then instead of an ununderstandable spaghetti mess of a TRAVERSE algorithm, there was a clean small jewel of an algorithm, and a much more versatile object model than before.
Then, I had to also finally understand the subtle difference between IN-ORDER-TO and DO-FIRST, that once again Andreas Fuchs had resolved in a clever undocumented kludge in POIU that I didn't understand and tried to remove, to my failure. The real issue is whether a dependency matters for its side effects in the file system or in the current image. In the former case (typically, compiling a lisp file to FASL), its associated timestamp is that of the most recent output file(s), it need not be redone if already done (i.e. its oldest output file later than its latest dependencies). In the latter case (typically, load an existing FASL), it must be redone in the current image if never done even if already done before in a previous image, and the timestamp associated to it must be that of its most recent file dependency, and not the wall clock time at time of loading. And the trick is that for correctness, we may traverse every action node twice: the first time, we only needed it to have happened once, in another image that created a FASL that we depend on; the second time, we realize we need it in the current image. That is why LOAD-OP depends directly on PREPARE-OP for its side-effects in the current image, and cannot "just" depend on it indirectly through COMPILE-OP, because the latter needs not have happened in the current image, and thus neither do its dependencies, even in-image dependencies (they would have been in-image for the previous image that did the COMPILE-OP). Once again, the DO-FIRST vs IN-ORDER-TO behavior of ASDF 1 was kind of working in the simple cases, but could not possibly handle more complex cases including timestamp propagation.
- that version is to be superseded with one (3.1.0.*) which binds the alternative traversal and application implementations to OPERATION specializations, where each specialization implements a specific combination of traversal and application. (the *-OPERATION classes)
That change ALREADY HAPPENED in ASDF 2.27 (a.k.a. ASDF 3 beta), and at that time, I got all OPERATION extensions in Quicklisp fixed. The problem is that some extensions are not in Quicklisp, and Robert was badly bitten, and so wants a solution to save other people from being bitten as badly.
- the 3.1.0.* version provides no implementation for the OPERATION class.
Starting with ASDF 2.27, OPERATION exists and has a trivial implementation that does not have any implicit dependencies, either through baked in TRAVERSE magic or COMPONENT-DEPENDS-ON methods. It is merely the base class for OPERATIONs, as many existing extensions assume that we don't want to break.
- the observed failure mode has been that in systems for which the definitions include an OPERATION specialization, the intended actions do not occur because no dependencies are observed.
Most extensions to OPERATION did not want the implicit DOWNWARD and SIDEWAY dependencies, but a few did. These extensions were broken by ASDF 3 (aka ASDF 2.27), and had to be fixed. I could only fix those in Quicklisp and at work. I assumed that and an annoucement in asdf-devel would be enough. I was wrong.
The problem is that for backward-compatibility, OPERATION has somehow to be *both* the abstract base class for every operation including those that don't propagate down and left, *and* the old default behavior of those operations that did.
We thought that was impossible; I had let down people with the second assumption with silent subtle breakage. Robert would rather break it loudly for everyone.
- in order to rectify these failures, it is proposed to reinstate the depth-first+post-order mechanism for the OPERATION class in addition to the alternative mechanisms for specializations of that class.
It is proposed to add DOWNWARD and SIDEWAY dependencies magically to operation classes that do not inherit from any of DOWNWARD-OPERATION, SIDEWAY-OPERATION, UPWARD-OPERATION and NON-PROPAGATING-OPERATION, as an emulated case of "non-monotonic inheritance". Ugly, but will work.
as yet further alternatives, it would be possible to implement the combinations of propagation and application by delegating the implementation to a collection of singleton instances. in this case the alternatives could be declared as the default initialization value(s) for the respective (ABSTRACT-)OPERATION subclass. this would no doubt require broader changes to the asdf code base, but would offer the advantage, that the combination could have a degree of independence from the class hierarchy. for the case at hand, it would then be more natural to declare the default behavior for OPERATION than the approach of duplicating the methods of other specializations.
Operations in ASDF 3 are already kind of singleton instances, via MAKE-OPERATION and FIND-OPERATION. However, since old user code here and there may still be using MAKE-INSTANCE, the unicity isn't really guaranteed. That doesn't matter too much so far, because ASDF equates all objects of the same class in its method COMPONENT-OPERATION-TIME, and it is therefore unsafe to ever perform two actions with the same component and semantically different operations of the same class.
Note to Robert: here is how you could do it if you wanted to allow semantically distinct operations of the same class. You'd need to have a protocol to canonicalize them in the *OPERATIONS* memoization table, not by class name, but by CONS of the class name and some CANONICAL-OPERATION-INITARGS. The latter would be a generic function called on the initargs, where any parasite initargs such as FORCE and FORCE-NOT have been removed, since they below to the PLAN, not the OPERATION: the OPERATE protocol would be refined to explicit split arguments to be passed to MAKE-PLAN or to MAKE-OPERATION. The default method for CANONICAL-OPERATION-INITARGS would SORT (a plist->alist of) the initargs, and that would replace the current ORIGINAL-INITARGS slot. For this scheme to work even in presence of undisciplined users using MAKE-INSTANCE on an operation class, the OPERATION class would have an extra slot EFFECTIVE-OPERATION, uninitialized by default (nil or unbound), whose accessor initializes it if it's uninitialized, by looking up a canonical instance in *OPERATIONS*, and if unfound registering the current operation as canonical. Then, each component's COMPONENT-OPERATION-TIME hash-table would be indexed by canonicalized operation object rather than by operation class, and POIU would have to be changed accordingly. Of course, this entire cleanup is incompatible with how SWANK and GBBopen currently abuse slots of operation, so these would have to be fixed first. And that's why I didn't do it. It looks like SWANK can be fixed soon, though, so we'll see.
PS: There was another question asked at the end of my walkthrough, for which I now remember the good answer I didn't have then. Yes, you *could* walk the data structure in two passes and only care to accumulate timestamps and planned flag only in the second pass; but then you would need to remember all the dependency arcs, instead of just being able to have them be implicit in the traversal; That's what POIU does. That would have simplified the 36-line TRAVERSE-ACTION function, but would have necessitate pulling in all the graph data structures from POIU, which would make the code overall code bigger and more complex. As for the ~50 line COMPUTE-ACTION-STAMP method, it's the other meaty part of the algorithm, generalizing and fixing ASDF 1's old OPERATION-DONE-P to correctly handle timestamp propagation, and parametrized by a plan object and a just-done flag.
There there. I hope I've explained all the important details before I go away from the project. Do not hesitate to ask more questions.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org So that there be reality, there must be an observer. "I am, therefore someone thinks." — Faré
-
Anton Vodonosov -
Daniel Herring -
Faré -
james anderson -
Pascal Costanza -
Robert Brown -
Robert Goldman -
Robert P. Goldman