On Thu, Jan 2, 2014 at 11:44 PM, Robert P. Goldman rpgoldman@sift.info wrote:

While there are bug fixes waiting to reach our users, I'm quite concerned by the loss of backwards compatibility in systems that defined their own OPERATION subclasses.

This backward incompatibility already happened a year ago: it was consubstantial with the refactoring that begat ASDF3, and necessary to fix the broken dependency model of ASDF2. NO, most operation MUST NOT propagate downward, or even sideways; indeed some like prepare-op must propagate upward instead. The downward and sideways propagations were baked into TRAVERSE; that was one of the deep conceptual bugs in the ASDF model. Now they are configurable via the (ill-named) COMPONENT-DEPENDS-ON; and the previous behavior is mere inheritance specification away, with the DOWNWARD-OPERATION and SIDEWAYS-OPERATION mixins.

(The others deep conceptual bugs in the ASDF model were the lack of transitive timestamp checking, and the mess of DEPENDS-ON vs DO-FIRST dependencies. These were loosely related deep bugs. Then there were shallower bugs like the IF-COMPONENT-FAILS horror, the inconsistency between system :depends-on and other :depends-on, and probably more small bugs I can't remember.)

As far as I can tell, *all* such systems will break, since the old solution was to subclass OPERATION and the new solution is to subclass DOWNWARD-OPERATION to achieve the same results. *ALL* programmers' locally-defined operation subclasses are now broken by this change.

We broke those that depended on downward and/or sideways propagation (don't forget sideways-operation). But we *FIXED* all those who didn't want it, and had to deal with it anyway! And that includes the vast majority of the operations I audited from quicklisp (see below), but also asdf's very test-op, its many bundle operations, and of course the model-fixing prepare-op.

I have personally seen multiple systems that are broken by this change, and I would like to see some hard thought put into repairing this.

Did you see the previous email where I audited the damage (and the fixes)? http://thread.gmane.org/gmane.lisp.asdf.devel/3581/focus=3582 Only 4-5 victims (depending on how you count), I fixed one (dependency-op), two (clean-op and revert-op) were never really working, and one (parenscript-compile-op) indeed needs love, but the fix is trivial, and no one complained so far.

Meanwhile 5-10 extensions were fixed by this change, and 4-5 were unaffected (or, for 1-2 of them, fixed already).

A somewhat drastic solution would be to make the name OPERATION now denote DOWNWARD-OPERATION (which would remain as a canonical name), and rename the common superclass of DOWNWARD-OPERATION and UPWARD-OPERATION to something like ABSTRACT-OPERATION or COMMON-OPERATION.

No, no, no. That would be really bad. The real solution is to fix the handful of broken extensions.

The current refactoring is quite problematic, since it moves some of the previously-existing characteristics of OPERATION out and into a sub-class that no one has ever heard of before.

That's what ASDF 3 has been doing for a year, and if no one has complained about those handful of broken systems, it's probably easier to fix them than to rebreak all the MANY MORE extensions that were either fixed or unaffected, and would now need to be fixed instead. The minimal change is to keep fixing things, not to revert to ASDF2 brokenness.

Unfortunately, the above solution is not ready for prime-time, either, since if we add COMMON-OPERATION, all programmers' methods that dispatch on OPERATION will break if used with PREPARE-OP. On the one hand, that's probably not a big deal, since no one will have been customizing UPWARD-OPERATIONs, since they haven't existed. On the other hand, programmers who want to write extensions that really are generic to *all* types of operation (e.g., EXPLAIN type methods) would be broken by this proposed repair.

Yes, many extensions rely on OPERATION being the top of the hierarchy, and you don't want to break all of them. That includes POIU, ITA's now published QUUX through its qres-build system, and at least 6 quicklisp systems that I can easily find grepping through ~/quicklisp/dist/quicklisp/software/**/*asd*. Several of these defmethods are probably obsolete, since most extensions shouldn't specialize operation-done-p anymore.

Really, the current ASDF3 architecture is much improved over ASDF2, though indeed it there are still active issues.

(This reminds me how deferred warnings broke 50-odd systems in quicklisp, out of which only about 25 were fixed, and 25 had unresponsive authors, even a year afterwards. In the end, I had the deferred warnings disabled by default. Good luck if you want to enable the feature at long last, either by getting everything fixed, or allowing out-of-band disabling of the warnings.)

This problem also exposes a HUGE hole in our regression-testing methods: we have nothing that tests extensions to the ASDF protocol.

I disagree. ASDF is defined in an incremental way, and all the code in ASDF itself is "extensions" to the protocol as defined by previous pieces. Consider asdf-bundle and concatenate-source, if nothing else. It is a testament to the overall good design of the original ASDF and its CLOS based architecture that the code is so clean and small, compared to other code that does equivalent things much worse (have you looked at mk-defsystem? Ugh! And let's not discuss some horrors in C or Java). I salute Dan Barlow, who did a lot of experimentation, and whose bigger success overshadows his smaller failures.

Now, we can always add more tests to the ASDF test suite. Hopefully, we can also add tests to the software in quicklisp, and get cl-test-grid to run them somehow. Then there's the yak shaving of establishing some standard for CL testing that would let us have test reports and detect regressions at the individual test level depending on various environment parameters, instead of having a coarse grain of erroring an entire system. Oh, the dream of Consolidating CL Test Libraries!

I also find that OUTPUT-FILES may now be called more eagerly than before, perhaps because of the new PREPARE-OP. I have seen systems (perhaps not well-built systems....) that defined OUTPUT-FILES on an operation, O, in a way that assumed that earlier operations (COMPILE-OP and LOAD-OP) would be called before the OUTPUT-FILES method on O. That actually seems wrong, since OUTPUT-FILES is part of the plan-making part of the ASDF protocol, rather than the plan-execution part, but it *used to work* and given the extremely sketchy information available to users who wish to customize ASDF, such disruptions need to be very gently handled.

OUTPUT-FILES is called much more than before, a bit because of PREPARE-OP, but a lot because of COMPUTE-ACTION-STAMP. Moreover, particularly with BUNDLE-OP, OUTPUT-FILES can be expensive to compute. That's why for performance reasons, we now cache the results of OUTPUT-FILES. (Many thanks to stassats for pushing me into caring for performance, and making ASDF3 no significantly slower than ASDF2, sometimes faster, even though it is now doing much more work, more correctly).

Finally, as the responsible party now, I'm not comfortable sending out another release until I have come to understand the new protocol better than I do now. Indeed, it was culpably negligent of me to release the last couple of versions, and I apologize for doing so.

I understand your concern. On the other hand, consider that

* This particular change has been here for a year with no bug report, despite indeed breaking a handful of extensions that no one uses. Only one known useful piece of code remains broken.

* Your standard should probably not be perfection, but improvement and non-regression, and I believe the current release candidate meets it. The failures that we are experiencing are either news tests, or new platforms that were previously untested or non-supported; meanwhile plenty of bugs have been fixed, with many tests added, and new functionality is at hand.

* Throughout all the history of ASDF1 and ASDF2, all authors and maintainers including danb and including me obviously didn't have a clear understanding of the old protocol, since it was so fundamentally buggy. ASDF3 fixed the protocol. Your lack of understanding of the new protocol is not worse than the previous maintainer's lack of understanding of the old protocol.

Of course, you're in charge now, and may validly decide that it's a blocking issue.

I believe the ASDF sources are much more readable than they used to be; if you need explanations on any part of it, I'll gladly explain, add new comments or revise existing ones, or give you an interactive walkthrough. That might be a good thing to do, anyway, to pass the torch; and we could even record that event for any interested party. Walking through ASDF/DEFSYSTEM should take a couple of hours. less $(make defsystem-files)

Yours in procrastination,

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org You think you know when you can learn, are more sure when you can write, even more when you can teach, but certain when you can program. — Alan Perlis

I get this, and the recoding is a HUGE improvement. Furthermore, you have convinced me that we can't unwind this change in any way that won't create yet more damage.

However: a. This change is not backward-compatible

Actually, I looked again, and paren-compile-op is actually unaffected, because it already chases its dependencies through component-depends-on methods. Therefore, the only broken things are 1- ASDF-DEPENDENCY-GROVEL, of which I am the de facto maintainer (oops), and 2- CLEAN-OP and REVERT-OP, which were never useful or fully working.

b. It's not announced to maintainers of ASDF systems in a way that gets it done and

Actually, I have vague memories of going through all these quicklisp systems last year and having them do the right thing, but I'd have to dig through my mail to confirm.

c. The bugs it gives rise to are subtle and confusing. There's virtually no way a person whose system suddenly stops performing the expected operations is going to say "I bet I need to change the superclasses of my specialized OPERATION subclasses."

Nevertheless, there are no instance of the bug in Quicklisp. If someone has some proprietary software, and things fail for him because of an ASDF upgrade, and he hasn't contacted us yet, he deserves to lose.

Unfortunately, because we reused the old name, OPERATION, it is *very* difficult to trap this in a helpful way. Ideally, I would suggest doing something like raising some sort of signal when the user makes a new OPERATION subclass to suggest that s/he might want DOWNWARD- or UPWARD-OPERATION instead. But I don't see that alley as open to us.

We *have* to reuse the old name, because the name captures SOME of the meanings of "operation" (like "root class of all operations") while it breaks some others (like "a class of instances of which propagate downward). That these meanings are incompatible is an irreducible fact.

I'm relatively familiar with ASDF's guts, and an update to Allegro (getting ASDF 3 in place of 2) still cost my company about four programmer days, and was only resolved because I thought to IM you about my problems.

Sorry about that :-(

You only audited publicly available libraries. Applications, like ours, which perform tasks of no interest to the general world, but critical to us, are invisible to you.

Well, I had to create job security for myself :-/ I mean, yes, the version number was incremented from 2 to 3 because there was *some* level of backward incompatibility, and I wanted to warn users of the risks, even though I believe it was all well worth it in the end.

I have always been available for questions, and yes, it's annoying when you upgrade libraries and things don't "just work", and I've played that game enough to know the pain, but that's the price to pay for using third party code.

At any rate, an upgrade to the build system that requires the build system's author to review all the available community libraries -- not to mention libraries that are not shared with the community! -- is simply not an approach that will scale.

Yet, there is no other approach in the end. At best, you can have some versioning system that lets different software use different sets of versions of the same libraries (see NixOS), but when things break (and they will, not just for ASDF), you ultimately have the choice of sticking to an old one or moving forward.

At work, we also have a build system used by thousands of engineers, and yes, making sure we keep being able to rebuild all the software is at the same time the goal and the daily testing constraint of our software developments. When things break, we try to detect it early and warn users, but eventually, they have to move forward.

We need a patch to ASDF that will indicate to programmers in a clear way that the behavior of ASDF has changed, and give them a hint about how to fix this change.

If you know how to do things better... well, you are now in a position to do it. As for me, I didn't know any, but to discuss on this mailing-list, and sometimes rummage through code and contact authors individually (many many thanks to Zach for Quicklisp and Anton for cl-test-grid).

Ideally, we would be able to fix this socially, rather than with code: we would simply shout off the rooftops that the OPERATION classes had changed, indicate the needed revisions, and all would be well.

I've tried doing that. Too few people listen. Yet eventually, we must move on.

Consider the changes needed to make ASDF support cross-compilation, parallelism and determinism, which is all the same thing, really. I've documented in the ASDF TODO file some steps needed. In the end, that will require backward-incompatible changes. There are lots of things we can do to paper over the incompatibility, in addition to contacting the authors of that dozen or two systems that extend ASDF in non-trivial ways and getting them to fix their code. But in the end, if a year after, they haven't changed their system, you can consider the code unmaintained and dead (like those 25 systems that still didn't reply to my bug reports about deferred warnings).

Unfortunately, we do not have the capability to shout off the rooftops in this way: ASDF quietly slips out into the community mediated by the implementation suppliers.

When you update your implementation, you're supposed to expect breakage and have backups ready. And frankly, I always recommend putting all your dependencies, including ASDF, under source control — and including your compiler, too, or at least an identifier for its exact version.

I suppose one could introduce an ASDF banner that would print once, as a means of shouting off the rooftops.

No. The default mode of operation of ASDF should be silent. But if/when you introduce further incompatibility (see above), you can e.g. have a defmethod operate :before, :around or :after method that checks newly loaded or defined systems for signs that it has not been updated, e.g. methods defined on now deprecated or unused API functions. Then, both issuing a warning and defining a compatibility wrapper is a good idea.

I'm open to alternative suggestions; this one does not seem excellent.

One possibility might be adopting Allegro's style of incompatible update. When they changed the behavior of the reader for consecutive reader macros being able to make multiple s-expressions skippable (like #-allegro #-allegro :foo :bar), they introduced a new variable, something like

EXCL:*READER-MACRO-COMPATIBILITY*

If this had a value like :OLD, it would raise a continuable error when it encountered one of these consecutive reader, explaining the change in behavior. Once you knew about the behavior, you would set (or bind) this variable, and the warning would go away.

So people who made new OPERATION subclasses would get a warning until they introduced a form into their system definition files indicating that they knew about the change.

That would let us explain the situation to programmers, and shut up when the programmers understand the situation.

That's a distinct possibility, using CLOS introspection to visit every class, and patch the problematic ones; except you must do it promptly enough to fix use of these classes before they are have visible bad effects.

I agree that it's easier to fix such libraries -- after all, it's typically just replacing

OPERATION

with

#-asdf3 OPERATION #+asdf DOWNWARD-OPERATION

To me the key issue (to reiterate) is how do we find the people who need to do this, and make it known to them that they need to make this change?

Actually, replacing the superclass lisp, from (operation) to (#+asdf3 downward-operation #+asdf3 sideway-operation operation), with possibly non default corresponding slot values.

At least for us, the reason that there has been no bug report is that Allegro pushed an intermediate version of ASDF that had a broken EXCL:RUN-SHELL-COMMAND. When we figured this out, we ripped out that patch, so all of our production code has been running on ASDF2 until this past week.

I presume you mean broken ASDF:RUN-SHELL-COMMAND. Yes, subtle incompatibilities were produced, and some of them only fixed recently. That was a bug. I apologize. In exchange for which, we now have a vastly more expressive and more portable UIOP:RUN-PROGRAM. Our test suite was just not good enough, but it's much improved since.

I don't think silence in this case can be taken to indicate the absence of bugs. There can be a huge lag between ASDF releases and ASDF penetration into the user community.

That's also one of the reasons I invite people on the mailing-list to test new versions of ASDF, and test them in isolation of other implementation upgrades, etc.: because that's how you detect bugs. If you just wait a long time for the upgrade, then experience massive bugs, well, you wrought the lagged failure upon yourself, sadly.

I'm willing to see less than perfection, but a key desideratum for me is "fail loudly and obviously," instead of "quietly and confusingly." I want the quiet and confusing OPERATION failures to be moved to being understandable before the next release.

I can appreciate the sentiment, but good luck with that. We possibly have exhausted the number of people who might care. Those who care and still haven't upgraded a year later, obviously don't care as much as they should.

While I see the advantages of getting bug fixes out there, I don't believe that we get that many opportunities to achieve uptake through the implementations (with the possible exception of SBCL). So I want the most improvement/release possible, and if there's something that seems critical to me, I'm disinclined to let it slip past a release.

I'm not sure what you mean, but you're the release manager now.

CCL these days is the implementation that upgrades ASDF fastest, with rme reacting within a day of being probed. Next, CMUCL and ABCL these days usually upgrade within a release cycle. SBCL upgrades maybe every so many months, with all maintainers now wary of some breakage or some other due to ASDF or the way it's used in SBCL (which I changed recently, for the better, but with a lot of pain in transition). ECL at one point was not just very responsive but ahead of the curve, with asdf-ecl, however it is now unmaintained. Allegro, CLISP, LispWorks can update very fast or take years, depending. Other implementations just generally lag behind (mocl, scl, mkcl, gcl) or are dead (xcl, mcl, corman, genera).

There was at least a time when I understood the *actual* (as opposed to the desirable) protocol. This has slipped away. I'd like it to come back before I put my name behind another release.

I definitely welcome your offer of a walkthrough. I think that will be hugely helpful.

If you want, we can do this video walkthrough this weekend or next week, rather than between Jan 19-26. You're the primary intended audience.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org My understanding of macros is expanding — Matthias Felleisen wrt the PLT Scheme module system