Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm.
ASDF had a few major bugs remaining from ASDF 1.
To fix them, its traverse algorithm was, once again, rewritten and vastly simplified. It's almost readable, now!
To start with, I fixed this long-standing bug, whereby when a system was recompiled, this didn't trigger the recompilation of systems that depended on it as part of the same operate session: If you have a system B that depends on a system A, and A is modified, re-loading B will cause A to be recompiled, but not B. https://bugs.launchpad.net/asdf/+bug/479522 This was starting to annoying me a lot, since it causes systems such as xcvb-utils to often fail to recompile, getting its automatically-generated package all out of sync. One reason this bug survived so long is that there were rumors of people relying on the bug as a "feature". But ever since 2.21, we had the ability with :force-not to selectively avoid rebuilding
But this led to the deeper issue that ASDF only knew about timestamps local to a immediate operation, and was not propagating timestamps from one component to those that depended on it, which means the previous fix only worked within the same session: if you loaded B in one session, modified A, recompiled A in another session, then loaded B in a third session, the latter wouldn't trigger a recompilation of B: each file looks locally up-to-date, and is blissfully unaware that it is too old as compared to its dependencies. https://bugs.launchpad.net/asdf/+bug/1087609
I fixed this bug too, which required some major modifications, overall a simplification, in the way ASDF works. For backwards compatibility (if it's not backwards, it's not compatible), I had to keep some of the previous APIs, such as the misnamed operation-done-p the signature of which I couldn't change. As users, you shouldn't notice that anything changed. If you are power users, you may notice that do-first dependencies have been eliminated. There is now only one kind of dependencies, in-order-to, which does the right thing in both cases of output-files or no output-files.
It was way too much debugging, but it's now done. Please test.
When all this is cleaned up and released, I'm tempted to declare the launchpad milestone "asdf 2.1" reached, and open a start a new milestone 2.2 (numbers obviously disconnected from actual version numbers, these days).
Enjoy!
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Thinking is the hardest work there is, which is probably the reason why so few engage in it. — Henry Ford
Faré fahree@gmail.com writes:
Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm.
When I tried this, most Quicklisp systems failed. I get this (or variations):
Error opening #p"/usr/local/lib/sbcl/sb-psoxi/defpackage-ASDF-TMP.fasl": Permission denied
Zach
Zach Beane xach@xach.com writes:
Faré fahree@gmail.com writes:
Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm.
When I tried this, most Quicklisp systems failed. I get this (or variations):
Error opening #p"/usr/local/lib/sbcl/sb-psoxi/defpackage-ASDF-TMP.fasl": Permission denied
Zach
Sorry, I did a cut & paste by hand and typo'd sb-posix. The pathname in the actual error is real.
Zach
Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm.
On Sat, Dec 8, 2012 at 2:54 PM, Zach Beane xach@xach.com wrote:
Zach Beane xach@xach.com writes:
When I tried this, most Quicklisp systems failed. I get this (or variations):
Error opening #p"/usr/local/lib/sbcl/sb-posix/defpackage-ASDF-TMP.fasl": Permission denied
Zach
Sorry, I did a cut & paste by hand and typo'd sb-posix. The pathname in the actual error is real.
Oops.
Were you trying 2.26.9?
I fixed a few problems that tickled the SBCL contribs in 2.26.10, and another in 2.26.12.
It's working for me currently. I hope it's now working for you, too.
My apologies for the bugs.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The highest goal of computer science is to automate that which can be automated. — Derek L. VerLee
Faré fahree@gmail.com writes:
Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm.
On Sat, Dec 8, 2012 at 2:54 PM, Zach Beane xach@xach.com wrote:
Zach Beane xach@xach.com writes:
When I tried this, most Quicklisp systems failed. I get this (or variations):
Error opening #p"/usr/local/lib/sbcl/sb-posix/defpackage-ASDF-TMP.fasl": Permission denied
Zach
Sorry, I did a cut & paste by hand and typo'd sb-posix. The pathname in the actual error is real.
Oops.
Were you trying 2.26.9?
I fixed a few problems that tickled the SBCL contribs in 2.26.10, and another in 2.26.12.
It's working for me currently. I hope it's now working for you, too.
My apologies for the bugs.
Ok, I'm now using 2.26.18 and I get similar-looking failures in umlisp and commonqt:
unhandled SB-FORMAT:FORMAT-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1003179343}>: error in FORMAT: no more arguments
~A completed without~*~@[ its input file~:p~{ ~S~}~]~ ~@[ or~]~*~@[ its output file~:p~*~{ ~S~}~] ^
Zach
Ok, I'm now using 2.26.18 and I get similar-looking failures in umlisp and commonqt:
unhandled SB-FORMAT:FORMAT-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1003179343}>: error in FORMAT: no more arguments ~A completed without~*~@[ its input file~:p~{ ~S~}~]~ ~@[ or~]~*~@[ its output file~:p~*~{ ~S~}~] ^
Dear Zach,
oops, apparently, I'm no good at writing non-trivial FORMAT strings.
It should be fixed now — but now it will issue a warning about something fishy in those builds: some output file is declared but never actually created, or something.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org A common man marvels at uncommon things; a wise man marvels at the commonplace. — Confucius
Summary: ASDF 2.26.15 yet improved to allow a new vastly improved POIU 1.29.
Did you ever notice that ASDF did *not* correctly propagate timestamps?
On ASDF 1 and up to 2.26.8, If component A changes, and component B depends on system A, but compilation stopped before B in a previous run, then B may never be compiled again, even though a crucial change in a macro in A would yield a different file. Phantom bugs! This was fixed in 2.26.9, by teaching ASDF about timestamp propagation, which was a major rewrite of the TRAVERSE internals.
I updated POIU to use those new internals, and the hell of debugging POIU led me to making those internals painfully simple (within the constraints of respecting the declared semantics) and obviously correct. Also, many kludges were fixed in the internal dependency machinery. Many methods to component-depends-on were refactored to do the Right Thing where they weren't, and if you ever had problems with defining dependencies, they will hopefully have vanished. The infamous do-first is notably gone. Only in-order-to remains.
POIU now demonstrates how you can build alternate strategies to plan a build and perform a plan — in this case, in parallel, using an API that abstracts away the gory details dependency management.
To get dependencies right, I introduced a new notion, parent-op: operations on a component depend on the parent-op of the parent (unlike other operations, parent-op propagate up the component hierarchy, not down). This allow for the dependency graph to be actually correct, rather than to rely on implicit details of the traversal and plan execution; and this is also how POIU can build a complete graph of the dependencies, without the previous ugly kluges (in case you ever wondered what additional-dependencies was in the POIU sources).
I gutted away much of POIU, and found that I had to put back all the pieces I had put away, albeit in a more organized way. Andreas Fuchs obviously faced the same horror before me, except he didn't have the luxury of being able to fix ASDF to get POIU working nicely with it.
I also fixed the a deadlock issue in POIU when trying to report deferred warnings that do not fit in an OS pipe buffer: POIU now uses files, not pipes, to communicate its results.
Anyway. POIU was an incredible HELL to debug, but here it is now. Nothing should change for you, except that bugs have silently been squashed.
POIU has been tested to work with SBCL and single-threaded-CCL; earlier versions kind of worked with CLISP, but this one looks like it won't.
Please test ASDF and POIU, tell me if they work for you, and send me your feedback.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it — Richard Feynman
-
Faré -
Zach Beane