December 2012 - asdf-devel - mailman3.common-lisp.net

[asdf-devel] ASDF 2.27 release candidate
by Faré 03 Jan '13

03 Jan '13

OK, so 2.26.45 is my new release candidate for ASDF 2.27. Please grab it and test it. Since last release candidate 2.26.24, * lots of work was done towards backwards compatibility, fixing bugs introduced or uncovered during the previous fixing of essential conceptual bugs through massive refactoring of internals. * tests were added for asdf-bundle, which now also runs on XCL, but not on ECL with bytecode compiler. * tests were ran on all platforms with a command line on Linux x86-64 and even on LispWorks Personal Edition and Genera that don't, and they all pass modulo a few implementation bugs that were #-'ed out. * upgrade tests were also made to pass everywhere that's meaningful * consequent minor fixes to warnings or corner case errors were issued for Allegro, CLISP, ECL, ECL with bytecode compiler, SBCL, XCL. * There was a fix for LispWorks Personal Edition. You need to update ironclad and nibbles to run with the new ASDF because of the dropped if-component-dep-fails feature, but all other software should continue to run unmodified. Since release 2.26, the main changes are as follows: * ASDF-BUNDLE was merged into ASDF. This notably means fewer headaches for users of ECL; it also means a new load-fasl-op feature for all users of a modern CL, making it easier to deliver software as a single fasl. All remnants of its predecessor asdf-ecl were removed. * *decades-old* bugs of notable significance were fixed: (a) changes in dependencies now trigger a rebuild across systems, and https://bugs.launchpad.net/asdf/+bug/479522 (b) timestamps are now correctly propagated https://bugs.launchpad.net/asdf/+bug/1087609 In olden days, some have argued for not rebuilding systems that :depend-on a modified one as a "feature". It really isn't. On the other hand, if they want to explicitly prevent a rebuild, the actual feature :force-not is there to help them (released as part of 2.21 in April 2012). I will write about those bugs. * PREPARE-OP was introduced to fix a conceptual bug in the ASDF object model. It corresponds to "loading the dependencies of a component and its parents" and is explicitly depends-on'ed before to load or compile the component, instead of implicitly depending on it via magic in the TRAVERSE algorithm, which magic proved insufficient in cases revealed by the above bugs. It propagates upward in the component hierarchy, rather than downward, like the TRAVERSE algorithm used to systematically propagate dependencies. * TRAVERSE was gutted out and factored into reusable higher-order functions, which not only fixes the above conceptual bug, but makes for semantics that are simpler to implement, possible to understand, easier to extend, and less limited in expressivity. TRAVERSE doesn't automatically propagate operations downward the component hierarchy anymore; propagation is now implemented through methods on COMPONENT-DEPENDS-ON. * COMPONENT-DEPENDS-ON is now more powerful: it can express dependencies on arbitrary operation objects (i.e. operations of the same class can have different options) acting on arbitrary component objects (i.e. not necessarily siblings of the current component). The ASDF object model can now express arbitrary build graphs, with special magic no more for children components vs other dependencies. * Internals have been refactored, and some sorry features were excised. The semantics of OPERATION-DONE-P is simplified and now well-specified. The dubious DOFIRST mechanism is gone. FIND-COMPONENT will pass component objects through, and a corresponding FIND-OPERATION replaces MAKE-SUB-OPERATION. Several internal accessors were renamed. * IF-FEATURE is a new attribute of components that accepts an arbitrary feature expression such as (:and :sbcl (:or :x86 :x86-64)), which when defined is a precondition to the component being enabled; when the expression is false, the component is disabled, and dependencies to it are removed. It replaces the misguided :if-component-dep-fails attribute of modules and the accompanying :feature feature, which were dropped. They were not very expressive, clunky to use, and relied on baking conceptually dubious non-local behavior in the old TRAVERSE algorithm, which belied the object model and defeated other features. Unhappily, this breaks backwards compatibility for the few who used it, namely sb-grovel, nibbles and ironclad. A minimal compatibility layer allows ASDF 2.27 or later to load the SB-GROVEL of old versions of SBCL, but it won't suffice to load an old version of nibbles or ironclad; please upgrade them a recent version that was updated to support ASDF 2.27. * ASDF2.27 was added to the *features* so you can detect the presence of this massively updated ASDF and its new features with #+asdf2.27. This is *not* ASDF3, for we painfully maintained backwards compatibility. * Portability updates were done for ABCL, CLISP, CMUCL, ECL, LispWorks Personal Edition, MKCL, XCL. * Documentation received some small updates. * Tests were updated, notably including support for asdf-bundle. They were successfully run on ccl clisp sbcl ecl ecl_bytecodes cmucl abcl scl allegro lispworks allegromodern xcl. Manual tests were run on lispworks-personal-edition genera. Untested were cormancl mkcl rmcl. * POIU was updated. This extension is distributed separately from ASDF. It is now much simplified and made more correct by construction by reusing the new TRAVERSE infrastructure of ASDF and now being able to assume the ASDF object model is a complete description of dependencies. On SBCL it will handle deferred warnings from background compilations. On platforms that cannot fork it will gracefully fall back to serial compilation. * SLIME support for ASDF was significantly enhanced. It is distributed separately from ASDF, with SLIME. See latest checkin from 2012-12-26. We recommend that in your ~/.swank.lisp you should: (in-package :swank) (defparameter *upgrade-asdf-p* t) (pushnew 'try-compile-file-with-asdf *compile-file-for-emacs-hook*) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He wa'n't no common dog, he wa'n't no mongrel; he was a composite. A composite dog is a dog that is made up of all the valuable qualities that's in the dog breed — kind of a syndicate; and a mongrel is made up of all riffraff that's left over. — Mark Twain

3 7

[asdf-devel] asdf bundle and do-first
by Faré 03 Jan '13

03 Jan '13

Another conceptual bug in ASDF. Checking the plans generated for load-fasl-op, I find that with my new ASDF 2.26.x (currently x=45), it systematically loads all the individual .fasl's before it loads the .system.fasl, defeating the purpose. This was not the case before: in 2.26.7 and earlier, the (separately distributed before 2.26.7) asdf-bundle has load-fasl-op depend-on fasl-op which depends-on compile-op, which doesn't depend-on load-op, but do-first's on it — so that the dependency is only considered but if the immediate timestamp of the fasl is older than the timestamp of its lisp source. If all fasl's satisfy this local timestamp check, nothing is recompiled, none of the do-first is triggered, and the load-op's are not attempted. But the old design also fails to propagate timestamp information from the other files that are depended-on, let alone do-firsted. Therefore, there is some notion in the old do-first that must be preserved in the new ASDF, even though the previous design was conceptually flawed overall. In the end, what must be preserved is that some dependencies need to have been performed in the current image, whereas other dependencies only need to have been performed in some previous (or current) image. This means a dependency can be walked over twice: once in without "image-necessity", and a second time with it. Without image-necessity, only the timestamp matters; with image-necessity, whether it was performed in current image matters, too. Presumably, a node with output-files clears the image-necessity flag before checking timestamps, but re-traverses its dependencies with the flag if the timestamp check fails. A node with no output-files on the other hand is done for in-image effect, and heeds and propagates the image-necessity flag to its subnodes. I believe that it's possible to continue the current strategy of traversal without an explicit graph, though nodes may now be traversed twice: once without image-necessity, once with. Sigh. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Success is getting what you want. Happiness is wanting what you get. — Dale Carnegie

1 1

[asdf-devel] trivial-timeout patch
by Faré 29 Dec '12

29 Dec '12

Dear Gary, here is a patch to trivial-timeout to prevent unnecessary recompilation, now that ASDF actually propagates timestamp information based on static file dependencies. It basically fixes the .asd file to not reference non-existing files, so that the absence of those files doesn't cause ASDF to systematically force compilation of everything afterwards. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Tunes: because C is not the be all and end all of system programming languages

1 0

[asdf-devel] failures with 2.26.36
by Zach Beane 21 Dec '12

21 Dec '12

Here are a few failures: - Weblocks: http://report.quicklisp.org/weblocks/2012-12-20/failtail.txt - GBBopen also has the issue with :PARENT in the system file - hu.dwim.reiterate: http://report.quicklisp.org/hu.dwim.reiterate/2012-12-20/failtail.txt - lisp-executable-example has the same :PARENT issue - folio: http://report.quicklisp.org/folio/2012-12-20/failtail.txt I don't really understand 'em, just passing them along. Zach

2 4

[asdf-devel] File type problem with the latest ASDF
by Zach Beane 20 Dec '12

20 Dec '12

A number of projects fail to build with the latest ASDF from git. A few of them seem to be related to defining a custom file component class that has a custom file type. For example, see jwacs and portable aserve: http://report.quicklisp.org/jwacs/2012-12-19/failtail.txt http://report.quicklisp.org/portableaserve/2012-12-19/failtail.txt For jwacs, the file component has a type of "js", and for portableaserve, it has a type of "cl", but it seems like ASDF is looking for different file names. FSet also fails with this error: Error while trying to load definition for system fset from pathname /[...elided...]/fset-20121219-svn/fset.asd: Invalid initialization argument: :PARENT in call for class #<STANDARD-CLASS SYSTEM>. Zach

2 7

[asdf-devel] suggestion for aserve.asd
by Robert Goldman 20 Dec '12

20 Dec '12

You don't need acl-file any more. You can now use cl-source-file.cl, which is exported from ASDF. You can make legacy-acl-source-file inherit from cl-source-file.cl and cut the ACL-FILE class entirely. That will take care of using the .cl extension instead of .lisp. Cheers, r

3 3

[asdf-devel] Insufficient :if-component-dep-fails compatibility for nibbles
by Faré 18 Dec '12

18 Dec '12

stassats remarks that my if-component-dep-fails compatibility code might be enough to compile sb-rotate-byte, but nibbles not so. Indeed, it so appears that the semantics of the (feature ...) pseudo-operation (note: note a pseudo-component) is that a component depends on not just on its local feature, but on all the recursive features of all the components and subcomponents it depends-on. This means that you really can't untangle that from the entire traverse protocol. Moreover, since the visit protocol does not cache dependency on a missing feature, the previous implementation had cost O(n^2) on the size of a module, instead of O(n). I really don't want to tuck that horror back into the traversal algorithm. I implemented the "compatibility mode" with an eye on all people with old versions of SBCL, who may have a hard time upgrading it. On the other hand, I believe that people who upgrade asdf can easily upgrade nibbles, too. Nathan, can you use #+sbcl and similar other conditionals in your nibbles.asd instead of :if-component-dep-fails? That would really help with compatibility, without requiring ASDF to support something I really want out of the implementation. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Vision is the art of seeing what is invisible to others. — Jonathan Swift

1 0

[asdf-devel] prepare-op
by Faré 18 Dec '12

18 Dec '12

I have finally fixed an essential "dependency problem" of ASDF by introducing a new operation "parent-load-op" which I'm going to rename "prepare-op". This in turns opens the way for many many improvements to the internals, which will make ASDF easier to explain and extend, so expect some work there — and I'm postponing any 2.27 release until that's stable. However, any and all changes introduces potential backwards-incompatibility, and I'll be searching through packages available in quicklisp for any potential such incompatibility before I go ahead with renaming internals. Some details: In 2.26.14, I introduced an operation called parent-load-op, which ensures the dependencies of the parent are loaded before the child is compiled. This ensures the graph produced by following component-depends-on isn't missing crucial dependencies instead of relying on the specifics of the TRAVERSE algorithm (and at long last I really understand what rpgoldman was telling me while we were writing the ASDF 2 paper). I had to tweak traverse, because parent-load-op trickles upward, not downward, the component hierarchy: you depend-on the parent-load-op of your parents, but not on that of your children (that would create a circular dependency). So I had to add a check in TRAVERSE (now — 2.26.20 — in its nice and short visit-children helper) to not automatically propagate subclasses of parent-op dependencies downward. I started thinking about a theory of parent-op and child-op dependencies and how to rewrite the current hierarchy, and as I was doing it found that actually, parent-load-op is not parent-specific: it's actually a dependency-op or prepare-op, that ensures that the image is prepared to compile or load the component, by having already loaded all the dependencies, and it could do it for the "normal" case, too: instead of having load-op or compile-op depend on the load-op of all the dependencies, they could simply depend on prepare-op. Then, we don't need to have traverse know anything about propagating dependencies either downward or upward, it can all be done with two methods on component-depends-on! [which is one of the many misnomers in ASDF: components don't depend on anything, it's (operation . component) pairs, which I'm calling "actions", like in Kent Pitman's article), that are nodes in the dependency graph]. It was all quite a revelation. And so I'll be working on drastically cutting down ASDF, and making the basic dependency graph cleaner, which will make ASDF much simpler and more reliable to extend. Some change I'd like to make if no one uses these internals (or only use them without relying on the current setup to define methods, in which case I can define an alias]: module ==> make it a subclass of both child-component and parent-component system ==> is a parent but not a child, and thus not a module. component-depends-on ==> rename it to action-depends-on if-component-fails ==> I'd like to remove it altogether, it's a crock that doesn't go well with the reified graph model. I suppose I could preserve it the hard way, but would be ugly. mark-operation-done ==> maybe rename to stamp-action ? Or at least mark-action-done. operation-description ==> action-description component-visited-p ==> action-visited-p component-depends-on ==> component-dependencies - likely not possible for backwards compatibility. component-load-dependencies ==> component-depends-on - same. Then component-sibling-dependencies. circular-dependencies-components ==> circular-dependencies-actions, it's a list of (operation . component) pairs hash-tables of symbol, component ==> hash-tables of operations and component, systematically using make-sub-operation make-sub-operation ==> find-operation, to work like find-component, and ignore the first contextual argument if the second is already an object. Canonicalizes named operations while still allowing explicitly different operations of the same class. module-components ==> component-children module-components-by-name ==> component-children-by-name load-fasl-op ==> load-system-fasl-op load-op ==> split between load-op and load-fasl-op (see below). Alternatively, for backwards compatibility, call the latter load-compiled-op My current temporary names would also be renamed: parent-load-op ==> prepare-op (or dependency-op?) visit-children ==> done away with -- replaced entirely by one component-depends-on method. parent-op ==> upward-operation child-op ==> downward-operation Chasing "horizontal" dependencies to siblings under the same parent would be done entirely by prepare-op, which is a subclass of upward-operation. load-op would remain a downward-operation, but would have a separate strategy for deciding whether to delegate to load-source-op, load-fasl-op or load-system-fasl-op or whatever (e.g. some load-instrumented-op extension), based on a generalization of the force / force-not mechanism that would decide the compilation strategy. These ones would NOT be downward-operations, unlike the main load-op which would be a downward-op and propagate everywhere as appropriate. Since parents vs children vs siblings are no longer a fixture of the TRAVERSE algorithm but something reified in the object graph through the normal use of simple methods, you can easily define arbitrary graphs of your choice, instead of ASDF only being applicable to one particular kind of graph shapes. etc. Then the base object hierarchy becomes robust and stable, and it makes sense and is easy to extend, with a nice well-define dependency graph, that can be reified thanks to POIU's current make-parallel-plan. A lot of that (and previous) work was prompted by the desire to keep POIU working as I refactored ASDF, and then the desire to fix ASDF based on the things understood while adapting POIU. POIU interestingly builds an actual complete graph of dependencies from the object model, or at least did it within each system, because, not being able to fix ASDF, Andreas Fuchs had intercepted perform, not operate or perform-plan (which didn't exist then). One ugly inefficient detail Andreas had in his graph-building function was an additional-dependencies argument he used to copy the parent's dependencies on each and every child. That's really ugly. I wanted to get rid of it, but discovered I couldn't... until I invented parent-load-op, which makes it unnecessary, and also promises to fix compute-action-stamp. In any case, Andreas Fuchs certainly deserves a lot of credit for his POIU hack, of which I didn't fully understand the elaborate until I had to fully reimplement it bit by bit, to make it cleaner and fit the new ASDF, and admire each detail he got just right. And he did it all as an add-on, without changing a single line of ASDF itself (though overriding a few definitions — which I promptly made unnecessary once I became ASDF maintainer). That's impressive. And the language is also impressive for allowing such things, at all. But the result was absolute ugly. Being the unexpected co-maintainer of the two sometimes conflicting pieces of software has led me to make both of them ugly no more. PS: In the spirit of people who credit John McCarthy with having discovered Lisp rather than created it, I like think ASDF was a great *discovery* made by Dan Barlow, rather than some badly unfinished program he created; I'm only just reaching what I believe is the bottom of the matter, and it's much nicer than I thought it was, or than Dan probably understood, either, at the time. PPS: I was going to sent a mail to this list, then it grew very long into what would be a blog post, then two, etc. Scrap it. Back to a notice to the list. Still grew up to be too long. Oh well. I hope at least one of you read that and wasn't bored. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Politics is the only profession that does without learning, probably because those who suffer from mistakes are not the same as those who make them. — Achille Tournier, Pensées d'automne

4 23

[asdf-devel] Latest patch fixes all the tests for me...
by Robert Goldman 18 Dec '12

18 Dec '12

Haven't yet run the tests with upgrade. Will do that next. cheers, r

1 0

[asdf-devel] Problems with tests: static file types
by Robert Goldman 18 Dec '12

18 Dec '12

The failing tests all seem to have a variation on failure to read the TYPE slot of static files: Error while trying to load definition for system test-module-pathnames from pathname /Users/rpg/lisp/asdf/test/test-module-pathnames.asd: The slot TYPE is unbound in the object #<STATIC-FILE "test-module-pathnames" "sources/level1" "level2/static.file"> of class #<STANDARD-CLASS STATIC-FILE>.

3 2