January 2013 - asdf-devel - mailman3.common-lisp.net

[asdf-devel] cl-test-grid questions
by Faré 07 Jan '13

07 Jan '13

Dear Anton, I probably do NOT want my cl-test-grid results to be mixed with the usual results, since I want to run with my own version of ASDF which will skew the results. But I DO want to compare my results to an existing baseline. How do I achieve this effect? Where in cl-test-grid is the correct place to hook a call to asdf:upgrade-asdf before trying to load stuff? And to (asdf:initialize-source-registry '(:source-registry :ignore-inherited-configuration)) to make sure I'm testing what's in quicklisp and now my own source checkouts? (or are you doing that part already?) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If a law is unjust, a man is not only right to disobey it, he is obligated to do so. — Thomas Jefferson

2 3

[asdf-devel] An interesting ASDF conundrum
by Robert Goldman 07 Jan '13

07 Jan '13

This one I think is relevant to the proposed reorganization of ASDF to have changes in upstream systems trigger recompilation in systems that depend on them. I have made a crude component which is a MADE-FILE. The made-file is built by invoking make to produce some output file. MADE-FILE's compile-op OPERATION-DONE-P method always returns NIL, because ASDF does not understand what files the made file depends on -- that information is buried in the makefile. We don't want to duplicate that logic in ASDF. If we always invoke make, it will do the Right Thing and only rebuild the file when necessary. *Unfortunately*, any components in a system that have a :depends-on relationship to a MADE-FILE, will always be recompiled, whether necessary or not. If I remember correctly, the logic in traversal is that you do the child if its input files are fresher than the output file *or* if one of its dependencies was run. Now, with current ASDF, I can work around this for MADE-FILEs with stable APIs: I just shove them out into systems of their own, so they don't trigger constant recompilations of systems that depend on them. But this behavior will be changing with the new ASDF. I don't *believe* the existing controls on :force will solve this problem, either. We can't control this by simply saying "recompile everything if any upstream system changes" or "don't recompile anything based on upstream system changes" (although the latter will restore the current buggy behavior). I can think of two solutions: 1. KLUDGY -- allow individual system definitions (like my MADE-FILE wrapper systems) to specify that they do not propagate recompilation forcing onto their dependents. 2. BETTER, but bigger effect on the protocol: Separate the question of whether an operation is performed on a dependency from whether that dependency propagates change downstream. We could do this by adding a function that is run to test whether an <operation,component> pair propagates forcing onto its dependents. By default it would be the same as "did the operation get performed?" as now, but it could be overridden by component classes like MADE-FILE where, e.g., I could say "did the file-write-date of the OUTPUT-FILES change?" Does this make sense? cheers, r

2 6

[asdf-devel] test-logical-pathnames hangs on ACL 8.2 on Mac OS X
by Robert Goldman 06 Jan '13

06 Jan '13

I get to Test logical pathnames in source-registry, recursive and ACL goes away seemingly forever. The reason, I believe, is that this: #+allegro #.(asdf::pathname-root #p"ASDF:") returns "/" for me. When I look, I see that pathname-root is losing the host name somehow: CL-USER> (pathname-host #P"HOME:") "HOME" CL-USER> (pathname-host (pathname-root #P"HOME:")) NIL As far as I can tell messing around in the REPL, it's the specification of both :defaults #P"HOME" *and* :host NIL when MAKE-PATHNAME* calls MAKE-PATHNAME that does the damage: on ACL the :host NIL seems to smash the value for :host that would come from the defaults: CL-USER> (make-pathname :defaults #P"HOME:" :name "foo" :type "lisp" :host nil) #P"foo.lisp" CL-USER> (make-pathname :defaults #P"HOME:" :name "foo" :type "lisp") #P"HOME:foo.lisp" The ANSI spec for make-pathname says "After the components supplied explicitly by host, device, directory, name, type, and version are filled in, the merging rules used by merge-pathnames are used to fill in any unsupplied components from the defaults supplied by defaults." So I think it's an acceptable interpretation that the :host field should be cleared in the return value of MAKE-PATHNAME. Would it be possible to modify make-pathname* so that instead of passing defaults through to make-pathname as it does now, it would *unpack* the defaults argument into fields of the return value? I will try some experiments. cheers, r

1 1

[asdf-devel] ql-test
by Faré 06 Jan '13

06 Jan '13

Dear ASDF developers, so as to not issue ASDF release candidates that are totally bogus anymore, I've written this program, ql-test, inspired by ql-mapper: ssh://common-lisp.net/home/frideau/git/ql-test.git Maybe some of you will be interested in this infrastructure. I'm currently grovelling its output to determine breakage is caused by ASDF 2.26.55. Already sent a patch to chipz. Will send more. In other news, fset and gbbopen have merged my patches, so at least these won't cause problem (I also removed for now the cleanup that caused the backward-incompatibility with old fset). I won't declare victory until all new packages work with ASDF 2.27. PS: I'm keeping POIU updated, which helps me keep the ASDF code clean. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The ultimate result of shielding men from the effects of folly is to fill the world with fools. — Herbert Spencer

2 3

[asdf-devel] Patch for chipz.asd
by Faré 06 Jan '13

06 Jan '13

Dear Nathan, while trying to mass-recompile stuff in quicklisp with my current ASDF, I found that a mistake in chipz.asd was causing it to trigger recompilation of everything that depends on it: the html file is misnamed "chipz" instead of "index". Also note that if there is no semantics attached, you might as well use (:static-file "chipz-doc.txt") instead of painfully defining a new class txt-file then using (:txt-file "chipz-doc"). BTW, which ASDF bogosity was your comment referring to, and is it now fixed? If not, is there already a bug in launchpad for it? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Wir Mathematiker sind alle ein bisschen meschugge. (We mathematicans are all a bit crazy). — Lev Landau

1 0

[asdf-devel] More ASDF changes ahead
by Faré 05 Jan '13

05 Jan '13

Since I'm waiting for some clients to make appropriate changes before I may release ASDF (notably FSet and GBBopen), I'm considering making a few additional backward-incompatible changes in interfaces that I believe are not otherwise used (after grepping for code in quicklisp). The crux of the change is that I would like to move to a model where operations are a place-holder with zero content besides the class name — or failing that, a model where the timestamps are associated to a specific instance of an operation, not a class name, at which point memoization of "equivalent" operations is necessary for #'equal hash-tables to contain timestamps. This is not just a matter of taste, but also of correctness: if the semantics of two instances of an operation is actually different, then considering one is done when the other was done is incorrect — yet that's what ASDF currently does. I recently added a PLAN object for the purpose of traversals, which made it possible for ASDF and POIU to share the TRAVERSE algorithm. (Incidentally, it also makes ASDF look more like XCVB or LIL.) I'm moving :force and :force-not to this PLAN object, removing them (and :verbose) from OPERATION, removing the whole OPERATION-ANCESTOR crud (which was only a dirty way of having the ancestor operation object behave as a kind-of backassward plan object) and having a global hash of *OPERATIONS* that are EQUAL iff created with the same initargs. Now for issues: * swank.asd relies on OPERATION-FORCED for some obscure RELOAD feature. I think this is a mistake. I'll discuss that with the slime developers. Meanwhile, I'll keep it aroud through a stub for backward-compatibility. * GBBopen (again!) uses operation initargs to determine the name of an application to be built. I think that's a mistake. This information should be in the component. I'll discuss the matter with the GBBopen developers. Meanwhile, I will keep the initargs around for backward-compatibility. * the :on-failure and :on-warnings flags to operation don't actually work well with the new model, and had quite fishy semantics in the old one. Grepping through quicklisp-distributed code, it looks like no one is actually *using* it, though it's been cargo-culted in a few places. I will remove it, just keeping a few stubs for backward-compatibility. In practice, the only thing that works is to either bind *compile-file-on-warning-behaviour* and/or *compile-file-on-failure-behaviour* around the build, or equivalently pass :on-warning and/or :on-failure arguments to operate. The internal ASDF machinery that used to make that happen is horribly complex (what with make-sub-operation, now called find-operation, to pass along the flags from each operation to the next in the traversal), and does no more than if operate just recognized those keywords to bind those variables — which is my proposed interim solution (the defaulting based on a dynamic variable otherwise breaks my model of operations being equal if created from the same keyword arguments). I don't like those variables, especially since they mean subtly different things on different implementations (as usual CLISP being an outlier), but I'll have to think on how to best eliminate or replace them. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org To surrender to ignorance and call it God has always been premature, and it remains premature today. — Isaac Asimov

1 0

[asdf-devel] ASDF 2.27 release candidate
by Faré 03 Jan '13

03 Jan '13

OK, so 2.26.45 is my new release candidate for ASDF 2.27. Please grab it and test it. Since last release candidate 2.26.24, * lots of work was done towards backwards compatibility, fixing bugs introduced or uncovered during the previous fixing of essential conceptual bugs through massive refactoring of internals. * tests were added for asdf-bundle, which now also runs on XCL, but not on ECL with bytecode compiler. * tests were ran on all platforms with a command line on Linux x86-64 and even on LispWorks Personal Edition and Genera that don't, and they all pass modulo a few implementation bugs that were #-'ed out. * upgrade tests were also made to pass everywhere that's meaningful * consequent minor fixes to warnings or corner case errors were issued for Allegro, CLISP, ECL, ECL with bytecode compiler, SBCL, XCL. * There was a fix for LispWorks Personal Edition. You need to update ironclad and nibbles to run with the new ASDF because of the dropped if-component-dep-fails feature, but all other software should continue to run unmodified. Since release 2.26, the main changes are as follows: * ASDF-BUNDLE was merged into ASDF. This notably means fewer headaches for users of ECL; it also means a new load-fasl-op feature for all users of a modern CL, making it easier to deliver software as a single fasl. All remnants of its predecessor asdf-ecl were removed. * *decades-old* bugs of notable significance were fixed: (a) changes in dependencies now trigger a rebuild across systems, and https://bugs.launchpad.net/asdf/+bug/479522 (b) timestamps are now correctly propagated https://bugs.launchpad.net/asdf/+bug/1087609 In olden days, some have argued for not rebuilding systems that :depend-on a modified one as a "feature". It really isn't. On the other hand, if they want to explicitly prevent a rebuild, the actual feature :force-not is there to help them (released as part of 2.21 in April 2012). I will write about those bugs. * PREPARE-OP was introduced to fix a conceptual bug in the ASDF object model. It corresponds to "loading the dependencies of a component and its parents" and is explicitly depends-on'ed before to load or compile the component, instead of implicitly depending on it via magic in the TRAVERSE algorithm, which magic proved insufficient in cases revealed by the above bugs. It propagates upward in the component hierarchy, rather than downward, like the TRAVERSE algorithm used to systematically propagate dependencies. * TRAVERSE was gutted out and factored into reusable higher-order functions, which not only fixes the above conceptual bug, but makes for semantics that are simpler to implement, possible to understand, easier to extend, and less limited in expressivity. TRAVERSE doesn't automatically propagate operations downward the component hierarchy anymore; propagation is now implemented through methods on COMPONENT-DEPENDS-ON. * COMPONENT-DEPENDS-ON is now more powerful: it can express dependencies on arbitrary operation objects (i.e. operations of the same class can have different options) acting on arbitrary component objects (i.e. not necessarily siblings of the current component). The ASDF object model can now express arbitrary build graphs, with special magic no more for children components vs other dependencies. * Internals have been refactored, and some sorry features were excised. The semantics of OPERATION-DONE-P is simplified and now well-specified. The dubious DOFIRST mechanism is gone. FIND-COMPONENT will pass component objects through, and a corresponding FIND-OPERATION replaces MAKE-SUB-OPERATION. Several internal accessors were renamed. * IF-FEATURE is a new attribute of components that accepts an arbitrary feature expression such as (:and :sbcl (:or :x86 :x86-64)), which when defined is a precondition to the component being enabled; when the expression is false, the component is disabled, and dependencies to it are removed. It replaces the misguided :if-component-dep-fails attribute of modules and the accompanying :feature feature, which were dropped. They were not very expressive, clunky to use, and relied on baking conceptually dubious non-local behavior in the old TRAVERSE algorithm, which belied the object model and defeated other features. Unhappily, this breaks backwards compatibility for the few who used it, namely sb-grovel, nibbles and ironclad. A minimal compatibility layer allows ASDF 2.27 or later to load the SB-GROVEL of old versions of SBCL, but it won't suffice to load an old version of nibbles or ironclad; please upgrade them a recent version that was updated to support ASDF 2.27. * ASDF2.27 was added to the *features* so you can detect the presence of this massively updated ASDF and its new features with #+asdf2.27. This is *not* ASDF3, for we painfully maintained backwards compatibility. * Portability updates were done for ABCL, CLISP, CMUCL, ECL, LispWorks Personal Edition, MKCL, XCL. * Documentation received some small updates. * Tests were updated, notably including support for asdf-bundle. They were successfully run on ccl clisp sbcl ecl ecl_bytecodes cmucl abcl scl allegro lispworks allegromodern xcl. Manual tests were run on lispworks-personal-edition genera. Untested were cormancl mkcl rmcl. * POIU was updated. This extension is distributed separately from ASDF. It is now much simplified and made more correct by construction by reusing the new TRAVERSE infrastructure of ASDF and now being able to assume the ASDF object model is a complete description of dependencies. On SBCL it will handle deferred warnings from background compilations. On platforms that cannot fork it will gracefully fall back to serial compilation. * SLIME support for ASDF was significantly enhanced. It is distributed separately from ASDF, with SLIME. See latest checkin from 2012-12-26. We recommend that in your ~/.swank.lisp you should: (in-package :swank) (defparameter *upgrade-asdf-p* t) (pushnew 'try-compile-file-with-asdf *compile-file-for-emacs-hook*) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He wa'n't no common dog, he wa'n't no mongrel; he was a composite. A composite dog is a dog that is made up of all the valuable qualities that's in the dog breed — kind of a syndicate; and a mongrel is made up of all riffraff that's left over. — Mark Twain

3 7

[asdf-devel] asdf bundle and do-first
by Faré 03 Jan '13

03 Jan '13

Another conceptual bug in ASDF. Checking the plans generated for load-fasl-op, I find that with my new ASDF 2.26.x (currently x=45), it systematically loads all the individual .fasl's before it loads the .system.fasl, defeating the purpose. This was not the case before: in 2.26.7 and earlier, the (separately distributed before 2.26.7) asdf-bundle has load-fasl-op depend-on fasl-op which depends-on compile-op, which doesn't depend-on load-op, but do-first's on it — so that the dependency is only considered but if the immediate timestamp of the fasl is older than the timestamp of its lisp source. If all fasl's satisfy this local timestamp check, nothing is recompiled, none of the do-first is triggered, and the load-op's are not attempted. But the old design also fails to propagate timestamp information from the other files that are depended-on, let alone do-firsted. Therefore, there is some notion in the old do-first that must be preserved in the new ASDF, even though the previous design was conceptually flawed overall. In the end, what must be preserved is that some dependencies need to have been performed in the current image, whereas other dependencies only need to have been performed in some previous (or current) image. This means a dependency can be walked over twice: once in without "image-necessity", and a second time with it. Without image-necessity, only the timestamp matters; with image-necessity, whether it was performed in current image matters, too. Presumably, a node with output-files clears the image-necessity flag before checking timestamps, but re-traverses its dependencies with the flag if the timestamp check fails. A node with no output-files on the other hand is done for in-image effect, and heeds and propagates the image-necessity flag to its subnodes. I believe that it's possible to continue the current strategy of traversal without an explicit graph, though nodes may now be traversed twice: once without image-necessity, once with. Sigh. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Success is getting what you want. Happiness is wanting what you get. — Dale Carnegie

1 1