- asdf-devel - mailman3.common-lisp.net

[asdf-devel] ASDF 3 news
by Faré 29 Jan '13

29 Jan '13

Getting closer to release. The codebase is stable now, and I'm just making minor adjustments. Now at 2.26.164. I'm going to release 2.27 as soon as all tests pass on all implementations. The magic feature is #+asdf3, and 2.27 should be considered a pre-release. Hopefully by next month, after some testing, we can declare the next one to be ASDF 3.0. There were previously performance issues, with ASDF 2.26.152 being over 37 times slower than 2.26 at making a plan for iolib than it was before (3s vs .08s). After a day spent optimizing away a lot of truename queries and other pathname computations, ASDF is back at being 70% slower with default settings (while doing much more work behind the scene), and slightly faster if you disable all truename queries via (setf *resolve-symlinks* nil) -- but goodbye linkfarms if you do. After the last round of upgrade cleanups, hot upgrade now works without punting on recent versions of ABCL or CMUCL. But still not on CLISP. There seem to be various optimizations in CLISP FASLs that defeat everything I've tried so far except punting by renaming away the ASDF package as the very first thing. Many thanks to Stelian Ionescu, Stas Boukarev, Anton Vodonosov and Robert Goldman for their help with testing. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org You may easily play a joke on a man who likes to argue — agree with him. — Edgar Waston Howe

1 0

[asdf-devel] ASDF 3?
by Faré 27 Jan '13

27 Jan '13

Many people have suggested making the ASDF 2.27 release an ASDF 3.0 release instead. I was previously hesitant, wanting to signal that I preserve compatibility, and not making it a release that doesn't preserve version-satisfies, since the major number is different. But I realize that (a) my code may be the only one that ever checks the version of ASDF, and I'm having to update it anyway, and (b) considering at the list of new features and the major rewrite of previous features, it's actually a bigger change in the making than there was from ASDF 1 to ASDF 2. Therefore I'm changing my mind and I am considering calling the next release ASDF 3, with feature :asdf3 instead of :asdf2.27. If you have objections, speak in the next few days. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org How can you find a person who loves you the way you are, if you constantly lie about who you are?

3 3

[asdf-devel] ASDF-USER ?
by Faré 27 Jan '13

27 Jan '13

I think the whole "temporary package" thing is a failure, and we should just have an ASDF-USER package that uses ASDF. Does anyone have any objection to having .asd files be loaded in a shared ASDF-USER rather than a temporary ASDF~D package? If you want to define your own private functions in their private namespace, there's defpackage for you. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There are two types of people in the world: People who think there are two kinds of people, and people who don't.

3 10

[asdf-devel] New features in ASDF 2.27
by Faré 25 Jan '13

25 Jan '13

OK, so ASDF 2.27 has been 2 months in the doing, plus one month initial hyatus, that's three months since 2.26. Since then it has a bit over doubled in size, and the testing of it has caused a lot of breakage, for which I apologize again. Here is a list of the major changes so far, glossing over buglet fixes and internal changes that should be invisible to users: * ASDF-BUNDLE was merged into ASDF. This notably means fewer headaches for users of ECL; it also means a new load-fasl-op feature for all users of a modern CL, making it easier to deliver software as a single fasl. After release, I plan to offer a patch so SBCL use that for its contrib's. All remnants of asdf-bundle's predecessor asdf-ecl were removed. * *decades-old* bugs of notable significance were fixed: (a) changes in dependencies now trigger a rebuild across systems, and https://bugs.launchpad.net/asdf/+bug/479522 (b) timestamps are now correctly propagated (they weren't at all!) https://bugs.launchpad.net/asdf/+bug/1087609 In olden days, some have argued for not rebuilding systems that :depend-on a modified one as a "feature". It really isn't. On the other hand, it is now possible to explicitly prevent a system from being rebuilt, using an actual feature, :force-not, which was released in 2.21 in April 2012. Someday I will write about those bugs. * PREPARE-OP was introduced to fix a conceptual bug in the ASDF object model. It corresponds to "loading the dependencies of a component and its parents" and is explicitly depends-on'ed by LOAD-OP and COMPILE-OP of the component, instead of their implicitly depending on it via the TRAVERSE algorithm, which implicit dependency proved fatal in cases revealed by the above bugs. It propagates upward in the component hierarchy, rather than downward, like the TRAVERSE algorithm used to systematically propagate dependencies. * TRAVERSE was gutted out and factored into reusable higher-order functions and objects, which not only fixes the above conceptual bug, but makes for semantics that are simpler to implement, possible to understand, easier to extend, and less limited in expressivity. TRAVERSE doesn't automatically propagate operations downward the component hierarchy anymore; propagation is now implemented through methods on COMPONENT-DEPENDS-ON. * COMPONENT-DEPENDS-ON is now more powerful: it can express dependencies on arbitrary operation objects (i.e. operations of the same class can have different options) acting on arbitrary component objects (i.e. not necessarily siblings of the current component). The ASDF object model can now express arbitrary build graphs, with no more special magic for children components vs other dependencies. * COMPONENT-DO-FIRST is no more. It used to specify some dependencies that were skipped if no re-build was triggered based on local timestamps. Instead, COMPONENT-DEPENDS-ON is used for all dependencies, and all are consulted to correctly propagate timestamps. However actions meant for in-image side-effects are included only if needed, and might be visited twice by TRAVERSE, first without the needed-in-image-p flag, another time with. * FORCE and FORCE-NOT now correctly distinguish T from :ALL in addition to a list of system name specifiers, as per the original specification (which was never implemented in ASDF 1). Additionally, builtin systems (such as SB-BSD-SOCKETS on SBCL) cannot be forced (which doesn't work on SBCL). * IF-FEATURE is a new attribute of components that accepts an arbitrary feature expression such as (:and :sbcl (:or :x86 :x86-64)), which when defined is a precondition to the component being enabled; when the expression is false, the component is disabled, and dependencies to it are removed. It replaces the misguided :if-component-dep-fails attribute of modules and the accompanying :feature feature, which were dropped. They were not very expressive, clunky to use, and relied on baking conceptually dubious non-local behavior in the old TRAVERSE algorithm, which belied the object model and defeated other features. Unhappily, this breaks backwards compatibility for the few who used it, namely sb-grovel, nibbles and ironclad. A minimal compatibility layer allows ASDF 2.27 or later to load the SB-GROVEL of old versions of SBCL, but it won't suffice to load an old version of nibbles or ironclad; please upgrade them a recent version that was updated to support ASDF 2.27. * CONCATENATE-SOURCE-OP is a new operation that builds a single Lisp file from all the source files in a system, which allows you to develop in a structured way with many compact files, yet deliver a large single file. Other related operations allow you to load that source file, or compile and load it, or do the same while including all other system dependencies. ASDF was broken up into lots of small files, to be delivered that way. * FOO/BAR/BAZ names will be recognized by defsystem as having to be located in a file called foo.asd. This is backward compatible in the case where you somehow arranged to load foo.asd before to use the other subsystems, except now ASDF can find the subsystems by name without the main system definition having been loaded. See uses of function PRIMARY-SYSTEM-NAME. * BUILD-OP is a generic operation that can do the "right thing" for each system, without users having to maintain the knowledge of what the right thing is for each system they may indirectly depend on. * Convenience methods were added to most exported generic functions, so you can (input-files 'compile-op '(system1 "file1")) instead of (input-files (make-instance 'compile-op) (find-component 'system1 "file1")) This notably makes it much easier to interact with ASDF at the REPL and debug your system definitions or ASDF extensions (or modifications). * Packages were introduced as ASDF was broken up into small files. Each file now has its own package, importing and exporting functionality, following the style made popular by faslpath and quick-build. ASDF does not yet include builtin support for quick-build or faslpath systems and dependencies, however. Possibly in ASDF 2.28. * ASDF-DRIVER is a separately-usable library for Common Lisp runtime support. It is transcluded in asdf.lisp (i.e. when delivering ASDF as a separate bootstrap file, the contents of ASDF-DRIVER have been included in it), but can also be used as a separate ASDF system on top of an old ASDF (within limits: CLISP and XCL still need a recent ASDF). The package ASDF/DRIVER (with short nickname :D) re-exports all the functions that ASDF needed to run portably. * Pathnames are a notable part of ASDF-DRIVER, required to deal portably with pathnames despite the many bugs and variations from platform to platform. See notably in pathname.lisp, MERGE-PATHNAMES*, SUBPATHNAME, PARSE-UNIX-NAMESTRING, PARSE-NATIVE-NAMESTRING, ENSURE-PATHNAME, and many many more. * Pathname use by ASDF was also improved in countless small ways, closing more buglets that appeared in as many corner cases. For instance, system pathname defaulting was improved, and systems not associated with a file will now have NIL as their pathname, which will otherwise be correctly treated. Many things that would vary depending on implementation will work better. Also, although we cannot strongly enough disrecommend the use of so-called "logical pathnames", especially where portability matters in any way, we support them better than ever within the limitations of the implementation. * Image lifecycle support is included in ASDF-DRIVER (and thus ASDF); this standardizes things like access to *COMMAND-LINE-ARGUMENTS*, declaring an entry-point to a program, hook functions and/or a prelude to run before to start the main entry, and a postlude and/or hook functions to run before to dump an image, dumping or creating an image, etc. * PROGRAM-OP allows you to create standalone executables on implementations that support it: CLISP, Clozure CL, CMUCL ECL, LispWorks, SBCL, SCL. A hello-world-example system shows demonstrates how to use it. * Run-program is a replacement for the old run-shell-command. Unlike run-shell-command, it is portable to Windows as well as Unix, it can avoid going through a shell and will sensibly escape arguments when it does go through a shell, and it has a usable interface to either capture program output or run the program interactively. This supersedes run-program/ previously available from xcvb-driver, including several improvements and portability fixes. * Conditions can be selectively muffled around compilation thanks to the an around-compile hook and the *UNINTERESTING-COMPILER-CONDITIONS* parameter, and around loading with *UNINTERESTING-LOADER-CONDITIONS* which is _appended_ to the former during loading. This supersedes the functionality previously available from xcvb-driver and its thin wrapper asdf-condition-control, which was retired. * Deferred warnings for yet undefined functions, that in a single-session build are usually signaled at the end of a compilation unit, will be saved and reassembled to make each system a compilation unit of its own, even when compilation of file happens across multiple sessions; only CCL and SBCL support this feature at this time. I welcome patches for other compilers, for which ASDF will _not_ wrap an implicit WITH-COMPILATION-UNIT around its operate method anymore. * COMPILE-FILE* was beefed up to support all these features and more, and the Right Thing(tm) on all implementations, including support for their various extensions. * Internals have been refactored, and some sorry features were excised. The semantics of OPERATION-DONE-P is simplified and now well-specified. FIND-COMPONENT will pass component objects through, and a corresponding FIND-OPERATION replaces MAKE-SUB-OPERATION. Many internal accessors were renamed, after checking on Quicklisp that no one was using them. * Portability updates were done for ABCL, CLISP, Clozure CL, CMUCL, ECL, LispWorks Personal Edition, MKCL, XCL. Working support for GCL 2.6 was restored, with notable limitations including lack of support for output-translations or logical-pathnames. * TRUENAME resolution can be reliably turned off in all cases where ASDF was using them by setting the variable *RESOLVE-SYMLINKS* to NIL. This is useful if your operating system makes that operation slow, or if your build system uses symlinks to content-addressed storage in which case TRUENAME just does the wrong thing. * VERSION strings can now be specified as having to be fetched from a file using a (:read-file-form <path>) specification, that can use an optional :at keyword argument to specify a form or subform other than the first form in the file, as per the utility asdf-driver:access-at. For instance, if your file specials.lisp is a defpackage, an in-package and a defparameter of the version, then you could use: (:read-file-form "specials.lisp" :at (2 2)) This makes it easier for developers to manage versioning of one or multiple systems with a single master version location. * Self-Upgrade of ASDF will happen automatically as the first thing before any attempt to build anything, as it was determined that doing it in the midst of a build breaks everything, and it is not otherwise possible to reliably detect in advance whether ASDF may be needed in such midst. If you don't want an ASDF upgrade, just don't configure your source-registry to include any copy of ASDF. * DEFINE-PACKAGE is provided as a replacement to DEFPACKAGE that supports hot upgrade of software where some packages have changed. It also provides new forms :MIX and :REEXPORT which can be quite practical. * ASDF2.27 was added to the *features* so you can detect the presence of this massively updated ASDF and its new features with #+asdf2.27. This is *not* ASDF3, for we painfully maintained backwards compatibility, though it may have been temporarily broken during development. * Documentation received some updates. * Tests were updated, notably including support for bundles, encodings, run-program, and more. They were successfully run on ccl clisp sbcl ecl ecl_bytecodes cmucl abcl scl allegro lispworks allegromodern xcl Manual tests were run on gcl2.6 genera lispworks-personal-edition. Untested remain cormancl mkcl rmcl. * ASDF-DEBUG is a simple utility to make your favorite debugging primitives available in the package you're debugging. See contrib/debug.lisp and the asdf/utility:*asdf-debug-utility* variable. * ASDF-UTILS is now but an empty shell that depends-on ASDF-DRIVER. The ASDF-DRIVER package has the ASDF-UTILS nickname, and exports all the functionality that used to be in ASDF-UTILS, and much more, minus a few misdesigned things from ASDF1 that you ought not to have been using, and for which there are non-broken replacements. ASDF-UTILS was distributed separately from ASDF; ASDF-DRIVER is available both as part of ASDF 2.27 and later and separately. * POIU was updated. This extension is still distributed separately from ASDF. It is now much simplified and made correct by construction by reusing the new TRAVERSE infrastructure of ASDF and now being able to assume the ASDF object model is a complete description of dependencies. On SBCL it will handle deferred warnings from background compilations. Where forking is not possible or not supported, it will gracefully fall back to serial compilation. * INFERIOR-SHELL (distributed separately) was updated to use asdf-driver instead of xcvb-driver, providing a richer interface to the RUN-PROGRAM functionality. * SLIME support for ASDF was significantly enhanced. It is distributed with SLIME, separately from ASDF. See latest slime checkin from 2013-01-20. We recommend that you should include slime-asdf in your slime-setup and that in your ~/.swank.lisp you should include: (in-package :swank) (pushnew 'try-compile-file-with-asdf *compile-file-for-emacs-hook*) This requires ASDF 2.014.6 or later (as used by older quicklisp releases). There's one more feature pending, courtesy of Stelian Ionescu, that will hopefully make it to the release that I hope will happen next week. In the meantime, I welcome all testing of the new ASDF and/or its ASDF-DRIVER. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Memory is like an orgasm... it's better when you dont have to fake it. — Seymour Cray

2 3

[asdf-devel] Another problem with the warnings deferrel
by Robert Goldman 25 Jan '13

25 Jan '13

If you get an error in compilation, and rebuild the system, even on SBCL, it doesn't seem like the warnings lists get cleared.... At least I see ones that are clearly fixed reappear when I use the restart.... Best, r

2 1

[asdf-devel] distributing asdf
by Faré 24 Jan '13

24 Jan '13

[Edited subject] >> That begs the question: how should asdf be distributed? > > Does it? I thought we were talking about asdf-driver. > Well, asdf-driver is currently developed together with asdf, which crucially depends on it. So how to distribute one with or without the other is a question; hopefully made less important by quicklisp. Also, distributing asdf-driver separately from asdf makes for quadratic growth of potential upgrade issues to be wary of. >> As asdf.lisp only? > > Yes. As it always has been. > That is not the whole truth. ASDF has been distributed in many ways; in order of chronology and progress, I can cite: * as a source file that you may load manually. * as code builtin to the image on Debian with C-L-C. * as part of some batteries-included Lisp distribution. * as a REQUIRE module provided by the Lisp implementation. * as a system and source file ready to upgrade any of the previous with a more advanced version on which you depend. The latter solution has only worked since ASDF 2 (indeed, was the whole point of ASDF 2), and has allowed for some decoupling between what providers offer and was users may require, thus fostering nicer development. These days, all actively-maintained implementations (except maybe SCL?) provide ASDF 2 as a REQUIRE module. Some of them are quite old (CLISP still provides 2.011), but all of them are able to locate a new ASDF in the source registry and upgrade themselves. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Economic illiteracy often leads one to take for wealth creation or cost reduction what is only a forced displacement of activity, with no primary gain, and a lot of secondary costs and negative side-effects. — Faré

2 18

[asdf-devel] clisp and asdf
by Faré 22 Jan '13

22 Jan '13

Of all the lisp implementations, CLISP has given me the most trouble with ASDF upgrade. Unlike all other Lisps, it won't let me undefine functions and redefine them in the same fasl file. Somehow, I always end up with CLISP either complaining that a function signature doesn't match (was or wasn't a generic function with an incompatible lambda-list), because I couldn't undefine it, or that it is undefined and never redefined, because I undefined it and CLISP throws away my redefinition somehow. I tried to selectively unintern symbols for functions thus upgraded, but that also fails. Yet all these things work in about all other Lisps. In the end, I punted: I just rename the old ASDF package to ASDF-${version} early on, but that messes with any package that :use's ASDF, and then CLISP will complain again, unless ASDF is the very first thing upgraded before anything uses it. Could CLISP be made more upgrade-friendly? (another thing I don't understand that may or may not be related is the .lib output files produced together with a fasl, when and how they are used, and what I should or should not do with them.) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Nostalgia isn’t what it used to be.

2 3

[asdf-devel] Upgrade issues
by Faré 21 Jan '13

21 Jan '13

ABCL is one of the three implementations, together with CLISP and CMUCL, where I ultimately punted on ASDF hot-upgrade, by just trying to rename away the ASDF package if it's too old. I don't know if you're interested in making the kind of package surgery I was indulging in work, but it involves rehoming symbols from ASDF to subpackages such as ASDF/COMPONENT, including for class slot names. Anyway, please test the latest ASDF with ABCL, I think it's ripe for release, but I'd like your feedback considering the massive amount of hacking that happened. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If the human mind were simple enough to understand, we'd be too simple to understand it. — Pat Bahn

2 2

[asdf-devel] Just pulled an update and now CCL fails the tests
by Robert Goldman 20 Jan '13

20 Jan '13

I am now getting 8 test failures: Ran 42 tests: 34 passing and 8 failing failing test(s): test-bundle.script test-compile-file-failure.script test-concatenate-source.script test-force.script test-nested-components.script test-package.script test-touch-system-2.script test-xach-update-bug.script Testing: test-bundle.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-bundle.script")' ; Registering #<SYSTEM "test-bundle-1"> ; Registering #<SYSTEM "test-bundle-2"> TEST-BUNDLE ASDF-TEST::BUNDLE-1 => #P"/Users/rpg/lisp/asdf/build/fasls/ccl-1.8-f95-macosx-x64/asdf/test/test-bundle-1\\.system.dx64fsl" ASDF-TEST::BUNDLE-2 => #P"/Users/rpg/lisp/asdf/build/fasls/ccl-1.8-f95-macosx-x64/asdf/test/test-bundle-2\\.system.dx64fsl" TEST ABORTED: Failed assertion: (PROBE-FILE BUNDLE-1) Testing: test-compile-file-failure.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-compile-file-failure.script")' ; Warning: Warning. ; While executing: #<Anonymous Function #x30200099F98F>, in process listener(1). TEST ABORTED: Failed assertion: (HANDLER-CASE (LET ((*COMPILE-FILE-FAILURE-BEHAVIOUR* :ERROR)) (LOAD-SYSTEM 'TEST-COMPILE-FILE-FAILURE :FORCE T) NIL) (COMPILE-ERROR NIL T)) Testing: test-concatenate-source.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-concatenate-source.script")' ; Registering #<SYSTEM "test-concatenate-source"> ; Loading system definition from /Users/rpg/lisp/asdf/test/file3-only.asd into #<Package "ASDF-0"> ; Registering #<SYSTEM "file3-only"> TEST ABORTED: These two expressions fail comparison with EQUAL: (PRINC-TO-STRING (INPUT-FILES MCSO SYS)) evaluates to "NIL" (PRINC-TO-STRING (LOOP :FOR N :IN '(3 1 2) :COLLECT (SUBPATHNAME *TEST-DIRECTORY* (FORMAT NIL "file~D.lisp" N)))) evaluates to "(/Users/rpg/lisp/asdf/test/file3.lisp /Users/rpg/lisp/asdf/test/file1.lisp /Users/rpg/lisp/asdf/test/file2.lisp)" Testing: test-force.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-force.script")' TEST ABORTED: Failed assertion: (>= (FILE-WRITE-DATE FILE1) FILE1-DATE) Testing: test-nested-components.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-nested-components.script")' ; Registering #<SYSTEM "test-nested-components-a"> TEST ABORTED: Invalid pathname #P"test-nested-components-1.asd": Expected an absolute pathname Testing: test-package.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-package.script")' ; Registering #<SYSTEM "test-package"> TEST ABORTED: Invalid relative pathname #P"test-package/" for component ("test-package") Testing: test-touch-system-2.script + /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch --eval '(load "test-touch-system-2.script")' ; Registering #<SYSTEM "test1"> TEST ABORTED: Invalid pathname #P"test1.asd": Expected an absolute pathname Now, change registry so foo is found from xach-foo-2/ load foo yet again. It should see the pathname has changed and load it anew TEST ABORTED: No package named "SECOND-VERSION" cheers, r

4 5

[asdf-devel] Tests still failing on allegro
by Robert Goldman 19 Jan '13

19 Jan '13

I checked test-force. This is failing because after the test-force system is loaded, we have this: ASDF-TEST> (asdf:compile-file-pathname* "file1") #P"/Users/rpg/lisp/asdf/build/fasls/acl-8.2-macosx-x64/root/file1.fasl" ASDF-TEST> (probe-file (asdf:compile-file-pathname* "file1")) NIL This despite the fact that the system is loaded: ASDF-TEST> (asdf:operate 'asdf:load-op 'test-force) #<LOAD-OP > ((#<PREPARE-OP > . #<SYSTEM "test-force">) (#<PREPARE-OP > . #<CL-SOURCE-FILE "test-force" "file1">) (#<LOAD-OP > . #<CL-SOURCE-FILE "test-force" "file1">) (#<COMPILE-OP > . #<SYSTEM "test-force">) (#<LOAD-OP > . #<SYSTEM "test-force">)) Looks like this might be the problem: ASDF-TEST> (asdf:output-files (make-instance 'asdf:compile-op) (asdf:find-component "test-force" "file1")) (#P"/Users/rpg/lisp/asdf/build/fasls/acl-8.2-macosx-x64/asdf/test/file1.fasl") See that the file that is generated by compiling "file1" does not have the same pathname as (ASDF:COMPILE-FILE-PATHNAME* "file1") This seems to give the right answer: ASDF-TEST> (asdf:compile-file-pathname* (asdf:component-pathname (asdf:find-component "test-force" "file1"))) #P"/Users/rpg/lisp/asdf/build/fasls/acl-8.2-macosx-x64/asdf/test/file1.fasl" Is this simply a bug in the test? Best, r

1 0