asdf-devel

asdf-devel@common-lisp.net

1378 discussions

[asdf-devel] suggestion for aserve.asd
by Robert Goldman 20 Dec '12

20 Dec '12

You don't need acl-file any more. You can now use cl-source-file.cl, which is exported from ASDF. You can make legacy-acl-source-file inherit from cl-source-file.cl and cut the ACL-FILE class entirely. That will take care of using the .cl extension instead of .lisp. Cheers, r

3 3

[asdf-devel] Insufficient :if-component-dep-fails compatibility for nibbles
by Faré 18 Dec '12

18 Dec '12

stassats remarks that my if-component-dep-fails compatibility code might be enough to compile sb-rotate-byte, but nibbles not so. Indeed, it so appears that the semantics of the (feature ...) pseudo-operation (note: note a pseudo-component) is that a component depends on not just on its local feature, but on all the recursive features of all the components and subcomponents it depends-on. This means that you really can't untangle that from the entire traverse protocol. Moreover, since the visit protocol does not cache dependency on a missing feature, the previous implementation had cost O(n^2) on the size of a module, instead of O(n). I really don't want to tuck that horror back into the traversal algorithm. I implemented the "compatibility mode" with an eye on all people with old versions of SBCL, who may have a hard time upgrading it. On the other hand, I believe that people who upgrade asdf can easily upgrade nibbles, too. Nathan, can you use #+sbcl and similar other conditionals in your nibbles.asd instead of :if-component-dep-fails? That would really help with compatibility, without requiring ASDF to support something I really want out of the implementation. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Vision is the art of seeing what is invisible to others. — Jonathan Swift

1 0

[asdf-devel] prepare-op
by Faré 18 Dec '12

18 Dec '12

I have finally fixed an essential "dependency problem" of ASDF by introducing a new operation "parent-load-op" which I'm going to rename "prepare-op". This in turns opens the way for many many improvements to the internals, which will make ASDF easier to explain and extend, so expect some work there — and I'm postponing any 2.27 release until that's stable. However, any and all changes introduces potential backwards-incompatibility, and I'll be searching through packages available in quicklisp for any potential such incompatibility before I go ahead with renaming internals. Some details: In 2.26.14, I introduced an operation called parent-load-op, which ensures the dependencies of the parent are loaded before the child is compiled. This ensures the graph produced by following component-depends-on isn't missing crucial dependencies instead of relying on the specifics of the TRAVERSE algorithm (and at long last I really understand what rpgoldman was telling me while we were writing the ASDF 2 paper). I had to tweak traverse, because parent-load-op trickles upward, not downward, the component hierarchy: you depend-on the parent-load-op of your parents, but not on that of your children (that would create a circular dependency). So I had to add a check in TRAVERSE (now — 2.26.20 — in its nice and short visit-children helper) to not automatically propagate subclasses of parent-op dependencies downward. I started thinking about a theory of parent-op and child-op dependencies and how to rewrite the current hierarchy, and as I was doing it found that actually, parent-load-op is not parent-specific: it's actually a dependency-op or prepare-op, that ensures that the image is prepared to compile or load the component, by having already loaded all the dependencies, and it could do it for the "normal" case, too: instead of having load-op or compile-op depend on the load-op of all the dependencies, they could simply depend on prepare-op. Then, we don't need to have traverse know anything about propagating dependencies either downward or upward, it can all be done with two methods on component-depends-on! [which is one of the many misnomers in ASDF: components don't depend on anything, it's (operation . component) pairs, which I'm calling "actions", like in Kent Pitman's article), that are nodes in the dependency graph]. It was all quite a revelation. And so I'll be working on drastically cutting down ASDF, and making the basic dependency graph cleaner, which will make ASDF much simpler and more reliable to extend. Some change I'd like to make if no one uses these internals (or only use them without relying on the current setup to define methods, in which case I can define an alias]: module ==> make it a subclass of both child-component and parent-component system ==> is a parent but not a child, and thus not a module. component-depends-on ==> rename it to action-depends-on if-component-fails ==> I'd like to remove it altogether, it's a crock that doesn't go well with the reified graph model. I suppose I could preserve it the hard way, but would be ugly. mark-operation-done ==> maybe rename to stamp-action ? Or at least mark-action-done. operation-description ==> action-description component-visited-p ==> action-visited-p component-depends-on ==> component-dependencies - likely not possible for backwards compatibility. component-load-dependencies ==> component-depends-on - same. Then component-sibling-dependencies. circular-dependencies-components ==> circular-dependencies-actions, it's a list of (operation . component) pairs hash-tables of symbol, component ==> hash-tables of operations and component, systematically using make-sub-operation make-sub-operation ==> find-operation, to work like find-component, and ignore the first contextual argument if the second is already an object. Canonicalizes named operations while still allowing explicitly different operations of the same class. module-components ==> component-children module-components-by-name ==> component-children-by-name load-fasl-op ==> load-system-fasl-op load-op ==> split between load-op and load-fasl-op (see below). Alternatively, for backwards compatibility, call the latter load-compiled-op My current temporary names would also be renamed: parent-load-op ==> prepare-op (or dependency-op?) visit-children ==> done away with -- replaced entirely by one component-depends-on method. parent-op ==> upward-operation child-op ==> downward-operation Chasing "horizontal" dependencies to siblings under the same parent would be done entirely by prepare-op, which is a subclass of upward-operation. load-op would remain a downward-operation, but would have a separate strategy for deciding whether to delegate to load-source-op, load-fasl-op or load-system-fasl-op or whatever (e.g. some load-instrumented-op extension), based on a generalization of the force / force-not mechanism that would decide the compilation strategy. These ones would NOT be downward-operations, unlike the main load-op which would be a downward-op and propagate everywhere as appropriate. Since parents vs children vs siblings are no longer a fixture of the TRAVERSE algorithm but something reified in the object graph through the normal use of simple methods, you can easily define arbitrary graphs of your choice, instead of ASDF only being applicable to one particular kind of graph shapes. etc. Then the base object hierarchy becomes robust and stable, and it makes sense and is easy to extend, with a nice well-define dependency graph, that can be reified thanks to POIU's current make-parallel-plan. A lot of that (and previous) work was prompted by the desire to keep POIU working as I refactored ASDF, and then the desire to fix ASDF based on the things understood while adapting POIU. POIU interestingly builds an actual complete graph of dependencies from the object model, or at least did it within each system, because, not being able to fix ASDF, Andreas Fuchs had intercepted perform, not operate or perform-plan (which didn't exist then). One ugly inefficient detail Andreas had in his graph-building function was an additional-dependencies argument he used to copy the parent's dependencies on each and every child. That's really ugly. I wanted to get rid of it, but discovered I couldn't... until I invented parent-load-op, which makes it unnecessary, and also promises to fix compute-action-stamp. In any case, Andreas Fuchs certainly deserves a lot of credit for his POIU hack, of which I didn't fully understand the elaborate until I had to fully reimplement it bit by bit, to make it cleaner and fit the new ASDF, and admire each detail he got just right. And he did it all as an add-on, without changing a single line of ASDF itself (though overriding a few definitions — which I promptly made unnecessary once I became ASDF maintainer). That's impressive. And the language is also impressive for allowing such things, at all. But the result was absolute ugly. Being the unexpected co-maintainer of the two sometimes conflicting pieces of software has led me to make both of them ugly no more. PS: In the spirit of people who credit John McCarthy with having discovered Lisp rather than created it, I like think ASDF was a great *discovery* made by Dan Barlow, rather than some badly unfinished program he created; I'm only just reaching what I believe is the bottom of the matter, and it's much nicer than I thought it was, or than Dan probably understood, either, at the time. PPS: I was going to sent a mail to this list, then it grew very long into what would be a blog post, then two, etc. Scrap it. Back to a notice to the list. Still grew up to be too long. Oh well. I hope at least one of you read that and wasn't bored. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Politics is the only profession that does without learning, probably because those who suffer from mistakes are not the same as those who make them. — Achille Tournier, Pensées d'automne

4 23

[asdf-devel] Latest patch fixes all the tests for me...
by Robert Goldman 18 Dec '12

18 Dec '12

Haven't yet run the tests with upgrade. Will do that next. cheers, r

1 0

[asdf-devel] Problems with tests: static file types
by Robert Goldman 18 Dec '12

18 Dec '12

The failing tests all seem to have a variation on failure to read the TYPE slot of static files: Error while trying to load definition for system test-module-pathnames from pathname /Users/rpg/lisp/asdf/test/test-module-pathnames.asd: The slot TYPE is unbound in the object #<STATIC-FILE "test-module-pathnames" "sources/level1" "level2/static.file"> of class #<STANDARD-CLASS STATIC-FILE>.

3 2

[asdf-devel] Just re-tested ASDF
by Robert Goldman 18 Dec '12

18 Dec '12

I pulled from git this morning and re-ran the tests and now the tests fail on CCL on Mac OS X: Using /Users/rpg/ccl/dx86cl64 --no-init --quiet --batch Ran 38 tests: 34 passing and 4 failing failing test(s): asdf-pathname-test.script test-module-pathnames.script test-static-and-serial.script test-system-pathnames.script That's the first test to run. ACL 8.2 fails as well, which suggests this might be a failure in the ASDF logic, instead of how it ties to a particular implementation. More information as I gather it. best, r

1 0

[asdf-devel] Please test ASDF 2.26.24
by Faré 16 Dec '12

16 Dec '12

Considering that I ran out of energy with respect to ASDF hacking following the marathons of refactoring last week, I'm declaring 2.26.24 a candidate for release of 2.27. Please test thoroughly, since this is quite a MAJOR release, with a lot of significant changes. The only reason I'm not bumping the version to ASDF 3 is that this would needlessly break compatibility with who depends-on (:version "asdf" "2.XXX") when ASDF 2.27 is probably compatible with most every system out there. There are a few incompatibilities, though. Here are the changes since 2.26: * ASDF-BUNDLE was merged into ASDF. This notably means fewer headaches for users of ECL; it also means a new load-fasl-op feature for all users of a modern CL, making it easier to deliver software as a single fasl. All remnants of its predecessor asdf-ecl were removed. * Two related significant *decades-old* bugs were fixed (see note 1): (a) changes in dependencies now trigger a rebuild across systems, and https://bugs.launchpad.net/asdf/+bug/479522 (b) timestamps are now correctly propagated https://bugs.launchpad.net/asdf/+bug/1087609 In olden days, some have argued for not rebuilding systems that :depend-on a modified one as a "feature". It really isn't. On the other hand, if they want to explicitly prevent a rebuild, the actual feature :force-not is there to help them (released as part of 2.21 in April 2012). * In the process, an essential bug in the ASDF object model was resolved, by introducing a new operation PREPARE-OP corresponding to "loading the dependencies of a component and its parents" and required before to load or compile the component, instead of depending on the implicit behavior of the TRAVERSE algorithm, which was breaking down in the above bugs. * The TRAVERSE algorithm was gutted out. The above conceptual bug was fixed, and instead it has been factored into reusable higher-order functions. It no longer assumes all operations propagate their dependencies downwards, which is not the case with PREPARE-OP where they propagate upwards. Instead, operations can easily propagate whichever way they want using a one-liner defmethod to COMPONENT-DEPENDS-ON. * COMPONENT-DEPENDS-ON is now more powerful: it can express dependencies on arbitrary operation objects (i.e. operations of the same class can have different options) acting on arbitrary component objects (i.e. not necessarily siblings of the current component). The ASDF object model can now express arbitrary build graphs, with special magic no more for children components vs other dependencies. * Many internals have been refactored, and some sorry features were excised. The semantics of OPERATION-DONE-P is now well-specified. The dubious DOFIRST mechanism is gone. FIND-COMPONENT will pass component objects through, and a corresponding FIND-OPERATION replaces MAKE-SUB-OPERATION. Some internal accessors were renamed. * The dubious :feature and :if-component-dep-fails :try-next features were removed. They were not very expressive, clunky to use, and relied on baking conceptually buggy behavior in the old TRAVERSE algorithm. Unhappily, this breaks backwards compatibility for the few who used it. Happily, a much cleaner :if-feature component option was added. However, if you use it you then depend on a recent ASDF 2.27 or later. * POIU was updated. It is still an extension not distributed with ASDF. It is now much simplified and made more correct by construction by reusing the new TRAVERSE infrastructure of ASDF and now being able to assume the ASDF object model is a complete description of dependencies. On SBCL it will handle deferred warnings from background compilations. On platforms that cannot fork it will gracefully fall back to serial compilation. * The documentation was updated a little. The test suite was updated. * There were some portability updates for ABCL, CMUCL, MKCL. Happy hacking! Notes 1- ASDF inherited those long-standing dependency bug from MK-DEFSYSTEM, which probably inherited it from its predecessors, and so these related bugs have probably lived for over 30 years, for as long as defsystem existed! I haven't seen the proprietary defsystems used by e.g. Allegro or LispWorks, but the fact that e.g. LispWorks has caused-by and requires dependencies (akin to in-order-to and do-first in asdf) makes it very suspect in my eyes. And yet, it all hasn't seem to affect people too much — certainly not so much that they would go a-hunting and a-fixing them! Indeed, if you're working a single system at a time, until it works, without interrupting the build (instead fixing the code and using a restart after errors), without leaving and restarting your image, then the bug won't hit you. If you occasionally abort a build, but no file depends on a modified macro or other compile-time state that wasn't itself modified to accommodate the change, then you won't hit. Therefore, it might never have been seen by a normal Lisp Machine developer. But when you develop plenty of Lisp libraries at a time, each its own system, while often restarting your Lisp to build from clean, or if you have a test-foo system separate from your foo system, then these bugs will become quite noticeable. And so people now notice and care. That's interesting in what that says about how Lisp hacking has evolved. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The truth of a proposition has nothing to do with its credibility. And vice versa. — Robert Heinlein, "Time Enough For Love"

1 0

[asdf-devel] Guidelines for ASDF system authors
by Felix Lange 16 Dec '12

16 Dec '12

Hi, I've been thinking a lot about the structure of Common Lisp projects lately and would like to turn those thoughts into a public discussion. I believe that some written guidelines for ASDF system authors could really improve how the Common Lisp ecosystem functions as a whole. The documentation on the ASDF website is comprehensive with respect to the capabilities of its implementation, but there are few guidelines on how developers are supposed to structure their systems. I perceive a need for a manual chapter or document that consolidates best practices advice and somehow documents the `intended' use of ASDFs countless features. This information is currently scattered all over the ASDF manual. Getting the big picture is possible but kind of difficult. The new section `Extensions' on the ASDF project website as well as the FAQ section in the manual are very much a step in the right direction. Some of the questions that I think need a documented solution which could also act as a community convention: - How to properly conditionalize on *FEATURES*. This is touched briefly in manual section 6.2.1.3 `Required features'. There's a note explaining that excluding components through the #+ syntax is a bad idea, and that the correct way of doing it i.e. (:file "foo" :depends-on (:feature :sbcl)) does not work at this time. In the meantime, Nikodemus has written MADEIRA-PORT for this exact purpose. - Implementing TEST-OP The manual tells me that `it has proven difficult to define how the test operation should signal its results' and that TEST-OP does nothing by default. Current practice seems to be defining TEST-OP as a method that loads another system and then somehow executes all tests, failing the operation when some test did not pass. This is noted in the FAQ section and should really be the official way. - How to specify optional dependencies in the `correct' way? Suppose I'm working on a library that can use either IRONCLAD or CL+SSL to generate random bytes using a strong crypto algorithm. I don't want to clutter the final application image with both crypto libraries but rather use whatever happens to be the application developer's choice. I have several options to do this. 1) Use :WEAKLY-DEPENDS-ON to rely on the application's transient dependencies to load either one. This is obviously a nice solution, but I have to document the requirement of loading at least one of them in the README and figure out which one to use at runtime. 2) Define two systems, MY-LIBRARY-IRONCLAD and MY-LIBRARY-CL+SSL. This would force the user to require one of them in order to use the library. I also need to document this, but do not have to switch at runtime. 3) Solve the problem outside of ASDF - How much `formalization' of the build process pays off in the long run? ASDF could obviously have more uses if people followed a common guideline. What's the advantage of custom component classes vs. a custom operation that is hooked into COMPILE-OP via :IN-ORDER-TO? What's used by existing systems? - Whether or not to define a package for the system definition The documentation uses (IN-PACKAGE :ASDF) in section 5.1. Common practice seems to be defining a package inside the system definition file if the system uses custom component classes or features that require toplevel forms other than ASDF:DEFSYSTEM. :DEFSYSTEM-DEPENDS-ON is also an option. - What are some good practices when building applications with ASDF? As the manual points out, `ASDF presents three faces'. People who want to use the ASDF for applications have very different needs from library developers and will often want tight control over their dependency graph. Are versioned dependencies usable and practical? How can I override decisions made by library authors in my application's build process? What's a good way to prevent package name conflicts? Some `war stories' from commercial application development could really help others arrive at a good solution quickly. I see that `style guide for .asd files' is a TODO item in the manual, so it seems some of those question are already being addressed. I am willing to do documentation work as soon as I understand people's intentions. Looking forward to your point of view, Felix

2 1

[asdf-devel] if-component-dep-fails
by Faré 14 Dec '12

14 Dec '12

Dear Nathan, at least as far as quicklisp-included software goes, your ironclad and nibbles seem to be the only user of ASDF's :if-component-dep-fails and :feature features. These features are a crock and I'd like to plainly remove them. In the future, I might provide a better-thought-out replacement that actually works. In the meantime, the backwards-compatible solution would be to use #+sbcl and such; yes it breaks an abstraction barrier — but at least it doesn't require ASDF itself to break its own abstraction barrier internally. Could you "fix" your .asd's to not use these features I'd like to drop? Does any other person who cares about ASDF cares about these features? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you make people think they're thinking, they'll love you; but if you really make them think, they'll hate you. — Don Marquis

3 8

[asdf-devel] :if-feature
by Faré 14 Dec '12

14 Dec '12

ASDF 2.26.21 removed the horribly clunky :if-component-dep-fails. ASDF 2.26.22 added a much more meaningful :if-feature. Any component can use the :if-feature option, which can be an arbitrary keyword expression, just like #+. The difference is that unlike #+ it doesn't magically change the *package* to keyword, so you have to include the colons, as in :if-feature (:and :sbcl :x86-64) The upside is that its semantics is simple, doesn't break the object model, and allows for arbitrary expressions. The downside is that it is not backwards compatible. It will be released with 2.27 only. (If you have a better name than :if-feature, now is the time to suggest it.) Hopefully, I will remember to include a #+asdf2.27 before I release. Please test asdf 2.27 and give me feedback. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The worst thing that can happen to a good cause is, not to be skillfully attacked, but to be ineptly defended. — F. Bastiat

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

asdf-devel