- asdf-devel - mailman3.common-lisp.net

[asdf-devel] Please test ASDF 2.26.24
by Faré 16 Dec '12

16 Dec '12

Considering that I ran out of energy with respect to ASDF hacking following the marathons of refactoring last week, I'm declaring 2.26.24 a candidate for release of 2.27. Please test thoroughly, since this is quite a MAJOR release, with a lot of significant changes. The only reason I'm not bumping the version to ASDF 3 is that this would needlessly break compatibility with who depends-on (:version "asdf" "2.XXX") when ASDF 2.27 is probably compatible with most every system out there. There are a few incompatibilities, though. Here are the changes since 2.26: * ASDF-BUNDLE was merged into ASDF. This notably means fewer headaches for users of ECL; it also means a new load-fasl-op feature for all users of a modern CL, making it easier to deliver software as a single fasl. All remnants of its predecessor asdf-ecl were removed. * Two related significant *decades-old* bugs were fixed (see note 1): (a) changes in dependencies now trigger a rebuild across systems, and https://bugs.launchpad.net/asdf/+bug/479522 (b) timestamps are now correctly propagated https://bugs.launchpad.net/asdf/+bug/1087609 In olden days, some have argued for not rebuilding systems that :depend-on a modified one as a "feature". It really isn't. On the other hand, if they want to explicitly prevent a rebuild, the actual feature :force-not is there to help them (released as part of 2.21 in April 2012). * In the process, an essential bug in the ASDF object model was resolved, by introducing a new operation PREPARE-OP corresponding to "loading the dependencies of a component and its parents" and required before to load or compile the component, instead of depending on the implicit behavior of the TRAVERSE algorithm, which was breaking down in the above bugs. * The TRAVERSE algorithm was gutted out. The above conceptual bug was fixed, and instead it has been factored into reusable higher-order functions. It no longer assumes all operations propagate their dependencies downwards, which is not the case with PREPARE-OP where they propagate upwards. Instead, operations can easily propagate whichever way they want using a one-liner defmethod to COMPONENT-DEPENDS-ON. * COMPONENT-DEPENDS-ON is now more powerful: it can express dependencies on arbitrary operation objects (i.e. operations of the same class can have different options) acting on arbitrary component objects (i.e. not necessarily siblings of the current component). The ASDF object model can now express arbitrary build graphs, with special magic no more for children components vs other dependencies. * Many internals have been refactored, and some sorry features were excised. The semantics of OPERATION-DONE-P is now well-specified. The dubious DOFIRST mechanism is gone. FIND-COMPONENT will pass component objects through, and a corresponding FIND-OPERATION replaces MAKE-SUB-OPERATION. Some internal accessors were renamed. * The dubious :feature and :if-component-dep-fails :try-next features were removed. They were not very expressive, clunky to use, and relied on baking conceptually buggy behavior in the old TRAVERSE algorithm. Unhappily, this breaks backwards compatibility for the few who used it. Happily, a much cleaner :if-feature component option was added. However, if you use it you then depend on a recent ASDF 2.27 or later. * POIU was updated. It is still an extension not distributed with ASDF. It is now much simplified and made more correct by construction by reusing the new TRAVERSE infrastructure of ASDF and now being able to assume the ASDF object model is a complete description of dependencies. On SBCL it will handle deferred warnings from background compilations. On platforms that cannot fork it will gracefully fall back to serial compilation. * The documentation was updated a little. The test suite was updated. * There were some portability updates for ABCL, CMUCL, MKCL. Happy hacking! Notes 1- ASDF inherited those long-standing dependency bug from MK-DEFSYSTEM, which probably inherited it from its predecessors, and so these related bugs have probably lived for over 30 years, for as long as defsystem existed! I haven't seen the proprietary defsystems used by e.g. Allegro or LispWorks, but the fact that e.g. LispWorks has caused-by and requires dependencies (akin to in-order-to and do-first in asdf) makes it very suspect in my eyes. And yet, it all hasn't seem to affect people too much — certainly not so much that they would go a-hunting and a-fixing them! Indeed, if you're working a single system at a time, until it works, without interrupting the build (instead fixing the code and using a restart after errors), without leaving and restarting your image, then the bug won't hit you. If you occasionally abort a build, but no file depends on a modified macro or other compile-time state that wasn't itself modified to accommodate the change, then you won't hit. Therefore, it might never have been seen by a normal Lisp Machine developer. But when you develop plenty of Lisp libraries at a time, each its own system, while often restarting your Lisp to build from clean, or if you have a test-foo system separate from your foo system, then these bugs will become quite noticeable. And so people now notice and care. That's interesting in what that says about how Lisp hacking has evolved. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The truth of a proposition has nothing to do with its credibility. And vice versa. — Robert Heinlein, "Time Enough For Love"

1 0

[asdf-devel] Guidelines for ASDF system authors
by Felix Lange 16 Dec '12

16 Dec '12

Hi, I've been thinking a lot about the structure of Common Lisp projects lately and would like to turn those thoughts into a public discussion. I believe that some written guidelines for ASDF system authors could really improve how the Common Lisp ecosystem functions as a whole. The documentation on the ASDF website is comprehensive with respect to the capabilities of its implementation, but there are few guidelines on how developers are supposed to structure their systems. I perceive a need for a manual chapter or document that consolidates best practices advice and somehow documents the `intended' use of ASDFs countless features. This information is currently scattered all over the ASDF manual. Getting the big picture is possible but kind of difficult. The new section `Extensions' on the ASDF project website as well as the FAQ section in the manual are very much a step in the right direction. Some of the questions that I think need a documented solution which could also act as a community convention: - How to properly conditionalize on *FEATURES*. This is touched briefly in manual section 6.2.1.3 `Required features'. There's a note explaining that excluding components through the #+ syntax is a bad idea, and that the correct way of doing it i.e. (:file "foo" :depends-on (:feature :sbcl)) does not work at this time. In the meantime, Nikodemus has written MADEIRA-PORT for this exact purpose. - Implementing TEST-OP The manual tells me that `it has proven difficult to define how the test operation should signal its results' and that TEST-OP does nothing by default. Current practice seems to be defining TEST-OP as a method that loads another system and then somehow executes all tests, failing the operation when some test did not pass. This is noted in the FAQ section and should really be the official way. - How to specify optional dependencies in the `correct' way? Suppose I'm working on a library that can use either IRONCLAD or CL+SSL to generate random bytes using a strong crypto algorithm. I don't want to clutter the final application image with both crypto libraries but rather use whatever happens to be the application developer's choice. I have several options to do this. 1) Use :WEAKLY-DEPENDS-ON to rely on the application's transient dependencies to load either one. This is obviously a nice solution, but I have to document the requirement of loading at least one of them in the README and figure out which one to use at runtime. 2) Define two systems, MY-LIBRARY-IRONCLAD and MY-LIBRARY-CL+SSL. This would force the user to require one of them in order to use the library. I also need to document this, but do not have to switch at runtime. 3) Solve the problem outside of ASDF - How much `formalization' of the build process pays off in the long run? ASDF could obviously have more uses if people followed a common guideline. What's the advantage of custom component classes vs. a custom operation that is hooked into COMPILE-OP via :IN-ORDER-TO? What's used by existing systems? - Whether or not to define a package for the system definition The documentation uses (IN-PACKAGE :ASDF) in section 5.1. Common practice seems to be defining a package inside the system definition file if the system uses custom component classes or features that require toplevel forms other than ASDF:DEFSYSTEM. :DEFSYSTEM-DEPENDS-ON is also an option. - What are some good practices when building applications with ASDF? As the manual points out, `ASDF presents three faces'. People who want to use the ASDF for applications have very different needs from library developers and will often want tight control over their dependency graph. Are versioned dependencies usable and practical? How can I override decisions made by library authors in my application's build process? What's a good way to prevent package name conflicts? Some `war stories' from commercial application development could really help others arrive at a good solution quickly. I see that `style guide for .asd files' is a TODO item in the manual, so it seems some of those question are already being addressed. I am willing to do documentation work as soon as I understand people's intentions. Looking forward to your point of view, Felix

2 1

[asdf-devel] if-component-dep-fails
by Faré 15 Dec '12

15 Dec '12

Dear Nathan, at least as far as quicklisp-included software goes, your ironclad and nibbles seem to be the only user of ASDF's :if-component-dep-fails and :feature features. These features are a crock and I'd like to plainly remove them. In the future, I might provide a better-thought-out replacement that actually works. In the meantime, the backwards-compatible solution would be to use #+sbcl and such; yes it breaks an abstraction barrier — but at least it doesn't require ASDF itself to break its own abstraction barrier internally. Could you "fix" your .asd's to not use these features I'd like to drop? Does any other person who cares about ASDF cares about these features? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you make people think they're thinking, they'll love you; but if you really make them think, they'll hate you. — Don Marquis

3 8

[asdf-devel] :if-feature
by Faré 15 Dec '12

15 Dec '12

ASDF 2.26.21 removed the horribly clunky :if-component-dep-fails. ASDF 2.26.22 added a much more meaningful :if-feature. Any component can use the :if-feature option, which can be an arbitrary keyword expression, just like #+. The difference is that unlike #+ it doesn't magically change the *package* to keyword, so you have to include the colons, as in :if-feature (:and :sbcl :x86-64) The upside is that its semantics is simple, doesn't break the object model, and allows for arbitrary expressions. The downside is that it is not backwards compatible. It will be released with 2.27 only. (If you have a better name than :if-feature, now is the time to suggest it.) Hopefully, I will remember to include a #+asdf2.27 before I release. Please test asdf 2.27 and give me feedback. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The worst thing that can happen to a good cause is, not to be skillfully attacked, but to be ineptly defended. — F. Bastiat

1 0

[asdf-devel] P.S. re patch...
by Robert Goldman 13 Dec '12

13 Dec '12

Locally we don't use any tabs in lisp files, and my emacs enforces that. Seems like that added a bunch of whitespace diffs to the patch. Sorry about that. Best, r

3 8

[asdf-devel] Tests were failing on ACL
by Robert Goldman 12 Dec '12

12 Dec '12

I was getting warnings from the compiler because method parameters used in dispatch were declared to be ignored. I replaced the IGNORE declarations in question with IGNORABLE declarations, and now the tests pass on all of the implementations I have. I sent a patch using git just a few minutes ago.... cheers, r

1 0

[asdf-devel] [PATCH] Replaced some IGNORE declarations with IGNORABLE.
by Robert P. Goldman 12 Dec '12

12 Dec '12

From: "Robert P. Goldman" <rpgoldman(a)gmail.com> ACL doesn't like method parameters that are used in method dispatch to be ignored. This was causing test failures. --- asdf.lisp | 56 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/asdf.lisp b/asdf.lisp index 5fb35d3..62943ed 100644 --- a/asdf.lisp +++ b/asdf.lisp @@ -2560,8 +2560,8 @@ PREVIOUS-TIME when not null is the time at which the PREVIOUS system was loaded. (input-files o c) #+(or ecl mkcl) (loop :for i :in (input-files o c) - :unless (string= (pathname-type i) "fas") - :collect (compile-file-pathname (lispize-pathname i))))) + :unless (string= (pathname-type i) "fas") + :collect (compile-file-pathname (lispize-pathname i))))) (defmethod component-depends-on ((o load-op) (c component)) (cons `(load-op ,@(component-load-dependencies c)) @@ -3132,7 +3132,7 @@ output to *VERBOSE-OUT*. Returns the shell's exit code." ;; This has next to no chance of working on basic Windows! ;; Your best hope is that Cygwin or MSYS is somewhere in the PATH. (multiple-value-bind (io process exit-code) - (apply #'mkcl:run-program #+windows "sh" #-windows "/bin/sh" + (apply #'mkcl:run-program #+windows "sh" #-windows "/bin/sh" (list "-c" command) :input nil :output t #|*verbose-out*|# ;; will be *verbose-out* when we support it #-windows '(:search nil)) @@ -3282,7 +3282,7 @@ located." ;; Note if not using International ACL ;; see http://www.franz.com/support/documentation/8.1/doc/operators/excl/ics-targe… (excl:ics-target-case (:-ics "8")) - (and (member :smp *features*) "S")) + (and (member :smp *features*) "S")) #+armedbear (format nil "~a-fasl~a" s system::*fasl-version*) #+clisp (subseq s 0 (position #\space s)) ; strip build information (date, etc.) @@ -3367,7 +3367,7 @@ located." (:local-appdata (getenv-absolute-directory "LOCALAPPDATA")) (:appdata (getenv-absolute-directory "APPDATA")) (:common-appdata (or (getenv-absolute-directory "ALLUSERSAPPDATA") - (subpathname* (getenv-absolute-directory "ALLUSERSPROFILE") "Application Data/")))))) + (subpathname* (getenv-absolute-directory "ALLUSERSPROFILE") "Application Data/")))))) (defun* user-configuration-directories () (let ((dirs @@ -3926,12 +3926,12 @@ effectively disabling the output translation facility." (if (absolute-pathname-p output-file) ;; what cfp should be doing, w/ mp* instead of mp (let* ((type (pathname-type (apply 'fasl-type keys))) - (defaults (make-pathname - :type type :defaults (merge-pathnames* input-file)))) - (merge-pathnames* output-file defaults)) + (defaults (make-pathname + :type type :defaults (merge-pathnames* input-file)))) + (merge-pathnames* output-file defaults)) (apply-output-translations (apply 'compile-file-pathname input-file - (if output-file keys (remove-keyword :output-file keys)))))) + (if output-file keys (remove-keyword :output-file keys)))))) (defun* delete-file-if-exists (x) (when (and x (probe-file* x)) @@ -4559,8 +4559,8 @@ with a different configuration, so the configuration would be re-read then." (if (bundle-op-monolithic-p instance) ".system-and-dependencies" ".system"))) (when (typep instance 'monolithic-bundle-op) (destructuring-bind (&rest original-initargs - &key lisp-files prologue-code epilogue-code - &allow-other-keys) + &key lisp-files prologue-code epilogue-code + &allow-other-keys) (slot-value instance 'original-initargs) (setf (slot-value instance 'original-initargs) (remove-keys '(lisp-files epilogue-code prologue-code) original-initargs) @@ -4748,8 +4748,8 @@ with a different configuration, so the configuration would be re-read then." (unless (trivial-system-p c) (output-files (make-sub-operation o 'fasl-op) c))) -(defmethod perform ((o load-fasl-op) (c t)) - (declare (ignore o c)) +(defmethod perform ((o load-fasl-op) c) + (declare (ignorable o c)) nil) (defmethod perform ((o load-fasl-op) (c system)) @@ -4772,20 +4772,20 @@ with a different configuration, so the configuration would be re-read then." (every #'(lambda (c) (typep c 'compiled-file)) (module-components s))) (defmethod output-files (o (c compiled-file)) - (declare (ignore o c)) + (declare (ignorable o c)) nil) (defmethod input-files (o (c compiled-file)) (declare (ignore o)) (load (component-pathname c))) (defmethod perform ((o load-op) (c compiled-file)) - (declare (ignore o)) + (declare (ignorable o)) (load (component-pathname c))) (defmethod perform ((o load-source-op) (c compiled-file)) (perform (make-sub-operation o 'load-op) c)) (defmethod perform ((o load-fasl-op) (c compiled-file)) (perform (make-sub-operation o 'load-op) c)) (defmethod perform (o (c compiled-file)) - (declare (ignore o c)) + (declare (ignorable o c)) nil) ;;; @@ -4795,9 +4795,9 @@ with a different configuration, so the configuration would be re-read then." ((static-library :accessor prebuilt-system-static-library :initarg :lib))) (defmethod output-files ((o lib-op) (c prebuilt-system)) - (declare (ignore o)) + (declare (ignorable o)) (values (list (prebuilt-system-static-library c)) - t)) ; Advertise that we do not want this path renamed by asdf-output-translations + t)) ; Advertise that we do not want this path renamed by asdf-output-translations (defmethod perform ((o lib-op) (c prebuilt-system)) (first (output-files o c))) @@ -4994,7 +4994,7 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (output (output-files o c))) (ensure-directories-exist (first output)) (apply #'c::builder (bundle-op-type o) (first output) - :lisp-files (append object-files (bundle-op-lisp-files o)) + :lisp-files (append object-files (bundle-op-lisp-files o)) (append (bundle-op-build-args o) (when (and (typep o 'monolithic-bundle-op) (monolithic-op-prologue-code o)) @@ -5014,8 +5014,8 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (defun mkcl-bundle-sub-operations (op sys) (gather-components (make-sub-operation op 'compile-op) sys - :filter-system sys - :filter-type '(not system))) + :filter-system sys + :filter-type '(not system))) (defun files-to-bundle (operation system) (loop :for (o . c) :in (mkcl-bundle-sub-operations operation system) @@ -5027,17 +5027,17 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (defmethod output-files ((o bundle-op) (c system)) (let* ((name (component-name c)) - (static-lib-name (merge-pathnames - (compiler::builder-internal-pathname name :static-library) - (component-relative-pathname c))) - (fasl-bundle-name (merge-pathnames - (compiler::builder-internal-pathname name :fasb) - (component-relative-pathname c)))) + (static-lib-name (merge-pathnames + (compiler::builder-internal-pathname name :static-library) + (component-relative-pathname c))) + (fasl-bundle-name (merge-pathnames + (compiler::builder-internal-pathname name :fasb) + (component-relative-pathname c)))) (list static-lib-name fasl-bundle-name))) (defmethod perform ((o bundle-op) (c system)) (let* ((object-files (files-to-bundle o c)) - (output (output-files o c))) + (output (output-files o c))) (ensure-directories-exist (first output)) (when (bundle-op-do-static-library-p o) (apply #'compiler::build-static-library (first output) -- 1.7.11.3

1 0

[asdf-devel] long-standing bugs fixed in traverse
by Faré 10 Dec '12

10 Dec '12

Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm. ASDF had a few major bugs remaining from ASDF 1. To fix them, its traverse algorithm was, once again, rewritten and vastly simplified. It's almost readable, now! To start with, I fixed this long-standing bug, whereby when a system was recompiled, this didn't trigger the recompilation of systems that depended on it as part of the same operate session: If you have a system B that depends on a system A, and A is modified, re-loading B will cause A to be recompiled, but not B. https://bugs.launchpad.net/asdf/+bug/479522 This was starting to annoying me a lot, since it causes systems such as xcvb-utils to often fail to recompile, getting its automatically-generated package all out of sync. One reason this bug survived so long is that there were rumors of people relying on the bug as a "feature". But ever since 2.21, we had the ability with :force-not to selectively avoid rebuilding But this led to the deeper issue that ASDF only knew about timestamps local to a immediate operation, and was not propagating timestamps from one component to those that depended on it, which means the previous fix only worked within the same session: if you loaded B in one session, modified A, recompiled A in another session, then loaded B in a third session, the latter wouldn't trigger a recompilation of B: each file looks locally up-to-date, and is blissfully unaware that it is too old as compared to its dependencies. https://bugs.launchpad.net/asdf/+bug/1087609 I fixed this bug too, which required some major modifications, overall a simplification, in the way ASDF works. For backwards compatibility (if it's not backwards, it's not compatible), I had to keep some of the previous APIs, such as the misnamed operation-done-p the signature of which I couldn't change. As users, you shouldn't notice that anything changed. If you are power users, you may notice that do-first dependencies have been eliminated. There is now only one kind of dependencies, in-order-to, which does the right thing in both cases of output-files or no output-files. It was way too much debugging, but it's now done. Please test. When all this is cleaned up and released, I'm tempted to declare the launchpad milestone "asdf 2.1" reached, and open a start a new milestone 2.2 (numbers obviously disconnected from actual version numbers, these days). Enjoy! —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Thinking is the hardest work there is, which is probably the reason why so few engage in it. — Henry Ford

2 6

[asdf-devel] Merging asdf-encodings into asdf?
by Faré 08 Dec '12

08 Dec '12

In the previous thread, I wondered whether to merge asdf-encodings into asdf. Any feedback on the proposal? SCL plans to essentially do this in their upcoming release (1.4?). It's again about 20KB, to add to the current 223KB (feature and size inflation). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The major advances in civilization are processes that all but wreck the societies in which they occur. — A.N. Whitehead

3 5

[asdf-devel] Fixed long-standing bug lp#479522
by Faré 07 Dec '12

07 Dec '12

In ASDF 2.26.8, I fixed long-standing bug lp#479522 "wanted: recompile system when dependency changes". https://bugs.launchpad.net/asdf/+bug/479522 It was this left-over from ASDF 1, whereby if a system A is modified, it is recompiled, but this does not cause the recompilation of a system B that depends-on A. Back in the dark days of ASDF 1, when computers were slow and compilation was at a premium, this was considered a "feature" by some, though a bug by many more. Bug-compatibility made us preserve this dubious behavior into ASDF 2. Since ASDF 2.21, the :FORCE-NOT feature of OPERATE allows you to avoid compilation of whichever systems you specify, so you don't need to rely on bugs of the implementation to achieve your desired effect. The bug is now fixed. If you were relying on it, please use :FORCE-NOT instead. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Wishing without work is like fishing without bait.

1 0