- asdf-devel - mailman3.common-lisp.net

[asdf-devel] Guidelines for ASDF system authors
by Felix Lange 16 Dec '12

16 Dec '12

Hi, I've been thinking a lot about the structure of Common Lisp projects lately and would like to turn those thoughts into a public discussion. I believe that some written guidelines for ASDF system authors could really improve how the Common Lisp ecosystem functions as a whole. The documentation on the ASDF website is comprehensive with respect to the capabilities of its implementation, but there are few guidelines on how developers are supposed to structure their systems. I perceive a need for a manual chapter or document that consolidates best practices advice and somehow documents the `intended' use of ASDFs countless features. This information is currently scattered all over the ASDF manual. Getting the big picture is possible but kind of difficult. The new section `Extensions' on the ASDF project website as well as the FAQ section in the manual are very much a step in the right direction. Some of the questions that I think need a documented solution which could also act as a community convention: - How to properly conditionalize on *FEATURES*. This is touched briefly in manual section 6.2.1.3 `Required features'. There's a note explaining that excluding components through the #+ syntax is a bad idea, and that the correct way of doing it i.e. (:file "foo" :depends-on (:feature :sbcl)) does not work at this time. In the meantime, Nikodemus has written MADEIRA-PORT for this exact purpose. - Implementing TEST-OP The manual tells me that `it has proven difficult to define how the test operation should signal its results' and that TEST-OP does nothing by default. Current practice seems to be defining TEST-OP as a method that loads another system and then somehow executes all tests, failing the operation when some test did not pass. This is noted in the FAQ section and should really be the official way. - How to specify optional dependencies in the `correct' way? Suppose I'm working on a library that can use either IRONCLAD or CL+SSL to generate random bytes using a strong crypto algorithm. I don't want to clutter the final application image with both crypto libraries but rather use whatever happens to be the application developer's choice. I have several options to do this. 1) Use :WEAKLY-DEPENDS-ON to rely on the application's transient dependencies to load either one. This is obviously a nice solution, but I have to document the requirement of loading at least one of them in the README and figure out which one to use at runtime. 2) Define two systems, MY-LIBRARY-IRONCLAD and MY-LIBRARY-CL+SSL. This would force the user to require one of them in order to use the library. I also need to document this, but do not have to switch at runtime. 3) Solve the problem outside of ASDF - How much `formalization' of the build process pays off in the long run? ASDF could obviously have more uses if people followed a common guideline. What's the advantage of custom component classes vs. a custom operation that is hooked into COMPILE-OP via :IN-ORDER-TO? What's used by existing systems? - Whether or not to define a package for the system definition The documentation uses (IN-PACKAGE :ASDF) in section 5.1. Common practice seems to be defining a package inside the system definition file if the system uses custom component classes or features that require toplevel forms other than ASDF:DEFSYSTEM. :DEFSYSTEM-DEPENDS-ON is also an option. - What are some good practices when building applications with ASDF? As the manual points out, `ASDF presents three faces'. People who want to use the ASDF for applications have very different needs from library developers and will often want tight control over their dependency graph. Are versioned dependencies usable and practical? How can I override decisions made by library authors in my application's build process? What's a good way to prevent package name conflicts? Some `war stories' from commercial application development could really help others arrive at a good solution quickly. I see that `style guide for .asd files' is a TODO item in the manual, so it seems some of those question are already being addressed. I am willing to do documentation work as soon as I understand people's intentions. Looking forward to your point of view, Felix

2 1

[asdf-devel] if-component-dep-fails
by Faré 15 Dec '12

15 Dec '12

Dear Nathan, at least as far as quicklisp-included software goes, your ironclad and nibbles seem to be the only user of ASDF's :if-component-dep-fails and :feature features. These features are a crock and I'd like to plainly remove them. In the future, I might provide a better-thought-out replacement that actually works. In the meantime, the backwards-compatible solution would be to use #+sbcl and such; yes it breaks an abstraction barrier — but at least it doesn't require ASDF itself to break its own abstraction barrier internally. Could you "fix" your .asd's to not use these features I'd like to drop? Does any other person who cares about ASDF cares about these features? —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you make people think they're thinking, they'll love you; but if you really make them think, they'll hate you. — Don Marquis

3 8

[asdf-devel] :if-feature
by Faré 15 Dec '12

15 Dec '12

ASDF 2.26.21 removed the horribly clunky :if-component-dep-fails. ASDF 2.26.22 added a much more meaningful :if-feature. Any component can use the :if-feature option, which can be an arbitrary keyword expression, just like #+. The difference is that unlike #+ it doesn't magically change the *package* to keyword, so you have to include the colons, as in :if-feature (:and :sbcl :x86-64) The upside is that its semantics is simple, doesn't break the object model, and allows for arbitrary expressions. The downside is that it is not backwards compatible. It will be released with 2.27 only. (If you have a better name than :if-feature, now is the time to suggest it.) Hopefully, I will remember to include a #+asdf2.27 before I release. Please test asdf 2.27 and give me feedback. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The worst thing that can happen to a good cause is, not to be skillfully attacked, but to be ineptly defended. — F. Bastiat

1 0

[asdf-devel] P.S. re patch...
by Robert Goldman 13 Dec '12

13 Dec '12

Locally we don't use any tabs in lisp files, and my emacs enforces that. Seems like that added a bunch of whitespace diffs to the patch. Sorry about that. Best, r

3 8

[asdf-devel] Tests were failing on ACL
by Robert Goldman 12 Dec '12

12 Dec '12

I was getting warnings from the compiler because method parameters used in dispatch were declared to be ignored. I replaced the IGNORE declarations in question with IGNORABLE declarations, and now the tests pass on all of the implementations I have. I sent a patch using git just a few minutes ago.... cheers, r

1 0

[asdf-devel] [PATCH] Replaced some IGNORE declarations with IGNORABLE.
by Robert P. Goldman 12 Dec '12

12 Dec '12

From: "Robert P. Goldman" <rpgoldman(a)gmail.com> ACL doesn't like method parameters that are used in method dispatch to be ignored. This was causing test failures. --- asdf.lisp | 56 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/asdf.lisp b/asdf.lisp index 5fb35d3..62943ed 100644 --- a/asdf.lisp +++ b/asdf.lisp @@ -2560,8 +2560,8 @@ PREVIOUS-TIME when not null is the time at which the PREVIOUS system was loaded. (input-files o c) #+(or ecl mkcl) (loop :for i :in (input-files o c) - :unless (string= (pathname-type i) "fas") - :collect (compile-file-pathname (lispize-pathname i))))) + :unless (string= (pathname-type i) "fas") + :collect (compile-file-pathname (lispize-pathname i))))) (defmethod component-depends-on ((o load-op) (c component)) (cons `(load-op ,@(component-load-dependencies c)) @@ -3132,7 +3132,7 @@ output to *VERBOSE-OUT*. Returns the shell's exit code." ;; This has next to no chance of working on basic Windows! ;; Your best hope is that Cygwin or MSYS is somewhere in the PATH. (multiple-value-bind (io process exit-code) - (apply #'mkcl:run-program #+windows "sh" #-windows "/bin/sh" + (apply #'mkcl:run-program #+windows "sh" #-windows "/bin/sh" (list "-c" command) :input nil :output t #|*verbose-out*|# ;; will be *verbose-out* when we support it #-windows '(:search nil)) @@ -3282,7 +3282,7 @@ located." ;; Note if not using International ACL ;; see http://www.franz.com/support/documentation/8.1/doc/operators/excl/ics-targe… (excl:ics-target-case (:-ics "8")) - (and (member :smp *features*) "S")) + (and (member :smp *features*) "S")) #+armedbear (format nil "~a-fasl~a" s system::*fasl-version*) #+clisp (subseq s 0 (position #\space s)) ; strip build information (date, etc.) @@ -3367,7 +3367,7 @@ located." (:local-appdata (getenv-absolute-directory "LOCALAPPDATA")) (:appdata (getenv-absolute-directory "APPDATA")) (:common-appdata (or (getenv-absolute-directory "ALLUSERSAPPDATA") - (subpathname* (getenv-absolute-directory "ALLUSERSPROFILE") "Application Data/")))))) + (subpathname* (getenv-absolute-directory "ALLUSERSPROFILE") "Application Data/")))))) (defun* user-configuration-directories () (let ((dirs @@ -3926,12 +3926,12 @@ effectively disabling the output translation facility." (if (absolute-pathname-p output-file) ;; what cfp should be doing, w/ mp* instead of mp (let* ((type (pathname-type (apply 'fasl-type keys))) - (defaults (make-pathname - :type type :defaults (merge-pathnames* input-file)))) - (merge-pathnames* output-file defaults)) + (defaults (make-pathname + :type type :defaults (merge-pathnames* input-file)))) + (merge-pathnames* output-file defaults)) (apply-output-translations (apply 'compile-file-pathname input-file - (if output-file keys (remove-keyword :output-file keys)))))) + (if output-file keys (remove-keyword :output-file keys)))))) (defun* delete-file-if-exists (x) (when (and x (probe-file* x)) @@ -4559,8 +4559,8 @@ with a different configuration, so the configuration would be re-read then." (if (bundle-op-monolithic-p instance) ".system-and-dependencies" ".system"))) (when (typep instance 'monolithic-bundle-op) (destructuring-bind (&rest original-initargs - &key lisp-files prologue-code epilogue-code - &allow-other-keys) + &key lisp-files prologue-code epilogue-code + &allow-other-keys) (slot-value instance 'original-initargs) (setf (slot-value instance 'original-initargs) (remove-keys '(lisp-files epilogue-code prologue-code) original-initargs) @@ -4748,8 +4748,8 @@ with a different configuration, so the configuration would be re-read then." (unless (trivial-system-p c) (output-files (make-sub-operation o 'fasl-op) c))) -(defmethod perform ((o load-fasl-op) (c t)) - (declare (ignore o c)) +(defmethod perform ((o load-fasl-op) c) + (declare (ignorable o c)) nil) (defmethod perform ((o load-fasl-op) (c system)) @@ -4772,20 +4772,20 @@ with a different configuration, so the configuration would be re-read then." (every #'(lambda (c) (typep c 'compiled-file)) (module-components s))) (defmethod output-files (o (c compiled-file)) - (declare (ignore o c)) + (declare (ignorable o c)) nil) (defmethod input-files (o (c compiled-file)) (declare (ignore o)) (load (component-pathname c))) (defmethod perform ((o load-op) (c compiled-file)) - (declare (ignore o)) + (declare (ignorable o)) (load (component-pathname c))) (defmethod perform ((o load-source-op) (c compiled-file)) (perform (make-sub-operation o 'load-op) c)) (defmethod perform ((o load-fasl-op) (c compiled-file)) (perform (make-sub-operation o 'load-op) c)) (defmethod perform (o (c compiled-file)) - (declare (ignore o c)) + (declare (ignorable o c)) nil) ;;; @@ -4795,9 +4795,9 @@ with a different configuration, so the configuration would be re-read then." ((static-library :accessor prebuilt-system-static-library :initarg :lib))) (defmethod output-files ((o lib-op) (c prebuilt-system)) - (declare (ignore o)) + (declare (ignorable o)) (values (list (prebuilt-system-static-library c)) - t)) ; Advertise that we do not want this path renamed by asdf-output-translations + t)) ; Advertise that we do not want this path renamed by asdf-output-translations (defmethod perform ((o lib-op) (c prebuilt-system)) (first (output-files o c))) @@ -4994,7 +4994,7 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (output (output-files o c))) (ensure-directories-exist (first output)) (apply #'c::builder (bundle-op-type o) (first output) - :lisp-files (append object-files (bundle-op-lisp-files o)) + :lisp-files (append object-files (bundle-op-lisp-files o)) (append (bundle-op-build-args o) (when (and (typep o 'monolithic-bundle-op) (monolithic-op-prologue-code o)) @@ -5014,8 +5014,8 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (defun mkcl-bundle-sub-operations (op sys) (gather-components (make-sub-operation op 'compile-op) sys - :filter-system sys - :filter-type '(not system))) + :filter-system sys + :filter-type '(not system))) (defun files-to-bundle (operation system) (loop :for (o . c) :in (mkcl-bundle-sub-operations operation system) @@ -5027,17 +5027,17 @@ using WRITE-SEQUENCE and a sensibly sized buffer." ; copied from xcvb-driver (defmethod output-files ((o bundle-op) (c system)) (let* ((name (component-name c)) - (static-lib-name (merge-pathnames - (compiler::builder-internal-pathname name :static-library) - (component-relative-pathname c))) - (fasl-bundle-name (merge-pathnames - (compiler::builder-internal-pathname name :fasb) - (component-relative-pathname c)))) + (static-lib-name (merge-pathnames + (compiler::builder-internal-pathname name :static-library) + (component-relative-pathname c))) + (fasl-bundle-name (merge-pathnames + (compiler::builder-internal-pathname name :fasb) + (component-relative-pathname c)))) (list static-lib-name fasl-bundle-name))) (defmethod perform ((o bundle-op) (c system)) (let* ((object-files (files-to-bundle o c)) - (output (output-files o c))) + (output (output-files o c))) (ensure-directories-exist (first output)) (when (bundle-op-do-static-library-p o) (apply #'compiler::build-static-library (first output) -- 1.7.11.3

1 0

[asdf-devel] long-standing bugs fixed in traverse
by Faré 10 Dec '12

10 Dec '12

Summary: please update asdf to 2.26.9 to test some major improvements in its build algorithm. ASDF had a few major bugs remaining from ASDF 1. To fix them, its traverse algorithm was, once again, rewritten and vastly simplified. It's almost readable, now! To start with, I fixed this long-standing bug, whereby when a system was recompiled, this didn't trigger the recompilation of systems that depended on it as part of the same operate session: If you have a system B that depends on a system A, and A is modified, re-loading B will cause A to be recompiled, but not B. https://bugs.launchpad.net/asdf/+bug/479522 This was starting to annoying me a lot, since it causes systems such as xcvb-utils to often fail to recompile, getting its automatically-generated package all out of sync. One reason this bug survived so long is that there were rumors of people relying on the bug as a "feature". But ever since 2.21, we had the ability with :force-not to selectively avoid rebuilding But this led to the deeper issue that ASDF only knew about timestamps local to a immediate operation, and was not propagating timestamps from one component to those that depended on it, which means the previous fix only worked within the same session: if you loaded B in one session, modified A, recompiled A in another session, then loaded B in a third session, the latter wouldn't trigger a recompilation of B: each file looks locally up-to-date, and is blissfully unaware that it is too old as compared to its dependencies. https://bugs.launchpad.net/asdf/+bug/1087609 I fixed this bug too, which required some major modifications, overall a simplification, in the way ASDF works. For backwards compatibility (if it's not backwards, it's not compatible), I had to keep some of the previous APIs, such as the misnamed operation-done-p the signature of which I couldn't change. As users, you shouldn't notice that anything changed. If you are power users, you may notice that do-first dependencies have been eliminated. There is now only one kind of dependencies, in-order-to, which does the right thing in both cases of output-files or no output-files. It was way too much debugging, but it's now done. Please test. When all this is cleaned up and released, I'm tempted to declare the launchpad milestone "asdf 2.1" reached, and open a start a new milestone 2.2 (numbers obviously disconnected from actual version numbers, these days). Enjoy! —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Thinking is the hardest work there is, which is probably the reason why so few engage in it. — Henry Ford

2 6

[asdf-devel] Merging asdf-encodings into asdf?
by Faré 08 Dec '12

08 Dec '12

In the previous thread, I wondered whether to merge asdf-encodings into asdf. Any feedback on the proposal? SCL plans to essentially do this in their upcoming release (1.4?). It's again about 20KB, to add to the current 223KB (feature and size inflation). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The major advances in civilization are processes that all but wreck the societies in which they occur. — A.N. Whitehead

3 5

[asdf-devel] Fixed long-standing bug lp#479522
by Faré 07 Dec '12

07 Dec '12

In ASDF 2.26.8, I fixed long-standing bug lp#479522 "wanted: recompile system when dependency changes". https://bugs.launchpad.net/asdf/+bug/479522 It was this left-over from ASDF 1, whereby if a system A is modified, it is recompiled, but this does not cause the recompilation of a system B that depends-on A. Back in the dark days of ASDF 1, when computers were slow and compilation was at a premium, this was considered a "feature" by some, though a bug by many more. Bug-compatibility made us preserve this dubious behavior into ASDF 2. Since ASDF 2.21, the :FORCE-NOT feature of OPERATE allows you to avoid compilation of whichever systems you specify, so you don't need to rely on bugs of the implementation to achieve your desired effect. The bug is now fixed. If you were relying on it, please use :FORCE-NOT instead. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Wishing without work is like fishing without bait.

1 0

[asdf-devel] Merging asdf-bundle into asdf
by Faré 07 Dec '12

07 Dec '12

I haven't heard any objection to my plan of merging asdf-bundle into asdf, but I have not any actively supporting mention either — just the fact that it will moderately enhance the lives of Quicklisp, ECL and MKCL maintainers. I'm leaving you a few more days to send me feedback before I actually do it. In the meantime, I added a precompiled-system feature to asdf-bundle. You can distribute a library as fasl-only, by first (1) compiling it with fasl-op on your build image: (asdf:operate 'asdf::fasl-op :my-library) Then copying the output file (first (asdf:output-files (make-instance 'asdf::fasl-op) (asdf:find-system :my-library)) to some destination of your choice. (2) in your user's image, have a .asd that looks like that, or better, evaluate the form as part of your initialization, without the need for a .asd: (defsystem :my-library :class :precompiled-system :fasl #p"/path/to/my-library.system.fasl") The fasl argument can be a logical pathname or an expression that will be evaluated later. This way, user libraries can use defsystem :depends-on and find the library whether its source is present or whether a .fasl was made available. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org We are always in anarchy. But we pay a hefty price maintaining the illusion that we aren't, and another one being misled by the illusion. On Thu, Nov 29, 2012 at 9:50 PM, Faré <fahree(a)gmail.com> wrote: > On Thu, Nov 29, 2012 at 11:18 AM, Juan Jose Garcia-Ripoll > <juanjose.garciaripoll(a)gmail.com> wrote: >> On Thu, Nov 29, 2012 at 2:41 PM, Faré <fahree(a)gmail.com> wrote: >>> >>> Now, do we want to distribute asdf-bundle.lisp as a separate system >>> asdf-bundle.asd? >>> Do we want to distribute it as part of asdf.asd the way we used to do >>> asdf-ecl.lisp? >>> Do we want to just add its contents to asdf.lisp, a growth of a bit over >>> 10% in size, >>> for additional functionality on many platforms and much less headaches on >>> ECL? >> >> ECL will pledge to whatever means of distribution you choose. Just keep me >> informed. >> > After carefully considering the headache for users of Quicklisp and ECL, > and the fact that asdf-bundle now supports load-fasl-op > for all actively maintained implementations, > except ABCL which has its own thing, > I've decided that it's probably best to fold asdf-bundle into ASDF. > (Note that I haven't tested the SCL, LispWorks or MKCL ports, and > that the LispWorks port I just threw together probably needs some love). > > The only objection I can imagine is file size, > but somehow, I fear we're past the point of being able to complain. > I fully assume the creeping featuritis since ASDF 1. > wc stats are as follows: > 4534 19011 200127 asdf.lisp > 597 2138 21861 asdf-bundle.lisp > 5131 21149 221988 total > So the increase is a bit under 11% increase in character file size. > > I'll give a few days for people to issue objections, > and if none is raised, I'll go ahead with the merge some time next week, > after hopefully testing on more implementations. > Hopefully, I will add some tests to the ASDF test suite.

2 2