asdf-devel April 2022

asdf-devel@common-lisp.net

5 participants
5 discussions

Re: Define a Simple Echo-Op
by zacque 28 Apr '22

28 Apr '22

Hi phoebe, Thanks for your thoughtful reply! > A few things. > > 1. ECHO-OP should not be selfward. SELFWARD-OPERATION is for (OPERATION COMPONENT) pairs which depend on > (DIFFERENT-OPERATION COMPONENT) for the same COMPONENT. For example, LOAD-OP is selfward with respect to > COMPILE-OP, because in order to perform (load-op FILE), you must first perform (compile-op FILE). Your ECHO-OP has no such > dependency. In this case, I believe you want ECHO-OP to be downward and sideways, meaning that (ECHO-OP MODULE) > depends on (ECHO-OP MODULE-COMPONENT) for each of the children MODULE-COMPONENTs of the MODULE, and that > (ECHO-OP SOURCE-FILE) depends on (ECHO-OP EARLIER-SOURCE-FILE) for each of the EARLIER-SOURCE-FILEs in > SOURCE-FILE's :DEPENDS-ON list. This way, when you call (OPERATE 'ECHO-OP (FIND-SYSTEM "whatever")), ASDF will do a > depth-first dependency-order traversal of your system. After playing a little with downward and sideway, I can observe that downward-operation provides a depth-search/"postorder" traversal for the current system; sideway-operation provides a dependency-first traversal for dependent systems. * Note: Use :force all with OPERATE to force reloading all dependent systems. E.g. (asdf:operate 'echo-op:echo-op :echo-op-test :force :all) * Question: Why can't I subclass ASDF:OPERATION? This is unexpected since ASDF:OPERATION is the base class for all operations. My code: ----------- file: echo-op.lisp ------------------- (in-package #:echo-op) (defclass echo-op (asdf:operation) ()) (defmethod asdf:perform ((op echo-op) c) (format t "~&Operation ~a on component ~a has input files:~{~% ~a~}~%" op c (asdf:input-files op c))) ------------------------------------------------- Output: ----------------- REPL -------------------------- CL-USER> (asdf:operate 'echo-op:echo-op :echo-op-test :force t) WARNING: DEPRECATED-FUNCTION-WARNING: Using deprecated function (ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON :FOR-OPERATION #<ECHO-OP:ECHO-OP >) -- please update your code to use a newer API. WARNING: DEPRECATED-FUNCTION-WARNING: Using deprecated function (ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON :FOR-OPERATION #<ECHO-OP:ECHO-OP >) -- please update your code to use a newer API. WARNING: DEPRECATED-FUNCTION-WARNING: Using deprecated function (ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON :FOR-OPERATION #<ECHO-OP:ECHO-OP >) -- please update your code to use a newer API. WARNING: DEPRECATED-FUNCTION-WARNING: Using deprecated function (ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON :FOR-OPERATION #<ECHO-OP:ECHO-OP >) -- please update your code to use a newer API. Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "package"> has input files: Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "example"> has input files: Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "main"> has input files: Operation #<ECHO-OP > on component #<SYSTEM "echo-op-test"> has input files: #<ECHO-OP:ECHO-OP > #<ASDF/PLAN:SEQUENTIAL-PLAN {1015FB0BC3}> ------------------------------------------------- > 2. Your COMPONENT-DEPENDS-ON method is wrong. No pair of (OPERATION COMPONENT) should ever depend on the same > (OPERATION COMPONENT). What you're saying is, "in order to perform (ECHO-OP FILE), you must first perform (ECHO-OP > FILE)." > > 3. For operations that subclass one or more of DOWNWARD- UPWARD- SIDEWAY- SELFWARD- or > NON-PROPOGATING-OPERATION, you don't need to define a COMPONENT-DEPENDS-ON method. Why is my COMPONENT-DEPENDS-ON method wrong? It's not obvious that one doesn't need to define a COMPONENT-DEPENDS-ON method for >=1 *-OPERATION. Is it stated anywhere? I'm following the manual: "If the action of performing the operation on a component has dependencies, you must define a method on component-depends-on." and following the examples in cffi-grovel: https://github.com/cffi/cffi/blob/3c76afe7ba03ce015e0df99ac9ddcd61320a44a4/… That said, my code works fine by removing my COMPONENT-DEPENDS-ON method. And I can see what you meant by (ECHO-OP FILE) depending on itself by printing it out: My code: ----------- file: echo-op.lisp ------------------- (in-package #:echo-op) (defclass echo-op (asdf:sideway-operation asdf:downward-operation) ()) (defmethod asdf:perform ((op echo-op) c) (format t "~&Operation ~a on component ~a depends on ~{~% ~a~}~%" op c (asdf:component-depends-on op c))) ------------------------------------------------- Output: ----------------- REPL -------------------------- CL-USER> (asdf:operate 'echo-op:echo-op :echo-op-test :force t) Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "package"> depends on (#<ECHO-OP >) Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "example"> depends on (#<ECHO-OP >) Operation #<ECHO-OP > on component #<CL-SOURCE-FILE "echo-op-test" "main"> depends on (#<ECHO-OP >) Operation #<ECHO-OP > on component #<SYSTEM "echo-op-test"> depends on (#<ECHO-OP >) (#<ECHO-OP > #<CL-SOURCE-FILE "echo-op-test" "package"> #<CL-SOURCE-FILE "echo-op-test" "example"> #<CL-SOURCE-FILE "echo-op-test" "main">) (DEFINE-OP echo-op-test) #<ECHO-OP:ECHO-OP > #<ASDF/PLAN:SEQUENTIAL-PLAN {10162397C3}> ------------------------------------------------- But still I don't get why echo-op depends on itself. I didn't specify it anyway. Or it that the default behaviour for all *-operation classes? > 4. Most (OPERATION COMPONENT) pairs have very uninteresting sets of input files. (COMPILE-OP CL-SOURCE-FILE) has one > input file, the .lisp source file. (LOAD-OP CL-SOURCE-FILE) has one input file, the .fasl compiled file. (ECHO-OP CL-SOURCE-FILE) > will have no input files at all, unless you define a method on INPUT-FILES to list them. Why do I get nothing for my input-files? I'm expecting it to print out the pathname to (:FILE "foo") object, very much like the input files into COMPILE-OP. Again, I thought ASDF is smart enough to infer it for me? To quote the manual: "A method for this function is often not needed, since ASDF has a pretty clever default input-files mechanism." Is there a way to obtain the pathname to (:FILE "foo") from the CL-SOURCE-FILE object like what COMPILE-OP does? > I think the following definition of ECHO-OP might be enlightening to you: > > (uiop:define-package :echo-op > (:use :cl) > (:export #:echo-op)) > (in-package :echo-op) > > (defclass echo-op (asdf:sideway-operation asdf:downward-operation) ()) > > (defun print-input-files (op c) > (format t "~&Operation ~a on component ~a has input files:~{~% ~a~}~%" > op c (asdf:input-files op c))) > > (defun print-dependencies (op c) > (format t "~&Operation ~a on component ~a depends on ~{~% ~a~}~%" > op c (asdf:component-depends-on op c))) > > (defmethod asdf:perform ((op echo-op) c) > (flet ((do-operations (thunk) > (funcall thunk op c) > (funcall thunk (asdf:make-operation 'asdf:compile-op) c) > (funcall thunk (asdf:make-operation 'asdf:load-op) c))) > (format t "~&~%Input files for component ~a with a variety of operations:~%~%" c) > (do-operations #'print-input-files) > (format t "~&~%Dependencies for component ~a with a variety of operations:~%~%" c) > (do-operations #'print-dependencies))) > > Note that: > 1. The only method I have defined is on PERFORM, and it is a primary method, not an :AROUND method. ASDF already has all > the COMPONENT-DEPENDS-ON methods I need. > 2. I print the COMPONENT-DEPENDS-ON list in addition to the INPUT-FILES list. > 3. I print both the COMPONENT-DEPENDS-ON and INPUT-FILES lists for all three of ECHO-OP, COMPILE-OP and LOAD-OP. > > I recommend you load this version, try (ASDF:OPERATE 'ECHO-OP:ECHO-OP (ASDF:FIND-SYSTEM "echo-op-test") :FORCE T) and > see what output you get. Thanks! It's indeed very enlightening. And I've separated it out into few parts to play with, as shown in the code snippet I pasted above. One question: How do you know whether to define PERFORM primary method or an :AROUND method for the custom operation class? It's not obvious to me which one to choose for different use cases. Thanks! -- Regards, zacque

3 4

Re: Define a Simple Echo-Op
by Robert Goldman 27 Apr '22

27 Apr '22

On 27 Apr 2022, at 11:15, phoebe Goldman wrote: >> According to the docs, most operations inherit from self ward-operation, which would take the component-pathname as an input file (and might take others). > > I think input file sets are orthogonal to dependency direction. SELFWARD-OPERATION means that (OP COMPONENT) depends on (DIFFERENT-OP COMPONENT), like how (LOAD-OP LISP-SOURCE-FILE) depends on (COMPILE-OP LISP-SOURCE-FILE). That doesn't mean that (LOAD-OP LISP-SOURCE-FILE) takes COMPONENT-PATHNAME as an INPUT-FILE. This is what I see in the source, but I haven't had enough time to read it over carefully (or really, at all): ``` ;; An action with a selfward-operation by default gets its input-files from the output-files of ;; the actions using selfward-operations it depends on (and the same component), ;; or if there are none, on the component-pathname of the component if it's a file ;; -- and then on the results of the next-method. (defmethod input-files ((o selfward-operation) (c component)) `(,@(or (loop :for dep-o :in (ensure-list (selfward-operation o)) :append (or (output-files dep-o c) (input-files dep-o c))) (if-let ((pathname (component-pathname c))) (and (file-pathname-p pathname) (list pathname)))) ,@(call-next-method)))) ```

1 0

can't reply to asdf list Fwd: Undelivered Mail Returned to Sender
by Robert Goldman 26 Apr '22

26 Apr '22

Response from Phoebe Goldman hand-forwarded because of problems with common-lisp.net's email server: A few things. 1. ECHO-OP should not be selfward. SELFWARD-OPERATION is for (OPERATION COMPONENT) pairs which depend on (DIFFERENT-OPERATION COMPONENT) for the same COMPONENT. For example, LOAD-OP is selfward with respect to COMPILE-OP, because in order to perform (load-op FILE), you must first perform (compile-op FILE). Your ECHO-OP has no such dependency. In this case, I believe you want ECHO-OP to be downward and sideways, meaning that (ECHO-OP MODULE) depends on (ECHO-OP MODULE-COMPONENT) for each of the children MODULE-COMPONENTs of the MODULE, and that (ECHO-OP SOURCE-FILE) depends on (ECHO-OP EARLIER-SOURCE-FILE) for each of the EARLIER-SOURCE-FILEs in SOURCE-FILE's :DEPENDS-ON list. This way, when you call (OPERATE 'ECHO-OP (FIND-SYSTEM "whatever")), ASDF will do a depth-first dependency-order traversal of your system. 2. Your COMPONENT-DEPENDS-ON method is wrong. No pair of (OPERATION COMPONENT) should ever depend on the same (OPERATION COMPONENT). What you're saying is, "in order to perform (ECHO-OP FILE), you must first perform (ECHO-OP FILE)." 3. For operations that subclass one or more of DOWNWARD- UPWARD- SIDEWAY- SELFWARD- or NON-PROPOGATING-OPERATION, you don't need to define a COMPONENT-DEPENDS-ON method. 4. Most (OPERATION COMPONENT) pairs have very uninteresting sets of input files. (COMPILE-OP CL-SOURCE-FILE) has one input file, the .lisp source file. (LOAD-OP CL-SOURCE-FILE) has one input file, the .fasl compiled file. (ECHO-OP CL-SOURCE-FILE) will have no input files at all, unless you define a method on INPUT-FILES to list them. I think the following definition of ECHO-OP might be enlightening to you: ``` (uiop:define-package :echo-op (:use :cl) (:export #:echo-op)) (in-package :echo-op) (defclass echo-op (asdf:sideway-operation asdf:downward-operation) ()) (defun print-input-files (op c) (format t "~&Operation ~a on component ~a has input files:~{~% ~a~}~%" op c (asdf:input-files op c))) (defun print-dependencies (op c) (format t "~&Operation ~a on component ~a depends on ~{~% ~a~}~%" op c (asdf:component-depends-on op c))) (defmethod asdf:perform ((op echo-op) c) (flet ((do-operations (thunk) (funcall thunk op c) (funcall thunk (asdf:make-operation 'asdf:compile-op) c) (funcall thunk (asdf:make-operation 'asdf:load-op) c))) (format t "~&~%Input files for component ~a with a variety of operations:~%~%" c) (do-operations #'print-input-files) (format t "~&~%Dependencies for component ~a with a variety of operations:~%~%" c) (do-operations #'print-dependencies))) ``` Note that: 1. The only method I have defined is on PERFORM, and it is a primary method, not an :AROUND method. ASDF already has all the COMPONENT-DEPENDS-ON methods I need. 2. I print the COMPONENT-DEPENDS-ON list in addition to the INPUT-FILES list. 3. I print both the COMPONENT-DEPENDS-ON and INPUT-FILES lists for all three of ECHO-OP, COMPILE-OP and LOAD-OP. I recommend you load this version, try (ASDF:OPERATE 'ECHO-OP:ECHO-OP (ASDF:FIND-SYSTEM "echo-op-test") :FORCE T) and see what output you get. cheers, phoebe > On Apr 26, 2022, at 3:40 AM, zacque <technical+asdf-devel(a)zacque.tk <mailto:technical+asdf-devel@zacque.tkwrote: >>> > Hi, >>> > I'm trying to learn how ASDF object model works by defining a simple > operation "echo-op". I want it to print out pathnames of all loaded lisp > files to *standard-output*. It sounds like a simple task, but I couldn't > get it to work. Thus I'm asking for help here. >>> > To do that, I define a project: > ----------- file: echo-op.asd ------------------- > (defsystem #:echo-op > :components > ((:file "package") > (:file "echo-op"))) > ------------------------------------------------- >>> > ----------- file: package.lisp ------------------- > (defpackage #:echo-op > (:use #:cl #:asdf) > (:export > #:selfward-echo-op)) > ------------------------------------------------- >>> > ----------- file: echo-op.lisp ------------------- > (in-package #:echo-op) >>> > (defclass selfward-echo-op (selfward-operation) > ()) >>> > (defmethod asdf:component-depends-on ((op selfward-echo-op) c) > `((selfward-echo-op ,c) ,@(call-next-method))) >>> > (defmethod asdf:output-files ((op selfward-echo-op) c) > nil) >>> > (defmethod asdf:perform :around ((op selfward-echo-op) c) > (let ((input-files (asdf:input-files op c))) > (loop for file in input-files > do (format t "~s" file)))) >>> > (setf (find-class 'asdf::selfward-echo-op) (find-class 'selfward-echo-op)) > ------------------------------------------------- >>> > Then, if I run these from the REPL: > ----------- REPL ------------------- > CL-USER> (asdf:load-system :echo-op :force t) > ; compiling file > ; compilation finished in 0:00:00.004 > ; compiling file > ; wrote > ; compilation finished in 0:00:00.036 > T > CL-USER> (asdf:operate 'asdf::selfward-echo-op :echo-op-test) > ; Evaluation aborted on #<SB-PCL::NO-APPLICABLE-METHOD-ERROR {1002341273}>. > ------------------------------------------------- >>> > I got the error: > ----------- SLIME *sldb* ------------------- > The slot ASDF/ACTION:SELFWARD-OPERATION is unbound in the object > #<SELFWARD-ECHO-OP >. > [Condition of type UNBOUND-SLOT] > ------------------------------------------------- >>> > The "echo-op-test" system is simply: > ----------- file: echo-op-test.lisp ------------------- > (defsystem #:echo-op-test > :defsystem-depends-on (#:echo-op) > :components > ((:file "package") > (:file "example") > (:file "main"))) > ------------------------------------------------- > with empty files package.lisp, example.lisp, and main.lisp. >>> > Now, if I redefine selfward-echo-op to subclass load-op, I got this > error instead: > ----------- SLIME *sldb* ------------------- > Circular dependency of > #1=(#<ECHO-OP:SELFWARD-ECHO-OP > > . #<ASDF/SYSTEM:SYSTEM "echo-op-test">) > on: > (#1#) > [Condition of type ASDF/ACTION:CIRCULAR-DEPENDENCY] > ------------------------------------------------- > which I have no idea what is going on. >>> > I'm still learning about ASDF, so any help to achieve what I want to do > would be very much appreciated. >>> > Thanks! >>> > -- > Regards, > zacque >>> >> > > >

1 0

Define a Simple Echo-Op
by zacque 26 Apr '22

26 Apr '22

Hi, I'm trying to learn how ASDF object model works by defining a simple operation "echo-op". I want it to print out pathnames of all loaded lisp files to *standard-output*. It sounds like a simple task, but I couldn't get it to work. Thus I'm asking for help here. To do that, I define a project: ----------- file: echo-op.asd ------------------- (defsystem #:echo-op :components ((:file "package") (:file "echo-op"))) ------------------------------------------------- ----------- file: package.lisp ------------------- (defpackage #:echo-op (:use #:cl #:asdf) (:export #:selfward-echo-op)) ------------------------------------------------- ----------- file: echo-op.lisp ------------------- (in-package #:echo-op) (defclass selfward-echo-op (selfward-operation) ()) (defmethod asdf:component-depends-on ((op selfward-echo-op) c) `((selfward-echo-op ,c) ,@(call-next-method))) (defmethod asdf:output-files ((op selfward-echo-op) c) nil) (defmethod asdf:perform :around ((op selfward-echo-op) c) (let ((input-files (asdf:input-files op c))) (loop for file in input-files do (format t "~s" file)))) (setf (find-class 'asdf::selfward-echo-op) (find-class 'selfward-echo-op)) ------------------------------------------------- Then, if I run these from the REPL: ----------- REPL ------------------- CL-USER> (asdf:load-system :echo-op :force t) ; compiling file ; compilation finished in 0:00:00.004 ; compiling file ; wrote ; compilation finished in 0:00:00.036 T CL-USER> (asdf:operate 'asdf::selfward-echo-op :echo-op-test) ; Evaluation aborted on #<SB-PCL::NO-APPLICABLE-METHOD-ERROR {1002341273}>. ------------------------------------------------- I got the error: ----------- SLIME *sldb* ------------------- The slot ASDF/ACTION:SELFWARD-OPERATION is unbound in the object #<SELFWARD-ECHO-OP >. [Condition of type UNBOUND-SLOT] ------------------------------------------------- The "echo-op-test" system is simply: ----------- file: echo-op-test.lisp ------------------- (defsystem #:echo-op-test :defsystem-depends-on (#:echo-op) :components ((:file "package") (:file "example") (:file "main"))) ------------------------------------------------- with empty files package.lisp, example.lisp, and main.lisp. Now, if I redefine selfward-echo-op to subclass load-op, I got this error instead: ----------- SLIME *sldb* ------------------- Circular dependency of #1=(#<ECHO-OP:SELFWARD-ECHO-OP > . #<ASDF/SYSTEM:SYSTEM "echo-op-test">) on: (#1#) [Condition of type ASDF/ACTION:CIRCULAR-DEPENDENCY] ------------------------------------------------- which I have no idea what is going on. I'm still learning about ASDF, so any help to achieve what I want to do would be very much appreciated. Thanks! -- Regards, zacque

1 0

[Openmcl-devel] My asdf problem
by Paul Krueger 05 Apr '22

05 Apr '22

I am pretty much certain that I now understand what happened to cause my asdf problem. At the end of this I have a few final questions that maybe somebody can answer definitively for me. If you are curious about the details, read on. Here are some background facts: What I was seeing is a failure of asdf to locate a new system in my ~/common-lisp directory as it should. This problem only occurred in the CCL IDE, not when running CCL from the command line. There is an asdf internal variable called asdf/source-registry:*source-registry* which contains a list of all the .asd files it can find as a result of searching all of the directories where asdf thinks they might be. In my case that was my ~/common-lisp/ directory. In asdf itself. That variable is only initialized when a file is searched for. That might happen either via an explicit asdf call (e.g.(asdf::find-system <something>) or indirectly via a quickload call such as (ql::quickload <something>). Just loading asdf will not cause the *source-registry* to be initialized However, it turns out that just loading quickload WILL cause the asdf *source-registry* variable to be initialized. Once that variable is initialized, asdf will never try to initialize it again since that would waste a lot of time looking for .asd files all over again. So knowing all that, here is what happened. I installed and rebuilt the latest CCL a couple of months ago. As part of building the CCL IDE after that, both asdf and quickload were loaded into the CCL IDE image that was created (one question later is related to this) As a normal result of loading quickload, the asdf/source-registry:*source-registry* variable was initialized (another question below is related to this) When the CCL IDE image was built it included the value of that asdf *source-registry* variable, so that when the IDE was launched that variable was initialized. Only recently did I add additional asdf-defined systems to the ~/common-lisp/ directory. But of course when asdf tried to find them it couldn’t because the *source-registry* variable was effectively initialized when the IDE was built (long before I added my new systems). I can explicitly call an asdf function to reinitialize that variable and then everything is found normally. One question I have left is whether loading asdf and quickload into the CCL IDE is something that is done explicitly by the process that is now used to build the IDE or whether there is some option to not include them. Was there something that I should have done or not done when building the IDE that would have prevented this problem? Had there been nothing in my ~/common-lisp directory to find, then the value of that asdf variable would have been nil and that is enough to make asdf reinitialize it. But assuring that each time I build seems like a pain. My next question is why loading quicklisp causes it to initialize that variable? Conceivably it needs to search for .asd files that might be in its load environment and uses asdf to do so. Next I’m wondering whether there is some change to asdf code that can be made to reinitialize the *source-registry* every time the IDE is re-started. As another possibility, perhaps the CCL code for dumping the IDE can be modified in some way. In a private email Robert suggested: "If you have time to look, take a peek at asdf/uiop/image.lisp and asdf/bundle.lisp which should have code Fare wrote about dumping an image in a way that it will restart happily.” That seems like something that CCL maintainers might want to consider. While waiting for fixes or other suggestions, I will put an explicit call to reinitialize the *source-registry* in my ccl-ide-init.lisp file: (asdf/source-registry::initialize-source-registry) If anyone else has an environment similar to mine and rebuilds their own version of the IDE, I would suggest they do the same.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

asdf-devel April 2022