December 2022 - asdf-devel - mailman3.common-lisp.net

Re: ADG: handling functions defined after a reference
by Faré 31 Dec '22

31 Dec '22

IIRC, at ITA we loaded all the source code of QPX as source code in a first pass before we compiled it, to get all the functions resolved (happily, there weren't any macro-defining macros requiring more than one level of such loading), due to the code having been defined without such a clean build in mind. I don't remember for sure, but I suspect we did that also when we used ADG on that code-base to detect the dependencies. Maybe we even ran ADG twice? In the end, we also generated fine-grained dependency reports so we could later split the files in multiple parts that had a proper dependency order, and even automatically did the file-splitting based on dependencies with toplevel-form granularity. (The output from this project was dropped for political reasons, though, sigh). Sorry, my memories are growing dim, and I don't have access to this codebase anymore. I might know people who have access and might be able to dig it out (or then again might not—the code might have been archived with the old revision control system when ITA was absorbed by Google, and no one is going to get paid to fight bureaucracy to get to ancient code; and it probably was in a branch which will make archaeology harder). —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “In reviewing the history of the English Government, its wars and its taxes, a bystander, not blinded by prejudice nor warped by interest, would declare that taxes were not raised to carry on wars, but that wars were raised to carry on taxes.” — Thomas Paine, “Rights of Man” On Sat, Dec 31, 2022 at 12:43 AM Robert Dodier <robert.dodier(a)gmail.com> wrote: > > Hi, thanks for your help so far. > > I have noticed that asdf-dependency-grovel doesn't handle functions > which are defined after a reference. E.g. suppose a file says (defun f > () (g)) and G hasn't been defined yet. Then a file that's loaded > afterwards says (defun g ()). If the loading order of the files were > reversed, ADG would detect the dependency, but as it stands, ADG sees > that there is a new symbol G but doesn't output a dependency for that. > > It seems like the problem might be originating from dependencies being > detected at the time that the reference is detected. It seems like if > the definitions and references were all generated, and then all of the > references matched with definitions after everything is loaded, then > ADG would detect out-of-order references. Just guessing there. > > Do you see an easy way to get ADG to detect out-of-order dependencies? > The reason I ask is that the practical example I am interested in > (Maxima) has a lot of those. > > Thanks for any insights, > > Robert

1 0

asdf-dependency-grovel tests mostly passing now
by Robert Dodier 20 Dec '22

20 Dec '22

I've made still more progress on getting asdf-dependency-grovel running. At this point most of the unit tests are successful; there are only a couple of failed tests, both having to do with DEFTYPE. Here's what I'm getting now. I am working on an ancient Ubuntu (14.04) system. In the interest of clarity I've suppressed the SBCL banner. $ sbcl --no-userinit --script tests/run-tests.lisp This is SBCL 2.1.6, an implementation of ANSI Common Lisp. [...] WARNING: :SB-EVAL is no longer present in *FEATURES* ;;; TESTS failed: ;; u-deftype.satisfies should have: ("p-deftype"), has NIL ;; u-deftype.bar should have: ("p-deftype"), has NIL The warning about :SB-EVAL is coming from loading ASDF. The failed test cases are, respectively, (in-package :asdf-dependency-grovel-test) (deftype uses-satisfies () `(satisfies satisfies-something)) and (in-package :asdf-dependency-grovel-test) (deftype uses-bar () `(or (not bar) (and bar rational))) while the corresponding provider is (in-package :asdf-dependency-grovel-test) (deftype foo () 'string) (deftype bar () 'integer) (defun satisfies-something (#1=#:a) (declare (ignore #1#)) t) I wonder why the appropriate dependency wasn't detected, but I'm not too worried about it. ADG is still useful even if it misses these. Here is the content of tests/run-tests.lisp, which I concocted after tinkering a bit. (load "/home/robert/by-others/asdf-3.1.5/build/asdf.lisp") (push "/home/robert/by-others/asdf-dependency-grovel-master/" asdf:*central-registry*) (push "/home/robert/by-others/asdf-dependency-grovel-master/tests/" asdf:*central-registry*) (load "tests/grovel-tests.lisp") (load "handlers/00-standard-handlers.lisp") (asdf-dependency-grovel-tester:test-result) There are various questionable aspects of that, which I'll point out, although they don't really bother me too much -- I'm not responsible for ADG. (1) I was able to get ADG working only with this particular combination of SBCL (2.1.6) and ASDF (3.1.5). There might be other workable combinations, however, some or all later versions of ASDF trigger errors in ADG. (2) Use of ASDF:*CENTRAL-REGISTRY* is terrible. Okay, well, it works. (3) It appears the handlers file is supposed to be loaded by ASDF, which, I guess, is supposed to scan the handlers directory, but something in that process didn't work, so it's loaded by hand. ADG constructs a new .asd file from the inferred dependencies (and writes it into the ASDF output cache -- maybe that could be mentioned in a readme or something) and I present it here as a kind of trophy. All the best, Robert Dodier PS. groveled-components.lisp: ;;; This file contains -*- lisp -*- expressions. ;;; AUTO-GENERATED file from system definition of system (test-serial-system). Instead of directly editing this file, please edit the system definition and re-generate this file. ( ("test-serial-system" :components ( (:file "package" :encoding :utf-8) (:file "p-defmacro" :depends-on ("package") :encoding :utf-8) (:file "p-define-method-combination" :depends-on ("package") :encoding :utf-8) (:file "p-feature" :depends-on ("package") :encoding :utf-8) (:file "p-defvar" :depends-on ("package") :encoding :utf-8) (:file "p-define-setf-expander" :depends-on ("package") :encoding :utf-8) (:file "p-defsetf" :depends-on ("package") :encoding :utf-8) (:file "p-defclass" :depends-on ("package") :encoding :utf-8) (:file "p-defstruct" :depends-on ("package") :encoding :utf-8) (:file "p-defgeneric" :depends-on ("package") :encoding :utf-8) (:file "p-defmethod" :depends-on ("package") :encoding :utf-8) (:file "p-define-condition" :depends-on ("package") :encoding :utf-8) (:file "p-defpackage" :depends-on ("package") :encoding :utf-8) (:file "p-defun" :depends-on ("package") :encoding :utf-8) (:file "p-define-symbol-macro" :depends-on ("package") :encoding :utf-8) (:file "p-defconstant" :depends-on ("package") :encoding :utf-8) (:file "p-deftype" :depends-on ("package") :encoding :utf-8) (:file "u-defmacro" :depends-on ("p-defmacro" "package") :encoding :utf-8) (:file "u-defmacro.indirect" :depends-on ("p-defmacro" "package") :encoding :utf-8) (:file "u-define-method-combination" :depends-on ("p-define-method-combination" "package") :encoding :utf-8) (:file "u-feature" :depends-on ("p-feature" "package") :encoding :utf-8) (:file "u-defvar" :depends-on ("p-defvar" "package") :encoding :utf-8) (:file "u-defvar.vector" :depends-on ("p-defvar" "package") :encoding :utf-8) (:file "u-defvar.setf" :depends-on ("p-defvar" "package") :encoding :utf-8) (:file "u-defvar.push" :depends-on ("p-defvar" "package") :encoding :utf-8) (:file "u-define-setf-expander" :depends-on ("p-define-setf-expander" "package") :encoding :utf-8) (:file "u-defsetf" :depends-on ("p-defsetf" "package") :encoding :utf-8) (:file "u-defclass.defclass" :depends-on ("p-defclass" "package") :encoding :utf-8) (:file "u-defclass.defmethod" :depends-on ("p-defclass" "package") :encoding :utf-8) (:file "u-defgeneric" :depends-on ("p-defgeneric" "package") :encoding :utf-8) (:file "u-defmethod" :depends-on ("p-defmethod" "package") :encoding :utf-8) (:file "u-define-condition" :depends-on ("p-define-condition" "package") :encoding :utf-8) (:file "u-defstruct" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defstruct.type" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defstruct.defmethod" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defstruct.functions" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defstruct.functions.sharpquote" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defstruct.include" :depends-on ("p-defstruct" "package") :encoding :utf-8) (:file "u-defpackage" :depends-on ("p-defpackage") :encoding :utf-8) (:file "u-defpackage.1" :depends-on ("p-defpackage") :encoding :utf-8) (:file "u-defpackage.2" :depends-on ("p-defpackage") :encoding :utf-8) (:file "u-defpackage.3" :depends-on ("p-defpackage") :encoding :utf-8) (:file "u-defun" :depends-on ("p-defun" "package") :encoding :utf-8) (:file "u-defun.sharpquote" :depends-on ("p-defun" "package") :encoding :utf-8) (:file "u-define-symbol-macro" :depends-on ("p-define-symbol-macro" "package") :encoding :utf-8) (:file "u-defconstant" :depends-on ("p-defconstant" "package") :encoding :utf-8) (:file "u-deftype.foo" :depends-on ("p-deftype" "package") :encoding :utf-8) (:file "u-deftype.bar" :depends-on ("package") :encoding :utf-8) (:file "u-deftype.satisfies" :depends-on ("package") :encoding :utf-8) (:file "u-deftype.etypecase" :depends-on ("p-deftype" "package") :encoding :utf-8) (:file "u-deftype.typecase" :depends-on ("p-deftype" "package") :encoding :utf-8))) )

2 1

progress on asdf-dependency-grovel
by Robert Dodier 17 Dec '22

17 Dec '22

I've managed to get asdf-dependency-grovel to load. There is a comment in adg/tests/grovel-test.lisp that ASDF 3.1 is required. Okay, I have ASDF 3.1.5 laying around so I loaded that. Now SBCL can load adg. The test script adg/tests/run-tests.sh runs (with modifications) but all of the tests fail -- it appears that no dependencies are identified. I enabled *DEBUG-TRACE* in variables.lisp, and I get some output which suggests something is happening, which is encouraging. Faré, can you say anything about what one should expect to see in the debug output if stuff is working correctly? Can you see anything amiss in the output shown below? best, Robert PS. Here is the output for adg/tests/u-defconstant.lisp. That file contains just (cl:in-package :asdf-dependency-grovel-test) (defvar test-defconstant-user.1 test-defconstant.1) which references TEST-DEFCONSTANT.1, which is defined in p-defconstant.lisp, so that's the dependency that's supposed to be detected. The output for other tests is similar. ----- begin output ----- DEBUG: Perform load-source-op #<ASDF-DEPENDENCY-GROVEL:INSTRUMENTED-CL-SOURCE-FILE "test-serial-system" "u-defconstant"> DEBUG: Operating on (:ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) DEBUG: Operating on (:FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) DEBUG: Operating on ((:FORM 0 :FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) 0 "(IN-PACKAGE :ASDF-DEPENDENCY-GROVEL-TEST)") DEBUG: Done operating on ((:FORM 0 :FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) 0 "(IN-PACKAGE :ASDF-DEPENDENCY-GROVEL-TEST)") DEBUG: Operating on ((:FORM 1 :FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) 44 "(DEFVAR TEST-DEFCONSTANT-USER.1 ...)") DEBUG: Done operating on ((:FORM 1 :FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) 44 "(DEFVAR TEST-DEFCONSTANT-USER.1 ...)") DEBUG: Done operating on (:FILE #P"/home/robert/by-others/asdf-dependency-grovel-master/tests/u-defconstant.lisp" :ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) DEBUG: Done operating on (:ASDF ("test-serial-system" "u-defconstant") :ASDF ("test-serial-system") . T) ----- end output -----

2 5

Lisp file and/or ASDF dependency analysis; trying to load asdf-dependency-grovel
by Robert Dodier 15 Dec '22

15 Dec '22

Hi, I'm interested in analyzing dependencies among a collection of Lisp files, or analyzing dependencies stated in a .asd file, or both. (1) Towards the goal of analyzing dependencies among Lisp files, I'm trying to load asdf-dependency-grovel, and I'm not having any luck. I obtained a tar.gz of the current version from Gitlab, from the project https://gitlab.common-lisp.net/xcvb/asdf-dependency-grovel. I found it necessary to patch tests/run-tests.sh to get anywhere (patch below) and I get the following error at this point: ------------------------------------------------ $ sh tests/run-tests.sh This is SBCL 2.1.6, an implementation of ANSI Common Lisp. WARNING: redefining UIOP/UTILITY:PARSE-BODY in DEFUN WARNING: Deprecated recursive use of (ASDF/OPERATE:OPERATE 'ASDF/LISP-ACTION:LOAD-OP '("test-serial-system")) while visiting (ASDF-DEPENDENCY-GROVEL:DEPENDENCY-OP "test-serial" "asdf-dependency-grovel-test/serial") - please use proper dependencies instead debugger invoked on a SIMPLE-ERROR in thread #<THREAD "main thread" RUNNING {988AB081}>: Just performed compiling #<INSTRUMENTED-CL-SOURCE-FILE "test-serial-system" "package"> but failed to mark it done ------------------------------------------------ Can anyone offer any advice about the "failed to mark it done" error? or the bit about "deprecated recursive use"? (2) About looking at dependencies stated in a .asd file, I have a .asd file which ASDF is refusing to load because it has a circular dependency. Is there a way to get ASDF to tell me what that circular dependency is? Failing that, can someone suggest any tools to extract a directed graph from a .asd file? I suppose it wouldn't be too hard to do it myself but no need to reinvent the wheel. I'm thinking I could find a graph analysis library to extract any cycles, if I had a directed graph in hand. I don't suppose someone has already automated looking for cycles in .asd files? Thanks in advance for any advice, I appreciate your help. best, Robert Dodier PS. Here's my patch for asdf-dependency-grovel. $ diff -u tests/run-tests.sh-original tests/run-tests.sh --- tests/run-tests.sh-original 2022-12-13 22:39:50.174826971 -0800 +++ tests/run-tests.sh 2022-12-13 22:43:41.675974917 -0800 @@ -5,7 +5,10 @@ ADG_TEST_DIR="$(dirname $0)" -$LISP --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval '(uiop:quit (asdf-dependency-grovel-tester:test-result))' +$LISP --eval '(require (quote asdf))'\ + --eval '(push "/home/robert/by-others/asdf-dependency-grovel-master/" asdf:*central-registry*)'\ + --eval '(push "/home/robert/by-others/asdf-dependency-grovel-master/tests/" asdf:*central-registry*)'\ + --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval '(uiop:quit (asdf-dependency-grovel-tester:test-result))' STATUS=$? rm -rf asdf-dependency-grovel-tmp-* #$LISP --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval '(asdf-dependency-grovel-tester:check-base-deps)' --eval '(uiop:quit)'

2 3

Fwd: Lisp file and/or ASDF dependency analysis; trying to load asdf-dependency-grovel
by Faré 15 Dec '22

15 Dec '22

---------- Forwarded message --------- From: Faré <fahree(a)gmail.com> Date: Thu, Dec 15, 2022, 04:02 Subject: Re: Lisp file and/or ASDF dependency analysis; trying to load asdf-dependency-grovel To: Robert Dodier <robert.dodier(a)gmail.com> asdf-dependency-grovel predates asdf 2, and probably bitrotted with asdf 3. My guess is that its dependency-op needs a prepare-dependency-op the same way that load-source-op has a prepare-load-source-op. The entire class hierarchy could probably be lifted the same way, except of course for the perform method. Sorry I have no time to dig into the code. Hope this helps. On Thu, Dec 15, 2022, 02:08 Robert Dodier <robert.dodier(a)gmail.com> wrote: > Hi, I'm interested in analyzing dependencies among a collection of > Lisp files, or analyzing dependencies stated in a .asd file, or both. > > (1) Towards the goal of analyzing dependencies among Lisp files, I'm > trying to load asdf-dependency-grovel, and I'm not having any luck. I > obtained a tar.gz of the current version from Gitlab, from the project > https://gitlab.common-lisp.net/xcvb/asdf-dependency-grovel. I found it > necessary to patch tests/run-tests.sh to get anywhere (patch below) > and I get the following error at this point: > > ------------------------------------------------ > $ sh tests/run-tests.sh > This is SBCL 2.1.6, an implementation of ANSI Common Lisp. > > WARNING: redefining UIOP/UTILITY:PARSE-BODY in DEFUN > WARNING: > Deprecated recursive use of (ASDF/OPERATE:OPERATE > 'ASDF/LISP-ACTION:LOAD-OP > '("test-serial-system")) while visiting > (ASDF-DEPENDENCY-GROVEL:DEPENDENCY-OP "test-serial" > "asdf-dependency-grovel-test/serial") > - please use proper dependencies instead > > debugger invoked on a SIMPLE-ERROR in thread > #<THREAD "main thread" RUNNING {988AB081}>: > Just performed compiling #<INSTRUMENTED-CL-SOURCE-FILE > "test-serial-system" "package"> but failed to mark it done > ------------------------------------------------ > > Can anyone offer any advice about the "failed to mark it done" error? > or the bit about "deprecated recursive use"? > > (2) About looking at dependencies stated in a .asd file, I have a .asd > file which ASDF is refusing to load because it has a circular > dependency. Is there a way to get ASDF to tell me what that circular > dependency is? Failing that, can someone suggest any tools to extract > a directed graph from a .asd file? I suppose it wouldn't be too hard > to do it myself but no need to reinvent the wheel. > > I'm thinking I could find a graph analysis library to extract any > cycles, if I had a directed graph in hand. I don't suppose someone has > already automated looking for cycles in .asd files? > > Thanks in advance for any advice, I appreciate your help. > > best, > > Robert Dodier > > PS. Here's my patch for asdf-dependency-grovel. > > $ diff -u tests/run-tests.sh-original tests/run-tests.sh > --- tests/run-tests.sh-original 2022-12-13 22:39:50.174826971 -0800 > +++ tests/run-tests.sh 2022-12-13 22:43:41.675974917 -0800 > @@ -5,7 +5,10 @@ > > ADG_TEST_DIR="$(dirname $0)" > > -$LISP --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval '(uiop:quit > (asdf-dependency-grovel-tester:test-result))' > +$LISP --eval '(require (quote asdf))'\ > + --eval '(push > "/home/robert/by-others/asdf-dependency-grovel-master/" > asdf:*central-registry*)'\ > + --eval '(push > "/home/robert/by-others/asdf-dependency-grovel-master/tests/" > asdf:*central-registry*)'\ > + --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval '(uiop:quit > (asdf-dependency-grovel-tester:test-result))' > STATUS=$? > rm -rf asdf-dependency-grovel-tmp-* > #$LISP --load ${ADG_TEST_DIR}/grovel-tests.lisp --eval > '(asdf-dependency-grovel-tester:check-base-deps)' --eval '(uiop:quit)' > >

1 0