On 2/6/13 Feb 6 -5:42 AM, Faré wrote:

Dear Robert,

it looks like the issue is that you loaded your methods to extend ASDF before upgrading ASDF itself, which won't work. Upgrading ASDF includes a FMAKUNBOUND on several functions, including PERFORM, which indeed UNTRACE's them on many implementations.

ASDF 3 will always try to upgrade itself before any operation, precisely to avoid squashing your methods in the middle of the operation, which would be far worse.

I'm afraid that this behavior needs refinement. Consider what we are doing: * we load my system definition * we load/upgrade ASDF3 * ASDF3 destroys part of my system definition and *quietly* goes ahead and does the wrong thing, loading a damaged system definition.

I can't imagine a case where the programmer wants this behavior.

I could be convinced that ASDF should upgrade itself, but I believe my example shows that any time ASDF upgrades itself it must reload all system definitions, because they have potentially been damaged.

That seems like a tall order, so a more modest alternative would be:

ASADF should raise an error and refuse to continue after reloading itself, forcing the programmer to re-do any operations that triggered the reload. In that case, it should mark all loaded system definitions as invalid, since they may contain method definitions that have been removed.

But if it was already upgraded, this should be a No-Op (unless there's a bug in ASDF). Moreover, :force t should not apply to ASDF, because it has the :builtin-system-p t flag. I just tried at home, and it worked.

I just retried and got the same behavior. No git operations done this morning. This is on ACL 8.2, Mac OS X, 64-bit Enterprise:

CL-USER> (push "home:lisp;asdf;" asdf:*central-registry*) ("home:lisp;asdf;" "home:lisp;franz-clx;" "~/lisp/asdf-systems/" "~/lisp/asdf-install-systems/systems/" "~/lisp/asdf-systems/" "~/lisp/asdf-install-systems/systems/") CL-USER> (asdf:load-system "asdf") ; Loading /Users/rpg/lisp/asdf/asdf.asd ; Fast loading ; /Users/rpg/.cache/common-lisp/acl-8.2-macosx-x64/Users/rpg/lisp/asdf/build/asdf.fasl ; Upgrading ASDF from version 2.23.7 to version 2.28 T CL-USER> (circa) Configuring for CIRCA systems... CIRCA ASDFs now in load path. NIL CL-USER> (asdf:load-system "circa-csm" :force t) ; Loading /Users/rpg/circa/code/csm/circa-csm.asd ; Loading /Users/rpg/circa/code/csm/logical-pathname.lisp ; Loading /Users/rpg/circa/code/csm/optimization-boilerplate.lisp ; Fast loading /usr/local/acl/acl82.64/code/CLX.001 ;;; Installing clx patch, version 1. ; Autoloading for package "CLTL1": ; Fast loading from bundle code/cltl1.fasl. ; Foreign loading clx:excldep.dylib. ; Loading /Users/rpg/lisp/asdf/asdf.asd ; Loading /Users/rpg/lisp/asdf/asdf-driver.asd ; Fast loading ; /Users/rpg/.cache/common-lisp/acl-8.2-macosx-x64/Users/rpg/lisp/asdf/build/asdf.fasl .....

I verify that this behavior does *NOT* occur in SBCL, loading the same system definition.

The only suspicious thing I see in the transcript from ACL is the loading of CLX. I wonder if there's any chance that this triggers some interaction with ASDF? That would be bad.

Considering that you seem to already have had ASDF 3 installed (since it did the automatic upgrade attempt), the issue seems to be that you git pull'ed after loading your ASDF overrides, and they thus got squashed during the implicit ASDF upgrade.

No, the above experiment shows that is not the explanation. Something stranger is going on.

Best, r

Just a question for understanding this better: CLOS provides clean ways to update methods with newer versions. Why do you use fmakunbound instead? Is this to stay backwards-compatible? Would it be a good idea to switch to a CLOS-based solution in the future to prevent such problems?

I'm probably missing something…

Thanks, Pascal

On 6 Feb 2013, at 12:42, Faré fahree@gmail.com wrote:

Dear Robert,

it looks like the issue is that you loaded your methods to extend ASDF before upgrading ASDF itself, which won't work. Upgrading ASDF includes a FMAKUNBOUND on several functions, including PERFORM, which indeed UNTRACE's them on many implementations.

ASDF 3 will always try to upgrade itself before any operation, precisely to avoid squashing your methods in the middle of the operation, which would be far worse. But if it was already upgraded, this should be a No-Op (unless there's a bug in ASDF). Moreover, :force t should not apply to ASDF, because it has the :builtin-system-p t flag. I just tried at home, and it worked.

Considering that you seem to already have had ASDF 3 installed (since it did the automatic upgrade attempt), the issue seems to be that you git pull'ed after loading your ASDF overrides, and they thus got squashed during the implicit ASDF upgrade.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Money doesn't change men, it merely unmasks them. If a man is naturally selfish or arrogant or greedy, the money brings that out, that's all. — Henry Ford

On Wed, Feb 6, 2013 at 1:41 AM, Robert Goldman rpgoldman@sift.info wrote:

...

I was having a case where some package wrangling in one of our systems behaved differently with new ASDF than with the ASDF 2 that ships with ACL.

It seemed like there was an :AFTER method on one of my modules that simply wasn't being called.

I took my usual resort and did

(TRACE ASDF:PERFORM).

I was very surprised to see that my

(asdf:load-system "circa-csm" :force t)

resulted in only four calls to perform, and all of them had to do with ASDF:

CL-USER> (asdf:load-system "circa-csm" :force t) ; Loading /Users/rpg/lisp/asdf/asdf.asd ; Loading /Users/rpg/lisp/asdf/asdf-driver.asd 0[7]: (ASDF/ACTION:PERFORM #<ASDF/LISP-ACTION:PREPARE-OP > #<ASDF/SYSTEM:SYSTEM "asdf">) 0* (MULTIPLE-VALUE-PROG1 (PROGN #<STANDARD-METHOD ASDF/ACTION:PERFORM :BEFORE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)> #<STANDARD-METHOD ASDF/ACTION:PERFORM (ASDF/LISP-ACTION:PREPARE-OP ASDF/COMPONENT:COMPONENT)>) #<STANDARD-METHOD ASDF/ACTION:PERFORM :AFTER (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)>) 0[7]: returned NIL 0[7]: (ASDF/ACTION:PERFORM #<ASDF/LISP-ACTION:PREPARE-OP > #<ASDF/COMPONENT:MODULE "asdf" "build">) 0* (MULTIPLE-VALUE-PROG1 (PROGN #<STANDARD-METHOD ASDF/ACTION:PERFORM :BEFORE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)> #<STANDARD-METHOD ASDF/ACTION:PERFORM (ASDF/LISP-ACTION:PREPARE-OP ASDF/COMPONENT:COMPONENT)>) #<STANDARD-METHOD ASDF/ACTION:PERFORM :AFTER (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)>) 0[7]: returned NIL 0[7]: (ASDF/ACTION:PERFORM #<ASDF/LISP-ACTION:PREPARE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "asdf" "build" "asdf">) 0* (MULTIPLE-VALUE-PROG1 (PROGN #<STANDARD-METHOD ASDF/ACTION:PERFORM :BEFORE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)> #<STANDARD-METHOD ASDF/ACTION:PERFORM (ASDF/LISP-ACTION:PREPARE-OP ASDF/COMPONENT:COMPONENT)>) #<STANDARD-METHOD ASDF/ACTION:PERFORM :AFTER (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)>) 0[7]: returned NIL 0[7]: (ASDF/ACTION:PERFORM #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "asdf" "build" "asdf">) 0* (MULTIPLE-VALUE-PROG1 (PROGN #<STANDARD-METHOD ASDF/ACTION:PERFORM :BEFORE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)> #<STANDARD-METHOD ASDF/ACTION:PERFORM (ASDF/LISP-ACTION:LOAD-OP ASDF/LISP-ACTION:CL-SOURCE-FILE)>) #<STANDARD-METHOD ASDF/ACTION:PERFORM :AFTER (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)>) ; Fast loading ; /Users/rpg/.cache/common-lisp/acl-8.2-macosx-x64/Users/rpg/lisp/asdf/build/asdf.fasl 0[7]: returned T

;;; from here on down there's lots of loading, but no recompilation (as I ;;; expected from :force t) and no calls to PERFORM. ; Fast loading ; /Users/rpg/.cache/common-lisp/acl-8.2-macosx-x64/Users/rpg/circa/code/csm/musliner-tools.fasl ; Fast loading ; /Users/rpg/.cache/common-lisp/acl-8.2-macosx-x64/Users/rpg/circa/code/csm/stochastic.fasl

Or -- oh, wait! Did the recompilation of ASDF killed the tracing?

If so, I don't think that :force t should be causing ASDF itself to rebuild. I understand why it's a dependency, but this shows why it's inappropriate: rebuilding asdf destroys our ability to inspect its behavior.

Oh, and maybe it's also killing the PERFORM :after method in my system definition?

I'd suggest :force t should stop before it gets to ASDF itself. Maybe we need to add :force :omg to do that! ;-)

---

asdf-devel mailing list asdf-devel@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

-- Pascal Costanza

On 6 Feb 2013, at 20:36, Faré fahree@gmail.com wrote:

On Wed, Feb 6, 2013 at 8:16 PM, Pascal Costanza pc@p-cos.net wrote:

...

Just a question for understanding this better: CLOS provides clean ways to update methods with newer versions. Why do you use fmakunbound instead? Is this to stay backwards-compatible? Would it be a good idea to switch to a CLOS-based solution in the future to prevent such problems?

I'm probably missing something…

Well, remove-method could be used in some cases, but it doesn't by far cover all cases: 1- sometimes, the function signature changed; &key arguments were added (or removed?), a function was made generic that wasn't before, or generic no more that was before. Only fmakunbound can handle these cases.

Adding or removing keyword arguments in methods is not a problem in CLOS.

Why not keep functions generic if your intent is to redefine functionality at runtime?

2- sometimes, other methods exist that were installed by asdf extensions that have been invalidated by other reasons (such as classes being updated, etc.) These methods should NOT be preserved, but wholly invalidated, until an updated version of the ASDF extension can themselves be loaded.

I don't understand this use case. Can you give more details?

3- CLOS is yet another vast source of portability issues, that I don't want to open. And for ASDF, I can't rely on an external library such as closer2mop (although, maybe now I can?).

CLOS is part of ANSI CL, and in that regard not more or less well specified than the rest of Common Lisp. The quality of the CLOS MOP specification is indeed lower, but so far none of the things you seem to want seem to require CLOS MOP functionality, so I'm not sure how relevant this is.

find-method and remove-method are part of CLOS, no MOP required. In many cases, a plain defmethod is sufficient, even for 'removing' methods (just use an empty method body).

Pascal

-- Pascal Costanza