- asdf-devel - mailman3.common-lisp.net

[asdf-devel] performance of find-system (on sbcl)
by Attila Lendvai 16 Nov '09

16 Nov '09

dear list, find-system is so slow that i took the time to look into it. this is what i've used to test: (sb-sprof:with-profiling () (trace sb-fasl::load-as-source) (time (labels ((recurse (system-name) (let ((system (asdf:find-system system-name nil))) (dolist (child (rest (first (asdf:component-depends-on 'asdf:compile-op system)))) (recurse (asdf:find-system child nil)))))) (recurse (asdf:find-system :hu.dwim.walker nil))))) out of the box run: Evaluation took: 3.233 seconds of real time 2.350000 seconds of total run time (2.120000 user, 0.230000 system) [ Run times consist of 0.410 seconds GC time, and 1.940 seconds non-GC time. ] 72.69% CPU 15,573 forms interpreted 2,688 lambdas converted 5,615,673,160 processor cycles 15 page faults 128,748,160 bytes consed when adding the following to asdf:find-system, just around the cl:load call: #+sbcl (sb-ext:*evaluator-mode* :interpret) Evaluation took: 1.506 seconds of real time 0.740000 seconds of total run time (0.640000 user, 0.100000 system) [ Run times consist of 0.110 seconds GC time, and 0.630 seconds non-GC time. ] 49.14% CPU 16,582 forms interpreted 56 lambdas converted 2,764,841,332 processor cycles 2 page faults 58,959,808 bytes consed any thoughts on this? -- attila

2 2

[asdf-devel] problems defining a new component class
by Robert Goldman 13 Nov '09

13 Nov '09

I'm finding that for some reason the :default-initargs to a new component class (a subclass of cl-source-file) I've defined are not getting applied. I have a vague memory that there's some weirdness in the way asdf creates components that means that this doesn't work, but I'd be obliged if anyone could provide more guidance. In particular, I would be very grateful for any suggested workaround if, as I suspect, this is something that ASDF simply won't let me do. many thanks, r

1 0

[asdf-devel] ASDF improvements from ECL
by Faré 11 Nov '09

11 Nov '09

(Moving the discussion from the ecl list to the asdf list, originally Re: ECL support for cl-launch and xcvb) 2009/11/9 Juan Jose Garcia-Ripoll <juanjose.garciaripoll(a)googlemail.com>: >> For the record, the issue I finally found and fixed in building an >> image was as follows: asdf:traverse is traversing system dependencies > > I have a problem with asdf:traverse. I mean, this function drives me > mad. Its logic for listing what has to be done and most important > _when_ is either broken or random. The last problem we had was that > for certain systems traverse lists the LOAD-OP on the system far away > from the LOAD-OPs on its individual components. ASDF:MAKE-BUILD was > previously using the components to decide what to link, but for those > systems arbitrary dependencies from other systems would be pushed in > between breaking everything. Now ECL does the following: it invokes > TRAVERSE with a hacked LOAD-OP method that reports NIL when queried > whether that operation was than; then it takes the list and groups all > subcomponents of the library that we want to turn into a big FASL, > program, etc, and then it compiles and links everything. I did not > change anything in TRAVERSE, but rather in how we use it. > >> I noticed that you're using what looks like an old version of >> asdf.lisp. There have been numerous improvements in asdf recently, and >> having been following them, I think that current revision 1.369 from >> git is probably a good pick for an update, except that it doesn't work >> in ECL as-is. It would be nice if you would merge your changes into >> upstream. > > This is another thing I have been asking help from the ASDF > maintainers. We cannot push our extensions upstream until they have > been agreed upon, I mean we need a blessing for what we are doing with > TRAVERSE and the like, and the assurance that the enging below is not > going to arbitrary change breaking the operations we have. Again, I > have written like five emails to the list without any answer. It is > discouraging. > The problem is that you'll find no one to bless your changes, as ASDF does not have a clearly defined dictator, the original author having left, our main maintainer not having the resources to become one, and a vast undefined community of users not being able to provide a clear permission wrt making incompatible changes since it's not clear what in the implementation or its documentation is binding for the future and what isn't. I propose you offer a patch, and convince people with lots of systems to try it. I'm volunteering to try your backwards-incompatible ASDF improvements on all the systems used at ITA. Possibly, the DWIM.hu guys will try it on all the systems they have, and so on. If no one complains (and why should they?) then your patch will be a strong candidate for inclusion. That automated test framework of yours is also a good thing to show that your changes are not harmful. >>> The other thing is that I have come to realize is that there is no >>> systematic way of testing libraries. ASDF does not provide support for >>> this. I have posted a call for help, or better for consensus the >>> Common Lisp community, but so far the best I got was either being >>> ignored or recriminated for being too emphatic about this. >>> >> Yeah, I saw your messages on the ASDF mailing-list. >> I think it's not going to happen there. >> I'd like to support something in XCVB, though, but I don't know what yet, >> and have been fighting more urgent lacks of feature, first. > > You seem to be also confused w.r.t. what is really needed. ASDF does > not have to support any single test library out there. What it needs > is to ask developers to provide a valid TEST-OP, subclassing an > operation it already provides. It is as simple as saying PERFORM on > (YOUR-OWN-TEST-OP ANY-COMPONENT) should return two values: OK-VALUE > and a SEQUENCE of failed tests inwhatever format you want. > Do you propose that people (or their libraries) should do things like that? (eval `(defmethod asdf:perform ((op asdf:test-op) (c (eql ,(find-system :foo)))) ...)) > If ASDF specifies rigorously how test operations should behave then > each test framework out there will be able to provide a uniform > interface for testing in a couple of lines. I already showed how to do > it for RT, for instance. > Sounds good. > The only complain on the ASDF side was that currently PERFORM can not > return values because it is not know the order in which operations are > performed and because a single operation may trigger A LOT of other > operations. That is why I developed a different implementation of > ASDF:OPERATE which works incrementally creating a log of operations > and output values. This log is extremely valuable because 1) by > filtering test operations we can know which tests fails, 2) in general > it allows a completely automated analysis of what ASDF did and what > operations failed, instead of the current ugly interface based on the > debugger. Both features are essential for automated building and > testing. > Interestingly, antifuchs' POIU also has a logging facility, to allow for replay of past builds (with non-deterministism introduced by the termination order of the parallel compilation jobs). > The patches have been submitted to the list but, as I said, > no answer so far. Yes, I saw the patch, but couldn't assume the authority to (in)validate it, and probably no one else could. Getting a patch widely tested could reassure us that "in practice, no one relies on the behavior of the old TRAVERSE where it is incompatibly redefined by the new one". [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] One can be so anxious to put his "best foot forward" that he doesn't even notice that it isn't his own foot. -- Harry Browne (HIFFIAUW)

4 8

[asdf-devel] All this activity is great and looking forward to catching up once...
by Gary King 10 Nov '09

10 Nov '09

I finish my business trip. Apologies for my apparent (actual?) vanishing! -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter

1 0

[asdf-devel] Recompile if dependent system changed
by Tobias C. Rittweiler 09 Nov '09

09 Nov '09

As far as I'm told (and that matches my experience), ASDF does not recompile the current system if a dependent system changed, unless of course :FORCE T is specified. This is logged as https://bugs.launchpad.net/asdf/+bug/479522 -T.

1 0

[asdf-devel] :IMPLEMENTATION keyword for TEST-OP
by Tobias C. Rittweiler 09 Nov '09

09 Nov '09

I want to run a test suite on an implementation I'm not currently working in. For example, TEST-OP could take an :IMPLEMENTATION argument (with one possible value being :ALL to test all "registered" implementations -- whatever "registered" means.) Fare nicely collected sexps describing how to do that for different implementations at http://common-lisp.net/gitweb?p=projects/xcvb/xcvb.git;a=blob;f=lisp-invoca… This is logged as https://bugs.launchpad.net/asdf/+bug/479513 -T.

1 0

[asdf-devel] recursiveness of :FORCE
by Tobias C. Rittweiler 09 Nov '09

09 Nov '09

When I use :FORCE, in most cases I do want to recompile solely the specified system, not all its dependencies. In my ideal world, :FORCE T would recompile the specified system only, and :FORCE :ALL would recompile it along its dependencies. Of course that would be backwards-incompatible, so perhaps better stick with the current behaviour for :FORCE T, but add :FORCE :SYSTEM, or :FORCE :SHALLOW, or similiar. -T.

2 5

Re: [asdf-devel] ASDF test-op question
by Juan Jose Garcia-Ripoll 09 Nov '09

09 Nov '09

Hi Robert, I hope you do not mind I CC this email to the list, because I think you made pretty good points and I think I also have some valid answers to them. 2009/10/19 Robert Goldman <rpgoldman(a)sift.info>: > Yes, that's right. You /do/ need to run through all of ASDF's code. I > have been going through that code. I am trying to tell the list what I > found there. And I thank you for that. I probably never got as far as understanding TRAVERSE and the way it works and this is one of the reasons why I feel my design of the ASDF-ECL extensions might be very fragile. > ASDF invokes TRAVERSE to make a /plan/ for performing an operation. A > plan is an ordered list of operation . component pairs. After traverse > returns this plan, OPERATE loops over the ordered list, calling PERFORM > on each pair. Understood. However note one thing: these are ASDF internals. They are not "exposed" to the users and not fixed by the manual. ASDF might work equally well if one changes the way OPERATE uses the output of TRAVERSE, or if we fix an order for TRAVERSE, or if we dispense the plan-and-then-execute setup. > This means that the return values of PERFORM are /discarded/. Not necesseraly. > It also means that it's not obvious to find which are the component > operations that are done for some operation X. That is, if you do (OP1 > . A) (OP2 . B) and (OP3 . C) in order to do (OP . X) all you know is > that the first three will appear /somewhere/ before (OP . X) in the plan > --- you have no stronger guarantee about where they are. In particular, > there is no top-level function call to do OP X that will be on the stack > when the three subsidiary operations are done. Understood as well. However, this also does not prevent ASDF from exposing a history of what it has done so far and let later operations use that. I attach a simple extension to ASDF's OPERATE that creates a log. The log is a list of objects, each one containing the operation that was performed, whether it failed, and the output of PERFORM. With that log it is easy to build a TEST function that parses only the log entries related to tests. * (multiple-value-setq (op log) (asdf:operate 'asdf:load-op 'cl-unicode)) [...] ;;; Computing Hangul syllable names #<ASDF:LOAD-OP NIL {12229E41}> * (mapc #'asdf::print-log-entry log) ;;; Log entry: ;;; #<LOAD-OP NIL {12229E41}> on #<SYSTEM "cl-unicode" {11A88DF9}> ;;; No return values [...] * (describe (first log)) #<ASDF:LOG-ENTRY {12E80A69}> is an instance of class #<STANDARD-CLASS ASDF:LOG-ENTRY>. The following slots have :INSTANCE allocation: COMPONENT #<ASDF:SYSTEM "cl-unicode" {11A88DF9}> OPERATION #<ASDF:LOAD-OP NIL {12229E41}> FAILED-P NIL RETURN-VALUES NIL Could we agree that this is a valid starting point? It is really compatible with previous uses of ASDF and it only asks one thing: PERFORM, applied on TEST-OP should return two values, success-p and a list of objects containing the tests, so that we can analyze the log. Juanjo -- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://juanjose.garciaripoll.googlepages.com

2 2

[asdf-devel] DOC-OP
by Tobias C. Rittweiler 09 Nov '09

09 Nov '09

I think GwKing uses automatically generated documentation for his libraries. Gary, do you use some hook into ASDF for that? In my experience, you always have to toggle a few switches for each project (name, version number, download url, etc), and I'd like to have a general interface that hides the details, say (ASDF:OOS 'ASDF:DOC-OP :TARGET-DIR #P"...") -T.

1 1

Re: [asdf-devel] loading a standalone lisp file
by Faré 29 Oct '09

29 Oct '09

2009/10/28 Robert Goldman <rpgoldman(a)sift.info>: > Faré wrote: >> 2009/10/28 Gary King <gwking(a)metabang.com>: >>>> Speaking of optimization settings, does anyone see any problem with >>>> putting in ASDF something that deterministically (re)sets the >>>> optimization setting before each and any perform operation? Otherwise, >>>> optimization settings will vary wildly depending on what >>>> implementation you use, which files get recompiled, etc. >>> I think we should do this; it's on my list. >>> >> Here's a skeleton of implementation (don't forget a defgeneric). >> >> (defmethod perform :before (operation component) >> (proclaim-optimization-settings-for operation component)) >> >> (defmethod proclaim-optimization-settings-for (op component) >> (declare (ignorable op component)) >> (proclaim `(optimize ,@*default-optimization-settings*))) >> >> Unrelatedly, a slight refactoring patch is attached. It would makes it >> easier for ADG and other ASDF add-ons to reuse in a future-proof way >> the ASDF pathname merging algorithm. > > Do we need a :after method to restore the old settings? I'm not sure > how to do that actually, since I don't believe there's a portable way to > record them. Do you have thoughts about this? > (rpg replied to me in private but I'm Cc'ing back the list) At ITA, we actually use a :around method with (defun call-with-local-compilation-settings (thunk) (let #+clozure ((ccl::*nx-speed* ccl::*nx-speed*) (ccl::*nx-space* ccl::*nx-space*) (ccl::*nx-safety* ccl::*nx-safety*) (ccl::*nx-cspeed* ccl::*nx-cspeed*) (ccl::*nx-debug* ccl::*nx-debug*)) #+sbcl ((sb-c::*policy* sb-c::*policy*)) #-(or clozure sbcl) () (funcall thunk))) (defmacro with-local-compilation-settings (() &body body) `(call-with-local-compilation-settings (lambda () ,@body))) [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] The American Republic will endure, until politicians realize they can bribe the people with their own money. -- Alexis de Tocqueville (attributed)

3 3