Juan Jose Garcia-Ripoll writes:

Now a couple of examples.

Suppose you have an ASDF system which is made of many sources with several dependencies. I assume the system does not rely on other resources (additional files, etc), but this could be extended.

If you want to build a single FASL file that contains EVERYTHING, use

(asdf:make-bundle 'asdf:monolithic-fasl-op :my-system :move-here "./")

If you want to build not only the FASL but also a system file that allows you to load it with ASDF, use

(asdf:make-bundle 'asdf:monolithic-binary-op :my-system :move-here "./")

It should be possible to port also the other operations, such as PROGRAM-OP, which might dump an image or do whatever implementation-specific magic there is, etc, etc.

Juanjo

Do people want something like that? Dear, yes! For example, Zach Beane's buildapp found great resonance:

http://www.xach.com/lisp/buildapp/

Is your stuff the essentials to have something like that be part of ASDF? If so, could you shortly write up a kind of roadmap?

-T.

Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com writes:

On Mon, Mar 29, 2010 at 10:01 AM, Tobias C. Rittweiler tcr@freebits.dewrote:

...

Do people want something like that? Dear, yes! For example, Zach Beane's buildapp found great resonance:

  http://www.xach.com/lisp/buildapp/

Is your stuff the essentials to have something like that be part of ASDF? If so, could you shortly write up a kind of roadmap?

What is needed for any kind of standalone system building that relies on ASDF system definitions and is capable of incorporating all dependencies?

• A way to gather the list of files that form part of that system

(gather-components in asdf-ext.lisp)

• A way to compile the components (provided by asdf.lisp and extended by

asdf-{ecl,sbcl,...}.lisp if needed)

• A way to pack all the compiled files into a single output file

(implementation-dependent and thus in asdf-{ecl,sbcl...}).

The first part is very critical and it needs the expertise of ALL of ASDF maintainers to get it right. As a bonus the result will be applicable to ALL functions that kind of grovel through ASDF system definitions.

Another real life usage case: Slime's asdf contrib does that, too, to provide functionality such as running the Emacs command rgrep, or query-replace over all files defined in a system.

-T.

On 3/29/10 Mar 29 -9:49 AM, Juan Jose Garcia-Ripoll wrote:

On Mon, Mar 29, 2010 at 4:43 PM, Tobias C. Rittweiler <tcr@freebits.de mailto:tcr@freebits.de> wrote:

Juan Jose Garcia-Ripoll
<juanjose.garciaripoll@googlemail.com
<mailto:juanjose.garciaripoll@googlemail.com>> writes:
> What is needed for any kind of standalone system building that
relies on
> ASDF system definitions and is capable of incorporating all
dependencies?
>
> - A way to gather the list of files that form part of that system
> (gather-components in asdf-ext.lisp)
> - A way to compile the components (provided by asdf.lisp and
extended by
> asdf-{ecl,sbcl,...}.lisp if needed)
> - A way to pack all the compiled files into a single output file
> (implementation-dependent and thus in asdf-{ecl,sbcl...}).
>
> The first part is very critical and it needs the expertise of ALL
of ASDF
> maintainers to get it right. As a bonus the result will be
applicable to ALL
> functions that kind of grovel through ASDF system definitions.

Another real life usage case: Slime's asdf contrib does that, too, to
provide functionality such as running the Emacs command rgrep, or
query-replace over all files defined in a system.

This is why I insist so often that the ASDF grovelling facility has to be part of the core. Right now ECL and the extensions above use a nasty trick: TRAVERSE is invoked using an operation COMPILE-OP and we wrap around LOAD-OP to inform ASDF that the operation was not done before. What this does is create a list of operations that ASDF would do assuming that absolutely no system was loaded. This has the advantage that one does not have to code special cases or traverse the tree manually looking for implicit or explicit dependencies, but I am afraid it is just as fragile.

I think that this is a laudable objective, but I'm inclined to suggest that it be pushed past ASDF 2 because it's a significant widening of the API with hard-to-predict effects on backward compatibility.

Aside from fixing the obvious bug in intra-system dependencies, we have been very delicate about rooting around inside TRAVERSE and, as you know, even my very delicate modifications caused significant pain.

My vote would be to make gathering components wait past ASDF 2.

That would allow us to ship pretty close to what we have now (I am painfully conscious that a number of the bottlenecks are down to me), and would also permit a changing of the guard in ASDF maintainership. Faré has expressed an interest in focusing his "fix the world" energy on an ASDF successor, instead of continuing to try to wrestle with the ASDF code base. Similarly, I'd like to get ASDF tolerably documented and be able to free up time for some other open source projects.

Would it be possible to take a straw poll about how to assign the gather-components protocol to a milestone?

Best, r

On Mon, Mar 29, 2010 at 5:47 PM, Robert Goldman rpgoldman@sift.info wrote:

I.e., can you characterize this declaratively in terms of the values of MODULE-COMPONENTS, and for some operation the values of INPUT-FILES and/or OUTPUT-FILES?

I do not understand your concerns or those in other messages. I do not intend to change the API, nor the way TRAVERSE works. I just want to code a _separate_ function that ensures we gather everything that is needed for a system to load. I see three uses of a better coded GATHER-COMPONENTS

1) Find the compiled files that make up a system 2) Find all the components that make up a system 3) Find all the files that make up a system

In case 1) which is the one I am most interested in the behavior should be as follows. Assume lisp L is a fresh new image with only ASDF in it and assume system S is the system we want to gather information about. Assume that L is used ONCE with S using operation LOAD-OP. Exit that lisp image and load L AGAIN and apply LOAD-OP on the same system S. GATHER-COMPONENTS should return the list of files that are LOADed in the second phase, for this is by definition the set of files that actually make up a system.

Case 2) could be coded as case 1) but instead of gathering the files at the end, gather the list of all components.

Case 3) requires new syntax because right now we do not have ways to list all the resources that belong to a system, do we?

The reason why I am calling for a consensus here is that this implies understanding how TRAVERSE works and finding the right way to operate on the chain of systems to gather the previous information.

Juanjo

On 3/29/10 Mar 29 -11:05 AM, Juan Jose Garcia-Ripoll wrote:

On Mon, Mar 29, 2010 at 5:47 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:

I.e., can you characterize this declaratively in terms of the values of
MODULE-COMPONENTS, and for some operation the values of INPUT-FILES
and/or OUTPUT-FILES?

I do not understand your concerns or those in other messages. I do not intend to change the API, nor the way TRAVERSE works. I just want to code a _separate_ function that ensures we gather everything that is needed for a system to load. I see three uses of a better coded GATHER-COMPONENTS

1. Find the compiled files that make up a system
2. Find all the components that make up a system
3. Find all the files that make up a system

In case 1) which is the one I am most interested in the behavior should be as follows. Assume lisp L is a fresh new image with only ASDF in it and assume system S is the system we want to gather information about. Assume that L is used ONCE with S using operation LOAD-OP. Exit that lisp image and load L AGAIN and apply LOAD-OP on the same system S. GATHER-COMPONENTS should return the list of files that are LOADed in the second phase, for this is by definition the set of files that actually make up a system.

Thanks, that helps. But it isn't really what I'm asking, which is "how would you characterize this in terms of existing ASDF functions (I think MODULE-COMPONENTS, INPUT-FILES, OUTPUT-FILES and, given what you've said, IN-ORDER-TO). I'm not trying to be pedantic, but we are allowing people to extend the ASDF object model by creating their own components and operations. So we need to understand what implications this new GATHER-COMPONENTS is to have on extensions.

Note that your definition of GATHER-COMPONENTS is made in terms of operations that are done outside the lisp itself. So I don't think it is well-defined in terms of ASDF operations (asdf is not set up to support, for example, "pretending" that it hasn't loaded a system). Without such a clear definition, that doesn't appeal to outside operations like "start a fresh lisp," we do not have a clear specification for implementing GATHER-COMPONENTS.

Note that this would /not/ be true if we were to implement GATHER-COMPONENTS by actually starting up a separate, clean lisp process, and harvesting information out of that lisp process. If we were to do that, then your definition would be sufficient as a specification.

Case 2) could be coded as case 1) but instead of gathering the files at the end, gather the list of all components.

Case 3) requires new syntax because right now we do not have ways to list all the resources that belong to a system, do we?

it's not even clear what is meant by "all the resources that belong to a system." Let's return to my protocol example. Consider, for the sake of example, that I have ".proto" files that are translated, using a perl script, into ".lisp" files (call this 'proto-to-lisp-op). Now I have a system that contains a component like this:

(:proto-file "IRC-proto")

and we specify that we must perform a proto-to-lisp-op to generate IRC-proto.lisp from IRC-proto.proto before we can do the COMPILE-OP.

What's the resource here? Is it the .proto file? the .lisp file? Which should be returned for your use case #3?

This is what I mean by "extending the API," although "API" might be wrong there. Maybe "protocol" is meant. People generating .asd files have had no reason to think about this question of "resource" that you are raising here, so it's quite likely that adding this new facility will not work on previously-existing systems.

That is the reason that I'm arguing for pushing this out into ASDF 3.

Note also that this will be much easier to do if a more declarative ASDF (where dependencies needed for system definition loading are made declarative, instead of requiring calls to asdf:oos in .asd files, etc.), so that would be another reason to push this into ASDF 3.

best, r

The idea of shipping an unguaranteed extension is fine with me. Let me just clarify, please: my concern was what seemed like a requirement for getting the component-gathering right. I'd rather not expose that to "outsiders" until we can get it right, and I'm worried, based on my experience with TRAVERSE, that we'd end up locked into some bug- compatibility.

Sounds like there's a good function that can be exposed as an extension without needing to get the component-gathering right, which would be great.

Maybe I got misled -- are there two issues: what can be done now and what we could do a lot better if component gathering worked reliably?

On Mar 29, 2010, at 13:43, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com

wrote:

Hi Robert,

I must say that I understand your concerns which are just, right now, *time*. And I also understand that in prolonging this thread I risk you lose your patience with a feeling that I am wasting your time, but please read this email and understand the following key points that indeed will save us all some time

1. We have a working set of ASDF extensions that ECL users apply

routinely to most systems out there. It works.

2. You asked me to extend those extensions to SBCL in an email to

the mailing list. I did it in a way that DOES NOT CHANGE the way ASDF behaves in any single way (*)

3. Those extensions can be shipped with ASDF without warranty of

100% work, and an explicit statement such as the following one:

The asdf-bundle operations are only designed to work with systems that contain basic ASDF operations and components. The protocol for systems that rely on external resources, such as shared libraries, compiled C files, bitmaps, etc, is not yet defined, so using it on such systems is not guaranteed to work.

4. Given this, the current implementation just works. No work on

implementors to do before ASDF 2.0

5. In order to remove the statement of 3) we have to code a better

map-over-ASDF or gather-components function. This can be done at any time and it is not a requirement for shipping the current extensions. This was what I said should be "discussed", but if you are too busy, simply forget it.

You can stop reading here if you wish, and I will continue in a separate email with a discussion of how this should be completed if there is ever a desire to include these features in ASDF.

Juanjo

(*) Only a small :around method for operation-done-p, but this method is only active when "performing" one of the bundle operators.

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

Now to the point.

ASDF right now has two modes of operation. One is "building" the other one is "loading". User defined operations belong to one or another class.

* compile-op is clearly a "building" operation, for it takes lisp files and compiles them * load-op is clearly a "loading" operation: it just loads a compiled file * load-source-op just as well. * cffi-grovel is a "building" operation: it takes some header and produces a lisp file * a ficticious ffi-dlopen would be a "loading" operation, installing a shared library in memory.

One would be tempted to classify these operations according to the value of OUTPUT-FILES. If it is NIL, then is a "builder", otherwise it is a "loader". However this is not always the case as ASDF has not imposed that restriction so far.

A simpler way to differentiate "builders" and "loaders" is the one I mentioned before: the two lisp images process, which is close to xvcb philosophy 1 Start a clean lisp image. 2 Apply a (asdf:oos 'load-op :target-system) 3 Quit the image 4 Start another clean lisp image 5 Apply (asdf:oos 'load-op :target-system) The second pass will give a set of operations that is just "loaders", for the first phase created all files that the lisp needs: shared libraries, etc, etc

We thus all agree that a standalone system consists only of "loaders": from the second lisp image I could dump whatever is needed and produce a fresh and ready lisp image.

The same concept can be applied to binary distributed, standalone replacements for an ASDF system. If in step "5" I identify all the operations that are done on a given system I can identify all "loaders" that relate to that particular system.

The only missing ingredient would be identifying that the system needs, such as "resources" that are neither produced nor actually related to an ASDF operation, because currently there is no need for that. This can be left for a second stage, an ASDF 2.1 if you wish, for 3.0 looks too far and more like an excuse to forget things. In that ASDF 2.1 we can add another component such as

(:resource "my-bitmap-file" :type "bmp")

and force that component have always OUTPUT-FILES the file itself. Actually I can happily do it right now it is just two lines.

Now the question is how do we "fake" the two lisp images process, which is a perfectly valid specification of what we need.

Right now the extensions do not implement the previous specification. Instead, at the risk of including spurious things, I just invoke TRAVERSE with a LOAD-OP and find out all files that are actually loaded. I know this is a hack but it worked so far.

One way that conforms better to the previous specification would be as follows: First force a COMPILE-OP on the system, either in the current or in a forked image. This ensures that al build operations are performed. A second LOAD-OP using (defmethod operation-done-p :around ((operation load-op) c) (if *force-load-p* nil (call-next-method))) and setting *force-load-p* to T will then identify which components actually form part of the lisp image.

This has the disadvantage that it triggers a full compilation of a system for a simple query. It is justified if the goal is just to make a bundle operation, but it may be too prohibiting for other extensions.

A third way would be to TRAVERSE using a system that assumes all builders are done and only "loaders" are required. How would I do that? Explicitely telling TRAVERSE what I assume that has been done and what not.

(defvar *gather-component-filter* nil)

(defmethod builder-operation-p ((o compile-op) c) t)

(defmethod builder-operation-p (o c) (and (output-files o c) t))

(defmethod operation-done-p :around ((o t) c) (if *gather-component-filter* (funcall *gather-component-filter* o c) (call-next-method)))

(defun gather-loaded-components (system &key (op-type 'compile-op) filter-system (filter-type t) include-self) "Finds all components that SYSTEM depends on when applying LOAD-OP. The output is a list of pairs of operations of the type OP-TYPE and components they act upon. The list can be filtered either by component type (FILTER-TYPE) or by imposing that the component belong to a given system (FILTER-SYSTEM). INCLUDE-SELF adds a final pair for the operation and the system itself." (let* ((system (find-system system)) (*gather-component-filter* #'builder-operation-p) (operation (make-instance op-type)) (tree (traverse (make-instance 'LOAD-OP) system))) (append (loop for (op . component) in tree when (and (typep component filter-type) (or (not filter-system) (eq (component-system component) filter-system))) collect (progn (when (eq component system) (setf include-self nil)) (cons operation component))) (and include-self (list (cons operation system))))))

This is an extension of what is currently being used in asdf-ext.lisp but in a more reasonable way because it would be extensible to other things like resources, etc. Note that the :around method is harmless: it only takes effect when gathering components and what it does is discarding the operations we assume done.

Juanjo

On Mon, Mar 29, 2010 at 5:37 PM, Robert Goldman rpgoldman@sift.info wrote:

Is this an alternative syntax? Do you expect:

(asdf:oos 'asdf:monolithic-binary-op :my-system :move-here "./")

to work as well?

No. The operations do not have a notion of "destination", because I did not want to mess up with pathname translations again. MOVE-HERE is implemented by MAKE-BUNDLE. I agree that we could find another way, perhaps writing an output-files that makes use of the new features (second value T = no translation), but I wanted to keep it as simple as possible as well as decoupled right now.

Juanjo