good morning;

On 2010-03-17, at 14:12 , Robert Goldman wrote:

On 3/17/10 Mar 17 -4:04 AM, Juan Jose Garcia-Ripoll wrote:

...

This brings back one of the things I discussed long ago. TRAVERSE's behavior has to be documented and its behavior has to be specified and fixed. Either that or ASDF should stabilize a way to write extensions on top of it. I can not be hanging out there waiting for every improvement in ASDF to break our build system.

Incidentally, another thing that breaks ECL is that INPUT-FILES now has a default method with signature ((o operation) ...) That broke our default methods with signature ((o T) (c compiled-file))

With all due respect, I think taking over primary methods for ANY of the existing ASDF generic functions and classes should be proclaimed to be out of bounds for extenders.

I propose that we modify the manual (if I can figure out where! the chapter on the object model is a real mess) to specify that extenders should only define primary methods on classes they define themselves. Otherwise the ASDF maintainers reserve the right to make arbitrary changes.

Modifications to generic functions --- even generic functions whose names are exported --- for classes --- even classes whose names are exported --- should be limited to :around, :before and :after method definitions.

If this proposal meets with approval, we should try to figure out where it can live in the manual.

one would be within reason to adopt the same constraints as the language standard: only the library itself is permitted to define methods (whether primary, or qualified) exclusively on "standard" classes. 11.1.2.1.2 #19 is very stingy, but it maintains a clear interface.

On 3/17/10 Mar 17 -9:00 AM, Faré wrote:

Juanjo: it is unfortunate that the recent fix to a bug in TRAVERSE led to breakage of the ECL extension. We'll try our best to accommodate your needs. Do you know how to fix the breakage on your side, or do you need us to change something?

Robert: for all purposes, Juanjo counts as one of the "ASDF maintainers", as far as the essential ECL extensions are concerned.

OK. I apologize if I seemed grumpy, but we beat this TRAVERSE issue to death on the mailing list a long time ago, and I'm getting very tired of having to explain the oddity of ASDF's component model. This is not an oddity for which I'm responsible, just one from which I suffer with the rest, and now all of my ASDF time for today has been burned on email, and I will not be able to do any of the ACL testing and debugging that I'd been hoping to get done.

I don't know how to fix things, but we certainly should fix them before we release 2.

It's not clear to me that these things can be fixed inside ASDF. At least some of them involve Juanjo being caught by an unfortunate lack of clarity in the nature of the API that ASDF supports.

E.g., it seems that we can fix one of Juanjo's two problems by simply recasting one of his INPUT-FILES (?) primary method as an :AROUND method.

As for the problem with TRAVERSE, I don't understand enough what the problem is. Can it be solved simply by adding something like:

(defmethod PERFORM ((op lib-op) (c cl-source-file)) (values))

?

I.e., without knowing why it's bad that LIB-OP visits each piece of a system, I can't say how to repair this.

best, r

On 3/17/10 Mar 17 -3:31 PM, Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 3:23 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:

E.g., it seems that we can fix one of Juanjo's two problems by simply
recasting one of his INPUT-FILES (?) primary method as an :AROUND
method.

Around methods are a hack. If you later on decide to add an around method to ASDF then you break whatever I do. If some user decides to add an around method then it will break again.

For the record, I don't agree that :around methods are a hack. Or at any rate, no worse than (call-next-method). If you don't like CLOS, you won't like ASDF. ASDF was explicitly designed to be a successor to MK-DEFSYSTEM that exploited CLOS.

I /do/ agree that we need to layer the protocol more. We had already had a layer that allowed us to separate ASDF internal around methods from ASDF user around methods. This was removed over my protests.

It seems reasonable to me that we might want to identify an API layer that is explicitly for people adapting ASDF to particular Lisp implementations. I would welcome suggestions for improving this.

As for the problem with TRAVERSE, I don't understand enough what the
problem is.  Can it be solved simply by adding something like:
 (defmethod PERFORM ((op lib-op) (c cl-source-file))
 (values))

What if somebody adds another component to a system? Right now this means I have to track all existing ASDF components, know in advance which ones may form part of the module, because as it was said I am not allowed to write a method with a generic signature PERFORM ((op lib-op) (c component)) because COMPONENT is a class that ASDF defines.

The inclusion of LIB-OP should open up

(defmethod PERFORM ((op LIB-OP) (c COMPONENT)) ...)

to you.

Summing up, please you all consider this: if you forbid --in the ASDF specification-- to write any method that uses directly ASDF classes that means we are not allowed to extend ASDF systems at all.

• We would have to define our own system, module, component,

operations, and these would no be allowed to coexist with ASDF systems.

• We would not be allowed to use the same *.asd systems that other

library defines because those use ASDF's system classes.

• We would not be allowed to combine ASDF's LOAD-OP operations with our

own components (precompiled files, etc) because we are not allowed to write methods using LOAD-OP and our extended classes.

I think the above is overboard. It would be more helpful for you to work to provide a clean description of what is an acceptable API and what is not than to indulge in this kind of overstatement.

We cannot allow people to write arbitrary methods on arbitrary bits of ASDF --- the whole system would become unstable and unmaintainable. So we MUST find a crisp way to characterize what extenders can and can't do.

Please focus your attention on helping us define a clear set of guidelines rather than attacking the notion of limitations on the extenders.

Robert

On 3/17/10 Mar 17 -4:57 PM, Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 9:46 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:

What we do need is clear and self explained in the asdf-ecl.lisp extension and I discussed this here long long ago. If this has to be done again, then so it be.

Well, turnabout is fair play, since you have made me explain the ASDF traverse operation to you again.

On 3/17/10 Mar 17 -4:57 PM, Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 9:46 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:

Second, ECL's extensions are part of ASDF. It is not just a game we play to be on this list or to ship ASDF with our system. It also sucks my development time and prevents me from working on other useful things like ECL's compiler, performance, etc, but I believe on the utility of the project: being able to build and ship arbitrary ASDF systems as libraries, executables and the like, and improving integration of ASDF with systems that work better with monolithic binary files instead of thousands of small compiled files. This is a path other implementations may follow, but no one seems to care right now.

OK, we need to be much more clear about what is under discussion at any point.

If you are talking about ECL's extensions which are part of ASDF, instead of outside ASDF, then the stricture I proposed does not apply to you.

The restrictions I proposed are for the kinds of ASDF extensions that are put in individual .asd files and that are put into asdf add-on libraries. For those it is clear we cannot allow people to simply go off and rewrite core bits of ASDF. For example, if I'm writing the ASDF definition for my FLOYD-WARSHALL transitive closure library, I can't be putting into it

(defmethod PERFORM ((op load-op) (c cl-source-file)) ....)

because that means that anyone who loads my library will be loading it into a radically different environment.

The ECL extensions are obviously not this sort of thing.

It seems to me that you have two choices:

1. really fuse the ECL extensions into ASDF proper, to the extent of putting the source in there. We will still rely on some periodic testing to make sure that any incompatibilities are caught, but James Anderson's work seems like it will fix this problem.

2. keep ECL separate and provide some statement, ideally one that can be programmatically checked, about the kind of API you expect. For example, you could add a test to the test suite that would check for the absence of a method for PERFORM on the signature (T COMPONENT). That would verify that you have the method free for your own use.

I agree that a common ground should be found, but be careful with your sentences.

This is not just 'people' and this is not just 'extenders'. Either you accept this is part of ASDF and accept its growing user base, or not. If asdf-ecl.lisp is part of ASDF then any maintainer should also care for what those other components need and do, and what they are based on.

Again, if this is part of ASDF, then you are correct, you are not just "people," and these strictures wouldn't apply to you.

However, I think you have contributed to this confusion yourself, by objecting to strictures meant only for ASDF system developers, on behalf of something that you say yourself does not apply to you.

Please focus your attention on helping us base define a clear set of
guidelines rather than attacking the notion of limitations on the
extenders.

What we do need is clear and self explained in the asdf-ecl.lisp extension and I discussed this here long long ago. If this has to be done again, then so it be.

asdf-ecl.lisp provides six new operations that have to integrate with ASDF and that can be used instead of LOAD-OP or COMPILE-OP. These operations are DLL-OP, LIB-OP, FASL-OP, monolithic versions (that is everything in one file) and a PROGRAM-OP.

These operations proceed in a simple way: they use TRAVERSE with a LOAD-OP to know what a system uses, and then they use PERFORM with a COMPILE-OP to actually compile it and from the list that TRAVERSE creates they learn the list of compiled files and build things with them (library, program, whatever)

Integration with TRAVERSE is crucial and depending on how _you_ _all_ decide that TRAVERSE should behave then we will have to adapt the code one way or another. This integration is fragile. If at some point the algorithm is randomly changed and you decide that systems should be listed before components, then we are f*ck*d.

Can you say what exactly has goofed this up for you? Note that TRAVERSE is not deterministic --- it's possible to specify multiple different orderings that satisfy the dependency specification in the ASDF file.

It's possible that what is going on is simply a bug, but I can't tell without knowing what has changed. Can you give me a concrete example? The behavior of TRAVERSE should not have changed for your application (except up to the nondeterminism to which I refer above); the changes should only appear on REcompilation.

Until now our own operations listed no dependencies other than the LOAD-OP and COMPILE-OP mentioned before. TRAVERSE was just happy with that. Right now ASDF is adding additional dependencies such as performing the same operations (DLL-OP, LIB-OP...) on the components itself. This was my source of surprise when I started this thread.

As far as I can tell, if ASDF was not doing that before, it was an unintended behavior.

If individual files are not to be subjected to these operations, would it be possible to define these operations as SYSTEM-OPERATIONS and simply define a method on PERFORM that would ensure that SYSTEM-OPERATIONS on COMPONENTS are no-ops?

asdf-ecl.lisp only defined methods on COMPONENT-DEPENDS-ON, INPUT-FILES, OUTPUT-FILES, PERFORM and OPERATION-DONE-P. All the signatures we used contain at least either OUR components our OUR operations, so in that sense they should be allowed to exist and ASDF should be careful on deciding what to do with these methods -- for instance, rewriting paths as we discussed on another thread.

I thought you were specifically referring to core methods. I see that I was not fully understanding.

Note that the protocol for writing these new operations was never clear and it has grown up with experience and by inspecting ASDF's codebase. At some point the extensions were accepted in the source tree, point at which I asked whether the integration that was done the right way or not. The lack of answer was understood as an ok.

This exhibits two problems ASDF development. The underlying problem is that people are doing this work on very thin time slices.

1. Lack of answer can never effectively be treated as "ok" --- lack of an answer is more likely to simply mean "I haven't had time to look at this." My changes to TRAVERSE went in a very long time ago, and only after a very long discussion in the mailing list. It's clear that you didn't have time to review them then, and so an issue that we thought was closed will have to be reopened.

2. The documentation is badly bit-rotted. The section on extending the ASDF object model is in particularly bad shape. It was never complete, as far as I can tell, and it has not been updated. We badly need a protocol specification stating exactly what has to be defined to make new components and operations. I can't say that I fully understand how to do this --- my experiments have often seemed to go just fine, then led to me stumbling much later on a problem that came up because some critical method was missing.

Any suggestion about how to document this protocol would be very helpful.

On Wed, Mar 17, 2010 at 11:54 PM, Juan Jose Garcia-Ripoll < juanjose.garciaripoll@googlemail.com> wrote:

Oh, there is nothing with TRAVERSE's output _right now_.

Let me clarify this again:

- The fact that TRAVERSE now adds the same operation for all components was new. That was my source of confusion. Before this did not happen with LIB-OP and the like. Maybe a straightforrwad solution is just writing

(defun perform ((o bundle-op) (c component)) nil)

so that all components which are not modules get a default PERFORM that does nothing. Is this safe?

- The order and format of TRAVERSE's output is important. Details like ensuring that the list includes the operated systems and that a system's components appear before the system that owns them, is also important, for this allows us to use TRAVERSE for identifying what files make up a module, using a variant of

(let ((*forcing* t)) (traverse (make-instance 'load-op) some-system))

which lists all components and all systems that should be loaded, sorted in some appropriate order. We rely critically on this, because otherwise I do not know a way to traverse a set of systems "portably" without redoing all of ASDF's logic.

Juanjo

On Thu, Mar 18, 2010 at 12:21 AM, Juan Jose Garcia-Ripoll < juanjose.garciaripoll@googlemail.com> wrote:

Narrowing down the problem, TRAVERSE now adds the LIB-OP operation to all components of a system. I added a method

(defmethod perform ((operation lib-op) (c component)) nil)

but this method gets overriden by a more specific one which is provided by asdf.lisp

(defmethod perform ((operation operation) (c source-file)) (sysdef-error "~@<required method PERFORM not implemented ~ for operation ~A, component ~A~@:>" (class-of operation) (class-of c)))

This method was the cause of all problems.

There is obviously something awry going on here because now I realize that this is not more specific, but less -- lib-op <= operation Anyway, I will leave this for tomorrow as I am obviously not fit for this anymore.

Juanjo

On 3/17/10 Mar 17 -6:47 PM, Juan Jose Garcia-Ripoll wrote:

On Thu, Mar 18, 2010 at 12:25 AM, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com mailto:juanjose.garciaripoll@googlemail.com> wrote:

On Thu, Mar 18, 2010 at 12:21 AM, Juan Jose Garcia-Ripoll
<juanjose.garciaripoll@googlemail.com
<mailto:juanjose.garciaripoll@googlemail.com>> wrote:

    Narrowing down the problem, TRAVERSE now adds the LIB-OP
    operation to all components of a system. I added a method

    (defmethod perform ((operation lib-op) (c component))
      nil)

    but this method gets overriden by a more specific one which is
    provided by asdf.lisp

    (defmethod perform ((operation operation) (c source-file))
      (sysdef-error
       "~@<required method PERFORM not implemented ~
        for operation ~A, component ~A~@:>"
       (class-of operation) (class-of c)))

    This method was the cause of all problems.


There is obviously something awry going on here because now I
realize that this is not more specific, but less -- lib-op <=
operation Anyway, I will leave this for tomorrow as I am obviously
not fit for this anymore.

This time I leave for bed, but while looking at the screen I think I found the problem: packages. The two symbols, PERFORM in one and in another file are different. I think this has to do with the way symbols are interned / uninterned during the compilation phase itself. A symptom is that loading the sources *.lisp everything works just fine. Another one is that the debugger prints the offending method as #:PERFORM

Is there any chance that you could somehow have run afoul of the package surgery at the top of asdf.lisp? I.e., could you have compiled /your/ PERFORM method into the ASDF package somehow before the package magic at the head of asdf.lisp destroyed it?

I confess that I can't say I understand how that package surgery in the eval-when will work. I believe it's going to be something like this:

1. compile asdf.lisp --- original ASDF package, if any, is destroyed by ENSURE-PACKAGE

2. compile asdf-ecl.lisp

3. load asdf.fasl --- package destroyed again

4. load asdf-ecl.fasl

Is it possible that somehow the compilation of asdf-ecl is recording some information about the package that is somehow damaged by the package surgery?

Just a wild guess.

good morning;

one will no doubt have noticed this message. one may well have objected, that it does not reflect reality. that is true.

it brings up an issue which should be addressed as part of the discussion about a more stable interface:

component-depends-on should be changed so that it uses subtype rather than eq as its test.

it has long been clear,[0] that this renders asdf's "O-O" aspirations for naught, but the issue has until now been dormant. from this point on, if asdf wants a stable interface, the internals will need to recognize that library classes will be specialized.

the recent point, about intended base methods, which defined instead - likely unintentionally, as default methods, expresses similar concerns.

--- [0] : http://github.com/lisp/de.setf.asdf.x/blob/master/asdf- x.lisp#L1992

Begin forwarded message:

From: james anderson james.anderson@setf.de Date: 2010-03-18 00:35:51GMT+01:00 To: ASDF-devel asdf-devel@common-lisp.net Subject: Re: [asdf-devel] ASDF traverse changed behavior?

On 2010-03-18, at 00:03 , Juan Jose Garcia-Ripoll wrote:

...

On Wed, Mar 17, 2010 at 11:54 PM, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote: Oh, there is nothing with TRAVERSE's output _right now_.

Let me clarify this again:

• The fact that TRAVERSE now adds the same operation for all

components was new. That was my source of confusion. Before this did not happen with LIB-OP and the like. Maybe a straightforrwad solution is just writing

(defun perform ((o bundle-op) (c component)) nil)

so that all components which are not modules get a default PERFORM that does nothing. Is this safe?

if the library adopts an ansi-11.1.2.1.2-like restriction on extensions, this would be compatible, because one of the specializers is not a standard class.

---

asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

good afternoon;

On 2010-03-18, at 00:03 , Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 11:54 PM, Juan Jose Garcia-Ripoll juanjose.garciaripoll@googlemail.com wrote: Oh, there is nothing with TRAVERSE's output _right now_.

Let me clarify this again:

• The fact that TRAVERSE now adds the same operation for all

components was new. That was my source of confusion. Before this did not happen with LIB-OP and the like. Maybe a straightforrwad solution is just writing

(defun perform ((o bundle-op) (c component)) nil)

so that all components which are not modules get a default PERFORM that does nothing. Is this safe?

• The order and format of TRAVERSE's output is important. Details

like ensuring that the list includes the operated systems and that a system's components appear before the system that owns them, is also important, for this allows us to use TRAVERSE for identifying what files make up a module, using a variant of

(let ((*forcing* t)) (traverse (make-instance 'load-op) some- system))

which lists all components and all systems that should be loaded, sorted in some appropriate order. We rely critically on this, because otherwise I do not know a way to traverse a set of systems "portably" without redoing all of ASDF's logic.

if asdf were to adopt an 11.1.2.1.2-rule, asdf-ecl.lisp would require a change.

it would not be supported for an extension to extend 'asdf:load-op such that load-op itself specialized an operation-done-p :around method which forced complete traversal results. it would be be necessary to specialize the load-op class as, for example, collect- op, and specialize operation-done-p on that class. in which case its own primary method could always return nil and an :around method would not be necessary.

good afternoon.

On 2010-03-18, at 15:02 , Juan Jose Garcia-Ripoll wrote:

On Thu, Mar 18, 2010 at 2:55 PM, james anderson james.anderson@setf.de wrote:

if asdf were to adopt an 11.1.2.1.2-rule, asdf-ecl.lisp would require a change.

it would not be supported for an extension to extend 'asdf:load-op such that load-op itself specialized an operation-done-p :around method which forced complete traversal results. it would be be necessary to specialize the load-op class as, for example, collect- op, and specialize operation-done-p on that class. in which case its own primary method could always return nil and an :around method would not be necessary.

What you suggest can not be done.

you type too fast. please see my next message.

While load-op dependencies are automatically added to all systems by some hard-coded logic in DEFSYSTEM, if we create a collect-op, traverse will not find any dependencies that are related to it and it will refuse to, say, traverse cl-ppcre when stumpwm depends on it.

A potential aternative would be to remove the :around method and use *forcing* but this again does not work, because *forcing* only forces the system we act upon, and not the dependecies it has.

So right now there is no way to traverse the ASDF system tree without such a hack.

Juanjo

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

On 3/18/10 Mar 18 -9:02 AM, Juan Jose Garcia-Ripoll wrote:

On Thu, Mar 18, 2010 at 2:55 PM, james anderson <james.anderson@setf.de mailto:james.anderson@setf.de> wrote:

if asdf were to adopt an 11.1.2.1.2-rule, asdf-ecl.lisp would
require a change.

it would not be supported for an extension to extend 'asdf:load-op
such that load-op itself specialized an operation-done-p :around
method which forced complete traversal results. it would be be
necessary to specialize the load-op class as, for example,
collect-op, and specialize operation-done-p on that class. in which
case its own primary method could always return nil and an :around
method would not be necessary.

What you suggest can not be done. While load-op dependencies are automatically added to all systems by some hard-coded logic in DEFSYSTEM, if we create a collect-op, traverse will not find any dependencies that are related to it and it will refuse to, say, traverse cl-ppcre when stumpwm depends on it.

A potential aternative would be to remove the :around method and use *forcing* but this again does not work, because *forcing* only forces the system we act upon, and not the dependecies it has.

So right now there is no way to traverse the ASDF system tree without such a hack.

FWIW, I think we are in agreement that ASDF-ECL should not be subject to the full stringency of the 11.1.2.1.2 rule, because it's part of ASDF proper.

OTOH, it's probably a good idea to figure out the implications of such a rule on true extensions, so I don't think James's attentions here are wasted.

Best, Robert

I think we've come to some conclusions about the /general/ problem of modifying the ASDF operations and the hazards of using CLOS in an exported API. But I wonder if we've answered the original issue, viz:

Juanjo --- do you have what you need to make asdf-ecl.lisp work again (wrt traverse)? Is there anything we (I) can do to help with this?

Also, if you find a place where you see a system operation being done /before/ the corresponding operation on a component, please drop a launchpad ticket in right away --- it means TRAVERSE is messed up.

cheers, r

On 3/17/10 Mar 17 -5:54 PM, Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 11:28 PM, Robert Goldman <rpgoldman@sift.info mailto:rpgoldman@sift.info> wrote:

For example, if I'm writing the ASDF
definition for my FLOYD-WARSHALL transitive closure library, I can't be
putting into it

(defmethod PERFORM ((op load-op) (c cl-source-file))
 ....)

because that means that anyone who loads my library will be loading it
into a radically different environment.

Understood. Then the general restriction would be that at least one of the classes in the method should be proprietary to allow the user to write such methods. Does this seem reasonable?

Correct.

And I think that you point out an important corollary to this --- we need to figure out what sorts of things the ASDF core is allowed to do.

For example, we should avoid method definitions with signatures like

(T <component-type>)

because these are likely to cause problems downstream when people try to extend them, because of multiple argument dispatch.

I'm hoping that we can come up with some form of specification that is more terse and general than simply specifying what we are going to do for each generic function that we export.

Perhaps this will emerge from trying to write a manual section on writing one's own components and operations.

> Integration with TRAVERSE is crucial and depending on how _you_ _all_
> decide that TRAVERSE should behave then we will have to adapt the code
> one way or another. This integration is fragile. If at some point the
> algorithm is randomly changed and you decide that systems should be
> listed before components, then we are f*ck*d.

Can you say what exactly has goofed this up for you?  Note that TRAVERSE
is not deterministic --- it's possible to specify multiple different
orderings that satisfy the dependency specification in the ASDF file.

Oh, there is nothing with TRAVERSE's output _right now_. It has bitten us hard in the past, though and I was just mentioning for the record, to give a complete overview of how this kind of operations bind to the ASDF core: these are operations rely on traversing a system and performing things on them, a task that can be very useful for many other purposes, such as grovelling information from libraries (kind of TAGs files), metaprogramming, etc. All these things should be doable with ASDF if the API is fixed with a long term scope.

> asdf-ecl.lisp only defined methods on COMPONENT-DEPENDS-ON,
INPUT-FILES,
> OUTPUT-FILES, PERFORM and OPERATION-DONE-P. All the signatures we used
> contain at least either OUR components our OUR operations, so in that
> sense they should be allowed to exist and ASDF should be careful on
> deciding what to do with these methods -- for instance, rewriting
paths
> as we discussed on another thread.

I thought you were specifically referring to core methods.  I see that I
was not fully understanding.

Ok, so then we may consider that these are non-volatile definitions, which is great because as I said this is basically all one really needs to build things on top of ASDF!

I think so, but see my notes up above. I don't know how to handle cases like

someone wants to write a method for (MY-OP T) but the core wants to write methods for (T COMPONENT-CLASS). How do we avoid or arbitrate these cases? Does anyone have a simple rule to apply here?

> At some point the extensions were accepted in the source tree,
point at
> which I asked whether the integration that was done the right way or
> not. The lack of answer was understood as an ok.

This exhibits two problems ASDF development.  The underlying problem is
that people are doing this work on very thin time slices.

1.  Lack of answer can never effectively be treated as "ok" --- lack of
an answer is more likely to simply mean "I haven't had time to look at
this."  My changes to TRAVERSE went in a very long time ago, and only
after a very long discussion in the mailing list.  It's clear that you
didn't have time to review them then, and so an issue that we thought
was closed will have to be reopened.

I miss a lot of emails when the subjects are not directly related. On reading the archive of emails I see what you were discussing -- to me it just looked like a minor detail in dependency binding and nothing that would change the logic of TRAVERSE.

Right. And I missed a lot of your ECL discussions for the same reasons.

2.  The documentation is badly bit-rotted.  The section on extending the
ASDF object model is in particularly bad shape.  It was never complete,
as far as I can tell, and it has not been updated.  We badly need a
protocol specification stating exactly what has to be defined to make
new components and operations.  I can't say that I fully understand how
to do this --- my experiments have often seemed to go just fine, then
led to me stumbling much later on a problem that came up because some
critical method was missing.

I must say I am basically in the same position. That is why I ask so often and may sound sometimes like nagging. Seems we are all a bit insecure on how ASDF should behave, maybe two things would help -- one would be writing down a description of what the main function, TRAVERSE, does, another one is perhaps enlarging the set of tests so that we all feel safer about experimentation.

I would actually like to have an abstract description of the task. That seems preferable to describing what TRAVERSE does, since I think the task would leave us free to implement TRAVERSE in different ways, and have users just know "we will invoke perform on all these nodes, under these circumstances," where "these circumstances" are dictated by dependencies and the state of the file system.

Also, I think the current TRAVERSE is somewhat broken with respect to the abstract problem description. Some early decisions, reasonable at the time, seem to me to have trapped us in unfortunate ways.

I've actually been trying to sketch out an abstract description of ASDF as an AI planning problem, but I keep being side-tracked by the need to work for money!

Best, R

Juanjo

-- Instituto de Física Fundamental, CSIC c/ Serrano, 113b, Madrid 28006 (Spain) http://tream.dreamhosters.com

On 3/17/10 Mar 17 -5:36 PM, Faré wrote:

Dear friends,

let's please focus on the technical problem and drop the nagging. We all want the same thing: an ASDF that works for everyone, including ECL, and ECL is not to be considered a mere client, but an integral part of ASDF. And we all suffer from the same breakages that waste our time. And so let's talk about solutions rather than about blame.

From Juanjo's and Robert's descriptions, it seems to me that the TRAVERSE API could be improved to satisfy everyone.

Maybe it's a matter of splitting TRAVERSE into parts, and allowing these parts to recurse or not on dependencies depending on various flags.

I think the main issue is being able to recurse within the scope of some system or module or not: we want to be able to tell TRAVERSE "here's a predicate or description that tells how far you go", maybe recording on the side a queue of things that haven't been traversed but need to be done nonetheless.

So, traverse would take as an additional argument some specifier for the scope of the traversal (system, module, predicate, etc.), and return multiple values: the list of inside ops (fully recursed into), and the list of outside ops (not recursed into).

Would something like that help?

For all TRAVERSE's failings, I'd prefer not to see the API complicated. I'm inclined to think it's already a bug that one has to understand TRAVERSE as much as one does. Also, if we expose TRAVERSE, we are more nailed into it. You and I have already discussed the fact that it would be nice if we could fix the INTER-system dependency problem. If we expose the TRAVERSE API, fixing this kind of bug will only get harder, because the slice we expose as supported will be wider.

Finally, I'm concerned that this would cause an explosion in our testing needs. If we have N different TRAVERSE configurations, we'll have to test them against N different situations.

How about my suggestion that we allow some operations to be defined as system operations that do not recurse into their children?

Would that solve the problem?

Note that that means traverse will visit those nodes, but we have not promised anything about the behavior of traverse, nor about its return value. To the limited extent that we can say that ASDF specifies anything, I think one could say that it tells you that it will do all the operations in its plan, and it actually tells you you need to define them in some cases to be No-ops.

best, r

On 3/17/10 Mar 17 -4:33 AM, Juan Jose Garcia-Ripoll wrote:

On Wed, Mar 17, 2010 at 9:58 AM, Juan Jose Garcia-Ripoll <juanjose.garciaripoll@googlemail.com mailto:juanjose.garciaripoll@googlemail.com> wrote:

Before, if an operation defined by me, such as LIB-OP, did not
traverse a system, then the only operation we got was that operation
applied on the system, as a generic object.

Now TRAVERSE not only imposes the LIB-OP operation on the system,
but it also scans the components and imposes the operation on the
components. 

Instead of having TRAVERSE use some kind of hard-coded logic as it does now

        (let ((module-ops
               (when (typep c 'module)

[...] (dolist (kid (module-components c)) (handler-case

why not go back to something more reasonable which is based on COMPONENT-DEPENDS-ON

(defmethod component-depends-on ((o operation) (c system)) ...)

I tried to come up with a cleaner method when I rewrote TRAVERSE to fix the module dependency bug. I couldn't find a way to do this.

The short answer is that the problem comes from the way ASDF uses MODULE components to represent two different things at once:

1. The operation to be done on the module as a whole (including the operation done on its children) and

2. Any clean up operations that don't live on a component (let's say you want to write an entry in a table after a module is combined).

Recall that PERFORM applied to a module is done AFTER any operations on the component's children. Similarly, OPERATION-DONE-P on the module pays no attention to whether the children are done, nor does OPERATION-DONE-P in general pay any attention to the dependencies.

If you think THIS bug fix was bad for backward compatibility, believe me, doing something "more reasonable" would make your head swim!

That said, if you have a better patch, feel free to provide it. Other than that, I discussed this on the list till I was blue in the face while the modifications were underway. This will be my last message on the subject; please read over the archive of the list to see why TRAVERSE was modified in the way it was. If you have any remaining questions, I will be happy to entertain them, but I am not going over this ground again.

Best, R