...

Perhaps a better solution would be to export OPERATION-FORCED and TRAVERSE, so that this could be deal with where-ever is needed -- especially in local initialization files for systems installed in read-only locations. The SBCL solution would look about the same -- it just would not reside inside asdf.lisp.

This seems like a reasonable solution.

I agree. There are probably two items here:

1. Allowing a system definer to say "don't descend into this sub- system" even when :force is t.

2. Allowing a system user to say "I want to force the recompilation of this system but not the systems on which it depends"

For #1, I'd suggest adding a slot to system that controls how force behaves. Ideas include

* read-only (to mean that the system cannot be edited or recompiled.) * ignore-force (to mean, well, ignore force).

For #2, I'd suggest using

* recompile everything

* :force t * :force :descend

* recompile only the current system

* :force :system-only

Both of these seem worthwhile to me. I'd even have to implement them tonight if people like the idea!? (he said optimistically).

P.S. I have a vague recollection about weakly dependent systems but don't remember ever really understanding them. Are they part of a solution to this problem.

-- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter

2009/5/12 Robert Goldman rpgoldman@sift.info:

...

I agree. There are probably two items here:

1. Allowing a system definer to say "don't descend into this sub-system"

even when :force is t.

Instead of (or in addition to) #1 I think there is call for a way to user (or sysadmin, etc) say "don't recompile these systems".

Consider a shared installation of lisp libraries under /usr/local/asdf/:

(defvar *ro-systems* (directory "/usr/local/lib/asdf/*.asd"))

(defmethod traverse :before ((o compile-op) (s system)) (when (and (operation-forced o) (member (system-definition-pathname s) *ro-systems* :test #'equal)) (warn "Ignoring :FORCE T for COMPILE-OP of system installed in ~ /usr/local/asdf/: ~:@(~A~)." (component-name c)) (setf (operation-forced o) nil)))

...

For #1, I'd suggest adding a slot to system that controls how force behaves. Ideas include

* read-only (to mean that the system cannot be edited or recompiled.)     * ignore-force (to mean, well, ignore force).

I think both of these are fairly limited in scope: libraries installed along with lisp implementations and libraries which break horribly upon recompilation seem like the only ones which should use these, and in in case of implementations installing 3rd-party libraries they should not really be editing the system -- so a protocol extensions seems necessary even in the presence of these options / slots.

...

For #2, I'd suggest using

• recompile everything

* :force t     * :force :descend

• recompile only the current system

* :force :system-only

Should there be a way for user to say "reload everything despite the timestamps, but don't recompile anything"?

Cheers,

-- Nikodemus

Gary King writes:

...

...

Perhaps a better solution would be to export OPERATION-FORCED and TRAVERSE, so that this could be deal with where-ever is needed -- especially in local initialization files for systems installed in read-only locations. The SBCL solution would look about the same -- it just would not reside inside asdf.lisp.

This seems like a reasonable solution.

I agree. There are probably two items here:

1. Allowing a system definer to say "don't descend into this sub-

system" even when :force is t.

This sounds fishy to me. AFAIU, system definers aren't supposed to be able to make decisions about installation, at least in part because the author of a system can't really know what the user's installation requirements are. (Cf. the rationale for telling users to customize fasl location with a method on OUTPUT-FILES.)

-- Richard

2009/5/12 Robert Goldman rpgoldman@sift.info:

If we wish to keep this design principle, it seems that we should provide a facility for installers to annotate asdf system definitions. Presumably these annotations should be "on the side" and not in the system definition files themselves.

This is what the original proposal to export COMPONENT-FORCED and TRAVERSE was for.

Cheers,

-- Nikodemus

2009/5/13 Robert Goldman rpgoldman@sift.info:

Ah.  I see.  Question:  do we foresee some protocol for infiltrating the find-system process so that an installer can drop an ancillary file in that will be loaded before or after the system definition file that is written by the developer.

I'm not sure I follow -- specifically I'm not sure what you mean by installer in this context.

For *my* purposes I see this being something that is best handled in site/user initialization files. In case of SBCL contribs, we might start installing a default

/usr/local/lib/sbcl/sbclrc

or alternatively appending the code to each .asd file we install (eugh).

Cheers,

-- Nikodemus

2009/5/13 Robert Goldman rpgoldman@sift.info:

I remain confused on "what installer" -- but nevermind.

I was also thinking "eugh" in response to that alternative.  So I

Just the be clear, the "eugh" did not apply to the first option, which seems the most natural one to me.

figured it might be desirable to allow an installer to take an upstream library, together with its .asd file, and modify only his/her version of find-system so that when <library>.asd was loaded, we would also look for (for example) <library>.install or <library>.config, and load those right after the .asd file to impose any necessary site-specific configuration.

I'm opposed to adding number of files responsible for the behaviour of the system as long as there are alternatives. They make debugging harder, and test case reproduction doubly so -- especially when they can contain arbitrary code.

That said, I think perhaps adding umpteen methods to TRAVERSE is not the right hammer here. Maybe a *READ-ONLY-SYSTEMS*, list, which it would be easy to push things onto.

Alternatively, if complex configuration is wanted, I would propose a hash-table mapping system names to asdf:configuration objects, and a DEFINE-SYSTEM-CONFIGURATION macro to populate it -- population would be done in initialization files (or lazy-loaded files via some hook if one really has to.)

Cheers,

-- Nikodemus

2009/5/13 Robert Goldman rpgoldman@sift.info:

I guess that this is a matter of aesthetics, about which we'll have to agree to disagree. I think small snippets of code that are located close to the things that they are supposed to configure are easier to track, understand, and even test than stuff that's squirreled away non-locally (e.g., in /usr/local/etc/sbclrc).

To a degree, yes. However, "X doesn't work on my system" is made a lot harder to reproduce / debug remotely the more sources of code there are. Also, ASFD just deprecated preference files some time back -- and I'm not sure how configuration files would differ from preference files.

I think there are a a couple of facts about what I'm doing, and how it differs from what you are doing that may make for this difference of opinion.

I'm concerned with delivering applications, rather than delivering a lisp implementation, which seems more like your concern.

This means that I'm typically not messing with a site's global lisp configuration, but rather local configurations.

I would have imagined that most lisp applications would be delivered independently of the lisp development framework possibly at place. (Which in my books includes all the .lisp, .asd and .fasl files: the application either delivers all it's own ancilliary files or is a single executable.) Maybe this is not so.

Can you describe your typical delivery scenario?

I'm also often working with multiple lisp implementations concurrently, which also leads me to see this as an issue of configuring the individual libraries.

For that matter, I'm typically working with > 1 different lisp project, in different lisp configurations (the projects have different source repositories, assume different libraries, etc.). So I typically work with multiple different asdf:*central-registry* configurations.

This is something I have almost no experience in. Can you outline the major issues as you see them?

Cheers,

-- Nikodemus

Nikodemus Siivola wrote:

2009/5/13 Robert Goldman rpgoldman@sift.info:

...

I guess that this is a matter of aesthetics, about which we'll have to agree to disagree. I think small snippets of code that are located close to the things that they are supposed to configure are easier to track, understand, and even test than stuff that's squirreled away non-locally (e.g., in /usr/local/etc/sbclrc).

To a degree, yes. However, "X doesn't work on my system" is made a lot harder to reproduce / debug remotely the more sources of code there are. Also, ASFD just deprecated preference files some time back -- and I'm not sure how configuration files would differ from preference files.

...

I think there are a a couple of facts about what I'm doing, and how it differs from what you are doing that may make for this difference of opinion.

I'm concerned with delivering applications, rather than delivering a lisp implementation, which seems more like your concern.

This means that I'm typically not messing with a site's global lisp configuration, but rather local configurations.

I would have imagined that most lisp applications would be delivered independently of the lisp development framework possibly at place. (Which in my books includes all the .lisp, .asd and .fasl files: the application either delivers all it's own ancilliary files or is a single executable.) Maybe this is not so.

Can you describe your typical delivery scenario?

We often work in a distributed way, so there are really two different "delivery" scenarios:

1. To the user. This is very much along the lines that you have outlined above --- we try to be independent of the lisp environment (if any) on the site. Indeed, when we can manage it, we find the best delivery solution is often a live cd. This is because we often want to work with other software packages (e.g., ImageMagick, Grass GIS, etc.). In that case, even if one can manage all the across lisp oddities, there's still the question of how daemons are started, what libraries and other applications are installed, etc. A nightmare!

2. To different developers, often at different companies. In this case, we typically cannot control the lisp development environment (and we may be working in a team where some developers are using ACL, CCL, and SBCL). In this case, which is probably most relevant to our discussion here, we try to make sure that all libraries are in the revision control system (which means we typically mirror open source CL libraries into our own SVN), and we use a solution that we have developed that will populate the developer's asdf:*central-registry* in a way that creates a coherent development environment across different developers, machines, and sites.

For these cases, we typically do not try to hijack the developer's lisp init file, for two reasons --- first, the user typically has configurations that make things easy and familiar for him/her and second, we would have to provide multiple init files (for the different lisps).

We'd probably be happiest doing something that I've described above --- adding modifications that are proximal to the individual libraries, rather than putting them in a central configuration file --- because this allows us to share those modifications across multiple copies of the same library, in different svn repositories. When we move from one project to another, we'd rather have our configuration for a library with the library, instead of remote.

I hope that I haven't bored you with this long account...

...

I'm also often working with multiple lisp implementations concurrently, which also leads me to see this as an issue of configuring the individual libraries.

For that matter, I'm typically working with > 1 different lisp project, in different lisp configurations (the projects have different source repositories, assume different libraries, etc.). So I typically work with multiple different asdf:*central-registry* configurations.

This is something I have almost no experience in. Can you outline the major issues as you see them?

I think the discussion above should mostly explain things. One additional point I should probably make:

1. We have developed an 'asd-finder' in pure common-lisp that can be used to find all the .asd files in a directory subtree(s). This allows us to populate asdf:*central-registry* from an SVN repository working copy without fussing with symbolic links, etc. It also is robust to the addition of new libraries w/o fussing with configuration files.

2. We have a macro that defines a simple command that points the asd finder at a bunch of directories and populates the asdf:*central-registry*. This means at the start of a programming session, I can start my lisp, type a one-word command that will give me the right asdf context, and then can start work from there (typically, but not always, by loading the top-level asdf system).

The above routine, at the expense of a limited overhead at the start of a programming session, saves us from the muss and fuss of building symlink farms, and makes starting work on a new machine a mere matter of checking out an svn repository.

Best, Robert