Dear Juanjo,

...

I must admit that after four hours fighting with this I am right now a bit pissed off, so please forgive the tone of the message.

I apologize for the complications.

...

ASDF has introduced a new caching system which alters the names of all output files. This is making me mad because I am trying to implement pre-built ASDF components for ECL  and ASDF insists on changing the value of output-files and input-files, looking for those prebuilt files at the location of the cache.

I forgot to say this is 1.604 That seems to be one source of problems: logical pathnames are only kept untouched in recent versions and . However, recent versions of ASDF do not work with ECL because a generic function has been removed, component-relative-pathname. I attach a diff of the required changes.

Oops. I suppose it is best to upgrade. In 1.633 we indeed don't apply translations to logical pathnames (LPNs), since whoever uses LPNs probably instead want his own LPN translations to take precedence. I merged your diff to asdf-ecl.lisp into 1.634. I'm sorry I didn't test and find the bug earlier. I would be wonderful if you contributed a test script to the test suite so I'd find breakages earlier.

In any case I still find it problematic that all physical pathnames are translated and that no simple way of keeping ECL's system directory untouch remains. I tried with

#+ecl (,(translate-logical-pathname "SYS:") ())

but the resulting initialization is

...

(asdf::initialize-output-translations)

(#P"/Users/jjgarcia/lib/ecl-10.3.1/" #P"/Users/jjgarcia/lib/ecl-10.3.1/")

so that the system directory pathname NEVER matches any translation. The previous list should instead read

(#P"/Users/jjgarcia/lib/ecl-10.3.1/**/*.*" #P"/Users/jjgarcia/lib/ecl-10.3.1/**/*.*")

You hit another corner case! To allow the user to specify arbitrary translations, not just ones with **/*.* at the end, we don't add **/*.* to specified pathnames, only to string-designated pathnames. Thus, what you wanted was:

#+ecl (,(translate-logical-pathname "SYS:**;*.*") ())

It's also what I committed to 1.634.

(This was documented in the manual, but it's a small note in a big manual indeed.)

PS: if you're going to have some magic ECL-provided ASDF systems that are tied to the implementation, you should probably want to:

A- add to asdf.lisp's wrapping-source-registry something like: #+ecl (:tree ,(translate-logical-pathname "SYS:magic-systems;")) This takes precedence over any user-supplied configuration. SBCL uses it for its internal magic systems. B- make sure you do *NOT* provide asdf.asd in *that* directory, since combined with the above, that would prevent users from installing their own asdf upgrade by taking precedence. C- if you want to bundle systems that are not too tied to ECL internals, you may want to add to default-source-registry something like: #+ecl (:tree ,(translate-logical-pathname "SYS:bundled-systems;")) There is no problem with providing asdf.asd there, though it's probably redundant with the magic from (require :asdf).

The horror! The horror!

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] There exist two ways to get rich: the way of economics, that of industrialists, workers and tradesmen, that consists in freely exchanging the fruits of one's labor; and the way of politics, that of robbers, bureaucrats and politicians, that consists in grabbing by force the fruits of the labor of others.

On Sat, Mar 13, 2010 at 5:08 PM, Juan Jose Garcia-Ripoll < juanjose.garciaripoll@googlemail.com> wrote:

Thanks again for your help. Using the latest changes in ASDF ECL is now capable of building and shipping its own versions of CLX and other tools in a way that can then be linked to other FASL, programs or shared libraries using the ASDF extensions we already had.

I was just thinking how to extend this functionality to user-defined systems and hit another problem. The goal is to take an ASDF system and prepare a bundle that contains a PREBUILT-SYSTEM file (*.asd), a library file (*.a or *.lib) and a FASL (*.fas) This set of files can then be distributed elsewhere.

The problem is that the prebuilt system definition must contain the location of these other files,

(DEFSYSTEM "user-file" :CLASS ASDF::PREBUILT-SYSTEM :LIB "user-file.a" :COMPONENTS ((:COMPILED-FILE "serve-event")))

Currently this works for ECL's contributed extensions because we hardcode logical pathnames, as "SYS:user-file.a"

However in general these files would be stored together with the *.asd file so their full pathnames could be resolved based on the former. How can I do that without getting the pathnames get overwritten by ASDF?

Changing output-translations is not an option, because this would have global effects.

Juanjo

On Sat, Mar 13, 2010 at 8:50 PM, Faré fahree@gmail.com wrote:

Where is the file to be when you compile? Where is the file to be when you load? How do you currently do things, and what breaks? Could say, a logical pathname host LIB: help locate the libraries? How do you do things without ASDF?

There is not going to be any file that gets compiled. There is not going to be any lisp sources.

The goal is to replace a set of sources + asdf file with a set of compiled files (one static library, one FASL files) and a hacked *.asd file that relies only on those binary files.

An important component is that users are able to build these sets of *.asd+*.lib+*.fasl file with something like (asdf:make-standalone-system 'some-system-name) and then installed anywhere.

I mean that the set of files should then be installed anywhere the user wants -- as far as it is a path reachable by ASDF and the files remain together, in the same directory. Using logical pathnames is not an option because we can not know where the user wants to have the files.

What is the purpose of this? Well, it would make software distribution very easy, in particular for Linux-like distributions, but it would also be useful for users that want to deliver a program and a set of extensions in the form of ASDF system definitions -- a tool they are already familiar with --.

Juanjo

On Sun, Mar 14, 2010 at 1:36 AM, Faré fahree@gmail.com wrote:

1- you want an ASDF operation that dumps some description of things to be loaded. 2- you want to use a .asd itself as the output of that operation. 3- you want some location-independence in that description.

I presume the role of 2- is so that the result be somewhat interchangeable with the original source being packaged? But is that the case? Does interchangeability really bring more value than it creates confusion?

No, interchangeability means not chance of using the library as it was used. Not having it means that new ways of loading code and linking it into executables of libraries should be found. New functions, new operations, new steps to configure. In other ways, it imposes and additional burden on the user.

The proof that the current scheme is good is the fact that ECL already delivers some key libraries in this form. For instance CLX. The result is that anyone who wants to build a standalone executable including Common Lisp, CLX and its own custom code may do so using the distributed binary -- no need for sources --. Stumpwm is one user of this, resulting in a really small executable, 1.5Mb, that just works.

For instance, XCVB-master (the module that can control XCVB from within current Lisp image) uses "manifest" files (or SEXPs) as a delivery mechanism telling it what files to load -- however, said mechanism is anything but location-independent, unless you use logical pathnames.

It's your design choice. I believe that distributing together, in the same directory, the *.ASD file and the binaries, is just fine.

Now, maybe logical pathnames are exactly the mechanism to use here, whether you use ASDF, XCVB-master, or some other homegrown delivery target.

No option. What you say is something like for every utility they should cook up a logical hostname? They would then have to store those logical pathname definitions in some other file and match the place where they install the files....

That sounds like imposing an unneeded burden on users of this extension just because there is one single problem in ASDF (not in the extension), namely __all physical pathnames gets translated into the cache directory__

We just need a way to selectively deactivate those translations from user written code. Say, I somebody writes an OUTPUT-FILES she should be able to tell ASDF that no translations should be applied to those paths. This is also important for platform independent resources, as it has been discused in c.l.l.

If you remove this problem somehow, I will stop bothering the mailing list :-) Really, if that is removed then the extension we cooked should just work for user code as well.

...

I mean that the set of files should then be installed anywhere the user wants -- as far as it is a path reachable by ASDF and the files remain together, in the same directory. Using logical pathnames is not an option because we can not know where the user wants to have the files.

I think there's a key constraint I don't understand here.

The desire is that people should install things wherever they want. Is that really a "constraint"?

Independently from how they are represented in the delivery file, how are you supposed to locate the components being delivered?

No common file. Just three files, say serve-event.asd, serve-event.fas and libserve-event.a are stored all together in some directory /usr/local/lib/asdf/x86-32/serve-event/ and then the user adds the tree /usr/local/lib/asdf to the set of directories that ASDF scans. Think of a software distribution that provides serve-event.rpm as a pre-compiled version of that library. This can be pretty well integrated with ASDF if, as I say, you just lift the restriction of translating ALL pathnames.

Are they to be in a path relative to e.g. the truename of the description? In some known system path such as /usr/lib?Anywhere in some library search path? Some combination of the above? Note that if a search path is required, you'll need some additional facility anyway.

Wherever they want. The point is that we are not allowed to tell people how they should install their own software. There is no additional facility needed: they just store the three files together in a search tree taught to ASDF and I can code the OUTPUT-FILES to resolve the component names (libserve-event.a, serve-event.fas) based on the location of the *.ASD file. This can be extended to libraries that carry resources with them -- bitmaps, binary files, and the like.

If they change the logic of how *.a and *.fas files are located then they could also write methods for OUTPUT-FILES, but the problem is that right now the values returned by their methods will always get rewritten by the pathname translation utility.

Please do not tell me that, given that the tree has to be registered with ASDF anyway, then they should also add the path to the list of pathnames that do not get translated. That is a very weak link prone to break. Furthermore, in the same tree of directories one might want source and binary versions of ASDF packages.

However, if you want to forget all this blah, blah, blah, it is also ok. Just remember that what I said above applies: we need a way to write code in ASDF whose pathnames do not get arbitrarily overwritten. This facility is seriously needed for ASDF extensibility. Another example. Say, for instance, that one ASDF extension decides to code temporary files and wants them stored in /var/tmp so that they get regularly cleaned up by the OS. Again, either does it push the path in the list of exceptions, or it is fracked.

Juanjo

On Mon, Mar 15, 2010 at 2:33 AM, Faré fahree@gmail.com wrote:

...

directory, the *.ASD file and the binaries, is just fine.

It's your design choice. I believe that distributing together, in the

same OK. But I don't see what is your problem, then?

I mean, you can either write a new component type for libraries to be loaded without being compiled (i.e. compile-op is a nop, load-op loads the binary), or just define a special method that does all the loading.

The problem is that this extends beyond load-op. There are currently six other operations for six other delivery modes in asdf-ecl.lisp -- fasl, shared and static library, monolithic or not --. The operations may depend and use those prepackaged binaries, not because they are applied by themselves, but because the component that they get use as input files the "output files" of the static bundle.

...

written code. We just need a way to selectively deactivate those translations from user

Why need translations at all? Just don't use OUTPUT-FILES in your target .asd. When you create a target .asd, use components that refer directly to the things to be loaded. Your class compiled-file looks great.

DLL-OP on system A that depends on system B, which is in binary form must look at the dependencies and see what output files they produce. B in this case has a static library libb.a which is used by its children and this is listed as "output-files".

So I suppose that what you were asking for all that time, that I was failing to understand all along, was the ability for OUTPUT-FILES to say return a (CONS PATHNAME NIL) meaning "I want this pathname, untranslated" and the outer-most :AROUND method would detect that, then pass the inner PATHNAME untranslated.

That's totally doable. Would adding a case

(cons (pathname (car path))) to apply-output-translations satisfy you?

Indeed. This or any other way that does not break current ASDF. What I would like is seamless integration without imposing something that is prone to break in the future. Another possibility would be for OUTPUT-FILES to return two values: the list and an optional second value saying "do not translate!" This second value will be ignored by existing functions but it may be used by apply-output-translations It also has the advantage that the main output value remains in the same format and we do not introduce a corner case that might break ASDF's logic.

Alternatively, I could cook up an API to easily add a mapping to output-translations on the fly (if not already there).

This could be problematic if we keep in the same tree ASDF files for projects that are shipped in binary and in source form.

So what about marking paths as "not to be translated" by returning

(list path) instead of path?

I will use whatever API you choose. Thanks again for taking your time to read my long emails!

Juanjo

good afternoon;

this does not sound like a case for specialization. it sounds more like delegation. what about hooking an output translator into the components. if it is there, it is used. if it is not, no translation happens. (as much as i may regret this) the "standard" output translation is supplied as the default initform / or default initarg. if one supplied nil in the initialization form, that is used instead and no translation happens.

On 2010-03-15, at 16:47 , Faré wrote:

Dear Juanjo,

...

Indeed. This or any other way that does not break current ASDF. What I would like is seamless integration without imposing something that is prone to break in the future. Another possibility would be for OUTPUT-FILES to return two values: the list and an optional second value saying "do not translate!" This second value will be ignored by existing functions but it may be used by apply-output-translations It also has the advantage that the main output value remains in the same format and we do not introduce a corner case that might break ASDF's logic.

Your API sounds great. What about this?

(defmethod output-files :around ((op operation) (c component)) "Translate output files, unless asked not to" (multiple-value-bind (files fixedp) (call-next-method) (if fixedp files (mapcar #'apply-output-translations files))))

If you're OK with it, I'll push it as ASDF 1.635.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http:// fare.tunes.org ] Faré's Second Law of Dissent: I am Right, whence it follows that all who disagree with me are either (1) evil, (2) stupid or (3) crazy. (The alternatives are not mutually exclusive.) This universal law is valid for all values of "me", including "you".

---

asdf-devel mailing list asdf-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

On Mon, Mar 15, 2010 at 4:47 PM, Faré fahree@gmail.com wrote:

(defmethod output-files :around ((op operation) (c component)) "Translate output files, unless asked not to" (multiple-value-bind (files fixedp) (call-next-method) (if fixedp files (mapcar #'apply-output-translations files))))

What about this:

(defmethod output-files :around ((op operation) (c component)) "Translate output files, unless asked not to" (values (multiple-value-bind (files fixedp) (call-next-method) (if fixedp files (mapcar #'apply-output-translations files))) t))

It has the advantage that the :AROUND method also follows the API and other functions calling OUTPUT-FILES get to know that the paths are now fixed.

...

(defmethod output-files :around ((op operation) (c component))  "Translate output files, unless asked not to"   (values    (multiple-value-bind (files fixedp) (call-next-method)      (if fixedp          files          (mapcar #'apply-output-translations files)))    t))

It has the advantage that the :AROUND method also follows the API and other functions calling OUTPUT-FILES get to know that the paths are now fixed.

Yup, that's better. I'm putting that in my local copy, to be released this week (hopefully after having a chance to look at janderson's pathname test).

Isn't this a job for

(defmethod output-files asdf:around ((op operation) (c component) ...)

?

Meh, right at the moment I was considering getting rid of asdf:around...

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] "I think sex is better than logic, but I can't prove it." — Monty Python

On 15 March 2010 15:37, Robert Goldman rpgoldman@sift.info wrote:

On 3/15/10 Mar 15 -3:20 PM, Faré wrote:

...

...

...

(defmethod output-files :around ((op operation) (c component))  "Translate output files, unless asked not to"   (values    (multiple-value-bind (files fixedp) (call-next-method)      (if fixedp          files          (mapcar #'apply-output-translations files)))    t))

It has the advantage that the :AROUND method also follows the API and other functions calling OUTPUT-FILES get to know that the paths are now fixed.

Yup, that's better. I'm putting that in my local copy, to be released this week (hopefully after having a chance to look at janderson's pathname test).

...

Isn't this a job for

(defmethod output-files asdf:around ((op operation) (c component) ...)

?

Meh, right at the moment I was considering getting rid of asdf:around...

Right, but the programmer is likely to want to be able to have his/her own :around method on this function signature....

What about we reserve (defmethod foo :around (op c) ...) but advertise (defmethod foo :around ((op operation) (c component)) ...) as user-definable?

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] You don't have to like everything about me, but if you don't love me the way I am, it's not me you love, only some fantasy of yours.

On Mon, Mar 15, 2010 at 5:19 PM, Robert Goldman rpgoldman@sift.info wrote:

Isn't this a job for

(defmethod output-files asdf:around ((op operation) (c component) ...)

Wrapping around a method that undoes the change that another around method

was doing looks definitely like a bad hack to me. Instead, providing an API for a method to advertise that no around method should mess with its output looks like something predictable and which does not lead to double-coding.

Juanjo