[asdf-devel] asdf output locations
My plan for ASDF 1.600 is to rename the (recently) builtin asdf-binary-locations to asdf-output-locations and make a few incompatible changes to it. * These changes will allow for configuration through an API similar to the one I introduced for the source-registry in 1.500. * I don't think many people use bleeding-edge enough ASDF to have the builtin ABL and these kinds of people would easily adapt anyway * This API will allow the future common-lisp-controller to plug nicely into ASDF through a supported API that doesn't break other stuff and isn't broken by it. * It will also bring all the binary location advantages of asdf-binary-locations, common-lisp-controller or CL-launch to everyone with a single, well-documented, supported configuration mechanism. I committed the following document to explain my current plans, and would appreciate comments: http://common-lisp.net/gitweb?p=projects/xcvb/asdf.git;a=blob;f=README.asdf-... [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Wishing without work is like fishing without bait. — Frank Tyger
Faré <...> writes:
My plan for ASDF 1.600 is to rename the (recently) builtin asdf-binary-locations to asdf-output-locations and make a few incompatible changes to it. * These changes will allow for configuration through an API similar to the one I introduced for the source-registry in 1.500. * I don't think many people use bleeding-edge enough ASDF to have the builtin ABL and these kinds of people would easily adapt anyway * This API will allow the future common-lisp-controller to plug nicely into ASDF through a supported API that doesn't break other stuff and isn't broken by it. * It will also bring all the binary location advantages of asdf-binary-locations, common-lisp-controller or CL-launch to everyone with a single, well-documented, supported configuration mechanism.
I committed the following document to explain my current plans, and would appreciate comments:
http://common-lisp.net/gitweb?p=projects/xcvb/asdf.git;a=blob;f=README.asdf-...
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Wishing without work is like fishing without bait. — Frank Tyger
Thanks for time you're investing in this. I think you could just have appended the writeup to your mail, as it results in one step less people have to take to read it, and it's easier to reply in context. I'll do so. The PS is README.asdf-output-locations written by Fare. -T. PS. ===================== ASDF Output Locations ===================== This file specifies how ASDF stores "binary" outputs for its operations, typically Lisp FASL files, but also any other files that may be generated, e.g. C files and executables from CFFI-GROVEL. Configurations ============== Configurations specify mappings from source locations to binary locations. 1- An application may explicitly initialize the binary-locations configuration using the `Configuration API`_ below, in which case this takes precedence. It may itself compute this configuration from the command-line, from a script, from its own configuration file, etc. 2- The source registry will be configured from the environment variable ``ASDF_OUTPUT_LOCATIONS`` if it exists. 3- The source registry will be configured from user configuration file ``~/.config/common-lisp/asdf-output-locations.conf`` if it exists. 4- The source registry will be configured from user configuration directory ``~/.config/common-lisp/asdf-output-locations.conf.d/`` if it exists. 5- The source registry will be configured from system configuration file ``/etc/common-lisp/asdf-output-locations.conf`` if it exists. 6- The source registry will be configured from system configuration directory ``/etc/common-lisp/asdf-output-locations.conf.d/`` if it exists. Each of these configurations is specified as a SEXP in a trival domain-specific language (defined below). Additionally, a more shell-friendly syntax is available for the environment variable (defined yet below). Each of these configurations is only used if the previous configuration explicitly or implicitly specifies that it includes its inherited configuration. Additionally, some implementation-specific directories may be automatically added to whatever mappings are specified in configuration files, no matter if the last one inherits or not. Backward Compatibility ====================== We purposefully do not provide backward compatibility with earlier versions of asdf-binary-locations (8 Sept 2009), common-lisp-controller (6.17) or cl-launch (2.35), each of which had similar general capabilities. Future versions of same packages (if any) will hopefully use the new ASDF API as defined below. Indeed, few people use and customize these packages; these people are experts who can trivially adapt to the new configuration. Other people will experience software that "just works". Configuration DSL ================= Here is the grammar of the SEXP DSL for asdf-output-locations configuration: ;; A configuration is single SEXP starting with keyword :source-registry ;; followed by a list of directives. CONFIGURATION := (:asdf-output-locations DIRECTIVE ...) ;; A directive is one of the following: DIRECTIVE := ;; add a single directory to be scanned (no recursion) (:map DIRECTORY-DESIGNATOR DIRECTORY-DESIGNATOR) | (:include PATHNAME-DESIGNATOR) ;; Your configuration expression MUST contain ;; exactly one of either of these: (:inherit-configuration) | ; splices contents of inherited configuration (:ignore-inherited-configuration) ; drop contents of inherited configuration DIRECTORY-DESIGNATOR := ABSOLUTE-COMPONENT-DESIGNATOR | (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) ABSOLUTE-COMPONENT-DESIGNATOR := STRING ;; namestring (directory is assumed, better be absolute or bust) PATHNAME ;; pathname (better be an absolute directory or bust) :HOME ;; designates the user-homedir-pathname ~/ :USER-CACHE ;; designates the default location for the user cache :SYSTEM-CACHE ;; designates the default location for the system cache :ROOT ;; the root of the filesystem hierarchy RELATIVE-COMPONENT-DESIGNATOR := STRING ;; namestring (directory is assumed) PATHNAME ;; pathname (better be a directory or bust) :IMPLEMENTATION ;; a directory based on implementation, e.g. sbcl-1.0.32.30-linux-x86-64 Relative components better be either relative or subdirectories of the path before them, or bust. You may use #+features to customize the configuration file. Include statements cause the search to recurse with the path specifications from the file specified. An inherit-configuration statement cause the search to recurse with the path specifications from the next configuration (see section Configurations_ above). Configuration Directories ========================= Configuration directories consist in files each contains a list of directives without any enclosing ``(:asdf-binary-locations ...)`` form. The files will be sorted by namestring as if by #'string< and the lists of directives of these files with be concatenated in order. An implicit ``:inherit-configuration`` will be included at the end of the list. This allows for packaging software that has file granularity (e.g. Debian's ``dpkg`` or some future version of ``clbuild``) to easily include configuration information about distributed software. Directories may be included by specifying a directory pathname or namestring in an ``:include`` directive, e.g.:: (:include "/foo/bar/") Shell-friendly syntax for configuration ======================================= When considering environment variable ``ASDF_OUTPUT_LOCATIONS`` ASDF will skip to next configuration if it's an empty string. It will ``READ`` the string as a SEXP in the DSL if it begins with a paren ``(`` and it will be interpreted as a colon-separated list of directories. Directories should come in pairs, each pair indicating a :map directive. The magic ``!`` entry indicates the splicing of inherited configuration rather than one of entry in a mapping pair. Semantics of Output Location Mappings =====================================
From the specified configuration, a list of mappings is extracted in a straightforward way: mappings are collected in order, recursing through included or inherited configuration as specified. To this list is prepended some implementation-specific mappings, and is appended a global default.
The list is then compiled to a mapping table as follows: for each entry, in order, resolve the first designated directory into an actual directory pathname for source locations. If no mapping was specified yet for that location, resolve the second designated directory to an output location directory add a mapping to the table mapping the source location to the output location, and add another mapping from the output location to itself (unless a mapping already exists for the output location). Based on the table, a mapping function is defined, mapping source pathnames to output pathnames: given a source pathname, locate the longest matching prefix in the source column of the mapping table. Replace that prefix by the corresponding output column in the same row of the table, and return the result. If no match is found, return the source pathname. (A global default mapping the filesystem root to itself may ensure that there will always be a match, with same fall-through semantics). Caching Results =============== The implementation is allowed to either eagerly compute the information from the configurations and file system, or to lazily re-compute it every time, or to cache any part of it as it goes. To explicitly flush any information cached by the system, use the API below. Output location API =================== The specified functions are exported from package ASDF. (initialize-output-locations) will read the configuration and initialize all internal variables. (clear-output-locations) undoes any output location configuration and clears any cache for the mapping algorithm. You might want to call that before you dump an image that would be resumed with a different configuration, and return an empty configuration. Note that this does not include clearing information about systems defined in the current image, only about where to look for systems not yet defined. (ensure-output-locations) checks whether output locations have been initialized. If not, initialize them. This function will be called before any attempt to operate on a system. If your application wants to override the provided defaults, it will have to use the below function process-output-locations. (process-output-locations X &key inherit collect) If X is a CONS, parse it as a SEXP in the configuration DSL, and extend or override inheritted configuration. If X is a STRING, first parse it into a SEXP as for the ASDF_OUTPUT_LOCATIONS environment variable (see above) then process it. If X is a PATHNAME, read the file as a single SEXP and process it. The inheritted configuration is provided in keyword argument inherit, itself a list of functions that take inherit and collect keyword arguments and defaulting to a list of functions that implements the default behavior. (translate-output-locations PATHNAME) Applies the configured output location translations to PATHNAME (calls ensure-output-locations for the locations). Credits ======= Thanks a lot to Bjorn Lindberg and Gary King for ASDF-Binary-Locations, and to Peter van Eynde for Common Lisp Controller. All bad design ideas and implementation bugs are to mine, not theirs. But so are good design ideas and elegant implementation tricks. -- Francois-Rene Rideau <...>
"Tobias C. Rittweiler" <...> writes:
I'll do so. The PS is README.asdf-output-locations written by Fare.
-T.
PS.
===================== ASDF Output Locations =====================
This file specifies how ASDF stores "binary" outputs for its operations, typically Lisp FASL files, but also any other files that may be generated, e.g. C files and executables from CFFI-GROVEL.
I think this is going to be base for the actual documentation. So I'll raise a few points I'd like to see in the final documentation. Two things: a) I don't particularly like the name Output Locations. I can understand that a different name is needed. "FASL Location", would probably be most intuitive to Lispers -- though it can be used for more stuff. A compromise would be to call the chapter in the actual documentation, "ASDF Output (FASL) Locations." b) The documentation should shortly reflect how this was, implicitly, done in the past, and to what problems it lead.
Backward Compatibility ======================
We purposefully do not provide backward compatibility with earlier versions of asdf-binary-locations (8 Sept 2009), common-lisp-controller (6.17) or cl-launch (2.35), each of which had similar general capabilities.
Future versions of same packages (if any) will hopefully use the new ASDF API as defined below.
Indeed, few people use and customize these packages; these people are experts who can trivially adapt to the new configuration. Other people will experience software that "just works".
Again two points: a) You should give short reason why backwards incompatibility is not provided. In particular, what the problems are with status quo. b) Let's say someone is using the old ASDF-Binary-Locations (the package as it was before the merge into ASDF proper), let's further say his implementation will upgrade to the new ASDF, and he will update to this new version of his implementation -- will the old ABL package break under his feet? Silently fail? -T.
Dear Tobias, thanks for your feedback.
a) I don't particularly like the name Output Locations. I can understand that a different name is needed. "FASL Location", would probably be most intuitive to Lispers -- though it can be used for more stuff. But there's is more than FASLs involved. For instance, SB-GROVEL and CFFI-GROVEL create intermediate C files, executable files, and generated Lisp files. Test operations could create test report files. Etc.
A compromise would be to call the chapter in the actual documentation, "ASDF Output (FASL) Locations." What about "ASDF Output Locations (FASL, etc.)"?
b) The documentation should shortly reflect how this was, implicitly, done in the past, and to what problems it lead. OK.
"Previously, asdf-binary-locations, common-lisp-controller, cl-launch, and possibly other hacks were being used to redirect where asdf stores fasls and other output files. These were hard to setup, as they required the user to insert configuration statements precisely at the right time between the point that these pieces of software were loaded and the point that ASDF was first actually used. With the new mechanism, users can maintain a global configuration file and not have to worry about loading an extension to ASDF, and enjoy all the benefits of this relocation mechanism, including any relocations managed by the operating system distribution."
a) You should give short reason why backwards incompatibility is not provided. In particular, what the problems are with status quo. "" These previous programs' API was not designed for easy configuration by the end-user in an easy way with configuration files, and their previous API didn't fit the new paradigm.
But this incompatibility won't inconvenience many people. Indeed, few people use and customize these packages; these people are experts who can trivially adapt to the new configuration. Most people are not experts, could not properly configure these features (except inasmuch as the default configuration of common-lisp-controller and/or cl-launch might have been doing the right thing for some users), and yet will experience software that "just works", as configured by the system distributor, or by default. ""
b) Let's say someone is using the old ASDF-Binary-Locations (the package as it was before the merge into ASDF proper), let's further say his implementation will upgrade to the new ASDF, and he will update to this new version of his implementation -- will the old ABL package break under his feet? Silently fail?
Depends. I think that the way I'm going to do things, if he (or someone else form him) configures only one of these mechanisms, only that mechanism will be used. But if he configures both, "interesting" conflicts may happen. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Faré's Second Law of Dissent: I am Right, whence it follows that all who disagree with me are either (1) evil, (2) stupid or (3) crazy. (The alternatives are not mutually exclusive.) This universal law is valid for all values of "me", including "you".
On 2/1/10 Feb 1 -2:53 AM, Faré wrote:
Dear Tobias,
thanks for your feedback.
....
a) You should give short reason why backwards incompatibility is not provided. In particular, what the problems are with status quo. "" These previous programs' API was not designed for easy configuration by the end-user in an easy way with configuration files, and their previous API didn't fit the new paradigm.
But this incompatibility won't inconvenience many people. Indeed, few people use and customize these packages; these people are experts who can trivially adapt to the new configuration. Most people are not experts, could not properly configure these features (except inasmuch as the default configuration of common-lisp-controller and/or cl-launch might have been doing the right thing for some users), and yet will experience software that "just works", as configured by the system distributor, or by default.
For the record, I would like to say that this is most emphatically not my experience. At my firm, we have long used asdf-binary-locations, since we develop software that runs on ACL, SBCL and may be branching out to CCL. I /never/ configure ASDF-BINARY-LOCATIONS -- it interrogates my lisp and computes a relative pathname like allegro-8.1m-64bit-macosx-x86-64 sbcl-1.0.28-darwin-x86-64 All I do is: (asdf:oos 'asdf:load-op :asdf-binary-locations) and I value this highly. I would not like to see this go away in favor of an opportunity to indulge in further configurations, especially since I am the de facto ASDF expert at my firm. I would be interested to work with you to provide some equivalent default behavior in the event of an otherwise-unconfigured ASDF Output Locations (AOL --- there's a snappy name....), and I would prefer that we not ship ASDF 2 without a similarly easy-to-use option. best, r
Having the documentation here is helpful, but a few examples would be even more helpful. For example, if I wanted to replicate Gary's A-B-L in the new configuration language, how would I do that? I don't see an opportunity in the language to grab info about the lisp implementation and compute a binary pathname. Is it possible to perform arbitrary computations in the configuration file? *features* is not enough to do what Gary does. Also, one obvious extension to the existing A-B-L would be to direct different kinds of components to write the outputs of their various operations into different places. Is that possible with this framework? E.g., write C object files to a specific location, dump parenscript preprocessing outputs (javascript files) into a directory from which they will be served by the web server, etc. A sample configuration file or two is needed before we can assess the framework. thanks, r
On 1 February 2010 10:34, Robert Goldman <rpgoldman@sift.info> wrote:
Having the documentation here is helpful, but a few examples would be even more helpful.
For example, if I wanted to replicate Gary's A-B-L in the new configuration language, how would I do that? I don't see an opportunity in the language to grab info about the lisp implementation and compute a binary pathname. Is it possible to perform arbitrary computations in the configuration file? *features* is not enough to do what Gary does.
mkdir ~/.config/common-lisp/asdf-output-locations.conf.d/ echo '(:root :user-cache)' > ~/.config/common-lisp/asdf-output-locations.conf.d/00-user-cache echo '("/my/special/path" ("/my/special/cache" :implementation "gratuitous/subdirs"))' > ~/.config/common-lisp/asdf-output-locations.conf.d/10-special Examples ought indeed to be in the final documentation. Note that with ABL, you must carefully load ABL and set whichever variable in each Lisp startup script, so the amount of work is no less.
Also, one obvious extension to the existing A-B-L would be to direct different kinds of components to write the outputs of their various operations into different places. Is that possible with this framework? I suppose you could use logical pathnames on the destination side of your translations. Configuring logical pathnames is out of the scope of AOL. But the docs should suggest the idea. logical pathnames are also a cthulhuesque piece of non-portability horror.
E.g., write C object files to a specific location, dump parenscript preprocessing outputs (javascript files) into a directory from which they will be served by the web server, etc. That's tricky. How do you currently do that with ABL?
A sample configuration file or two is needed before we can assess the framework. Certainly.
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] In a family argument, if it turns out you are right — apologize at once. — Robert Heinlein, "Time Enough For Love"
On 2/1/10 Feb 1 -9:48 AM, Faré wrote:
On 1 February 2010 10:34, Robert Goldman <rpgoldman@sift.info> wrote:
Having the documentation here is helpful, but a few examples would be even more helpful.
For example, if I wanted to replicate Gary's A-B-L in the new configuration language, how would I do that? I don't see an opportunity in the language to grab info about the lisp implementation and compute a binary pathname. Is it possible to perform arbitrary computations in the configuration file? *features* is not enough to do what Gary does.
mkdir ~/.config/common-lisp/asdf-output-locations.conf.d/ echo '(:root :user-cache)' > ~/.config/common-lisp/asdf-output-locations.conf.d/00-user-cache echo '("/my/special/path" ("/my/special/cache" :implementation "gratuitous/subdirs"))' > ~/.config/common-lisp/asdf-output-locations.conf.d/10-special
Examples ought indeed to be in the final documentation.
Note that with ABL, you must carefully load ABL and set whichever variable in each Lisp startup script, so the amount of work is no less.
I don't see this --- all I do is load it and I'm done, and it's not so carefully, either. It's just a one-liner. Your experience of A-B-L and mine seem radically different in ways I can't account for. Feel like I'm missing something here.... Contrariwise, I don't see how the above does what I am doing now -- it looks like that redirection line (/my/special/cache) will only work on one system's files, rather than pervasively as A-B-L does. Do we get the behavior of asdf-binary-locations by doing (:asdf-output-locations (:map "*" :implementation)) ? This seems forbidden by the grammar. My concern is that I don't see how to get the one-liner pervasive behavior of A-B-L yet. Once I see that, I suspect I'll try to provide backward-compatible behavior by providing a one-liner.... A couple of related questions: I'm concerned that the above notation only provides mappings for absolute components (since the start of a directory-designator must be an /absolute/ component designator. Here's why: I often find myself working with Lispers who are not sophisticated about ASDF (or even very sympathetic towards it). For working with them we typically provide a top-level checkout from the version control system which will contain a file that will, when loaded, populate their *central-registry*, using some magic applied to *load-truename*. Would that be relatively straightforward here? Should we add :this-directory to the set of absolute-component-designators? Are logical pathnames permitted in a directory-designator?
Also, one obvious extension to the existing A-B-L would be to direct different kinds of components to write the outputs of their various operations into different places. Is that possible with this framework? I suppose you could use logical pathnames on the destination side of your translations. Configuring logical pathnames is out of the scope of AOL. But the docs should suggest the idea. logical pathnames are also a cthulhuesque piece of non-portability horror.
E.g., write C object files to a specific location, dump parenscript preprocessing outputs (javascript files) into a directory from which they will be served by the web server, etc. That's tricky. How do you currently do that with ABL?
I don't --- I write my own OUTPUT-FILES method in a .asd file that's a crawling horror involving caching the value of *load-truename* into a global and doing pathname-wrangling. In the spirit of ASDF, we should be writing output-files methods and then allowing your code to shuffle them, but that use-case seems to break down a bit here. Consider providing an asdf system definition for a web site. You want to specify the relative location of the content files, which may be generated by asdf operations (e.g., my parenscript example). Those content files must be findable by the code in your system. So this seems like a case where the "let the end user decide where to put the output files" breaks down.... Hm. Needs more thought. These web content files are a case where the code/data distinction is not so clear, and ASDF has not handled data files well (indeed, by unpredictably relocating code files, it has broken our ability to find data files using *load-truename*). best, r
Note that with ABL, you must carefully load ABL and set whichever variable in each Lisp startup script, so the amount of work is no less.
I don't see this --- all I do is load it and I'm done, and it's not so carefully, either. It's just a one-liner. Your experience of A-B-L and mine seem radically different in ways I can't account for. Feel like I'm missing something here....
I suppose this redirection might be enabled in ABL by default when you're using the separate ABL, but ABL is disabled by default when using the current ASDF, for obvious reasons of backward compatibility. And so you currently need to explicitly enable the builtin ABL where you used to load an external ABL. In either case, that's a one-liner to be inserted early in your Lisp startup, between (require :asdf) and the first time asdf is used. Annoying stuff suitable only to turn newbies away. Once I replace the builtin ABL by AOL, the configuration will still be a single line, but for the whole user or machine, instead of each and every Lisp startup script. Or you can continue to do it in a startup script with a one liner, if you want. The line will have changed, but it'll be documented and if you're doing that, you're an expert anyway. Something like (asdf:ensure-output-locations '(:asdf-output-locations :enable-user-cache))
Contrariwise, I don't see how the above does what I am doing now -- it looks like that redirection line (/my/special/cache) will only work on one system's files, rather than pervasively as A-B-L does. You can redirect "/" or :root: ("/" (:home ".fasl" :implementation)) (:root ("/var/cache/common-lisp" :uid :implementation-type)) Actually, I'd like to have a keyword for :uid and/or :user, but I don't know how to portably (across OS and implementation) extract the uid or username or whatelse, and I'm not sure I want to add such machinery to asdf. But I'll accept one if contributed.
Do we get the behavior of asdf-binary-locations by doing
(:asdf-output-locations (:map "*" :implementation))
? This seems forbidden by the grammar. We can simplify the common case in the API, so I propose :enable-user-cache no paren, no nothing, be how you enable ABL-style global fasl redirection.
echo "(:asdf-output-locations :enable-user-cache)" > ~/.config/common-lisp/asdf-output-locations.conf
I'm concerned that the above notation only provides mappings for absolute components (since the start of a directory-designator must be an /absolute/ component designator. Here's why: I often find myself working with Lispers who are not sophisticated about ASDF (or even very sympathetic towards it). For working with them we typically provide a top-level checkout from the version control system which will contain a file that will, when loaded, populate their *central-registry*, using some magic applied to *load-truename*. Would that be relatively straightforward here? Should we add :this-directory to the set of absolute-component-designators? I could trivially add :current-directory -- but understand the risks if you mix and match absolute and relative. Maybe I should recognize specially :current-directory as a relative directory component, too.
I could add that to the source-registry mini-language, too.
Are logical pathnames permitted in a directory-designator? Yes, but they will be truename'd on the source side, because ASDF truenames its .asd files to locate source code. On the destination side is where you can and want to do funky mappings, anyway.
E.g., write C object files to a specific location, dump parenscript preprocessing outputs (javascript files) into a directory from which they will be served by the web server, etc. That's tricky. How do you currently do that with ABL?
I don't --- I write my own OUTPUT-FILES method in a .asd file that's a crawling horror involving caching the value of *load-truename* into a global and doing pathname-wrangling.
Gah! Well, if you have any bright idea, I'm all ears. I suppose you can use logical pathnames (cough, cough) as destination mappings for AOL, but oh my.
In the spirit of ASDF, we should be writing output-files methods and then allowing your code to shuffle them, but that use-case seems to break down a bit here.
Consider providing an asdf system definition for a web site. You want to specify the relative location of the content files, which may be generated by asdf operations (e.g., my parenscript example). Those content files must be findable by the code in your system. So this seems like a case where the "let the end user decide where to put the output files" breaks down.... Hm. Needs more thought. These web content files are a case where the code/data distinction is not so clear, and ASDF has not handled data files well (indeed, by unpredictably relocating code files, it has broken our ability to find data files using *load-truename*).
I don't have a clear solution for that problem at this point. If anyone has one, I'm curious. ASDF has no notion of "installed file". [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] I know that most men, including those at ease with problems of the greatest complexity, can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives. — Tolstoy
....
In the spirit of ASDF, we should be writing output-files methods and then allowing your code to shuffle them, but that use-case seems to break down a bit here.
Consider providing an asdf system definition for a web site. You want to specify the relative location of the content files, which may be generated by asdf operations (e.g., my parenscript example). Those content files must be findable by the code in your system. So this seems like a case where the "let the end user decide where to put the output files" breaks down.... Hm. Needs more thought. These web content files are a case where the code/data distinction is not so clear, and ASDF has not handled data files well (indeed, by unpredictably relocating code files, it has broken our ability to find data files using *load-truename*).
I don't have a clear solution for that problem at this point. If anyone has one, I'm curious.
ASDF has no notion of "installed file".
Agreed, and this can be a real nuisance if your system carries data along with it. I thought I would mention this now. If we're reforming file writing, we might want to try to pick off this problem as well. The web application is a plausible one as is, for example, a graphical app that has image files contained with it. Using *load-truename* to find such files is, as I mentioned earlier, broken by use of A-B-L or any other code that wrangles the output files of the compile-op. What I typically do is something unpleasant like this (from memory; possibly slightly inaccurate): 1. In the asdf system definition file, do something like this: (defparameter cl-user::*<sysname>-root-directory* (merge-pathnames (make-pathname :type :unspecific :name :unspecific) *load-truename*)) ;; use cl-user because you don't have access to the package(s) in the ;; as yet not loaded system... 2. In the loaded files at some point (with-open-file (str (merge-pathnames "imagefile.png" cl-user::*<sysname>-root-directory*)) ....) I'd love to see a clean way to avoid this.... Cheers, r
Robert Goldman writes:
....
In the spirit of ASDF, we should be writing output-files methods and then allowing your code to shuffle them, but that use-case seems to break down a bit here.
Consider providing an asdf system definition for a web site. You want to specify the relative location of the content files, which may be generated by asdf operations (e.g., my parenscript example). Those content files must be findable by the code in your system. So this seems like a case where the "let the end user decide where to put the output files" breaks down.... Hm. Needs more thought. These web content files are a case where the code/data distinction is not so clear, and ASDF has not handled data files well (indeed, by unpredictably relocating code files, it has broken our ability to find data files using *load-truename*).
I don't have a clear solution for that problem at this point. If anyone has one, I'm curious.
ASDF has no notion of "installed file".
Agreed, and this can be a real nuisance if your system carries data along with it.
I thought I would mention this now. If we're reforming file writing, we might want to try to pick off this problem as well.
The web application is a plausible one as is, for example, a graphical app that has image files contained with it.
Using *load-truename* to find such files is, as I mentioned earlier, broken by use of A-B-L or any other code that wrangles the output files of the compile-op.
What I typically do is something unpleasant like this (from memory; possibly slightly inaccurate):
1. In the asdf system definition file, do something like this:
(defparameter cl-user::*<sysname>-root-directory* (merge-pathnames (make-pathname :type :unspecific :name :unspecific) *load-truename*)) ;; use cl-user because you don't have access to the package(s) in the ;; as yet not loaded system...
2. In the loaded files at some point
(with-open-file (str (merge-pathnames "imagefile.png" cl-user::*<sysname>-root-directory*)) ....)
I'd love to see a clean way to avoid this....
I do not understand the issue at hand -- just want provide a comment on the side: Instead of *LOAD-TRUENAME* in the .asd file, you could probably get away with using *COMPILE-FILE-PATHNAME*, say, in the your package.lisp? Which would not be wrangled by ASDF-Binary-Locations and consorts. (It also solves the package problem.) -T.
On 2/2/10 Feb 2 -2:39 AM, Tobias C. Rittweiler wrote:
Robert Goldman writes:
...
Instead of *LOAD-TRUENAME* in the .asd file, you could probably get away with using *COMPILE-FILE-PATHNAME*, say, in the your package.lisp? Which would not be wrangled by ASDF-Binary-Locations and consorts. (It also solves the package problem.)
Thank you very much, tcr, this seems like a simple and elegant solution to the problem. Much simpler than messing with ASDF proper! I hadn't thought hard about how this would play with output-file wrangling; now I see that it will be unaffected. Good! One problem down! R
Once I replace the builtin ABL by AOL, the configuration will still be a single line, but for the whole user or machine, instead of each and every Lisp startup script. Or you can continue to do it in a startup script with a one liner, if you want. The line will have changed, but it'll be documented and if you're doing that, you're an expert anyway. Something like (asdf:ensure-output-locations '(:asdf-output-locations :enable-user-cache))
how about making that be the default? it's transparent enough and i don't think anyone will mind not seeing fasls thrashing all their source directories as the default behavior...
Consider providing an asdf system definition for a web site. You want to specify the relative location of the content files, which may be generated by asdf operations (e.g., my parenscript example). Those content files must be findable by the code in your system. So this seems like a case where the "let the end user decide where to put the output files" breaks down.... Hm. Needs more thought. These web content files are a case where the code/data distinction is not so clear, and ASDF has not handled data files well (indeed, by unpredictably relocating code files, it has broken our ability to find data files using *load-truename*).
I don't have a clear solution for that problem at this point. If anyone has one, I'm curious.
imho that's stretching the limits of asdf and output directories... in hu.dwim.wui we have a special web broker that caches the js compilation results and manages recompilations, last-modified http header. also, some of our js scripts are fixed to the preferred locale order of the request (at js compile time), etc... if you really want to do that using asdf, then i'd suggest adding a custom my-parenscript-file component class and override the generic method to customize where the output goes (assuming there's such a generic). and maybe renaming that machinery to asdf-temporary-output to avoid confusion. but here i might be the one stretching the limits... :) -- attila
My plan for ASDF 1.600 is to rename the (recently) builtin asdf-binary-locations to asdf-output-locations and make a few incompatible changes to it.
Please don't. I don't have time for a long explanation; but A-B-L is a completely accurate description of 90% of what it does, and ASDF has certain options completely backwards. (I think it inherited these from the single-user/single-address-space lisp machine mentality.) Rather than institutionalize these, I think there are straightforward ways to start turning towards a better path. Please ping me if I don't get back to this in a couple days (I had a few comments to make re 1.5.0 that have also languished). Or maybe we could meet somewhere to hash out some better ideas. - Daniel
participants (5)
-
Attila Lendvai -
dherring＠tentpost.com -
Faré -
Robert Goldman -
Tobias C. Rittweiler