[asdf-devel] summary proposal for COMMON_LISP_PATH and system search changes
Here is a summary of what I've heard and what I think we should do. # Proposal Summary These are independent but make the most sense if taken as a whole. * Specify new defaults for `*central-registry*`, * use an environment variable named `COMMON_LISP_PATH` to override ASDF's defaults, * use `*central-registry*` to find and load an ASDF configuration file (if present) and * modify the default behavior of `sysdef-central-registry-search` to search deeply. ## default value for *central-registry* COMMON_LISP_PATH environment variable with syntax like path[;path]* This is used by ASDF to help build the default value for `*central-registry*`. If not specified, `*central-registry*` defaults to (on *nix-like systems): (list *default-pathname-defaults* (merge-pathnames (make-pathname :directory '(:relative ".common-lisp")) (user-homedir-pathname))) (merge-pathnames (make-pathname :directory '(:relative ".local" "share" "common-lisp")) (user-homedir-pathname))) (make-pathname :directory '(:absolute "usr" "local" "share" "common-lisp")) (make-pathname :directory '(:absolute "usr" "share" "common- lisp")) ) (The last two would need to be different for Windows-based systems.) ## modification for `sysdef-central-registry-search` Currently `sysdef-central-registry-search` searches in the top-level of each directory in `*central-registry*` to find systems. If we use the above, then a more useful approach would be to search in the top-level and in a sub-directory named `systems`. An even more useful approach would to search in all sub-directories of each entry. ## configuration file When ASDF starts, it will: 1. setup `*central-registry*` using its own defaults and the `COMMON_LISP_PATH` environment variable. 2. look for (and load if it is found) a file named `asdf-config.lisp` in each of these paths traversed in _reverse_ order (so that user settings take precedence overs system ones). Note that the ASDF package will exist when this file is read. I.e., I'm suggesting that we go the "user settings overwrite system settings" route rather than the "if there are any user settings, then ignore the system settings" one. -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
On 2009-09-27, at 15:16 , Gary King wrote:
Here is a summary of what I've heard and what I think we should do.
# Proposal Summary
[...]
## default value for *central-registry*
COMMON_LISP_PATH environment variable with syntax like
path[;path]*
This is used by ASDF to help build the default value for `*central-registry*`.
If not specified, `*central-registry*` defaults to (on *nix-like systems):
(list *default-pathname-defaults* (merge-pathnames (make-pathname :directory '(:relative ".common-lisp")) (user-homedir-pathname))) (merge-pathnames (make-pathname :directory '(:relative ".local" "share" "common-lisp")) (user-homedir-pathname))) (make-pathname :directory '(:absolute "usr" "local" "share" "common-lisp")) (make-pathname :directory '(:absolute "usr" "share" "common- lisp")) )
please have the initialization process filter this to remove anything for which probe-file fails.
(The last two would need to be different for Windows-based systems.)
## modification for `sysdef-central-registry-search`
Currently `sysdef-central-registry-search` searches in the top-level of each directory in `*central-registry*` to find systems. If we use the above, then a more useful approach would be to search in the top-level and in a sub-directory named `systems`. An even more useful approach would to search in all sub-directories of each entry.
if you intend to take the trouble to do this, please consider a convention which treats system names as path components. if they are taken to denote the tail end of the path (ie. as jave package names), it is possible to unambiguously store and locate the entire lisp world in a single source tree. yes, i realize, this is not the way most people use asdf now.
## configuration file
this one confuses me. which process is it intended to effect, which cannot be effected by the contents of the runtime's respective initialization file and the respective .asd file(s)?
When ASDF starts, it will:
1. setup `*central-registry*` using its own defaults and the `COMMON_LISP_PATH` environment variable.
2. look for (and load if it is found) a file named `asdf-config.lisp` in each of these paths traversed in _reverse_ order (so that user settings take precedence overs system ones). Note that the ASDF package will exist when this file is read.
? what does this load order accomplish which the load order of the runtime init and the respective .asd files does not? [...]
On Sep 27, 2009, at 3:16 PM, Gary King wrote:
COMMON_LISP_PATH environment variable with syntax like
path[;path]*
It is custommary in POSIX systems to use ':' as separtor in PATH variables, since ';' has a special meaning for POSIX shells, and therefore would require escape. It should also be specified that if a path doesn't end with a '/', then it will be automatically added, that is, these are considered to be directory pathnames. It should also be specified what happens if there's an empty path. We can easily get an empty path, notably at the end, with careless shell programming (it must be said it's rather awkward to write it right in shell). I would specify that empty paths are ignored. Since it is an environment variable designed to set the ASDF central registry paths, why not call it ASDF_CENTRAL_REGISTRY_PATH ? Or ASDF_SYSTEMS_PATH ? Cannot ASDF be used to compile C programs? -- __Pascal Bourguignon__ http://www.informatimago.com
"Pascal J. Bourguignon" writes:
Since it is an environment variable designed to set the ASDF central registry paths, why not call it ASDF_CENTRAL_REGISTRY_PATH ? Or ASDF_SYSTEMS_PATH ? Cannot ASDF be used to compile C programs?
It has been suggested that such an environment variable should be shared among system definition facilities (du jour, that's asdf and xcvb) and should hence be named generally. -T.
Tobias C. Rittweiler wrote:
"Pascal J. Bourguignon" writes:
Since it is an environment variable designed to set the ASDF central registry paths, why not call it ASDF_CENTRAL_REGISTRY_PATH ? Or ASDF_SYSTEMS_PATH ? Cannot ASDF be used to compile C programs?
It has been suggested that such an environment variable should be shared among system definition facilities (du jour, that's asdf and xcvb) and should hence be named generally.
How about LISP_SYSTEMS_PATH then? COMMON_LISP_PATH seems to me to suggest a path to the lisp environment.... Cheers, r
2009/9/27 Robert Goldman <rpgoldman@sift.info>:
Tobias C. Rittweiler wrote:
"Pascal J. Bourguignon" writes:
Since it is an environment variable designed to set the ASDF central registry paths, why not call it ASDF_CENTRAL_REGISTRY_PATH ? Or ASDF_SYSTEMS_PATH ? Cannot ASDF be used to compile C programs?
It has been suggested that such an environment variable should be shared among system definition facilities (du jour, that's asdf and xcvb) and should hence be named generally.
How about LISP_SYSTEMS_PATH then? COMMON_LISP_PATH seems to me to suggest a path to the lisp environment....
XCVB doesn't have systems, it has builds - though I suppose they could be renamed under community pressure. Nevertheless, Python has PYTHONPATH, it doesn't confuse anyone, and non-pythonistas don't have to know if Python has systems, builds, modules, libraries, widgets or foobars. PATH as the postfix for colon-separated (under Unix, semi-colon under DOS/Windows) list of directories where to look for stuff is a well-established tradition. PATH, LD_LIBRARY_PATH, MANPATH, PYTHONPATH, CDPATH FPATH MAILPATH MODULE_PATH (for zsh), etc. I think the questions are whether to 1- (syntax) make it always colon-separated, or have it semi-colon separated in non-cygwin Windows. 2- (semantics) whether and how to specify recursion for search in the PATH. The important point to discuss is 2 - especially if the path is to be shared between ASDF and XCVB, since (a) ASDF doesn't currently recurse on entries of its *central-registry*, whereas (b) XCVB does recurse on entries of its *xcvb-path*, and eagerly searches for entries, avoiding VCS caches (old svn and darcs) and detecting and discarding any conflicts (and this recursing is essential to the design of XCVB). You can see how XCVB does it there: http://common-lisp.net/gitweb?p=projects/xcvb/xcvb.git;a=blob;f=search-path.... Of course, (i) ASDF has a requirement of backwards compatibility, and (ii) recursing in a way both efficient and portable might be tricky, (iii) ASDF can't afford to pull external libraries for recursing, (iv) in practice, recursing one level of directories might be enough for ASDF (or not). And so the choice is not as easy as might seem. As for other suggestions, James Anderson was probably right to request that bad paths included in the PATH should be discarded from the registry, though (IGNORE-ERRORS (TRUENAME x)) rather than (PROBE-FILE x) since a directory is not a file and CL will PROBE-FILE will bork. [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] Big Business has to *sell* the Vietnam War to you. Big Government can just draft your ass and ship you overseas under threat of imprisonment and/or death. Now, which one is better again? -- rho@netdoor.com
Faré wrote:
2009/9/27 Robert Goldman <rpgoldman@sift.info>:
Tobias C. Rittweiler wrote:
"Pascal J. Bourguignon" writes:
[...snip...] I think the questions are whether to [...snip...] 2- (semantics) whether and how to specify recursion for search in the PATH.
The important point to discuss is 2 - especially if the path is to be shared between ASDF and XCVB, since (a) ASDF doesn't currently recurse on entries of its *central-registry*, whereas (b) XCVB does recurse on entries of its *xcvb-path*, and eagerly searches for entries, avoiding VCS caches (old svn and darcs) and detecting and discarding any conflicts (and this recursing is essential to the design of XCVB).
You can see how XCVB does it there: http://common-lisp.net/gitweb?p=projects/xcvb/xcvb.git;a=blob;f=search-path....
Of course, (i) ASDF has a requirement of backwards compatibility, and (ii) recursing in a way both efficient and portable might be tricky,
I'm reluctant to add default recursion for reasons of efficiency. I work on a number of projects that contain both Java and CL code. Auto-recursion fits this case particularly poorly, because the java compiler and the namespacing conventions cause java code to excrete directories at a ferocious pace. Tree-searching one of these projects is very painful. Now, one could provide the ability to turn this behavior off, of course, but that raises the bar to entry. I'd prefer this recursion be turned OFF by default, and enabled by people who know they want it. Possible happy compromise: allow a convention like /path/to/recursive/directory//:/another/path to specify when you want recursion in the variable? That works for the shell variable, which is a string; not sure how to mark the corresponding CL pathname object. best, r
On 2009-09-29, at 16:09 , Robert Goldman wrote:
[...]
Possible happy compromise: allow a convention like
/path/to/recursive/directory//:/another/path
to specify when you want recursion in the variable?
That works for the shell variable, which is a string; not sure how to mark the corresponding CL pathname object.
how about "**". as in, /path/to/recursive/directory/**/:/another/path/
james anderson wrote:
On 2009-09-29, at 16:09 , Robert Goldman wrote:
[...]
Possible happy compromise: allow a convention like
/path/to/recursive/directory//:/another/path
to specify when you want recursion in the variable?
That works for the shell variable, which is a string; not sure how to mark the corresponding CL pathname object.
how about "**". as in,
/path/to/recursive/directory/**/:/another/path/
WFM. Is there any chance that this choice would cause awkwardness with the shell's parsing? r
On 2009-09-29, at 17:20 , Robert Goldman wrote:
james anderson wrote:
On 2009-09-29, at 16:09 , Robert Goldman wrote:
[...]
Possible happy compromise: allow a convention like
/path/to/recursive/directory//:/another/path
to specify when you want recursion in the variable?
That works for the shell variable, which is a string; not sure how to mark the corresponding CL pathname object.
how about "**". as in,
/path/to/recursive/directory/**/:/another/path/
WFM. Is there any chance that this choice would cause awkwardness with the shell's parsing?
there is always /path/to/recursive/directory/:wild-inferiors/:/another-path which evil is less?
On Sep 29, 2009, at 5:20 PM, Robert Goldman wrote:
james anderson wrote:
On 2009-09-29, at 16:09 , Robert Goldman wrote:
[...]
Possible happy compromise: allow a convention like
/path/to/recursive/directory//:/another/path
to specify when you want recursion in the variable?
That works for the shell variable, which is a string; not sure how to mark the corresponding CL pathname object.
how about "**". as in,
/path/to/recursive/directory/**/:/another/path/
WFM. Is there any chance that this choice would cause awkwardness with the shell's parsing?
Well of course: you will have to quote the path or escape the stars. But substituting the star with another character is even less tasty. Allow "*" for all the subdirectories (not recursive) too. -- __Pascal Bourguignon__ http://www.informatimago.com
Take 2. Thanks for all the comments and ideas; and is a new summary. # Proposal Summary These are independent but make the most sense if taken as a whole. * Specify new defaults for `*central-registry*`, * use an environment variable named `CL_SYSTEMS_PATH` to override ASDF's defaults, * use `*central-registry*` to find and load an ASDF configuration file (if present) and * modify the default behavior of `sysdef-central-registry-search` to search deeply. ## default value for *central-registry* CL_SYSTEMS_PATH environment variable with syntax like path[:path]* (on *nix) or path[;path]* (on Windows) I.e., path designators separated by an OS dependent marker (colons or semi-colons). This is used by ASDF to help build the default value for `*central-registry*`. Each `path` will be interpreted as follows: * a `!` will be replaced by the default value (see below) so that it is easy to extend the path rather than just replacing it. * empty ones will be removed from the list, * since these represent _directories_, trailing slashes will be added if necessary, * if the last component is a `*` or `**` then sub-directories of this path will be searched recursively either one level deep (if `*`) or all the way down (if `**`). If not specified, `*central-registry*` defaults to (on *nix-like systems): (list *default-pathname-defaults* (merge-pathnames (make-pathname :directory '(:relative ".common-lisp")) (user-homedir-pathname))) (merge-pathnames (make-pathname :directory '(:relative ".local" "share" "common-lisp")) (user-homedir-pathname))) (make-pathname :directory '(:absolute "usr" "local" "share" "common-lisp")) (make-pathname :directory '(:absolute "usr" "share" "common- lisp")) ) (On a windows system, the last two would be replaced by c:\common-lisp and c:\windows\common-lisp). Note that this list would be automatically filtered for any entries for which the moral equivalent of `(ignore-errors (truename x))` fails. ## modification for `sysdef-central-registry-search` Currently `sysdef-central-registry-search` searches in the top-level of each directory in `*central-registry*` to find systems. If we use the above, then a more useful approach would be to search in the top-level and in a sub-directory named `systems`. An even more useful approach would to search in all sub-directories of each entry. ## configuration file When ASDF starts, it will: 1. setup `*central-registry*` using its own defaults and the `CL_SYSTEMS_PATH` environment variable. 2. look for and load the *first* file named `asdf-config.lisp` in each of these paths traversed in order (so that user settings take precedence over system ones). Note that the ASDF package will exist when this file is read. I.e., I'm suggesting "if there are any user settings, then ignore the system settings" one rather than the "user settings overwrite system settings".
On Oct 1, 2009, at 3:55 PM, Gary King wrote:
Take 2.
Thanks for all the comments and ideas; and is a new summary.
# Proposal Summary
* a `!` will be replaced by the default value (see below) so that it is easy to extend the path rather than just replacing it.
I don't like too much using random characters like this. On the other hand, I'm not sure I've got a good proposition here. I've seen programs using the ${VAR} or $(VAR) syntax. So instead of matching "!", we'd match "${DEFAULT}". We'd write: export CL_SYSTEM_PATH="${HOME}"'/lisp/*/:/data/lisp/**/system/:$ {DEFAULT}' or: export CL_SYSTEM_PATH="${HOME}/lisp/\*/:/data/lisp/\*\*/system/:\$ {DEFAULT}" which is nice when there is several such variable the program provides, but is also confusing the user with environment variables vs. the program variables.
* if the last component is a `*` or `**` then sub-directories of this path will be searched recursively either one level deep (if `*`) or all the way down (if `**`).
I see no reason to restrict it to the last component. On the contrary, we may assume that each system is delivered with a specific directory structure, and that a system X could have its various asdf, defsystem, xcvb files in a specific subdirectory: X/system/X.asd in which case we would set: CL_SYSTEM_PATH=/src/lisp/\*/system/:\${DEFAULT} -- __Pascal Bourguignon__ http://www.informatimago.com
Gary King writes:
## default value for *central-registry*
...
* a `!` will be replaced by the default value (see below) so that it is easy to extend the path rather than just replacing it.
Is this really necessary?
## configuration file
When ASDF starts, it will:
1. setup `*central-registry*` using its own defaults and the `CL_SYSTEMS_PATH` environment variable.
2. look for and load the *first* file named `asdf-config.lisp` in each of these paths traversed in order (so that user settings take precedence over system ones). Note that the ASDF package will exist when this file is read.
"each of these paths" -- which paths? The ones in *CENTRAL-REGISTRY*? What's the use case for that? -T.
Gary King writes:
## default value for *central-registry*
COMMON_LISP_PATH environment variable with syntax like
path[;path]*
Shouldn't this be path[:path]* on Unix systems, by analogy to PATH, MANPATH, LD_LIBRARY_PATH, et al.? -- Richard
Gary King wrote:
Here is a summary of what I've heard and what I think we should do.
# Proposal Summary
These are independent but make the most sense if taken as a whole.
* Specify new defaults for `*central-registry*`,
* use an environment variable named `COMMON_LISP_PATH` to override ASDF's defaults,
Here's a use case for which I don't think this is good --- consider a person who uses more than a single Common Lisp implementation (e.g., ACL + SBCL + CCL). Is it problematic to have a single COMMON_LISP_PATH environment variable? Maybe it's not, and we should just say "if you're going to do something this complicated, you'll have to do some engineering of your init files that's more complicated than setting a variable /anyway/, so you're not the sort of person for whom this was intended." Thoughts?
2009/9/27 Robert Goldman <rpgoldman@sift.info>:
Here's a use case for which I don't think this is good --- consider a person who uses more than a single Common Lisp implementation (e.g., ACL + SBCL + CCL). Is it problematic to have a single COMMON_LISP_PATH environment variable?
Actually, we have at ITA some big application that can be compiled by at least two Lisp implementations and would live quite happily with the same COMMON_LISP_PATH, thanks to conditional compilation. There might indeed (or then again might not) be a need to keep any implementation-specific directories in the central registry directories. For instance, the current XCVB_PATH thingie allows you to specify "!" for "the previous value of the path", that you can insert anywhere in your PATH, e.g. /home/fare/cl:!:/home/fare/.local/share/common-lisp [ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ] A tautology is a thing which is tautological.
Robert Goldman writes:
Gary King wrote:
Here is a summary of what I've heard and what I think we should do.
# Proposal Summary
These are independent but make the most sense if taken as a whole.
* Specify new defaults for `*central-registry*`,
* use an environment variable named `COMMON_LISP_PATH` to override ASDF's defaults,
Here's a use case for which I don't think this is good --- consider a person who uses more than a single Common Lisp implementation (e.g., ACL + SBCL + CCL). Is it problematic to have a single COMMON_LISP_PATH environment variable?
Maybe it's not, and we should just say "if you're going to do something this complicated, you'll have to do some engineering of your init files that's more complicated than setting a variable /anyway/, so you're not the sort of person for whom this was intended."
If you read my original posting, I was looking for a shared registry place exactly for the case of using several implementations at once, and not having to bother with all their idiosyncratic init files. -T.
On Sun, 27 Sep 2009, Gary King wrote:
Here is a summary of what I've heard and what I think we should do.
Looks like a good direction. These changes may justify bumping the ASDF version number to 2.0. My comments follow.
## default value for *central-registry*
COMMON_LISP_PATH environment variable with syntax like path[;path]*
How about CL_PATH or CL_SYSTEM_PATH? As others noted, the value should have an OS-dependent syntax. POSIX: path[:path]* Microsoft: path[;path]* Both syntaxes should ignore an empty path (e.g. "path1:path2:") and should not require paths to have a trailing slash (a common source of errors).
This is used by ASDF to help build the default value for `*central-registry*`.
If not specified, `*central-registry*` defaults to (on *nix-like systems): ... These look fine.
(The last two would need to be different for Windows-based systems.)
Something like c:\common-lisp and c:\windows\common-lisp ? It would be nice if the defaults could be invoked by shell users. Then they could write CL_SYSTEM_PATH=DEFAULT_CL_SYSTEM_PATH:/opt/common-lisp.
## modification for `sysdef-central-registry-search`
Currently `sysdef-central-registry-search` searches in the top-level of each directory in `*central-registry*` to find systems. If we use the above, then a more useful approach would be to search in the top-level and in a sub-directory named `systems`. An even more useful approach would to search in all sub-directories of each entry.
Searching an extra "systems" subdirectory seems redundant; to follow common-lisp-controller, make it an explicit part of DEFAULT_CL_SYSTEM_PATH. Searching subdirectories for arbitrary *.asd files seems too loose. If ASDF chooses to search subdirectories, it might be good to follow the pattern set by make, Java, and Apple; search for a single file in each subdirectory (e.g. the makefile, the jar manifest, the .app Info.plist, an ASDF system.asd). I like James's idea. Let CL_SYSTEM_PATH=path1:path2. Then (asdf:oos 'asdf:load-op :babel) will use the first of path1/babel/system.asd and path2/babel/system.asd. Nothing else to search. No loading asd files until the proper one is found. For backwards compatibility, ASDF can continue looking for $PATH/*.asd without checking subdirectories.
## configuration file
When ASDF starts, it will:
1. setup `*central-registry*` using its own defaults and the `COMMON_LISP_PATH` environment variable.
2. look for (and load if it is found) a file named `asdf-config.lisp` in each of these paths traversed in _reverse_ order (so that user settings take precedence overs system ones). Note that the ASDF package will exist when this file is read.
I.e., I'm suggesting that we go the "user settings overwrite system settings" route rather than the "if there are any user settings, then ignore the system settings" one.
That works, but I think shells use the first available for a reason: it is easier to explicitly include a system file than it is to undo unwanted changes. Also, such config files generally don't belong in PATH; how about ~/.asdf.lisp and /etc/asdf.lisp. This file will be used to configure stuff like asdf-binary-locations, correct? IMO, the config file should be loaded before the CL_SYSTEM_PATH variable is used. They could usefully modify DEFAULT_CL_SYSTEM_PATH; but they should never modify the actual CL_SYSTEM_PATH (e.g. a shell script may need custom settings). Later, Daniel
This file will be used to configure stuff like asdf-binary-locations, correct? IMO, the config file should be loaded before the CL_SYSTEM_PATH variable is used. They could usefully modify DEFAULT_CL_SYSTEM_PATH; but they should never modify the actual CL_SYSTEM_PATH (e.g. a shell script may need custom settings).
Can you provide some examples of what you see folks doing with this? thanks, -- Gary Warren King, metabang.com Cell: (413) 559 8738 Fax: (206) 338-4052 gwkkwg on Skype * garethsan on AIM * gwking on twitter
On Thu, 1 Oct 2009, Gary King wrote:
This file will be used to configure stuff like asdf-binary-locations, correct? IMO, the config file should be loaded before the CL_SYSTEM_PATH variable is used. They could usefully modify DEFAULT_CL_SYSTEM_PATH; but they should never modify the actual CL_SYSTEM_PATH (e.g. a shell script may need custom settings).
Can you provide some examples of what you see folks doing with this?
This would be a central spot to customize ASDF for all lisp implementations on a system. It would have stuff like. (setf asdf:*centralize-lisp-binaries* t asdf:*default-toplevel-directory* #P"/fasls" asdf:*compile-file-failure-behaviour* :warn asdf:*default-central-registry* (list p1 p2 p3) ...) Again, it should be searched in the usual configuration paths (e.g. ~/.asdf-config.lisp, /etc/asdf-config.lisp), not in CL_SYSTEM_PATH. A CL_SYSTEM_ENV variable could replace CL_SYSTEM_PATH by pointing directly to this configuration file... Later, Daniel P.S. If these are to be used by non-asdf systems, then it may be appropriate to move such symbols outside the asdf package. Similarly, there would be a need to specify initialization semantics so the different systems set them cooperatively. Fortunately, these issues can be addressed at a later date when such systems exist.
participants (8)
-
Daniel Herring -
Faré -
Gary King -
james anderson -
Pascal J. Bourguignon -
Richard M Kreuter -
Robert Goldman -
Tobias C. Rittweiler