Hello all,
Moving this question from https://gitlab.common-lisp.net/asdf/asdf/-/issues/52
To reiterate, I believe that the instructions in https://common-lisp.net/project/asdf/asdf/Upgrading-ASDF.html 'Upgrading ASDF' could use some clarification.
I am attempting to run a newer version of ASDF than that provided by my impl. The text I'm having issue with is:
If your implementation already provides ASDF 3 or later (and it should),
but you want a more recent ASDF version than your implementation provides, then you just need to ensure the more recent ASDF is installed in a configured path, like any other system. We recommend you download an official tarball or checkout a release from git into ~/common-lisp/asdf/.
Once the source code for ASDF is installed, you don’t need any extra step
to load it beyond the usual (require "asdf"): ASDF 3 will automatically look whether an updated version of itself is available amongst the regularly configured systems, before it compiles anything else.
First, it says I should download an official tarball. I looked to the best of my ability and could not find one. Even if I had found one, I don't know what I'd do with it. So I moved to the second option and cloned ASDF to `~/common-lisp/asdf/` and built it successfully.
I verify that indeed I am running a newer ASDF. However, I noticed `~/comon-lisp/asdf/ext/` includes certain third-party libraries bundled in the repo. These are being prioritized over Quicklisp libraries, and it's causing conflicts.
So my question is:
May I get clarification on how I may build & install ASDF such that the implementation-bundled one will automatically find the newer version, while omitting the libraries ext/ directory from being picked up?
Thanks all!
On 18 Jan 2021, at 17:37, Wilfredo Velazquez wrote:
Hello all,
Moving this question from https://gitlab.common-lisp.net/asdf/asdf/-/issues/52
To reiterate, I believe that the instructions in https://common-lisp.net/project/asdf/asdf/Upgrading-ASDF.html 'Upgrading ASDF' could use some clarification.
I am attempting to run a newer version of ASDF than that provided by my impl. The text I'm having issue with is:
If your implementation already provides ASDF 3 or later (and it should), but you want a more recent ASDF version than your implementation provides, then you just need to ensure the more recent ASDF is installed in a configured path, like any other system. We recommend you download an official tarball or checkout a release from git into ~/common-lisp/asdf/. Once the source code for ASDF is installed, you don’t need any extra step to load it beyond the usual (require "asdf"): ASDF 3 will automatically look whether an updated version of itself is available amongst the regularly configured systems, before it compiles anything else.
First, it says I should download an official tarball. I looked to the best of my ability and could not find one. Even if I had found one, I don't know what I'd do with it.
There should be tarballs here:
https://common-lisp.net/project/asdf/
Look for the "Getting it" section.
So I moved to the second option and cloned ASDF to `~/common-lisp/asdf/` and built it successfully.
I verify that indeed I am running a newer ASDF. However, I noticed `~/comon-lisp/asdf/ext/` includes certain third-party libraries bundled in the repo. These are being prioritized over Quicklisp libraries, and it's causing conflicts.
So my question is:
May I get clarification on how I may build & install ASDF such that the implementation-bundled one will automatically find the newer version, while omitting the libraries ext/ directory from being picked up?
As far as I can tell, if you put a directory tree into `~/common-lisp/`, ASDF will traverse it looking for system definitions, and there is nothing you can do to override this. I have come to dislike the `~/common-lisp/` solution for this reason.
There are other ways you can make the new directory visible to ASDF *without* exposing `ext/`:
* The libraries in `ext/` are there so that ASDF can be downloaded in isolation and tested without any external dependencies. If you don't pull the git submodules for the ASDF repository, they won't be there, and you won't have the problem you are having. You could do "git get rid of my submodules" somehow, but this is the ASDF list, not the git help list -- you'll have to figure that out on your own!
* You can make the directory find-able using Faré's configuration language either directly in your lisp init file, or by using the environment variables. I'm not a fan of environment variables, because they add more complexity to my lisp configuration. I like to keep all of the configuration in my lisp init file, so I don't have to worry about *where* a setting is made. If you add `(:directory <my asdf directory>)` to the configuration, then ASDF will not search it recursively (the alternative is the `:tree` keyword, which tells ASDF you *do* want it to search recursively). There's a way of making this same distinction in the environment variables: it's explained in the manual.
* If you are old school and don't want to fuss with debugging the configuration DSL, you can just do: `(push <my asdf directory> asdf:*central-registry*)` -- again, ASDF will not search this directory recursively.
I hope that will solve your problem!
On 18 Jan 2021, at 17:37, Wilfredo Velazquez wrote:
To reiterate, I believe that the instructions in https://common-lisp.net/project/asdf/asdf/Upgrading-ASDF.html 'Upgrading ASDF' could use some clarification.
I am attempting to run a newer version of ASDF than that provided by my impl. The text I'm having issue with is:
If your implementation already provides ASDF 3 or later (and it should), but you want a more recent ASDF version than your implementation provides, then you just need to ensure the more recent ASDF is installed in a configured path, like any other system. We recommend you download an official tarball or checkout a release from git into ~/common-lisp/asdf/. Once the source code for ASDF is installed, you don’t need any extra step to load it beyond the usual (require "asdf"): ASDF 3 will automatically look whether an updated version of itself is available amongst the regularly configured systems, before it compiles anything else.
Personally I've never really been a fan of installing ASDF this way since a) it requires loading ASDF twice: the implementation built-in and then the upgrade and b) it requires that ~/common-lisp/ be on ASDF's search path (I mess around with CL_SOURCE_REGISTRY envvar quite a bit and frequently end up excluding ~/common-lisp/). I tend to replace the implementation's built-in version https://common-lisp.net/project/asdf/asdf.html#Replacing-your-implementation_0027s-ASDF. But do note, I have seen this method occasionally break and I fall back to the ~/common-lisp/asdf/ apprach when it does. But I'm pretty sure it works in the latest release on at least SBCL, ECL, and CCL.
As far as I can tell, if you put a directory tree into `~/common-lisp/`, ASDF will traverse it looking for system definitions, and there is nothing you can do to override this. I have come to dislike the `~/common-lisp/` solution for this reason.
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
-Eric
**Warning -- opinionated rant follows!**
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
`asdf:*central-registry*` is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, `*central-registry*` is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my `.emacs` is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that `ls` won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
Can I upgrade ASDF by simply placing the built asdf.lisp file in ~/common-lisp/asdf/asdf.lisp and omitting everything else? Will ASDF find it?
On Tue, Jan 19, 2021 at 9:44 AM Robert Goldman rpgoldman@sift.info wrote:
*Warning -- opinionated rant follows!*
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
asdf:*central-registry* is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, *central-registry* is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my .emacs is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that ls won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
On 19 Jan 2021, at 9:05, Wilfredo Velazquez wrote:
Can I upgrade ASDF by simply placing the built asdf.lisp file in ~/common-lisp/asdf/asdf.lisp and omitting everything else? Will ASDF find it?
I don't *believe* so -- the ASDF search mechanisms are all aimed at finding files with the `.asd` extension. I'd have to look at the code (I'm not able to do this right now), but there would have to be a special case for files named "asdf.lisp". But I could be wrong.
On Tue, Jan 19, 2021 at 9:44 AM Robert Goldman rpgoldman@sift.info wrote:
*Warning -- opinionated rant follows!*
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
asdf:*central-registry* is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, *central-registry* is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my .emacs is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that ls won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
-- Wilfredo Velázquez-Rodríguez
In upgrade.lisp we see:
``` (defun upgrade-asdf () "Try to upgrade of ASDF. If a different version was used, return T. We need do that before we operate on anything that may possibly depend on ASDF." (let ((*load-print* nil) (*compile-print* nil)) (handler-bind (((or style-warning) #'muffle-warning)) (symbol-call :asdf :load-system :asdf :verbose nil)))) ```
This suggests to me that upgrading will require finding "asdf.asd". So I would have thought that this would rely on finding the source in `asdf/` -- where there's "asdf.asd" -- rather than the merged single file in "asdf.lisp".
On 19 Jan 2021, at 9:05, Wilfredo Velazquez wrote:
Can I upgrade ASDF by simply placing the built asdf.lisp file in ~/common-lisp/asdf/asdf.lisp and omitting everything else? Will ASDF find it?
On Tue, Jan 19, 2021 at 9:44 AM Robert Goldman rpgoldman@sift.info wrote:
*Warning -- opinionated rant follows!*
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
asdf:*central-registry* is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, *central-registry* is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my .emacs is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that ls won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
-- Wilfredo Velázquez-Rodríguez
Ahh I see. Yes, that's the specific line I was looking for. Thank you. Though I don't fully understand how it goes about checking a newer version, this at least gives me the information I need.
Thanks!
On Tue, Jan 19, 2021 at 11:07 AM Robert Goldman rpgoldman@sift.info wrote:
In upgrade.lisp we see:
(defun upgrade-asdf () "Try to upgrade of ASDF. If a different version was used, return T. We need do that before we operate on anything that may possibly depend on ASDF." (let ((*load-print* nil) (*compile-print* nil)) (handler-bind (((or style-warning) #'muffle-warning)) (symbol-call :asdf :load-system :asdf :verbose nil))))
This suggests to me that upgrading will require finding "asdf.asd". So I would have thought that this would rely on finding the source in asdf/ -- where there's "asdf.asd" -- rather than the merged single file in "asdf.lisp".
On 19 Jan 2021, at 9:05, Wilfredo Velazquez wrote:
Can I upgrade ASDF by simply placing the built asdf.lisp file in ~/common-lisp/asdf/asdf.lisp and omitting everything else? Will ASDF find it?
On Tue, Jan 19, 2021 at 9:44 AM Robert Goldman rpgoldman@sift.info wrote:
*Warning -- opinionated rant follows!*
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
asdf:*central-registry* is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, *central-registry* is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my .emacs is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that ls won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
-- Wilfredo Velázquez-Rodríguez
Standing ovation.
On Tue, Jan 19, 2021 at 3:45 PM Robert Goldman rpgoldman@sift.info wrote:
*Warning -- opinionated rant follows!*
On 18 Jan 2021, at 20:50, Eric Timmons wrote:
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results https://urldefense.proofpoint.com/v2/url?u=https-3A__common-2Dlisp.net_project_asdf_asdf.html-23Caching-2DResults&d=DwMFAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=neRclckOj-HiRNxvlVT0Yw&m=AM8JvYqHdPyu4xUFYUNXVbteBkq7DApCwGFFWclUgHE&s=q1CpKZhFB4J6ZnLXKHTiA4nRWs-R3ABoU_8wTHpINXs&e=. ASDF will stop recursing if it finds that file and just use the info it contains.
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
I have repeatedly had to help people where I work who have put one of these magical -- and invisible -- files or configurations somewhere, forgotten it, and then don't understand why some aspect of their configuration is misbehaving. Now I tell people *only* to configure things in their lisp init files, *not* to use magical directories, and *not* to use environment variables. Then if something goes wrong, you know where to look for the culprit.
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
asdf:*central-registry* is terribly inefficient, *but it is simple* and *it can be inspected when things go wrong*. None of these other schemes share that feature. I have tried to trace the control flow for interpreting the configuration DSL and it's a mass of twisty passages all exactly alike. Lots of key functionality is in variables, or in anonymous lambdas, making tracing effectively impossible. (Adding configuration logging would be a big help)
So -- if you are at Google or some other shop where you have a zillion systems coming together, yes, *central-registry* is too slow, and will kill you. For most of us, though, using one of the alternatives is premature optimization.
I am a strong believer that the only way to keep track of the burgeoning complexity of today's systems is to *localize* and *simplify* configuration, rather than disperse it. For some reason, the dispersion faction is winning the design game, though. I'm not sure why -- perhaps there's some notion of tidiness and elegance that encourages all of these configuration layering (look how elegant! You can override things or accept the default!) and dispersion (there's an individual config file for every purpose, instead of one monster -- and I admit my .emacs is a thing to strike fear in the heart).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that ls won't show it to you?
Finally, *everything* will break, so make sure you know how the poor user will debug it when it does. Make sure things are traceable in CL. Log things. Make your error messages understandable. Lots of ASDF doesn't follow this principle, and I would love to move towards making it easier to diagnose and debug.
"Robert Goldman" rpgoldman@sift.info writes:
This is true, but it causes just the kind of problems I have alluded to -- your ASDF configuration is now smeared all over your system, and debugging it becomes essentially impossible.
You're right that understanding what the source registry is doing is extremely difficult right now. (And I also find the twisty turny passages of the code difficult to follow). I started https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/156 to try and make it better.
The source registry computation works in two phases: the DSL is flattened into a list of directories to search, and then the directories are actually searched. There's a ton more info on the first commit to !156, but now provenance info is recorded in the flattening step. Recording it in the second step should also be trivial once we decide where to record it (*source-registry* or another variable).
Also not a fan of invisible config files. Yeah -- it's ok to have dot files in your home dir so that you don't get overwhelmed, but why is it a good idea to hide the file in your repo that configures the CI so that `ls` won't show it to you?
I personally lean toward hiding the files that are kind of background infrastructure and won't be edited all that frequently. But I also now not everyone does (obviously!). Perhaps one thing we can do is change ASDF to use cl-source-registry.cache if it exists, falling back to .cl-source-registry.cache if it doesn't? (Also mentioned this on !156)
Note also that if ASDF isn't configured in your lisp init file, but by a config file or environment variable, *it will be configured as it loads*. This means that all of your debugging tools are taken away from you. There will be no tracing, because by the time you can set up a trace, the damage will be done.
I don't think this is true. I thought ASDF configured the source registry on the first invocation of find-system? So you can set up traces before that happens. But that may not be useful since the first thing most people do when starting Lisp is load swank which ends up initializing ASDF (on SBCL at least) through a REQUIRE.
If you wanted to trace or otherwise debug a catastrophically broken source registry configuration using SLIME, I think adding a (asdf:initialize-source-registry '(:source-registry :ignore-inherited-configuration)) to your init file would let you get SLIME up and running. Then you can trace the config like normal by evaluating (asdf:initialize-source-registry nil)
And now for my turn on the soap box:
While configuring only through an init file may help localize the configuration from the perspective of a single developer, I think the system level fallbacks are useful to localize the configuration from the perspective of an integrator for a loosely coupled team. I manage multiple development Docker images for my group and have routinely used the system level source registry config to ensure that some of our development tools are available. This is nice because I don't have to badger N people to update their init files if I need to tweak something and they can mount their init files directly from their Docker host into multiple containers (with varying tools prebundled in each) and know that things will Just Work.
On the other end of the spectrum, I've found a use for CL_SOURCE_REGISTRY that I use to hyper-localize my configuration. Unless you jump through hoops, your Lisp init file is generally user wide config. I've been developing CLPM https://www.clpm.dev that configures ASDF on a per-project basis. One of the features it provides is a Ruby Bundler-like exec feature where you can run a command (say in CI or your terminal) and know that any instance of ASDF in that environment will see only the systems you've specified for that project[1]. CL_SOURCE_REGISTRY is the only way I can think of doing this that works seamlessly across all Lisps no matter how they're invoked.
-Eric
[1] plus the systems that are forced upon you by WRAPPING-SOURCE-REGISTRY.
I hadn't considered replacing the impl's ASDF this way. It sounds good, except I do need to remember then that my impl install is 'dirty', and also need to remember to apply this each time I upgrade or if I use a different impl. It does give me the idea of customizing my impl's 'require' hook in my userinit file, however.
This is fine for my purposes since I prefer to isolate my 'builds', I'm just looking to have ASDF upgraded for my day-to-day dev.
On Mon, Jan 18, 2021 at 9:51 PM Eric Timmons etimmons@mit.edu wrote:
On 18 Jan 2021, at 17:37, Wilfredo Velazquez wrote:
To reiterate, I believe that the instructions in https://common-lisp.net/project/asdf/asdf/Upgrading-ASDF.html 'Upgrading ASDF' could use some clarification.
I am attempting to run a newer version of ASDF than that provided by my impl. The text I'm having issue with is:
If your implementation already provides ASDF 3 or later (and it should), but you want a more recent ASDF version than your implementation provides, then you just need to ensure the more recent ASDF is installed in a configured path, like any other system. We recommend you download an official tarball or checkout a release from git into ~/common-lisp/asdf/. Once the source code for ASDF is installed, you don’t need any extra step to load it beyond the usual (require "asdf"): ASDF 3 will automatically look whether an updated version of itself is available amongst the regularly configured systems, before it compiles anything else.
Personally I've never really been a fan of installing ASDF this way since a) it requires loading ASDF twice: the implementation built-in and then the upgrade and b) it requires that ~/common-lisp/ be on ASDF's search path (I mess around with CL_SOURCE_REGISTRY envvar quite a bit and frequently end up excluding ~/common-lisp/). I tend to replace the implementation's built-in version < https://common-lisp.net/project/asdf/asdf.html#Replacing-your-implementation.... But do note, I have seen this method occasionally break and I fall back to the ~/common-lisp/asdf/ apprach when it does. But I'm pretty sure it works in the latest release on at least SBCL, ECL, and CCL.
As far as I can tell, if you put a directory tree into `~/common-lisp/`, ASDF will traverse it looking for system definitions, and there is nothing you can do to override this. I have come to dislike the `~/common-lisp/` solution for this reason.
That's not quite right. It could definitely be more friendly, but there are a few ways to better control it.
To completely prevent ~/common-lisp/ from being traversed you could put an :ignore-inherited-configuration directive somewhere in the CL_SOURCE_REGISTRY envvar or $XDG_CONFIG_DIRS/common-lisp/source-registry.conf. But that approach also would prevent the files in $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d from being parsed (as well as any system level config). It might be nice to allow an inheritance config directive to be specified in the configuration directory parsing if it isn't already (there's an implicit :inherit-configuration tacked on the end of the directory based config).
Another option is to drop a .cl-source-registry.cache file in ~/common-lisp/ or one of the sub directories. https://common-lisp.net/project/asdf/asdf.html#Caching-Results. ASDF will stop recursing if it finds that file and just use the info it contains.
-Eric
-
Eric Timmons -
Marco Antoniotti -
Robert Goldman -
Wilfredo Velazquez