Balancing ASDF Configuration for Precompiled OS and User-Compiled FASL Files
Hello, I'm attempting to adjust the ASDF configuration for FreeBSD CL ports/packages to balance the use of precompiled FASL files from OS packages with the ability to compile and write code under users' home directories. Details are below. Any advice would be appreciated. The system-wide configuration [0] I'm attempting to adjust is used both when building FreeBSD CL OS packages and on target systems. Specifically, it: - sets up a central registry, - enforces a filesystem layout, - and configures output translations. With this configuration, users can run `pkg install cl-alexandria-sbcl` and two packages will be installed: - cl-alexandria, essentially the upstream repository, - cl-alexandria-sbcl, the compiled code. From their Lisp implementation, users only need to execute (asdf:load-system :alexandria) and no local compilation is required. However, if a user installs only cl-alexandria without cl-alexandria-sbcl, issues arise. Due to the output translations, ASDF attempts to write compiled code to a location where the user lacks write access. On the other hand, if we remove the output translations on target systems, all FASL files provided by the OS packages are ignored, and ASDF tries to recompile and write them under ~/.cache/common-lisp/. Is there a way to configure ASDF to recognize and accept the FASL files from the OS packages and compile and write any missing FASL files to the default untranslated location under ~/.cache/common-lisp/? Thank you, Joe [0] https://cgit.freebsd.org/ports/tree/devel/cl-asdf/bsd.cl-asdf.mk
I haven't verified the details, but I believe the short answer to your question is "yes." What one would need to do is to find where packages such as `cl-alexandria-sbcl` write their compiled code and add that location to the exceptions to output translations. See [the asdf manual](https://asdf.common-lisp.dev/asdf.html#Controlling-where-ASDF-saves-compiled...), and particularly look at the definition of the configuration DSL. This may require the user to appropriately employ `:inherit-configuration` in any user-specific configuration they do, but in general it's not a good idea to `:ignore-inherited-configuration` unless you have a very good reason. Best, R On 24 Nov 2024, at 11:08, Joseph Mingrone wrote:
Hello,
I'm attempting to adjust the ASDF configuration for FreeBSD CL ports/packages to balance the use of precompiled FASL files from OS packages with the ability to compile and write code under users' home directories. Details are below. Any advice would be appreciated.
The system-wide configuration [0] I'm attempting to adjust is used both when building FreeBSD CL OS packages and on target systems. Specifically, it:
- sets up a central registry, - enforces a filesystem layout, - and configures output translations.
With this configuration, users can run `pkg install cl-alexandria-sbcl` and two packages will be installed:
- cl-alexandria, essentially the upstream repository, - cl-alexandria-sbcl, the compiled code.
From their Lisp implementation, users only need to execute (asdf:load-system :alexandria) and no local compilation is required.
However, if a user installs only cl-alexandria without cl-alexandria-sbcl, issues arise. Due to the output translations, ASDF attempts to write compiled code to a location where the user lacks write access. On the other hand, if we remove the output translations on target systems, all FASL files provided by the OS packages are ignored, and ASDF tries to recompile and write them under ~/.cache/common-lisp/.
Is there a way to configure ASDF to recognize and accept the FASL files from the OS packages and compile and write any missing FASL files to the default untranslated location under ~/.cache/common-lisp/?
Thank you, Joe
[0] https://cgit.freebsd.org/ports/tree/devel/cl-asdf/bsd.cl-asdf.mk
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC) 319 N. First Ave., Suite 400 Minneapolis, MN 55401 Google Voice: (612) 326-3934 Cell: (612) 384-3454 Email: rpgoldman@SIFT.net
On Sun, 2024-11-24 at 14:09, Robert Goldman <rpgoldman@sift.info> wrote:
I haven't verified the details, but I believe the short answer to your question is "yes."
What one would need to do is to find where packages such as `cl-alexandria-sbcl` write their compiled code and add that location to the exceptions to output translations.
See [the asdf manual](https://asdf.common-lisp.dev/asdf.html#Controlling-where-ASDF-saves-compiled...), and particularly look at the definition of the configuration DSL.
This may require the user to appropriately employ `:inherit-configuration` in any user-specific configuration they do, but in general it's not a good idea to `:ignore-inherited-configuration` unless you have a very good reason.
Best, R
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)
Hello Robert, Thank you for your response. We loop over the locations where compiled code is stored to build the output translations, as shown in the code below. (defvar *freebsd-output-translations* ()) (pushnew :inherit-configuration *freebsd-output-translations*) (dolist (path (directory "/usr/local/lib/common-lisp/*/")) (unless (and (pathnamep path) (search *system-registry* (namestring path) :start1 0 :end1 (length *system-registry*))) (let ((source (make-pathname :directory (append (pathname-directory path) (list :wild-inferiors)))) (target (make-pathname :directory (append (pathname-directory path) (list (lisp-specific-fasl-subdir) :wild-inferiors))))) (pushnew (list source target) *freebsd-output-translations*)))) (asdf:initialize-output-translations (cons :output-translations *freebsd-output-translations*)) The files in our CL OS packages are stored under /usr/local/lib/common-lisp/<project>/. The FASL packages put compiled code in directories unique to each lisp implementation such as /usr/local/lib/common-lib/<project>/sbclfasl, /usr/local/lib/common-lisp/<package_name>/cclfasl/, and so on. Before the call to asdf:initialize-output-translations, *freebsd-output-translations* looks correct (if I'm understanding the DSL format): (:OUTPUT-TRANSLATIONS (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") :INHERIT-CONFIGURATION) The part that I'm still missing is how to add locations to the exceptions to output translations. Kind regards, Joe P.S. After the call to asdf:initialize-output-translations, dumping asdf::*output-translations* gives: (((#P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/" T) (#P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*" T) (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") (#P"/home/jrm/.cache/common-lisp/my-dir/**/*.*" T) (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/**/*.*" #P"/home/jrm/.cache/common-lisp/my-dir/**/*.*") (T #P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*")))
I thought I had understood your question, but now I see that I did not. Let me see if I have this right: 1. There are pre-packaged CL systems (sources) like `cl-alexandria` 2. There are additionally binary systems corresponding to the pre-packaged systems, like `cl-alexandria-sbcl` Are you trying to do the following: - If the prepackaged CL source systems and their binaries are present, then ASDF should be configured so that the output-translations point ASDF to the packaged binaries. - If a prepackaged CL source system is present and the binary package *is not* present, then ASDF should be configured so that the binaries from building the prepackaged systems go into the *user's* cache directory somewhere, because the user will typically not be able to write to the location of the packaged binaries ? If that is so, maybe when a package like `cl-alexandria` is installed, it should write an entry to the output translations to write fasls to the user's directory, and then when a package like `cl-alexandria-sbcl` is written, the output translations should be updated to delete the original entry and add a new entry that points to the packaged fasl directory. Note that I think this could be done by writing to files in `/etc` or some similar location. Honestly, I don't know what to do if the binary package has some but not all fasls. That would be a mess.
Hello Robert,
Thank you for your response.
We loop over the locations where compiled code is stored to build the output translations, as shown in the code below.
(defvar *freebsd-output-translations* ()) (pushnew :inherit-configuration *freebsd-output-translations*) (dolist (path (directory "/usr/local/lib/common-lisp/*/")) (unless (and (pathnamep path) (search *system-registry* (namestring path) :start1 0 :end1 (length *system-registry*))) (let ((source (make-pathname :directory (append (pathname-directory path) (list :wild-inferiors)))) (target (make-pathname :directory (append (pathname-directory path) (list (lisp-specific-fasl-subdir) :wild-inferiors))))) (pushnew (list source target) *freebsd-output-translations*)))) (asdf:initialize-output-translations (cons :output-translations *freebsd-output-translations*))
The files in our CL OS packages are stored under /usr/local/lib/common-lisp/<project>/. The FASL packages put compiled code in directories unique to each lisp implementation such as /usr/local/lib/common-lib/<project>/sbclfasl, /usr/local/lib/common-lisp/<package_name>/cclfasl/, and so on.
Before the call to asdf:initialize-output-translations, *freebsd-output-translations* looks correct (if I'm understanding the DSL format):
(:OUTPUT-TRANSLATIONS (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") :INHERIT-CONFIGURATION)
The part that I'm still missing is how to add locations to the exceptions to output translations.
Kind regards, Joe
P.S. After the call to asdf:initialize-output-translations, dumping asdf::*output-translations* gives:
(((#P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/" T) (#P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*" T) (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") (#P"/home/jrm/.cache/common-lisp/my-dir/**/*.*" T) (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/**/*.*" #P"/home/jrm/.cache/common-lisp/my-dir/**/*.*") (T #P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*")))
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC) 319 N. First Ave., Suite 400 Minneapolis, MN 55401 Google Voice: (612) 326-3934 Cell: (612) 384-3454 Email: rpgoldman@SIFT.net
Hello Robert, Your description of what we are trying to accomplish is accurate. In short, we aim to: - Use existing binaries when they are installed as OS packages. - Allow things to work as expected when those binaries are not installed, i.e., write binaries to the user's cache. Your suggestion to write configuration files along with the packages to control output translation seems like a good approach. That is, when a user installs the cl-alexandria-sbcl package, we could write ${PREFIX}/etc/common-lisp/asdf-output-translations.conf.d/10-cl-alexandria-sbcl.conf. Without that package installed, I believe the default behaviour - writing to the user's cache - works. I'm working on implementing this approach but have encountered hurdles: 1. Custom Configuration Directory Our packages should only write under ${PREIX} (typically /usr/local/). Is it possible to tell ASDF to read from /usr/local/etc/common-lisp/asdf-output-translations.conf.d instead of /etc/common-lisp/asdf-output-translations.conf.d? Ideally, we could do this from our asdf-init.lisp. I notice that #:system-output-translations-pathname is exported. Could it be used for this purpose? 2. Configuration Error I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL: % cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) However, ASDF reports an error: % sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl" * (asdf:load-system :foo) debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped) Do you know what's wrong with this DSL? 3. Distinguishing Lisp Implementations This approach will only work if we can distinguish the running Lisp implementation. The manual mentions, "You may use #+features to customize the configuration file." Can we use something like #+sbcl in the .conf files to conditionally set up the output translations for SBCL? Thank you for your time and insight. Kind regards, Joe
On 28 Nov 2024, at 19:10, Joseph Mingrone wrote:
Hello Robert,
Your description of what we are trying to accomplish is accurate. In short, we aim to:
- Use existing binaries when they are installed as OS packages. - Allow things to work as expected when those binaries are not installed, i.e., write binaries to the user's cache.
Your suggestion to write configuration files along with the packages to control output translation seems like a good approach. That is, when a user installs the cl-alexandria-sbcl package, we could write ${PREFIX}/etc/common-lisp/asdf-output-translations.conf.d/10-cl-alexandria-sbcl.conf. Without that package installed, I believe the default behaviour - writing to the user's cache - works. I'm working on implementing this approach but have encountered hurdles:
1. Custom Configuration Directory
Our packages should only write under ${PREIX} (typically /usr/local/). Is it possible to tell ASDF to read from /usr/local/etc/common-lisp/asdf-output-translations.conf.d instead of /etc/common-lisp/asdf-output-translations.conf.d? Ideally, we could do this from our asdf-init.lisp. I notice that #:system-output-translations-pathname is exported. Could it be used for this purpose?
That's a function binding, so no, it cannot. I think you *could* change the function definition of `system-config-pathnames` to change *all* ASDF configuration locations from `/etc/` to `/usr/local/etc/`. I do not know enough about how FreeBSD works to know if this is OK for your use case. I'm afraid that I don't understand how XDG works, either: that was Faré's addition. The API does not look to me to be able to set a special configuration directory for only a specific kind of configuration (such as only for output translations and not for source-finding).
2. Configuration Error
I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL:
% cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/"))
However, ASDF reports an error:
% sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/> .
SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl" * (asdf:load-system :foo)
debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped)
Do you know what's wrong with this DSL?
I do not. Do you have a backtrace? I suspect the problem is the use of "**", but I'm not sure. The manual states that ``` DIRECTORY-DESIGNATOR := NIL | ; As source: skip this entry. As destination: same as source T | ; as source matches anything, as destination ; maps pathname to itself. ABSOLUTE-COMPONENT-DESIGNATOR ; same as in the source-registry language ``` And going to the page for the source registry DSL: ``` ABSOLUTE-COMPONENT-DESIGNATOR := (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) | STRING | ;; namestring (better be absolute or bust, directory assumed where ;; applicable). In output-translations, directory is assumed and ;; **/*.*.* added if it's last. On MCL, a MacOSX-style POSIX ;; namestring (for MacOS9 style, use #p"..."); Note that none of the ;; above applies to strings used in *central-registry*, which ;; doesn't use this DSL: they are processed as normal namestrings. ;; however, you can compute what you put in the *central-registry* ;; based on the results of say ;; (asdf::resolve-location "/Users/fare/cl/cl-foo/") PATHNAME | ;; pathname (better be an absolute path, or bust) ;; In output-translations, unless followed by relative components, ;; it better have appropriate wildcards, as in **/*.*.* :HOME | ; designates the user-homedir-pathname ~/ :USER-CACHE | ; designates the default location for the user cache :HERE | ;; designates the location of the configuration file ;; (or *default-pathname-defaults*, if invoked interactively) :ROOT ;; magic, for output-translations source only: paths that are relative ;; to the root of the source host and device ``` Possibly you need a trailing `*.*.*`? Also, as I look at this, do you want your new output translations to shadow old ones? In that case you should put `:inherit-configuration` last instead of first.
3. Distinguishing Lisp Implementations
This approach will only work if we can distinguish the running Lisp implementation. The manual mentions, "You may use #+features to customize the configuration file." Can we use something like #+sbcl in the .conf files to conditionally set up the output translations for SBCL?
Honestly, I don't know. But when I look at the source for `parse-output-translations-string`, it looks like it's a string parser, and does not use `read`, in which case reader macros would not work. But TBH I cannot see how the How about starting up a lisp with `process-output-translations` and `parse-output-translations-string` traced and posting the results here. Note that this is one of the things I don't like about the way that ASDF uses configuration files: it's hard to load ASDF and configure it for debugging *before* it sees its configuration files. You could actually edit the ASDF source to inject the tracing (or `format` statements), of course. (I have been meaning to add some switch to startup so that one can debug its initialization, but have not had the time.)
Thank you for your time and insight.
Kind regards, Joe
Hello Robert, Things are progressing, and the issues I described in the first message of this thread have been solved. As I write this, we have support for CCL, CLISP, and SBCL. I'm now working on adding support for ECL, but facing a problem with ASDF output translations being ignored. The ASDF configuration code [0] generates these output translations: ;; CCL (((#P"/wrkdirs/usr/ports/devel/cl-asdf-ccl/work/asdf-3.3.7/**/" T) (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/wrkdirs/usr/ports/devel/cl-asdf-ccl/work/asdf-3.3.7/**/") (#P"/root/.cache/common-lisp/ccl-1.13-f98-bsd-x64/**/*.*" T) (T #P"/root/.cache/common-lisp/ccl-1.13-f98-bsd-x64/**/*.*"))) ;; CLISP (((#P"/wrkdirs/usr/ports/devel/cl-asdf-clisp/work/asdf-3.3.7/**/" T) (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/wrkdirs/usr/ports/devel/cl-asdf-clisp/work/asdf-3.3.7/**/") (#P"/root/.cache/common-lisp/clisp-2.49.93+-unix-x64/**/*.*" T) (T #P"/root/.cache/common-lisp/clisp-2.49.93+-unix-x64/**/*.*"))) ;; ecl (((#P"/wrkdirs/usr/ports/devel/cl-asdf-ecl/work/asdf-3.3.7/**/" T) (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/wrkdirs/usr/ports/devel/cl-asdf-ecl/work/asdf-3.3.7/**/") (#P"/root/.cache/common-lisp/ecl-24.5.10-unknown-bsd-x64/**/*.*" T) (#P"SYS:**;*.*.*" T) (T #P"/root/.cache/common-lisp/ecl-24.5.10-unknown-bsd-x64/**/*.*"))) ;; SBCL (((#P"/wrkdirs/usr/ports/devel/cl-asdf-sbcl/work/asdf-3.3.7/**/" T) (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/wrkdirs/usr/ports/devel/cl-asdf-sbcl/work/asdf-3.3.7/**/") (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/root/.cache/common-lisp/sbcl-2.5.0-bsd-x64/**/*.*" T) (T #P"/root/.cache/common-lisp/sbcl-2.5.0-bsd-x64/**/*.*"))) As you can see, in this particular case, the ASDF package itself is being built. For CCL, CLISP, and SBCL, everything works as expected, and the FASL files are written under /wrkdirs/. However, when building packages for ECL, ASDF tries to write the FASL files to ${HOME}/.cache/common-lisp, which fails due to restrictions in our package-building tool. I noticed some #+ecl conditions in uiop/lisp-build.lisp, but have not been able to pinpoint the issue. The ECL maintainer guessed that ASDF might fail to write to the expected location because it's trying to rename files between different mount points when POSIX says it should copy them. Our build system does use a few different nullfs mounts from outside of the isolated build environment (jail) to create a single mount inside the jail. Do you have any advice for debugging the issue? Kind regards, Joe P.S. For posterity, below is the solution to the problem I described with configuration under asdf-output-translations.conf.d/. On Sun, 2024-12-01 at 15:33, Robert Goldman <rpgoldman@sift.info> wrote:
On 28 Nov 2024, at 19:10, Joseph Mingrone wrote:
2. Configuration Error
I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL:
% cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/"))
However, ASDF reports an error:
% sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/> .
SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl" * (asdf:load-system :foo)
debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped)
Do you know what's wrong with this DSL?
I do not. Do you have a backtrace? I suspect the problem is the use of "**", but I'm not sure.
After a brainstorming session with Robert Brown (reb) on #common-lisp@Libera.Chat and a re-read of the manual, we determined that what's expected in asdf-output-translations.conf.d/whatever.conf is different than what's expected in asdf-output-translations.conf. In .d/, there must only be a set of translation forms like ((:home "foo") (:home "bar")) [0] https://github.com/freebsd/freebsd-ports/blob/main/devel/cl-freebsd-asdf-ini... [1] https://github.com/freebsd/freebsd-ports/blob/main/Mk/Uses/cl.mk#L102-L126
On 1 Jan 2025, at 16:33, Joseph Mingrone wrote:
As you can see, in this particular case, the ASDF package itself is being built. For CCL, CLISP, and SBCL, everything works as expected, and the FASL files are written under /wrkdirs/. However, when building packages for ECL, ASDF tries to write the FASL files to ${HOME}/.cache/common-lisp, which fails due to restrictions in our package-building tool.
I noticed some #+ecl conditions in uiop/lisp-build.lisp, but have not been able to pinpoint the issue. The ECL maintainer guessed that ASDF might fail to write to the expected location because it's trying to rename files between different mount points when POSIX says it should copy them. I haven't had a chance to check the source, but it seems like this explanation is probably wrong. Here's why: if the problem was not being able to write to the expected location, I don't believe that the code would fall back to writing in `~/.cache`. I would have expected the output translations to fail and the fasl file (possibly with a temp name) be left in the compilation directory.
To debug, the first thing I would do is look at what's in `asdf/output-translations::*output-translations*` when running in ECL and compare that with what's there in one of the implementations where the output translations are working properly. That will tell us if the OT specifications are being processed correctly, at least. If that looks good, then I would see what happens with some calls to `asdf/output-translations::apply-output-translations` in the REPL. If you find errors there, please post a bug to the ASDF issues site. If you *don't* please email again and we can try to figure out what's going wrong. Worst case, you could modify your copy of ASDF to add a call to `(trace asdf/output-translations::apply-output-translations)` and see what happens then. Also, are you using this with ASDF taken from our git repo? It looks like you are working with some packaged version. Please see if you get the same behavior after loading the latest `asdf.lisp` instead of whatever is coming from FreeBSD.
(Sorry for an almost year-late comment.) Note that for many years ECL has been shipping with its own "ASDF 3.1.8" forked from our 3.1.7 instead of adopting 3.2.0 (we never released a 3.1.8). What more, its maintainer has refused to work with the ASDF maintainers to even discuss which use cases we failed to properly support, so we could fix them and amend our test suite. I haven't been following this for years, though, so can't speak of the current status for sure, though I suspect things haven't changed. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “When everything seems to be going against you, remember that the airplane takes off against the wind, not with it.” — Henry Ford On Thu, Jan 2, 2025 at 5:00 PM Robert Goldman <rpgoldman@sift.info> wrote:
On 1 Jan 2025, at 16:33, Joseph Mingrone wrote:
As you can see, in this particular case, the ASDF package itself is being built. For CCL, CLISP, and SBCL, everything works as expected, and the FASL files are written under /wrkdirs/. However, when building packages for ECL, ASDF tries to write the FASL files to ${HOME}/.cache/common-lisp, which fails due to restrictions in our package-building tool.
I noticed some #+ecl conditions in uiop/lisp-build.lisp, but have not been able to pinpoint the issue. The ECL maintainer guessed that ASDF might fail to write to the expected location because it's trying to rename files between different mount points when POSIX says it should copy them. I haven't had a chance to check the source, but it seems like this explanation is probably wrong. Here's why: if the problem was not being able to write to the expected location, I don't believe that the code would fall back to writing in ~/.cache. I would have expected the output translations to fail and the fasl file (possibly with a temp name) be left in the compilation directory.
To debug, the first thing I would do is look at what's in asdf/output-translations::*output-translations* when running in ECL and compare that with what's there in one of the implementations where the output translations are working properly. That will tell us if the OT specifications are being processed correctly, at least.
If that looks good, then I would see what happens with some calls to asdf/output-translations::apply-output-translations in the REPL. If you find errors there, please post a bug to the ASDF issues site. If you don't please email again and we can try to figure out what's going wrong.
Worst case, you could modify your copy of ASDF to add a call to (trace asdf/output-translations::apply-output-translations) and see what happens then.
Also, are you using this with ASDF taken from our git repo? It looks like you are working with some packaged version. Please see if you get the same behavior after loading the latest asdf.lisp instead of whatever is coming from FreeBSD.
participants (3)
-
Faré -
Joseph Mingrone -
Robert Goldman