Hello,
I'm attempting to adjust the ASDF configuration for FreeBSD CL ports/packages to balance the use of precompiled FASL files from OS packages with the ability to compile and write code under users' home directories. Details are below. Any advice would be appreciated.
The system-wide configuration [0] I'm attempting to adjust is used both when building FreeBSD CL OS packages and on target systems. Specifically, it:
- sets up a central registry, - enforces a filesystem layout, - and configures output translations.
With this configuration, users can run `pkg install cl-alexandria-sbcl` and two packages will be installed:
- cl-alexandria, essentially the upstream repository, - cl-alexandria-sbcl, the compiled code.
From their Lisp implementation, users only need to execute (asdf:load-system :alexandria) and no local compilation is required.
However, if a user installs only cl-alexandria without cl-alexandria-sbcl, issues arise. Due to the output translations, ASDF attempts to write compiled code to a location where the user lacks write access. On the other hand, if we remove the output translations on target systems, all FASL files provided by the OS packages are ignored, and ASDF tries to recompile and write them under ~/.cache/common-lisp/.
Is there a way to configure ASDF to recognize and accept the FASL files from the OS packages and compile and write any missing FASL files to the default untranslated location under ~/.cache/common-lisp/?
Thank you, Joe
[0] https://cgit.freebsd.org/ports/tree/devel/cl-asdf/bsd.cl-asdf.mk
I haven't verified the details, but I believe the short answer to your question is "yes."
What one would need to do is to find where packages such as `cl-alexandria-sbcl` write their compiled code and add that location to the exceptions to output translations.
See [the asdf manual](https://asdf.common-lisp.dev/asdf.html#Controlling-where-ASDF-saves-compiled...), and particularly look at the definition of the configuration DSL.
This may require the user to appropriately employ `:inherit-configuration` in any user-specific configuration they do, but in general it's not a good idea to `:ignore-inherited-configuration` unless you have a very good reason.
Best, R
On 24 Nov 2024, at 11:08, Joseph Mingrone wrote:
Hello,
I'm attempting to adjust the ASDF configuration for FreeBSD CL ports/packages to balance the use of precompiled FASL files from OS packages with the ability to compile and write code under users' home directories. Details are below. Any advice would be appreciated.
The system-wide configuration [0] I'm attempting to adjust is used both when building FreeBSD CL OS packages and on target systems. Specifically, it:
- sets up a central registry, - enforces a filesystem layout, - and configures output translations.With this configuration, users can run `pkg install cl-alexandria-sbcl` and two packages will be installed:
- cl-alexandria, essentially the upstream repository, - cl-alexandria-sbcl, the compiled code.From their Lisp implementation, users only need to execute (asdf:load-system :alexandria) and no local compilation is required.
However, if a user installs only cl-alexandria without cl-alexandria-sbcl, issues arise. Due to the output translations, ASDF attempts to write compiled code to a location where the user lacks write access. On the other hand, if we remove the output translations on target systems, all FASL files provided by the OS packages are ignored, and ASDF tries to recompile and write them under ~/.cache/common-lisp/.
Is there a way to configure ASDF to recognize and accept the FASL files from the OS packages and compile and write any missing FASL files to the default untranslated location under ~/.cache/common-lisp/?
Thank you, Joe
[0] https://cgit.freebsd.org/ports/tree/devel/cl-asdf/bsd.cl-asdf.mk
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)
319 N. First Ave., Suite 400 Minneapolis, MN 55401
Google Voice: (612) 326-3934 Cell: (612) 384-3454 Email: rpgoldman@SIFT.net
On Sun, 2024-11-24 at 14:09, Robert Goldman rpgoldman@sift.info wrote:
I haven't verified the details, but I believe the short answer to your question is "yes."
What one would need to do is to find where packages such as `cl-alexandria-sbcl` write their compiled code and add that location to the exceptions to output translations.
See [the asdf manual](https://asdf.common-lisp.dev/asdf.html#Controlling-where-ASDF-saves-compiled...), and particularly look at the definition of the configuration DSL.
This may require the user to appropriately employ `:inherit-configuration` in any user-specific configuration they do, but in general it's not a good idea to `:ignore-inherited-configuration` unless you have a very good reason.
Best, R
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)
Hello Robert,
Thank you for your response.
We loop over the locations where compiled code is stored to build the output translations, as shown in the code below.
(defvar *freebsd-output-translations* ()) (pushnew :inherit-configuration *freebsd-output-translations*) (dolist (path (directory "/usr/local/lib/common-lisp/*/")) (unless (and (pathnamep path) (search *system-registry* (namestring path) :start1 0 :end1 (length *system-registry*))) (let ((source (make-pathname :directory (append (pathname-directory path) (list :wild-inferiors)))) (target (make-pathname :directory (append (pathname-directory path) (list (lisp-specific-fasl-subdir) :wild-inferiors))))) (pushnew (list source target) *freebsd-output-translations*)))) (asdf:initialize-output-translations (cons :output-translations *freebsd-output-translations*))
The files in our CL OS packages are stored under /usr/local/lib/common-lisp/<project>/. The FASL packages put compiled code in directories unique to each lisp implementation such as /usr/local/lib/common-lib/<project>/sbclfasl, /usr/local/lib/common-lisp/<package_name>/cclfasl/, and so on.
Before the call to asdf:initialize-output-translations, *freebsd-output-translations* looks correct (if I'm understanding the DSL format):
(:OUTPUT-TRANSLATIONS (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") :INHERIT-CONFIGURATION)
The part that I'm still missing is how to add locations to the exceptions to output translations.
Kind regards, Joe
P.S. After the call to asdf:initialize-output-translations, dumping asdf::*output-translations* gives:
(((#P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/" T) (#P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*" T) (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") (#P"/home/jrm/.cache/common-lisp/my-dir/**/*.*" T) (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/**/*.*" #P"/home/jrm/.cache/common-lisp/my-dir/**/*.*") (T #P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*")))
I thought I had understood your question, but now I see that I did not. Let me see if I have this right:
1. There are pre-packaged CL systems (sources) like `cl-alexandria` 2. There are additionally binary systems corresponding to the pre-packaged systems, like `cl-alexandria-sbcl`
Are you trying to do the following:
- If the prepackaged CL source systems and their binaries are present, then ASDF should be configured so that the output-translations point ASDF to the packaged binaries. - If a prepackaged CL source system is present and the binary package *is not* present, then ASDF should be configured so that the binaries from building the prepackaged systems go into the *user's* cache directory somewhere, because the user will typically not be able to write to the location of the packaged binaries
?
If that is so, maybe when a package like `cl-alexandria` is installed, it should write an entry to the output translations to write fasls to the user's directory, and then when a package like `cl-alexandria-sbcl` is written, the output translations should be updated to delete the original entry and add a new entry that points to the packaged fasl directory.
Note that I think this could be done by writing to files in `/etc` or some similar location.
Honestly, I don't know what to do if the binary package has some but not all fasls. That would be a mess.
Hello Robert,
Thank you for your response.
We loop over the locations where compiled code is stored to build the output translations, as shown in the code below.
(defvar *freebsd-output-translations* ()) (pushnew :inherit-configuration *freebsd-output-translations*) (dolist (path (directory "/usr/local/lib/common-lisp/*/")) (unless (and (pathnamep path) (search *system-registry* (namestring path) :start1 0 :end1 (length *system-registry*))) (let ((source (make-pathname :directory (append (pathname-directory path) (list :wild-inferiors)))) (target (make-pathname :directory (append (pathname-directory path) (list (lisp-specific-fasl-subdir) :wild-inferiors))))) (pushnew (list source target) *freebsd-output-translations*)))) (asdf:initialize-output-translations (cons :output-translations *freebsd-output-translations*))
The files in our CL OS packages are stored under /usr/local/lib/common-lisp/<project>/. The FASL packages put compiled code in directories unique to each lisp implementation such as /usr/local/lib/common-lib/<project>/sbclfasl, /usr/local/lib/common-lisp/<package_name>/cclfasl/, and so on.
Before the call to asdf:initialize-output-translations, *freebsd-output-translations* looks correct (if I'm understanding the DSL format):
(:OUTPUT-TRANSLATIONS (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") :INHERIT-CONFIGURATION)
The part that I'm still missing is how to add locations to the exceptions to output translations.
Kind regards, Joe
P.S. After the call to asdf:initialize-output-translations, dumping asdf::*output-translations* gives:
(((#P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/" T) (#P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*" T) (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") (#P"/home/jrm/.cache/common-lisp/my-dir/**/*.*" T) (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/**/*.*" #P"/home/jrm/.cache/common-lisp/my-dir/**/*.*") (T #P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*")))
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)
319 N. First Ave., Suite 400 Minneapolis, MN 55401
Google Voice: (612) 326-3934 Cell: (612) 384-3454 Email: rpgoldman@SIFT.net
Hello Robert,
Your description of what we are trying to accomplish is accurate. In short, we aim to:
- Use existing binaries when they are installed as OS packages. - Allow things to work as expected when those binaries are not installed, i.e., write binaries to the user's cache.
Your suggestion to write configuration files along with the packages to control output translation seems like a good approach. That is, when a user installs the cl-alexandria-sbcl package, we could write ${PREFIX}/etc/common-lisp/asdf-output-translations.conf.d/10-cl-alexandria-sbcl.conf. Without that package installed, I believe the default behaviour - writing to the user's cache - works. I'm working on implementing this approach but have encountered hurdles:
1. Custom Configuration Directory
Our packages should only write under ${PREIX} (typically /usr/local/). Is it possible to tell ASDF to read from /usr/local/etc/common-lisp/asdf-output-translations.conf.d instead of /etc/common-lisp/asdf-output-translations.conf.d? Ideally, we could do this from our asdf-init.lisp. I notice that #:system-output-translations-pathname is exported. Could it be used for this purpose?
2. Configuration Error
I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL:
% cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/"))
However, ASDF reports an error:
% sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at http://www.sbcl.org/.
SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl" * (asdf:load-system :foo)
debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped)
Do you know what's wrong with this DSL?
3. Distinguishing Lisp Implementations
This approach will only work if we can distinguish the running Lisp implementation. The manual mentions, "You may use #+features to customize the configuration file." Can we use something like #+sbcl in the .conf files to conditionally set up the output translations for SBCL?
Thank you for your time and insight.
Kind regards, Joe
On 28 Nov 2024, at 19:10, Joseph Mingrone wrote:
Hello Robert,
Your description of what we are trying to accomplish is accurate. In short, we aim to:
- Use existing binaries when they are installed as OS packages. - Allow things to work as expected when those binaries are not installed, i.e., write binaries to the user's cache.Your suggestion to write configuration files along with the packages to control output translation seems like a good approach. That is, when a user installs the cl-alexandria-sbcl package, we could write ${PREFIX}/etc/common-lisp/asdf-output-translations.conf.d/10-cl-alexandria-sbcl.conf. Without that package installed, I believe the default behaviour - writing to the user's cache - works. I'm working on implementing this approach but have encountered hurdles:
- Custom Configuration Directory
Our packages should only write under ${PREIX} (typically /usr/local/). Is it possible to tell ASDF to read from /usr/local/etc/common-lisp/asdf-output-translations.conf.d instead of /etc/common-lisp/asdf-output-translations.conf.d? Ideally, we could do this from our asdf-init.lisp. I notice that #:system-output-translations-pathname is exported. Could it be used for this purpose?
That's a function binding, so no, it cannot. I think you *could* change the function definition of `system-config-pathnames` to change *all* ASDF configuration locations from `/etc/` to `/usr/local/etc/`. I do not know enough about how FreeBSD works to know if this is OK for your use case. I'm afraid that I don't understand how XDG works, either: that was Faré's addition.
The API does not look to me to be able to set a special configuration directory for only a specific kind of configuration (such as only for output translations and not for source-finding).
- Configuration Error
I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL:
% cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/"))
However, ASDF reports an error:
% sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at http://www.sbcl.org/ .
SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl"
- (asdf:load-system :foo)
debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped)
Do you know what's wrong with this DSL?
I do not. Do you have a backtrace? I suspect the problem is the use of "**", but I'm not sure. The manual states that
``` DIRECTORY-DESIGNATOR := NIL | ; As source: skip this entry. As destination: same as source T | ; as source matches anything, as destination ; maps pathname to itself. ABSOLUTE-COMPONENT-DESIGNATOR ; same as in the source-registry language ```
And going to the page for the source registry DSL:
``` ABSOLUTE-COMPONENT-DESIGNATOR := (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) | STRING | ;; namestring (better be absolute or bust, directory assumed where ;; applicable). In output-translations, directory is assumed and ;; **/*.*.* added if it's last. On MCL, a MacOSX-style POSIX ;; namestring (for MacOS9 style, use #p"..."); Note that none of the ;; above applies to strings used in *central-registry*, which ;; doesn't use this DSL: they are processed as normal namestrings. ;; however, you can compute what you put in the *central-registry* ;; based on the results of say ;; (asdf::resolve-location "/Users/fare/cl/cl-foo/") PATHNAME | ;; pathname (better be an absolute path, or bust) ;; In output-translations, unless followed by relative components, ;; it better have appropriate wildcards, as in **/*.*.* :HOME | ; designates the user-homedir-pathname ~/ :USER-CACHE | ; designates the default location for the user cache :HERE | ;; designates the location of the configuration file ;; (or *default-pathname-defaults*, if invoked interactively) :ROOT ;; magic, for output-translations source only: paths that are relative ;; to the root of the source host and device ```
Possibly you need a trailing `*.*.*`?
Also, as I look at this, do you want your new output translations to shadow old ones? In that case you should put `:inherit-configuration` last instead of first.
- Distinguishing Lisp Implementations
This approach will only work if we can distinguish the running Lisp implementation. The manual mentions, "You may use #+features to customize the configuration file." Can we use something like #+sbcl in the .conf files to conditionally set up the output translations for SBCL?
Honestly, I don't know. But when I look at the source for `parse-output-translations-string`, it looks like it's a string parser, and does not use `read`, in which case reader macros would not work.
But TBH I cannot see how the
How about starting up a lisp with `process-output-translations` and `parse-output-translations-string` traced and posting the results here. Note that this is one of the things I don't like about the way that ASDF uses configuration files: it's hard to load ASDF and configure it for debugging *before* it sees its configuration files. You could actually edit the ASDF source to inject the tracing (or `format` statements), of course. (I have been meaning to add some switch to startup so that one can debug its initialization, but have not had the time.)
Thank you for your time and insight.
Kind regards, Joe
-
Joseph Mingrone -
Robert Goldman