I thought I had understood your question, but now I see that I did not. Let me see if I have this right: 1. There are pre-packaged CL systems (sources) like `cl-alexandria` 2. There are additionally binary systems corresponding to the pre-packaged systems, like `cl-alexandria-sbcl` Are you trying to do the following: - If the prepackaged CL source systems and their binaries are present, then ASDF should be configured so that the output-translations point ASDF to the packaged binaries. - If a prepackaged CL source system is present and the binary package *is not* present, then ASDF should be configured so that the binaries from building the prepackaged systems go into the *user's* cache directory somewhere, because the user will typically not be able to write to the location of the packaged binaries ? If that is so, maybe when a package like `cl-alexandria` is installed, it should write an entry to the output translations to write fasls to the user's directory, and then when a package like `cl-alexandria-sbcl` is written, the output translations should be updated to delete the original entry and add a new entry that points to the packaged fasl directory. Note that I think this could be done by writing to files in `/etc` or some similar location. Honestly, I don't know what to do if the binary package has some but not all fasls. That would be a mess.

Hello Robert,

Thank you for your response.

We loop over the locations where compiled code is stored to build the output translations, as shown in the code below.

(defvar *freebsd-output-translations* ()) (pushnew :inherit-configuration *freebsd-output-translations*) (dolist (path (directory "/usr/local/lib/common-lisp/*/")) (unless (and (pathnamep path) (search *system-registry* (namestring path) :start1 0 :end1 (length *system-registry*))) (let ((source (make-pathname :directory (append (pathname-directory path) (list :wild-inferiors)))) (target (make-pathname :directory (append (pathname-directory path) (list (lisp-specific-fasl-subdir) :wild-inferiors))))) (pushnew (list source target) *freebsd-output-translations*)))) (asdf:initialize-output-translations (cons :output-translations *freebsd-output-translations*))

The files in our CL OS packages are stored under /usr/local/lib/common-lisp/<project>/. The FASL packages put compiled code in directories unique to each lisp implementation such as /usr/local/lib/common-lib/<project>/sbclfasl, /usr/local/lib/common-lisp/<package_name>/cclfasl/, and so on.

Before the call to asdf:initialize-output-translations, *freebsd-output-translations* looks correct (if I'm understanding the DSL format):

(:OUTPUT-TRANSLATIONS (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") :INHERIT-CONFIGURATION)

The part that I'm still missing is how to add locations to the exceptions to output translations.

Kind regards, Joe

P.S. After the call to asdf:initialize-output-translations, dumping asdf::*output-translations* gives:

(((#P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/" T) (#P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/" T) (#P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*" T) (#P"/usr/local/lib/common-lisp/gpgme/**/" #P"/usr/local/lib/common-lisp/gpgme/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/asdf/**/" #P"/usr/local/lib/common-lisp/asdf/sbclfasl/**/") (#P"/usr/local/lib/common-lisp/alexandria/**/" #P"/usr/local/lib/common-lisp/alexandria/sbclfasl/**/") (#P"/home/jrm/.cache/common-lisp/my-dir/**/*.*" T) (#P"/usr/local/lib/sbcl/**/*.*" T) (#P"/**/*.*" #P"/home/jrm/.cache/common-lisp/my-dir/**/*.*") (T #P"/usr/home/jrm/.cache/common-lisp/sbcl-2.4.10-bsd-x64-s/**/*.*")))

Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC) 319 N. First Ave., Suite 400 Minneapolis, MN 55401 Google Voice: (612) 326-3934 Cell: (612) 384-3454 Email: rpgoldman@SIFT.net

On 28 Nov 2024, at 19:10, Joseph Mingrone wrote:

Hello Robert,

Your description of what we are trying to accomplish is accurate. In short, we aim to:

- Use existing binaries when they are installed as OS packages. - Allow things to work as expected when those binaries are not installed, i.e., write binaries to the user's cache.

Your suggestion to write configuration files along with the packages to control output translation seems like a good approach. That is, when a user installs the cl-alexandria-sbcl package, we could write ${PREFIX}/etc/common-lisp/asdf-output-translations.conf.d/10-cl-alexandria-sbcl.conf. Without that package installed, I believe the default behaviour - writing to the user's cache - works. I'm working on implementing this approach but have encountered hurdles:

1. Custom Configuration Directory

Our packages should only write under ${PREIX} (typically /usr/local/). Is it possible to tell ASDF to read from /usr/local/etc/common-lisp/asdf-output-translations.conf.d instead of /etc/common-lisp/asdf-output-translations.conf.d? Ideally, we could do this from our asdf-init.lisp. I notice that #:system-output-translations-pathname is exported. Could it be used for this purpose?

That's a function binding, so no, it cannot. I think you *could* change the function definition of `system-config-pathnames` to change *all* ASDF configuration locations from `/etc/` to `/usr/local/etc/`. I do not know enough about how FreeBSD works to know if this is OK for your use case. I'm afraid that I don't understand how XDG works, either: that was Faré's addition. The API does not look to me to be able to set a special configuration directory for only a specific kind of configuration (such as only for output translations and not for source-finding).

2. Configuration Error

I manually created a test configuration file at /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf, with this DSL:

% cat /etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf (:output-translations :inherit-configuration ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/"))

However, ASDF reports an error:

% sbcl This is SBCL 2.4.10, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/> .

SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ;;; loading #P"/usr/local/lib/common-lisp/asdf/sbclfasl/build/asdf.fasl" * (asdf:load-system :foo)

debugger invoked on a ASDF/OUTPUT-TRANSLATIONS:INVALID-OUTPUT-TRANSLATION in thread #<THREAD tid=103125 "main thread" RUNNING {1103F50093}>: Invalid asdf output-translation (:OUTPUT-TRANSLATIONS :INHERIT-CONFIGURATION ("/usr/local/lib/common-lisp/asdf/**/" "/usr/local/lib/common-lisp/asdf/sbclfasl/**/")) in #P"/etc/common-lisp/asdf-output-translations.conf.d/10-asdf-sbcl.conf" (will be skipped)

Do you know what's wrong with this DSL?

I do not. Do you have a backtrace? I suspect the problem is the use of "**", but I'm not sure. The manual states that ``` DIRECTORY-DESIGNATOR := NIL | ; As source: skip this entry. As destination: same as source T | ; as source matches anything, as destination ; maps pathname to itself. ABSOLUTE-COMPONENT-DESIGNATOR ; same as in the source-registry language ``` And going to the page for the source registry DSL: ``` ABSOLUTE-COMPONENT-DESIGNATOR := (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) | STRING | ;; namestring (better be absolute or bust, directory assumed where ;; applicable). In output-translations, directory is assumed and ;; **/*.*.* added if it's last. On MCL, a MacOSX-style POSIX ;; namestring (for MacOS9 style, use #p"..."); Note that none of the ;; above applies to strings used in *central-registry*, which ;; doesn't use this DSL: they are processed as normal namestrings. ;; however, you can compute what you put in the *central-registry* ;; based on the results of say ;; (asdf::resolve-location "/Users/fare/cl/cl-foo/") PATHNAME | ;; pathname (better be an absolute path, or bust) ;; In output-translations, unless followed by relative components, ;; it better have appropriate wildcards, as in **/*.*.* :HOME | ; designates the user-homedir-pathname ~/ :USER-CACHE | ; designates the default location for the user cache :HERE | ;; designates the location of the configuration file ;; (or *default-pathname-defaults*, if invoked interactively) :ROOT ;; magic, for output-translations source only: paths that are relative ;; to the root of the source host and device ``` Possibly you need a trailing `*.*.*`? Also, as I look at this, do you want your new output translations to shadow old ones? In that case you should put `:inherit-configuration` last instead of first.

3. Distinguishing Lisp Implementations

This approach will only work if we can distinguish the running Lisp implementation. The manual mentions, "You may use #+features to customize the configuration file." Can we use something like #+sbcl in the .conf files to conditionally set up the output translations for SBCL?

Honestly, I don't know. But when I look at the source for `parse-output-translations-string`, it looks like it's a string parser, and does not use `read`, in which case reader macros would not work. But TBH I cannot see how the How about starting up a lisp with `process-output-translations` and `parse-output-translations-string` traced and posting the results here. Note that this is one of the things I don't like about the way that ASDF uses configuration files: it's hard to load ASDF and configure it for debugging *before* it sees its configuration files. You could actually edit the ASDF source to inject the tracing (or `format` statements), of course. (I have been meaning to add some switch to startup so that one can debug its initialization, but have not had the time.)

Thank you for your time and insight.

Kind regards, Joe

On 1 Jan 2025, at 16:33, Joseph Mingrone wrote:

As you can see, in this particular case, the ASDF package itself is being built. For CCL, CLISP, and SBCL, everything works as expected, and the FASL files are written under /wrkdirs/. However, when building packages for ECL, ASDF tries to write the FASL files to ${HOME}/.cache/common-lisp, which fails due to restrictions in our package-building tool.

I noticed some #+ecl conditions in uiop/lisp-build.lisp, but have not been able to pinpoint the issue. The ECL maintainer guessed that ASDF might fail to write to the expected location because it's trying to rename files between different mount points when POSIX says it should copy them. I haven't had a chance to check the source, but it seems like this explanation is probably wrong. Here's why: if the problem was not being able to write to the expected location, I don't believe that the code would fall back to writing in `~/.cache`. I would have expected the output translations to fail and the fasl file (possibly with a temp name) be left in the compilation directory.

To debug, the first thing I would do is look at what's in `asdf/output-translations::*output-translations*` when running in ECL and compare that with what's there in one of the implementations where the output translations are working properly. That will tell us if the OT specifications are being processed correctly, at least. If that looks good, then I would see what happens with some calls to `asdf/output-translations::apply-output-translations` in the REPL. If you find errors there, please post a bug to the ASDF issues site. If you *don't* please email again and we can try to figure out what's going wrong. Worst case, you could modify your copy of ASDF to add a call to `(trace asdf/output-translations::apply-output-translations)` and see what happens then. Also, are you using this with ASDF taken from our git repo? It looks like you are working with some packaged version. Please see if you get the same behavior after loading the latest `asdf.lisp` instead of whatever is coming from FreeBSD.