Dear users and developers,
I'm having issues using the bundling ops, deliver-asd-op to be specific.
Apparently, since commit c213e319fd9f6dc53570f0a05034ee31a24ae1ae, the generated system definitions get mangled when deliver-asd-op is used on slashy subsystems. Instead of foo/bar, the system name becomes foo--bar. Previously, the pathname-name was set to the full complex component name (sneakily including the forward slash character).
Consider this ABCL session (ASDF 3.3.7):
(load "https://common-lisp.net/project/asdf/archives/asdf.lisp")
(ensure-directories-exist "~/common-lisp/testsys1/")
(uiop:with-output-file (o "~/common-lisp/testsys1/testsys1.asd") (print '(defsystem "testsys1" :components ((:file "test1"))) o) (print '(defsystem "testsys1/sub" :components ((:file "test2"))) o))
(uiop:with-output-file (o "~/common-lisp/testsys1/test1.lisp") (print '(defpackage test1 (:use :cl)) o) (print '(in-package test1) o) (print '(defun test1 () 42) o))
(uiop:with-output-file (o "~/common-lisp/testsys1/test2.lisp") (print '(defpackage test2 (:use :cl)) o) (print '(in-package test2) o) (print '(defun test2 () 43) o))
(asdf:clear-configuration) (asdf:operate 'asdf:deliver-asd-op "testsys1") (asdf:operate 'asdf:deliver-asd-op "testsys1/sub")
(uiop:println (list (asdf:output-files 'asdf:compile-bundle-op "testsys1") (asdf:output-files 'asdf:deliver-asd-op "testsys1") (asdf:output-files 'asdf:compile-bundle-op "testsys1/sub") (asdf:output-files 'asdf:deliver-asd-op "testsys1/sub")))
The result:
(("~/.cache/common-lisp/.../testsys1/testsys1--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1--sub--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1--sub.asd"))
Concatenating these .asd files and replacing the invalid system names becomes an epic hack. At which point it might be easier to implement a replacement asdf:operation. But it might be better to do it in ASDF itself.
To keep these systems loadable, their definitions should land in the parent .asd file. But if ASDF checks whether an operation is already completed based on probe-file, it becomes a problem when multiple slashy systems are involved. It could cause early cutoff. The deliver-asd-op would have to check for the existence of a corresponding defsystem form in the file, rather than the mere existence of that file.
The incoming defsystems would have to be appended. I don't know whether that should be automatic or require an explicit invoke-restart. There might be some validation required to verify that the primary system is also being delivered. (Correct?)
Fasls can probably stay mangled, since they can be targeted independently, as opposed to .asd files... well, that's unless someone decides to create a (defsystem "testsys1--sub") - though it does seem unconventional. Maybe that would be an acceptable limitation.
Here are the results from SBCL, which still uses uses ASDF 3.3.1:
(("~/.cache/common-lisp/.../testsys1/testsys1--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1/sub--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1/sub.asd"))
(By the way, I'm not sure how did the testsys1/testsys1 subdirectory manage to get created without an error, since it seems that testsys/sub is the pathname-name, and I believe ensure-directory-exists works with pathname-directory.)
In this case, I'm lucky that the testsys1 project doesn't contain an inner testsys1 directory containing another ASD file "sub.asd" - that would cause another conflict, but I digress.
To summarize, regarding deliver-asd-op:
- The slashy system names and the .asd pathname-names are being mangled - The system definitions need to stay in one file - DIY merging un-mangling them afterwards is too hairy - Let's discuss encapsulating that hairiness in ASDF itself
The goal: to support binary deployment of slashy systems (non-slashy systems already work), so that the generated .asd and .fasl can be installed, without source, on client systems, all the while being ASDF-discoverable.
An alternative workaround might be to create an ephemeral uber system just for delivery, one that depends-on the parent system and every desired slashy system. But that prevents fine grained updates and also creates one more system to remember.
I'm so sorry to have taken so long to answer: work has been terribly busy.
I think it would be best to post this as an issue to the ASDF git repo [here](https://gitlab.common-lisp.net/asdf/asdf/-/issues). I have never even used the bundling operations (except for the one that assembles `build/asdf.lisp`), much less looked into the code.
I'd suggest reading [ASDF issue 23](https://gitlab.common-lisp.net/asdf/asdf/-/issues/23) since it was the attempt to fix this issue that led to the commit you pointed to.
I think in the short term you are best off creating the uber system.
On 17 Sep 2024, at 18:50, Kasper Gałkowski wrote:
Dear users and developers,
I'm having issues using the bundling ops, deliver-asd-op to be specific.
Apparently, since commit c213e319fd9f6dc53570f0a05034ee31a24ae1ae, the generated system definitions get mangled when deliver-asd-op is used on slashy subsystems. Instead of foo/bar, the system name becomes foo--bar. Previously, the pathname-name was set to the full complex component name (sneakily including the forward slash character).
Consider this ABCL session (ASDF 3.3.7):
(load "https://common-lisp.net/project/asdf/archives/asdf.lisp")
(ensure-directories-exist "~/common-lisp/testsys1/")
(uiop:with-output-file (o "~/common-lisp/testsys1/testsys1.asd") (print '(defsystem "testsys1" :components ((:file "test1"))) o) (print '(defsystem "testsys1/sub" :components ((:file "test2"))) o))
(uiop:with-output-file (o "~/common-lisp/testsys1/test1.lisp") (print '(defpackage test1 (:use :cl)) o) (print '(in-package test1) o) (print '(defun test1 () 42) o))
(uiop:with-output-file (o "~/common-lisp/testsys1/test2.lisp") (print '(defpackage test2 (:use :cl)) o) (print '(in-package test2) o) (print '(defun test2 () 43) o))
(asdf:clear-configuration) (asdf:operate 'asdf:deliver-asd-op "testsys1") (asdf:operate 'asdf:deliver-asd-op "testsys1/sub")
(uiop:println (list (asdf:output-files 'asdf:compile-bundle-op "testsys1") (asdf:output-files 'asdf:deliver-asd-op "testsys1") (asdf:output-files 'asdf:compile-bundle-op "testsys1/sub") (asdf:output-files 'asdf:deliver-asd-op "testsys1/sub")))
The result:
(("~/.cache/common-lisp/.../testsys1/testsys1--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1--sub--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1--sub.asd"))
Concatenating these .asd files and replacing the invalid system names becomes an epic hack. At which point it might be easier to implement a replacement asdf:operation. But it might be better to do it in ASDF itself.
To keep these systems loadable, their definitions should land in the parent .asd file. But if ASDF checks whether an operation is already completed based on probe-file, it becomes a problem when multiple slashy systems are involved. It could cause early cutoff. The deliver-asd-op would have to check for the existence of a corresponding defsystem form in the file, rather than the mere existence of that file.
The incoming defsystems would have to be appended. I don't know whether that should be automatic or require an explicit invoke-restart. There might be some validation required to verify that the primary system is also being delivered. (Correct?)
Fasls can probably stay mangled, since they can be targeted independently, as opposed to .asd files... well, that's unless someone decides to create a (defsystem "testsys1--sub") - though it does seem unconventional. Maybe that would be an acceptable limitation.
Here are the results from SBCL, which still uses uses ASDF 3.3.1:
(("~/.cache/common-lisp/.../testsys1/testsys1--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1/sub--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1/sub.asd"))
(By the way, I'm not sure how did the testsys1/testsys1 subdirectory manage to get created without an error, since it seems that testsys/sub is the pathname-name, and I believe ensure-directory-exists works with pathname-directory.)
In this case, I'm lucky that the testsys1 project doesn't contain an inner testsys1 directory containing another ASD file "sub.asd" - that would cause another conflict, but I digress.
To summarize, regarding deliver-asd-op:
- The slashy system names and the .asd pathname-names are being
mangled
- The system definitions need to stay in one file
- DIY merging un-mangling them afterwards is too hairy
- Let's discuss encapsulating that hairiness in ASDF itself
The goal: to support binary deployment of slashy systems (non-slashy systems already work), so that the generated .asd and .fasl can be installed, without source, on client systems, all the while being ASDF-discoverable.
An alternative workaround might be to create an ephemeral uber system just for delivery, one that depends-on the parent system and every desired slashy system. But that prevents fine grained updates and also creates one more system to remember.
Robert P. Goldman Research Fellow Smart Information Flow Technologies (d/b/a SIFT, LLC)
319 N. First Ave., Suite 400 Minneapolis, MN 55401
Voice: (612) 326-3934 Email: rpgoldman@SIFT.net
My use case is very similar to what Jason Miller described[1] previously. I'm also using Nix to build ASDF systems, and turned to bundle ops after finding source registry and output translations configuration too hard to manage. I also had problems with symlink behavior, which are often used in Nix but are hairy in CL[2]. Anyway, I got stuck working on the right Nix-side representation of ASDF systems before being forced to deal with the mentioned slashy system bundling issues. But with other stuff that has come up, I've shelved the topic and sticked with the SR/OT solution for deployments. Let's see if I'll be able to revisit in 2025. When that happens, I'll post a fresh bug report/patches to gitlab. In the meantime, anyone descending down the Lisp+Nix rabbit hole, feel free to get in touch.
[1]: https://mailman3.common-lisp.net/hyperkitty/list/asdf-devel@common-lisp.net/... [2]: https://patch-diff.githubusercontent.com/raw/Uthar/nix-cl/pull/42.patch
Maybe the Right Thing™ might be to walk the set of all slashy subsystems in the original asd, and create the same set, with the same names and same dependencies, in the delivered asd—just precompiled as .fasl instead of .lisp.
I'm not volunteering to do any of the work (until someday hired as a CL developer again, maybe). But as usual, I'm willing to consult wrt what the current implementation does and what I think it should be doing.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org “The Government is just the armed gang you're afraid would take over in the absence of Government.”
On Sun, Oct 20, 2024 at 8:06 AM Kasper Gałkowski kasgal@posteo.net wrote:
My use case is very similar to what Jason Miller described[1] previously. I'm also using Nix to build ASDF systems, and turned to bundle ops after finding source registry and output translations configuration too hard to manage. I also had problems with symlink behavior, which are often used in Nix but are hairy in CL[2]. Anyway, I got stuck working on the right Nix-side representation of ASDF systems before being forced to deal with the mentioned slashy system bundling issues. But with other stuff that has come up, I've shelved the topic and sticked with the SR/OT solution for deployments. Let's see if I'll be able to revisit in 2025. When that happens, I'll post a fresh bug report/patches to gitlab. In the meantime, anyone descending down the Lisp+Nix rabbit hole, feel free to get in touch.
-- -Kasper
-
Faré -
Kasper Gałkowski -
Robert Goldman