September 2024 - asdf-devel - mailman3.common-lisp.net

Expected deliver-asd-op behavior involving slashy subsystems
by Kasper Gałkowski 21 Oct '24

21 Oct '24

Dear users and developers, I'm having issues using the bundling ops, deliver-asd-op to be specific. Apparently, since commit c213e319fd9f6dc53570f0a05034ee31a24ae1ae, the generated system definitions get mangled when deliver-asd-op is used on slashy subsystems. Instead of foo/bar, the system name becomes foo--bar. Previously, the pathname-name was set to the full complex component name (sneakily including the forward slash character). Consider this ABCL session (ASDF 3.3.7): (load "https://common-lisp.net/project/asdf/archives/asdf.lisp") (ensure-directories-exist "~/common-lisp/testsys1/") (uiop:with-output-file (o "~/common-lisp/testsys1/testsys1.asd") (print '(defsystem "testsys1" :components ((:file "test1"))) o) (print '(defsystem "testsys1/sub" :components ((:file "test2"))) o)) (uiop:with-output-file (o "~/common-lisp/testsys1/test1.lisp") (print '(defpackage test1 (:use :cl)) o) (print '(in-package test1) o) (print '(defun test1 () 42) o)) (uiop:with-output-file (o "~/common-lisp/testsys1/test2.lisp") (print '(defpackage test2 (:use :cl)) o) (print '(in-package test2) o) (print '(defun test2 () 43) o)) (asdf:clear-configuration) (asdf:operate 'asdf:deliver-asd-op "testsys1") (asdf:operate 'asdf:deliver-asd-op "testsys1/sub") (uiop:println (list (asdf:output-files 'asdf:compile-bundle-op "testsys1") (asdf:output-files 'asdf:deliver-asd-op "testsys1") (asdf:output-files 'asdf:compile-bundle-op "testsys1/sub") (asdf:output-files 'asdf:deliver-asd-op "testsys1/sub"))) The result: (("~/.cache/common-lisp/.../testsys1/testsys1--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1--sub--system.abcl") ("~/.cache/common-lisp/.../testsys1/testsys1--sub.asd")) Concatenating these .asd files and replacing the invalid system names becomes an epic hack. At which point it might be easier to implement a replacement asdf:operation. But it might be better to do it in ASDF itself. To keep these systems loadable, their definitions should land in the parent .asd file. But if ASDF checks whether an operation is already completed based on probe-file, it becomes a problem when multiple slashy systems are involved. It could cause early cutoff. The deliver-asd-op would have to check for the existence of a corresponding defsystem form in the file, rather than the mere existence of that file. The incoming defsystems would have to be appended. I don't know whether that should be automatic or require an explicit invoke-restart. There might be some validation required to verify that the primary system is also being delivered. (Correct?) Fasls can probably stay mangled, since they can be targeted independently, as opposed to .asd files... well, that's unless someone decides to create a (defsystem "testsys1--sub") - though it does seem unconventional. Maybe that would be an acceptable limitation. Here are the results from SBCL, which still uses uses ASDF 3.3.1: (("~/.cache/common-lisp/.../testsys1/testsys1--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1.asd") ("~/.cache/common-lisp/.../testsys1/testsys1/sub--system.fasl") ("~/.cache/common-lisp/.../testsys1/testsys1/sub.asd")) (By the way, I'm not sure how did the testsys1/testsys1 subdirectory manage to get created without an error, since it seems that testsys/sub is the pathname-name, and I believe ensure-directory-exists works with pathname-directory.) In this case, I'm lucky that the testsys1 project doesn't contain an inner testsys1 directory containing another ASD file "sub.asd" - that would cause another conflict, but I digress. To summarize, regarding deliver-asd-op: - The slashy system names and the .asd pathname-names are being mangled - The system definitions need to stay in one file - DIY merging un-mangling them afterwards is too hairy - Let's discuss encapsulating that hairiness in ASDF itself The goal: to support binary deployment of slashy systems (non-slashy systems already work), so that the generated .asd and .fasl can be installed, without source, on client systems, all the while being ASDF-discoverable. An alternative workaround might be to create an ephemeral uber system just for delivery, one that depends-on the parent system and every desired slashy system. But that prevents fine grained updates and also creates one more system to remember.

3 3

Re: Working around nonconforming names in system definitions
by Robert Dodier 27 Sep '24

27 Sep '24

On Sat, Sep 14, 2024 at 11:01 AM Robert P. Goldman <rpg(a)sift.net> wrote: > I’ve got to push back on your rant. There’s a perfectly good reason for those two limitations you dislike: they are the same as the limitations on logical pathnames. So if you wish to be able to find an ASDF system relative to a directory designated by a logical pathname, those limitations are pretty much forced on you. All right. Help me sort this out. I actually want to make this work. I have a directory Foo_Bar and I want to create a .asd file (in any directory which is known to ASDF) which describes the stuff within Foo_Bar. From what I have sorted out so far, that .asd file must be named something other than "Foo_Bar.asd" and the system it describes must be named something other than "Foo_Bar". The ideal outcome is that a user would say (asdf:load-system "Foo_Bar") and that the system which describes the stuff in "Foo_Bar" would get loaded. How close can I get to that, and what do I have to do, involving logical pathnames if need be, in order to get it? I am thinking I need to do something like (I emphasize "like", as I am aware that the following code doesn't work): (setf (logical-pathname-translations "abc") ("**;foobar.asd" "Foo_Bar.asd")) and that must be executed before the system is defined (defsystem "foobar" ...) and before the user calls ASDF:LOAD-SYSTEM. (asdf:load-system "Foo_Bar") Can this work, at all? As I was saying, I actually want this (or anything) to work. I don't have any point to make except to get something working. Thanks for your help. Robert

4 13

Re: Working around nonconforming names in system definitions
by Robert Dodier 16 Sep '24

16 Sep '24

On Sun, Sep 15, 2024 at 7:29 PM phoebe Goldman <phoebe(a)goldman-tribe.org> wrote: > if you're newly adding system definitions, it seems like you must be at liberty to choose whatever names you want for those systems, right? Yes, and I might end up there, but I'd rather keep the name of the system the same as the name of the project, because it's going to cause unnecessary confusion to try to ensure that users refer to the "right" name. Maybe I can encapsulate some heuristics so that nonconforming names get mapped into the corresponding conforming name or names, so that users can refer to the original, nonconforming name ... That's not foolproof, so, again, I'd rather not if I don't have to. Robert

2 1

Re: Working around nonconforming names in system definitions
by Robert P. Goldman 16 Sep '24

16 Sep '24

Hi Robert, I’m away from my desk, so can’t check, but If I recall correctly there are name canonicalization functions that downcase. I don’t know if these are only applied when a system designator is a symbol or to strings as well. The underscores would be an issue if the system name was merged with a logical pathname, if a logical pathname was part of the ASDF configuration. In that case, some lisp implementations would reject the resulting pathname as illegal. Some might be more permissive. I have had experiences with path names that work on allegro and not on SBCL. CL-JSON replaces underscores with 2 hyphens, perhaps you could do something similar and double letters to indicate caps. You could provide some translation of system names to file names like that if you wanted to permit non standard system names. Of course, you would have to ensure that anyone using one of these systems would also have loaded the extension that found it properly. Honestly, this seems a lot more trouble than it’s worth: the name restrictions aren’t that onerous. On Sep 14, 2024 at 23:50:11 CDT, Robert Dodier <robert.dodier(a)gmail.com> wrote: On Sat, Sep 14, 2024 at 8:45 PM Faré <fahree(a)gmail.com> wrote: > If you're using logical pathnames, you're asking for trouble > (especially on SBCL). I wasted weeks of my life on logical pathnames, > and STRONGLY recommend that no one should ever use them anymore under > any circumstances. Just to be clear, I don't *want* to use logical pathnames, but I will if it provides a solution to the problem. > If you're not, there should be no problem having a system called > Foo_Bar, is there? > > Just make sure you use a (defsystem |Foo_Bar| ...) with the same case, no? I dunno, that doesn't seem to work ... I tried this in SBCL which has ASDF 3.3.something bundled. (asdf:load-system "Foo_Bar") provokes a warning: "WARNING: System definition file #P"/home/dodier/tmp/Foo_Bar.asd" contains definition for system "foo_bar". Please only define "Foo_Bar" and secondary systems with a name starting with "Foo_Bar/" (e.g. "Foo_Bar/test") in that file. and an error: Component "Foo_Bar" not found Anyway, the documentation says (https://asdf.common-lisp.dev/asdf.html#rule_002dsystem_002ddesignator) that "Foo_Bar" is an unacceptable name since it has both an underscore and uppercase letters. I find that (defsystem baz_quux ...) in a file named "baz_quux.asd" is loaded successfully without complaint -- I'm surprised that works; I guess the no underscores rule is not actually enforced? I guess the stated rules about simple component names are not actually enforced by the DEFSYSTEM macro itself ... the do-nothing system * (asdf:defsystem "Blurf_Mumble") * (asdf:operate 'asdf:load-op *) succeeds. I guess the stuff about string case only comes into play when the file system gets involved? FWIW & all the best. Robert

2 1

Working around nonconforming names in system definitions
by Robert Dodier 14 Sep '24

14 Sep '24

I am interested in automatically generating ASDF system definitions for projects which lack them. These projects have names, both the project name and the names of files or directories within them, which might not conform to ASDF's requirements for names. In particular, project names containing uppercase letters and underscores.These are other peoples' projects, so I'm not at liberty to rename them to conform to ASDF. RANT. The limitations that ASDF puts on names are just bananas. I have to wonder who thought it would be a good idea to force people to work around them. I know this has been discussed before, without any resolution, so I won't pursue it any further. There's no need to explain about logical pathnames, I know already. Thanks for understanding. /RANT For a project which has a conforming name, say foobar, I can create foobar.asd and say that the name of the system is "foobar". That's great. Let's say the name of the project is Foo_Bar. What are some options for naming the .asd system and the system which it contains, ideally so that when a user tries (asdf:load-system "Foo_Bar") they get the appropriate system? Perhaps that's impossible; if so I'd just like to know. I've tried some different things and haven't come up with anything that seems to work. A web search finds a few aging and/or obsolete references, and in any event, nothing specifically directed towards solving this problem. Thanks for any insights. Robert Dodier

2 1

Return value of UIOP:CALL-WITH-CURRENT-DIRECTORY
by Kevin Zheng 02 Sep '24

02 Sep '24

Hi asdf developers, I recently was writing some code with UIOP:CALL-WITH-CURRENT-DIRECTORY and came across something slightly unexpected. In particular, running CALL-WITH-CURRENT-DIRECTORY or WITH-CURRENT-DIRECTORY forms evaluates to NIL, whereas one may expect that the form evaluates to the return value of THUNK or the macro body, respectively. In my program, I'm trying to do something like: (uiop:with-current-directory (dir) (uiop:run-program (list "some" "program") :output :string)) Which doesn't quite do what I want it to do. I believe this has to do with the definition, which ends in a trailing (chdir dir) form to get out of the directory which unfortunately returns NIL instead of the value from evaluating THUNK. So my question is: would it be acceptable if a future version of UIOP changes call-with-current-directory to return the value of evaluating THUNK? Thanks, Kevin

3 4