[asdf-devel] ASDF 2.016 breaks ABCL translations for jar files
Stellian's [normalization to ANSI semantics of COMPILE-FILE-PATHNAME*][1] has the unfortunate effect of breaking ABCL's translation of systems packaged in jar files. [1]: http://common-lisp.net/gitweb?p=projects/asdf/asdf.git;a=commit;h=ed4cd32932... I have applied the subsequent patch to the asdf-2.016 packaged with ABCL to restore our needed functionality but am unhappy at the necessity of introducing a runtime implementation conditional, as sure a sign as anything that I'm "doing it wrong," and can't really request that ASDF patch itself in this manner. Maybe if I explain things as I have analyzed them, someone else can suggest a better way forward. Recall that ABCL uses the presence of a cons in the DEVICE component to indicate that a Pathname represents an entry in a jar archive. A pathname located within a jar archive is not a writable location, so we introduced ABCL specific ASDF output translation rules which translate such a source location to an appropriately differentiated output location in the ASDF cache hierarchy. This results in ASDF making the following kind of call in preparation of the output file location it will pass to COMPILE-FILE: (compile-file-pathname* <path with a DEVICE> :output-file <path without a DEVICE>) Stellian's patch correctly fixes behavior of COMPILE-FILE-PATHNAME* to use pathname components in input-file if none are specified in output-file. For ABCL systems in a jar, this results in returning an output location within the jar file, which being a non-valid output location because it is not writable causes ASDF to error out in compiling the system. If there were some mechanism to indicate that the invocation of APPLY-OUTPUT-TRANSLATIONS had really "finished" so that no further processing was necessary, we could possibly plausibly short-circuit this call to COMPILE-FILE-PATHAME*. I suppose my recursive invocation of APPLY-OUTPUT-TRANSLATIONS in the ABCL-specific TRANSLATE-JAR-PATHNAME adds the missing TYPE to the pathname which other code paths don't have at this point, which is why the call to COMPILE-FILE-PATHNAME* is necessary. ANSI allows the addition of implementation specific arguments to COMPILE-FILE-PATHNAME, so we could maybe add a :strip-device for ABCL but this seems even less elegant. Any suggestions? @@ -3523,7 +3525,16 @@ --- a/asdf.lisp Thu Jun 09 06:03:20 2011 +0000 +++ b/asdf.lisp Thu Jun 09 15:00:45 2011 +0200 (defun* compile-file-pathname* (input-file &rest keys &key output-file &allow-other-keys) (if (absolute-pathname-p output-file) - (apply 'compile-file-pathname (lispize-pathname input-file) keys) + ;;; If the default ABCL rules for translating from a jar path to + ;;; a non-jar path have been affected, no further computation of + ;;; the output location is necessary. + (if (and (find :abcl *features*) + (pathname-device input-file) ; input-file is in a jar + (not (pathname-device output-file)) ; output-file is not in a jar + (equal (pathname-type input-file) "lisp") + (equal (pathname-type output-file) "abcl")) + output-file + (apply 'compile-file-pathname (lispize-pathname input-file) keys)) (apply-output-translations (apply 'compile-file-pathname (truenamize (lispize-pathname input-file)) -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 9 June 2011 09:35, Mark Evenson <evenson@panix.com> wrote:
Stellian's [normalization to ANSI semantics of COMPILE-FILE-PATHNAME*][1] has the unfortunate effect of breaking ABCL's translation of systems packaged in jar files.
Oops. This wasn't caught by my tests. My sincere apologies. Can you write a simple test to be added to test/asdf-pathnames.script, and/or send me an example such that I can reproduce the bug at home? Maybe just a trace output of the relevant functions: (trace translate-pathname* and compile-file-pathname* apply-output-translations) (add any function there necessary to show where it breaks). Note that when I (describe #p"jar:file:///foo/bar.jar!/baz/quux.lisp") I get: #P"jar:file:/foo/bar.jar!/baz/quux.lisp" is an object of type EXTENSIONS:JAR-PATHNAME: HOST NIL DEVICE (#P"/foo/bar.jar") DIRECTORY (:ABSOLUTE "baz") NAME "quux" TYPE "lisp" VERSION NIL I believe that a pathname instead of a namestring makes that non-conformant. Sigh. Note that if requires you could keep a cache of the pathname in addition (or underlying store?) to any namestring in the device.
I have applied the subsequent patch to the asdf-2.016 packaged with ABCL to restore our needed functionality but am unhappy at the necessity of introducing a runtime implementation conditional, as sure a sign as anything that I'm "doing it wrong," and can't really request that ASDF patch itself in this manner. Maybe if I explain things as I have analyzed them, someone else can suggest a better way forward.
Yes, something is wrong.
Recall that ABCL uses the presence of a cons in the DEVICE component to indicate that a Pathname represents an entry in a jar archive. A pathname located within a jar archive is not a writable location, so we introduced ABCL specific ASDF output translation rules which translate such a source location to an appropriately differentiated output location in the ASDF cache hierarchy. This results in ASDF making the following kind of call in preparation of the output file location it will pass to COMPILE-FILE:
(compile-file-pathname* <path with a DEVICE> :output-file <path without a DEVICE>)
What output-file do you get and where is it from? Is any path there not absolute, and if so why?
If there were some mechanism to indicate that the invocation of APPLY-OUTPUT-TRANSLATIONS had really "finished" so that no further processing was necessary, we could possibly plausibly short-circuit this call to COMPILE-FILE-PATHAME*. Well, output-files accepts a secondary value of T to specify that no more translations are needed. I don't understand yet what to change into ASDF, but either we can somehow reuse that mechanism, or we can provide something similar inside apply-output-translations, or somehow be more clever in translate-jar-pathname.
I suppose my recursive invocation of APPLY-OUTPUT-TRANSLATIONS in the ABCL-specific TRANSLATE-JAR-PATHNAME adds the missing TYPE to the pathname which other code paths don't have at this point, which is why the call to COMPILE-FILE-PATHNAME* is necessary.
My head hurts. What type is added to what where? Can you give examples of functions giving the "wrong" results?
ANSI allows the addition of implementation specific arguments to COMPILE-FILE-PATHNAME, so we could maybe add a :strip-device for ABCL but this seems even less elegant.
Probably. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org First they ignore you. Then they laugh at you. Then they fight you. Then you win. — Gandhi
On 6/9/11 16:37 , Faré wrote: […]
Can you write a simple test to be added to test/asdf-pathnames.script,
I'll work on that. Can I just create a new top-level form or do you prefer I wedge it into TEST-COMPONENT-PATHNAMES with an #+abcl/#-abcl conditional?
and/or send me an example such that I can reproduce the bug at home? Maybe just a trace output of the relevant functions: (trace translate-pathname* and compile-file-pathname* apply-output-translations) (add any function there necessary to show where it breaks).
A trace for the JSS system packaged in abcl-contrib.jar: CL-USER> (asdf:compile-system :jss :force t) 0: (ASDF:APPLY-OUTPUT-TRANSLATIONS #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl") 1: (ASDF::TRANSLATE-PATHNAME* #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl" #P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION ASDF::TRANSLATE-JAR-PATHNAME {66EF7D74}> NIL #P"jar:file:/**/*.jar!/**/*.*") 2: (ASDF:APPLY-OUTPUT-TRANSLATIONS #P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl") 3: (ASDF::TRANSLATE-PATHNAME* #P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" #P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" NIL #P"/___jar___file___root___/**/*.*") 3: TRANSLATE-PATHNAME* returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 2: APPLY-OUTPUT-TRANSLATIONS returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 1: TRANSLATE-PATHNAME* returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 0: APPLY-OUTPUT-TRANSLATIONS returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 0: (ASDF:APPLY-OUTPUT-TRANSLATIONS #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl") 1: (ASDF::TRANSLATE-PATHNAME* #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.abcl" #P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION ASDF::TRANSLATE-JAR-PATHNAME {66EF7D74}> NIL #P"jar:file:/**/*.jar!/**/*.*") 2: (ASDF:APPLY-OUTPUT-TRANSLATIONS #P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl") 3: (ASDF::TRANSLATE-PATHNAME* #P"/___jar___file___root___/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" #P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" NIL #P"/___jar___file___root___/**/*.*") 3: TRANSLATE-PATHNAME* returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 2: APPLY-OUTPUT-TRANSLATIONS returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 1: TRANSLATE-PATHNAME* returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 0: APPLY-OUTPUT-TRANSLATIONS returned #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" 0: (ASDF:COMPILE-FILE-PATHNAME* #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.lisp" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl") 0: COMPILE-FILE-PATHNAME* returned #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" And the subsequent COMPILE-FILE fails as it can't write to the jar file returned by COMPILE-FILE-PATHNAME*.
Note that when I (describe #p"jar:file:///foo/bar.jar!/baz/quux.lisp") I get: #P"jar:file:/foo/bar.jar!/baz/quux.lisp" is an object of type EXTENSIONS:JAR-PATHNAME: HOST NIL DEVICE (#P"/foo/bar.jar") DIRECTORY (:ABSOLUTE "baz") NAME "quux" TYPE "lisp" VERSION NIL
I believe that a pathname instead of a namestring makes that non-conformant. Sigh.
You mean the list with pathname in the device field? Yes, ABCL is non-conformant in this manner in order to support addressing entries in jar files.
Note that if requires you could keep a cache of the pathname in addition (or underlying store?) to any namestring in the device.
I don't think I understand this suggestion. […]
Recall that ABCL uses the presence of a cons in the DEVICE component to indicate that a Pathname represents an entry in a jar archive. A pathname located within a jar archive is not a writable location, so we introduced ABCL specific ASDF output translation rules which translate such a source location to an appropriately differentiated output location in the ASDF cache hierarchy. This results in ASDF making the following kind of call in preparation of the output file location it will pass to COMPILE-FILE:
(compile-file-pathname* <path with a DEVICE> :output-file<path without a DEVICE>)
What output-file do you get and where is it from?
0: (ASDF:COMPILE-FILE-PATHNAME* #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.lisp" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl") 0: COMPILE-FILE-PATHNAME* returned #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" The arg to :output-file has come from the OUTPUT-FILES :around method passing in "jar:file:…" and then hitting the TRANSLATE-JAR-PATHNAME rule.
Is any path there not absolute, and if so why?
No, it is always absolute as TRANSLATE-JAR-PATHNAME returns the fully translated pathname.
If there were some mechanism to indicate that the invocation of APPLY-OUTPUT-TRANSLATIONS had really "finished" so that no further processing was necessary, we could possibly plausibly short-circuit this call to COMPILE-FILE-PATHAME*. Well, output-files accepts a secondary value of T to specify that no more translations are needed.
I missed that: I'll look into this further when I get a chance.
I don't understand yet what to change into ASDF, but either we can somehow reuse that mechanism, or we can provide something similar inside apply-output-translations, or somehow be more clever in translate-jar-pathname.
I suppose my recursive invocation of APPLY-OUTPUT-TRANSLATIONS in the ABCL-specific TRANSLATE-JAR-PATHNAME adds the missing TYPE to the pathname which other code paths don't have at this point, which is why the call to COMPILE-FILE-PATHNAME* is necessary.
My head hurts. What type is added to what where?
My understanding is that ASDF uses the COMPILE-FILE-PATHNAME* to supply the TYPE of fasl, which is implementation dependent. Otherwise I didn't have an explanation as to why we run COMPILE-FILE-PATHNAME* before COMPILE-FILE which is going to return the same results, as in my testing the value that :output-file has at this point is already the final value anyways.
Can you give examples of functions giving the "wrong" results?
No. As I stated, it is correct that COMPILE-FILE-PATHNAME* merges in the device from input-file when the output-file arg has NIL for device. This is the expected behavior from the ANSI perspective, but what we are trying to get away from in ABCL's case. Thanks for thinking with me on this. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Note that when I (describe #p"jar:file:///foo/bar.jar!/baz/quux.lisp") I get: #P"jar:file:/foo/bar.jar!/baz/quux.lisp" is an object of type EXTENSIONS:JAR-PATHNAME: HOST NIL DEVICE (#P"/foo/bar.jar") DIRECTORY (:ABSOLUTE "baz") NAME "quux" TYPE "lisp" VERSION NIL
I believe that a pathname instead of a namestring makes that non-conformant. Sigh.
You mean the list with pathname in the device field? Yes, ABCL is non-conformant in this manner in order to support addressing entries in jar files.
Note that if requires you could keep a cache of the pathname in addition (or underlying store?) to any namestring in the device.
I don't think I understand this suggestion.
You could be conformant if the device component were ("/foo/bar.jar") instead of (#P"/foo/bar.jar"). Internally, you could even store the pre-parsed pathname in a slot rather than the namestring, as long as the pathname-device accessor returns a namestring and as equal does the right thing.
What output-file do you get and where is it from?
0: (ASDF:COMPILE-FILE-PATHNAME* #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/jss/packages.lisp" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl" :OUTPUT-FILE #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl") 0: COMPILE-FILE-PATHNAME* returned #P"jar:file:/Users/evenson/work/abcl/dist/abcl-contrib.jar!/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/Users/evenson/work/abcl/dist/abcl-contrib.jar/jss/packages.abcl"
At this point can you trace COMPILE-FILE-PATHNAME, too? Since the output-file satisfies absolute-pathname-p, the branch taken by COMPILE-FILE-PATHNAME* should be to just call COMPILE-FILE-PATHNAME, at which point I expect it to *not* merge the device of the source file. OH I GET IT! (I think.) The bug is that (1) COMPILE-FILE-PATHNAME uses MERGE-PATHNAMES instead of ASDF:MERGE-PATHNAMES* (as you would, I assume). (2) Your absolute pathnames have a host and device of NIL, instead of a non-nil host device of say "FILE" and "LOCALHOST" or say "" and "". Sigh. I would have suggested fixing things by having ABCL's pathname scheme not use NIL for HOST and DEVICE in such cases, but who knows what other interesting (positive or negative) consequences that might or might not have. It could be a big win or a big headache. If you can't fix ABCL this way, if such "fix" actually breaks other stuff, or if you don't dare try, I suppose I need to reimplement in ASDF what CL implementations *should* have done, i.e. using MERGE-PATHNAMES* instead of MERGE-PATHNAMES in handling the OUTPUT-FILE.
My understanding is that ASDF uses the COMPILE-FILE-PATHNAME* to supply the TYPE of fasl, which is implementation dependent. Otherwise I didn't have an explanation as to why we run COMPILE-FILE-PATHNAME* before COMPILE-FILE which is going to return the same results, as in my testing the value that :output-file has at this point is already the final value anyways.
The reason why we pass :output-file is that we want to preserve the API of compile-file-pathname, to which it is valid to pass arguments such as :output-file "/tmp/" or :output-file "foo" (without type) or :output-file (make-pathname :name nil :type "a") (without name). If feel like more headaches are on their way... What is the PROPER output-file defaulting algorithm for CL, and what SHOULD it be for ASDF? Stelian -- I demand your opinion on that one, since you started it all.
Can you give examples of functions giving the "wrong" results?
No. As I stated, it is correct that COMPILE-FILE-PATHNAME* merges in the device from input-file when the output-file arg has NIL for device. This is the expected behavior from the ANSI perspective, but what we are trying to get away from in ABCL's case.
Thanks for thinking with me on this.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you make people think they're thinking, they'll love you; but if you really make them think, they'll hate you. — Don Marquis
Thanks for thinking with me on this.
OK. I believe the following definition might make each of Stelian, you and I happy. Can you try it? (defun* compile-file-pathname* (input-file &rest keys &key output-file &allow-other-keys) (if (absolute-pathname-p output-file) (apply 'compile-file-pathname (make-pathname :host nil :device nil :defaults (lispize-pathname input-file)) keys) (apply-output-translations (apply 'compile-file-pathname (truenamize (lispize-pathname input-file)) keys)))) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org All programmers are playwrights and all computers are lousy actors.
On 6/9/11 7:06 PM, Faré wrote:
Thanks for thinking with me on this.
OK. I believe the following definition might make each of Stelian, you and I happy. Can you try it?
(defun* compile-file-pathname* (input-file&rest keys&key output-file&allow-other-keys) (if (absolute-pathname-p output-file) (apply 'compile-file-pathname (make-pathname :host nil :device nil :defaults (lispize-pathname input-file)) keys) (apply-output-translations (apply 'compile-file-pathname (truenamize (lispize-pathname input-file)) keys))))
Yes, this works fine with ABCL. But for implementations that need a device (like Windows using device for the drive letter), won't this result in stripping the device out of all output-file locations? As I mentioned in the previous message, I think the better fix is to have TRANSLATE-JAR-PATHNAME do something like the following (untested): index a8339f7..0395f56 100755 --- a/asdf.lisp +++ b/asdf.lisp @@ -3569,12 +3569,14 @@ effectively disabling the output translation facility." (root (format nil "/___jar___file___root___/~@[~A/~]" (and (find :windows *features*) (pathname-device p))))) - (apply-output-translations - (merge-pathnames* - (relativize-pathname-directory source) - (merge-pathnames* - (relativize-pathname-directory (ensure-directory-pathname p)) - root))))) + (make-pathname :defaults + (apply-output-translations + (merge-pathnames* + (relativize-pathname-directory source) + (merge-pathnames* + (relativize-pathname-directory (ensure-directory-pathname p)) + root))) + :device :unspecific))) ;;;; ----------------------------------------------------------------- ;;;; Compatibility mode for ASDF-Binary-Locations -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/9/11 7:42 PM, Mark Evenson wrote: […]
As I mentioned in the previous message, I think the better fix is to have TRANSLATE-JAR-PATHNAME do something like the following (untested):
The patch is bogus as a direct fix. Something (TRUENAMIZE?) has stripped the :unspecific device component by the time it gets to COMPILE-FILE-PATHNAME*. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/9/11 6:44 PM, Faré wrote:
Note that when I (describe #p"jar:file:///foo/bar.jar!/baz/quux.lisp") I get: #P"jar:file:/foo/bar.jar!/baz/quux.lisp" is an object of type EXTENSIONS:JAR-PATHNAME: HOST NIL DEVICE (#P"/foo/bar.jar") DIRECTORY (:ABSOLUTE "baz") NAME "quux" TYPE "lisp" VERSION NIL
I believe that a pathname instead of a namestring makes that non-conformant. Sigh.
You mean the list with pathname in the device field? Yes, ABCL is non-conformant in this manner in order to support addressing entries in jar files.
Note that if requires you could keep a cache of the pathname in addition (or underlying store?) to any namestring in the device.
On re-reading CLHS, I retract my admission that ABCL is non-conformant. From the CLHS glossary a "valid pathname device [is a] a string, nil, :unspecific, or some other object defined by the implementation to be a valid pathname device." ABCL is using the "some other object" ability here. The "might" description in CLHS 19.2.2.4.2 offers guidance as to the allowed types, but doesn't put a firm requirement. Or is this understanding incorrect somehow? Note I am trying to use "conformant" here in the manner prescribed by CLHS 1.5, not in the informal sense of "what a friendly CL implementation" should do. The two main design reasons for allowing a list of pathnames in our device component: they allow pathnames representing URI locations so our jars may be loaded over the network, and they allow us to reference jar archives stored within archives which is important for ABCL fasls. We could plausibly convert the device pathname to namestrings, as every URI has an suitably encoded string representation, but without the ability to use a cons in the device slot we would lose the ability to refer to a nested entry.
You could be conformant if the device component were ("/foo/bar.jar") instead of (#P"/foo/bar.jar").
But have the device component be a cons violates the first sentence of CLHS 19.2.2.4.2 that you are presumably referencing. Or how do I misunderstand? […]
At this point can you trace COMPILE-FILE-PATHNAME, too? Since the output-file satisfies absolute-pathname-p, the branch taken by COMPILE-FILE-PATHNAME* should be to just call COMPILE-FILE-PATHNAME, at which point I expect it to *not* merge the device of the source file. OH I GET IT! (I think.) The bug is that (1) COMPILE-FILE-PATHNAME uses MERGE-PATHNAMES instead of ASDF:MERGE-PATHNAMES* (as you would, I assume). (2) Your absolute pathnames have a host and device of NIL, instead of a non-nil host device of say "FILE" and "LOCALHOST" or say "" and "".
Yes, it is our use of NIL here that is causing problems. What I was going to suggest is that the TRANSLATE-JAR-PATHNAME function return a pathname with an :UNSPECIFIC device component. This should allow the COMPILE-FILE-PATHNAME merge algorithm to do what we want, but I see you have incorporated this into your next message, which I will reply to.
Sigh. I would have suggested fixing things by having ABCL's pathname scheme not use NIL for HOST and DEVICE in such cases, but who knows what other interesting (positive or negative) consequences that might or might not have. It could be a big win or a big headache.
I think we should probably insert :UNSPECIFIC for more cases than where we do from MAKE-PATHNAME. […] -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On re-reading CLHS, I retract my admission that ABCL is non-conformant. From the CLHS glossary a "valid pathname device [is a] a string, nil, :unspecific, or some other object defined by the implementation to be a valid pathname device." ABCL is using the "some other object" ability here. The "might" description in CLHS 19.2.2.4.2 offers guidance as to the allowed types, but doesn't put a firm requirement. Or is this understanding incorrect somehow? Note I am trying to use "conformant" here in the manner prescribed by CLHS 1.5, not in the informal sense of "what a friendly CL implementation" should do.
The CLHS sometimes contains mutually contradictory statements, or statements that have subtle interactions or reinforcing meaning. I seem to remember from previous discussions with SBCL language lawyers that there were somewhere in the CLHS restrictions in addition to these, but at this point it's third hand remembrance of second hand knowledge. Sigh.
But have the device component be a cons violates the first sentence of CLHS 19.2.2.4.2 that you are presumably referencing. Or how do I misunderstand? Indeed. This paragraph suggests the component can be a string but not a cons except for wild pathnames.
OH I GET IT! (I think.) The bug is that (1) COMPILE-FILE-PATHNAME uses MERGE-PATHNAMES instead of ASDF:MERGE-PATHNAMES* (as you would, I assume). (2) Your absolute pathnames have a host and device of NIL, instead of a non-nil host device of say "FILE" and "LOCALHOST" or say "" and "".
Yes, it is our use of NIL here that is causing problems.
Good. At least, we understand the issue if not the solution.
What I was going to suggest is that the TRANSLATE-JAR-PATHNAME function return a pathname with an :UNSPECIFIC device component. This should allow the COMPILE-FILE-PATHNAME merge algorithm to do what we want, but I see you have incorporated this into your next message, which I will reply to.
That could be a solution. Another solution would be for ASDF to somewhat distrust the implementation, and do its own thing, using merge-pathnames* where the implementation would likely use the error-prone merge-pathnames. How does THIS work for you? (defun* compile-file-pathname* (input-file &rest keys &key output-file &allow-other-keys) (flet ((cfp (i o) ; what compile-file-pathname should actually do (let ((fasl (pathname-type (apply 'compile-file-pathname "foo.lisp" keys)))) (merge-pathnames* ; NB: the caller must (merge-pathnames* i) if o not absolute. o (make-pathname :type fasl :defaults i))))) (if (absolute-pathname-p output-file) (cfp input-file output-file) ; if user specifies absolute output, use it (apply-output-translations (cfp (truenamize input-file) output-file))))) —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org "If men are good, you don't need government; if men are evil or ambivalent, you don't dare have one." — Robert LeFevre
Please try 2.016.1 and see if it satisfies you. Pathnames are a big FAIL of CL. —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Slogans rarely convince the unconvinced. However, they do rally the troops already on your side. — John McCarthy
On 6/10/11 00:57 , Faré wrote: On 6/10/11 00:57 , Faré wrote:
Please try 2.016.1 and see if it satisfies you.
Yes, that works for [ABCL and I have pushed][1]. [1]: http://trac.common-lisp.net/armedbear/changeset/13319
Pathnames are a big FAIL of CL.
So thinks Stellian as well, so IOLIB tries to get away from. I admire CL Pathnames given the different types of abstractions that early 1980s filesystems had to normalize, but certainly that is scant consolation for dealing with them in the 2010s. Certainly it seems that the behavior of COMPILE-FILE-PATHNAME not matching expectations of MERGE-PATHNAMES is a CL FAIL. With a little more specification on how to handle the implementation-specific parts, I think things would be better off. At this point, the code of ASDF2 for handling these differences is probably the most advanced--and widely tested--stab at such abstraction. It some point in the future one might consider breaking it out into a separate library, as it would undoubtedly have wide utility outside of ASDF. Thanks again for the help, Mark -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/10/11 00:57 , Faré wrote: On 6/10/11 00:57 , Faré wrote:
Please try 2.016.1 and see if it satisfies you.
Yes, that works for [ABCL and I have pushed][1]. [1]: http://trac.common-lisp.net/armedbear/changeset/13319
Pathnames are a big FAIL of CL.
So thinks Stellian as well, so IOLIB tries to get away from. I admire CL Pathnames given the different types of abstractions that early 1980s filesystems had to normalize, but certainly that is scant consolation for dealing with them in the 2010s. Certainly it seems that the behavior of COMPILE-FILE-PATHNAME not matching expectations of MERGE-PATHNAMES is a CL FAIL. With a little more specification on how to handle the implementation-specific parts, I think things would be better off. At this point, the code of ASDF2 for handling these differences is probably the most advanced--and widely tested--stab at such abstraction. It some point in the future one might consider breaking it out into a separate library, as it would undoubtedly have wide utility outside of ASDF. Thanks again for the help, Mark -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
participants (2)
-
Faré -
Mark Evenson