[armedbear-devel] Spaces in pathnames (jar:file:/x/y/z/a space/...)
Dear ABCL developers, While ABCL can handle spaces (& others) in file pathnames just fine, a space in jar file pathnames still doesn't work. I have confirmed this on Linux, Mac and Windows. An example using Cygwin, $ java -jar abcl.jar Armed Bear Common Lisp 0.24.0 Java 1.6.0_20 Sun Microsystems Inc. Java HotSpot(TM) Client VM Low-level initialization completed in 0.625 seconds. Startup completed in 4.438 seconds. Type ":help" for a list of available commands. CL-USER(1): (load "jar:file:e:/My Dropbox/temp/a space/hello.jar!/hello.abcl") #<THREAD "interpreter" {9AC0F5}>: Debugger invoked on condition of type FILE-ERROR Failed to create URI from 'e:/My Dropbox/temp/a space/hello.jar': Illegal character in opaque part at index 10: file:e:/My Dropbox/temp/a space/hello.jar Restarts: 0: TOP-LEVEL Return to top level. (and yeah, this particular path already has a space character in "My Dropbox" along with my original intended "a space" folder) By the way, thanks for the :wildcard directory support (I've only just tested that, it works for me now!). Yong.
On 2/15/11 10:37 PM, Theam Yong Chew wrote:
Dear ABCL developers,
While ABCL can handle spaces (& others) in file pathnames just fine, a space in jar file pathnames still doesn't work. I have confirmed this on Linux, Mac and Windows. An example using Cygwin,
[…] When a Pathname starts using URI scheme identifiers like 'jar', the strings are then required to obey URI encoding rules. So try replacing #\Space characters with the string "%20". #<THREAD "interpreter" {9AC0F5}>: Debugger invoked on condition of
type FILE-ERROR Failed to create URI from 'e:/My Dropbox/temp/a space/hello.jar': Illegal character in opaque part at index 10: file:e:/My Dropbox/temp/a space/hello.jar Restarts: 0: TOP-LEVEL Return to top level.
[…] Try with the URI escaped version, i.e. "jar:file:e/My%20Dropbox/temp/a%20spac/hello.jar!/hello.abcl", instead. A quick glance through the [ABCL JAR-PATHNAME specification][1] shows that it doesn't explicity mention this design. Implicitly, since a JAR-PATHNAME is a URI-PATHNAME it has to obey the [URI-PATHNAME rules][2], but we could certainly make things clearer. [1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname... [2]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname... We tried to make the Lisp reader friendly is this regard by DWIMing a String inputs into Pathname, but if you find an unambiguous case that could be improved to be friendly, we would solicit a specification or patches. Enjoy the bear! -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 2/16/11, Mark Evenson <evenson@panix.com> wrote:
On 2/15/11 10:37 PM, Theam Yong Chew wrote:
Dear ABCL developers,
While ABCL can handle spaces (& others) in file pathnames just fine, a space in jar file pathnames still doesn't work. I have confirmed this on Linux, Mac and Windows. An example using Cygwin,
[…]
When a Pathname starts using URI scheme identifiers like 'jar', the strings are then required to obey URI encoding rules. So try replacing #\Space characters with the string "%20".
#<THREAD "interpreter" {9AC0F5}>: Debugger invoked on condition of
type FILE-ERROR Failed to create URI from 'e:/My Dropbox/temp/a space/hello.jar': Illegal character in opaque part at index 10: file:e:/My Dropbox/temp/a space/hello.jar Restarts: 0: TOP-LEVEL Return to top level.
[…]
Sorry Mark, I was going to add some more clarification to my original email, but you were too quick to reply! The above error message was actually slightly different from how I originally stumbled on the problem, which looked somewhat like this: [1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._' Restarts: 0: ABORT Return to debug level 1. In fact, originally, I encountered this from the command line like so: java -cp abcl.jar:.... org.armedbear.lisp.Main --load 'jar:file:main.jar!/main.abcl' So there weren't even any obvious spaces, since I only provided relative paths. So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
Try with the URI escaped version, i.e. "jar:file:e/My%20Dropbox/temp/a%20spac/hello.jar!/hello.abcl", instead.
Yes, the original error message I had showed just such an escape sequence, but I wasn't sure what to make of it! For simple folks like me, if we load "/home/..." as a file, it's not a URL (and does not require escaping), whereas the jar: path uses URL 'encoding'. Would that be right? (Simplifying of course)
A quick glance through the [ABCL JAR-PATHNAME specification][1] shows that it doesn't explicity mention this design. Implicitly, since a JAR-PATHNAME is a URI-PATHNAME it has to obey the [URI-PATHNAME rules][2], but we could certainly make things clearer.
[1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname...
[2]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname...
We tried to make the Lisp reader friendly is this regard by DWIMing a String inputs into Pathname, but if you find an unambiguous case that could be improved to be friendly, we would solicit a specification or patches.
Not sure what you mean here, but just being aware of the difference between jar and normal file-paths (vis a vis URL vs "plain strings") would be more than sufficient I feel. I vaguely knew about the URL stuff, but must have discounted it due to heavy use of the normal pathname form in ABCL. Thanks, and sorry for the confusion. I hope this error is clear enough this time, and is reproducible. Yong.
Enjoy the bear!
Yes, very much so!
On 2/16/11 14:34 , Theam Yong Chew wrote: […]
Sorry Mark, I was going to add some more clarification to my original email, but you were too quick to reply! The above error message was actually slightly different from how I originally stumbled on the problem, which looked somewhat like this:
[1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._' Restarts: 0: ABORT Return to debug level 1.
[…]
So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
It looks like you might have encountered a more serious problem with our classloader. Can you mail me a copy of the failing jar file? You might try recompiling all the FASLs if you have upgraded the 'abcl.jar' you use recently, but this probably won't help. When I get the time, I'll try to examine your failure, synthesize a failing test, and we'll see from there. Finding the time will probably happen over the next couple days -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Wed, Feb 16, 2011 at 5:16 PM, Mark Evenson <evenson@panix.com> wrote:
On 2/16/11 14:34 , Theam Yong Chew wrote: […]
Sorry Mark, I was going to add some more clarification to my original email, but you were too quick to reply! The above error message was actually slightly different from how I originally stumbled on the problem, which looked somewhat like this:
[1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._' Restarts: 0: ABORT Return to debug level 1.
[…]
So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
It looks like you might have encountered a more serious problem with our classloader. Can you mail me a copy of the failing jar file?
Notice the %2520 in the failing path: "in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._'" It looks like something is URI-encoding the already-URI-encoded pathname, replacing the % in %20 with %25 (thus obtaining %2520). I don't think this has anything to do with the classloader, but I may be wrong here. Generally, we might need to distinguish between URI-escaping Pathname constructors (for strings coming from the outside world) and non-URI-escaping constructors for use by ABCL internals. Bye, Alessio
On 2/17/11, Alessio Stalla <alessiostalla@gmail.com> wrote:
On Wed, Feb 16, 2011 at 5:16 PM, Mark Evenson <evenson@panix.com> wrote:
On 2/16/11 14:34 , Theam Yong Chew wrote: […]
Sorry Mark, I was going to add some more clarification to my original email, but you were too quick to reply! The above error message was actually slightly different from how I originally stumbled on the problem, which looked somewhat like this:
[1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._' Restarts: 0: ABORT Return to debug level 1.
[…]
So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
It looks like you might have encountered a more serious problem with our classloader. Can you mail me a copy of the failing jar file?
Notice the %2520 in the failing path: "in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._'" It looks like something is URI-encoding the already-URI-encoded pathname, replacing the % in %20 with %25 (thus obtaining %2520). I don't think this has anything to do with the classloader, but I may be wrong here. Generally, we might need to distinguish between URI-escaping Pathname constructors (for strings coming from the outside world) and non-URI-escaping constructors for use by ABCL internals.
Bye, Alessio
Hi everyone, I had a quick look at the source over last week. I'm not completely confident if the attached patch fixes it, or if I've broken anything else... I'm also not sure if I'm running the unit tests properly, but at least the number of failures hasn't gone up! According to my understanding, truename already has some uriEncoding logic in it (which I didn't quite follow), so it seemed to me like the second uriEncode is the redundant culprit. Am I right? Yong.
On 10-Mar-2011 3:58 PM, "Theam Yong Chew" <senatorzergling@gmail.com> wrote:
On 2/17/11, Alessio Stalla <alessiostalla@gmail.com> wrote:
On Wed, Feb 16, 2011 at 5:16 PM, Mark Evenson <evenson@panix.com> wrote:
On 2/16/11 14:34 , Theam Yong Chew wrote: […]
Sorry Mark, I was going to add some more clarification to my original email, but you were too quick to reply! The above error message was actually slightly different from how I originally stumbled on the problem, which looked somewhat like this:
[1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in
'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._'
Restarts: 0: ABORT Return to debug level 1.
[…]
So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
It looks like you might have encountered a more serious problem with our classloader. Can you mail me a copy of the failing jar file?
Notice the %2520 in the failing path: "in
'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._'"
It looks like something is URI-encoding the already-URI-encoded pathname, replacing the % in %20 with %25 (thus obtaining %2520). I don't think this has anything to do with the classloader, but I may be wrong here. Generally, we might need to distinguish between URI-escaping Pathname constructors (for strings coming from the outside world) and non-URI-escaping constructors for use by ABCL internals.
Bye, Alessio
Hi everyone,
I had a quick look at the source over last week. I'm not completely confident if the attached patch fixes it, or if I've broken anything else... I'm also not sure if I'm running the unit tests properly, but at least the number of failures hasn't gone up!
According to my understanding, truename already has some uriEncoding logic in it (which I didn't quite follow), so it seemed to me like the second uriEncode is the redundant culprit. Am I right?
Yong.
Hi everyone, What's the effect of the latest Pathname.java work on this bug I reported a few months ago? The one line patched I included then seemed to fix the loading of jar files with spaces in their pathnames, but then fails when loading .abcl files with spaces in their pathnames... so it still wasn't right. I haven't been able to follow in detail (but I haven't looked too deeply) the pathname creation, defaulting of various fields and path escaping logic myself. Any ideas on the right thing to do? Yong.
On 6/4/11 6:43 AM, Theam Yong Chew wrote: […]
What's the effect of the latest Pathname.java work on this bug I reported a few months ago?
If you are referring to the changes culminating [r13302 as the latest work on Pathname.java][r13302], I wouldn't expect any change in the URI escaping behavior as this work only dealt with the version component. Ville had done some other work about a month ago, but I think he retracted his patch as it caused more problems than it solved. [r13302]: http://trac.common-lisp.net/armedbear/changeset/13302
The one line patched I included then seemed to fix the loading of jar files with spaces in their pathnames, but then fails when loading .abcl files with spaces in their pathnames... so it still wasn't right.
I haven't been able to follow in detail (but I haven't looked too deeply) the pathname creation, defaulting of various fields and path escaping logic myself. Any ideas on the right thing to do?
Initially, I couldn't reproduce your problems, so I didn't file a bug, apply your patch or investigate much further. From re-reading the conversation, it looks like I assumed that your initial misunderstanding of the URI escaping bit was cleared up, and that things were working for you. Sorry for my confusion. After your latest email, I was able to find a problem, but not in any of the cases your explicitly mention: To be clear I can a) LOAD a fasl ("an *.abcl file") with a space in its pathname (load "/Users/evenson/work/abcl/bugs/foo bar.abcl") b) LOAD a fasl from a jar file with a space in the jar pathname (load "jar:file:/Users/evenson/work/abcl/bugs/a%20space/foo%20bar.abcl!/foo._") what I can't do with the current trunk code is c) Load a fasl with a space in its name from a jar ;;; FAILS (load "jar:file:/Users/evenson/work/abcl/bugs/xx.jar!/xx/foo bar.abcl") Can you verify that only the last case fails for you without your patch? Can you provide additional examples of bad behavior? Alessio's point that we need to clearly identify which parts of the Pathname need escaping should clearly be addressed. I've filed [ticket #151][#151] to keep track of this issue. When we agree to the extent of the problem, i.e. identify all the failures, we'll take a stab at fixing things. Currently, I'm thinking we should add flags for Pathname components that can be strings to record when we have already performed URI conversions. Apologies for my initial misinterpretation: let's try to be as clear as possible to fix this correctly. [#151]: http://trac.common-lisp.net/armedbear/ticket/151 -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 4 June 2011 09:59, Mark Evenson <evenson@panix.com> wrote:
If you are referring to the changes culminating [r13302 as the latest work on Pathname.java][r13302], I wouldn't expect any change in the URI escaping behavior as this work only dealt with the version component. Ville had done some other work about a month ago, but I think he retracted his patch as it caused more problems than it solved.
My earlier patches don't handle spaces, but tried to solve handling of filenames that start with strings such as "File::", which happens in perl documentation, for instance. I do need to resurrect that work at some point. The null handling fixes done for Pathname might help there. But, as I said, I didn't do any fixes for spaces-in-names.
On 6/4/11, Mark Evenson <evenson@panix.com> wrote:
On 6/4/11 6:43 AM, Theam Yong Chew wrote:
[…]
To be clear I can
a) LOAD a fasl ("an *.abcl file") with a space in its pathname
(load "/Users/evenson/work/abcl/bugs/foo bar.abcl")
b) LOAD a fasl from a jar file with a space in the jar pathname
(load "jar:file:/Users/evenson/work/abcl/bugs/a%20space/foo%20bar.abcl!/foo._")
what I can't do with the current trunk code is
c) Load a fasl with a space in its name from a jar
;;; FAILS (load "jar:file:/Users/evenson/work/abcl/bugs/xx.jar!/xx/foo bar.abcl")
Can you verify that only the last case fails for you without your patch? Can you provide additional examples of bad behavior?
[...]
Hi, Sorry for the slow reply, I've been thinking about a good way to go about this problem. test.lisp is my initial attempt at exposing all possible space-in-path possibilities (probably not complete yet). It's also not very systematic yet, but for now, I'm trying to find out both cases that work and cases that don't. In the following, I'm trying to compare the differences between an older abcl and my single-line patched version. I'm sorry the test indices are not too informative, as this is just a rough first try, and I need to go to sleep soon (please double check test.lisp, I might have been sloppy too). "0.25.0-dev-svn-13229" ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /home/tyc20/lisp/abcl-space-in-pathname/foo.abcl (0.003 seconds) ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /tmp/foo.abcl (0.003 seconds) ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /tmp/foo bar.abcl (0.004 seconds) 4: Failed to create URI from '/tmp/foo bar.abcl': Illegal character in path at index 13: file:/tmp/foo bar.abcl 5: File not found. 6: File not found. 7: File not found. 8: File not found. 13: Loadable FASL not found for 'jar:file:/tmp/a%20space/xx.jar!/foo.abcl' in 'jar:jar:file:/tmp/a%2520space/xx.jar!/foo.abcl!/foo._' 14: Failed to create URI from '/tmp/a space/xx.jar': Illegal character in path at index 11: file:/tmp/a space/xx.jar 15: File not found. 16: File not found. 17: File not found. Using my "one-line patch" to the above branch, ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /home/tyc20/lisp/abcl-space-in-pathname/foo.abcl (0.003 seconds) ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /tmp/foo.abcl (0.003 seconds) ; Compiling /home/tyc20/lisp/abcl-space-in-pathname/foo.lisp ... ; Wrote /tmp/foo bar.abcl (0.004 seconds) 3: Failed to create URI from '/tmp/foo bar.abcl': Illegal character in path at index 13: file:/tmp/foo bar.abcl 4: Failed to create URI from '/tmp/foo bar.abcl': Illegal character in path at index 13: file:/tmp/foo bar.abcl 5: File not found. 6: File not found. 7: File not found. 8: File not found. 14: Failed to create URI from '/tmp/a space/xx.jar': Illegal character in path at index 11: file:/tmp/a space/xx.jar 15: File not found. 16: File not found. 17: File not found. Hope this is a good starting point? I believe this can later be folded into the unit tests? Yong
On Jun 6, 2011, at 22:46 , Theam Yong Chew wrote: […]
Sorry for the slow reply, I've been thinking about a good way to go about this problem. test.lisp is my initial attempt at exposing all possible space-in-path possibilities (probably not complete yet). It's also not very systematic yet, but for now, I'm trying to find out both cases that work and cases that don't.
In the following, I'm trying to compare the differences between an older abcl and my single-line patched version. I'm sorry the test indices are not too informative, as this is just a rough first try, and I need to go to sleep soon (please double check test.lisp, I might have been sloppy too).
[…]
Hope this is a good starting point? I believe this can later be folded into the unit tests?
Your tests certainly provide good starting point. Some of the failures I get end from running a slightly modified version of 'test.lisp' end in restarts that only allow me to abort the thread, so I don't currently get a clean version to compare to your results. I will try to incorporate these tests into the ABCL RT test suite over the next few days, and then look at a strategy for dealing with the problems. My general plan for addressing the "when do I need escaping?" is to use the hitherto non-obvious feature that ABCL actually has two subtypes of PATHNAME, URL-PATHNAME and JAR-PATHNAME: CL-USER> (type-of #p"foo/bar") PATHNAME CL-USER> (type-of #p"http://foo/bar") URL-PATHNAME CL-USER> (type-of #p"jar:http://foo/bar!/") JAR-PATHNAME The rule should be that both URL-PATHNAME and JAR-PATHNAME would have URI escaped components on creation. I think we just need to have the various escaping/unescaping snippets internal to PATHNAME be mindful of this distinction. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/6/11 22:46 , Theam Yong Chew wrote: […]
Hope this is a good starting point? I believe this can later be folded into the unit tests?
With [r13353](), I believe that I finally have addressed your concerns. I merged what I interpreted to be the meaning of your tests in [abcl/test/lisp/jar-pathnames.lisp][2] avoiding the use of Alexandria to not have dependencies in our test suite. You may want to verify again with your "local conditions" (i.e. I haven't tested under windows") that ticket #151 is truly fixed for you. [r13353]: http://trac.common-lisp.net/armedbear/changeset/13353 [2]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/test/lisp/abcl/jar-... The rules adopted for URI encoding are 1. All pathname components are represented "as is" without escaping. 2. Namestrings are suitably escaped if the Pathname is a URL-PATHNAME or a JAR-PATHNAME. 3. Namestrings should all "round-trip": (when (typep p 'pathname) (equal (namestring p) (namestring (pathname p)))) The basics are that you should always encode #\Space as "%20" when you use any of the "jar:", "jar:file:", "file:", or anything starting with a URL schema as a namestring to construct a pathname. The functions EXT:URI-ENCODE and EXT:URI-DECODE give the user a chance to manipulate strings independently of the PATHNAME constrictors. Strictly speaking these functions are not really RFC compliant routines because what is escaped and what isn't is dependent on what section of the URI one is dealing with, but they "seem to do the right thing" for the basic URI schemas that I have tested. Someday this should be done differently (unsure of how). Hopefully this establishes a reasonable behavior for users wrt. whitespace in pathnames. If this is not the case, please let us know. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/21/11 12:27 , Mark Evenson wrote:
The rules adopted for URI encoding are
1. All pathname components are represented "as is" without escaping.
2. Namestrings are suitably escaped if the Pathname is a URL-PATHNAME
or a JAR-PATHNAME. […]
Alan asked me to include some docstrings for this behavior, so in writing them up I discovered two points that need clarification. a) For URL-PATHNAME the directory components are stored with the URI escaping, contradicting the first rule (pathname-directory #p"http://example.org/bar%20with%20space/there") ==> (:ABSOLUTE "bar%20with%20space") b) For a JAR-PATHNAME the DIRECTORY-NAMESTRING returns an unescaped string: (directory-namestring #p"jar:file:foo.jar!/a%20spaced%20directory/bar.lisp") ==> "/a spaced directory/" It might be nice to allow behavior a) to stand with the reasoning that the typical usage might be to splice up URI paths but in the end the result would be reconstituted as a URL-PATHNAME, but this contradicts the simplicity of rule #1. But I think behavior b) should be fixed to emit the uri encoded representation. If the user wishes to take components from a URL-PATHNAME to refer to files on the local filesystem with #\Space instead of %20, she would need to explicitly specify such a transformation. Thoughts? I hope this isn't the tip of another iceberg… -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 6/21/11, Mark Evenson <evenson@panix.com> wrote:
On 6/6/11 22:46 , Theam Yong Chew wrote: […]
Hope this is a good starting point? I believe this can later be folded into the unit tests?
With [r13353](), I believe that I finally have addressed your concerns. I merged what I interpreted to be the meaning of your tests in [abcl/test/lisp/jar-pathnames.lisp][2] avoiding the use of Alexandria to not have dependencies in our test suite. You may want to verify again with your "local conditions" (i.e. I haven't tested under windows") that ticket #151 is truly fixed for you.
[r13353]: http://trac.common-lisp.net/armedbear/changeset/13353 [2]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/test/lisp/abcl/jar-...
The rules adopted for URI encoding are
1. All pathname components are represented "as is" without escaping.
2. Namestrings are suitably escaped if the Pathname is a URL-PATHNAME
or a JAR-PATHNAME.
3. Namestrings should all "round-trip":
(when (typep p 'pathname)
(equal (namestring p)
(namestring (pathname p))))
The basics are that you should always encode #\Space as "%20" when you use any of the "jar:", "jar:file:", "file:", or anything starting with a URL schema as a namestring to construct a pathname.
The functions EXT:URI-ENCODE and EXT:URI-DECODE give the user a chance to manipulate strings independently of the PATHNAME constrictors. Strictly speaking these functions are not really RFC compliant routines because what is escaped and what isn't is dependent on what section of the URI one is dealing with, but they "seem to do the right thing" for the basic URI schemas that I have tested. Someday this should be done differently (unsure of how).
Hopefully this establishes a reasonable behavior for users wrt. whitespace in pathnames. If this is not the case, please let us know.
--
Hi everyone, Sorry for the slow reply, I've just got around to testing the pathname stuff. Here're my discoveries, using "0.26.0-dev-svn-13379M". This is a long email. First, I needed to patch jar-pathname, like so (truename only works on a pre-existing file/folder). Feel free to do something that makes more sense. tyc20@ackbar:~/lisp/imp/abcl$ svn diff Index: test/lisp/abcl/jar-pathname.lisp =================================================================== --- test/lisp/abcl/jar-pathname.lisp (revision 13379) +++ test/lisp/abcl/jar-pathname.lisp (working copy) @@ -9,10 +9,11 @@ (let ((temp-file (java:jcall "getAbsolutePath" (java:jstatic "createTempFile" "java.io.File" "jar" "tmp")))) (setf *tmp-directory* - (truename (make-pathname :directory - (append - (pathname-directory (pathname temp-file)) - '("jar-pathname-tests")))) + (truename (ensure-directories-exist + (make-pathname :directory + (append + (pathname-directory (pathname temp-file)) + '("jar-pathname-tests"))))) *tmp-directory-whitespace* (merge-pathnames "a/directory with/s p a/" *tmp-directory*)))) After running the tests, I could see "25 out of 548 total tests failed", and "23 unexpected failures". [java] 23 unexpected failures: ABCL.TEST.LISP::UNUSED.1, [java] ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, [java] ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, [java] ABCL.TEST.LISP::MAKE-CLASS-FILE.1, [java] ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, [java] ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, [java] ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, [java] ABCL.TEST.LISP::GENERATE-METHOD.5, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, [java] ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, [java] ABCL.TEST.LISP::PHYSICAL.21, ABCL.TEST.LISP::PHYSICAL.27, [java] ABCL.TEST.LISP::SILLY.5, ABCL.TEST.LISP::TRANSLATE-PATHNAME.5, [java] ABCL.TEST.LISP::PATHNAME.URI-ENCODING.2, [java] ABCL.TEST.LISP::JAR-PATHNAME.LOAD.1, [java] ABCL.TEST.LISP::MATH.READ-FROM-STRING.1. PROBE-FILE.4 and PROBE-FILE.5 are supposed to fail (expected), but is LOAD.1 supposed to fail? Another user facing issue, not reflected in the tests (yet) is differentiating between a normal PATHNAME and URL-PATHNAME for .abcl files. CL-USER> (load "/tmp/nospace.lisp") HI T CL-USER> (load "/tmp/nospace.abcl") HI T CL-USER> (load "/tmp/a space.lisp") HI T CL-USER> (load "/tmp/a space.abcl") Failed to create URI from '/tmp/a space.abcl': Illegal character in path at index 11: file:/tmp/a space.abcl [Condition of type SIMPLE-ERROR] I assume this now requires escaping. But as I write this email, I found that I don't know how exactly to do this: CL-USER> (load "file:/tmp/a%20space.abcl") Failed to create URI from '/tmp/a space.abcl': Illegal character in path at index 11: file:/tmp/a space.abcl [Condition of type SIMPLE-ERROR] CL-USER> (load "jar:file:/tmp/a%20space.abcl") File not found. [Condition of type FILE-ERROR] Is it possible to load .abcl files? The only working example I could come up with seems to break abstraction barriers, by having to know about the internal format of .abcl files (name._ etc) CL-USER> (load #P"jar:file:/tmp/a%20space.abcl!/a%20space._") HI T Now that I look again, the test cases haven't covered any .abcl path loading. I also discovered a corner case (which had been around, but that I wasn't as aware of before) with renamed abcl files: If we compile a file with space in its name, but then rename the resultant abcl file to have no spaces. (load "/tmp/a space.lisp") => ok (compile-file "/tmp/a space.lisp") => "/tmp/a space.abcl" (abcl.test.lisp::cl-fad-copy-file "/tmp/a space.abcl" "/tmp/renamed-to-no-space.abcl") While we have a different name, the contents "a space._" remain the same. Now (load "/tmp/renamed-to-no-space.abcl") => gives error Improper URI syntax for 'file:/a space._': java.net.URISyntaxException: Illegal character in path at index 7: file:/a space._ [Condition of type SIMPLE-ERROR] This is inconsistent with an example with no spaces (or other special characters). Again: (load "/tmp/nospace.lisp") (compile-file "/tmp/nospace.lisp") (abcl.test.lisp::cl-fad-copy-file "/tmp/nospace.abcl" "/tmp/no-space-renamed.abcl") CL-USER> (load "/tmp/no-space-renamed.abcl") HI T Perhaps it's easier to just ban the renaming of .abcl files? (Or deprecate, or declare to be undefined behaviour...). If someone needs a differently named .abcl file, it's simple to use (compile-file "/tmp/a space.lisp" :output-file "/tmp/renamed-to-no-space.abcl") That's all I've tested so far on Ubuntu. ;; ======================================== Unfortunately, on Windows, the .abcl file loading (pathname) issue is much worse, since we see lots of spaces in pathnames. Armed Bear Common Lisp 0.26.0-dev-svn-13379M Java 1.6.0_23 Sun Microsystems Inc. Java HotSpot(TM) Client VM Low-level initialization completed in 1.191 seconds. Startup completed in 3.725 seconds. Loading C:\Documents and Settings\tyc\.abclrc completed in 0.0 seconds. Type ":help" for a list of available commands. CL-USER(1): (require :asdf) ("MOP" "CLOS" "PRINT-OBJECT" "FORMAT" "ASDF") CL-USER(2): (asdf:oos 'asdf:test-op :abcl) ; Compiling C:/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.lisp ... ; (LET* (# #) ...) ; Wrote C:/Documents and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-133 79m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl/ASDF-TMP-rt-package. abcl (0.771 seconds) #<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type SIMPLE-E RROR Failed to create URI from 'C:/Documents and Settings/tyc/.cache/common-lisp/ab cl-0.26.0-dev-svn-13379m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl /rt-package.abcl': Illegal character in opaque part at index 17: file:C:/Documen ts and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-13379m-fasl37-win-x86 /C/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.abcl Restarts: 0: TRY-RECOMPILING Recompile rt-package and try loading it again 1: RETRY Retry loading FASL for #<ASDF:CL-SOURCE-FILE "abcl-test-lis p" "abcl-rt" "rt-package">. 2: ACCEPT Continue, treating loading FASL for #<ASDF:CL-SOURCE-FILE " abcl-test-lisp" "abcl-rt" "rt-package"> as having been successful. 3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS TEM "abcl">. 4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYSTEM "abcl"> as having been successful. 5: TOP-LEVEL Return to top level. [1] CL-USER(3): I temporarily worked around the above problem by setting asdf output translation to a folder path without spaces. I bumped into some compilation errors when running the tests. Here's one: ... ; (DEFTEST PATHNAME.WINDOWS.6 ...) #<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type ERROR The URI has no path: file:z:/foo/bar Restarts: 0: TRY-RECOMPILING Try recompiling pathname-tests 1: RETRY Retry compiling #<ASDF:CL-SOURCE-FILE "abcl-test-lisp" "tes t" "pathname-tests">. 2: ACCEPT Continue, treating compiling #<ASDF:CL-SOURCE-FILE "abcl-te st-lisp" "test" "pathname-tests"> as having been successful. 3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS TEM "abcl">. 4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYSTEM "abcl"> as having been successful. 5: TOP-LEVEL Return to top level. I don't have a z drive. I can type some of the forms in the REPL without any issues, but not others: CL-USER(24): (pathname-device #P"z:foo/bar") "z" CL-USER(25): (pathname-device #P"z:/foo/bar") "z" Trying #P"file:z:foo/bar" leads to the "URI has no path" error. Finally, because I used the asdf pretend-file-compiled-successfully-continue restart a few times, lots of tests were not run in the end, but whatever ran gave the following errors: 20 out of 290 total tests failed: ABCL.TEST.LISP::UNUSED.1, ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, ABCL.TEST.LISP::MAKE-CLASS-FILE.1, ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, ABCL.TEST.LISP::GENERATE-METHOD.5, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2, ABCL.TEST.LISP::MATH.READ-FROM-STRING.1, ABCL.TEST.LISP::ZIP.1. 19 unexpected failures: ABCL.TEST.LISP::UNUSED.1, ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, ABCL.TEST.LISP::MAKE-CLASS-FILE.1, ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, ABCL.TEST.LISP::GENERATE-METHOD.5, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2, ABCL.TEST.LISP::MATH.READ-FROM-STRING.1. It sounds like the .abcl loading mechanism needs to support spaces too (especially on Windows). That's all for now (I hope). I have seen a few other puzzling things I'll try and investigate later. I guess all the above problems relate to .abcl paths, so do they count as jar-pathnames or normal pathnames, or what? Thanks for looking into this. Yong.
On Jul 9, 2011, at 12:57 , Theam Yong Chew wrote: […]
Sorry for the slow reply, I've just got around to testing the pathname stuff. Here're my discoveries, using "0.26.0-dev-svn-13379M". This is a long email.
[…] Thank for the considered reply! It will take a bit of time to go through everything in detail to give your information the attention it deserves, but rest assured that it is in my TODO pile. […]
After running the tests, I could see "25 out of 548 total tests failed", and "23 unexpected failures".
[java] 23 unexpected failures: ABCL.TEST.LISP::UNUSED.1, [java] ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, [java] ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, [java] ABCL.TEST.LISP::MAKE-CLASS-FILE.1, [java] ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, [java] ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, [java] ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, [java] ABCL.TEST.LISP::GENERATE-METHOD.5, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, [java] ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, [java] ABCL.TEST.LISP::PHYSICAL.21, ABCL.TEST.LISP::PHYSICAL.27, [java] ABCL.TEST.LISP::SILLY.5, ABCL.TEST.LISP::TRANSLATE-PATHNAME.5, [java] ABCL.TEST.LISP::PATHNAME.URI-ENCODING.2, [java] ABCL.TEST.LISP::JAR-PATHNAME.LOAD.1, [java] ABCL.TEST.LISP::MATH.READ-FROM-STRING.1.
PROBE-FILE.4 and PROBE-FILE.5 are supposed to fail (expected), but is LOAD.1 supposed to fail?
[…] JAR-PATHNAME.LOAD.1 is funny. The first time it is run it fails, but then subsequent invocations seem to succeed. I haven't figured out what is going on with it quite yet, but I presume it has something to do with being the first test to use the WITH-JAR-INIT macro. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 2/17/11, Mark Evenson <evenson@panix.com> wrote:
On 2/16/11 14:34 , Theam Yong Chew wrote:
[…]
[1] CL-USER(1): (load "jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl") #<THREAD "interpreter" {12C9557}>: Debugger invoked on condition of type FILE-ERROR Loadable FASL not found for 'jar:file:/home/tyc20/Dropbox/temp/a%20space/hello.jar!/hello.abcl' in 'jar:jar:file:/home/tyc20/Dropbox/temp/a%2520space/hello.jar!/hello.abcl!/hello._' Restarts: 0: ABORT Return to debug level 1.
[…]
So to clarify, it wasn't a "Failed to create URI" error, but a "Loadable FASL not found" error.
It looks like you might have encountered a more serious problem with our classloader. Can you mail me a copy of the failing jar file?
I've included that, but I don't think there's anything significant about it. Here's the original source: ==== hello.lisp ==== (format t "Hello.~%") ================ Compile and jar up like so: (compile-file "...hello.lisp") Then at a command line, jar cf hello.jar hello.abcl That's it.
You might try recompiling all the FASLs if you have upgraded the 'abcl.jar' you use recently, but this probably won't help.
Yeah, I've check that. I'm quite sure I'm consistently using abcl-0.24.0 Thanks for your prompt attention. Yong.
participants (4)
-
Alessio Stalla -
Mark Evenson -
Theam Yong Chew -
Ville Voutilainen