On 6/21/11, Mark Evenson evenson@panix.com wrote:
On 6/6/11 22:46 , Theam Yong Chew wrote: […]
Hope this is a good starting point? I believe this can later be folded into the unit tests?
With [r13353](), I believe that I finally have addressed your concerns. I merged what I interpreted to be the meaning of your tests in [abcl/test/lisp/jar-pathnames.lisp][2] avoiding the use of Alexandria to not have dependencies in our test suite. You may want to verify again with your "local conditions" (i.e. I haven't tested under windows") that ticket #151 is truly fixed for you.
The rules adopted for URI encoding are
All pathname components are represented "as is" without escaping.
Namestrings are suitably escaped if the Pathname is a URL-PATHNAME
or a JAR-PATHNAME.
Namestrings should all "round-trip":
(when (typep p 'pathname)
(equal (namestring p) (namestring (pathname p))))The basics are that you should always encode #\Space as "%20" when you use any of the "jar:", "jar:file:", "file:", or anything starting with a URL schema as a namestring to construct a pathname.
The functions EXT:URI-ENCODE and EXT:URI-DECODE give the user a chance to manipulate strings independently of the PATHNAME constrictors. Strictly speaking these functions are not really RFC compliant routines because what is escaped and what isn't is dependent on what section of the URI one is dealing with, but they "seem to do the right thing" for the basic URI schemas that I have tested. Someday this should be done differently (unsure of how).
Hopefully this establishes a reasonable behavior for users wrt. whitespace in pathnames. If this is not the case, please let us know.
--
Hi everyone,
Sorry for the slow reply, I've just got around to testing the pathname stuff. Here're my discoveries, using "0.26.0-dev-svn-13379M". This is a long email.
First, I needed to patch jar-pathname, like so (truename only works on a pre-existing file/folder). Feel free to do something that makes more sense.
tyc20@ackbar:~/lisp/imp/abcl$ svn diff Index: test/lisp/abcl/jar-pathname.lisp =================================================================== --- test/lisp/abcl/jar-pathname.lisp (revision 13379) +++ test/lisp/abcl/jar-pathname.lisp (working copy) @@ -9,10 +9,11 @@ (let ((temp-file (java:jcall "getAbsolutePath" (java:jstatic "createTempFile" "java.io.File" "jar" "tmp")))) (setf *tmp-directory* - (truename (make-pathname :directory - (append - (pathname-directory (pathname temp-file)) - '("jar-pathname-tests")))) + (truename (ensure-directories-exist + (make-pathname :directory + (append + (pathname-directory (pathname temp-file)) + '("jar-pathname-tests"))))) *tmp-directory-whitespace* (merge-pathnames "a/directory with/s p a/" *tmp-directory*))))
After running the tests, I could see "25 out of 548 total tests failed", and "23 unexpected failures".
[java] 23 unexpected failures: ABCL.TEST.LISP::UNUSED.1, [java] ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, [java] ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, [java] ABCL.TEST.LISP::MAKE-CLASS-FILE.1, [java] ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, [java] ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, [java] ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, [java] ABCL.TEST.LISP::GENERATE-METHOD.5, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, [java] ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, [java] ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, [java] ABCL.TEST.LISP::PHYSICAL.21, ABCL.TEST.LISP::PHYSICAL.27, [java] ABCL.TEST.LISP::SILLY.5, ABCL.TEST.LISP::TRANSLATE-PATHNAME.5, [java] ABCL.TEST.LISP::PATHNAME.URI-ENCODING.2, [java] ABCL.TEST.LISP::JAR-PATHNAME.LOAD.1, [java] ABCL.TEST.LISP::MATH.READ-FROM-STRING.1.
PROBE-FILE.4 and PROBE-FILE.5 are supposed to fail (expected), but is LOAD.1 supposed to fail?
Another user facing issue, not reflected in the tests (yet) is differentiating between a normal PATHNAME and URL-PATHNAME for .abcl files.
CL-USER> (load "/tmp/nospace.lisp") HI T CL-USER> (load "/tmp/nospace.abcl") HI T CL-USER> (load "/tmp/a space.lisp") HI T CL-USER> (load "/tmp/a space.abcl") Failed to create URI from '/tmp/a space.abcl': Illegal character in path at index 11: file:/tmp/a space.abcl [Condition of type SIMPLE-ERROR]
I assume this now requires escaping. But as I write this email, I found that I don't know how exactly to do this:
CL-USER> (load "file:/tmp/a%20space.abcl") Failed to create URI from '/tmp/a space.abcl': Illegal character in path at index 11: file:/tmp/a space.abcl [Condition of type SIMPLE-ERROR]
CL-USER> (load "jar:file:/tmp/a%20space.abcl") File not found. [Condition of type FILE-ERROR]
Is it possible to load .abcl files? The only working example I could come up with seems to break abstraction barriers, by having to know about the internal format of .abcl files (name._ etc)
CL-USER> (load #P"jar:file:/tmp/a%20space.abcl!/a%20space._") HI T
Now that I look again, the test cases haven't covered any .abcl path loading.
I also discovered a corner case (which had been around, but that I wasn't as aware of before) with renamed abcl files:
If we compile a file with space in its name, but then rename the resultant abcl file to have no spaces.
(load "/tmp/a space.lisp") => ok (compile-file "/tmp/a space.lisp") => "/tmp/a space.abcl" (abcl.test.lisp::cl-fad-copy-file "/tmp/a space.abcl" "/tmp/renamed-to-no-space.abcl")
While we have a different name, the contents "a space._" remain the same. Now
(load "/tmp/renamed-to-no-space.abcl") => gives error
Improper URI syntax for 'file:/a space._': java.net.URISyntaxException: Illegal character in path at index 7: file:/a space._ [Condition of type SIMPLE-ERROR]
This is inconsistent with an example with no spaces (or other special characters). Again:
(load "/tmp/nospace.lisp") (compile-file "/tmp/nospace.lisp") (abcl.test.lisp::cl-fad-copy-file "/tmp/nospace.abcl" "/tmp/no-space-renamed.abcl")
CL-USER> (load "/tmp/no-space-renamed.abcl") HI T
Perhaps it's easier to just ban the renaming of .abcl files? (Or deprecate, or declare to be undefined behaviour...). If someone needs a differently named .abcl file, it's simple to use
(compile-file "/tmp/a space.lisp" :output-file "/tmp/renamed-to-no-space.abcl")
That's all I've tested so far on Ubuntu.
;; ========================================
Unfortunately, on Windows, the .abcl file loading (pathname) issue is much worse, since we see lots of spaces in pathnames.
Armed Bear Common Lisp 0.26.0-dev-svn-13379M Java 1.6.0_23 Sun Microsystems Inc. Java HotSpot(TM) Client VM Low-level initialization completed in 1.191 seconds. Startup completed in 3.725 seconds. Loading C:\Documents and Settings\tyc.abclrc completed in 0.0 seconds. Type ":help" for a list of available commands. CL-USER(1): (require :asdf) ("MOP" "CLOS" "PRINT-OBJECT" "FORMAT" "ASDF") CL-USER(2): (asdf:oos 'asdf:test-op :abcl) ; Compiling C:/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.lisp ... ; (LET* (# #) ...) ; Wrote C:/Documents and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-133 79m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl/ASDF-TMP-rt-package. abcl (0.771 seconds) #<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type SIMPLE-E RROR Failed to create URI from 'C:/Documents and Settings/tyc/.cache/common-lisp/ab cl-0.26.0-dev-svn-13379m-fasl37-win-x86/C/me/progs/abcl-0.26-test/test/lisp/abcl /rt-package.abcl': Illegal character in opaque part at index 17: file:C:/Documen ts and Settings/tyc/.cache/common-lisp/abcl-0.26.0-dev-svn-13379m-fasl37-win-x86 /C/me/progs/abcl-0.26-test/test/lisp/abcl/rt-package.abcl Restarts: 0: TRY-RECOMPILING Recompile rt-package and try loading it again 1: RETRY Retry loading FASL for #<ASDF:CL-SOURCE-FILE "abcl-test-lis p" "abcl-rt" "rt-package">. 2: ACCEPT Continue, treating loading FASL for #<ASDF:CL-SOURCE-FILE " abcl-test-lisp" "abcl-rt" "rt-package"> as having been successful. 3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS TEM "abcl">. 4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYSTEM "abcl"> as having been successful. 5: TOP-LEVEL Return to top level. [1] CL-USER(3):
I temporarily worked around the above problem by setting asdf output translation to a folder path without spaces. I bumped into some compilation errors when running the tests. Here's one:
... ; (DEFTEST PATHNAME.WINDOWS.6 ...) #<THREAD "interpreter" {9D3F8B}>: Debugger invoked on condition of type ERROR The URI has no path: file:z:/foo/bar Restarts: 0: TRY-RECOMPILING Try recompiling pathname-tests 1: RETRY Retry compiling #<ASDF:CL-SOURCE-FILE "abcl-test-lisp" "tes t" "pathname-tests">. 2: ACCEPT Continue, treating compiling #<ASDF:CL-SOURCE-FILE "abcl-te st-lisp" "test" "pathname-tests"> as having been successful. 3: RETRY Retry #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYS TEM "abcl">. 4: ACCEPT Continue, treating #<STANDARD-CLASS ASDF:TEST-OP {4BD173}> on #<ASDF:SYSTEM "abcl"> as having been successful. 5: TOP-LEVEL Return to top level.
I don't have a z drive. I can type some of the forms in the REPL without any issues, but not others:
CL-USER(24): (pathname-device #P"z:foo/bar") "z" CL-USER(25): (pathname-device #P"z:/foo/bar") "z"
Trying #P"file:z:foo/bar" leads to the "URI has no path" error.
Finally, because I used the asdf pretend-file-compiled-successfully-continue restart a few times, lots of tests were not run in the end, but whatever ran gave the following errors:
20 out of 290 total tests failed: ABCL.TEST.LISP::UNUSED.1, ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, ABCL.TEST.LISP::MAKE-CLASS-FILE.1, ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, ABCL.TEST.LISP::GENERATE-METHOD.5, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2, ABCL.TEST.LISP::MATH.READ-FROM-STRING.1, ABCL.TEST.LISP::ZIP.1. 19 unexpected failures: ABCL.TEST.LISP::UNUSED.1, ABCL.TEST.LISP::UNUSED.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.2, ABCL.TEST.LISP::DERIVE-TYPE-LOGXOR.3, ABCL.TEST.LISP::MAKE-CLASS-FILE.1, ABCL.TEST.LISP::FINALIZE-CLASS-FILE.1, ABCL.TEST.LISP::GENERATE-METHOD.1, ABCL.TEST.LISP::GENERATE-METHOD.2, ABCL.TEST.LISP::GENERATE-METHOD.3, ABCL.TEST.LISP::GENERATE-METHOD.4, ABCL.TEST.LISP::GENERATE-METHOD.5, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.1, ABCL.TEST.LISP::WITH-CODE-TO-METHOD.2, ABCL.TEST.LISP::DMC-RETURN.1, ABCL.TEST.LISP::DMC-RETURN.2, ABCL.TEST.LISP::DMC-RETURN.3, ABCL.TEST.LISP::URL-PATHNAME.1, ABCL.TEST.LISP::URL-PATHNAME.2, ABCL.TEST.LISP::MATH.READ-FROM-STRING.1.
It sounds like the .abcl loading mechanism needs to support spaces too (especially on Windows). That's all for now (I hope). I have seen a few other puzzling things I'll try and investigate later.
I guess all the above problems relate to .abcl paths, so do they count as jar-pathnames or normal pathnames, or what?
Thanks for looking into this.
Yong.