On Mar 11, 2010, at 4:15 AM, Alan Ruttenberg wrote:

And related,

(MAKE-PATHNAME :TYPE "abcl" :DIRECTORY (:ABSOLUTE "lsw" "jss" "armedbear-0.19.0-dev-darwin-unknown") :DEFAULTS #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.abcl")

->

#P"jar:file:/Users/alanr/Desktop/lsw.jar!//lsw/jss/armedbear-0.19.0-dev-darwin-unknown/invoke.abcl"

Note the "//"

Which fails.

Fails what? Does lsw.jar actually contain relative jar entries as is the usual behavior for JARs?

-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

On Thu, Mar 11, 2010 at 9:19 AM, Mark Evenson evenson@panix.com wrote:

On Mar 11, 2010, at 3:00 AM, Alan Ruttenberg wrote:

...

CL-USER(1): (PATHNAME-DIRECTORY #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.abcl") -> (:RELATIVE "lsw" "jss")

I think that's wrong. Should be :absolute, right?

:RELATIVE is the desired behavior because "!/" is actually a terminal symbol that separates the URL from the ENTRY, i.e.

JAR-PATHNAME ::= "jar:" URL "!/" ENTRY

If you look at the contents of a jar file via java.util.ZipFile.entries(), all the hierarchal entries do not have an initial slash, so are best expressed as relative pathnames.

True, they don't have an initial slash, but they're all relative to the root of the jar i.e. they are absolute, to my understanding. If they were relative, they could be interpreted differently depending to some reference directory.

Or at least that was true in my testing.  Although I didn't look a specification of Zip file structure to confirm this but rather observed how the JVM tools on various platforms behaved, so perhaps this is an incorrect assumption.

Do you have an example of a JAR where this isn't the case?  If so, please give details so we can somehow figure out how to accommodate this behavior.

-- "A screaming comes across the sky.  It has happened before, but there is nothing to compare to it now."

---

armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

The problem surfaced with what seems to me to be a regression in the use of the abcl-binary-locations system, which puts compiled files in separate directories according to implementation. I'm working on packaging up my application in a single jar. I move the systems inside the jar and then try to load using asdf. Because of the use of :relative, the call to OUTPUT-FILES-USING-MAPPINGS duplicates the leading directory components, and then the compiled file is not found.

The thing is that with the current setup has it that there is never an :absolute directory. I think this will violate assumptions of programs beyond abcl-binary-locations. I believe the assumption will be that truename will give you something that has an :absolute pathname.

-Alan

CL-USER(37): (asdf::oos 'asdf::load-op (asdf::find-system "jss")) 0: (ASDF::OUTPUT-FILES-USING-MAPPINGS #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.lisp" (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.abcl") NIL) 1: (ASDF:IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME) 1: IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME returned "armedbear-0.19.0-dev-darwin-unknown" 0: OUTPUT-FILES-USING-MAPPINGS returned (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/lsw/jss/armedbear-0.19.0-dev-darwin-unknown/invoke.abcl") 0: (ASDF::OUTPUT-FILES-USING-MAPPINGS #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.lisp" (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.abcl") NIL) 1: (ASDF:IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME) 1: IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME returned "armedbear-0.19.0-dev-darwin-unknown" 0: OUTPUT-FILES-USING-MAPPINGS returned (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/lsw/jss/armedbear-0.19.0-dev-darwin-unknown/invoke.abcl") 0: (ASDF::OUTPUT-FILES-USING-MAPPINGS #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.lisp" (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/invoke.abcl") NIL) 1: (ASDF:IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME) 1: IMPLEMENTATION-SPECIFIC-DIRECTORY-NAME returned "armedbear-0.19.0-dev-darwin-unknown" 0: OUTPUT-FILES-USING-MAPPINGS returned (#P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/lsw/jss/armedbear-0.19.0-dev-darwin-unknown/invoke.abcl") 0: (LOAD #P"jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/jss/lsw/jss/armedbear-0.19.0-dev-darwin-unknown/invoke.abcl") Debugger invoked on condition of type FILE-ERROR: File not found.

On 3/11/10, Mark Evenson evenson@panix.com wrote:

On Mar 11, 2010, at 9:37 AM, Alessio Stalla wrote:

[…]

...

...

If you look at the contents of a jar file via java.util.ZipFile.entries(), all the hierarchal entries do not have an initial slash, so are best expressed as relative pathnames.

True, they don't have an initial slash, but they're all relative to the root of the jar i.e. they are absolute, to my understanding. If they were relative, they could be interpreted differently depending to some reference directory.

Since we control the behavior of Pathname completely as a Java primitive, we can change the underlying implementation to properly treat the :ABSOLUTE component of the DIRECTORY list by special-casing the searching/iteration of jar file contents. This change would preclude the distinction between a JAR that has entries with an initial '/' and those without, but we could pave over those differences via some amount of if/thens sprinkled throughout the code.

But before digging everything up again and testing it, I'd like to understand what we want to accomplish here.

Since MERGE-PATHNAMES currently has special semantics with jar pathnames already, we wouldn't really gain any useful behavior that I can immediately think of.

Would you propose that the namestring of a jar pathname remain "jar:URL!/foo/bar.lisp" or would it now become "jar:URL!//foo/bar.lisp" (which would break the behavior documented in java.net.JarURLConnection)?

So, if y'all really want to change the :RELATIVE to :ABSOLUTE here, please convince me, as I currently a net negative in this approach.

I've updated [the Pathname implementation notes][1] to explain the current behavior better.

--

"A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

On Thu, Mar 11, 2010 at 8:06 AM, Mark Evenson evenson@panix.com wrote:

On Mar 11, 2010, at 1:53 PM, Alan Ruttenberg wrote:

...

The problem surfaced with what seems to me to be a regression in the use of the abcl-binary-locations system, which  puts compiled files in separate directories according to implementation. I'm working on packaging up my application in a single jar. I move the systems inside the jar and then try to load using asdf. Because of the use of :relative, the call to OUTPUT-FILES-USING-MAPPINGS duplicates the leading directory components, and then the compiled file is not found.

In your usage of ASDF-BINARY-LOCATIONS the system in the JAR does not have FASLs, which are compiled somewhere to the local filesystem, right?  Just packaging the entire ASDF system with FASLs in the JAR does work (c.f. the abcl-contrib.jar containing ASDF-INSTALL built by the 'abcl.contrib' target).  Maybe you could circumvent the usage of ASDF-BINARY-LOCATIONS for jars for the time being?

Ok. I'll do that for now.

...

The thing is that with the current setup has it that there is never an :absolute directory. I think this will violate assumptions of programs beyond abcl-binary-locations. I believe the assumption will be that truename will give you something that has an :absolute pathname.

The assertion that TRUENAME by convention always returns an :ABSOLUTE directory component certainly leans the argument towards implementing that behavior.  We'd have to examine the Zip specification (I assume there's a RFC) to see if doing this would be ambiguous for actual absolute entries in the jar.

Once I test loading from OSGi bundles, I'll try to see what's blocking ASDF-BINARY-LOCATIONS from working.

Ok, thanks. An also thank you both very much(!) for looking athe OSGi stuff. I'm completely clueless about how any of that works so I don't even know where to start.

-Alan

--

"A screaming comes across the sky.  It has happened before, but there is nothing to compare to it now."

On Fri, Mar 12, 2010 at 11:05 AM, Mark Evenson evenson@panix.com wrote:

On Mar 11, 2010, at 2:15 PM, Alan Ruttenberg wrote:

[…]

...

...

In your usage of ASDF-BINARY-LOCATIONS the system in the JAR does not have FASLs, which are compiled somewhere to the local filesystem, right?  Just packaging the entire ASDF system with FASLs in the JAR does work (c.f. the abcl-contrib.jar containing ASDF-INSTALL built by the 'abcl.contrib' target).  Maybe you could circumvent the usage of ASDF-BINARY-LOCATIONS for jars for the time being?

Ok. I'll do that for now.

[…]

I got a little time today to try out ASDF-BINARY-LOCATIONS with ABCL, but didn't get that far, maybe you can explain a little more about your setup.

I downloaded ASDF-BINARY-LOCATIONS via ASDF-INSTALL which reports a version of "0.3.9".  I assume for your usage of ASDF-BINARY-LOCATIONS you want to have a centralized directory as opposed to a per-jar mapping, so I configured .abclrc with the following:

(require :asdf-binary-locations)  (setf asdf:*centralize-lisp-binaries* t        asdf:*default-toplevel-directory* (truename "~/.asdf-binaries/"))

I load it and leave it at default settings. It puts subdirectories where my asd files are.

The idea is to do as little special-case packaging as possible I want to have a loaded project and say: (ship-it)

To package I plan to unpack all jars used by my system into a single directory, add one more directory for my source, copy all source directories to there (except omitting any dynamic jars also placed there), modify system.lisp to load all my systems (disabling dynamic jar loading), and have some code that interprets the command line arguments to do stuff.

Here's the start of the code for managing the jars. http://svn.mumble.net:8080/svn/lsw/trunk/util/jar-current-classpath.lisp

The problem occurred when I tried to load the compiled system code so prepared.

-Alan

This seems to not have the desired behavior, as ASDF now compiles the FASLS to a subdirectory of the location of the Lisp files.  This part seems to be failing the included tests as well.

Could you correct my assumptions about your setup if wrong, and/or explain a little more about how you are attempting to use ASDF-BINARY-LOCATIONS with jars?

--

"A screaming comes across the sky.  It has happened before, but there is nothing to compare to it now."

After further analysis, I concede that the directory portion of jar Pathnames should be :ABSOLUTE and not :RELATIVE. I'm not sure how long such a re-implementation will take, but it's in-progress. I'd like to get it finished before we release 0.19, but we'll see.

I'd like to preserve keeping the namestring of a jar pathname congruent to the JAR URL syntax (i.e. "jar:URL!/ENTRY") so that even when ENTRY contains an :ABSOLUTE directory, the printing of a double slash is suppressed. But whether this means I should not allow :RELATIVE directories in a jar pathname or not I am not sure. Maybe a jar pathname with a :RELATIVE directory doesn't have a namestring representation? Opinions on this are welcome.

On Sun, Mar 14, 2010 at 9:33 AM, Mark Evenson evenson@panix.com wrote:

On 3/14/10 11:30 AM, Mark Evenson wrote:

...

After further analysis, I concede that the directory portion of jar Pathnames should be :ABSOLUTE and not :RELATIVE.  I'm not sure how long such a re-implementation will take, but it's in-progress. I'd like to get it finished before we release 0.19, but we'll see.

[…]

An [implementation has been committed to trunk in r12531][1].  Having a test suite for the behavior made this go a lot faster than I had thought, and increases my confidence in the changes.

This needs to be backported to 0.19 before that release.

-- "A screaming comes across the sky.  It has happened before, but there is nothing to compare to it now."

---

armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

Hi Mark,

Thanks for this.

Things load up now, however there is something that confuses me:

I see load messages like the following (note that there are two "!/" in the pathname.

; Loaded #P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/" (0.179 seconds)

Describe shows that the device is a list of the jar path and directory, and that there is no directory.

CL-USER(6): (describe #P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/") #P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/" is an object of type PATHNAME: HOST NIL DEVICE (#P"/Users/alanr/Desktop/lsw.jar" #P"lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl") DIRECTORY NIL NAME NIL TYPE NIL VERSION NIL

ps. I think I'm losing on the jar combination thing. It looks like (still need to confirm this) that there are classes called EDU.xxxx as well as ones called "edu.yyyy". These get smushed together into one "edu" directory because of OS X default case-insensitive file system, and when jarred back together one of them loses. Sigh. Might have to take another approach :(. Hmm, wonder if I can create a case sensitive disk image on the fly. Or maybe jarjar (http://code.google.com/p/jarjar/) is the thing to use...

-Alan

On Mon, Mar 15, 2010 at 4:08 AM, Mark Evenson evenson@panix.com wrote:

On 3/15/10 6:20 AM, Alan Ruttenberg wrote: […]

...

Things load up now, however there is something that confuses me:

I see load messages like the following (note that there are two "!/" in the pathname.

; Loaded #P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/" (0.179 seconds)

Describe shows that the device is a list of the jar path and directory, and that there is no directory.

CL-USER(6): (describe

#P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/")

#P"jar:jar:file:/Users/alanr/Desktop/lsw.jar!/lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl!/" is an object of type PATHNAME:   HOST         NIL   DEVICE       (#P"/Users/alanr/Desktop/lsw.jar" #P"lsw/owl/tests/armedbear-0.20.0-dev-darwin-unknown/test-suite.abcl")   DIRECTORY    NIL   NAME         NIL   TYPE         NIL   VERSION      NIL

This is how the implementation represents a jar within a jar, which is necessary for loading FASLs to work.  Remember that a packed ABCL FASL is a zip (i.e. jar) file that contains an init FASL (the "*._" file), plus the compiled classes (the "*-nnn.cls" where nnn is an increasing integer).  When the system loads the packed FASL, it uses MERGE-PATHNAME to combine the name of the sub-FASL (i.e. "foo-1.cls") with the *LOAD-TRUENAME* of the FASL (i.e "jar:jar:file:/a/b/bar.jar!/foo.abcl!/").

In writing this reply, I realize this probably breaks the use of *LOAD-TRUENAME*/*LOAD-PATHNAME* to code that finds itself in a packed FASL in a jar.  Such code would probably be much happier if its *LOAD-PATHNAME* were

jar:file:/a/b/bar.jar!/foo.abcl

while its *LOAD-TRUENAME* remained

jar:jar:file/a/b/bar.jar!/foo.abcl!/

Or maybe both should not have the double jar syntax, so that MERGE-PATHNAME would do the right thing?  I introduced another special (SYS::*LOAD-TRUENAME-FASL*) where we can keep the "real" value.

OK, I admit my head is hurting now. My gut is that when loading compiled files, one expects *load-truename* and *load-pathname* to be the same sort of thing as when loading the lisp source. It's an implementation detail that the compiled file is a jar.

In other cases, I don't know.

Holler back if y'all agree with this modification, and I'll try to find time to implement it soon.

-- "A screaming comes across the sky.  It has happened before, but there is nothing to compare to it now."

On Mon, Mar 15, 2010 at 1:20 AM, Alan Ruttenberg alanruttenberg@gmail.com wrote:

ps. I think I'm losing on the jar combination thing. It looks like (still need to confirm this) that there are classes called EDU.xxxx as well as ones called "edu.yyyy". These get smushed together into one "edu" directory because of OS X default case-insensitive file system, and when jarred back together one of them loses. Sigh. Might have to take another approach :(. Hmm, wonder if I can create a case sensitive disk image on the fly. Or maybe jarjar (http://code.google.com/p/jarjar/) is the thing to use...

FWIW, the approach of using the case-sensitive disk image to unjar to does indeed work. I might still look into jarjar, as it would be nice to have something that didn't depend on running in os x to work.

There has been discussion in the past about implementing save-image, which is hard. However, having a function that creates a single jar, arranging to include all your compiled files and reload them on start would be a nice step towards that goal. I'll keep y'all informed of my progress.

-Alan

Sounds really cool.

Long-term, is it conceivable to use a meta-class for improved Java integration? (External) Java classes would be instances and ABCL classes that are instances would be compiled to .class files. Obviously, instances of this meta-class would only be able to dispatch on the first argument.

Too complicated? Doable?

Thanks!

Axel

On Mar 15, 2010, at 14:58 , Alan Ruttenberg wrote:

On Mon, Mar 15, 2010 at 1:20 AM, Alan Ruttenberg alanruttenberg@gmail.com wrote:

...

ps. I think I'm losing on the jar combination thing. It looks like (still need to confirm this) that there are classes called EDU.xxxx as well as ones called "edu.yyyy". These get smushed together into one "edu" directory because of OS X default case-insensitive file system, and when jarred back together one of them loses. Sigh. Might have to take another approach :(. Hmm, wonder if I can create a case sensitive disk image on the fly. Or maybe jarjar (http://code.google.com/p/jarjar/) is the thing to use...

FWIW, the approach of using the case-sensitive disk image to unjar to does indeed work. I might still look into jarjar, as it would be nice to have something that didn't depend on running in os x to work.

There has been discussion in the past about implementing save-image, which is hard. However, having a function that creates a single jar, arranging to include all your compiled files and reload them on start would be a nice step towards that goal. I'll keep y'all informed of my progress.

-Alan

---

armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel