Implementing true support for JAR files in ABCL has taken quite a bit more time and effort as can be seen by the rejection of my mail of few minutes ago to the list for being too long.
I've posted my [patches with explanation to the web][1], as I am not sure when or if the moderator will let the original message through.
[1]: http://slack.net/~evenson/abcl/jar-file
About the only disadvantage for the current patch is that I had to remove the use of ZipCache for entries as I was developing as it didn't make much sense. I haven't really noticed too much of a time difference for loading or compiling ABCL.
Comments? Once I get things tested under Windows, and play around with the loading from http support, I intend to commit to trunk.
On Sun, Jan 24, 2010 at 4:14 PM, Mark Evenson evenson@panix.com wrote:
Implementing true support for JAR files in ABCL has taken quite a bit more time and effort as can be seen by the rejection of my mail of few minutes ago to the list for being too long.
I've posted my [patches with explanation to the web][1], as I am not sure when or if the moderator will let the original message through.
[1]: http://slack.net/~evenson/abcl/jar-file
About the only disadvantage for the current patch is that I had to remove the use of ZipCache for entries as I was developing as it didn't make much sense. I haven't really noticed too much of a time difference for loading or compiling ABCL.
Comments? Once I get things tested under Windows, and play around with the loading from http support, I intend to commit to trunk.
This is great news for me. Thanks for all your efforts!
The only thing that I have not clear is the integration with ASDF: is it possible to have a .asd inside a jar and using it to load a Lisp library from the same jar? If yes, great :) if no, what is missing?
Incidentally, we may be able to autodiscover .asd files by scanning the classpath, and suitably update asdf:*central-registry*.
Alessio
On 1/26/10 11:39 AM, Alessio Stalla wrote: […]
The only thing that I have not clear is the integration with ASDF: is it possible to have a .asd inside a jar and using it to load a Lisp library from the same jar? If yes, great :) if no, what is missing?
I'll check that out now, as this is one think that I want to work for ease of use, so I'll let you know in a bit.
Incidentally, we may be able to autodiscover .asd files by scanning the classpath, and suitably update asdf:*central-registry*.
We first need to agree on a convention to wedge the concept of Pathname into CLASSPATH. I now support your earlier proposal to keep the two separate, as it makes more sense. And I'd like to get the "jar:" part of my changes into the trunk, and tested for a bit before introducing another the next step. Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
Incidentally, we may be able to autodiscover .asd files by scanning the classpath, and suitably update asdf:*central-registry*.
We first need to agree on a convention to wedge the concept of Pathname into CLASSPATH. I now support your earlier proposal to keep the two separate, as it makes more sense. And I'd like to get the "jar:" part of my changes into the trunk, and tested for a bit before introducing another the next step. Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
With Java resources, one is always warned against using the classpath that way. Looking for a particular path is possible, but listing paths (or files) not. It might be safer to introduce an additional concept. This could be an explicit list of Java resource paths or a new kind of ASDF classpath.
Here is how Spring does auto-detection: http://static.springsource.org/spring/docs/2.5.x/reference/beans.html#beans-...
Here is more information from Sun on finding classes: http://java.sun.com/j2se/1.5.0/docs/tooldocs/findingclasses.html
On Tue, Jan 26, 2010 at 1:21 PM, Axel Rauschmayer axel@rauschma.de wrote:
Incidentally, we may be able to autodiscover .asd files by scanning the classpath, and suitably update asdf:*central-registry*.
We first need to agree on a convention to wedge the concept of Pathname into CLASSPATH. I now support your earlier proposal to keep the two separate, as it makes more sense. And I'd like to get the "jar:" part of my changes into the trunk, and tested for a bit before introducing another the next step. Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
With Java resources, one is always warned against using the classpath that way. Looking for a particular path is possible, but listing paths (or files) not. It might be safer to introduce an additional concept. This could be an explicit list of Java resource paths or a new kind of ASDF classpath.
Scanning the entire "classpath" (or better, the resources known to classloaders) in general is impossible, since a classloader can create a resource from anywhere (e.g. from a buffer in memory, like ABCL itself does for runtime compilation).
However, it is possible to scan the classpath in the most common cases, i.e. when the resources are either on the filesystem or in some Jar loaded by a regular URLClassLoader.
I agree that doing it right, for some value of right, is far from easy as there are tricky cases (e.g. classloaders used by some application servers) that you might still want to handle.
Alessio
Here is how Spring does auto-detection: http://static.springsource.org/spring/docs/2.5.x/reference/beans.html#beans-...
Here is more information from Sun on finding classes: http://java.sun.com/j2se/1.5.0/docs/tooldocs/findingclasses.html
-- Axel.Rauschmayer@ifi.lmu.de http://www.pst.ifi.lmu.de/~rauschma/
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On Tue, Jan 26, 2010 at 2:17 PM, Alessio Stalla alessiostalla@gmail.com wrote:
On Tue, Jan 26, 2010 at 1:21 PM, Axel Rauschmayer axel@rauschma.de wrote:
Incidentally, we may be able to autodiscover .asd files by scanning the classpath, and suitably update asdf:*central-registry*.
We first need to agree on a convention to wedge the concept of Pathname into CLASSPATH. I now support your earlier proposal to keep the two separate, as it makes more sense. And I'd like to get the "jar:" part of my changes into the trunk, and tested for a bit before introducing another the next step. Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
With Java resources, one is always warned against using the classpath that way. Looking for a particular path is possible, but listing paths (or files) not. It might be safer to introduce an additional concept. This could be an explicit list of Java resource paths or a new kind of ASDF classpath.
Scanning the entire "classpath" (or better, the resources known to classloaders) in general is impossible, since a classloader can create a resource from anywhere (e.g. from a buffer in memory, like ABCL itself does for runtime compilation).
However, it is possible to scan the classpath in the most common cases, i.e. when the resources are either on the filesystem or in some Jar loaded by a regular URLClassLoader.
I agree that doing it right, for some value of right, is far from easy as there are tricky cases (e.g. classloaders used by some application servers) that you might still want to handle.
FWIW, DWIMming the classpath doesn't seem worth the effort. If we can do directory inside and outside jars and get access to the classpath then people can easily build things on top of that.
-Alan
On Tue, Jan 26, 2010 at 6:54 AM, Mark Evenson evenson@panix.com wrote:
Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
Recall that I had a patch for that that could probably be easily adapted.
(advise directory (let* ((translated (translate-logical-pathname (car arglist))) (device (pathname-device translated))) (if (and device (eql 0 (search "jar:file:" (namestring device)))) (directory-in-jar translated) (:do-it))) :when :around :name jar-file)
(defun directory-in-jar (pathname) (let* ((jarfile (subseq (namestring (pathname-device pathname)) 9)) (rest-pathname (namestring (make-pathname :directory (pathname-directory pathname) :name (pathname-name pathname) :type (pathname-type pathname))))) (if (or (position #* (namestring rest-pathname)) (wild-pathname-p rest-pathname)) (let ((jar (jnew "java.util.zip.ZipFile" jarfile))) (let ((els (jcall "entries" jar))) (loop while (#"hasMoreElements" els) for name = (jcall "getName" (#"nextElement" els)) when (pathname-match-p (concatenate 'string "/" name) rest-pathname) collect (make-pathname :device (pathname-device pathname) :name (pathname-name name) :type (pathname-type name) :directory `(:absolute ,@(cdr (pathname-directory name))))))) (let ((truename (probe-file-in-jar pathname))) (if truename (list truename) nil)))))
-Alan
On 1/26/10 8:43 PM, Alan Ruttenberg wrote:
On Tue, Jan 26, 2010 at 6:54 AM, Mark Evensonevenson@panix.com wrote:
Right now, wildcards (and DIRECTORY) don't work inside JARs which would be the first step to autodiscovery.
Recall that I had a patch for that that could probably be easily adapted.
[…]
I haven't forgotten your code, and am studying it intensely.
Right now I have a nasty bug with Pathname.mergePathnames() sharing structure that shows up when using ASDF in JARs.
On 1/24/10 4:14 PM, Mark Evenson wrote: […]
Once I get things tested under Windows, and play around with the loading from http support, I intend to commit to trunk.
I've now relased an [updated patchset against trunk][1] that passes all tests under Windows.
In order to get the Windows tests to work, you'll need an post [svn r2402][2] version of SVN trunk.
Things I am still not entirely happy with
1) Unfortunately there are a lot of dangling references to open files in the code that will never be properly closed until ABCL exits. Since we are only reading files, this isn't too tragic, but under Windows with that q#*$Q(*! behavior of preventing files from being deleted if *anyone* has an open handle, can, for example, prevent a FASL from being recompiled.
2) I need to restabish the use of ZipCache, but I'm not sure it fits. It might help with tracking down problems to 1)
[1]: http://slack.net/~evenson/abcl/jar-file/abcl-jar-file-20100126.patch
[2]: http://code.google.com/p/abcl-dynamic-install/source/detail?r=bf9aa23d2841d7...
A [third version of patches][1] are now available to load from JARs in ABCL.
This version can load FASLs from JARs with a asdf:load-just-fasls-op (thanks Alan!) implementation (see web page for instructions).
In general this work seems to be approaching something I would merge to trunk with the following outstanding issues to consider:
1) There is a regression in failing logical pathname translation tests in ABCL.TEST.LISP having to do with improper case conversion. Probably introduced with the Pathname copy constructor I introduced to get around the MERGE-PATHNAME implementation not treating its arguments in final (which is currently a bug on trunk in certain conditions as well).
2) Dangling open InputStreams need to be closed
3) Possibly reestablish the use ZipCache, although I don't think its a win in its previous form. Maybe something using a timer thread to invalidate? Seems wrong.
4) Test the use of URLs in JAR pathnames
5) Wider testing with the ANSI test suite to make sure the Pathname behavior is at least as correct as the previous version.
[1]: http://slack.net/~evenson/abcl/jar-file/
2010/1/28 Mark Evenson evenson@panix.com:
- Possibly reestablish the use ZipCache, although I don't think its a
win in its previous form. Maybe something using a timer thread to invalidate? Seems wrong.
*shudder* starting threads without somebody asking for it is a big no-no for a library. I find the mention of it appalling.
On 1/28/10 1:36 PM, Ville Voutilainen wrote:
2010/1/28 Mark Evensonevenson@panix.com:
- Possibly reestablish the use ZipCache, although I don't think its a
win in its previous form. Maybe something using a timer thread to invalidate? Seems wrong.
*shudder* starting threads without somebody asking for it is a big no-no for a library. I find the mention of it appalling.
Yeah, yeah. I hear you loud and clear, as "appalling" certainly confirms my own aversion to this approach.
Can either you or Erik provide me with a before/after scenario that led Erik to code ZipCache as it stands? There seems to be little subjective difference that I can tell loading without ZipCache in my patches. Alesssio's information about the "hidden" ZipCache in the JVM would explain this absence of difference, but I would like to at least take a stab at "proving" that not using it in Load hasn't made a difference.
2010/1/28 Mark Evenson evenson@panix.com:
Can either you or Erik provide me with a before/after scenario that led Erik to code ZipCache as it stands? There seems to be little subjective
Erik did the zipcache code based on his measurements, so he knows best.
Hi.
On Thu, Jan 28, 2010 at 3:33 PM, Ville Voutilainen ville.voutilainen@gmail.com wrote:
2010/1/28 Mark Evenson evenson@panix.com:
Can either you or Erik provide me with a before/after scenario that led Erik to code ZipCache as it stands? There seems to be little subjective
Erik did the zipcache code based on his measurements, so he knows best.
The ZipCache was introduced to reduce the time required to load zipped .abcl files (the ones which are not embedded inside any other type of archive). The effect was established using a profiler; I'm not sure if it was measurable by just starting up the abcl executable.
HTH,
Erik.
On Thu, Jan 28, 2010 at 1:30 PM, Mark Evenson evenson@panix.com wrote:
A [third version of patches][1] are now available to load from JARs in ABCL.
This version can load FASLs from JARs with a asdf:load-just-fasls-op (thanks Alan!) implementation (see web page for instructions).
In general this work seems to be approaching something I would merge to trunk with the following outstanding issues to consider:
- There is a regression in failing logical pathname translation tests in
ABCL.TEST.LISP having to do with improper case conversion. Probably introduced with the Pathname copy constructor I introduced to get around the MERGE-PATHNAME implementation not treating its arguments in final (which is currently a bug on trunk in certain conditions as well).
- Dangling open InputStreams need to be closed
This seems the most dangerous issue. IIRC in some cases abcl has this problem even without your patch. However, it should be just a matter of putting some try-finally blocks in the right places.
- Possibly reestablish the use ZipCache, although I don't think its a win
in its previous form. Maybe something using a timer thread to invalidate? Seems wrong.
Since you used JarURLConnection to access resources inside JARs, you are already using the cache which is built in the JVM, and ZipCache is, or should be, superfluous. This cache is not mentioned in Java's standard API documentation, but it is a known feature of at least Sun's JVM; see for example http://www.szegedi.org/articles/remotejars.html. I expect this feature to be present in all sane JVMs (loading classes from remote Jars without it would be much slower).
Test the use of URLs in JAR pathnames
Wider testing with the ANSI test suite to make sure the Pathname
behavior is at least as correct as the previous version.
I can only say: thanks for your work, it is going to make abcl more deploy-friendly and that's a huge bonus.
Alessio
The [first release candidate of the JAR file support patches][1] are now available.
[1]: http://slack.net/~evenson/abcl/jar-file/
Notable changes in this version:
o The ASDF:LOAD-JUST-FASLS-OP has been replaced by the use of the normal ASDF:LOAD-OP verb for loading from JAR files. Now if the output file location of ASDF's compilation phase would be a JAR file, the compilation is simply skipped.
o Loading from network JARs now works
o Use of JAR pathnames in OPEN now signals an error. The dependency of ABCL's streams on java.io.File looks too strong to easily graft my changes in at this point, so I'd rather get the current code in rather than waiting even longer by adding new features.
I'm pretty happy with this version having cleaned up numerous bugs so I intend to commit it to the trunk unless there are vocal objections in the next few days.
The extensively reworked new implementation for specifiying jar pathnames has now been merged to trunk as [svn r12422][1].
[1]: http://trac.common-lisp.net/armedbear/changeset/12422
Excerpts from the commit message:
Pathname namestrings that have the form "jar:URL!/ENTRY" now construct references to the ENTRY within a jar file that is located by URL. The most common use is the "file:" form of URL (e.g. 'jar:file:/home/me/foo.jar!/foo.lisp') although any valid syntax accepted by the java.net.URL constructor should work (such as 'jar:http://abcl-dynamic-install.googlecode.com/files/baz.jar!/a/b/eek.lisp').
The internal structure of a jar pathname has changed. Previously a pathname with a DEVICE that was itself a pathname referenced a jar. This convention was not able to simultaneously represent bothjar entries that were themselves jar files as occurs with packed FASLs within JARs and devices which refer to drive letters under Windows. Now, a pathname which refers to a jar has a DEVICE which is a proper list of at most two entries. The first entry always references the "outer jar", and the second entry (if it exists) references the "inner jar". Casual users are encouraged not to manipulate the "internal structure" of jar pathname by setting its DEVICE directly, but instead rely on namestring <--> pathname conversions.
Jar pathnames are only currently valid for use with LOAD, TRUENAME, PROBE-FILE and pathname translation related functions (such as MERGE-PATHNAMES, TRANSLATE-PATHNAME, etc.) Passing one to OPEN currently signals an error. Jar pathnames do not currently work with DIRECTORY or PROBE-DIRECTORY.
Jar pathnames work for ASDF systems packaged within JARs. We override ASDF:LOAD-OP to load ASDF from JAR Pathnames by bypassing compilation if the output location would be in a JAR file. Interaction with ASDF-BINARY-LOCATIONS is currently untested.
Pathname now used as the basis of ABCL's internal routines for loading FASLs replacing the use of strings, which simplifies a lot of the behavior in looking for things to LOAD.
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Alessio Stalla -
Axel Rauschmayer -
Erik Huelsmann -
Mark Evenson -
Ville Voutilainen