Hello everyone,
I have a question which is not exactly pertinent with the development of ABCL, but which imho can start a useful discussion. Please forgive me if this is too OT.
I have a project with sources in two languages, Java and Lisp. It is an Eclipse project, but the IDE does not make difference. The project is a library, not an end-user application.
Currently it is structured as follows:
src/java - Java source code src/lisp - Lisp source code lib/ - third-party libraries, both Java (jars) and Lisp (directories).
The Java code is compiled by the IDE and put in an output directory. The Lisp source files are treated by the IDE as normal, non-source files, and copied verbatim into the same ouput folder as the Java classes.
When I run the project from the IDE, I launch a class which compiles and loads the Lisp systems using asdf. This works for me locally, but thinking about distributing the project to users, this approach has some drawbacks:
- Lisp files must be true files, they can't be in a Jar (asdf doesn't understand jars). - Lisp files are recompiled every time, since the IDE always overwrites the Lisp source files in the output directory. - If I make a jar, it will contain Lisp source files, not binaries, since binaries are unknown to the IDE.
Ideally I'd like to do what ABCL does, i.e. create one big jar containing .class files and compiled Lisp files. Lisp sources could be compiled by the IDE or by a script, I don't care. Still, I would like to maintain the use of asdf (since some of my dependencies use it and loading them manually is too much of a hassle). I don't know how I can do this.
I have also thought about leaving everything as-is, and when I'll distribute the library as a jar, have it extract at runtime the Lisp source files from the jar in a temporary directory and instruct asdf to search for systems in that directory. I will also need the lib/ folder to be in a known place, since asdf has to load stuff from there too (but this is less of a problem since I can make the library user configure that place, and use a sensible default otherwise). I am wondering if a cleaner approach exists, and if someone has experience in this regard.
Cheers, Alessio
2009/8/24 Alessio Stalla alessiostalla@gmail.com:
When I run the project from the IDE, I launch a class which compiles and loads the Lisp systems using asdf. This works for me locally, but thinking about distributing the project to users, this approach has some drawbacks:
- Lisp files must be true files, they can't be in a Jar (asdf doesn't
understand jars).
- Lisp files are recompiled every time, since the IDE always
overwrites the Lisp source files in the output directory.
- If I make a jar, it will contain Lisp source files, not binaries,
since binaries are unknown to the IDE.
Create an ANT project, that allows you to do almost anything. Many java IDEs will be able to build with the ANT project (without trying to do their own clever, or not so clever wizardry), thus allowing all the flexibility that ANT allows.
On Mon, Aug 24, 2009 at 5:18 PM, Ville Voutilainenville.voutilainen@gmail.com wrote:
2009/8/24 Alessio Stalla alessiostalla@gmail.com:
When I run the project from the IDE, I launch a class which compiles and loads the Lisp systems using asdf. This works for me locally, but thinking about distributing the project to users, this approach has some drawbacks:
- Lisp files must be true files, they can't be in a Jar (asdf doesn't
understand jars).
- Lisp files are recompiled every time, since the IDE always
overwrites the Lisp source files in the output directory.
- If I make a jar, it will contain Lisp source files, not binaries,
since binaries are unknown to the IDE.
Create an ANT project, that allows you to do almost anything. Many java IDEs will be able to build with the ANT project (without trying to do their own clever, or not so clever wizardry), thus allowing all the flexibility that ANT allows.
Ok, good advice: this way I can compile the Java files first, then have ANT launch ABCL and compile my system with ASDF, and then package everything in a jar (depending on the target). However, when distributing the compiled library, I will still need code to load the Lisp parts (e.g. by extracting them and using asdf), am I correct? It's not a big deal actually... too bad ANT doesn't understand sexps, only xml ;)
Ale
On Aug 24, 2009, at 11:14, Alessio Stalla wrote:
- Lisp files must be true files, they can't be in a Jar (asdf doesn't
understand jars).
Perhaps ABCL should support CL pathnames referring to jar entries? You might still have to extend ASDF to deal with compiled-but-no-sources, but it would be better-integrated.
Last I checked, asdf can already load files from the jar, but apparently only from one folder. I tried to fix this at one point, but didn't understand the details of how it worked well enough.
-Alan
On Mon, Aug 24, 2009 at 9:10 AM, Kevin Reidkpreid@mac.com wrote:
On Aug 24, 2009, at 11:14, Alessio Stalla wrote:
- Lisp files must be true files, they can't be in a Jar (asdf doesn't
understand jars).
Perhaps ABCL should support CL pathnames referring to jar entries? You might still have to extend ASDF to deal with compiled-but-no-sources, but it would be better-integrated.
-- Kevin Reid http://switchb.org/kpreid/
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
"ar" == Alan Ruttenberg alanruttenberg@gmail.com writes:
ar> ar> Last I checked, asdf can already load files from the jar, but ar> apparently only from one folder. I tried to fix this at one point, but ar> didn't understand the details of how it worked well enough.
Indeed, ABCL can only load from the org/armedbear/lisp directory. The following patch to ABCL makes it possible to load from an arbitrary directory of the jar file.
http://article.gmane.org/gmane.lisp.armedbear.devel/62
Can this be pulled into the current abcl? -Alan
On Mon, Aug 24, 2009 at 10:10 AM, Eric Marsdeneric.marsden@free.fr wrote:
"ar" == Alan Ruttenberg alanruttenberg@gmail.com writes:
ar> ar> Last I checked, asdf can already load files from the jar, but ar> apparently only from one folder. I tried to fix this at one point, but ar> didn't understand the details of how it worked well enough.
Indeed, ABCL can only load from the org/armedbear/lisp directory. The following patch to ABCL makes it possible to load from an arbitrary directory of the jar file.
http://article.gmane.org/gmane.lisp.armedbear.devel/62
-- Eric Marsden
On 8/24/09 7:13 PM, Alan Ruttenberg wrote:
Can this be pulled into the current abcl?
Seems quite reasonable: I'll try to look into it over the course of today. Sorry I missed this useful looking patch earlier in the armedbear development thread.
On Mon, Aug 24, 2009 at 7:10 PM, Eric Marsdeneric.marsden@free.fr wrote:
"ar" == Alan Ruttenberg alanruttenberg@gmail.com writes:
ar> ar> Last I checked, asdf can already load files from the jar, but ar> apparently only from one folder. I tried to fix this at one point, but ar> didn't understand the details of how it worked well enough.
Indeed, ABCL can only load from the org/armedbear/lisp directory. The following patch to ABCL makes it possible to load from an arbitrary directory of the jar file.
I was aware of this patch, however I don't know asdf well enough...
Suppose I have:
/home/alessio/my.jar my/ my.asd a.lisp b.lisp
Where system.asd is something like
(asdf:defsystem :my (:components ((:file "a") (:file "b"))))
Then, can I (push #P"jar:file:/home/alessio/my.jar!my/" asdf:*central-registry*) and have it Just Work with (asdf:oos 'asdf:load-op :my)?
Ale
On 8/24/09 7:10 PM, Eric Marsden wrote:
Indeed, ABCL can only load from the org/armedbear/lisp directory. The following patch to ABCL makes it possible to load from an arbitrary directory of the jar file.
http://article.gmane.org/gmane.lisp.armedbear.devel/62
Patching trunk via the attached patch, trying for a simple case of a file "foo.lisp" containing the form
(defun foo () (format t "sdfsadfsdf"))
compiled with
(let ((sys::*compile-file-zip* t)) (compile-file "foo.lisp" :output-file "foo.jar"))
leads to the following stack trace. I have run out of time right now to investigate more thoroughly, but will get back to this when I can later.
0: (INVOKE-DEBUGGER #<ERROR {CDFD504}>) 1: org.armedbear.lisp.Lisp.error(Lisp.java:348) 2: org.armedbear.lisp.Lisp.loadCompiledFunction(Lisp.java:1136) 3: org.armedbear.lisp.CompiledClosure$1.execute(CompiledClosure.java:221) 4: org.armedbear.lisp.LispThread.execute(LispThread.java:511) 5: org.armedbear.lisp.Lisp.evalCall(Lisp.java:488) 6: org.armedbear.lisp.Lisp.eval(Lisp.java:453) 7: org.armedbear.lisp.Lisp.evalCall(Lisp.java:490) 8: org.armedbear.lisp.Lisp.eval(Lisp.java:453) 9: org.armedbear.lisp.Load.faslLoadStream(Load.java:550) 10: org.armedbear.lisp.Load.access$100(Load.java:47) 11: org.armedbear.lisp.Load$1.execute(Load.java:408) 12: org.armedbear.lisp.LispThread.execute(LispThread.java:528) 13: org.armedbear.lisp.Lisp.evalCall(Lisp.java:495) 14: org.armedbear.lisp.Lisp.eval(Lisp.java:453) 15: org.armedbear.lisp.Load.loadStream(Load.java:518) 16: org.armedbear.lisp.Load.loadFileFromStream(Load.java:479) 17: org.armedbear.lisp.Load.load(Load.java:192) 18: org.armedbear.lisp.Load.load(Load.java:680) 19: org.armedbear.lisp.Load.access$200(Load.java:47) 20: org.armedbear.lisp.Load$2.execute(Load.java:632) 21: org.armedbear.lisp.Symbol.execute(Symbol.java:816) 22: org.armedbear.lisp.LispThread.execute(LispThread.java:563) 23: org.armedbear.lisp.load_1.execute(load.lisp:33) 24: org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:91) 25: org.armedbear.lisp.LispThread.execute(LispThread.java:511) 26: org.armedbear.lisp.Lisp.evalCall(Lisp.java:488) 27: org.armedbear.lisp.Lisp.eval(Lisp.java:453) 28: org.armedbear.lisp.Primitives$16.execute(Primitives.java:305) 29: (SYSTEM:LOAD-COMPILED-FUNCTION "foo-1.cls") 30: (SYSTEM:INIT-FASL :VERSION 32) 31: (SYSTEM::%LOAD #P"jar:file:/Users/evenson/work/abcl/foo.jar!/foo" NIL NIL T) 32: (LOAD "jar:file:/Users/evenson/work/abcl/foo.jar!/foo") 33: (SYSTEM::%EVAL (LOAD "jar:file:/Users/evenson/work/abcl/foo.jar!/foo")) 34: (EVAL (LOAD "jar:file:/Users/evenson/work/abcl/foo.jar!/foo"))
Attached is a patch that allows LOAD to work on arbitrary JAR files. It uses the semi-official ([according to Wikipedia][1] and as implemented in java.net.URL) URI Schema:
jar:file:PATH!/JAR_ENTRY
where "file:PATH" is a filepath URI (meaning that absolute pathnames need to have a double leading slash, i.e. '//var/tmp/foo.jar'), and JAR_ENTRY is the path in the jar file.
I never could get Eric Marsden's patch to work, as it would load the initial FASL entry (the one ending in '._' containing the load instructions), but not the compiled classes. It bothers me a little that following his instructions
Next steps:
1) making this work for JARs containing "foo.abcl" zipped FASLs which are themselves zipped collections of a single FASL (ABCL overloads the ".abcl" suffix to mean either a initial FASL OR a zipped collection of the initial FASL plus associated compiled toplevel forms). Such zipped FASLs would be the natural unit for dealing with ABCL compile artifacts, right?
2) Developing packaging utilities for putting together compilation units. This would probably not be so necessary if step 1) can be made to work, right?
3) Check how this works under Microsoft Windows.
Question:
A) There probably should be some way to load all the FASLs contained in a JAR. Would
(load "jar:file:foo.jar")
work as a syntax with the semantics of loading all the compiled classes that one can find in the associated JAR?
B) I have deviated a little from Eric's original idea to bundle additional classes with the current abcl.jar for distribution, into having additional JARs with the extra classes. If anything on the path that I am going does not meet up with use case expectations, please let me know.
Bugs:
I) Currently the code has a bug that if a file whose name is a duplicate of one in the initial abcl.jar, it will not be loaded.
[1]: http://en.wikipedia.org/wiki/URI_scheme#Unofficial_but_common_URI_schemes
On 8/26/09 2:43 PM, Mark Evenson wrote:
I never could get Eric Marsden's patch to work, as it would load the initial FASL entry (the one ending in '._' containing the load instructions), but not the compiled classes. It bothers me a little that following his instructions
… didn't work for me, making me wonder if I missed something. Did anyone else get his patch to work, or can Eric comment?
Thanks, Mark
On Wed, Aug 26, 2009 at 2:43 PM, Mark Evensonevenson@panix.com wrote:
Attached is a patch that allows LOAD to work on arbitrary JAR files. It uses the semi-official ([according to Wikipedia][1] and as implemented in java.net.URL) URI Schema:
jar:file:PATH!/JAR_ENTRY
where "file:PATH" is a filepath URI (meaning that absolute pathnames need to have a double leading slash, i.e. '//var/tmp/foo.jar'), and JAR_ENTRY is the path in the jar file.
Looks nice. One piece of advice, if it can be useful to you: since the file:PATH part is an URL, you could use URL.openStream combined with ZipInputStream: that way all the quirks about paths on different OSes, URL-encoded characters, and the like would be handled automatically by the Java standard library.
Also, how does this patch behave with recursive loading - i.e. I load x from file.jar and x contains (load "y"), is "y" relative to x?
Alessio
On Aug 26, 2009, at 8:55, Alessio Stalla wrote:
Also, how does this patch behave with recursive loading - i.e. I load x from file.jar and x contains (load "y"), is "y" relative to x?
CL:LOAD does not rebind *default-pathname-defaults*. The proper way for a file to do this (insofar as it's appropriate to do at all) is:
(load (merge-pathnames #p"y" *load-truename*))
ABCL's jar file support should be made such that this works.
On Wed, Aug 26, 2009 at 3:25 PM, Kevin Reidkpreid@mac.com wrote:
On Aug 26, 2009, at 8:55, Alessio Stalla wrote:
Also, how does this patch behave with recursive loading - i.e. I load x from file.jar and x contains (load "y"), is "y" relative to x?
CL:LOAD does not rebind *default-pathname-defaults*. The proper way for a file to do this (insofar as it's appropriate to do at all) is:
(load (merge-pathnames #p"y" *load-truename*))
Right, sorry - I was thinking more of asdf, which loads everything relative to the location of the .asd file, and if that is in a jar, ...
ABCL's jar file support should be made such that this works.
Ale
----- "Mark Evenson" evenson@panix.com a écrit :
I never could get Eric Marsden's patch to work, as it would load the initial FASL entry (the one ending in '._' containing the load instructions), but not the compiled classes. It bothers me a little that following his instructions
I looked at my patch again, and don't understand how it ever worked; it doesn't handle the LOAD-COMPILED-FILE aspect of things. Probably during my quick testing I had some files lying around in the org/armedbear/lisp directory.
Thanks for fixing it; it should improve application startup times enormously.
Eric
I just noticed that this patch breaks SLIME, so it definitely needs more work around the corner cases.
Attached is a revised patch for loading ABCL FASLs from jar files.
Now loading unpacked and packed FASLs work, so it is considerably easier to package things.
To use the packed FASL variant:
0) For 'bar.lisp' containing
(defun foo () (format t ("Foo here!"))
1) Compile normally with COMPILE-FILE
CL-USER(1): (compile-file "bar.lisp")
2) Package the resulting packed FASL ('*.abcl') with JAR (or zip):
cmd$ jar cfv baz.jar bar.abcl
You can add as many packed FASLs to the jar archive as you want.
To load issue
CL-USER(1): (load "jar:file:baz.jar!/bar")
and voila!
CL-USER(2): (foo) Foo here! NIL
Note that you can't currently specify "bar.abcl" explicitly in the load form (a bug).
The code needs some refactoring love, and much further testing, but I wanted to show some progress to get feedback. Other than both Load.load() and Lisp.loadCompiledFunction() getting seriously hard to understand, there is a conceptual problem with expressing "the sub-entry of a zipped entry of a zip file" with Lisp pathnames that if someone could suggest a way around, it would clear things up. ABCL uses the trick that if a pathname has a device component and the device is actually a pathname, then the entire pathname represents a entry in a zip file with device refering to the enclosing zip file, with the enclosing pathname directory, name, and type components identifying which entry within the zip file. If we could figure out a way to specify the sub-entry of an entry it would help out the abstraction. Does it make sense to use the device component recursively something like:
pathname--+-->device["jar:file:baz.jar"]-->device["jar:file:bar.abcl"] | +-->name["bar-1.cls"]
I think that helps, but it's late, so I'll check back later to see if this still makes sense.
Mark
Attached is a further revision of the patch for loading Lisp from JAR files. This version works in about all the cases we need to support. This feature will be one of the first to hit to 0.17 trunk when we branch, because I don't want to commit such a potentially destabilizing change in the week before 0.16.
This patch includes additions to the ABCL test suite for this change, since there are enough corner cases that I seem to miss testing them occasionally in writing this code. Working with our test suite via ASDF is not currently the most obvious thing in the world, so some notes on that: To get the tests to work, get a REPL that loads the ASDF "abcl-test-lisp" system, and run the tests once (needed to bind *this-directory* variable). Often one must tell the ASDF methods that they must be 'forced' to get to work, so use the 'force-system-test' via SLIME or pass the ':force t' option if you are issuing commands directly from the REPL. After the "abcl-test-lisp" system has run once, issue na (ABCL-TEST:REM-ALL-TESTS) in that REPL then load the "test/lisp/abcl/load.lisp" file, which will create a JAR ('baz.jar') that the tests work on (only works under *NIX for the moment, sorry). Then, one get issue (ABCL-TEST:DO-TESTS) to run the testing of LOAD specific tests.
Some notes on our use RT package, the basis for the ABCL Lisp test suite: RT lacks a mechanism for expressing "build-up/tear-down" abstractions that I am familiar with in test suites. Included in this would be an easy way to specify a value for *DEFAULT-PATHNAME-DEFAULTS* when the tests are run (I provide this in the ABCL-TEST-LISP:RUN method, but there should be a way to specify per test so that individual tests can be run easier when developing).
All the cases in the test suite currently work except for LOAD.9, which tries to explicitly load a "*.lisp" file from a JAR. I need to refactor the Load.findLoadableFile() to handle working in JAR files.
My work on [loading Lisp from JAR files has been committed to trunk][1].
With this modification, you may now package up the results of COMPILE-FILE (the packed FASLs ending in '.abcl') in a JAR file for loading with ABCL.
For example,
(load "jar:file:///PATH/TO/the.jar!/foo")
would load the forms contained in "foo.abcl" packaged in the "/PATH/TO/the.jar" (relative paths for the 'file:' URI work as well).
[1]: http://trac.common-lisp.net/armedbear/changeset/12141
Test code has been included, but currently only works under UNIX due to needing to pack the JAR file for testing via an external script.
Further things that should be eventually be addressed (by me, I presume):
1) The Load.load() function could use some re-factoring. This has proved to be rather difficult (or maybe I am just not thinking correctly in Java anymore), as my refactorings keep turning out longer and *more* convoluted than what is present. I wanted to get the functionality out there with some tests as it seems the community (i.e. Alan) would like to use this sooner rather than later.
2) Load from JARs not on local filesystem ("jar:http://here.we.go/application.jar").
3) Test what the *load-truename* variable gets bound to a little more rigorously
armedbear-devel@common-lisp.net
-
Alan Ruttenberg -
Alessio Stalla -
Eric Marsden -
eric.marsden＠free.fr
-
Kevin Reid -
Mark Evenson -
Ville Voutilainen