[armedbear-devel] packaging external libs with abcl
I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies? Thanks, Matt -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.
On Fri, Sep 11, 2009 at 3:20 PM, Matthew D. Swank <akopa@charter.net> wrote:
I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies?
This has been somewhat discussed in the thread "[OT] best way to structure and deploy a project" which I started. What I ended up doing is to use ANT to build my system, both the Java part (natively) and the Lisp part, by making it invoke ABCL with --load compile-system.lisp, where compile-system.lisp is (push such-and-such asdf:*central-registry*) (asdf:oos 'asdf:compile-op :the-system). ANT packages it all in a jar, with a directory structure such as, e.g., system-name/ bin/ --classes, .lisp, fasls lib/ --lisp libraries This is similar to what ABCL itself does (in fact, my ant file is basically a trimmed down version of abcl's), except abcl has no library dependencies. Now, afaik asdf can only load from files, so the jar contains a main class written in Java that detects if it's been launched from a jar, and if it is, it extracts the jar in a temporary directory and uses asdf to load the system from there. This of course cannot work on AppEngine; maybe the patch to load from jars addresses this issue as well? hth, Alessio
On Fri, 11 Sep 2009 16:25:07 +0200 Alessio Stalla <alessiostalla@gmail.com> wrote:
On Fri, Sep 11, 2009 at 3:20 PM, Matthew D. Swank <akopa@charter.net> wrote:
I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies?
...
Now, afaik asdf can only load from files, so the jar contains a main class written in Java that detects if it's been launched from a jar, and if it is, it extracts the jar in a temporary directory and uses asdf to load the system from there. This of course cannot work on AppEngine; maybe the patch to load from jars addresses this issue as well?
Well whether it's in a jar, or on the (r/o) file system, the classes in the .abcl files have to loaded in to the "image" at runtime, correct? I could just make the precompiled lisp files available on the file system, and let asdf load them in my app entry point. Also, if I needed the files to be packaged as jars, or if I didn't want the overhead of asdf searching the system dependencies, I could capture the load order that asdf generates when it compiles the systems and load them in proper order directly. Of course this all gets a bit messy if library code depends on the asdf systems being available at runtime. Also, can all this happen fast enough to serve a request? Matt -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.
On 9/20/09 7:32 AM, Matthew D. Swank wrote: […]
I could just make the precompiled lisp files available on the file system, and let asdf load them in my app entry point. Also, if I needed the files to be packaged as jars, or if I didn't want the overhead of asdf searching the system dependencies, I could capture the load order that asdf generates when it compiles the systems and load them in proper order directly.
Loading ASDF packaged code from the filesystem
Of course this all gets a bit messy if library code depends on the asdf systems being available at runtime. Also, can all this happen fast enough to serve a request?
Why not package the library code in ASDF itself, using ASDF's dependency mechanism to pull in the necessary code? For packaging JARs with Java code as ASDF entries, [Alan Ruttenberg's JSS extension][1] allows the specification of JAR files in ADSF under ABCL. The [following snippet][asdf-jar] from private unreleased work of my own, wraps the IRIS WSML reasoner jars necessary to work with in ABCL with a thin Lisp bridge. As for "can all this happen fast enough to server a request", assuming your application is for user response time for HTTP applications (i.e. on the order of 100ms), the answer is probably no if every request is going to be a "cold start." But this is more a limitation that any JVM-based application faces which is commonly architected around by having workers "warmed up" to the point of serving a request waiting for incoming requests. Practically all modern HTTP applications use some sort of warm-up. [asdf-jar]: (defsystem :wsml2reasoner-jars :version "0.6.4" ;; last sync with SVN :depends-on (:abcld) :components ((:module wsml2reasoner :pathname "lib/" :components ((:jar-file "wsml2reasoner"))) (:module iris-libs :pathname "lib/ext/iris-reasoner/iris/" :components ((:jar-file "iris-0.58"))) (:module jgrapht-libs :pathname "lib/ext/iris-reasoner/jgrapht/" :components ((:jar-file "jgrapht-jdk1.5-0.7.1"))) (:module wsmo-libs :pathname "lib/ext/wsmo/" :components ((:jar-file "WSML-grammar-20081202") (:jar-file "wsmo-api-0.6.2") (:jar-file "wsmo4j-0.6.2"))) (:module log4j-libs :pathname "lib/ext/log4j/" :components ((:jar-file "log4j-1.2.14"))))) [1]: http://mumble.net:8080/svn/lsw/trunk/ -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Sun, 20 Sep 2009 10:12:48 +0200 Mark Evenson <evenson@panix.com> wrote:
Why not package the library code in ASDF itself, using ASDF's dependency mechanism to pull in the necessary code? For packaging JARs with Java code as ASDF entries, [Alan Ruttenberg's JSS extension][1] allows the specification of JAR files in ADSF under ABCL. The [following snippet][asdf-jar]
Doesn't this still extract the components in the jar to the file system (I am trying to do this on the appengine), or does this use the new jar path functionality (e.g. (load "jar:file:///PATH/TO.jar!/foo")) in rev-12141? Matt -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.
On 9/20/09 6:32 PM, Matthew D. Swank wrote:
On Sun, 20 Sep 2009 10:12:48 +0200 Mark Evenson<evenson@panix.com> wrote:
Why not package the library code in ASDF itself, using ASDF's dependency mechanism to pull in the necessary code? For packaging JARs with Java code as ASDF entries, [Alan Ruttenberg's JSS extension][1] allows the specification of JAR files in ADSF under ABCL. The [following snippet][asdf-jar]
Doesn't this still extract the components in the jar to the file system (I am trying to do this on the appengine), or does this use the new jar path functionality (e.g. (load "jar:file:///PATH/TO.jar!/foo")) in rev-12141?
No, JSS loads code straight from the JAR files, currently it can only reference Java classes. JSS uses the Bean Shell custom classpath loader to dynamically add JAR files to the classpath as represented Lisp-side as CL-USER:*ADDED-TO-CLASSPATH*. What that syntax shows is ABCL under JSS dynamically modifying the classpath to include additional JARs after starting up (in my example, JARs containing a semantic web reasoner from STI Innsbruck known as IRIS). Practically what this means is that one just has to specify abcl.jar, jscheme.jar, and bsh-2.0b4.jar on ABCL's initial classpath, after which you can programatically add additional JARs via ASDF systems. This is very useful when using ABCL to "wrap" Java libraries with a Lisp layer. Since you indicate in a separate post that Google App Engine does not allow even read-only access to the filesystem, I suspect that this won't help with your problem. N.B. there is a Ant build property named 'additional.jars' which includes additional JARs when creating the ABCL wrapper to facilitate this kind of interaction if you find yourself *always* needing to refer to JAR files from ABCL. I usually create a 'build.properties' file with references to my local copies 'jscheme.jar' and 'bsh-2.04.jar' in my top-level ABCL build directory. That way, when I need the JSS extension, I can just load it in via ASDF. [svn 12141] allows one to start referring to Lisp code contained in JARs other than 'abcl.jar', but I need to get a few more improvements into ABCL before it becomes a reasonable method to use ASDF to wrap Lisp code contained in JARs. The first improvement would be to allow MERGE-PATHNAMES to work with jar:file pathnames so one could specify relative pathnames to package an ASDF system within a JAR file. The second improvement would to be use wildcard jar:file pathnames to specify things like "the first JAR in the CLASSPATH which has a '!/foo.asd' entry". The third thing would probably be to create an ABCL custom classpath loader like that provided in Bean Shell (and used by JSS as CL-USER:*ADDED-TO-CLASSPATH*) so that dynamically adding references to JARs while running ABCL would be easy. After all this, I think deploying ABCL applications in JAR files would be a lot simpler. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Mon, 21 Sep 2009 12:11:58 +0200 Mark Evenson <evenson@panix.com> wrote:
On 9/20/09 6:32 PM, Matthew D. Swank wrote:
On Sun, 20 Sep 2009 10:12:48 +0200 Mark Evenson<evenson@panix.com> wrote:
Why not package the library code in ASDF itself, using ASDF's dependency mechanism to pull in the necessary code? For packaging JARs with Java code as ASDF entries, [Alan Ruttenberg's JSS extension][1] allows the specification of JAR files in ADSF under ABCL. The [following snippet][asdf-jar]
Doesn't this still extract the components in the jar to the file system (I am trying to do this on the appengine), or does this use the new jar path functionality (e.g. (load "jar:file:///PATH/TO.jar!/foo")) in rev-12141?
... [blah blah JSS ... blah blah build properties ... blah bah jar head] Java infrastructure incantations hurt my head. Matt -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.
On Fri, 11 Sep 2009 16:25:07 +0200 Alessio Stalla <alessiostalla@gmail.com> wrote:
On Fri, Sep 11, 2009 at 3:20 PM, Matthew D. Swank <akopa@charter.net> wrote:
I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies?
This has been somewhat discussed in the thread "[OT] best way to structure and deploy a project" which I started. What I ended up doing is to use ANT to build my system, both the Java part (natively) and the Lisp part, by making it invoke ABCL with --load compile-system.lisp, where compile-system.lisp is (push such-and-such asdf:*central-registry*) (asdf:oos 'asdf:compile-op :the-system). ANT packages it all in a jar, with a directory structure such as, e.g.,
system-name/ bin/ --classes, .lisp, fasls lib/ --lisp libraries
This is similar to what ABCL itself does (in fact, my ant file is basically a trimmed down version of abcl's), except abcl has no library dependencies. Now, afaik asdf can only load from files, so the jar contains a main class written in Java that detects if it's been launched from a jar, and if it is, it extracts the jar in a temporary directory and uses asdf to load the system from there. This of course cannot work on AppEngine; maybe the patch to load from jars addresses this issue as well?
I must have miss read the doc, I don't even have r/o access to the file system. I'm not sure how I'd bootstrap an abcl application on the google app engine. java.security.AccessControlException: access denied (java.io.FilePermission /home/singollo/root/appengine-proj/abcl-test/war/load.lisp read) at java.security.AccessControlContext.checkPermission(AccessControlContext.java:323) at java.security.AccessController.checkPermission(AccessController.java:546) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:139) at java.lang.SecurityManager.checkRead(SecurityManager.java:871) at java.io.File.isFile(File.java:776) at org.armedbear.lisp.Load.findLoadableFile(Load.java:67) at org.armedbear.lisp.Load.load(Load.java:132) at org.armedbear.lisp.Load.load(Load.java:728) at org.armedbear.lisp.Load.access$200(Load.java:50) at org.armedbear.lisp.Load$2.execute(Load.java:680) at org.armedbear.lisp.Symbol.execute(Symbol.java:816) at org.armedbear.lisp.LispThread.execute(LispThread.java:563) at org.armedbear.lisp.load_1.execute(load.lisp:33) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:91) at org.armedbear.lisp.LispThread.execute(LispThread.java:511) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:489) at org.armedbear.lisp.Lisp.eval(Lisp.java:454) at org.armedbear.lisp.Lisp.eval(Lisp.java:452) at org.armedbear.lisp.Lisp.eval(Lisp.java:395) at org.armedbear.lisp.Interpreter.eval(Interpreter.java:156) -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.
On 9/21/09 12:59 AM, Matthew D. Swank wrote:
On Fri, 11 Sep 2009 16:25:07 +0200 Alessio Stalla<alessiostalla@gmail.com> wrote:
On Fri, Sep 11, 2009 at 3:20 PM, Matthew D. Swank<akopa@charter.net> wrote:
I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies?
This has been somewhat discussed in the thread "[OT] best way to structure and deploy a project" which I started. What I ended up doing is to use ANT to build my system, both the Java part (natively) and the Lisp part, by making it invoke ABCL with --load compile-system.lisp, where compile-system.lisp is (push such-and-such asdf:*central-registry*) (asdf:oos 'asdf:compile-op :the-system). ANT packages it all in a jar, with a directory structure such as, e.g.,
system-name/ bin/ --classes, .lisp, fasls lib/ --lisp libraries
This is similar to what ABCL itself does (in fact, my ant file is basically a trimmed down version of abcl's), except abcl has no library dependencies. Now, afaik asdf can only load from files, so the jar contains a main class written in Java that detects if it's been launched from a jar, and if it is, it extracts the jar in a temporary directory and uses asdf to load the system from there. This of course cannot work on AppEngine; maybe the patch to load from jars addresses this issue as well?
I must have miss read the doc, I don't even have r/o access to the file system. I'm not sure how I'd bootstrap an abcl application on the google app engine.
I think one has to package everything inside 'abcl.jar', placing your code in the 'org.armedbear.lisp' directory. This should be what *LISP-HOME* gets set to. I have yet to get enough time to figure out how to deploy to Google App Engine (GAE) myself: it's on a long TODO list which got pushed further down with having to figure out how to use the Eclipse plugin to deploy to GAE. -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
participants (3)
-
Alessio Stalla -
Mark Evenson -
Matthew D. Swank