Hello,
I'm almost done with my small experiment:
* I'm loading lisp code via a Java file using Load.loadSystemFile in an init method
* I'm adding the file file to *lisp-home* via gradle
* I'm using gradle tasks to create a single JAR with abcl and abcl-contrib added (not as JAR files)
Note that I need to have only one jar file and with specific requirements around the main method; the above allows me to have it working fine, a single JAR which is essentially standalone.
My only remaining doubt (apart from other possible solutions which could also work and be more elegant, I'm open to suggestions of course) is around loading abcl-contrib.jar or more specifically the included packages (JSS and JFLI specifically). I've spent some hours tracking the loading mechanism and learned a bit about the process but I'm still missing some pieces.
My current solution (which works) is:
(require :abcl-contrib) (pushnew (make-pathname :device (pathname-device *lisp-home*) :directory '(:absolute "jss")) asdf:*central-registry*) (require :jss)
This adds #P"jar:file:/foo/myapp.jar!/jss" to ASDF, but I would like something more direct.
With my setup (system::find-contrib) returns NIL; I've tried to add the abcl-contrib.jar to myapp.jar (which is in the CLASSPATH of course) but the code on abcl-contrib.lisp (and specifically find-contrib) doesn't seem to seach inside a jar for another one. I also tried to use something like
(system::add-contrib (make-pathname :device (pathname-device *lisp-home*) :directory '(:absolute "abcl-contrib.jar")))
to see if it would pick up the jar but no luck.
I think there is something obvious I'm missing, the pieces are all there (the jar handling code works fine, etc) but I can't seem to find the right way to put them together.
Best regards,
On 4/29/17 21:48, Frederico Munoz wrote:
Hello,
I'm almost done with my small experiment:
I'm loading lisp code via a Java file using Load.loadSystemFile in an init method
I'm adding the file file to *lisp-home* via gradle
I'm using gradle tasks to create a single JAR with abcl and abcl-contrib added (not as JAR files)
Note that I need to have only one jar file and with specific requirements around the main method; the above allows me to have it working fine, a single JAR which is essentially standalone.
If you could detail the requirements you have around the main method a little bit, that would help understand more exactly where you are heading. From reading between the lines based on your [comments around your Watson IoT example][1], I *think* you are trying to make a WAR that will deploy to Jetty, so I am going to take an educated guess that that is what you are trying to create.
[1]: https://developer.ibm.com/recipes/tutorials/watson-iot-with-common-lisp/
A couple pointers on making WAR with ABCL:
1) [abcl-servlet][2] provides infrastructure for making an example WAR archive which has ABCL-CONTRIB available. The WAR only works for Java Servlet containers that provide an "exploded WAR" deployment strategy, i.e. at deployment the WAR archive is unzipped onto the local filesystem.
[2]: https://bitbucket.org/easye/abcl-servlet
2) Mucking with Load.loadSystemFile() and *LISP-HOME* should not be necessary, as these are really meant to be internal interfaces to boot the Common Lisp environment. Of course, you are welcome to experiment, but it would be best not to rely on these interfaces if at all possible. The values for the logical pathname SYS *should* be the "right way" forward to identify what *LISP-HOME* points to. Logical pathnames are better as they easily allow us to "update" the directory with the runtime value without having to explicitly re-merge the whole slew of PATHNAME objects derived from the build-time value.
3) Gradle recipes are cool. We would like to eventually include Gradle support in the ABCL build mechanism as it seems the most progressive of current ecosystem (especially now that Groovy has matured a bit). We would like to "call into" Ant for parts of the build as the current mechanism for controlling the artifacts and contents of 'abcl.jar' and 'abcl-contrib.jar' reside there and duplication of code paths would incur maintenance overhead.
My only remaining doubt (apart from other possible solutions which could also work and be more elegant, I'm open to suggestions of course) is around loading abcl-contrib.jar or more specifically the included packages (JSS and JFLI specifically). I've spent some hours tracking the loading mechanism and learned a bit about the process but I'm still missing some pieces.
The as yet unreleased abcl-1.5.0 contains [a top-level Ant target 'abcl-aio.jar'][4] that creates a single archive containing both abcl.jar, abcl-contrib.jar, plus whatever has been added to the build-time file:abcl/contrib/ directory.
[4]: http://abcl.org/trac/browser/trunk/abcl/build.xml#L521
With [abcl-aio][] we introduced a mechanism that introspects the jar manifest for declarations of ASDF systems to load when (require :abcl-contrib) is issued.
[abcl-aio]: http://abcl.org/trac/changeset/14908
We intend for 'abcl-aio' to serve as the mechanism for the end-user/developer to craft custom jar files.
Separately, there has been a long-standing mechanism to [include arbitrary code at system boot in a 'system.lisp' file][5] that could plausibly act as a hook for system intitialization, but I have never needed it to get WAR archives loaded in Java Servlet contexts.
[5]: http://abcl.org/trac/browser/trunk/abcl/build.xml#L281
In short, if you were to build from ABCL trunk, I think you should have the mechanisms to do what you want. ABCL trunk (aka 'abcl-1.5.0-dev') is quite stable, essentially representing the current release candidate for abcl-1.5.0. I am still working on a couple things (Java 6 support for UIOP/RUN-PROGRAM, using ABCL-BUILD to install Ant and/or Maven via interactive restart, better tests for PATHNAME shennigans), but I keep the patches for this work separate from the ABCL trunk source until it passes tests.
[…]
I think there is something obvious I'm missing, the pieces are all there (the jar handling code works fine, etc) but I can't seem to find the right way to put them together.
ABCL-AIO isn't currently documented (a bug), so I would not consider it obvious. [ABCL-JAR][] essentially uses the same mechanism (i.e. pushing PATHNAMEs to ASDF:*CENTRAL-REGISTRY*) you are prototyping here, but of course you are trying to get ABCL-CONTRIB to work in the first place here so it won't be much help for the immediate task at hand.
[abcl-jar]: http://abcl.org/trac/browser/trunk/abcl/contrib/asdf-jar/README.markdown
Good luck, and lemme know what else you need, Mark
Hello,
Thank you for your comprehensive answer, much appreciated.
Mark Evenson evenson@panix.com wrote:
On 4/29/17 21:48, Frederico Munoz wrote:
Hello,
(...)
Note that I need to have only one jar file and with specific requirements around the main method; the above allows me to have it working fine, a single JAR which is essentially standalone.
If you could detail the requirements you have around the main method a little bit, that would help understand more exactly where you are heading. From reading between the lines based on your [comments around your Watson IoT example][1], I *think* you are trying to make a WAR that will deploy to Jetty, so I am going to take an educated guess that that is what you are trying to create.
Ahh, yes, I can see how it could appear that way, I should have been clearer but I didn't want to add to much information; currently I'm taking pieces from my recent articles in dW - Common Lisp and OpenWhisk - and using the ability to write Java functions to use ABCL:
https://console.ng.bluemix.net/docs/openwhisk/openwhisk_actions.html#openwhi...
So not a WAR but a compiled Java class with a certain main method (must receive and return a JsonObject), which mustn't depend on things which are not guaranteed to exist remotely.
A couple pointers on making WAR with ABCL:
- [abcl-servlet][2] provides infrastructure for making an example WAR
archive which has ABCL-CONTRIB available. The WAR only works for Java Servlet containers that provide an "exploded WAR" deployment strategy, i.e. at deployment the WAR archive is unzipped onto the local filesystem.
I actually read that during my investigations and it was exactly what would latter lead me to loadSystemFile(): I can't use loadLispResource() since the jar isn't unzipped at the server. This lead me to some initial strange errors because locally it works since I'm running it in a place where the directory structure supports it, and this is directly related to your second point.
- Mucking with Load.loadSystemFile() and *LISP-HOME* should not be
necessary, as these are really meant to be internal interfaces to boot the Common Lisp environment. Of course, you are welcome to experiment, but it would be best not to rely on these interfaces if at all possible. The values for the logical pathname SYS *should* be the "right way" forward to identify what *LISP-HOME* points to. Logical pathnames are better as they easily allow us to "update" the directory with the runtime value without having to explicitly re-merge the whole slew of PATHNAME objects derived from the build-time value.
I'm using loadSystemFile() exactly because I need to work within the constraints of an (unzipped) jar file, and after some time trying to find how to make loadLispResources() work for files within the jar I found http://www.didierverna.net/blog/index.php?post/2011/01/22/Towards-ABCL-Stand... which allowed me to have it working. *LISP-HOME* was the first thing I found that could be used when I started looking around for an answer to "how do I get the path of the jar file so that I can use it to load abcl-contrib or the included packages?".
I tried finding information on the "logical pathname SYS" without success, if you could point to the right documentation about this it would be great.
- Gradle recipes are cool. We would like to eventually include Gradle
support in the ABCL build mechanism as it seems the most progressive of current ecosystem (especially now that Groovy has matured a bit). We would like to "call into" Ant for parts of the build as the current mechanism for controlling the artifacts and contents of 'abcl.jar' and 'abcl-contrib.jar' reside there and duplication of code paths would incur maintenance overhead.
I'm new to Java development toolchain (I used Eclipse here and there to make some trivial things but that automates most of this), I used gradle because it was being used by the OpenWhisk team and also because it allowed me to quickly achieve my goal: compared with ant XML files it was a breeze to have something that produced a single JAR file with the lisp file in place and the ABCL classes inside. This from my perspective which is not tied with building ABCL itself, of course. I will share all of it in some repo of course.
My only remaining doubt (apart from other possible solutions which could also work and be more elegant, I'm open to suggestions of course) is around loading abcl-contrib.jar or more specifically the included packages (JSS and JFLI specifically). I've spent some hours tracking the loading mechanism and learned a bit about the process but I'm still missing some pieces.
The as yet unreleased abcl-1.5.0 contains [a top-level Ant target 'abcl-aio.jar'][4] that creates a single archive containing both abcl.jar, abcl-contrib.jar, plus whatever has been added to the build-time file:abcl/contrib/ directory.
With [abcl-aio][] we introduced a mechanism that introspects the jar manifest for declarations of ASDF systems to load when (require :abcl-contrib) is issued.
We intend for 'abcl-aio' to serve as the mechanism for the end-user/developer to craft custom jar files.
Oh, this looks good. I will look into it. Currently I'm not building ABCL itself, I'm using the existing jar files as dependencies (and, btw, it seems that the ones in the Maven repository are rather old, 1.2 IIRC, would be good to update them).
Separately, there has been a long-standing mechanism to [include arbitrary code at system boot in a 'system.lisp' file][5] that could plausibly act as a hook for system intitialization, but I have never needed it to get WAR archives loaded in Java Servlet contexts.
Since I'm not building ABCL (and I think that going that route could appear a bit to complex to others, but maybe I'm wrong, more on that further down) I could use that if system.lisp is being automatically loaded, but OTOH using loadSystemFile() is not overly complex. BTW, system.lisp has this which seems related to the SYS logical pathname.
(SETF (LOGICAL-PATHNAME-TRANSLATIONS "sys") '(("SYS:SRC;**;*.*" "/Users/evenson/work/abcl/src/org/armedbear/lisp/**/*.*") ("SYS:JAVA;**;*.*" "/Users/evenson/work/abcl/src/org/armedbear/lisp/../../../**/*.*")))
In short, if you were to build from ABCL trunk, I think you should have the mechanisms to do what you want. ABCL trunk (aka 'abcl-1.5.0-dev') is quite stable, essentially representing the current release candidate for abcl-1.5.0. I am still working on a couple things (Java 6 support for UIOP/RUN-PROGRAM, using ABCL-BUILD to install Ant and/or Maven via interactive restart, better tests for PATHNAME shennigans), but I keep the patches for this work separate from the ABCL trunk source until it passes tests.
The way I've done it this far was from a different angle: instead of building ABCL and making changes to the ABCL build mechanism to produce my desired outcome, I'm using the binary ABCL distribution as a dependency, making changes during the construction of the JAR file. This can be the wrong approach but given my lack of familiarity with ABCL's (and Java in general) build process it seemed the most direct way because I started from an available Java sample and went from there.
That said I will investigate using ABCL build itself to do this, using AIO. Since I currently have it working I can showcase something based on what I have and then do a second part using that approach.
I think there is something obvious I'm missing, the pieces are all there (the jar handling code works fine, etc) but I can't seem to find the right way to put them together.
ABCL-AIO isn't currently documented (a bug), so I would not consider it obvious. [ABCL-JAR][] essentially uses the same mechanism (i.e. pushing PATHNAMEs to ASDF:*CENTRAL-REGISTRY*) you are prototyping here, but of course you are trying to get ABCL-CONTRIB to work in the first place here so it won't be much help for the immediate task at hand.
That's mildly reassuring to hear :) I generally go towards a goal by learning what I need as I go which can lead me to some less-than-obvious solutions. Still, I have learned quite a bit about how things work under the hood in the process.
Thank you for your help and support!
Best regards,
armedbear-devel@common-lisp.net
-
Frederico Munoz -
Mark Evenson