Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks for abcl-contrib in the following locations.
1. Looks for a jar named abcl-contrib.jar in the classpath determined by probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The following code proves that it is relatively easy to add this capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
This change makes it possible to include the abcl-contrib "dependency" directly inside the jar application that is currently running, regardless of the name of the jar file. Therefore, applications which make use of abcl-contrib can be distributed as a single jar.
For maven users, this means that their uberjars will work by just including abcl and abcl-contrib as dependencies.
For ant users they would probably need to extract abcl-contrib and include the extracted files in their generated jar which I believe can be fully automated.
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private (not exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
I would like your feedback on this issue. If this is a feature you want, or if you have any specific objections to including this feature in ABCL, please let me know.
Best Regards, Ralph Ritoch
Hi,
There is one small correction to the code used for testing.
(in-package :sys) (add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (boot-classloader)) 0))))
The code I sent in the proposal will not work because the functions from the sys package are private.
Best Regards, Ralph Ritoch
On Sun, Jul 26, 2015 at 8:29 PM, Ralph Ritoch rritoch@gmail.com wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks for abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The following code proves that it is relatively easy to add this capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
This change makes it possible to include the abcl-contrib "dependency" directly inside the jar application that is currently running, regardless of the name of the jar file. Therefore, applications which make use of abcl-contrib can be distributed as a single jar.
For maven users, this means that their uberjars will work by just including abcl and abcl-contrib as dependencies.
For ant users they would probably need to extract abcl-contrib and include the extracted files in their generated jar which I believe can be fully automated.
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private (not exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
I would like your feedback on this issue. If this is a feature you want, or if you have any specific objections to including this feature in ABCL, please let me know.
Best Regards, Ralph Ritoch
On 2015/7/26 14:29, Ralph Ritoch wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks for abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The contrib file may also be named via a versioning suffix something like 'abcl-contrib-m.n.o.jar'.
Some background on the current design: abcl-contrib.jar was explicitly separated from abcl.jar in order to have a clean separation between the code providing an ANSI Common Lisp, and the ABCL-specific extensions to this. The goal was to bound the size of the abcl.jar to some reasonably constant (over time) amount. In addition, function symbols in abcl.jar are subject to an "autoload" mechanism which delays the loading of their implementation until their first invocation. Consequently, the Lisp code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE (which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar may assume the full ANSI environment so may be addressed via CL:LOAD.
The following code proves that it is relatively easy to add this capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
A better implementation would be to add this capability to the mechanism used by (require :abcl-contrib) which corresponds to directly to [code contained in abcl-contrib.lisp][1].
[1]: http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib....
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
I think 'version.lisp' is a little too generic, and having its contents be irrelevant missing a design opportunity.
All SYS:ADD-CONTRIB essentially does is add locations to ASDF. Therefore it would be better to use the facilities of ASDF with [ABCL's implementation of JAR-PATHNAME][2] to search for an ASDF source-registry configuration at a well known path within a JAR, something like jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf. The files in this zip entry would then further specify specific ASDF locations within the jar. This has an advantage over the current mechanism which just looks for *.asd files in the second level directories, as supporting much more flexible packaging.
[2]: http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.m...
While I don't think we lose anything, I would like to note that implementing this proposal would weaken the difference between an ABCL-CONTRIB provided system and any other system packaged by ASDF. With the proposed (on the asdf development mailing list) strengthening of the rules and facilities for metadata in ASDF system descriptions, we should be able to (eventually) differentiate an ABCL-CONTRIB system from any other ASDF system if there were a valid need for this.
As part of the modification of SYS:ADD-CONTRIB, the current use of ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit configuration API exported by ASDF.
[…]
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private (not exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
The argument that the symbols one needs in SYS are not exported makes no functional difference to implement anything: you can still call them. What difficulty/impossibility do you see here?
That the symbols which are exported from SYS and EXT should be done in a cleaner, more consistent, manner is a point I readily accept. Ideally over time, I wish to only external symbols in the EXT package to constitute the API for extending ABCL.
[…]
In order to sponsor these changes to ABCL trunk, I would like to see:
1. The build instructions for Ant and Maven must share the same instructions. I do not want to chase the maintenance of two build processes. The path of least resistance to the rest of the current ABCL community would be to have Maven slave into the Ant build process at all necessary points, including sharing intermediate artifact locations and packaging instructions.
2. SYS:ADD-CONTRIB should use the ASDF configuration API as the mechanism to add new system locations based on the presence of a special directory such as jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf.
3. The code in SYS:ADD-CONTRIB should initially try to locate abcl-contrib{,-m.n.o}.jar files, falling back to searching the classpath. Since searching the classpath may be a potentially expensive operation--especially if it contains locations on a remote network--we may want to consider a runtime option and/or special variable to limit the search strategy.
4. Since the classpath can be modified, there should be a mechanism to explicitly request the ABCL-CONTRIB initialization strategy be re-run. I'm not sure the current code does this very well.
5. We should to test the new loading strategy in a) Java Servlet (Tomcat/Jetty) b) Google App Engine and c) OSGi containers.
Otherwise, good work! I think with a little back-and-forth on design, we can get something like this into ABCL in the near term.
Mark,
I have reviewed your suggestions and they are certainly progress in the right direction. I have two concerns, the first is with the path-spec for locating configuration files.
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf
It generally considered a bad practice for two libraries to contribute to the same package and while the META-INF folder falls into system dependent functionality I believe a better pathspec would be accommodate this to "namespace" the configuration files. I'm aware that this isn't the standard practice, but if the namespace is properly named it is will be easier to track configuration files back to the library they came from.
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf
My second concern is the method of configuring the contrib search. The default should certainly support the "legacy" search methods and fallback to the classpath, but to avoid imposing limitations on future systems I believe a better algorithm would be to allow registration of custom configuration search handlers such as having a sys:*configuration-providers* which contains and ordered list of configuration provider functions. In the case of web based apps, this opens the potential to cache remote configuration files.
I am looking forward to the progress of this, expanding this functionality beyond the configuration of one specific library could greatly improves the application lifecycle for ABCL applications.
As for the maintenance of the maven build system, the pom's I submitted compile the java files, and builds the jar, but is calling ant targets. While ant can call maven it still comes down to the key issue that maven doesn't pick-up a jar created by an external process. Maven central requires a source release, a binary release, and javadoc release, which all must be GPG signed. The complexity of trying to make maven a slave to ant is comparable to the complexity of the current "manual" system and I believe it will lead to the same situation you have now where no one does releases to maven. While there may be room for improvement with the maven system, there's normally little to no maintenance needed, once it is properly configured. The only thing that must be changed are the version numbers, and any dependencies. If you can make maven a slave to ant such that it will deploy to central and local repositories, that would be fine, but other than creating a custom maven plugin I don't know any way it would be possible without allowing maven to package and sign the jars.
I was advised that it would be possible to do away with the ant system all together. What was suggested to me by java developers was to have the java sources built in one module, which has a post-build script to compile the lisp files, and a second module which uses the target (generated) files of the first module as it's resource path. Calling ant is much easier since the ant system is already built, but it is possible to completely deprecate ant in favor of maven.
Best Regards, Ralph Ritoch
Date: Mon, 27 Jul 2015 10:42:30 +0200 From: Mark Evenson evenson@panix.com To: armedbear-devel@common-lisp.net Subject: Re: Proposal to improve the loading of the abcl-contrib dependency. Message-ID: 55B5EEF6.5030601@panix.com Content-Type: text/plain; charset=utf-8
On 2015/7/26 14:29, Ralph Ritoch wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks
for
abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The contrib file may also be named via a versioning suffix something like 'abcl-contrib-m.n.o.jar'.
Some background on the current design: abcl-contrib.jar was explicitly separated from abcl.jar in order to have a clean separation between the code providing an ANSI Common Lisp, and the ABCL-specific extensions to this. The goal was to bound the size of the abcl.jar to some reasonably constant (over time) amount. In addition, function symbols in abcl.jar are subject to an "autoload" mechanism which delays the loading of their implementation until their first invocation. Consequently, the Lisp code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE (which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar may assume the full ANSI environment so may be addressed via CL:LOAD.
The following code proves that it is relatively easy to add this
capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
A better implementation would be to add this capability to the mechanism used by (require :abcl-contrib) which corresponds to directly to [code contained in abcl-contrib.lisp][1].
[1]: http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib....
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
I think 'version.lisp' is a little too generic, and having its contents be irrelevant missing a design opportunity.
All SYS:ADD-CONTRIB essentially does is add locations to ASDF. Therefore it would be better to use the facilities of ASDF with [ABCL's implementation of JAR-PATHNAME][2] to search for an ASDF source-registry configuration at a well known path within a JAR, something like jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf. The files in this zip entry would then further specify specific ASDF locations within the jar. This has an advantage over the current mechanism which just looks for *.asd files in the second level directories, as supporting much more flexible packaging.
[2]: http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.m...
While I don't think we lose anything, I would like to note that implementing this proposal would weaken the difference between an ABCL-CONTRIB provided system and any other system packaged by ASDF. With the proposed (on the asdf development mailing list) strengthening of the rules and facilities for metadata in ASDF system descriptions, we should be able to (eventually) differentiate an ABCL-CONTRIB system from any other ASDF system if there were a valid need for this.
As part of the modification of SYS:ADD-CONTRIB, the current use of ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit configuration API exported by ASDF.
[?]
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private
(not
exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
The argument that the symbols one needs in SYS are not exported makes no functional difference to implement anything: you can still call them. What difficulty/impossibility do you see here?
That the symbols which are exported from SYS and EXT should be done in a cleaner, more consistent, manner is a point I readily accept. Ideally over time, I wish to only external symbols in the EXT package to constitute the API for extending ABCL.
[?]
In order to sponsor these changes to ABCL trunk, I would like to see:
1. The build instructions for Ant and Maven must share the same instructions. I do not want to chase the maintenance of two build processes. The path of least resistance to the rest of the current ABCL community would be to have Maven slave into the Ant build process at all necessary points, including sharing intermediate artifact locations and packaging instructions.
2. SYS:ADD-CONTRIB should use the ASDF configuration API as the mechanism to add new system locations based on the presence of a special directory such as jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf.
3. The code in SYS:ADD-CONTRIB should initially try to locate abcl-contrib{,-m.n.o}.jar files, falling back to searching the classpath. Since searching the classpath may be a potentially expensive operation--especially if it contains locations on a remote network--we may want to consider a runtime option and/or special variable to limit the search strategy.
4. Since the classpath can be modified, there should be a mechanism to explicitly request the ABCL-CONTRIB initialization strategy be re-run. I'm not sure the current code does this very well.
5. We should to test the new loading strategy in a) Java Servlet (Tomcat/Jetty) b) Google App Engine and c) OSGi containers.
Otherwise, good work! I think with a little back-and-forth on design, we can get something like this into ABCL in the near term.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Sun, Jul 26, 2015 at 8:29 PM, Ralph Ritoch rritoch@gmail.com wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks for abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The following code proves that it is relatively easy to add this capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
This change makes it possible to include the abcl-contrib "dependency" directly inside the jar application that is currently running, regardless of the name of the jar file. Therefore, applications which make use of abcl-contrib can be distributed as a single jar.
For maven users, this means that their uberjars will work by just including abcl and abcl-contrib as dependencies.
For ant users they would probably need to extract abcl-contrib and include the extracted files in their generated jar which I believe can be fully automated.
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private (not exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
I would like your feedback on this issue. If this is a feature you want, or if you have any specific objections to including this feature in ABCL, please let me know.
Best Regards, Ralph Ritoch
Mark,
I have one additional issue to add to the configuration discovery process. The original concept of using a /META-INF/abcl-contrib/version.lisp file accounts for web based classpaths while the *-conf.d pattern does not due to the fact that it is often not possible to get a list of files for web based classpaths. To accommodate this a default configuration file list would also be needed, such as /META-INF/abcl-config-sources.lisp . I am aware that this creates a huge potential for conflict as various libraries could each contain this file, but the web deployment process would need to ensure that all of the configuration files in the *-conf.d are included in this file when the classpath will be accessed remotely but not from within a jar.
Best Regards, Ralph Ritoch
On Sat, Aug 1, 2015 at 4:20 PM, Ralph Ritoch rritoch@gmail.com wrote:
Mark,
I have reviewed your suggestions and they are certainly progress in the right direction. I have two concerns, the first is with the path-spec for locating configuration files.
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf
It generally considered a bad practice for two libraries to contribute to the same package and while the META-INF folder falls into system dependent functionality I believe a better pathspec would be accommodate this to "namespace" the configuration files. I'm aware that this isn't the standard practice, but if the namespace is properly named it is will be easier to track configuration files back to the library they came from.
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf
My second concern is the method of configuring the contrib search. The default should certainly support the "legacy" search methods and fallback to the classpath, but to avoid imposing limitations on future systems I believe a better algorithm would be to allow registration of custom configuration search handlers such as having a sys:*configuration-providers* which contains and ordered list of configuration provider functions. In the case of web based apps, this opens the potential to cache remote configuration files.
I am looking forward to the progress of this, expanding this functionality beyond the configuration of one specific library could greatly improves the application lifecycle for ABCL applications.
As for the maintenance of the maven build system, the pom's I submitted compile the java files, and builds the jar, but is calling ant targets. While ant can call maven it still comes down to the key issue that maven doesn't pick-up a jar created by an external process. Maven central requires a source release, a binary release, and javadoc release, which all must be GPG signed. The complexity of trying to make maven a slave to ant is comparable to the complexity of the current "manual" system and I believe it will lead to the same situation you have now where no one does releases to maven. While there may be room for improvement with the maven system, there's normally little to no maintenance needed, once it is properly configured. The only thing that must be changed are the version numbers, and any dependencies. If you can make maven a slave to ant such that it will deploy to central and local repositories, that would be fine, but other than creating a custom maven plugin I don't know any way it would be possible without allowing maven to package and sign the jars.
I was advised that it would be possible to do away with the ant system all together. What was suggested to me by java developers was to have the java sources built in one module, which has a post-build script to compile the lisp files, and a second module which uses the target (generated) files of the first module as it's resource path. Calling ant is much easier since the ant system is already built, but it is possible to completely deprecate ant in favor of maven.
Best Regards, Ralph Ritoch
Date: Mon, 27 Jul 2015 10:42:30 +0200 From: Mark Evenson evenson@panix.com To: armedbear-devel@common-lisp.net Subject: Re: Proposal to improve the loading of the abcl-contrib dependency. Message-ID: 55B5EEF6.5030601@panix.com Content-Type: text/plain; charset=utf-8
On 2015/7/26 14:29, Ralph Ritoch wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks
for
abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The contrib file may also be named via a versioning suffix something like 'abcl-contrib-m.n.o.jar'.
Some background on the current design: abcl-contrib.jar was explicitly separated from abcl.jar in order to have a clean separation between the code providing an ANSI Common Lisp, and the ABCL-specific extensions to this. The goal was to bound the size of the abcl.jar to some reasonably constant (over time) amount. In addition, function symbols in abcl.jar are subject to an "autoload" mechanism which delays the loading of their implementation until their first invocation. Consequently, the Lisp code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE (which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar may assume the full ANSI environment so may be addressed via CL:LOAD.
The following code proves that it is relatively easy to add this
capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
A better implementation would be to add this capability to the mechanism used by (require :abcl-contrib) which corresponds to directly to [code contained in abcl-contrib.lisp][1].
[1]:
http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib....
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this
file
is not important.
I think 'version.lisp' is a little too generic, and having its contents be irrelevant missing a design opportunity.
All SYS:ADD-CONTRIB essentially does is add locations to ASDF. Therefore it would be better to use the facilities of ASDF with [ABCL's implementation of JAR-PATHNAME][2] to search for an ASDF source-registry configuration at a well known path within a JAR, something like jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf. The files in this zip entry would then further specify specific ASDF locations within the jar. This has an advantage over the current mechanism which just looks for *.asd files in the second level directories, as supporting much more flexible packaging.
[2]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.m...
While I don't think we lose anything, I would like to note that implementing this proposal would weaken the difference between an ABCL-CONTRIB provided system and any other system packaged by ASDF. With the proposed (on the asdf development mailing list) strengthening of the rules and facilities for metadata in ASDF system descriptions, we should be able to (eventually) differentiate an ABCL-CONTRIB system from any other ASDF system if there were a valid need for this.
As part of the modification of SYS:ADD-CONTRIB, the current use of ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit configuration API exported by ASDF.
[?]
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private
(not
exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
The argument that the symbols one needs in SYS are not exported makes no functional difference to implement anything: you can still call them. What difficulty/impossibility do you see here?
That the symbols which are exported from SYS and EXT should be done in a cleaner, more consistent, manner is a point I readily accept. Ideally over time, I wish to only external symbols in the EXT package to constitute the API for extending ABCL.
[?]
In order to sponsor these changes to ABCL trunk, I would like to see:
- The build instructions for Ant and Maven must share the same
instructions. I do not want to chase the maintenance of two build processes. The path of least resistance to the rest of the current ABCL community would be to have Maven slave into the Ant build process at all necessary points, including sharing intermediate artifact locations and packaging instructions.
- SYS:ADD-CONTRIB should use the ASDF configuration API as the
mechanism to add new system locations based on the presence of a special directory such as jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf.
- The code in SYS:ADD-CONTRIB should initially try to locate
abcl-contrib{,-m.n.o}.jar files, falling back to searching the classpath. Since searching the classpath may be a potentially expensive operation--especially if it contains locations on a remote network--we may want to consider a runtime option and/or special variable to limit the search strategy.
- Since the classpath can be modified, there should be a mechanism to
explicitly request the ABCL-CONTRIB initialization strategy be re-run. I'm not sure the current code does this very well.
- We should to test the new loading strategy in a) Java Servlet
(Tomcat/Jetty) b) Google App Engine and c) OSGi containers.
Otherwise, good work! I think with a little back-and-forth on design, we can get something like this into ABCL in the near term.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Sun, Jul 26, 2015 at 8:29 PM, Ralph Ritoch rritoch@gmail.com wrote:
Hi,
I have run into an issue with making executable abcl jars when the jar depends on, and provides, abcl-contrib. As far as I can tell abcl looks for abcl-contrib in the following locations.
- Looks for a jar named abcl-contrib.jar in the classpath determined by
probing the classloader for the jars it provides. 2. Looks in the directory of the jars in the classpath for a file named abcl-contrib.jar
Neither of these solutions work if the currently executing jar includes abcl-contrib and isn't named abcl-contrib.jar.
The following code proves that it is relatively easy to add this capability.
(in-package 'sys) (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref (java:jcall "getURLs" (sys:boot-classloader)) 0))))
This code properly loads abcl-contrib when the executing jar provides abcl-contrib, while (require 'abcl-contrib) does not.
My proposal is to add a file named abcl-contrib/version.lisp to the abcl-contrib.jar, possibly in the /META-INF directory to avoid any potential conflicts with java, and to have sys:find-contrib look for this file in the resources provided by the classpath. The contents of this file is not important.
This change makes it possible to include the abcl-contrib "dependency" directly inside the jar application that is currently running, regardless of the name of the jar file. Therefore, applications which make use of abcl-contrib can be distributed as a single jar.
For maven users, this means that their uberjars will work by just including abcl and abcl-contrib as dependencies.
For ant users they would probably need to extract abcl-contrib and include the extracted files in their generated jar which I believe can be fully automated.
It would be relatively easy for me to produce a patch to provide this feature as part of ABCL. It may be possible to provide this feature as an add-on, but the functions needed within the system package are private (not exported) so it would be difficult, if not impossible, to implement this feature as an add-on.
I would like your feedback on this issue. If this is a feature you want, or if you have any specific objections to including this feature in ABCL, please let me know.
Best Regards, Ralph Ritoch
On 01 Aug 2015, at 12:00, Ralph Ritoch rritoch@gmail.com wrote:
Mark,
I have one additional issue to add to the configuration discovery process. The original concept of using a /META-INF/abcl-contrib/version.lisp file accounts for web based classpaths while the *-conf.d pattern does not due to the fact that it is often not possible to get a list of files for web based classpaths. To accommodate this a default configuration file list would also be needed, such as /META-INF/abcl-config-sources.lisp . I am aware that this creates a huge potential for conflict as various libraries could each contain this file, but the web deployment process would need to ensure that all of the configuration files in the *-conf.d are included in this file when the classpath will be accessed remotely but not from within a jar.
Then I would suggest that we further ape the ASDF source registry mechanism by checking for the existence of a
jar:file:SOME.jar!/META-INF/common-lisp/source-registy.conf
file as well.
On 01 Aug 2015, at 10:20, Ralph Ritoch rritoch@gmail.com wrote:
[…]
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf
Might as well just take all files recursively:
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/**/*.conf
which will allow both styles to be used.
My second concern is the method of configuring the contrib search. The default should certainly support the "legacy" search methods and fallback to the classpath, but to avoid imposing limitations on future systems I believe a better algorithm would be to allow registration of custom configuration search handlers such as having a sys:*configuration-providers* which contains and ordered list of configuration provider functions. In the case of web based apps, this opens the potential to cache remote configuration files.
We’re not really defining a contrib search anymore, per se, but rather an additional mechanism to configure ASDF locations in jar files. ASDF already has a robust set of APIs for configuring itself, so I’m not entirely convinced that there is a need for anything beyond defining a well-known location within JAR archives to search. Note that ABCL already has an extension mechanism for CL:REQUIRE in the SYS:*MODULE-PROVIDER-FUNCTIONS* hook. Could you give me a couple more specific use cases in which one would need additional mechanisms to specify the location of ASDF source registry configuration files?
I am looking forward to the progress of this, expanding this functionality beyond the configuration of one specific library could greatly improves the application lifecycle for ABCL applications.
Again, ABCL already has a rich configuration mechanism for specifying ASDF system locations except for the location of the systems that constitute ABCL-CONTRIB. Could you explain what you mean by application lifecycle here or what domain you are using the term from?
As for the maintenance of the maven build system, the pom's I submitted compile the java files, and builds the jar, but is calling ant targets. While ant can call maven it still comes down to the key issue that maven doesn't pick-up a jar created by an external process. Maven central requires a source release, a binary release, and javadoc release, which all must be GPG signed. The complexity of trying to make maven a slave to ant is comparable to the complexity of the current "manual" system and I believe it will lead to the same situation you have now where no one does releases to maven. While there may be room for improvement with the maven system, there's normally little to no maintenance needed, once it is properly configured. The only thing that must be changed are the version numbers, and any dependencies. If you can make maven a slave to ant such that it will deploy to central and local repositories, that would be fine, but other than creating a custom maven plugin I don't know any way it would be possible without allowing maven to package and sign the jars.
One can make releases to Maven using the existing POMs as was done in the past: all we need is someone with the correct amount of upstream juice to establish the political connections.
There is nothing about the complexity of the current build process that prohibits releases to Maven, just a lack of a volunteer to ensure that it happens. The [instructions have been written][1]
[1]: http://abcl.org/trac/browser/trunk/abcl/maven-release.txt
I was advised that it would be possible to do away with the ant system all together. What was suggested to me by java developers was to have the java sources built in one module, which has a post-build script to compile the lisp files, and a second module which uses the target (generated) files of the first module as it's resource path. Calling ant is much easier since the ant system is already built, but it is possible to completely deprecate ant in favor of maven.
As I pointed out earlier, your POMs do not share the intermediate build artifact locations nor packaging instructions with the Ant system. In addition, there is a fair amount of other functionality in the Ant file which is definitely not duplicated in your POMs (running tests, running debugging instances, generating TAGS, usw.) Until we approach a level of competence with Maven that allows us to adapt Maven to the needs of ABCL, I think discussions about deprecating Ant in favor of Maven are premature.
Mark,
When I refer to the application lifecycle I am specifically referring to the "start" phase of an application where dependencies need to be loaded and configured. ASDF is a lisp system with no specific support for java. Java 9 is coming and is scheduled to include new semantics for modules and that may carry with it a need for additional configuration semantics related to the loading of modules. I wasn't aware that ABCL already has an extensibility system for the lifecycle, such as SYS:*MODULE-PROVIDER-FUNCTIONS*. If that can be used to override the default places to search for the abcl-contrib packages, than great. I didn't notice it in the code I was looking at. The core problem that needs to be solved is making a reasonable deployment system for distributed single-jar (uberjar) applications. I don't see much need to go beyond that since quicklisp is included in the contrib library. In the current state of ABCL the contrib package won't load in an uberjar without executing system implementation code that carries with it no guarantees of future support. The same technology that will enable execution from an uberjar, will also make it possible to integrate ABCL into many java systems, with one notable exception of OSGI based systems. These lifecycle issues are something that should be dealt with at the system level, otherwise every ABCL application will need to provide it's own lifecycle code and applications will likely not be interoperable with each-other. When it comes time to deal with OSGI and Java 9 integration issues, the problem of loading contrib libraries, or other libraries, will only get worse if this isn't solved within ABCL. The Java logging systems is a good example of how bad it can get when everyone decides to make their own implementation of core system features. If all libraries end up implementing their own loading lifecycle implementations, it is extremely likely that they will conflict with each-other. ASDF is a good system, but it will be rendered useless in a java environment if no uniform method is provided to properly configure it. In the current state, uberjars produced with abcl and abcl-contrib do not properly configure asdf so the standard code for loading the contrib packages in such applications does not work.
Best Regards, Ralph Ritoch
On Sun, Aug 2, 2015 at 12:39 AM, Mark Evenson evenson@panix.com wrote:
On 01 Aug 2015, at 10:20, Ralph Ritoch rritoch@gmail.com wrote:
[…]
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf
Might as well just take all files recursively:
jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/**/*.conf
which will allow both styles to be used.
My second concern is the method of configuring the contrib search. The
default should certainly support the "legacy" search methods and fallback to the classpath, but to avoid imposing limitations on future systems I believe a better algorithm would be to allow registration of custom configuration search handlers such as having a sys:*configuration-providers* which contains and ordered list of configuration provider functions. In the case of web based apps, this opens the potential to cache remote configuration files.
We’re not really defining a contrib search anymore, per se, but rather an additional mechanism to configure ASDF locations in jar files. ASDF already has a robust set of APIs for configuring itself, so I’m not entirely convinced that there is a need for anything beyond defining a well-known location within JAR archives to search. Note that ABCL already has an extension mechanism for CL:REQUIRE in the SYS:*MODULE-PROVIDER-FUNCTIONS* hook. Could you give me a couple more specific use cases in which one would need additional mechanisms to specify the location of ASDF source registry configuration files?
I am looking forward to the progress of this, expanding this
functionality beyond the configuration of one specific library could greatly improves the application lifecycle for ABCL applications.
Again, ABCL already has a rich configuration mechanism for specifying ASDF system locations except for the location of the systems that constitute ABCL-CONTRIB. Could you explain what you mean by application lifecycle here or what domain you are using the term from?
As for the maintenance of the maven build system, the pom's I submitted
compile the java files, and builds the jar, but is calling ant targets. While ant can call maven it still comes down to the key issue that maven doesn't pick-up a jar created by an external process. Maven central requires a source release, a binary release, and javadoc release, which all must be GPG signed. The complexity of trying to make maven a slave to ant is comparable to the complexity of the current "manual" system and I believe it will lead to the same situation you have now where no one does releases to maven. While there may be room for improvement with the maven system, there's normally little to no maintenance needed, once it is properly configured. The only thing that must be changed are the version numbers, and any dependencies. If you can make maven a slave to ant such that it will deploy to central and local repositories, that would be fine, but other than creating a custom maven plugin I don't know any way it would be possible without allowing maven to package and sign the jars.
One can make releases to Maven using the existing POMs as was done in the past: all we need is someone with the correct amount of upstream juice to establish the political connections.
There is nothing about the complexity of the current build process that prohibits releases to Maven, just a lack of a volunteer to ensure that it happens. The [instructions have been written][1]
I was advised that it would be possible to do away with the ant system
all together. What was suggested to me by java developers was to have the java sources built in one module, which has a post-build script to compile the lisp files, and a second module which uses the target (generated) files of the first module as it's resource path. Calling ant is much easier since the ant system is already built, but it is possible to completely deprecate ant in favor of maven.
As I pointed out earlier, your POMs do not share the intermediate build artifact locations nor packaging instructions with the Ant system. In addition, there is a fair amount of other functionality in the Ant file which is definitely not duplicated in your POMs (running tests, running debugging instances, generating TAGS, usw.) Until we approach a level of competence with Maven that allows us to adapt Maven to the needs of ABCL, I think discussions about deprecating Ant in favor of Maven are premature.
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On 2015/8/1 19:30, Ralph Ritoch wrote:
Mark,
When I refer to the application lifecycle I am specifically referring to the "start" phase of an application where dependencies need to be loaded and configured.
"Birth. School. Work. Death" is a lifecycle. "The loading of dependencies" is but a phase in a lifecycle.
[…]
ASDF is a lisp system with no specific support for java.
Not true: ASDF contains ABCL specific code to load systems from jar files, among other specializations. And ABCL-ASDF extends ASDF to include support for specifying Maven artifacts and additional points to load artifacts into the classloader.
[…]
I don't see much need to go beyond that since quicklisp is included in the contrib library.
Quicklisp is not included in the contrib library: QUICKLISP-ABCL is a loading shim that can fetch Quicklisp from the network.
[…[
When it comes time to deal with OSGI and Java 9 integration issues, the problem of loading contrib libraries, or other libraries, will only get worse if this isn't solved within ABCL.
Other libraries *can* be loaded via ASDF; it is merely the location of ABCL-CONTRIB that is currently special.
[…]
On 2015/8/1 19:30, Ralph Ritoch wrote: […]
I wasn't aware that ABCL already has an extensibility system for the lifecycle, such as SYS:*MODULE-PROVIDER-FUNCTIONS*. If that can be used to override the default places to search for the abcl-contrib packages, than great. I didn't notice it in the code I was looking at. The core problem that needs to be solved is making a reasonable deployment system for distributed single-jar (uberjar) applications.
[…]
An additional mechanism for modifying ABCL behavior at startup lies in the ability to [add arbitrary code to the contents of system.lisp][1]. One could use this to install an additional hook to SYS:*MODULE-PROVIDER-FUNCTIONS* which would be able to satisfy the (REQUIRE :abcl-contrib) by referring to PATHNAMEs within the current jar. Thus, one could boot an überjar without recourse to anything additional in ABCL.
[1]: http://abcl.org/trac/browser/trunk/abcl/abcl.properties.in#L13
Mark,
I don't know if you are planning on coding the features that are being speculated but I have implemented this feature per the original specs. In addition to adding the version file I also moved some things around to facilitate resource filtering to fill in the version number on the version file. This replacement doesn't happen with ant but the lack of replacement doesn't break the ant build either. There are two commits which produce this feature.
https://github.com/rritoch/abcl/commit/34feb3baa2bf6372e81fb91b176ac376d469e... https://github.com/rritoch/abcl/commit/fd0ecafc28b909a484436691dc7bf83b97875...
This repairs the bug where the abcl-contrib package isn't being detected in single-jar applications. I have tested it and it is fully functional.
Best Regards, Ralph Ritoch
On Sun, Aug 2, 2015 at 6:15 AM, Mark Evenson evenson@panix.com wrote:
On 2015/8/1 19:30, Ralph Ritoch wrote: […]
I wasn't aware that ABCL already has an extensibility system for the lifecycle, such as SYS:*MODULE-PROVIDER-FUNCTIONS*. If that can be used to override the default places to search for the abcl-contrib packages, than great. I didn't notice it in the code I was looking at. The core problem that
needs
to be solved is making a reasonable deployment system for distributed single-jar (uberjar) applications.
[…]
An additional mechanism for modifying ABCL behavior at startup lies in the ability to [add arbitrary code to the contents of system.lisp][1]. One could use this to install an additional hook to SYS:*MODULE-PROVIDER-FUNCTIONS* which would be able to satisfy the (REQUIRE :abcl-contrib) by referring to PATHNAMEs within the current jar. Thus, one could boot an überjar without recourse to anything additional in ABCL.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Mark,
I made one additional change to the code and I believe the performance is now faster than the original code. Instead of searching the classpath, I used the classloader to find the version file.
https://github.com/rritoch/abcl/commit/fd8bf57f921dbb015cbb8401d49b3427eec60...
Best Regards, Ralph Ritoch
On Mon, Aug 3, 2015 at 11:18 PM, Ralph Ritoch rritoch@gmail.com wrote:
Mark,
I don't know if you are planning on coding the features that are being speculated but I have implemented this feature per the original specs. In addition to adding the version file I also moved some things around to facilitate resource filtering to fill in the version number on the version file. This replacement doesn't happen with ant but the lack of replacement doesn't break the ant build either. There are two commits which produce this feature.
https://github.com/rritoch/abcl/commit/34feb3baa2bf6372e81fb91b176ac376d469e...
https://github.com/rritoch/abcl/commit/fd0ecafc28b909a484436691dc7bf83b97875...
This repairs the bug where the abcl-contrib package isn't being detected in single-jar applications. I have tested it and it is fully functional.
Best Regards, Ralph Ritoch
On Sun, Aug 2, 2015 at 6:15 AM, Mark Evenson evenson@panix.com wrote:
On 2015/8/1 19:30, Ralph Ritoch wrote: […]
I wasn't aware that ABCL already has an extensibility system for the lifecycle, such as SYS:*MODULE-PROVIDER-FUNCTIONS*. If that can be used to override the default places to search for the abcl-contrib packages, than great. I didn't notice it in the code I was looking at. The core problem that
needs
to be solved is making a reasonable deployment system for distributed single-jar (uberjar) applications.
[…]
An additional mechanism for modifying ABCL behavior at startup lies in the ability to [add arbitrary code to the contents of system.lisp][1]. One could use this to install an additional hook to SYS:*MODULE-PROVIDER-FUNCTIONS* which would be able to satisfy the (REQUIRE :abcl-contrib) by referring to PATHNAMEs within the current jar. Thus, one could boot an überjar without recourse to anything additional in ABCL.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel@common-lisp.net
-
Mark Evenson -
Ralph Ritoch