[armedbear-devel] ASDF-JAR vs ASDF3 deployment support
Hi all, Some years ago, Mark wrote ASDF-JAR; a solution to package ASDF systems and their dependencies in JARs. As it turns out, the solution chosen has some problems with correctly packaging ASDs for systems which use the #. reader macro. As an example: bordeaux-threads uses a version.lisp-sexp file to canonically define the version number and the system definition reads that file in a #. form. However, this file isn't packaged because it's not part of the file listing in the system definition. This means asdf can't load the definition after deployment. With the advent of ASDF3, its developers have created infrastructure to combine fasls of a single system and even fasls of a system and all its dependencies into a single large combined FASL. Among others, this new infrastructure does not run into the issue ASDF-JAR is running into. While not completely equivalent to the JAR solution, using this infrastructure would help us a huge step along that way of getting better deployment. My current thinking is that if the single-packed-FASL needs to end up as a jar, we can work on that as a next step. The first step to take would be to create "concatenate fasls" functionality, which would do the following: * Detect all the file entries in all the passed-in fasls * Create a new output zip file with directories for each fasl being packed, populated with the files and directories of that fasl * create a toplevel ._ file which loads the sub-fasls' ._ files Comments? Bye, Erik.
Wasn't #. deprecated in favor of load-time-value? On Mon, Apr 1, 2013 at 8:15 AM, Erik Huelsmann <ehuels@gmail.com> wrote:
Hi all,
Some years ago, Mark wrote ASDF-JAR; a solution to package ASDF systems and their dependencies in JARs.
As it turns out, the solution chosen has some problems with correctly packaging ASDs for systems which use the #. reader macro. As an example: bordeaux-threads uses a version.lisp-sexp file to canonically define the version number and the system definition reads that file in a #. form. However, this file isn't packaged because it's not part of the file listing in the system definition. This means asdf can't load the definition after deployment.
With the advent of ASDF3, its developers have created infrastructure to combine fasls of a single system and even fasls of a system and all its dependencies into a single large combined FASL. Among others, this new infrastructure does not run into the issue ASDF-JAR is running into. While not completely equivalent to the JAR solution, using this infrastructure would help us a huge step along that way of getting better deployment. My current thinking is that if the single-packed-FASL needs to end up as a jar, we can work on that as a next step.
The first step to take would be to create "concatenate fasls" functionality, which would do the following:
* Detect all the file entries in all the passed-in fasls * Create a new output zip file with directories for each fasl being packed, populated with the files and directories of that fasl * create a toplevel ._ file which loads the sub-fasls' ._ files
Comments?
Bye,
Erik.
_______________________________________________ armedbear-devel mailing list armedbear-devel@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
Hi Alan, On Mon, Apr 1, 2013 at 6:09 PM, Alan Ruttenberg <alanruttenberg@gmail.com>wrote:
Wasn't #. deprecated in favor of load-time-value?
Since one is resolved at read time and the other after compilation, I can't imagine how one can replace the other. I don't think either is deprecated. Regards, Erik.
This may not be directly related, but I am reminded of some peculiar behavior I encountered with #. and fasls. I reduced it to http://lists.common-lisp.net/pipermail/armedbear-devel/2012-July/002468.html The latest ABCL still has this issue (if it is an issue). On Mon, Apr 1, 2013 at 8:15 AM, Erik Huelsmann <ehuels@gmail.com> wrote:
Hi all,
Some years ago, Mark wrote ASDF-JAR; a solution to package ASDF systems and their dependencies in JARs.
As it turns out, the solution chosen has some problems with correctly packaging ASDs for systems which use the #. reader macro. As an example: bordeaux-threads uses a version.lisp-sexp file to canonically define the version number and the system definition reads that file in a #. form. However, this file isn't packaged because it's not part of the file listing in the system definition. This means asdf can't load the definition after deployment.
With the advent of ASDF3, its developers have created infrastructure to combine fasls of a single system and even fasls of a system and all its dependencies into a single large combined FASL. Among others, this new infrastructure does not run into the issue ASDF-JAR is running into. While not completely equivalent to the JAR solution, using this infrastructure would help us a huge step along that way of getting better deployment. My current thinking is that if the single-packed-FASL needs to end up as a jar, we can work on that as a next step.
The first step to take would be to create "concatenate fasls" functionality, which would do the following:
* Detect all the file entries in all the passed-in fasls * Create a new output zip file with directories for each fasl being packed, populated with the files and directories of that fasl * create a toplevel ._ file which loads the sub-fasls' ._ files
Comments?
Bye,
Erik.
_______________________________________________ armedbear-devel mailing list armedbear-devel@common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
01.04.2013, 16:28, "Erik Huelsmann" <ehuels@gmail.com>:
Hi all,
Some years ago, Mark wrote ASDF-JAR; a solution to package ASDF systems and their dependencies in JARs.
As it turns out, the solution chosen has some problems with correctly packaging ASDs for systems which use the #. reader macro. As an example: bordeaux-threads uses a version.lisp-sexp file to canonically define the version number and the system definition reads that file in a #. form. However, this file isn't packaged because it's not part of the file listing in the system definition. This means asdf can't load the definition after deployment.
With the advent of ASDF3, its developers have created infrastructure to combine fasls of a single system and even fasls of a system and all its dependencies into a single large combined FASL. Among others, this new infrastructure does not run into the issue ASDF-JAR is running into. While not completely equivalent to the JAR solution, using this infrastructure would help us a huge step along that way of getting better deployment. My current thinking is that if the single-packed-FASL needs to end up as a jar, we can work on that as a next step.
The first step to take would be to create "concatenate fasls" functionality, which would do the following:
* Detect all the file entries in all the passed-in fasls * Create a new output zip file with directories for each fasl being packed, populated with the files and directories of that fasl * create a toplevel ._ file which loads the sub-fasls' ._ files
Comments?
Bye,
Erik.
General note about deployment CL systems. Often the deployment package should contain not only compiled code, but also resources: web libraries and frameworks often contain .css, .html, .javascript; bordeaux-threads uses version.lisp-sexp files, and so on. In addition to the #. reader macro as used in bordeaux-threads, applications/libraries access their resources during run-time using asdf:system-relative-pathname or similar functions. In other words, such libraries assume presence during run-time of their full source control checkout . A general purpose deployment solutions should account this aspect. Of course, if an application is free from such problems, one can use concatenated fasls, lisp images or similar ways. One way to deploy applications with full content of the dependencies code and resources: 1. install quicklisp in a custom directory, 2. (ql:quickload :my-application) 3. copy the directory of my-application, the custom quicklisp directory and probably prebuild .fals files to server. A little bit more docs and example of this approach is here: https://github.com/avodonosov/heroku-buildpack-cl2 Best regards, - Anton
On 4/4/13 12:44 AM, Anton Vodonosov wrote: […]
General note about deployment CL systems.
Often the deployment package should contain not only compiled code, but also resources: web libraries and frameworks often contain .css, .html, .javascript; bordeaux-threads uses version.lisp-sexp files, and so on.
In addition to the #. reader macro as used in bordeaux-threads, applications/libraries access their resources during run-time using asdf:system-relative-pathname or similar functions. In other words, such libraries assume presence during run-time of their full source control checkout .
A general purpose deployment solutions should account this aspect.
One way to deploy applications with full content of the dependencies code and resources: 1. install quicklisp in a custom directory, 2. (ql:quickload :my-application) 3. copy the directory of my-application, the custom quicklisp directory and probably prebuild .fals files to server.
A little bit more docs and example of this approach is here: https://github.com/avodonosov/heroku-buildpack-cl2
In my opinion of "the right way", system definitions should contain a comprehensive listing of all the resources in the DEFSYSTEM form. Anything referenced by ASDF:SYSTEM-RELATIVE-PATHNAME or as a target of a CL:SHARPSIGN-DOT reader macro should be declared as a STATIC-FILE ASDF component. If this were somehow universally the case, then we could reliably use the machinery of ASDF3 to transverse its components, and determine both the "source" and the "compiled" components. But because "universal distribution" (i.e. for the majority of users of a given system) has more-or-less subsumed by Quicklisp, there is no real constraint on the developers of system definitions to ensure that all components are so enumerated: loading the system from Quicklisp works so nothing is broken, right? After Xach, Anton probably has the most practical experience with packaging ASDF definitions for deployment due to his pioneering work on CL-TEST-GRID. I find Anton comments to be a succinct summary of the issues involved in packaging a given ASDF system in Quicklisp. But when facing deployment for an ASDF system that is not in Quicklisp, but has dependencies on Quicklisp systems, Anton's approach won't work so well. One would have to find the .asd definition on the filesystem, then transverse its sub-directories to find the source files. This has the problem that there is no guarantee that the components of an asd definition exist in subdirectories, as theoretically the PATHNAME could point to an ancestor. And even if one discarded this possiblity, often directory hierarchies share multiple ASDF definitions, so one would package "source" components that aren't really source components. So, assuming that "deployment outside of Quicklisp from an ASDF definition" would be a greater good for the community, we seem to be stuck at a bootstrap problem. I can't think of a way to force ASDF authors to enumerate their static components, but until such enumeration is widespread, I can't demonstrate its utility by developing an ASDF/BUNDLE operation. Any ideas? -- "A screaming comes across the sky. It has happened before, but there is nothing to compare it to now."
On Thu, Apr 4, 2013 at 10:50 AM, Mark Evenson <evenson@panix.com> wrote:
On 4/4/13 12:44 AM, Anton Vodonosov wrote: […]
General note about deployment CL systems.
Often the deployment package should contain not only compiled code, but also resources: web libraries and frameworks often contain .css, .html, .javascript; bordeaux-threads uses version.lisp-sexp files, and so on.
In addition to the #. reader macro as used in bordeaux-threads, applications/libraries access their resources during run-time using asdf:system-relative-pathname or similar functions. In other words, such libraries assume presence during run-time of their full source control checkout .
Well, systems should declare the resources they want packaged into their applications in their system files. That's a requirement both ECL and ABCL now put forward, but I think other implementations which are able to deliver more or less stand alone programs would want the same.
A general purpose deployment solutions should account this aspect.
One way to deploy applications with full content of the dependencies code and resources: 1. install quicklisp in a custom directory, 2. (ql:quickload :my-application) 3. copy the directory of my-application, the custom quicklisp directory and probably prebuild .fals files to server.
A little bit more docs and example of this approach is here: https://github.com/avodonosov/heroku-buildpack-cl2
[ ... ]
But because "universal distribution" (i.e. for the majority of users of a given system) has more-or-less subsumed by Quicklisp, there is no real constraint on the developers of system definitions to ensure that all components are so enumerated: loading the system from Quicklisp works so nothing is broken, right?
Surely we can add tests for that in cl-test-grid as well?
After Xach, Anton probably has the most practical experience with packaging ASDF definitions for deployment due to his pioneering work on CL-TEST-GRID. I find Anton comments to be a succinct summary of the issues involved in packaging a given ASDF system in Quicklisp. But when facing deployment for an ASDF system that is not in Quicklisp, but has dependencies on Quicklisp systems, Anton's approach won't work so well. One would have to find the .asd definition on the filesystem, then transverse its sub-directories to find the source files.
The above is no longer true: with MONOLITHIC-FASL-OP you can build a fasl which holds a system and all its depedencies, including QL provided dependencies. This means you only need a working quicklisp setup on the development machine.
This has the problem that there is no guarantee that the components of an asd definition exist in subdirectories, as theoretically the PATHNAME could point to an ancestor.
True, but with the above, they don't need to anymore: they'll be included in the monolithic fasl. Even better: the monolithic fasl really doesn't require ASDF at all: you can simply load the FASL with a normal LOAD call and it will load all the embedded FASLs in the right (ASDF derived) order.
And even if one discarded this possiblity, often directory hierarchies share multiple ASDF definitions, so one would package "source" components that aren't really source components.
I'm sorry, I don't understand this bit. Could you try to explain your concern with different words? So, assuming that "deployment outside of Quicklisp from an ASDF
definition" would be a greater good for the community, we seem to be stuck at a bootstrap problem. I can't think of a way to force ASDF authors to enumerate their static components, but until such enumeration is widespread, I can't demonstrate its utility by developing an ASDF/BUNDLE operation.
Well, there is no need to develop the ASDF/BUNDLE operation: Fare did. Buth other than that, the fact that I can now succesfully package bordeaux-threads which we couldn't with ASDF-JAR, I think this is really a step ahead? Any ideas?
[I actually started writing this mail bottom to top, so what's below may be a reiteration...] Well, the great thing about the solution that Fare has now rolled out is that I was able to package and load Bordeax-threads even though it is problematic because it uses the #. reader macro and does not include the version.lisp-sexp in the ASDF definition... The infrastructure I've added to ABCL has been hooked up into ASDF with his new release. Much more doesn't seem to be required to get it all correctly packaged. Package providers can simply use #. reader macros to inject the list of :static-files they need to be distributed with their code and asdf will pick it up. We'll need to work and test ASDF + ABCL to see if everything works as intened though. Bye, Erik.
On 4/4/13 12:39 PM, Erik Huelsmann wrote: […]
Well, systems should declare the resources they want packaged into their applications in their system files. That's a requirement both ECL and ABCL now put forward, but I think other implementations which are able to deliver more or less stand alone programs would want the same.
Updating the ASDF documentation on how to use the new ops as well as recommendations for ASDF definitions would be good steps in this direction. When time and resources are available, of course… […]
But because "universal distribution" (i.e. for the majority of users of a given system) has more-or-less subsumed by Quicklisp, there is no real constraint on the developers of system definitions to ensure that all components are so enumerated: loading the system from Quicklisp works so nothing is broken, right?
Surely we can add tests for that in cl-test-grid as well?
Maybe we could implement an ASDF/LINT that checks the "well-formedness" of a given .asd file, reporting potential problems? Conceptually such checking needs to be maintained "closer" to ASDF than CL-TEST-GRID. Implementation is the sincerest form of flattery, of course… […]
And even if one discarded this possiblity, often directory hierarchies share multiple ASDF definitions, so one would package "source" components that aren't really source components.
I'm sorry, I don't understand this bit. Could you try to explain your concern with different words?
Think of the ABCL-TEST-LISP definition in abcl.asd: it only references files in the test/lisp/abcl sub-directory. So, if one one were to treat the pathname of the .asd file it is defined in, one would include the abcl Java source, possibly the build artifacts, etc. which is probably not what one wants. Or consider CFFI with the cffi-ffi.asd definition, which needs one specific subdirectory.
So, assuming that "deployment outside of Quicklisp from an ASDF definition" would be a greater good for the community, we seem to be stuck at a bootstrap problem. I can't think of a way to force ASDF authors to enumerate their static components, but until such enumeration is widespread, I can't demonstrate its utility by developing an ASDF/BUNDLE operation.
Well, there is no need to develop the ASDF/BUNDLE operation: Fare did. Buth other than that, the fact that I can now succesfully package bordeaux-threads which we couldn't with ASDF-JAR, I think this is really a step ahead?
Conceptually, it sounds like a step ahead but how do I use it. Could you write up a quick document on how to use it? I'm still not sure how things like ASDF:SYSTEM-RELATIVE-PATHNAME would work in the monolithic fasl scenario, so I would like to get my hands dirty a bit here.
Any ideas?
[I actually started writing this mail bottom to top, so what's below may be a reiteration...]
Well, the great thing about the solution that Fare has now rolled out is that I was able to package and load Bordeax-threads even though it is problematic because it uses the #. reader macro and does not include the version.lisp-sexp in the ASDF definition...
The infrastructure I've added to ABCL has been hooked up into ASDF with his new release. Much more doesn't seem to be required to get it all correctly packaged. Package providers can simply use #. reader macros to inject the list of :static-files they need to be distributed with their code and asdf will pick it up. We'll need to work and test ASDF + ABCL to see if everything works as intened though.
And then deprecate ASDF-JAR? In some ways, I thought that being able to present ABCL components as jar files (abcl.jar, abcl-contrib.jar, hunchentoot-all-1.2.14.jar) would be a more natural fit to typical Java server deployment scenarios. Other than this more natural fit to JVM deployments (whereby the pointy-headed boss would be none-the-wiser as "everything just looks like jar dependencies") the only advantage of using a jar format that still might exist would be to leverage the ["One Jar" approach][one-jar] to actually use a custom classloader to make JVM artifacts (libraries, resources, etc.) packaged in the deployment as well. With this sort of approach we could include the jna.jar needed for CFFI in the deployment artifact, as opposed to needing to override going for the resolving the Maven dependency at deployment runtime. [one-jar]: http://one-jar.sourceforge.net/ -- "A screaming comes across the sky. It has happened before, but there is nothing to compare it to now."
Well, systems should declare the resources they want packaged into their
applications in their system files. That's a requirement both ECL and ABCL now put forward, but I think other implementations which are able to deliver more or less stand alone programs would want the same.
Updating the ASDF documentation on how to use the new ops as well as recommendations for ASDF definitions would be good steps in this direction. When time and resources are available, of course…
Completely agreed. I'm going to write a blog item about my own use case, but true, there's a lot more to it than just that.
But because "universal distribution" (i.e. for the majority of users of
a given system) has more-or-less subsumed by Quicklisp, there is no
real
constraint on the developers of system definitions to ensure that all components are so enumerated: loading the system from Quicklisp
works
so nothing is broken, right?
Surely we can add tests for that in cl-test-grid as well?
Maybe we could implement an ASDF/LINT that checks the "well-formedness" of a given .asd file, reporting potential problems? Conceptually such checking needs to be maintained "closer" to ASDF than CL-TEST-GRID. Implementation is the sincerest form of flattery, of course
That's not really what I meant, if I understand you correctly. What I meant is: if we can use cl-test-grid to test if libraries load, then we surely can use cl-test-grid to test if a "compiled-and-loaded-from-monolithic-fals" version works. If it doesn't, I'd take that as an indication the ASDF system is broken (that is, if the loader would work for the non-monolithic case).
And even if one discarded this possiblity, often directory hierarchies share multiple ASDF definitions, so one would package "source" components that aren't really source components.
I'm sorry, I don't understand this bit. Could you try to explain your concern with different words?
Think of the ABCL-TEST-LISP definition in abcl.asd: it only references files in the test/lisp/abcl sub-directory.
That's fine. In this case, the current ASDF system builder includes only the lisp files it knows about.
So, if one one were to treat the pathname of the .asd file it is defined in, one would include the abcl Java source, possibly the build artifacts, etc. which is probably not what one wants.
Right, so I'm not arguing we should include "everything below the system file". I'm saying that system definitions can use #. macros to include files dynamically into the system definition and once that definition is compiled and packed, there's no need to evaluate the #. macro anymore so, loading off a monolithic fasl is still a viable option.
Or consider CFFI with the cffi-ffi.asd definition, which needs one specific subdirectory.
True. Isn't it the case that those subdirectories are currently included using reader conditionals? I'm assuming we're not cross-compiling systems for ABCL on SBCL here. I have a feeling I'm not understanding you on this point.
So, assuming that "deployment outside of Quicklisp from an ASDF
definition" would be a greater good for the community, we seem to be stuck at a bootstrap problem. I can't think of a way to force ASDF authors to enumerate their static components, but until such
enumeration
is widespread, I can't demonstrate its utility by developing an ASDF/BUNDLE operation.
Well, there is no need to develop the ASDF/BUNDLE operation: Fare did. Buth other than that, the fact that I can now succesfully package bordeaux-threads which we couldn't with ASDF-JAR, I think this is really a step ahead?
Conceptually, it sounds like a step ahead but how do I use it. Could you write up a quick document on how to use it? I'm still not sure how things like ASDF:SYSTEM-RELATIVE-PATHNAME would work in the monolithic fasl scenario, so I would like to get my hands dirty a bit here.
I'm not certain with respect to this point. Lets ask Fare about that :-)
[I actually started writing this mail bottom to top, so what's below may be a reiteration...]
Well, the great thing about the solution that Fare has now rolled out is that I was able to package and load Bordeax-threads even though it is problematic because it uses the #. reader macro and does not include the version.lisp-sexp in the ASDF definition...
The infrastructure I've added to ABCL has been hooked up into ASDF with his new release. Much more doesn't seem to be required to get it all correctly packaged. Package providers can simply use #. reader macros to inject the list of :static-files they need to be distributed with their code and asdf will pick it up. We'll need to work and test ASDF + ABCL to see if everything works as intened though.
And then deprecate ASDF-JAR?
Only if that's the right thing to do in the sense that ASDF-JAR's functionality has been subsumed into ASDF3's build facility. However, I don't think we need to: it's a contrib in its own right until it's broken and nobody can be motivated to unbreak it.
In some ways, I thought that being able to present ABCL components as jar files (abcl.jar, abcl-contrib.jar, hunchentoot-all-1.2.14.jar) would be a more natural fit to typical Java server deployment scenarios.
Well, the only difference betweer a FASL and a JAR is the MANIFEST file... I'm sure we can come up with a way to make the .abcls look like jars or even turn them into real ones. However, if there's a file called "my-application--systems.jar", why
Other than this more natural fit to JVM deployments (whereby the pointy-headed boss would be none-the-wiser as "everything just looks like jar dependencies") the only advantage of using a jar format that still might exist would be to leverage the ["One Jar" approach][one-jar] to actually use a custom classloader to make JVM artifacts (libraries, resources, etc.) packaged in the deployment as well. With this sort of approach we could include the jna.jar needed for CFFI in the deployment artifact, as opposed to needing to override going for the resolving the Maven dependency at deployment runtime.
[one-jar]: http://one-jar.sourceforge.net/
Bye, Erik.
participants (5)
-
Alan Ruttenberg -
Anton Vodonosov -
Erik Huelsmann -
James M. Lawrence -
Mark Evenson