ASDF contains some specialized code for ABCL that enable the translation of ASDF systems whose source is packaged into jar files to locate the corresponding object files in ASDF's user cache. In trying to figure out the correct mechanism to allow for "binary" ASDF distribution under ABCL that would include both the source and the fasls in the same jar file, I need to disable the jar file translation mechanism.
I expected that invoking DISABLE-OUTPUT-TRANSLATIONS would clear this built-in translation mechanism, but it doesn't seem to do this. Only invoking CLEAR-OUTPUT-TRANSLATIONS seems to actually change the output translation map. To wit:
ASDF> (disable-output-translations) ((T T) (#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (output-translations) ((#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T T) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (disable-output-translations) ((T T) (#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (output-translations) ((#P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*" T) (#P"/___jar___file___root___/**/*.*" #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*") (#P"jar:file:/**/*.jar!/**/*.*" #<FUNCTION TRANSLATE-JAR-PATHNAME {771931F8}>) (T T) (T #P"/Users/evenson/.cache/common-lisp/abcl-0.26.0-dev-fasl37-macosx-java/**/*.*")) ASDF> (clear-output-translations) ; No value ASDF> (output-translations) NIL ASDF>
Should DISABLE-OUTPUT-TRANSLATIONS actually push that (T T) list into the beginning of that (OUTPUT-TRANSLATION) list or does the API expect that the user use it something like
ASDF> (setf (output-translations) (disable-output-translations))
Should ASDF under ABCL strip out the jar translation mechanism as well, or should the ASDF API expect that the user would actual issue a (CLEAR-OUTPUT-TRANSLATIONS)? I confess that I am a bit confused when one would ever use DISABLE-OUTPUT-TRANSLATIONS as opposed to CLEAR-OUTPUT-TRANSLATIONS.
Once I understand the ASDF API here a bit better, I will work on the necessary ABCL specific ASDF patches if necessary.
On 17 May 2011 07:41, Mark Evenson evenson@panix.com wrote:
ASDF contains some specialized code for ABCL that enable the translation of ASDF systems whose source is packaged into jar files to locate the corresponding object files in ASDF's user cache.
The ASDF API: * clear-output-translations means "clear the current state, reload it from configuration when needed". It's useful when the configuration has or will change, e.g. before saving an image. It might not help you much here.
* disable-output-translations means "disable any output translation except those necessary for the system to run" as defined in wrapping-output-translations. Unhappily, that's where we put those jar file translations for ABCL, instead of, say, some entry at the bottom of *default-output-translations*.
Also, importantly, output-translations are sorted from most-specific to least-specific, so when you disable, even though the default for everything is to translate to same directory, jar files will still be translated to the cache. As they probably should be by default indeed.
Instead of disabling translations, you ought to add some specific translation (more specific than the default jar translation) that will tell ASDF that indeed for this or that jar, the fasls are stored in the jar, with an entry like: (:output-translations (#p"jar:file:/path/to/my.jar!/**/*.*") :inherit-output-translations)
Hope this helps.
BTW, for cl-launch and XCVB, I indeed am looking for a way to create bundles from compiled stuff. How do I create a jar from ABCL and a set of lisp files, precompiled or not?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Most people think they need a ruler. Perhaps we should give them a fake one that doesn't actually do anything, and then they won't think about it. It is sort of like giving an infant a pacifier. — Perry Metzger
On 5/17/11 3:28 PM, Faré wrote: […]
Hope this helps.
I'm gonna need a bit more time to chew over the ASDF code, but thanks for the general direction. I must confess that in spite of the reasonable looking documentation and having contributed the function translation code to ASDF, its whole output translation API has never really gelled in my understanding as a totality. Maybe I can contribute some examples to the texinfo file when the fuzz in my understanding resolves a bit.
BTW, for cl-launch and XCVB, I indeed am looking for a way to create bundles from compiled stuff. How do I create a jar from ABCL and a set of lisp files, precompiled or not?
A reasonable packaging mechanism for ABCL plus "compiled stuff" composed of ASDF system definitions that works well in a Java ecosystem is precisely the itch I'm in the process of scratching for ABCL right now. The base unit of packaging in Java is the jar file, which is nothing more than a ZIP archive with (possibly) some Java-the-language specific metadata. Currently, one can package an ASDF definition in a jar file, push the location of the asd file into ASDF:*CENTRAL-REGISTRY* using the [ABCL JAR-PATHNAME conventions][1] for which a subsequent ASDF:LOAD-SYSTEM will do the right thing.
[1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathname...
For instance, if you were to package up cl-ppcre installed via [quicklisp][10^9monkey-wants-this] in a jar file via
unix$ cd ~/quicklisp/dists/quicklisp/software && jar cfv /tmp/cl-ppcre-2.0.3.jar cl-ppcre-2.0.3
[10^9monkey-wants-this]: http://www.quicklisp.org
one could subsequently enable an ASDF controlled load of this system via
CL-USER> (push "jar:file:/tmp/cl-ppcre-2.0.3.jar!/cl-ppcre-2.0.3/" asdf:*central-registry)
When loaded, ASDF will compile the system to the user cache.
Some problems here that I'm working through:
1) You can't immediately load FASL out of the jar. In my current hackish way—i.e. without comprehending your advice yet—one has first ASDF compile the system with output translations disabled, and then
(defmethod asdf:perform ((o asdf:compile-op) (c asdf:cl-source-file))) (setf (asdf::output-translations) '((t t))))
in the target JVM to load the ABCL FASLS from the jar.
2) One has to specify the absolute path on the local filesystem (or potentially via a URI) for the jar, which makes things a bit fragile in the typical Java ecosystem usage which shuffles jars around like win32 DLLs (or, to be fair, Unix dynamic libraries) relying merely on the filename to keep things straight. My current insight into a way around would be to define another PATHNAME extension in ABCL for CLASSPATH entries, i.e. "classpath:cl-ppcre-2.0.3.jar!/cl-ppcre-2.0.3/" would refer to the named jar in the JVM CLASSPATH if it exists.
3) ABCL [has a bug][#149] that currently prevents ASDF systems located in the top-level entry of a JAR from being accessed.
[#149]: http://trac.common-lisp.net/armedbear/ticket/149
4) The extremely nice use of [JSS][jss] and [ABCLD's slight modification][abcld] of ASDF to also refer to jar files to dynamically load into the JVM probably needs to be rewritten, otherwise we run into the situation whereby we have jars (i.e. the packaged Java code) within the ASDF packaged jar which A) needs changes within ABCL to completely work and B) would be rather inefficient in that the naive implementation each request for a new entry in the JAR within a JAR would require a complete "reseek" through the enclosed ZIP file.
[jss]: http://code.google.com/p/lsw2/source/browse/#svn%2Ftrunk%2Fjss [abcld]: http://code.google.com/p/abcl-dynamic-install/source/browse?repo=abcld
5) A fear of mine: if I enable all this, I presume that people would start going around creating 'abcl.jar' files with different inclusions of different ASDF packagings. Without a real smart dynamic introspection system that essentially solves the problems in JVM's classpath we would end up with, at the minimum, a rather hostile situation for the end user. "Put 'abcl.jar' in your classpath." "I did, and it still didn't load Maxima." "Well, what's the checksum of your abcl.jar?" "c48d359a23ee" "Oh, you need 846f78c279cb". Ideally, I would like to come up with a mechanism that would require that 'abcl.jar' come from "official" ABCL packaging, but would somehow be able to introspect the JVM classpath to include ASDF definitions.
Comments solicited.
I'm gonna need a bit more time to chew over the ASDF code, but thanks for the general direction. I must confess that in spite of the reasonable looking documentation and having contributed the function translation code to ASDF, its whole output translation API has never really gelled in my understanding as a totality. Maybe I can contribute some examples to the texinfo file when the fuzz in my understanding resolves a bit.
output-translations is not THAT hard to understand, if you read the source code with my previous comments in mind. Patches to the docs are welcome, too.
A reasonable packaging mechanism for ABCL plus "compiled stuff" composed of ASDF system definitions that works well in a Java ecosystem is precisely the itch I'm in the process of scratching for ABCL right now.
Excellent. I hope you provide an API that isn't so ASDF-specific that it's a pain to use with XCVB -- though XCVB can use ASDF as a backend, and considering the slow ABCL start time, it's probably to be the preferred backend on ABCL.
- You can't immediately load FASL out of the jar.
Is that an ABCL limitation?
In my current hackish way—i.e. without comprehending your advice yet—one has first ASDF compile the system with output translations disabled, and then
(defmethod asdf:perform ((o asdf:compile-op) (c asdf:cl-source-file))) (setf (asdf::output-translations) '((t t))))
in the target JVM to load the ABCL FASLS from the jar.
Why that pain? Doesn't the zip process preserve the timestamps?
- One has to specify the absolute path on the local filesystem (or
potentially via a URI) for the jar, which makes things a bit fragile in the typical Java ecosystem usage which shuffles jars around like win32 DLLs (or, to be fair, Unix dynamic libraries) relying merely on the filename to keep things straight. My current insight into a way around would be to define another PATHNAME extension in ABCL for CLASSPATH entries, i.e. "classpath:cl-ppcre-2.0.3.jar!/cl-ppcre-2.0.3/" would refer to the named jar in the JVM CLASSPATH if it exists.
You want CL systems to be in a variety of jars, rather than just a one jar with everything in it?
- The extremely nice use of [JSS][jss] and [ABCLD's slight
modification][abcld] of ASDF to also refer to jar files to dynamically load into the JVM probably needs to be rewritten, otherwise we run into the situation whereby we have jars (i.e. the packaged Java code) within the ASDF packaged jar which A) needs changes within ABCL to completely work and B) would be rather inefficient in that the naive implementation each request for a new entry in the JAR within a JAR would require a complete "reseek" through the enclosed ZIP file.
I welcome patches to ASDF and/or implementation-specific contributions, just like our asdf-ecl.lisp.
- A fear of mine: if I enable all this, I presume that people would start
going around creating 'abcl.jar' files with different inclusions of different ASDF packagings. Without a real smart dynamic introspection system that essentially solves the problems in JVM's classpath we would end up with, at the minimum, a rather hostile situation for the end user. "Put 'abcl.jar' in your classpath." "I did, and it still didn't load Maxima." "Well, what's the checksum of your abcl.jar?" "c48d359a23ee" "Oh, you need 846f78c279cb". Ideally, I would like to come up with a mechanism that would require that 'abcl.jar' come from "official" ABCL packaging, but would somehow be able to introspect the JVM classpath to include ASDF definitions.
Maybe make it hard for anything but the original to be named abcl.jar?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Backwards compatible — If it's not backwards it's not compatible — Greg Newton gregnewton@netscape.net
On 5/17/11 10:01 PM, Faré wrote:
I'm gonna need a bit more time to chew over the ASDF code, but thanks for the general direction. I must confess that in spite of the reasonable looking documentation and having contributed the function translation code to ASDF, its whole output translation API has never really gelled in my understanding as a totality. Maybe I can contribute some examples to the texinfo file when the fuzz in my understanding resolves a bit.
output-translations is not THAT hard to understand, if you read the source code with my previous comments in mind. Patches to the docs are welcome, too.
I agree it shouldn't be that hard. And every time I read the code and docs I go "Uh-huh, now I understand." But somehow, like Perl code, when I come back to it two weeks later, I have to start from the beginning. I guess I'm just putting the proverbial stake in the ether to mark my frustration in case others have the same problem, as a) I'm old enough not to care that much if other people think I'm an idiot and b) maybe it's not just me. In amplifying my concern here, I definitely owe you a diff or two to asdf.texinfo to reward your patient explanations…
[…]
- You can't immediately load FASL out of the jar.
Is that an ABCL limitation?
Certainly not: abcl.jar contains FASLs, after all.
In my current hackish way—i.e. without comprehending your advice yet—one has first ASDF compile the system with output translations disabled, and then
(defmethod asdf:perform ((o asdf:compile-op) (c asdf:cl-source-file))) (setf (asdf::output-translations) '((t t))))
in the target JVM to load the ABCL FASLS from the jar.
Why that pain? Doesn't the zip process preserve the timestamps?
Timestamps should be preserved, but the current output translation code looks for FASLs in the user cache. Well, by that reasoning including both statements should be redundant. I need to look at this again…
[…]
You want CL systems to be in a variety of jars, rather than just a one jar with everything in it?
Yep. I wouldn't mind being able to include multiple systems in a single jar, but I want to keep the core abcl separate from the systems in order to be able to consistently rev the core (and, of course, the systems) via checksums.
- The extremely nice use of [JSS][jss] and [ABCLD's slight
modification][abcld] of ASDF to also refer to jar files to dynamically load into the JVM probably needs to be rewritten, otherwise we run into the situation whereby we have jars (i.e. the packaged Java code) within the ASDF packaged jar which A) needs changes within ABCL to completely work and B) would be rather inefficient in that the naive implementation each request for a new entry in the JAR within a JAR would require a complete "reseek" through the enclosed ZIP file.
I welcome patches to ASDF and/or implementation-specific contributions, just like our asdf-ecl.lisp.
I have my eye on that, and when I have mature patches to ASDF that work with ABCL out of the box, I will submit them for consideration. But currently, one needs a specialized build of ABCL with the jscheme and bsh jars in the classpath in order to use the jar-file extensions so this isn't practical. Given the amount of ABCL's lunch that Clojure has eaten, maybe it's time to make some post-ANSI CL decisions in our implementation wrt. ABCL Java/CL interop.
- A fear of mine: if I enable all this, I presume that people would start
going around creating 'abcl.jar' files with different inclusions of different ASDF packagings. Without a real smart dynamic introspection system that essentially solves the problems in JVM's classpath we would end up with, at the minimum, a rather hostile situation for the end user. "Put 'abcl.jar' in your classpath." "I did, and it still didn't load Maxima." "Well, what's the checksum of your abcl.jar?" "c48d359a23ee" "Oh, you need 846f78c279cb". Ideally, I would like to come up with a mechanism that would require that 'abcl.jar' come from "official" ABCL packaging, but would somehow be able to introspect the JVM classpath to include ASDF definitions.
Maybe make it hard for anything but the original to be named abcl.jar?
Not really possible as far as I can figure things to avoid the ability to name things 'abcl.jar' (or are you being sarcastic?) Solving the dynamic introspection problem seems more tractable, at the corresponding runtime cost of building a cache of the contents jars that the JVM knows about, but memory is getting cheaper by the ns, n'est-ce pas?
On 5/17/11 15:28 , Faré wrote: […]
Instead of disabling translations, you ought to add some specific translation (more specific than the default jar translation) that will tell ASDF that indeed for this or that jar, the fasls are stored in the jar, with an entry like: (:output-translations (#p"jar:file:/path/to/my.jar!/**/*.*") :inherit-output-translations)
Hope this helps.
I think you meant
(:output-translation (#p"jar:file:/path/to/my.jar!/**/*.*") :inherit-configuration)
which does indeed help.
On 5/17/11 15:28 , Faré wrote:
BTW, for cl-launch and XCVB, I indeed am looking for a way to create bundles from compiled stuff. How do I create a jar from ABCL and a set of lisp files, precompiled or not?
I've recently committed [ASDF-JAR which is a new contrib for ABCL][1] that should allow you package ASDF systems possibly with dependencies. When you get the time to check it out, please let me know if it meets your needs. I would be grateful for feedback of all types.
[1]: http://trac.common-lisp.net/armedbear/browser/trunk/abcl/contrib/asdf-jar/RE...
-
Faré -
Mark Evenson