[armedbear-devel] In-memory compilation status
Hello list, I'm interested in trying to get ABCL running (and preferably REPL-ing & compiling) on a certain cloud computing platform that has a JVM with no access to a file system. The Wiki page for JSR-223 and the sample code both mention that "compilation using the runtime compiler has been removed due to inconsistencies with evaluation and file-based compilation", since May '09. Browsing the mailing list archives suggests that people have been working on removing the need for temporary files since then, but I can't find any mention of "inconsistencies". The thread "Save-image w/Serialization - progress report" from April ends with Sig. Stalla in the process of implementing a byte-array-output-stream, then silence... Can someone comment on the state of runtime compilation, preferably entirely in-memory without reference to a file system? Basically I'd like to be able to compile stuff to a byte array, store the array as a blob in a memcache and/or other datastore, then pull it out again as required and load it via a custom classloader. Is this doable, or would it be easier to port ParenScript to Clojure (which is currently my Plan B)? Cheers, and it's great to see the ABCL project still going strong! John :^P P.S. If anyone is going to be in Hamburg at the weekend and wants to chat about this stuff, that'd be great! -- John Pallister john@johnp.net john@synchromesh.com
On Tue, Sep 8, 2009 at 2:19 AM, John Pallister<john@synchromesh.com> wrote:
Hello list,
I'm interested in trying to get ABCL running (and preferably REPL-ing & compiling) on a certain cloud computing platform that has a JVM with no access to a file system.
The Wiki page for JSR-223 and the sample code both mention that "compilation using the runtime compiler has been removed due to inconsistencies with evaluation and file-based compilation", since May '09. Browsing the mailing list archives suggests that people have been working on removing the need for temporary files since then, but I can't find any mention of "inconsistencies". The thread "Save-image w/Serialization - progress report" from April ends with Sig. Stalla in the process of implementing a byte-array-output-stream, then silence...
Hello John, Sig. Stalla sounds pompous, you can call me Alessio ;) Those you cite are really 2 separate issues. 1. For the JSR-223 part, it used to use the cl function COMPILE to compile code using the runtime compiler. However its behavior differs from COMPILE-FILE in ways that could result in unexpected behavior, so now the ABCL script engine uses COMPILE-FILE on its own, by saving the code to compile to temporary files. This could be fixed by using a primitive that works like COMPILE-FILE but is able to read from any stream, included from a string in memory. I don't exclude such a thing already exists, and only needs to be polished and made public. 2. However in general, JSR-223 or not, the ABCL compiler currently always uses files at least when compiling local functions, even if you use the runtime compiler. That is what you see in the serialization thread. Changing this is more problematic, since it probably requires changing the way compiled Lisp code is loaded and the instructions emitted by the compiler to load compiled local functions, and I'm afraid also the order in which the compiler does some things (wild guess here!). I'm also interested in this change, but I don't have much time to work on it.
Can someone comment on the state of runtime compilation, preferably entirely in-memory without reference to a file system?
Basically I'd like to be able to compile stuff to a byte array, store the array as a blob in a memcache and/or other datastore, then pull it out again as required and load it via a custom classloader. Is this doable, or would it be easier to port ParenScript to Clojure (which is currently my Plan B)?
A relatively quick fix would be to implement a virtual file API that writes either on disk or in memory (in a dummy, virtual filesystem). The compiler would be only changed slightly to use the more high-level API instead of direct file access. Primitives to load compiled code would need to look at some variable to determine where to load code from. The virtual filesystem could be exposed to the user so it could be persisted & restored across sessions. I think this approach is not enough to solve the problems with serialization (because the virtual fs needs to be manually saved/restored, since loaded code does not know where it is loaded from), but it's probably ok for situations like yours. Note that you'd have to use COMPILE-FILE and not COMPILE with the virtual fs, in order to have a known entry point (the virtual file name) when loading code later. Also maybe a finer-grained control on the virtual fs could be given to the user, so you'd be able to decide the life span of the fs. If others have ideas in this regard, please share. I'd like to avoid Plan B (having users resort to the competition :D) Bye, Alessio
Cheers, and it's great to see the ABCL project still going strong!
John :^P
P.S. If anyone is going to be in Hamburg at the weekend and wants to chat about this stuff, that'd be great! -- John Pallister john@johnp.net john@synchromesh.com
_______________________________________________ armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On 9/8/09 10:27 AM, Alessio Stalla wrote: […]
If others have ideas in this regard, please share. I'd like to avoid Plan B (having users resort to the competition :D)
This seems somewhat related to my recent work to load ABCL FASLs from JAR files, in that I have had to change Lisp.loadCompiledFile() to get things to work. In that new code, if I can address a FASL via a URL resolvable via java.net.URL, I can load the FASL. Does the cloud platform being considered here have a way to address its storage as URLs? I don't quite see the parts of the compiler that need to write to the filesystem. Can anyone point me to the places where that happens? -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Tue, Sep 8, 2009 at 12:45 PM, Mark Evenson<evenson@panix.com> wrote:
On 9/8/09 10:27 AM, Alessio Stalla wrote: […]
If others have ideas in this regard, please share. I'd like to avoid Plan B (having users resort to the competition :D)
This seems somewhat related to my recent work to load ABCL FASLs from JAR files, in that I have had to change Lisp.loadCompiledFile() to get things to work. In that new code, if I can address a FASL via a URL resolvable via java.net.URL, I can load the FASL.
Does the cloud platform being considered here have a way to address its storage as URLs?
I don't quite see the parts of the compiler that need to write to the filesystem. Can anyone point me to the places where that happens?
Going from memory, when it compiles local functions/closures, it needs to write each one to its own class file. The compiled top-level function will contain instructions like loadCompiledFunction("tempClassFileFoo1234.cls"). After the compiled function is loaded, if the runtime compiler has been used, temporary files are deleted. I also vaguely remember that the compiler uses temporary memory (remember/recall) for constants and other things, and generates code in the compiled class files like 'fetch contant 1234 from the compiler'. So, if it's still like this, that's a problem for the OP, too. But I haven't got the abcl source handy now, so that might be incorrect. hth, Alessio
On 9/8/09 1:04 PM, Alessio Stalla wrote: […]
I don't quite see the parts of the compiler that need to write to the filesystem. Can anyone point me to the places where that happens?
Going from memory, when it compiles local functions/closures, it needs to write each one to its own class file. The compiled top-level function will contain instructions like loadCompiledFunction("tempClassFileFoo1234.cls"). After the compiled function is loaded, if the runtime compiler has been used, temporary files are deleted.
I also vaguely remember that the compiler uses temporary memory (remember/recall) for constants and other things, and generates code in the compiled class files like 'fetch contant 1234 from the compiler'. So, if it's still like this, that's a problem for the OP, too.
But I haven't got the abcl source handy now, so that might be incorrect.
Ok, I understand the first use now, still hunting for the second. A possible strategy to cover the first use is by binding a per-thread special variable on entrance to CL:LOAD that would tell Lisp.loadCompiledFunction() where to look if it can't find the requested 'file'. I started to follow that strategy for the "load from JAR files patch", but it turned out not to be necessary. More investigation in a bit… -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Hi Mark, That sounds interesting. I take it you're not using the compiler as such, just loading code from across the network? Google provide their own Java API to the App Engine datastore, and provide JDO-compatible wrappers. There's another JSR-compliant interface to the memcache. So something that allowed the compiler to use JDO rather than the file system might be a worthwhile idea? Cheers, John :^P
Hi Alessio, thanks for your quick response. So, from reading your message the options would seem to be: 1. Fix COMPILE so it no longer produces "unexpected behaviour". In what way was it (mis-)behaving, and how was this unexpected? 2. Alter COMPILE so it just uses the some code as COMPILE-FILE, only with output to a byte stream instead of a file stream. Is this what you were doing? If so, how did you get on? 3. Alter COMPILE-FILE so that it doesn't use files for local functions. (This would be required for 2 above, I guess.) From your comments, this sounds like it might be tricky. Are the files just used as a cache, or are they re-loaded again at a later date? 4. Add some sort of "VFS factory" that can provide either the real file I/O calls or some "in-memory JAR" thing, and then modify COMPILE-FILE to use that. This would imply a performance penalty for the current "use real files" case due to the additional layer of indirection. This boils down to "remove the need for files during compilation" vs. "remove the need for a file system during compilation". Obviously one would be much easier than the other. It sounds like you and I would be interested in an ABCL that didn't rely on files for compilation, but I don't know whether that would be considered a Good Thing by everyone else. Re: Clojure, it looks OK, but I'd be happy to stick with good old Lisp for now (especially since my JVM target has no threads as well as no file system, so all its concurrency benefits are moot). And having to hack Java to get ABCL to work is OK, as long as I'm on the right path (and not inadvertently forking the project). Cheers, John :^P P.S. I'm talking about Google App Engine, in case anyone hasn't guessed yet. -- John Pallister john@johnp.net john@synchromesh.com
On Tue, Sep 8, 2009 at 1:05 PM, John Pallister<john@synchromesh.com> wrote:
Hi Alessio, thanks for your quick response.
So, from reading your message the options would seem to be:
1. Fix COMPILE so it no longer produces "unexpected behaviour". In what way was it (mis-)behaving, and how was this unexpected?
Sorry, I wasn't clear enough. COMPILE behaves differently from COMPILE-FILE as per the CL standard, to the point that (load (compile-file "file.lisp")) might not have the same effects as something like (loop :for x :in (read-all-forms-in "file.lisp") :do (funcall (compile `(lambda () ,x)))) - I don't remember the exact details, but I explicitly asked in comp.lang.lisp and I was advised against using COMPILE for that sort of thing. (the loop is only pseudo-code, just to make an example). Of the two, COMPILE-FILE is the one that is closer to LOADing the uncompiled source, so I chose that for uniformity, that way evaluate(script) behaves the same as (LOAD script) and compile(script).evaluate(), modulo compilation of course.
2. Alter COMPILE so it just uses the some code as COMPILE-FILE, only with output to a byte stream instead of a file stream. Is this what you were doing? If so, how did you get on?
No, IIRC COMPILE and COMPILE-FILE both use lower-level functions and it's those functions that presently require temporary files for some of their work. I actually just read some parts of the compiler sources and discussed a couple of things with Erik, but never got to the point of modifying the compiler.
3. Alter COMPILE-FILE so that it doesn't use files for local functions. (This would be required for 2 above, I guess.) From your comments, this sounds like it might be tricky. Are the files just used as a cache, or are they re-loaded again at a later date?
Heh, it depends. When COMPILE-FILE is called, files for local functions are kept, because you'll need them when you'll load the main FASL. When COMPILE is called, they are removed after compilation, as they are only used when loading the compiled function, before COMPILE returns. This again is from memory, but I'm pretty sure about it.
4. Add some sort of "VFS factory" that can provide either the real file I/O calls or some "in-memory JAR" thing, and then modify COMPILE-FILE to use that. This would imply a performance penalty for the current "use real files" case due to the additional layer of indirection.
Yes, but only for the compiler; I don't think this is much of a problem performance wise.
This boils down to "remove the need for files during compilation" vs. "remove the need for a file system during compilation". Obviously one would be much easier than the other.
It sounds like you and I would be interested in an ABCL that didn't rely on files for compilation, but I don't know whether that would be considered a Good Thing by everyone else.
I believe at least Erik wanted this, too. Generally I don't think it has negative side effects - it would be just a matter of adding new possibilities, not removing anything.
Re: Clojure, it looks OK, but I'd be happy to stick with good old Lisp for now (especially since my JVM target has no threads as well as no file system, so all its concurrency benefits are moot). And having to hack Java to get ABCL to work is OK, as long as I'm on the right path (and not inadvertently forking the project).
Hehe I was only half serious about Clojure. As always, the right thing is to use the tool that is best suited for the job. Still, if ABCL has a problem that makes it not be a viable option in certain cases, I'd like to remove the problem if it can be done (and if it doesn't introduce worse problems, of course).
P.S. I'm talking about Google App Engine, in case anyone hasn't guessed yet.
Interesting, since GAE is one of the platforms ABCL is advertised to work on ;) Bye, Alessio
Hi Alessio, thanks for that explanation (and sorry for my potentially ill-informed questions...). So if the (compressed) FASL format is a Zip archive containing the class files, would it be worth compiling local functions to an in-memory Zip/JAR image along with the top-level forms' classes, and then COMPILE-FILE can write the whole JAR to disk as the FASL (and everyone else can discard it)? (This may be like what happens now; the Wiki page on FASL formats is a bit ambiguous, and I admit I haven't looked through the ABCL code yet.) Anyway, on the strength of this brief discussion I can/should/will now go away and poke through the code. If/when I come up with anything interesting I shall report back to the list. Cheers, John :^P
On Tue, Sep 8, 2009 at 5:19 PM, John Pallister<john@synchromesh.com> wrote:
Hi Alessio, thanks for that explanation (and sorry for my potentially ill-informed questions...).
So if the (compressed) FASL format is a Zip archive containing the class files, would it be worth compiling local functions to an in-memory Zip/JAR image along with the top-level forms' classes, and then COMPILE-FILE can write the whole JAR to disk as the FASL (and everyone else can discard it)? (This may be like what happens now; the Wiki page on FASL formats is a bit ambiguous, and I admit I haven't looked through the ABCL code yet.)
Well that would be the optimal solution imho. What happens now IIRC is something like this: an uncompressed FASL file is created for each top-level function (form?), along with companion fasls for local functions. Then all these files are packaged in a jar/zip if compile-file is used, or the only one top-level function is loaded via loadCompiledFunction and all the files deleted, if compile is used.
Anyway, on the strength of this brief discussion I can/should/will now go away and poke through the code. If/when I come up with anything interesting I shall report back to the list.
That is much appreciated! Thanks, Alessio
participants (3)
-
Alessio Stalla -
John Pallister -
Mark Evenson