Following up on this,
I have some code by modifying some of the call chain for compile, jvm-compile etc in compiler-pass2.lisp that lets me generate class files where the compiled code for top level functions are stored. I have checked that I can load this and call this (and managed to do this without initializing the interpreter) from pure java land. One case that is not covered is a set of defuns within a let binding, for example
(let (y) (defun get-y() y) (defun set-y (x) (setq y x)) (defun add-x (x) (+ x y)))
right now compile-to-file only lets me compile these separately, and creates separate symbols static fields in each file separately and tries to initialize them in the static initialization blocks using a Lisp.recall function call. There is something funky going on in the compilation re whether this is in memory or not, but each function gets its own class and there does not seem an easy way to compile just the let form in the top level.
I will write and tests to see if my changes to jnew-runtime-class can work as well, and would like to co-ordinate on how to do more extensive testing and merging.
Archi
On Fri, Sep 21, 2012 at 11:52 AM, Alessio Stalla
<alessiostalla@gmail.com> wrote:
On Fri, Sep 21, 2012 at 4:39 PM, archisman rudra <
archi.rudra@gmail.com> wrote:
> Hi,
>
> hopefully this is the correct list. If not please let me know.
yes, you are writing to the correct list - welcome!
> Background:
> I am trying to use abcl to generate jars to send off to a hadoop cluster for
> execution.
Interesting! I'm answering because I wrote the current (incomplete)
version of jnew-runtime-class. Just FYI, you can find some notes about
it at <http://trac.common-lisp.net/armedbear/wiki/JavaFfi/RuntimeClass>.
> The svn version of the jnew-runtime-class does not store the
> actual method bodies in the generated class file, but rather loads the
> generated class file and sets the relevant field. Here is the relevant code
> :
>
> (multiple-value-bind (class-file method-implementation-fields)
> (apply #'java::%jnew-runtime-class class-name stream args)
> (sys::put-memory-function memory-class-loader
> class-name (sys::%get-output-stream-bytes
> stream))
> (let ((jclass (java:jcall "loadClass" memory-class-loader
> class-name)))
> (dolist (method method-implementation-fields)
> (setf (java:jfield jclass (car method)) (cdr method)))
> jclass))))
>
>
> This means that the generated jar file cannot be easily used from standard
> java code.
This is a byproduct of the design - although the design is mine, and
hasn't been discussed much among the other developers, so I'm not
representing anyone but myself. Let me clarify below.
> I would like to add some code in the static initialization block
> of generated class file to do something equivalent. I will have to save the
> lambda expression for the method and have been experimenting as follows:
>
> (defvar *cf*)
> (defvar *mif*)
>
> (with-open-file (stream "test.class" :direction :output :element-type
> '(unsigned-byte 8))
> (multiple-value-bind (class-file method-implementation-field)
> (java::%jnew-runtime-class "test" stream :methods
> `(("forty_two_method1" :int () ,(lambda (this) 42))))
> (setf *cf* class-file)
> (setf *mif* method-implementation-field)))
>
> (with-open-file (stream "test_forms" :direction :output)
> (dolist (m *mif*)
> (system:dump-form (list (car m) (function-lambda-expression (cdr m)))
> stream)))
>
>
> This assumes that the lambda expression does not refer to any variable or
> user defined function, but perhaps the asdf-jar contrib might help here.
>
> I was going to write the relevant code and start using it for my project,
> but I was wondering if I am duplicating work that others have already done
> or are doing and whether there are other libraries (jffli perhaps, I am not
> very familiar with it) that have similar functionality. Also if there are
> other interactions with other parts of the system that I am perhaps missing.
>
> I would e grateful for any advice.
What I wanted to achieve is to have methods that can be:
* arbitrary functions, including closures
* possibly redefined after the class has been loaded. Redefinition is
possible by using symbols as method implementations.
There's also a hidden goal that is to keep the implementation simple
enough, and compiling Lisp code to the body of methods in an arbitrary
class is not easy at the moment.
What explicitly is NOT a requirement is to have the generated class be
independent from ABCL (the methods are implemented in Lisp and will
always need runtime support). So, no matter how you arrange things,
you will never be able to distribute your class without abcl.jar. Note
that it is the same for Clojure, Scala, Groovy, etc.
I don't know specifically how Hadoop works, but, as I see things, you need to
1. provide abcl.jar + your class to it
2. have Hadoop execute enough code to load ABCL and initialize your class.
Now, probably the key point is who loads your class.
* if you can arrange to have ABCL load it and pass it to Hadoop, you
should be good with the current design.
* if Hadoop wants to load it by itself from a jar, then yes, you have
to somehow dump the method implementations somewhere and have the
class auto-load them from there when it is initialized.
For the second case, I propose an intermediate design that is, imho,
simpler than dumping arbitrary forms: special-case the situation when
a method implementation is a symbol. In that case, directly generate
an instruction in the class file to find that symbol in its package
and assign it to the appropriate field. If the implementation of a
method is not a symbol nothing changes. That way, you will still need
to init ABCL before loading your class, but, when all your method
implementations are symbols, the class will be able to initialize
itself without external support. If some methods are proper functions,
instead, the class won't be self-contained, but will need some Lisp
code to be executed to be properly initialized.
Does it make sense to you? Do you find it acceptable? If yes, I can
give a shot at implementing it, if you want.
Peace,
Alessio
--
Some gratuitous spam:
http://ripple-project.org Ripple, social credit system
http://villages.cc Villages.cc, Ripple-powered community economy
http://common-lisp.net/project/armedbear ABCL, Common Lisp on the JVM
http://code.google.com/p/tapulli my current open source projects
http://www.manydesigns.com/ ManyDesigns Portofino, open source
model-driven Java web application framework