On Mon, Oct 1, 2012 at 11:06 PM, archisman rudra archi.rudra@gmail.com wrote:

Following up on this,

I have some code by modifying some of the call chain for compile, jvm-compile etc in compiler-pass2.lisp that lets me generate class files where the compiled code for top level functions are stored. I have checked that I can load this and call this (and managed to do this without initializing the interpreter) from pure java land. One case that is not covered is a set of defuns within a let binding, for example

(let (y) (defun get-y() y) (defun set-y (x) (setq y x)) (defun add-x (x) (+ x y)))

right now compile-to-file only lets me compile these separately, and creates separate symbols static fields in each file separately and tries to initialize them in the static initialization blocks using a Lisp.recall function call. There is something funky going on in the compilation re whether this is in memory or not, but each function gets its own class and there does not seem an easy way to compile just the let form in the top level.

I will write and tests to see if my changes to jnew-runtime-class can work as well, and would like to co-ordinate on how to do more extensive testing and merging.

Archi

Hi Archi,

sorry for not following up for a long time. I really appreciate what you're doing, but unfortunately I have little time to dedicate to ABCL these days. If you are willing to cook up a patch, I could review it. Also, I'm rarely on #abcl on IRC, but we can schedule a virtual meeting there to better discuss the details if you want (and to hear ideas from other devs as well!).

Cheers, Alessio

On Fri, Sep 21, 2012 at 11:52 AM, Alessio Stalla alessiostalla@gmail.com wrote:

...

On Fri, Sep 21, 2012 at 4:39 PM, archisman rudra archi.rudra@gmail.com wrote:

...

Hi,

hopefully this is the correct list. If not please let me know.

Hi,

yes, you are writing to the correct list - welcome!

...

Background: I am trying to use abcl to generate jars to send off to a hadoop cluster for execution.

Interesting! I'm answering because I wrote the current (incomplete) version of jnew-runtime-class. Just FYI, you can find some notes about it at http://trac.common-lisp.net/armedbear/wiki/JavaFfi/RuntimeClass.

...

The svn version of the jnew-runtime-class does not store the actual method bodies in the generated class file, but rather loads the generated class file and sets the relevant field. Here is the relevant code :

(multiple-value-bind (class-file method-implementation-fields)
    (apply #'java::%jnew-runtime-class class-name stream args)
  (sys::put-memory-function memory-class-loader
                            class-name

(sys::%get-output-stream-bytes stream)) (let ((jclass (java:jcall "loadClass" memory-class-loader class-name))) (dolist (method method-implementation-fields) (setf (java:jfield jclass (car method)) (cdr method))) jclass))))

This means that the generated jar file cannot be easily used from standard java code.

This is a byproduct of the design - although the design is mine, and hasn't been discussed much among the other developers, so I'm not representing anyone but myself. Let me clarify below.

...

I would like to add some code in the static initialization block of generated class file to do something equivalent. I will have to save the lambda expression for the method and have been experimenting as follows:

(defvar *cf*) (defvar *mif*)

(with-open-file (stream "test.class" :direction :output :element-type '(unsigned-byte 8)) (multiple-value-bind (class-file method-implementation-field) (java::%jnew-runtime-class "test" stream :methods `(("forty_two_method1" :int () ,(lambda (this) 42)))) (setf *cf* class-file) (setf *mif* method-implementation-field)))

(with-open-file (stream "test_forms" :direction :output) (dolist (m *mif*) (system:dump-form (list (car m) (function-lambda-expression (cdr m))) stream)))

This assumes that the lambda expression does not refer to any variable or user defined function, but perhaps the asdf-jar contrib might help here.

I was going to write the relevant code and start using it for my project, but I was wondering if I am duplicating work that others have already done or are doing and whether there are other libraries (jffli perhaps, I am not very familiar with it) that have similar functionality. Also if there are other interactions with other parts of the system that I am perhaps missing.

I would e grateful for any advice.

What I wanted to achieve is to have methods that can be:

• arbitrary functions, including closures
• possibly redefined after the class has been loaded. Redefinition is

possible by using symbols as method implementations.

There's also a hidden goal that is to keep the implementation simple enough, and compiling Lisp code to the body of methods in an arbitrary class is not easy at the moment.

What explicitly is NOT a requirement is to have the generated class be independent from ABCL (the methods are implemented in Lisp and will always need runtime support). So, no matter how you arrange things, you will never be able to distribute your class without abcl.jar. Note that it is the same for Clojure, Scala, Groovy, etc.

I don't know specifically how Hadoop works, but, as I see things, you need to

1. provide abcl.jar + your class to it
2. have Hadoop execute enough code to load ABCL and initialize your

class.

Now, probably the key point is who loads your class.

• if you can arrange to have ABCL load it and pass it to Hadoop, you

should be good with the current design.

• if Hadoop wants to load it by itself from a jar, then yes, you have

to somehow dump the method implementations somewhere and have the class auto-load them from there when it is initialized.

For the second case, I propose an intermediate design that is, imho, simpler than dumping arbitrary forms: special-case the situation when a method implementation is a symbol. In that case, directly generate an instruction in the class file to find that symbol in its package and assign it to the appropriate field. If the implementation of a method is not a symbol nothing changes. That way, you will still need to init ABCL before loading your class, but, when all your method implementations are symbols, the class will be able to initialize itself without external support. If some methods are proper functions, instead, the class won't be self-contained, but will need some Lisp code to be executed to be properly initialized.

Does it make sense to you? Do you find it acceptable? If yes, I can give a shot at implementing it, if you want.

Peace, Alessio

-- Some gratuitous spam:

http://ripple-project.org Ripple, social credit system http://villages.cc Villages.cc, Ripple-powered community economy http://common-lisp.net/project/armedbear ABCL, Common Lisp on the JVM http://code.google.com/p/tapulli my current open source projects http://www.manydesigns.com/ ManyDesigns Portofino, open source model-driven Java web application framework