September 2009 - armedbear-devel - mailman3.common-lisp.net

[armedbear-devel] Ugh! java.lang.OutOfMemoryError: PermGen space
by Alessio Stalla 26 Sep '09

26 Sep '09

(dotimes (i 100000) (compile 'f (lambda () 42))) abcl dies with java.lang.OutOfMemoryError: PermGen space if I use nil instead of f it doesn't die. Also with nil is noticeably faster. Adding (when (= 999 (mod i 1000)) (room)) in the loop shows that gc is performed and memory is reclaimed, but apparently there's a leak somewhere. I'll try to find out more. Later, Ale

2 2

[armedbear-devel] In-memory compilation!
by Alessio Stalla 26 Sep '09

26 Sep '09

Hello list, I managed to get temp-file-less compilation with COMPILE working. And the hilarious things is: the classloader experiment was not necessary for this, as the compiler already behaved differently according the the value of *file-compilation*, and the use of loadCompiledFunction that troubled me was only done for the file compiler! So it's sufficient to move file creation from inner functions to outer ones and to pass an additional stream parameter to inner functions. Things can be made clearer - now the structure used to represent compilation results, class-file, is required to always have a pathname set, even when no file is created, for example. But I believe we're on the right track. I cannot provide a patch now because I have still have to clean up the classloader changes. Btw I believe that the use of classloaders is a good thing even if it's not used for memory compilation (for one thing, it would play nicer with serialization of lisp functions, if we ever want that). Cheers, Ale

1 1

[armedbear-devel] Some thoughts on classloaders, inner classes and in-memory compilation
by Alessio Stalla 24 Sep '09

24 Sep '09

Hello folks, I'd like to share some thoughts on the "usual" problem of in-memory compilation in abcl. I haven't written any code to support my thoughts - I just hope to make things clearer for me and other people on the list, and I'm asking you to share your ideas and comment mine. Warning - this is going to be a long post. Let's start by describing how abcl works now. Every piece of code in the JVM needs to be contained in a method of some class, so when abcl compiles a function it produces - surprise - a class, or better, a stream of bytes that the JVM knows how to interpret to create a class. Classes produced by the compiler extend a common abcl class providing the methods that will be used to actually invoke the function; the compiler, among other things, will override some of those methods. When using the runtime compiler, the class is immediately loaded; when using the file compiler, it is stored on the filesystem for later use. So far, nothing requires temporary files to work. However, a Lisp function can contain nested functions introduced with FLET or LABELS, and those have to be compiled to classes as well. Also, they need to be loaded contextually with the main function. How does abcl solve this? By adding instructions in the main class to load the local functions when the class itself is loaded, relatively to where it is loaded from. And here lies the problem. That location is always assumed to be a file in some filesystem subtree. So even when using the runtime compiler all the classes must be, at least temporarily, stored in files for the load machinery to work. Changing this is hard; classes are resolved from strings, how do know when a given string represents a file and when it represents some object in memory instead? There are workarounds, but I think it's the approach itself that's brittle. So let's step back a bit and take a look at how the JVM and Java the language work with respect to loading classes. The JVM uses dedicated objects called classloaders. They are responsible of translating from a class' symbolic name (a string) to a class metaobject, much like the CLOS find-class function does. Classloaders are organized hierarchically: every classloader has a parent which is first consulted to see if it already has the class (there is of course a built-in bootstrap classloader to break the circularity); if it has, it returns it, if it hasn't, it is loaded in a manner dependent to the particular classloader (e.g. from a file, from http, from memory, ...). The process of loading a class from a byte array is native in the JVM, so classloaders only get to decide where the byte array comes from and what it contains. Now to the more interesting things: 1. A class never exists in isolation; to do its work it will need to refer to other classes (at a bare minimum, its superclass and any interface it implements). The JVM - automatically! - uses the same classloader to load a class and, at linking time, all of its dependencies. 2. If I had to manually redo in Java what the abcl compiler does with functions, I'd use static inner classes to represent local functions. Inner classes are classes which are textually defined inside another class and share some data with it. Inner classes do not exist at the bytecode level, only at the Java language level: the compiler (javac) translates them to regular classes, with their name mangled. For example, a class Inner defined inside a class Outer will be referred to Outer.Inner in Java, but compiled to Outer$Inner.class by javac. 3. Inner classes then are treated exactly like the others: referred using strings inside code, resolved by a classloader (generally - always? - by the classloader of the containing class). Return to abcl. As you may have guessed, I propose that we no longer make classes autoload their dependencies, but properly use classloaders instead, in a fashion similar to how inner classes work. We will have an InMemoryClassLoader which will load classes from a Map<String, byte[]>, and a slightly extended URLClassLoader to load classes from the filesystem. Both, in addition to load classes, will be used by the compiler to write classes as well, so it will continue to use the class-file abstraction, changing only the code that actually writes the bytes. Every time the compiler would have written a call to loadCompiledFunction(classname) it will now use something like functionFoo.class.getClassLoader().loadClass(classname) where functionFoo is the compiler-generated name of the class representing the compiled Lisp function. Everything else should stay the same. Does this sound convincing? I admit I have left many things to elaborate on, and I haven't rehashed the code in the compiler, going mainly from memory instead. But I believe this approach has not been proposed before and looks doable. The next few days I'll try writing some sketch code to back up my ideas, if no-one finds any serious problem with them. Peace, Alessio

7 23

[armedbear-devel] [BUG] SET-SYNTAX-FROM-CHAR should clear dispatch tables
by Tobias C. Rittweiler 22 Sep '09

22 Sep '09

(let ((rt (copy-readtable nil))) (set-syntax-from-char #\# #\A rt (copy-readtable nil)) (get-dispatch-macro-character #\# #\x rt)) should signal an error as #\# is no longer a dispatching macro character. -T.

1 0

[armedbear-devel] packaging external libs with abcl
by Matthew D. Swank 22 Sep '09

22 Sep '09

I've been messing with running abcl on the google appengine, and I was wondering what was the best way to package libraries, especially the results of loading an asdf system and its dependencies? Thanks, Matt -- "You do not really understand something unless you can explain it to your grandmother." -- Albert Einstein.

3 8

[armedbear-devel] Bug: Attempt to GO to a tag whose extent has ended quits ABCL
by Brian Mastenbrook 21 Sep '09

21 Sep '09

When attempting to GO to a tag whose dynamic extent has ended, ABCL quits and prints the name of the exception used for internally handling GO (org.armedbear.lisp.Go). I see the same behavior in compiled or interpreted code. CL-USER(1): (lisp-implementation-version) "0.16.0" CL-USER(2): (let ((f nil)) (tagbody (setf f (lambda () (go foo))) foo) (funcall f)) org.armedbear.lisp.Go abcl-src-0.16.0 chandler$ -- Brian Mastenbrook brian(a)mastenbrook.net http://brian.mastenbrook.net/

6 10

[armedbear-devel] [BUG] make-thread and :NAME nil
by Tobias C. Rittweiler 21 Sep '09

21 Sep '09

Not providing a :NAME parameter to MAKE-THREAD will currently fail: CL-USER(1): (threads:make-thread (constantly t)) Debugger invoked on condition of type TYPE-ERROR: The value NIL is not of type STRING. Funnily enough, explicitly providing NIL kind of works: CL-USER(2): (threads:make-thread (constantly t) :name nil) #<THREAD "NIL" {1EEB372}> But, as you can see, the passed NIL is taken as a string designator, but probably shouldn't. -T.

2 1

[armedbear-devel] Your NAMED-READTABLES test
by Erik Huelsmann 20 Sep '09

20 Sep '09

Hi Tobias, The named-readtables test you sent me, does show an obvious bug in ABCL, but not the one you're trying to point me at: when loading your files on Windows, I'm getting an Index out of bounds error: the dispatch tables think our characters range from 0 - 255 in value whereas in practice that number is much higher. I'm working on a fix right now (setting CHAR_MAX to a higher value) but I don't feel too comfortable with it: it'll make us consume much more memory on a number of things. However, what's a K of memory worth these days... ? Bye, Erik.

2 2

[armedbear-devel] JSR-223: Returning immutable lists in ScriptEngineFactory
by Clemens Blamauer 20 Sep '09

20 Sep '09

Hi everyone, I was looking through the JSR-223 code in org.armedbear.lisp.scripting as I noticed that AbclScriptEngineFactory is not fully compliant to JSR-223 [1] (p. 101-102 in the Final Draft Specification, v1.0). For the methods: - java.util.List<java.lang.String> getExtensions() - java.util.List<java.lang.String> getMimeTypes() - java.util.List<java.lang.String> getNames() _immutable_ Lists should be returned. Please find a remedying patch below. Cheerz, Clemens [1] http://jcp.org/en/jsr/detail?id=223 ---- Index: AbclScriptEngineFactory.java =================================================================== --- AbclScriptEngineFactory.java (revision 12157) +++ AbclScriptEngineFactory.java (working copy) @@ -21,6 +21,7 @@ package org.armedbear.lisp.scripting; import java.util.ArrayList; +import java.util.Collections; import java.util.List; import javax.script.ScriptEngine; @@ -44,7 +45,7 @@ public List<String> getExtensions() { List<String> extensions = new ArrayList<String>(1); extensions.add("lisp"); - return extensions; + return Collections.unmodifiableList(extensions); } @Override @@ -89,7 +90,7 @@ @Override public List<String> getMimeTypes() { - return new ArrayList<String>(); + return Collections.unmodifiableList(new ArrayList<String>()); } @Override @@ -99,7 +100,7 @@ names.add("cl"); names.add("Lisp"); names.add("Common Lisp"); - return names; + return Collections.unmodifiableList(names); } @Override -- # Trust in code: # lynx http://lxr.linux.no/linux+v2.6.23.14/arch/i386/kernel/cpu/common.c#L677

2 1

[armedbear-devel] [PATCH] incorporating java stack frames into backtrace
by Tobias C. Rittweiler 20 Sep '09

20 Sep '09

When evaluating (ext:make-thread #'(lambda ())) you get an error "The value NIL is not of type STRING." -- this is a bug in ABCL itself, by the way, but that's not the matter of this mail. The concern of this mail is the backtrace that'd you get: Backtrace: 0: (INVOKE-DEBUGGER #<TYPE-ERROR {B1A201}>) 1: (MAKE-THREAD #<FUNCTION (LAMBDA ()) {15343C2}>) 2: (SYSTEM::%EVAL (MAKE-THREAD #'(LAMBDA NIL))) 3: (EVAL (MAKE-THREAD #'(LAMBDA NIL))) ... The only thing the backtrace reveales is that the error was signalled somewhere in MAKE-THREAD. With the patch of this mail, the backtrace will also contain the relevant Java stack frames for errors that were signalled from within ABCL's Java runtime: Backtrace: 0: (INVOKE-DEBUGGER #<TYPE-ERROR {B1A201}>) 1: org.armedbear.lisp.Lisp.error(Lisp.java:375) 2: org.armedbear.lisp.Lisp.type_error(Lisp.java:390) 3: org.armedbear.lisp.LispObject.getStringValue(LispObject.java:1061) 4: org.armedbear.lisp.LispThread.<init>(LispThread.java:111) 5: org.armedbear.lisp.LispThread.<init>(LispThread.java:39) 6: org.armedbear.lisp.LispThread$3.execute(LispThread.java:909) 7: org.armedbear.lisp.Primitive.execute(Primitive.java:108) 8: (MAKE-THREAD #<FUNCTION (LAMBDA ()) {15343C2}>) 9: (SYSTEM::%EVAL (MAKE-THREAD #'(LAMBDA NIL))) 10: (EVAL (MAKE-THREAD #'(LAMBDA NIL))) Another example: (LENGTH 12) Backtrace: 0: (INVOKE-DEBUGGER #<TYPE-ERROR {814AF5}>) 1: org.armedbear.lisp.Lisp.error(Lisp.java:375) 2: org.armedbear.lisp.Lisp.type_error(Lisp.java:390) 3: org.armedbear.lisp.LispObject.length(LispObject.java:403) 4: org.armedbear.lisp.LispObject.LENGTH(LispObject.java:410) 5: org.armedbear.lisp.Primitives$24.execute(Primitives.java:427) 6: (LENGTH 12) 7: (SYSTEM::%EVAL (LENGTH 12)) 8: (EVAL (LENGTH 12)) -T. -- Diese Nachricht wurde auf Viren und andere gefaerliche Inhalte untersucht und ist - aktuelle Virenscanner vorausgesetzt - sauber. Freebits E-Mail Virus Scanner

4 18