- armedbear-devel - mailman3.common-lisp.net

[armedbear-devel] Less-reflection branch: profiling and comparison with trunk
by Alessio Stalla 18 May '10

18 May '10

Hello, as I have already informally discussed with Erik and David Miles, I have been working on a branch which changes how functions are compiled in FASLs (and by consequence how they are loaded from FASLs). The purpose was to reduce the use of reflection by using a single classloader per FASL (as opposed to one per function) plus a master "loader" function that will initialize all other function classes using the new operator, rather than passing through reflection. Now such loading is done eagerly, as soon as the FASL is loaded; but with a bit more work, I should be able to restore autoloading semantics, so that a given function object is only instantiated the first time it is called (but the class representing it will always be loaded eagerly). I suspect it won't change much, though. The overall performance results (reported below) are not very encouraging. However, I do think that the branch has a better design than our current approach, and goes towards the objective of having a per-fasl object holding various things (e.g. a FASL constant pool). In the long run, not having functions rely on loadCompiledFunction in their static initialization code is a nice step towards serializable closures, if we'll ever want to have them. The results on load times and overall runtime speed have been insignificant. Apparently by large the most important factor in our load time is I/O and the branch doesn't change anything about that. It just saves ~1% of the loading time spent on reflection. The results on memory consumption are mixed: they do show a great decrease in the number of classloaders, as expected, but the numbers are not tremendously life-changing: 56 objects (live and allocated, for a total of 4032B, 0.1% of total live objects) vs 789 live objects of 1014 allocated, 44.184B, 1.1% of total live objects. There are also less reflection-related objects, but more Lisp objects, presumably due to eager loading. The numbers for those from a quick glance more or less balance out with each other. Doing these profiles showed that a lot of our memory is actually made of the stored byte[] containing class data for compiled functions. Those amount to ~4M on trunk and almost 8M on my branch (probably due the eager loading of compiled functions). Given that I now load all functions with a classloader that knows where functions have been loaded from, I think I should not store the byte[] in the function at all, but rely on the fasl classloader to re-extract them from the FASL on demand. However, I haven't still implemented this. For those who are interested, I'm attaching the memory profiles done with NetBeans. I left out the CPU profiles because they're fatter (~6M) and I'm on a slow connection now; but I can send them separately. Cheers, Alessio

2 3

[armedbear-devel] ABCL 0.20 stabilization status
by Erik Huelsmann 16 May '10

16 May '10

Last week we branched for 0.20. After branching, a number of issues were found. Some with respect to our relatively new pathname code, others with respect to the reader reorganization we've had. All of the issues *I* know of should be fixed. I've run the ansi tests on the branch (on Windows) and the Maxima test suite too. Both succeed without any surprises right now. So, from where I stand it looks like we'll be able to release with our current status. Please test the branch with your own normal usage patterns to make sure the release will work for you too. If things stay stable during the next week and no further regressions will be reported, I'll create a release next weekend. Bye, Erik.

1 0

[armedbear-devel] Objects sharing structure at compile time, get "destructured" in our FASL format
by Erik Huelsmann 16 May '10

16 May '10

The other day, Alessio and I were talking about how unfortunate it is that we use the reader to create relatively simple structures, such as lists in our Java classes. To illustrate: (defun foo () (let ((x '(1 2 3))) x)) In the above definition, if being file-compiled, the litteral list (1 2 3) gets serialized into a string which is - at load time - converted back to an object like this: LispObject theObject = Lisp.readObjectFromString("(1 2 3)"); In our minds, it would be faster to generate: LispObject theObject = new Cons(new Fixnum(1), new Cons(new Fixnum(2), new Cons(new Fixnum(3))))); Because that eliminates the need to set up a stream, reading from a string, running all the characters through the reader, including running of dispatch functions, etc. While that would be a nice optimization, I've run into a correctness issue investigating our current serialization. Now consider the following macro: (defmacro foo (x) `(defun bar () (let ((a ,x) (b ,x)) (eq a b)))) (foo '(1 2 3)) This macro can be used to generate an example of 2 objects sharing structure to the extreme: they are the same object. The same object is supposed to be assigned to A and B. In the interpreter, the result of a function call to BAR returns T. However, after filecompiling the above forms and loading the resulting fasl, the outcome of BAR is NIL. Suddenly, the forms are not the same anymore. The answer to that is too simple: the object assigned to A and B is being read for A and B separately with a readObjectFromString call, like this: LispObject theObjectA = Lisp.readObjectFromString("(1 2 3)"); LispObject theObjectB = Lisp.readObjectFromString("(1 2 3)"); The above is clearly wrong. The solution to this case is very simple: we should detect "duplicates" being serialized by the DECLARE-OBJECT-AS-STRING function. Although the above solution would solve the simple case of object equality, there are other examples which it won't solve, think about the following macro: (defmacro foo (x) `(defun bar () (let ((a ,x) (b `(0 ,,x))) (eq a (cdr b))))) The macro above will generate 2 different values to be serialized by the DECLARE-OBJECT-AS-STRING function, failing to detect the fact that the two share structure. Possibly, we could come up with a way to detect objects with shared structure and generate the right class-setup code to make the above example work. But that's not all there is to this problem: the objects sharing structure can be distributed over multiple Java classes; consider the fact that our local functions are Java classes of their own, each with their own constants. One of the ideas that Alessio and I had was that it would be nice to have a single pool of constants for an entire FASL, for the separate function classes to refer to when required. Looking at the above issue, I'd say we really do need such a facility. The only per-fasl facility for objects being equal that we currently have, is the vector with anonymous symbols. Any ideas on how to tackle this issue? I suppose other lisps don't have this issue because they don't use separate classes for their functions and have more influence over the storage format of their data in the fasl... Bye, Erik.

3 4

[armedbear-devel] more directory and jars
by Alan Ruttenberg 15 May '10

15 May '10

CL-USER> (DIRECTORY "jar:file:/Users/alanr/Desktop/jar2go.jar!/abcl-module-*.lisp") Calling (DIRECTORY "jar:file:/Users/alanr/Desktop/jar2go.jar!/abcl-module-*.lisp") DIRECTORY returned #<FILE-ERROR {E2C42F}> #<FILE-ERROR {E2C42F}> CL-USER> (describe *) #<FILE-ERROR {E2C42F}> is an instance of #<STANDARD-CLASS FILE-ERROR {8917A2}>. The following slots have :INSTANCE allocation: FORMAT-CONTROL "Not a wild pathname." FORMAT-ARGUMENTS NIL PATHNAME #P"jar:file:/Users/alanr/Desktop/jar2go.jar!/abcl-module-*.lisp" ; No value Below is the function I use for directory in jars. Please consider including it until there is a correct implementation in java. (defun directory-in-jar (pathname) (let* ((device (pathname-device pathname)) (jarfile (namestring (car device))) (rest-pathname (namestring (make-pathname :directory `(:absolute ,@(cdr (pathname-directory pathname))) :name (pathname-name pathname) :type (pathname-type pathname))))) (if (or (position #\* (namestring rest-pathname)) (wild-pathname-p rest-pathname)) (let ((jar (jnew "java.util.zip.ZipFile" jarfile))) (let ((els (jcall "entries" jar))) (loop while (jcall "hasMoreElements" els) for name = (jcall "getName" (jcall "nextElement" els)) when (pathname-match-p (concatenate 'string "/" name) rest-pathname) collect (make-pathname :device (pathname-device pathname) :name (pathname-name name) :type (pathname-type name) :directory `(:relative ,@(cdr (pathname-directory name))))))) (let ((truename (probe-file pathname))) (if truename (list truename) nil)))))

2 2

[armedbear-devel] system.lisp and handler-case
by Alan Ruttenberg 14 May '10

14 May '10

I'm trying to use system.lisp as the hook into writing code that reads command line parameters and does stuff - to use abcl for scripting. The problem is that debugging it is ... uncomfortable... , because (handler-case (require 'system) (t ())) Means that any error just stops what's happening. How can I establish that my code running as called from system.lisp runs in a normal debugging environment? Thanks, Alan

2 7

[armedbear-devel] over-agressive handler for module-provide-system
by Alan Ruttenberg 12 May '10

12 May '10

Current: (defun module-provide-system (module) (let ((*readtable* (copy-readtable nil))) (handler-case (load-system-file (string-downcase (string module))) (t (e) (unless (and (typep e 'error) (search "Failed to find loadable system file" (format nil "~A" e))) (format *error-output* "Failed to require ~A because '~A'~%" module e)) nil)))) First, t is too strong - warns will kill it. Second, why not let the debugger handle it if there is an error? -Alan

2 1

[armedbear-devel] lambda-list error at compilation
by Alan Ruttenberg 11 May '10

11 May '10

The following works as expected when evaluated, but throws an error when compiling. (defun hashmap-to-hashtable (hashmap &key (keyfun #'identity) (valfun #'identity) (invert? nil) &allow-other-keys &rest rest) (let ((keyset (#"keySet" hashmap)) (table (apply 'make-hash-table (remf :invert? (remf :valfun (remf :keyfun rest)))))) (with-constant-signature ((iterator "iterator" t) (hasnext "hasNext") (next "next")) (loop with iterator = (iterator keyset) while (hasNext iterator) for item = (next iterator) do (if invert? (setf (gethash (funcall valfun (#"get" hashmap item)) table) (funcall keyfun item)) (setf (gethash (funcall keyfun item) table) (funcall valfun (#"get" hashmap item))))) table))) -> Error &REST/&BODY must be followed by a variable. [Condition of type PROGRAM-ERROR] Restarts: 0: [ABORT] Abort compilation. 1: [ABORT] Return to SLIME's top level. 2: [ABORT] Abort thread. Backtrace: 0: (SYSTEM:MAKE-CLOSURE (LAMBDA (#:G29656 &KEY (# NIL #:G29659) (# NIL #:G29661) (# NIL) &ALLOW-OTHER-KEYS ...) NIL) NIL) 1: (JVM::P1-COMPILAND #S(JVM::COMPILAND :NAME HASHMAP-TO-HASHTABLE :LAMBDA-EXPRESSION (LAMBDA (#1=#:G29656 &KEY ((:KEYFUN #:G29658) NIL #:G29659) ((:VALFUN #:G29660) NIL #:G29661) ((:INVERT? #2=#:G29662).. 2: (JVM::P1-FUNCTION #'(SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE (HASHMAP &KEY (KEYFUN #'IDENTITY) (VALFUN #'IDENTITY) (INVERT? NIL) &ALLOW-OTHER-KEYS ...) (BLOCK HASHMAP-TO-HASHTABLE (LET # #)))) 3: (JVM::P1 #'(SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE (HASHMAP &KEY (KEYFUN #'IDENTITY) (VALFUN #'IDENTITY) (INVERT? NIL) &ALLOW-OTHER-KEYS ...) (BLOCK HASHMAP-TO-HASHTABLE (LET # #)))) Locals: "??" = "??" 4: (JVM::P1 (SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE (HASHMAP &KEY (KEYFUN #'IDENTITY) (VALFUN #'IDENTITY) (INVERT? NIL) &ALLOW-OTHER-KEYS ...) (BLOCK HASHMAP-TO-HASHTABLE (LET # #)))) 5: (JVM::P1-LET*-VARS #S(JVM::LET/LET*-NODE :FORM NIL :COMPILAND #S(JVM::COMPILAND :NAME NIL :LAMBDA-EXPRESSION (LAMBDA NIL (SYSTEM:%DEFUN 'HASHMAP-TO-HASHTABLE (SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE .. 6: (JVM::P1-LET/LET* (LET* ((#:G29655 #)) (SYSTEM:%DEFUN 'HASHMAP-TO-HASHTABLE #:G29655))) 7: (JVM::P1 (LET* ((#:G29655 #)) (SYSTEM:%DEFUN 'HASHMAP-TO-HASHTABLE #:G29655))) 8: (JVM::P1-FUNCTION-CALL (SYSTEM:%DEFUN 'HASHMAP-TO-HASHTABLE (SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE (HASHMAP &KEY # # # &ALLOW-OTHER-KEYS ...) (BLOCK HASHMAP-TO-HASHTABLE #)))) 9: (JVM::P1-BODY ((SYSTEM:%DEFUN 'HASHMAP-TO-HASHTABLE (SYSTEM:NAMED-LAMBDA HASHMAP-TO-HASHTABLE # #)))) 10: (#<FUNCTION {79F502}>) 11: (#<FUNCTION {C49395}>) 12: (#<FUNCTION {E4ED69}> NIL (LAMBDA NIL (DEFUN HASHMAP-TO-HASHTABLE (HASHMAP &KEY # # # &ALLOW-OTHER-KEYS ...) (LET # #)))) 13: (#<FUNCTION (LAMBDA ()) {B5506C}>) 14: (JRUN-EXCEPTION-PROTECTED #<FUNCTION (LAMBDA ()) {B5506C}>) 15: ((FLET SWANK-BACKEND:SWANK-COMPILE-STRING) "(defun hashmap-to-hashtable (hashmap &key (keyfun #'identity) (valfun #'identity) (invert? nil) &allow-other-keys &rest rest) (let ((keyset (#\"..

4 5

[armedbear-devel] asdf2 meets jar files
by Alan Ruttenberg 11 May '10

11 May '10

(asdf::initialize-source-registry '(:source-registry :ignore-inherited-configuration (:tree "jar:file:/Users/alanr/Desktop/jar2go.jar!/"))) -> WARNING: Error while scanning system definitions under directory #P"jar:file:/Users/alanr/Desktop/jar2go.jar!/": Unsupported directory component :WILD-INFERIORS.

2 1

[armedbear-devel] expecting to find integer on stack
by Alan Ruttenberg 10 May '10

10 May '10

Fresh svn update. Compiled with Java 1.5 on OSX File attached. laptop:~/repos/abcl % ./abcl --noinit Armed Bear Common Lisp 0.20.0-dev Java 1.5.0_19 Apple Inc. Java HotSpot(TM) Client VM Low-level initialization completed in 0.363 seconds. Startup completed in 0.914 seconds. Type ":help" for a list of available commands. CL-USER(1): (compile-file "/Users/alanr/repos/lsw/trunk/owlapi3/patmatch.lisp") ; Compiling /Users/alanr/repos/lsw/trunk/owlapi3/patmatch.lisp ... ; (DEFUN VARIABLE-P ...) ; (DEFUN PAT-MATCH ...) ; in (DEFUN PAT-MATCH ...) ; Caught STYLE-WARNING: ; Undefined variable NO-BINDINGS assumed special ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; Caught STYLE-WARNING: ; Undefined variable *OWL2-VOCABULARY-FORMS* assumed special ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (SETF (GET # ...) ...) ; (DEFUN SEGMENT-PATTERN-P ...) ; (DEFUN SINGLE-PATTERN-P ...) ; (DEFUN SEGMENT-MATCHER ...) ; (DEFUN SINGLE-MATCHER ...) ; (DEFUN SEGMENT-MATCH-FN ...) ; (DEFUN SINGLE-MATCH-FN ...) ; (DEFUN MATCH-IS ...) ; in (DEFUN MATCH-IS ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN MATCH-AND ...) ; in (DEFUN MATCH-AND ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN MATCH-OR ...) ; in (DEFUN MATCH-OR ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN MATCH-NOT ...) ; in (DEFUN MATCH-NOT ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN SEGMENT-MATCH ...) ; in (DEFUN SEGMENT-MATCH ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN FIRST-MATCH-POS ...) ; (DEFUN SEGMENT-MATCH+ ...) ; (DEFUN SEGMENT-MATCH? ...) ; (DEFUN SEGMENT-ATOM-MATCH? ...) ; (DEFUN MATCH-OPTIONAL-ATOM ...) ; in (DEFUN MATCH-OPTIONAL-ATOM ...) ; Caught STYLE-WARNING: ; Undefined variable FAIL assumed special ; (DEFUN MATCH-IF ...) ; Compilation unit finished ; Caught 9 STYLE-WARNING conditions ; The following functions were used but not defined: ; MATCH-VARIABLE java.lang.VerifyError: (class: org/armedbear/lisp/patmatch_19, method: execute signature: (Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject;) Expecting to find integer on stack at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2357) at java.lang.Class.getConstructor0(Class.java:2671) at java.lang.Class.newInstance0(Class.java:321) at java.lang.Class.newInstance(Class.java:303) at org.armedbear.lisp.Lisp.makeCompiledFunctionFromClass(Lisp.java:1286) at org.armedbear.lisp.Lisp.loadClassBytes(Lisp.java:1347) at org.armedbear.lisp.Lisp.loadClassBytes(Lisp.java:1340) at org.armedbear.lisp.AutoloadedFunctionProxy.loadPreloadedFunction(AutoloadedFunctionProxy.java:224) at org.armedbear.lisp.CompiledClosure$1.execute(CompiledClosure.java:232) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.compile_file_7.execute(compile-file.lisp:71) at org.armedbear.lisp.LispThread.execute(LispThread.java:552) at org.armedbear.lisp.Java$pf_jrun_exception_protection.execute(Java.java:1128) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.compile_file_5.execute(compile-file.lisp:71) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.compile_file_10.execute(compile-file.lisp:103) at org.armedbear.lisp.Symbol.execute(Symbol.java:799) at org.armedbear.lisp.LispThread.execute(LispThread.java:602) at org.armedbear.lisp.compile_file_40.execute(compile-file.lisp:486) at org.armedbear.lisp.LispThread.execute(LispThread.java:552) at org.armedbear.lisp.Java$pf_jrun_exception_protection.execute(Java.java:1128) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.compile_file_37.execute(compile-file.lisp:486) at org.armedbear.lisp.compiler_pass2_579.execute(compiler-pass2.lisp:8685) at org.armedbear.lisp.LispThread.execute(LispThread.java:552) at org.armedbear.lisp.Java$pf_jrun_exception_protection.execute(Java.java:1128) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.compiler_pass2_575.execute(compiler-pass2.lisp:8685) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:101) at org.armedbear.lisp.AutoloadedFunctionProxy.execute(AutoloadedFunctionProxy.java:145) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.compile_file_35.execute(compile-file.lisp:486) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:101) at org.armedbear.lisp.AutoloadedFunctionProxy.execute(AutoloadedFunctionProxy.java:145) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:497) at org.armedbear.lisp.Lisp.eval(Lisp.java:462) at org.armedbear.lisp.Lisp.eval(Lisp.java:460) at org.armedbear.lisp.Primitives$pf__eval.execute(Primitives.java:328) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.Lisp.evalCall(Lisp.java:497) at org.armedbear.lisp.Lisp.eval(Lisp.java:462) at org.armedbear.lisp.Lisp.progn(Lisp.java:631) at org.armedbear.lisp.Primitives$sf_block.execute(Primitives.java:3720) at org.armedbear.lisp.Lisp.eval(Lisp.java:452) at org.armedbear.lisp.Lisp.progn(Lisp.java:631) at org.armedbear.lisp.Closure.bindParametersAndExecute(Closure.java:446) at org.armedbear.lisp.Closure.execute(Closure.java:479) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.Lisp$1.execute(Lisp.java:278) at org.armedbear.lisp.Symbol.execute(Symbol.java:776) at org.armedbear.lisp.LispThread.execute(LispThread.java:568) at org.armedbear.lisp.top_level_50.execute(top-level.lisp:407) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:92) at org.armedbear.lisp.AutoloadedFunctionProxy.execute(AutoloadedFunctionProxy.java:139) at org.armedbear.lisp.Symbol.execute(Symbol.java:766) at org.armedbear.lisp.LispThread.execute(LispThread.java:552) at org.armedbear.lisp.top_level_51.execute(top-level.lisp:415) at org.armedbear.lisp.AutoloadedFunctionProxy.execute(AutoloadedFunctionProxy.java:139) at org.armedbear.lisp.LispThread.execute(LispThread.java:552) at org.armedbear.lisp.Interpreter.run(Interpreter.java:319) at org.armedbear.lisp.Main$1.run(Main.java:50) at java.lang.Thread.run(Thread.java:613) #<THREAD "interpreter" {1A3492}>: Debugger invoked on condition of type ERROR Caught java.lang.VerifyError: (class: org/armedbear/lisp/patmatch_19, method: execute signature: (Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject;) Expecting to find integer on stack. Restarts: 0: TOP-LEVEL Return to top level. [1] CL-USER(2):

4 10

[armedbear-devel] add fasl version to default directory for asdf cache
by Alan Ruttenberg 10 May '10

10 May '10

Otherwise one gets the like of: Error loading /Users/alanr/repos/lsw2/trunk/scripts/system-registry.lisp at line 42 (offset 1356) #<THREAD "interpreter" {7A8516}>: Debugger invoked on condition of type SIMPLE-ERROR Incorrect fasl version: jar:file:/Users/alanr/.cache/common-lisp/armedbear-0.20.0-dev-darwin-unknown/Users/alanr/repos/lsw2/trunk/jss/invoke.abcl!/invoke._ Restarts: -Alan

2 4