armedbear-devel

armedbear-devel@common-lisp.net

1 participants
1200 discussions

[armedbear-devel] Autoloaders: status report and a question
by Erik Huelsmann 18 Aug '12

18 Aug '12

Hi all, Sorry for the bumpy ride while I reorganized the auto loader system. Thanks for taking the time to respond to my earlier questions on auto loaders. This is the time to explain what I did with your answers (and what I did in general). To reiterate what this is all about: the autoloader is an important part of the start up efficiency and internal file dependency resolution of ABCL: it establishes function and macro bindings on symbols without loading the actual code to bind to. Instead, loading the FASL is being delayed until some part of the system (or a user) needs it. There are two variants of autoloaders: Java class autoloaders and lisp (fasl) autoloaders. Their purpose is the same and they operate roughly the same way, but one operates by forcing a class to be loaded while the other causes a FASL to be loaded. Most parts of the system don't know about autoloaders and simply evaluate code. The autoloaders take special action when evaluated: their purpose is not to execute the intended action, but instead, they load the fasl in which the actual code is located. Then, when the actual code is loaded, the function (or macro) call is forwarded to the loaded code. The FASL loader process replaces the content of the function symbol slot with the true function so the autoloader gets garbage collected after replacement. This entire system works pretty well, that is: *if* you know which symbols you have to install an autoloader on. Before my changes, we didn't know which symbols we had to install autoloaders on. Instead, we maintained a list of symbols by hand through trial and error. When a change caused total havoc, you'd look for the cause and - hopefully - debugged the problem to a missing autoloader. That's when you would add an autoloader to autoloads.lisp. That works as long as our only purpose is to build ABCL. However, we actually intend for ABCL to be used by our users. Users may not be aware of what autoloaders do for them and choose to start calling some random function in the AMOP definition. Since we hand-maintain a list of functions to be decorated with autoloaders, that probably doesn't work. So, in order to provide more stability for our users we need to decorate all functions with autoloaders. What's more: having all functions decorated with autoloaders -if done well- should also help ABCL development quite a bit: the chances of a change causing havoc because a non-decorated function is being used have been reduced to next-to-none. What then did I do? The change involves a few steps: 1. Extraction of symbols to be decorated ============================= The file compiler now generates (when compiling our own system, that is) two extra output files: basename.funcs and basename.macs. These files contain a single s-exp with all the symbols which have been DEFUNed or DEFMACROed respectively. With this information, we can generate decoration information. Analysis of the extracted symbols leads to the conclusion we can't assign an autoloader to each and every extracted symbol for 2 reasons: a. There is a built-in function associated with the symbol to help bootstrapping b. There are multiple DEFUNs or DEFGENERIC/DEFMETHODs 2. Determination of what to decorate symbols with ==================================== Autoloaders - at this time - can only be used to load a single FASL. So, when a symbol is in multiple files, we need to choose which file it load the FASL for (and hence not the others). Most symbols which occur in multiple files are DEFMETHODs which is perfectly fine. Most symbols with double locations stem from the EXTENSIBLE-SEQUENCES implementation. We want to exclude those FASLs, which reduces the number of multi-homed symbols to only a few. All single-homed symbols excluding those which have bootstrap functions attached can be decorated with autoloaders. 3. Generate code to install the autoloaders =============================== We just used to have autoloads.lisp (and Autoload.java). But that file will contain only the manual bits which can't be auto-generated. So there's now a second file autoload-gen.lisp. This file is being generated during the build and populated with all symbols which need an autoloader. Today's autoload.lisp has been stripped to the bare minimum of AUTOLOAD and AUTOLOAD-MACRO forms. With the stripped autoload.lisp and the fact that the autoloaders are bootstrapping helpers, I had to take additional measures to help us bootstrap again. The result is autoloads-gen.lisp in the source directory. It is loaded at build time to help bootstrapping. It's an unmodified version I took from the build directory: build/classes/org/armedbear/lisp/autoloads-gen.lisp - the one which is generated on each build. 4. Exporting the right symbols ====================== Not a problem related to autoloading per se, but definitely related to the fact that we don't load all fasls at boot time: autoload.lisp - as the canonical location being loaded at startup - had grown to be a listing of EXPORT forms. Having EXPORT forms and actual code widely apart is counter-intuitive: to the occasional developer it's inexplicable how some symbols become exported and some don't. (We had files with additional EXPORT forms.) So, when the above was "solved", I created another facility which collects all the symbols in the second argument of EXPORT throughout our source tree. Those symbols are added to the autoload-gen file and the EXPORT forms executed at start up as well. Even though the developer still doesn't know what's going on, the system should be working transparently. All of the above has happened over the last week(s) - last week, mostly. Then there is the question: As it turns out, the macroexpansion for DEFSTRUCT happens in a way such that the file compiler can't detect the slot accessors as being defined as functions. The advantage is that we're *not* seeing a whole slew of symbols to be attached to autoloaders and hence limiting the number of autoloaders (the contructor function *is* detected). Now: what do others think: should we depend on the constructor being called before any of the accessors to make sure the autoloaders have been run? Or should we attach an autoloader to each accessor function just to be sure? (This requires small changes to the file compiler and defstruct macro expansion.) Bye, Erik.

1 0

[armedbear-devel] SLIME && ABCL autoloaders
by Erik Huelsmann 18 Aug '12

18 Aug '12

Hi Mark, While working on the autoloaders, I found a very interesting thing: SLIME doesn't know about ABCL's autoloaders. To maximize the benefit of SLIME for ABCL (yea, I even started using it to debug ABCL itself, after you pointed out to me that these tools generally help productivity), I think it would be beneficial to teach it about autoloaders. The case I'm talking about is this: When using M-. SLIME looks up the source code location for the source associated with the symbol. However, when you do that to a symbol which hasn't been autoloaded yet, that source code location information isn't available. SLIME could easily resolve that by evaluating the form (SYS::RESOLVE <symbol>) and look for the info after that. SLIME could do the same when looking up function lambda list information to present in the minibuffer for auto-completion. How about that? Bye, Erik.

2 1

[armedbear-devel] SLIME broken on current svn because export statement is in autoloaded file
by Rudolf Schlatte 18 Aug '12

18 Aug '12

Hi, Subject says most of it, I guess. Currently, threads.lisp line 36ff. contains the export form that exports make-thread-lock from the threads package ... which will be evaluated as soon as one of the autoloaded functions is called. This first function call must use notation for unexported symbols to avoid a reader error. Cheers, Rudi

2 1

[armedbear-devel] Forward referenced classes woes
by Stas Boukarev 15 Aug '12

15 Aug '12

Consider the following code: (defclass whitespace-normalizer (sax-proxy)) (defclass sax-proxy (broadcast-handler) ()) (defclass broadcast-handler () ((handlers :initform nil :initarg :handlers :accessor broadcast-handler-handlers))) Doing the followoing results in errors: (setf (broadcast-handler-handlers (make-instance 'whitespace-normalizer)) 10) There is no applicable method for the generic function #<STANDARD-GENERIC-FUNCTION (SETF BROADCAST-HANDLER-HANDLERS) {6621477C}> when called with arguments (10 #<WHITESPACE-NORMALIZER {14BAAEA8}>). (setf (broadcast-handler-handlers (make-instance 'sax-proxy)) 10) The slot HANDLERS is missing from the class #<STANDARD-CLASS SAX-PROXY {1700FFDB}>. The problem is that whitespace-normalizer and sax-proxy are finalized too early. Finalization happens in std-after-initialization-for-classes http://trac.common-lisp.net/armedbear/browser/trunk/abcl/src/org/armedbear/… by calling maybe-finalize-class-subtree, which in its turn finalizes the class and its subclasses in case the class has all its superclasses finalized. When (defclass whitespace-normalizer (sax-proxy)) is executed it creates a forward-referenced-class for sax-proxy, and when sax-proxy is defined it calls a method for ensure-class-using-class http://trac.common-lisp.net/armedbear/browser/trunk/abcl/src/org/armedbear/… for forward-referenced-classes. Before calling reinitialize-instance it does (change-class class metaclass) to change from forward-referenced to normal class. change-class in its turn calls shared-initialize method, which calls the aforementioned std-after-initialization-for-classes, but it doesn't supply any direct-superclass arguments. So, when maybe-finalize-class-subtree is called it thinks that sax-proxy's sole superclass is standard-object, so it goes ahead finalizing sax-proxy and whitespace-normalizer at the wrong time with wrong information. And ensure-class-using-class later call with the right direct-superclass information doesn't matter anymore, since the classes were already badly finalized -- With best regards, Stas.

2 2

[armedbear-devel] [patch] compile slightly broken for compiled functions
by Vladimir Sedach 14 Aug '12

14 Aug '12

Hi, (compile nil (compile nil (lambda ()))) => NIL (compile '+) => + The first result is obviously wrong, and the second one is actually as well (according to the HyperSpec, compile should return the function definition of +). The fix is rather trivial: Index: src/org/armedbear/lisp/compiler-pass2.lisp =================================================================== --- src/org/armedbear/lisp/compiler-pass2.lisp (revision 14055) +++ src/org/armedbear/lisp/compiler-pass2.lisp (working copy) @@ -7483,7 +7483,7 @@ (resolve name) ;; Make sure the symbol has been resolved by the autoloader (setf definition (fdefinition name))) (when (compiled-function-p definition) - (return-from jvm-compile (values name nil nil))) + (return-from jvm-compile (values definition nil nil))) (let ((catch-errors *catch-errors*) (warnings-p nil) (failure-p nil) Index: test/lisp/abcl/compiler-tests.lisp =================================================================== --- test/lisp/abcl/compiler-tests.lisp (revision 14055) +++ test/lisp/abcl/compiler-tests.lisp (working copy) @@ -472,11 +472,10 @@ (not (null result)))) t) +(deftest compile-named + (compile '+) + (symbol-function '+)) - - - - - - - \ No newline at end of file +(deftest compile-compile + (compiled-function-p (compile nil (compile nil (lambda ())))) + t) Happy hacking, Vladimir

2 3

[armedbear-devel] Wrongly initialized generic function method combination
by Stas Boukarev 13 Aug '12

13 Aug '12

When compile a file with (defgeneric test (a b) (:method-combination progn :most-specific-last)) (defmethod test progn (a b) (list a b)) TEST receives a wrong method-combination (mop:generic-function-method-combination #'test) => #<METHOD-COMBINATION STANDARD {6E99175D}> When defgeneric is called, the generic function TEST is already created by the defmethod macro at macroexpansion time, but not completely initialized, so defgeneric ends up calling reinitialize-instance with :method-combination, but the slot for method-combination is called %METHOD-COMBINATION and its initarg is :%METHOD-COMBINATION, so nothing gets initialized. This was changed in the following commit: http://trac.common-lisp.net/armedbear/changeset?new=13947%40trunk%2Fabcl%2F… -- With best regards, Stas.

2 1

[armedbear-devel] [Patch]: Make listen work on gray-streams.
by Stas Boukarev 13 Aug '12

13 Aug '12

gray-listen functions weren't plugged into cl:listen for some reason. -- With best regards, Stas.

2 1

[armedbear-devel] LISTEN doesn't work on non-character streams
by Stas Boukarev 13 Aug '12

13 Aug '12

(with-open-file (stream "/dev/zero" :element-type '(unsigned-byte 8)) (listen stream)) #<FILE-STREAM {5CF9DE08}> is not a character input stream. -- With best regards, Stas.

1 0

[armedbear-devel] Stack overflow when compiler macro with fallback is triggered
by James M. Lawrence 12 Aug '12

12 Aug '12

(eval-when (:compile-toplevel :load-toplevel :execute) (defun foo () 99) (define-compiler-macro foo () `(locally (declare (notinline foo)) (foo)))) (defun call-foo () (foo)) Of course, the use case is a compiler macro that says, "OK, let's optimize! ... Never mind, I don't want to optimize that." Armed Bear Common Lisp 1.0.1-svn-13750-13751 Java 1.7.0_04 Oracle Corporation [...] ; (DEFUN FOO ...) ; (DEFINE-COMPILER-MACRO FOO ...) ; (DEFUN CALL-FOO ...) java.lang.StackOverflowError at org.armedbear.lisp.StructureObject$pf_make_structure.execute(StructureObject.java:671) at org.armedbear.lisp.Symbol.execute(Symbol.java:865) at org.armedbear.lisp.LispThread.execute(LispThread.java:760) at org.armedbear.lisp.jvm_714.execute(jvm.lisp:567) at org.armedbear.lisp.CompiledClosure.execute(CompiledClosure.java:92) at org.armedbear.lisp.Symbol.execute(Symbol.java:775) at org.armedbear.lisp.LispThread.execute(LispThread.java:633) at org.armedbear.lisp.jvm_746.execute(jvm.lisp:572) at org.armedbear.lisp.Symbol.execute(Symbol.java:775) at org.armedbear.lisp.LispThread.execute(LispThread.java:633) at org.armedbear.lisp.compiler_pass1_26.execute(compiler-pass1.lisp:487) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_61.execute(compiler-pass1.lisp:1204) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_61.execute(compiler-pass1.lisp:1204) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.compiler_pass1_20.execute(compiler-pass1.lisp:383) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_26.execute(compiler-pass1.lisp:487) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_61.execute(compiler-pass1.lisp:1204) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_61.execute(compiler-pass1.lisp:1204) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.compiler_pass1_20.execute(compiler-pass1.lisp:383) at org.armedbear.lisp.Symbol.execute(Symbol.java:785) at org.armedbear.lisp.LispThread.execute(LispThread.java:649) at org.armedbear.lisp.compiler_pass1_26.execute(compiler-pass1.lisp:487) [...]

5 5

[armedbear-devel] Double printing of the prompt
by Blake McBride 12 Aug '12

12 Aug '12

Greetings, The latest version of ABCL is double printing the prompt at times as follows: Armed Bear Common Lisp 1.1.0-dev-svn-13783 Java 1.6.0_29 Apple Inc. Java HotSpot(TM) 64-Bit Server VM Low-level initialization completed in 0.465 seconds. Startup completed in 1.453 seconds. Type ":help" for a list of available commands. CL-USER(1): 3 3 CL-USER(2): 'hello HELLO CL-USER(3): '(a b c) (A B C) CL-USER(4): nil NIL CL-USER(5): CL-USER(5): 77 77 CL-USER(6): (cdr '(hello)) NIL CL-USER(7): nil NIL CL-USER(8): CL-USER(8): 3 3 CL-USER(9):

3 3

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

armedbear-devel