In analyzing the [bug with CL-JSON][1], I think the culprit lies in improper compilation of PROGV forms, although I have yet to come up with a simple example factored out of the CL-JSON codebase. Reading last night's #abcl logs, I saw that Erik made a comment about problems with PROGV as well, without specifics. Thinking that if I had other examples of how PROGV misbehaves it might shed some light on my problems, I'd like to ask for details about other misbehavior of compiled PROGV forms however scant. Cases that are reproducible would be most welcome.
[1]: http://trac.common-lisp.net/armedbear/ticket/85
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Hi Mark,
On Fri, Mar 5, 2010 at 8:57 AM, Mark Evenson evenson@panix.com wrote:
In analyzing the [bug with CL-JSON][1], I think the culprit lies in improper compilation of PROGV forms, although I have yet to come up with a simple example factored out of the CL-JSON codebase. Reading last night's #abcl logs, I saw that Erik made a comment about problems with PROGV as well, without specifics. Thinking that if I had other examples of how PROGV misbehaves it might shed some light on my problems, I'd like to ask for details about other misbehavior of compiled PROGV forms however scant. Cases that are reproducible would be most welcome.
Well, the problem is that I looked at the implementation of interpreted PROGV and then at what the compiler generates. The structure of the code looks the same: interpreted uses a try/finally block and that's what compiled does too.
The difference that I can see is in the way the compiler and the interpreter do local transfer of control (like RETURN-FROM, which CL-JSON uses): the interpreter uses the same means as for non-local: raising a Java exception (and thus always triggering the finally). The compiler restores a specific dynamic environment and uses a JMP instruction to jump out of the PROGV block.
However, this mechanism is used for other blocks which binding specials too and I'm not aware of issues with those. This is why I would like to know more specific reproduction recipies, because the cl-json code is a bit too complex to test this entirely.
One thing to try would be to add (progv-node-p block) to the BLOCK-REQUIRES-NON-LOCAL-EXIT-P function, making the compiler generate non-local exits (exceptions) just like the interpreter. If that fixes it: we have a problem with the dyn-env determination or some other part of dyn-env management with local transfer of control blocks.
HTH,
Erik.
On 3/5/10 4:13 PM, Erik Huelsmann wrote: […]
One thing to try would be to add (progv-node-p block) to the BLOCK-REQUIRES-NON-LOCAL-EXIT-P function, making the compiler generate non-local exits (exceptions) just like the interpreter. If that fixes it: we have a problem with the dyn-env determination or some other part of dyn-env management with local transfer of control blocks.
As mentioned on #abcl, Erik's suggestion as in the attached patch fixes the behavior in CL-JSON. We're debating now on how to proceed with fixing ABCL.
In working on the PROGV problems, I can't understand why the following simple case works with JVM.
For the forms (attached in progv.lisp):
(defvar *a* "initial")
(defun foo () (progv '(*a*) '((symbol-value '*a*)) (setf *a* "foo") (return-from foo 42)))
You get the following JVM code:
0: invokestatic #27; //Method org/armedbear/lisp/LispThread.currentThread:()Lorg/armedbear/lisp/LispThread; 3: astore_1 4: getstatic #31; //Field OBJSTR106:Lorg/armedbear/lisp/LispObject; 7: getstatic #33; //Field OBJSTR107:Lorg/armedbear/lisp/LispObject; 10: aload_1 11: invokevirtual #37; //Method org/armedbear/lisp/LispThread.markSpecialBindings:()Lorg/armedbear/lisp/SpecialBindingsMark; 14: astore_2 15: aload_1 16: invokestatic #43; //Method org/armedbear/lisp/Lisp.progvBindVars:(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispThread;)V 19: aload_1 20: getstatic #45; //Field SYM108_A:Lorg/armedbear/lisp/Symbol; 23: getstatic #47; //Field STR109:Lorg/armedbear/lisp/SimpleString; 26: invokevirtual #51; //Method org/armedbear/lisp/LispThread.setSpecialVariable:(Lorg/armedbear/lisp/Symbol;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; 29: pop 30: getstatic #53; //Field FIXNUM_42:Lorg/armedbear/lisp/LispInteger; 33: areturn 34: aload_1 35: aload_2 36: invokevirtual #57; //Method org/armedbear/lisp/LispThread.resetSpecialBindings:(Lorg/armedbear/lisp/SpecialBindingsMark;)V 39: athrow 40: aload_1 41: aload_2 42: invokevirtual #57; //Method org/armedbear/lisp/LispThread.resetSpecialBindings:(Lorg/armedbear/lisp/SpecialBindingsMark;)V 45: areturn
What I don't understand is how the *A* binding is restored at all. But ABCL executes this code correctly, restoring *A* to the string "initial" after an execution of FOO. But as you see in the code, the SETF is executed in line 26, then the return value is discarded via pop, then the return value is put on the stack in line 30, and then the method returns via ARETURN. None of the exception handlers which invoke resetSpecialBindings at the end of the the code (lines 34-39, 40-45) are executed at all. So how does the special variable get reset? Is there something happening in the autoloaded that intercepts with a proxy that resets things?
Searching for a reasonable JVM debugger that doesn't assume you are compiling from Java doesn't seem to turn up much. Does anyone have a trick for following the JVM executing such code?
Very confused, Mark
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Tue, Mar 9, 2010 at 11:51 AM, Mark Evenson evenson@panix.com wrote:
In working on the PROGV problems, I can't understand why the following simple case works with JVM.
For the forms (attached in progv.lisp):
(defvar *a* "initial")
(defun foo () (progv '(*a*) '((symbol-value '*a*)) (setf *a* "foo") (return-from foo 42)))
You get the following JVM code:
0: invokestatic #27; //Method org/armedbear/lisp/LispThread.currentThread:()Lorg/armedbear/lisp/LispThread; 3: astore_1 4: getstatic #31; //Field OBJSTR106:Lorg/armedbear/lisp/LispObject; 7: getstatic #33; //Field OBJSTR107:Lorg/armedbear/lisp/LispObject; 10: aload_1 11: invokevirtual #37; //Method org/armedbear/lisp/LispThread.markSpecialBindings:()Lorg/armedbear/lisp/SpecialBindingsMark; 14: astore_2 15: aload_1 16: invokestatic #43; //Method org/armedbear/lisp/Lisp.progvBindVars:(Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispObject;Lorg/armedbear/lisp/LispThread;)V 19: aload_1 20: getstatic #45; //Field SYM108_A:Lorg/armedbear/lisp/Symbol; 23: getstatic #47; //Field STR109:Lorg/armedbear/lisp/SimpleString; 26: invokevirtual #51; //Method org/armedbear/lisp/LispThread.setSpecialVariable:(Lorg/armedbear/lisp/Symbol;Lorg/armedbear/lisp/LispObject;)Lorg/armedbear/lisp/LispObject; 29: pop 30: getstatic #53; //Field FIXNUM_42:Lorg/armedbear/lisp/LispInteger; 33: areturn 34: aload_1 35: aload_2 36: invokevirtual #57; //Method org/armedbear/lisp/LispThread.resetSpecialBindings:(Lorg/armedbear/lisp/SpecialBindingsMark;)V 39: athrow 40: aload_1 41: aload_2 42: invokevirtual #57; //Method org/armedbear/lisp/LispThread.resetSpecialBindings:(Lorg/armedbear/lisp/SpecialBindingsMark;)V 45: areturn
What I don't understand is how the *A* binding is restored at all. But ABCL executes this code correctly, restoring *A* to the string "initial" after an execution of FOO. But as you see in the code, the SETF is executed in line 26, then the return value is discarded via pop, then the return value is put on the stack in line 30, and then the method returns via ARETURN. None of the exception handlers which invoke resetSpecialBindings at the end of the the code (lines 34-39, 40-45) are executed at all. So how does the special variable get reset? Is there something happening in the autoloaded that intercepts with a proxy that resets things?
Are you sure those are both exception handlers, and there's not a finally block instead? I don't know how those constructs translate to bytecode, but if I had to implement special variables in Java, I would use try-finally to restore them.
Alessio
Searching for a reasonable JVM debugger that doesn't assume you are compiling from Java doesn't seem to turn up much. Does anyone have a trick for following the JVM executing such code?
Very confused, Mark
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On Mar 9, 2010, at 11:56 AM, Alessio Stalla wrote: […]
What I don't understand is how the *A* binding is restored at all. But ABCL executes this code correctly, restoring *A* to the string "initial" after an execution of FOO. But as you see in the code, the SETF is executed in line 26, then the return value is discarded via pop, then the return value is put on the stack in line 30, and then the method returns via ARETURN. None of the exception handlers which invoke resetSpecialBindings at the end of the the code (lines 34-39, 40-45) are executed at all. So how does the special variable get reset? Is there something happening in the autoloaded that intercepts with a proxy that resets things?
Are you sure those are both exception handlers, and there's not a finally block instead? I don't know how those constructs translate to bytecode, but if I had to implement special variables in Java, I would use try-finally to restore them.
Exception handlers are listed in a table (which is in the javap output but I didn't list in the post):
Exception table: from to target type 15 34 34 any
This exception table instructs the JVM that from index 15 to 33 (the "to" is exclusive), on any exception, start executing the code at index 34. The code from 40-45 isn't executed at all as far as I can tell. What *would* make sense would be that something was inserting a JSR instruction to the main method with the return set to index 40, but as far as I understand the JVM this isn't possible.
As far as I understand it [finally doesn't have JVM primitives][1] to support it: its implemented in terms of the exception handlers with JSR and GOTOs to handle the specific cases.
[1]: http://java.sun.com/docs/books/jvms/second_edition/html/Compiling.doc.html#1...
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Tue, Mar 9, 2010 at 3:17 AM, Mark Evenson evenson@panix.com wrote:
On Mar 9, 2010, at 11:56 AM, Alessio Stalla wrote: […]
What I don't understand is how the *A* binding is restored at all. But ABCL executes this code correctly, restoring *A* to the string "initial" after an execution of FOO. But as you see in the code, the SETF is executed in line 26, then the return value is discarded via pop, then the return value is put on the stack in line 30, and then the method returns via ARETURN. None of the exception handlers which invoke resetSpecialBindings at the end of the the code (lines 34-39, 40-45) are executed at all. So how does the special variable get reset? Is there something happening in the autoloaded that intercepts with a proxy that resets things?
Are you sure those are both exception handlers, and there's not a finally block instead? I don't know how those constructs translate to bytecode, but if I had to implement special variables in Java, I would use try-finally to restore them.
Exception handlers are listed in a table (which is in the javap output but I didn't list in the post):
Exception table: from to target type 15 34 34 any
any = finally block
This exception table instructs the JVM that from index 15 to 33 (the "to" is exclusive), on any exception, start executing the code at index 34. The code from 40-45 isn't executed at all as far as I can tell. What *would* make sense would be that something was inserting a JSR instruction to the main method with the return set to index 40, but as far as I understand the JVM this isn't possible.
But yes.. I am used to seeing at least a JSR to call 34
As far as I understand it [finally doesn't have JVM primitives][1] to support it: its implemented in terms of the exception handlers with JSR and GOTOs to handle the specific cases.
So definately confusing
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel mailing list armedbear-devel@common-lisp.net http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
On Mar 9, 2010, at 12:22 PM, dmiles@users.sourceforge.net wrote:
[…]
Exception handlers are listed in a table (which is in the javap output but I didn't list in the post):
Exception table: from to target type 15 34 34 any
any = finally block
But this won't get executed unless an exception is invoked, so its not at all the equivalent of a finally block.
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
armedbear-devel@common-lisp.net
-
Alessio Stalla -
dmiles＠users.sourceforge.net -
Erik Huelsmann -
Mark Evenson