Hi all,
In an attempt to solve ticket 201 and everything related (now that I'm into it anyway), I've been staring at the failure of one of ABCL's own tests and can't seem to understand what's going on.
So, DMC-TEST-MC.1 is failing because it uses a non-standard METHOD-QUALIFIER. STD-COMPUTE-EFFECTIVE-METHOD throws an error when matching the qualifiers of the defined methods with the method groups and finds that the method combination isn't one of type long-form. However, the problem is: it really *is* of type long form. However, this seems to happen *only* in the compiled case....s
So, I inserted a trace:
(eval-when (:compile-toplevel) (trace mop::STD-COMPUTE-EFFECTIVE-METHOD))
and extracted the test form from outside the test case, in order to be able to see the error triggered (RT just prints #<simple-error >):
(dmc-test-mc.1 :k 1)
The result is this backtrace:
[java] 0: (MOP::STD-COMPUTE-EFFECTIVE-METHOD #<STANDARD-GENERIC-FUNCTION DMC-TEST-MC.1 {10D3D99}> #<METHOD-COMBINATION {69B861}> (#<STANDARD-METHOD {C821EF}>))
which clearly shows the method combination is of type METHOD-COMBINATION and not of type LONG-METHOD-COMBINATION.
I do understand that this is the error, but I can't for the life of me figure out how we get into the situation that the method combination being instantiated is of the resulting type METHOD-COMBINATION...
Any help on debugging or additional insights would be extremely appreciated!
Bye,
Erik.
Even more puzzling:
There seems to be a difference in behaviour between the slime C-c C-c, C-c C-r behaviour (compile a form, evaluate a region) and running a PROGN with the forms from the slime repl.
Sure hope others can help out here. All I can do now is create the correct D-M-C implementation but writing tests and verifying them for their behaviour is not possible this way.
BTW: the PROGN and the C-c C-c approach probably differ in that one is interpreted and the other compiled. So, it looks like an interaction between compilation and D-M-C.
Bye,
Erik.
On Sat, Aug 4, 2012 at 11:31 AM, Erik Huelsmann ehuels@gmail.com wrote:
Hi all,
In an attempt to solve ticket 201 and everything related (now that I'm into it anyway), I've been staring at the failure of one of ABCL's own tests and can't seem to understand what's going on.
So, DMC-TEST-MC.1 is failing because it uses a non-standard METHOD-QUALIFIER. STD-COMPUTE-EFFECTIVE-METHOD throws an error when matching the qualifiers of the defined methods with the method groups and finds that the method combination isn't one of type long-form. However, the problem is: it really *is* of type long form. However, this seems to happen *only* in the compiled case....s
So, I inserted a trace:
(eval-when (:compile-toplevel) (trace mop::STD-COMPUTE-EFFECTIVE-METHOD))
and extracted the test form from outside the test case, in order to be able to see the error triggered (RT just prints #<simple-error >):
(dmc-test-mc.1 :k 1)
The result is this backtrace:
[java] 0: (MOP::STD-COMPUTE-EFFECTIVE-METHOD#<STANDARD-GENERIC-FUNCTION DMC-TEST-MC.1 {10D3D99}> #<METHOD-COMBINATION {69B861}> (#<STANDARD-METHOD {C821EF}>))
which clearly shows the method combination is of type METHOD-COMBINATION and not of type LONG-METHOD-COMBINATION.
I do understand that this is the error, but I can't for the life of me figure out how we get into the situation that the method combination being instantiated is of the resulting type METHOD-COMBINATION...
Any help on debugging or additional insights would be extremely appreciated!
Bye,
Erik.
Hi,
I see similar errors in other situations. My guess is that at load time, classes get redefined but object literals from compile-time (with the compile-time class) stay around in the fasl. These literals then have (come load-time) a class with the same name but different identity than the one that find-class returns.
Rudi
Sent from my iPad
On 4 Aug 2012, at 14:00, Erik Huelsmann ehuels@gmail.com wrote:
Even more puzzling:
There seems to be a difference in behaviour between the slime C-c C-c, C-c C-r behaviour (compile a form, evaluate a region) and running a PROGN with the forms from the slime repl.
Sure hope others can help out here. All I can do now is create the correct D-M-C implementation but writing tests and verifying them for their behaviour is not possible this way.
BTW: the PROGN and the C-c C-c approach probably differ in that one is interpreted and the other compiled. So, it looks like an interaction between compilation and D-M-C.
Bye,
Erik.
On Sat, Aug 4, 2012 at 11:31 AM, Erik Huelsmann ehuels@gmail.com wrote:
Hi all,
In an attempt to solve ticket 201 and everything related (now that I'm into it anyway), I've been staring at the failure of one of ABCL's own tests and can't seem to understand what's going on.
So, DMC-TEST-MC.1 is failing because it uses a non-standard METHOD-QUALIFIER. STD-COMPUTE-EFFECTIVE-METHOD throws an error when matching the qualifiers of the defined methods with the method groups and finds that the method combination isn't one of type long-form. However, the problem is: it really *is* of type long form. However, this seems to happen *only* in the compiled case....s
So, I inserted a trace:
(eval-when (:compile-toplevel) (trace mop::STD-COMPUTE-EFFECTIVE-METHOD))
and extracted the test form from outside the test case, in order to be able to see the error triggered (RT just prints #<simple-error >):
(dmc-test-mc.1 :k 1)
The result is this backtrace:
[java] 0: (MOP::STD-COMPUTE-EFFECTIVE-METHOD #<STANDARD-GENERIC-FUNCTION DMC-TEST-MC.1 {10D3D99}> #<METHOD-COMBINATION {69B861}> (#<STANDARD-METHOD {C821EF}>))which clearly shows the method combination is of type METHOD-COMBINATION and not of type LONG-METHOD-COMBINATION.
I do understand that this is the error, but I can't for the life of me figure out how we get into the situation that the method combination being instantiated is of the resulting type METHOD-COMBINATION...
Any help on debugging or additional insights would be extremely appreciated!
Bye,
Erik.
Hi Rudi,
I see similar errors in other situations. My guess is that at load time,
classes get redefined but object literals from compile-time (with the compile-time class) stay around in the fasl. These literals then have (come load-time) a class with the same name but different identity than the one that find-class returns.
Thinking about this assumption, I'm still very much puzzled what in our code base would instate an object of class METHOD-COMBINATION. That is, not of one of its two subclasses (SHORT-* or LONG-*), but of the direct type.
From what I searched our code base I can't find any code which would do
that (i.e. instantiate an object of that type).
Also, when looking at the MACROEXPAND-ALL output of DEFINE-METHOD-COMBINATION, DEFGENERIC and DEFMETHOD, there don't seem to be any direct references to any objects or classes. E.g. the method combination designation in DEFGENERIC gets saved as a symbol which is passed through to our internal %DEFGENERIC function which is called upon evaluation of the DEFGENERIC form be it compiled or interpreted.
Also, compilation of DEFINE-METHOD-COMBINATION, DEFGENERIC and DEFMETHOD don't have any other compile-time side effects than that the symbol of the generic function is (temporarily) assigned a function to its symbol function slot. Meaning D-M-C compilation is completely compile-time side effect free, as far as I can tell. Which would be completely contrary to your expectation.
So, I'm still very puzzled. What would be our best way forward? I'd love to fix this, because I think we'll run into it later as well in "real world" cases. Would getting together on IRC to chat about it be a good next step? (One issue that I see is that all these tests used to pass when I wrote them. So at some point something must have changed which made us incompatible with ourselves. That's another reason why I at least want to understand what's going on...)
Bye,
Erik.
On Aug 7, 2012, at 13:47, Erik Huelsmann ehuels@gmail.com wrote:
[...]
(One issue that I see is that all these tests used to pass when I wrote them. So at some point something must have changed which made us incompatible with ourselves. That's another reason why I at least want to understand what's going on...)
That I can answer, I think: generic functions used to store the method combination _name_, and the actual method combination would be looked up during the computation of the effective method. Now the generic function object stores the method combination object itself (purely because the amop test suite checks the type of the return value of generic-function-method-combination).
Rudi
(One issue that I see is that all these tests used to pass when I wrote them. So at some point something must have changed which made us incompatible with ourselves. That's another reason why I at least want to understand what's going on...)
That I can answer, I think: generic functions used to store the method combination _name_, and the actual method combination would be looked up during the computation of the effective method. Now the generic function object stores the method combination object itself (purely because the amop test suite checks the type of the return value of generic-function-method-combination).
Ok. I figured out what's happening, at least inside SLIME, but I assume the same happens for the compiler. At least, if we can fix this happening in SLIME, we can probably fix it for the compiler as well.
So, just to show the code that's failing to be correctly compiled:
Form 1: (define-method-combination dmc-test-args-with-optional.4 () ((methods ())) (:arguments &optional (a :default sup-p)) `(progn ,@(mapcar (lambda (method) `(call-method ,method)) methods) (values ,a ,sup-p)))
Form 2: (defgeneric dmc-test-args-with-optional.4a (x &optional b) (:method-combination dmc-test-args-with-optional.4) (:method (x &optional b) (progn x b)))
My compilation steps are to go into Form 1 and hit C-c C-c. Then do the same with Form 2.
Every time when I do that, the method combination of the generic function comes out with #<METHOD-COMBINATION STANDARD>. Which is clearly wrong.
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION 'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
I'm now wondering what the next step should be to resolve this situation...
Ah. Going back 10 months and looking at the DEFMETHOD macro there, I see now what the difference is between then and now: back then we didn't depend on the GF existing during the macro expansion. Now we do. Can we generate a different macroexpansion which defers the GF-being-defined requirement to evaluation time?
Bye,
Erik.
On Aug 7, 2012, at 20:41, Erik Huelsmann ehuels@gmail.com wrote:
Ok. I figured out what's happening, at least inside SLIME, but I assume the same happens for the compiler. At least, if we can fix this happening in SLIME, we can probably fix it for the compiler as well.
So, just to show the code that's failing to be correctly compiled:
Form 1: (define-method-combination dmc-test-args-with-optional.4 () ((methods ())) (:arguments &optional (a :default sup-p)) `(progn ,@(mapcar (lambda (method) `(call-method ,method)) methods) (values ,a ,sup-p)))
Form 2: (defgeneric dmc-test-args-with-optional.4a (x &optional b) (:method-combination dmc-test-args-with-optional.4) (:method (x &optional b) (progn x b)))
My compilation steps are to go into Form 1 and hit C-c C-c. Then do the same with Form 2.
Every time when I do that, the method combination of the generic function comes out with #<METHOD-COMBINATION STANDARD>. Which is clearly wrong.
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION 'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
I'm now wondering what the next step should be to resolve this situation...
Ah. Going back 10 months and looking at the DEFMETHOD macro there, I see now what the difference is between then and now: back then we didn't depend on the GF existing during the macro expansion. Now we do. Can we generate a different macroexpansion which defers the GF-being-defined requirement to evaluation time?
Cute. I think we can beef up %defgeneric to accept a list of method descriptions and the macro environment (for make-method-lambda) and call ensure-method inside %defgeneric, after creating the generic function. I can have a look at that.
Rudi
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION
'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before
compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
I'm now wondering what the next step should be to resolve this
situation...
Ah. Going back 10 months and looking at the DEFMETHOD macro there, I see
now what the difference is between then and now: back then we didn't depend on the GF existing during the macro expansion. Now we do. Can we generate a different macroexpansion which defers the GF-being-defined requirement to evaluation time?
Cute. I think we can beef up %defgeneric to accept a list of method descriptions and the macro environment (for make-method-lambda) and call ensure-method inside %defgeneric, after creating the generic function. I can have a look at that.
I can see how doing that would work at run-time, ie where %defgeneric would be called without the image having been exited (simply grab the environment and tuck it in a closure, or even simply stick it in the expansion). However, how do you expect to do the same at file-compilation time where the %defgeneric gets called after the image (possibly) has been closed and cleanly started up?
Bye,
Erik.
On Tue, Aug 7, 2012 at 9:09 PM, Erik Huelsmann ehuels@gmail.com wrote:
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION
'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before
compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
I'm now wondering what the next step should be to resolve this
situation...
Ah. Going back 10 months and looking at the DEFMETHOD macro there, I
see now what the difference is between then and now: back then we didn't depend on the GF existing during the macro expansion. Now we do. Can we generate a different macroexpansion which defers the GF-being-defined requirement to evaluation time?
Cute. I think we can beef up %defgeneric to accept a list of method descriptions and the macro environment (for make-method-lambda) and call ensure-method inside %defgeneric, after creating the generic function. I can have a look at that.
I can see how doing that would work at run-time, ie where %defgeneric would be called without the image having been exited (simply grab the environment and tuck it in a closure, or even simply stick it in the expansion). However, how do you expect to do the same at file-compilation time where the %defgeneric gets called after the image (possibly) has been closed and cleanly started up?
Reading this quote:
*Note:* The body of methods can also appear in the *:initial-methods* option of *defgeneric* forms. Initial methods are not considered by any of the protocols specified in this document.
from http://www.alu.org/mop/concepts.html
leaves me wondering if we're too strict to apply the requirement to use DEFMETHOD and MAKE-METHOD-LAMBDA processing the way we do now. Does this provide a way out?
Bye,
Erik.
And the attachment...
On Tue, Aug 7, 2012 at 9:09 PM, Erik Huelsmann ehuels@gmail.com wrote:
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION
'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before
compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
I'm now wondering what the next step should be to resolve this
situation...
Ah. Going back 10 months and looking at the DEFMETHOD macro there, I
see now what the difference is between then and now: back then we didn't depend on the GF existing during the macro expansion. Now we do. Can we generate a different macroexpansion which defers the GF-being-defined requirement to evaluation time?
Cute. I think we can beef up %defgeneric to accept a list of method descriptions and the macro environment (for make-method-lambda) and call ensure-method inside %defgeneric, after creating the generic function. I can have a look at that.
I can see how doing that would work at run-time, ie where %defgeneric would be called without the image having been exited (simply grab the environment and tuck it in a closure, or even simply stick it in the expansion). However, how do you expect to do the same at file-compilation time where the %defgeneric gets called after the image (possibly) has been closed and cleanly started up?
Bye,
Erik.
Hi!
Ok. I figured out what's happening, at least inside SLIME, but I assume the same happens for the compiler. At least, if we can fix this happening in SLIME, we can probably fix it for the compiler as well.
So, just to show the code that's failing to be correctly compiled:
Form 1: (define-method-combination dmc-test-args-with-optional.4 () ((methods ())) (:arguments &optional (a :default sup-p)) `(progn ,@(mapcar (lambda (method) `(call-method ,method)) methods) (values ,a ,sup-p)))
Form 2: (defgeneric dmc-test-args-with-optional.4a (x &optional b) (:method-combination dmc-test-args-with-optional.4) (:method (x &optional b) (progn x b)))
My compilation steps are to go into Form 1 and hit C-c C-c. Then do the
same with Form 2.
Every time when I do that, the method combination of the generic
function comes out with #<METHOD-COMBINATION STANDARD>. Which is clearly wrong.
What happens is that the second form expands into:
(PROG1 (SYSTEM:%DEFGENERIC 'DMC-TEST-ARGS-WITH-OPTIONAL.4A :LAMBDA-LIST '(X &OPTIONAL B) :METHOD-COMBINATION '(DMC-TEST-ARGS-WITH-OPTIONAL.4)) (PUSH (DEFMETHOD DMC-TEST-ARGS-WITH-OPTIONAL.4A (X &OPTIONAL B) (PROGN X B)) (SYSTEM:GENERIC-FUNCTION-INITIAL-METHODS (FDEFINITION
'DMC-TEST-ARGS-WITH-OPTIONAL.4A))))
Because the macro expander continues to expand the forms before
compiling them, it'll expand the DEFMETHOD above. When the DEFMETHOD gets expanded however, it's expansion is dependent on the existence of the generic function. However, since the form as a whole hasn't been interpreted yet, there *is* no generic function: after all, that's being created in the first form of the PROG1.
Something similar probably happens during compilation.
Cute. I think we can beef up %defgeneric to accept a list of method descriptions and the macro environment (for make-method-lambda) and call ensure-method inside %defgeneric, after creating the generic function. I can have a look at that.
Well, I hope you don't mind, but I quickly looked at the problem last week myself as well. From all the scenario's that I could come up with the simplest (and most unsatisfying in terms of "everything is delayed to load time and I want more at compile time") is what I tried (see attached patch).
This patch seems to solve the "compile in memory" issue, but for some reason, it doesn't seem to solve the compilation problem.
Also, note that the patch is incomplete, because the forms - before being stored for load-time evaluation - should at least be macro-expanded to expand macrolet and symbol-macrolet in the body.
Hopefully it's something to give you a head start in your analysis. (I'm not going to commit this, unless it's by accident.)
Bye,
Erik.
On Aug 7, 2012, at 20:41, Erik Huelsmann ehuels@gmail.com wrote:
[...]
So, just to show the code that's failing to be correctly compiled:
Form 1: (define-method-combination dmc-test-args-with-optional.4 () ((methods ())) (:arguments &optional (a :default sup-p)) `(progn ,@(mapcar (lambda (method) `(call-method ,method)) methods) (values ,a ,sup-p)))
Form 2: (defgeneric dmc-test-args-with-optional.4a (x &optional b) (:method-combination dmc-test-args-with-optional.4) (:method (x &optional b) (progn x b)))
My compilation steps are to go into Form 1 and hit C-c C-c. Then do the same with Form 2.
Every time when I do that, the method combination of the generic function comes out with #<METHOD-COMBINATION STANDARD>. Which is clearly wrong.
After fixing Stas's bug report (missing :method-combination initarg for standard-generic-functions), following this recipe gets me the right method combination. Can you re-check as well?
Thanks,
Rudi
Yup. It works very well now!
Thanks!
Bye,
Erik.
On Mon, Aug 13, 2012 at 3:39 PM, Rudolf Schlatte rudi@constantly.at wrote:
On Aug 7, 2012, at 20:41, Erik Huelsmann ehuels@gmail.com wrote:
[...]
So, just to show the code that's failing to be correctly compiled:
Form 1: (define-method-combination dmc-test-args-with-optional.4 () ((methods ())) (:arguments &optional (a :default sup-p)) `(progn ,@(mapcar (lambda (method) `(call-method ,method)) methods) (values ,a ,sup-p)))
Form 2: (defgeneric dmc-test-args-with-optional.4a (x &optional b) (:method-combination dmc-test-args-with-optional.4) (:method (x &optional b) (progn x b)))
My compilation steps are to go into Form 1 and hit C-c C-c. Then do the
same with Form 2.
Every time when I do that, the method combination of the generic
function comes out with #<METHOD-COMBINATION STANDARD>. Which is clearly wrong.
After fixing Stas's bug report (missing :method-combination initarg for standard-generic-functions), following this recipe gets me the right method combination. Can you re-check as well?
Thanks,
Rudi
armedbear-devel@common-lisp.net
-
Erik Huelsmann -
Rudi Schlatte -
Rudolf Schlatte