I got curious about how I might generate better code for JSS. The JSS reader macro generates a lambda, so the typical pattern one sees is: (#"matches" 'integer ".*a" ) -> ((LAMBDA (#:|#"matches"-first| &REST #:|#"matches"-rest|) (JSS:INVOKE-RESTARGS "matches" #:|#"matches"-first| #:|#"matches"-rest| NIL)) 'INTEGER ".*a") (yes I know this will generate an error at runtime) Right now the compiled code for one such call goes through 3 functions: (JSTATIC "matches" #<java class java.lang.Integer {3BBD30F}> ".*a") (APPLY #<JSTATIC {41D16045}> "matches" #<java class java.lang.Integer {3BBD30F}> (".*a")) (INVOKE-RESTARGS "matches" INTEGER (".*a") NIL) Really we only need: (JSTATIC "matches" #<java class java.lang.Integer {3BBD30F}> ".*a") and we know enough at compile time to generate that form. If only I could figure out where the right hook would be. The only place I could figure to do this is in precompile-function-call [1] With a hook in place, i define the hook as [2] After checking if the function call is one of the JSS ones, the hook transforms ((lambda(a b ) (jss::invoke-restargs-experimental method a b raw?) c d) to (jss::invoke-restargs-experimental method c '(d) raw? t) jss::invoke-restargs-experimental is macro that does the transformation I want. [3] The question is: Is there a more elegant way to do this, or a hook already built that I could use instead of redefining precompile-function-call If not, would it be reasonable to add a hook in the ABCL source so I don't need to patch it to do the optimization. Thanks, Alan [1] (defun precompile-function-call (form) (let ((op (car form))) (when (and (consp op) (eq (%car op) 'LAMBDA)) (return-from precompile-function-call I added this line --- (or (jss-fix-precompile op (mapcar #'precompile1 (cdr form))) --- (cons (precompile-lambda op) (mapcar #'precompile1 (cdr form)))))) (when (or (not *in-jvm-compile*) (notinline-p op)) (return-from precompile-function-call (precompile-cons form))) (when (source-transform op) (let ((new-form (expand-source-transform form))) (when (neq new-form form) (return-from precompile-function-call (precompile1 new-form))))) (when *enable-inline-expansion* (let ((expansion (inline-expansion op))) (when expansion (let ((explain *explain*)) (when (and explain (memq :calls explain)) (format t "; inlining call to ~S~%" op))) (return-from precompile-function-call (precompile1 (expand-inline form expansion)))))) (cons op (mapcar #'precompile1 (cdr form))))) [2] (defun jss-fix-precompile (op args) "Check if this is one of mine, and do the rewrite, otherwise pass" (ignore-errors (let ((body (cddr op))) (if (and (= (length body) 1) (consp (car body)) (eq (caar body) 'jss::invoke-restargs-experimental)) (precompile-function-call `(jss::invoke-restargs-experimental ,(second (car body)) ,(car args) ,(cdr args) ,(fifth (car body)) t)) nil)))) [3] (defmacro invoke-restargs-experimental (&whole form method object args &optional (raw? nil) (precompile nil)) "If I'm precompiling then I can do the transformation. If not I revert to the original method" (if precompile (if (and (consp object) (eq (car object) 'quote)) (let ((object (eval object))) (let* ((object-as-class-name (if (symbolp object) (maybe-resolve-class-against-imports object) )) (object-as-class (if object-as-class-name (find-java-class object-as-class-name)))) (cl-user::print-db object object-as-class-name object-as-class) (if raw? `(jstatic-raw ,method ,object-as-class ,@args) `(jstatic ,method ,object-as-class ,@args)))) (if raw? `(if (symbolp ,object) (jstatic-raw ,method (find-java-class ,object) ,@args) (jcall-raw ,method ,object ,@args)) `(if (symbolp ,object) (jstatic ,method (find-java-class ,object) ,@args) (jcall ,method ,object ,@args)))) `(invoke-restargs ,method ,object ,args ,raw?)))
On 2016/10/27 05:18, Alan Ruttenberg wrote:
I got curious about how I might generate better code for JSS.
[…]
The question is: Is there a more elegant way to do this, or a hook already built that I could use instead of redefining precompile-function-call
If not, would it be reasonable to add a hook in the ABCL source so I don't need to patch it to do the optimization.
A quick grep of the source does not seem to indicate any hooks into the controlling the behavior/implementation PRECOMPILER::PRECOMPILE-FUNCTION-CALL, supporting your finding of no currently reasonable mechanism for a user to manipulate. Therefore, I agree that creating a hook mechanism would be the reasonable way forward. I worry a little about compiler speed decrease if we do a naive hook implementation (i.e. something that MAPs APPLY). Would it be more reasonable to memoize possible values of PRECOMPILE-FUNCTION-CALL, providing an API to switch at runtime? Since PRECOMPILER isn't a documented ABCL package, it would be best to define the API in PRECOMPILER, but have hooks that inserts the public API in the SYSTEM package via the appropriate import and export of the symbols. Cool work! What sort of efficiency gains do you expect for JSS here? -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Hi, Did you look at define-compiler-macro as a possible hook mechanism? Regards, Erik On Oct 27, 2016 09:28, "Mark Evenson" <evenson@panix.com> wrote:
On 2016/10/27 05:18, Alan Ruttenberg wrote:
I got curious about how I might generate better code for JSS.
[…]
The question is: Is there a more elegant way to do this, or a hook already built that I could use instead of redefining precompile-function-call
If not, would it be reasonable to add a hook in the ABCL source so I don't need to patch it to do the optimization.
A quick grep of the source does not seem to indicate any hooks into the controlling the behavior/implementation PRECOMPILER::PRECOMPILE-FUNCTION-CALL, supporting your finding of no currently reasonable mechanism for a user to manipulate.
Therefore, I agree that creating a hook mechanism would be the reasonable way forward.
I worry a little about compiler speed decrease if we do a naive hook implementation (i.e. something that MAPs APPLY). Would it be more reasonable to memoize possible values of PRECOMPILE-FUNCTION-CALL, providing an API to switch at runtime?
Since PRECOMPILER isn't a documented ABCL package, it would be best to define the API in PRECOMPILER, but have hooks that inserts the public API in the SYSTEM package via the appropriate import and export of the symbols.
Cool work! What sort of efficiency gains do you expect for JSS here?
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
I was thinking the same thing. IIRC there's even some operator to emit your own assembly from the compiler macro. Il 27 ott 2016 11:01, "Erik Huelsmann" <ehuels@gmail.com> ha scritto:
Hi,
Did you look at define-compiler-macro as a possible hook mechanism?
Regards,
Erik
On Oct 27, 2016 09:28, "Mark Evenson" <evenson@panix.com> wrote:
On 2016/10/27 05:18, Alan Ruttenberg wrote:
I got curious about how I might generate better code for JSS.
[…]
The question is: Is there a more elegant way to do this, or a hook already built that I could use instead of redefining precompile-function-call
If not, would it be reasonable to add a hook in the ABCL source so I don't need to patch it to do the optimization.
A quick grep of the source does not seem to indicate any hooks into the controlling the behavior/implementation PRECOMPILER::PRECOMPILE-FUNCTION-CALL, supporting your finding of no currently reasonable mechanism for a user to manipulate.
Therefore, I agree that creating a hook mechanism would be the reasonable way forward.
I worry a little about compiler speed decrease if we do a naive hook implementation (i.e. something that MAPs APPLY). Would it be more reasonable to memoize possible values of PRECOMPILE-FUNCTION-CALL, providing an API to switch at runtime?
Since PRECOMPILER isn't a documented ABCL package, it would be best to define the API in PRECOMPILER, but have hooks that inserts the public API in the SYSTEM package via the appropriate import and export of the symbols.
Cool work! What sort of efficiency gains do you expect for JSS here?
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
I could use that instead of the macro but would still need the hook, which enabled getting rid of the lambda in the function position. That's beta-reduction, yes? Alan On Thursday, October 27, 2016, Erik Huelsmann <ehuels@gmail.com> wrote:
Hi,
Did you look at define-compiler-macro as a possible hook mechanism?
Regards,
Erik
On Oct 27, 2016 09:28, "Mark Evenson" <evenson@panix.com <javascript:_e(%7B%7D,'cvml','evenson@panix.com');>> wrote:
On 2016/10/27 05:18, Alan Ruttenberg wrote:
I got curious about how I might generate better code for JSS.
[…]
The question is: Is there a more elegant way to do this, or a hook already built that I could use instead of redefining precompile-function-call
If not, would it be reasonable to add a hook in the ABCL source so I don't need to patch it to do the optimization.
A quick grep of the source does not seem to indicate any hooks into the controlling the behavior/implementation PRECOMPILER::PRECOMPILE-FUNCTION-CALL, supporting your finding of no currently reasonable mechanism for a user to manipulate.
Therefore, I agree that creating a hook mechanism would be the reasonable way forward.
I worry a little about compiler speed decrease if we do a naive hook implementation (i.e. something that MAPs APPLY). Would it be more reasonable to memoize possible values of PRECOMPILE-FUNCTION-CALL, providing an API to switch at runtime?
Since PRECOMPILER isn't a documented ABCL package, it would be best to define the API in PRECOMPILER, but have hooks that inserts the public API in the SYSTEM package via the appropriate import and export of the symbols.
Cool work! What sort of efficiency gains do you expect for JSS here?
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On Thu, Oct 27, 2016 at 3:27 AM, Mark Evenson <evenson@panix.com> wrote:
Cool work! What sort of efficiency gains do you expect for JSS here?
Not sure yet. The thing that actually drove it was that I didn't like seeing the redundancy in the stack traces when I was debugging. At first I played around with trimming them out of the stack trace but that ran in an issue - I couldn't figure out exactly how the indexing of stack frames was coordinated between emacs and lisp. Then it occurred to me that they didn't need to be there in the first place. So it's more of an aesthetic thing for now. Alan
So I made a pass at doing this cleanly. Please check out the changes at https://github.com/alanruttenberg/abcl/commit/4e824ddd8f054eee4d943d0fdbbfcd... (precompiler.lisp, optimize-java-call.lisp) https://github.com/alanruttenberg/abcl/commit/b46c704b1460ccd75035c3f590cdc5... (test-optimize-java-call.lisp) (second one has commit message with documentation) It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests. For Mark, the test- file has this info but here it is again - 10000 calls of (#"compile" 'regex.Pattern ".*") With optimization: (INVOKE-RESTARGS-MACRO "compile" (QUOTE REGEX.PATTERN) (LIST ".*") NIL T) Without optimization: ((LAMBDA (#:G85648 &REST #:G85649) (INVOKE-RESTARGS "compile" #:G85648 #:G85649 NIL)) (QUOTE REGEX.PATTERN) ".*") JUST-LOOP 0.0 seconds real time 0 cons cells OPTIMIZED-JSS 0.011 seconds real time 0 cons cells UNOPTIMIZED-JSS 0.325 seconds real time 800156 cons cells On Thu, Oct 27, 2016 at 12:46 PM, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
On Thu, Oct 27, 2016 at 3:27 AM, Mark Evenson <evenson@panix.com> wrote:
Cool work! What sort of efficiency gains do you expect for JSS here?
Not sure yet. The thing that actually drove it was that I didn't like seeing the redundancy in the stack traces when I was debugging. At first I played around with trimming them out of the stack trace but that ran in an issue - I couldn't figure out exactly how the indexing of stack frames was coordinated between emacs and lisp. Then it occurred to me that they didn't need to be there in the first place. So it's more of an aesthetic thing for now.
Alan
Looks great!! On 4 November 2016 at 05:39, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
So I made a pass at doing this cleanly. Please check out the changes at
https://github.com/alanruttenberg/abcl/commit/ 4e824ddd8f054eee4d943d0fdbbfcd83c17b26d7 (precompiler.lisp, optimize- java-call.lisp) https://github.com/alanruttenberg/abcl/commit/ b46c704b1460ccd75035c3f590cdc52fff29886e (test-optimize-java-call.lisp)
(second one has commit message with documentation)
It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests.
For Mark, the test- file has this info but here it is again - 10000 calls of (#"compile" 'regex.Pattern ".*")
With optimization: (INVOKE-RESTARGS-MACRO "compile" (QUOTE REGEX.PATTERN) (LIST ".*") NIL T) Without optimization: ((LAMBDA (#:G85648 &REST #:G85649) (INVOKE-RESTARGS "compile" #:G85648 #:G85649 NIL)) (QUOTE REGEX.PATTERN) ".*")
JUST-LOOP 0.0 seconds real time 0 cons cells
OPTIMIZED-JSS 0.011 seconds real time 0 cons cells
UNOPTIMIZED-JSS 0.325 seconds real time 800156 cons cells
On Thu, Oct 27, 2016 at 12:46 PM, Alan Ruttenberg < alanruttenberg@gmail.com> wrote:
On Thu, Oct 27, 2016 at 3:27 AM, Mark Evenson <evenson@panix.com> wrote:
Cool work! What sort of efficiency gains do you expect for JSS here?
Not sure yet. The thing that actually drove it was that I didn't like seeing the redundancy in the stack traces when I was debugging. At first I played around with trimming them out of the stack trace but that ran in an issue - I couldn't figure out exactly how the indexing of stack frames was coordinated between emacs and lisp. Then it occurred to me that they didn't need to be there in the first place. So it's more of an aesthetic thing for now.
Alan
On 2016/11/4 05:39, Alan Ruttenberg wrote:
So I made a pass at doing this cleanly. Please check out the changes at
https://github.com/alanruttenberg/abcl/commit/4e824ddd8f054eee4d943d0fdbbfcd... (precompiler.lisp, optimize-java-call.lisp) https://github.com/alanruttenberg/abcl/commit/b46c704b1460ccd75035c3f590cdc5... (test-optimize-java-call.lisp)
(second one has commit message with documentation)
It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests.
Look into a mirror: there's one of the folks "who have worked on the compiler". Seriously, we don't really have such experience at the moment. Alessio and Erik are probably the most knowledgeable. Unfortunately there are two problems with the current state Alan's work: 1) With *INHIBIT-JSS-OPTIMIZATION* as nil, JSS won't compile any reference to a Java class that it not present in the classpath at compile time. Previously this wasn't the behavior. 2) There are a slew of additional failures in the ANSI-TEST LAMBDA section, which indicates that we need to work through some wrinkles in the behavior of the patch. LAMBDA.1, LAMBDA.2, LAMBDA.3, LAMBDA.4, LAMBDA.5, LAMBDA.6, LAMBDA.7, LAMBDA.8, LAMBDA.9, LAMBDA.10, LAMBDA.21, LAMBDA.22, LAMBDA.54, LAMBDA.57, LAMBDA.63, LAMBDA.64 Obviously we at least need to analyze/fix the problems with the LAMBDA form revealed by the ANSI-TEST suite, and I would really like to somehow have the previous behavior of JSS to somehow lazily reference Java callsites, but that is probably incompatible with optimization. I am tracking this issue as [420][] [420]: http://abcl.org/trac/ticket/420 The [patch that I tested with][1] contains to small changes to Alan's work: 1. Export JSS:*INHIBIT-JSS-OPTIMIZATION*; have it default to T 2. Revbump the JSS ASDF definition. [1]: http://abcl.org/trac/attachment/ticket/420/beta-reduce-20161117a.diff -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Hi, by 1) won't compile you mean that compilation fails? The compiler macro could simply revert to the unoptimized behavior in that case instead. On 17 November 2016 at 12:00, Mark Evenson <evenson@panix.com> wrote:
On 2016/11/4 05:39, Alan Ruttenberg wrote:
So I made a pass at doing this cleanly. Please check out the changes at
https://github.com/alanruttenberg/abcl/commit/ 4e824ddd8f054eee4d943d0fdbbfcd83c17b26d7 (precompiler.lisp, optimize-java-call.lisp) https://github.com/alanruttenberg/abcl/commit/ b46c704b1460ccd75035c3f590cdc52fff29886e (test-optimize-java-call.lisp)
(second one has commit message with documentation)
It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests.
Look into a mirror: there's one of the folks "who have worked on the compiler". Seriously, we don't really have such experience at the moment. Alessio and Erik are probably the most knowledgeable.
Unfortunately there are two problems with the current state Alan's work:
1) With *INHIBIT-JSS-OPTIMIZATION* as nil, JSS won't compile any reference to a Java class that it not present in the classpath at compile time. Previously this wasn't the behavior.
2) There are a slew of additional failures in the ANSI-TEST LAMBDA section, which indicates that we need to work through some wrinkles in the behavior of the patch.
LAMBDA.1, LAMBDA.2, LAMBDA.3, LAMBDA.4, LAMBDA.5, LAMBDA.6, LAMBDA.7, LAMBDA.8, LAMBDA.9, LAMBDA.10, LAMBDA.21, LAMBDA.22, LAMBDA.54, LAMBDA.57, LAMBDA.63, LAMBDA.64
Obviously we at least need to analyze/fix the problems with the LAMBDA form revealed by the ANSI-TEST suite, and I would really like to somehow have the previous behavior of JSS to somehow lazily reference Java callsites, but that is probably incompatible with optimization.
I am tracking this issue as [420][] [420]: http://abcl.org/trac/ticket/420
The [patch that I tested with][1] contains to small changes to Alan's work:
1. Export JSS:*INHIBIT-JSS-OPTIMIZATION*; have it default to T 2. Revbump the JSS ASDF definition.
[1]: http://abcl.org/trac/attachment/ticket/420/beta-reduce-20161117a.diff
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
Yup, compile time error. Unintended - thanks for catching that! I've fixed that in my source. Now if it can't find the class at compile time it will defer class lookup to runtime and not warn. I'll look into the test failure issues. Best, Alan On Thu, Nov 17, 2016 at 6:13 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
Hi, by 1) won't compile you mean that compilation fails? The compiler macro could simply revert to the unoptimized behavior in that case instead.
On 17 November 2016 at 12:00, Mark Evenson <evenson@panix.com> wrote:
On 2016/11/4 05:39, Alan Ruttenberg wrote:
So I made a pass at doing this cleanly. Please check out the changes at
https://github.com/alanruttenberg/abcl/commit/4e824ddd8f054e ee4d943d0fdbbfcd83c17b26d7 (precompiler.lisp, optimize-java-call.lisp) https://github.com/alanruttenberg/abcl/commit/b46c704b1460cc d75035c3f590cdc52fff29886e (test-optimize-java-call.lisp)
(second one has commit message with documentation)
It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests.
Look into a mirror: there's one of the folks "who have worked on the compiler". Seriously, we don't really have such experience at the moment. Alessio and Erik are probably the most knowledgeable.
Unfortunately there are two problems with the current state Alan's work:
1) With *INHIBIT-JSS-OPTIMIZATION* as nil, JSS won't compile any reference to a Java class that it not present in the classpath at compile time. Previously this wasn't the behavior.
2) There are a slew of additional failures in the ANSI-TEST LAMBDA section, which indicates that we need to work through some wrinkles in the behavior of the patch.
LAMBDA.1, LAMBDA.2, LAMBDA.3, LAMBDA.4, LAMBDA.5, LAMBDA.6, LAMBDA.7, LAMBDA.8, LAMBDA.9, LAMBDA.10, LAMBDA.21, LAMBDA.22, LAMBDA.54, LAMBDA.57, LAMBDA.63, LAMBDA.64
Obviously we at least need to analyze/fix the problems with the LAMBDA form revealed by the ANSI-TEST suite, and I would really like to somehow have the previous behavior of JSS to somehow lazily reference Java callsites, but that is probably incompatible with optimization.
I am tracking this issue as [420][] [420]: http://abcl.org/trac/ticket/420
The [patch that I tested with][1] contains to small changes to Alan's work:
1. Export JSS:*INHIBIT-JSS-OPTIMIZATION*; have it default to T 2. Revbump the JSS ASDF definition.
[1]: http://abcl.org/trac/attachment/ticket/420/beta-reduce- 20161117a.diff
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
I updated the pull request and the trac ticket to address the issues Mark found. Alan On Thu, Nov 17, 2016 at 9:06 AM, Alan Ruttenberg <alanruttenberg@gmail.com> wrote:
Yup, compile time error. Unintended - thanks for catching that! I've fixed that in my source. Now if it can't find the class at compile time it will defer class lookup to runtime and not warn. I'll look into the test failure issues. Best, Alan
On Thu, Nov 17, 2016 at 6:13 AM, Alessio Stalla <alessiostalla@gmail.com> wrote:
Hi, by 1) won't compile you mean that compilation fails? The compiler macro could simply revert to the unoptimized behavior in that case instead.
On 17 November 2016 at 12:00, Mark Evenson <evenson@panix.com> wrote:
On 2016/11/4 05:39, Alan Ruttenberg wrote:
So I made a pass at doing this cleanly. Please check out the changes at
https://github.com/alanruttenberg/abcl/commit/4e824ddd8f054e ee4d943d0fdbbfcd83c17b26d7 (precompiler.lisp, optimize-java-call.lisp) https://github.com/alanruttenberg/abcl/commit/b46c704b1460cc d75035c3f590cdc52fff29886e (test-optimize-java-call.lisp)
(second one has commit message with documentation)
It would be good to get one of the folks who have worked on the compiler to see if I've done anything wrong. ABCL builds fine with this and I didn't see any changes in the abcl.lisp tests.
Look into a mirror: there's one of the folks "who have worked on the compiler". Seriously, we don't really have such experience at the moment. Alessio and Erik are probably the most knowledgeable.
Unfortunately there are two problems with the current state Alan's work:
1) With *INHIBIT-JSS-OPTIMIZATION* as nil, JSS won't compile any reference to a Java class that it not present in the classpath at compile time. Previously this wasn't the behavior.
2) There are a slew of additional failures in the ANSI-TEST LAMBDA section, which indicates that we need to work through some wrinkles in the behavior of the patch.
LAMBDA.1, LAMBDA.2, LAMBDA.3, LAMBDA.4, LAMBDA.5, LAMBDA.6, LAMBDA.7, LAMBDA.8, LAMBDA.9, LAMBDA.10, LAMBDA.21, LAMBDA.22, LAMBDA.54, LAMBDA.57, LAMBDA.63, LAMBDA.64
Obviously we at least need to analyze/fix the problems with the LAMBDA form revealed by the ANSI-TEST suite, and I would really like to somehow have the previous behavior of JSS to somehow lazily reference Java callsites, but that is probably incompatible with optimization.
I am tracking this issue as [420][] [420]: http://abcl.org/trac/ticket/420
The [patch that I tested with][1] contains to small changes to Alan's work:
1. Export JSS:*INHIBIT-JSS-OPTIMIZATION*; have it default to T 2. Revbump the JSS ASDF definition.
[1]: http://abcl.org/trac/attachment/ticket/420/beta-reduce-20161 117a.diff
-- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
On 2016/11/17 17:30, Alan Ruttenberg wrote:
I updated the pull request and the trac ticket to address the issues Mark found.
A good day for the Bear… Works! Patch [accepted in trunk][420]. [420]: http://abcl.org/trac/ticket/420 -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."
participants (4)
-
Alan Ruttenberg -
Alessio Stalla -
Erik Huelsmann -
Mark Evenson