Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
Check out Bike's compiler-macro library, specifically the `form-type` function: https://github.com/Bike/compiler-macro
On Mon, Nov 6, 2017 at 3:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
On 7 November 2017 at 02:54, Steve Losh steve@stevelosh.com wrote:
Check out Bike's compiler-macro library, specifically the `form-type` function: https://github.com/Bike/compiler-macro
This is exactly what I was looking for. Thanks a ton!
Chaitanya
On Mon, Nov 6, 2017 at 3:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
A compiler-macro, just like a regular macro, receives an &environment argument which is supposed to contain (amongst other things) the lexical and global properties of bindings in the environment in which the macro is expanded. In the original language definition proposed by CLtL there was an environment access interface with functions like variablle-information which could return (for instance) what is known about the type of A in the lexical environment in which FOO is expanded.
Unfortunately, the environment access system in CLtL was underspecified, there were no existing implementations, and no one was sure an implementation was even possible without imposing large efficiency hits at compile time. So we of X3J13 removed it from the standard.
Duane Rettig of Franz later implemented a portable open-source sane environment module inspired by the original CLtL specification. This can be used to write a portable CL code walker of any kind, but it won't do you any good unless the code walker that is your favorite implementation's compiler happens to use it. But fortunately, several implementations provide documented environment access interfaces built into their compilers. Unfortunately, these interfaces are not defined by the standard, so their use generally makes code nonportable.
BTW, a plain INTEGER type declaration is often not as effective as you'd expect because INTEGER type is the union of FIXNUM and BIGNUM. Arithmetic on the former can generally be inlined in a few machine instructions (particularly if the integer range can be narrowed further) but if any arguments or the result could be bignums, further runtime type tests and/or callouts will be generated.
On Mon, Nov 6, 2017 at 12:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
There is a portable (*) re-implementation of the Environment API (including the re-normalization of Franz return values switch) in CLAST.
Cheers --
MA
On Nov 7, 2017, at 06:00 , Steve Haflich shaflich@gmail.com wrote:
A compiler-macro, just like a regular macro, receives an &environment argument which is supposed to contain (amongst other things) the lexical and global properties of bindings in the environment in which the macro is expanded. In the original language definition proposed by CLtL there was an environment access interface with functions like variablle-information which could return (for instance) what is known about the type of A in the lexical environment in which FOO is expanded.
Unfortunately, the environment access system in CLtL was underspecified, there were no existing implementations, and no one was sure an implementation was even possible without imposing large efficiency hits at compile time. So we of X3J13 removed it from the standard.
Duane Rettig of Franz later implemented a portable open-source sane environment module inspired by the original CLtL specification. This can be used to write a portable CL code walker of any kind, but it won't do you any good unless the code walker that is your favorite implementation's compiler happens to use it. But fortunately, several implementations provide documented environment access interfaces built into their compilers. Unfortunately, these interfaces are not defined by the standard, so their use generally makes code nonportable.
BTW, a plain INTEGER type declaration is often not as effective as you'd expect because INTEGER type is the union of FIXNUM and BIGNUM. Arithmetic on the former can generally be inlined in a few machine instructions (particularly if the integer range can be narrowed further) but if any arguments or the result could be bignums, further runtime type tests and/or callouts will be generated.
On Mon, Nov 6, 2017 at 12:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
On 7 November 2017 at 13:56, Antoniotti Marco antoniotti.marco@disco.unimib.it wrote:
There is a portable (*) re-implementation of the Environment API (including the re-normalization of Franz return values switch) in CLAST.
Thanks, CLAST looks like a very useful library.
Chaitanya
Cheers
MA
On Nov 7, 2017, at 06:00 , Steve Haflich shaflich@gmail.com wrote:
A compiler-macro, just like a regular macro, receives an &environment argument which is supposed to contain (amongst other things) the lexical and global properties of bindings in the environment in which the macro is expanded. In the original language definition proposed by CLtL there was an environment access interface with functions like variablle-information which could return (for instance) what is known about the type of A in the lexical environment in which FOO is expanded.
Unfortunately, the environment access system in CLtL was underspecified, there were no existing implementations, and no one was sure an implementation was even possible without imposing large efficiency hits at compile time. So we of X3J13 removed it from the standard.
Duane Rettig of Franz later implemented a portable open-source sane environment module inspired by the original CLtL specification. This can be used to write a portable CL code walker of any kind, but it won't do you any good unless the code walker that is your favorite implementation's compiler happens to use it. But fortunately, several implementations provide documented environment access interfaces built into their compilers. Unfortunately, these interfaces are not defined by the standard, so their use generally makes code nonportable.
BTW, a plain INTEGER type declaration is often not as effective as you'd expect because INTEGER type is the union of FIXNUM and BIGNUM. Arithmetic on the former can generally be inlined in a few machine instructions (particularly if the integer range can be narrowed further) but if any arguments or the result could be bignums, further runtime type tests and/or callouts will be generated.
On Mon, Nov 6, 2017 at 12:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
On 7 November 2017 at 10:30, Steve Haflich shaflich@gmail.com wrote:
A compiler-macro, just like a regular macro, receives an &environment argument which is supposed to contain (amongst other things) the lexical and global properties of bindings in the environment in which the macro is expanded. In the original language definition proposed by CLtL there was an environment access interface with functions like variablle-information which could return (for instance) what is known about the type of A in the lexical environment in which FOO is expanded.
Unfortunately, the environment access system in CLtL was underspecified, there were no existing implementations, and no one was sure an implementation was even possible without imposing large efficiency hits at compile time. So we of X3J13 removed it from the standard.
Duane Rettig of Franz later implemented a portable open-source sane environment module inspired by the original CLtL specification. This can be used to write a portable CL code walker of any kind, but it won't do you any good unless the code walker that is your favorite implementation's compiler happens to use it. But fortunately, several implementations provide documented environment access interfaces built into their compilers. Unfortunately, these interfaces are not defined by the standard, so their use generally makes code nonportable.
Thank you for sharing this piece of CL history. I never knew CLtL had an environment access facility.
BTW, a plain INTEGER type declaration is often not as effective as you'd expect because INTEGER type is the union of FIXNUM and BIGNUM. Arithmetic on the former can generally be inlined in a few machine instructions (particularly if the integer range can be narrowed further) but if any arguments or the result could be bignums, further runtime type tests and/or callouts will be generated.
Yes I realized this too recently.. the performance gains with fixnum are pretty good!
Chaitanya
On Mon, Nov 6, 2017 at 12:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
So thanks to the replies on this I now know that most of the popular Lisps do support inspecting the environment to figure out declared types.
But what about inferred types (e.g. in CMUCL, SBCL)? Do these Lisps provide a way to know the inferred type of a variable if no declaration was made explicitly?
Chaitanya
On 7 November 2017 at 02:13, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
CMUCL and SBCL have the Environment API somewhat available (*)
You can see whether they store the inferred values in there.
Marco
(*) Again, you can have a look at CLAST to see how to access it; I know: it is a shameless plug.
On Nov 9, 2017, at 09:18 , Chaitanya Gupta mail@chaitanyagupta.com wrote:
So thanks to the replies on this I now know that most of the popular Lisps do support inspecting the environment to figure out declared types.
But what about inferred types (e.g. in CMUCL, SBCL)? Do these Lisps provide a way to know the inferred type of a variable if no declaration was made explicitly?
Chaitanya
On 7 November 2017 at 02:13, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
I tried this on SBCL:
(defun bar (x) x)
(define-compiler-macro bar (&whole form x &environment env) (when (symbolp x) (print (multiple-value-list (clast:variable-information x env)))) form)
The form below, with the declaration, prints type information correctly:
(compile nil (lambda () (let ((y 10)) (declare (fixnum y)) (bar y))))
(:LEXICAL T ((TYPE . FIXNUM)))
However, the one below with no declarations, doesn't give any info:
(compile nil (lambda () (let ((y 10)) (bar y)) (let ((y '(1 2 3))) (bar y)) (let ((y 'foo)) (bar y))))
(:LEXICAL T NIL) (:LEXICAL T NIL) (:LEXICAL T NIL)
So I guess that, at least on SBCL, this is not possible.
Chaitanya
On 9 November 2017 at 14:09, Antoniotti Marco antoniotti.marco@disco.unimib.it wrote:
CMUCL and SBCL have the Environment API somewhat available (*)
You can see whether they store the inferred values in there.
Marco
(*) Again, you can have a look at CLAST to see how to access it; I know: it is a shameless plug.
On Nov 9, 2017, at 09:18 , Chaitanya Gupta mail@chaitanyagupta.com wrote:
So thanks to the replies on this I now know that most of the popular Lisps do support inspecting the environment to figure out declared types.
But what about inferred types (e.g. in CMUCL, SBCL)? Do these Lisps provide a way to know the inferred type of a variable if no declaration was made explicitly?
Chaitanya
On 7 November 2017 at 02:13, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
On Nov 9, 2017, at 10:17 , Chaitanya Gupta mail@chaitanyagupta.com wrote:
I tried this on SBCL:
(defun bar (x) x)
(define-compiler-macro bar (&whole form x &environment env) (when (symbolp x) (print (multiple-value-list (clast:variable-information x env)))) form)
The form below, with the declaration, prints type information correctly:
(compile nil (lambda () (let ((y 10)) (declare (fixnum y)) (bar y))))
(:LEXICAL T ((TYPE . FIXNUM)))
However, the one below with no declarations, doesn't give any info:
(compile nil (lambda () (let ((y 10)) (bar y)) (let ((y '(1 2 3))) (bar y)) (let ((y 'foo)) (bar y))))
(:LEXICAL T NIL) (:LEXICAL T NIL) (:LEXICAL T NIL)
So I guess that, at least on SBCL, this is not possible.
Seems plausible. I am not an expert on CMUCL/SBCL type inference, but I’d wager that it does not propagate types as we expect. BTW. LW behaves in the same way (plus: thank you; I discovered a shortcoming of CLAST LW wrapping that I just fixed).
Marco
Chaitanya
On 9 November 2017 at 14:09, Antoniotti Marco antoniotti.marco@disco.unimib.it wrote:
CMUCL and SBCL have the Environment API somewhat available (*)
You can see whether they store the inferred values in there.
Marco
(*) Again, you can have a look at CLAST to see how to access it; I know: it is a shameless plug.
On Nov 9, 2017, at 09:18 , Chaitanya Gupta mail@chaitanyagupta.com wrote:
So thanks to the replies on this I now know that most of the popular Lisps do support inspecting the environment to figure out declared types.
But what about inferred types (e.g. in CMUCL, SBCL)? Do these Lisps provide a way to know the inferred type of a variable if no declaration was made explicitly?
Chaitanya
On 7 November 2017 at 02:13, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
Compiler macros are expanded too early to get inferred type info. For example,
(let ((y 10)) (loop (bar y) (setq y :not-a-fixnum)))
Compiler macros are expanded too early to get inferred type info. For example,
(let ((y 10)) (loop (bar y) (setq y :not-a-fixnum)))
Would it be against the standard to do some type analysis before expanding compiler macros ? Without that, compiler macros can only make optimization decisions based on arguments which are literal values, which is still useful but not in all cases.
On Thu, 09 Nov 2017 12:38:56 +0100, Stelian Ionescu said:
Compiler macros are expanded too early to get inferred type info. For example,
(let ((y 10)) (loop (bar y) (setq y :not-a-fixnum)))
Would it be against the standard to do some type analysis before expanding compiler macros ?
Probably not against the standard, but not much analysis is possible. Afterall, you have to expand macros and compiler macros before their expansions can be analyzed. You can't infer the type of a variable unless you know the types of all of the values assigned to it.
Without that, compiler macros can only make optimizationdecisions based on arguments which are literal values, which is still useful but not in all cases.
Yes, that's really all they can be used for.
On 9 November 2017 at 16:42, Martin Simmons martin@lispworks.com wrote:
Compiler macros are expanded too early to get inferred type info. For example,
(let ((y 10)) (loop (bar y) (setq y :not-a-fixnum)))
Right. That makes sense. So it seems declarations is the way to go if you want to make type based optimizations.
Chaitanya
-- Martin Simmons LispWorks Ltd http://www.lispworks.com/
On Thu, 9 Nov 2017 14:47:53 +0530, Chaitanya Gupta said:
I tried this on SBCL:
(defun bar (x) x)
(define-compiler-macro bar (&whole form x &environment env) (when (symbolp x) (print (multiple-value-list (clast:variable-information x env)))) form)
The form below, with the declaration, prints type information correctly:
(compile nil (lambda () (let ((y 10)) (declare (fixnum y)) (bar y))))
(:LEXICAL T ((TYPE . FIXNUM)))
However, the one below with no declarations, doesn't give any info:
(compile nil (lambda () (let ((y 10)) (bar y)) (let ((y '(1 2 3))) (bar y)) (let ((y 'foo)) (bar y))))
(:LEXICAL T NIL) (:LEXICAL T NIL) (:LEXICAL T NIL)
So I guess that, at least on SBCL, this is not possible.
Chaitanya
On 9 November 2017 at 14:09, Antoniotti Marco antoniotti.marco@disco.unimib.it wrote:
CMUCL and SBCL have the Environment API somewhat available (*)
You can see whether they store the inferred values in there.
Marco
(*) Again, you can have a look at CLAST to see how to access it; I know: it is a shameless plug.
On Nov 9, 2017, at 09:18 , Chaitanya Gupta mail@chaitanyagupta.com wrote:
So thanks to the replies on this I now know that most of the popular Lisps do support inspecting the environment to figure out declared types.
But what about inferred types (e.g. in CMUCL, SBCL)? Do these Lisps provide a way to know the inferred type of a variable if no declaration was made explicitly?
Chaitanya
On 7 November 2017 at 02:13, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
Chaitanya
-- Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY
Please check: http://troncopackage.org
Please note that I am not checking my Spam-box anymore. Please do not forward this email without asking me first (cum grano salis).
On Mon, Nov 6, 2017 at 3:43 PM, Chaitanya Gupta mail@chaitanyagupta.com wrote:
Let's say I have created a function called `FOO`:
(defun foo (x) ...)
To optimize this function, I write a compiler macro. Can I make use of type declarations that users of `FOO` might have made for the argument `X` that is passed in to it?
That is, let's say `FOO` is used like this:
(let ((a (something-that-returns-an-integer))) (declare (integer a)) (foo a))
The compiler macro for `FOO` cannot make any optimizations on the value of `A`, but can it take advantage of the fact that `A` is declared as in `INTEGER` here?
There has been plenty of discussion about how you might access the environment, etc., to get this information, and whether it's portable, etc. Depending on your use case, you could make use of the [THE] special operator at the call site, and you'd be guaranteed to have it available to the compiler macro. That is, you'd now write
(foo (the fixnum a))
That works regardless of what environment introspection you can get, because you see (THE FIXNUM A) when you're expanding the compiler macro, and from that you could check the type, and if the use case is that it's very important to do some optimization at compile time, this might be an acceptable tradeoff.
[THE]: http://www.lispworks.com/documentation/HyperSpec/Body/s_the.htm
-
Antoniotti Marco -
Chaitanya Gupta -
Joshua TAYLOR -
Martin Simmons -
Stelian Ionescu -
Steve Haflich -
Steve Losh