[cffi-devel] %expand-type-to-foreign-dyn vs *runtime-translator-form*
I was looking at the new type translator expansion interface and noticed that one would experience different behavior depending on method definitions. I'll use expand-to-foreign-dyn as an example: If a translation method is available [for some defn thereof], as found by compute-applicable-methods, but that method answered *runtime-translator-form*, the final result of the expander would be something like `(multiple-value-bind (,var ,param) (translate-type-to-foreign ,value ,type) (unwind-protect (progn ,@body) (free-type-translated-object ,var ,type ,param))) Otherwise, both implementations of expand-type-to-foreign-dyn would short-circuit the generic function call and return `(let ((,var ,(expand-type-to-foreign value type))) ,@body) Though only if by similar heuristic they found that a one-way expander was available. I've rearranged things here to combine the cases, eliminate the *runtime-translator-form* exportation, and make the code simpler. Currently, both cases are collapsed into the former expansion. However, this breaks some misc-types.expand tests, as it breaks this guarantee listed in that file: Ensure that macroexpansion-time translators are called where this is guaranteed (defcfun, defcvar, foreign-funcall and defcallback) Now, expand-to-foreign is undocumented (:), so I am not sure what its semantics are supposed to be anyway. (In particular, I don't see whether the expand-type-* specializations for foreign-typedef are following the actual-type chain.) If it is precisely like a compiler macro, this can easily be fixed by changing translate-type-to-foreign to expand-type-to-foreign in the current expansion, and things will work nicely. However, I must be able to call free-type-translated-object on the results of the forward-expanded form for this to work. Can I rely on this guarantee, so that I can in return reoffer the aforementioned guarantee? -- Stephen Compall http://scompall.nocandysw.com/blog
On Tue, 2006-02-28 at 23:58 -0600, Stephen Compall wrote:
If a translation method is available [for some defn thereof], as found by compute-applicable-methods, but that method answered *runtime-translator-form*, the final result of the expander would be something like
`(multiple-value-bind (,var ,param) (translate-type-to-foreign ,value ,type) (unwind-protect (progn ,@body) (free-type-translated-object ,var ,type ,param)))
Otherwise, both implementations of expand-type-to-foreign-dyn would short-circuit the generic function call and return
`(let ((,var ,(expand-type-to-foreign value type))) ,@body)
I fixed the macroexpansion at http://paste.lisp.org/display/17379 and added a couple tests demonstrating this behavior to misc-types.lisp in http://scompall.nocandysw.com/cffi/foreign-dyn-behavior-tests.darcs.patch Wed Mar 1 13:41:22 CST 2006 Stephen Compall <scompall@nocandysw.com> * demonstrate differing behavior between foreign-dyn expansions - change expand-type-to-foreign-dyn for foreign-typedef to not short-circuit expand-type-to-foreign when falling back (see paste 17379) - add tests showing that you can change the foreign-dyn expansion semantics by providing an expand-to-foreign method that falls back -- Stephen Compall http://scompall.nocandysw.com/blog
On 2006-mar-01, at 19:53, Stephen Compall wrote:
Wed Mar 1 13:41:22 CST 2006 Stephen Compall <scompall@nocandysw.com> * demonstrate differing behavior between foreign-dyn expansions
;; free-translated-object must not be called when there is an etf, but ;; they answer *runtime-translator-form*
I think this isn't a good idea. Here's an example: (defmethod expand-to-foreign-dyn (value var body (name (eql :string))) (if <string's size is appropriate for stack allocation> `(with-foreign-string (,var ,value) ,@body) *runtime-translator-form*)) Surely we want free-translated-object to be called in this case. Or did I miss something? -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Mon, 2006-03-13 at 17:11 +0000, Luís Oliveira wrote:
I think this isn't a good idea. Here's an example:
(defmethod expand-to-foreign-dyn (value var body (name (eql :string))) (if <string's size is appropriate for stack allocation> `(with-foreign-string (,var ,value) ,@body) *runtime-translator-form*))
Surely we want free-translated-object to be called in this case.
It will be. The problem appears in the current rules for providing a fallback when expand-to-foreign-dyn is not provided. This is exemplified in the new tests: one answers T and the other answers NIL, when both should answer T or NIL. According to node "Optimizing Type Translators", several forms are guaranteed to use `expand-type-to-foreign' even when they want the semantics of foreign-dyn expansion, when such an explicit expansion is unavailable. I ran into this problem while eliminating *runtime-translator-form* from the interface. I have a patch that implements option 1; it can be trivially changed to implement either other option, however. 1) Remove the guarantee; fall back on the translate-* gfs when no foreign-dyn expansion is available. This is currently implemented by answering *runtime-translator-form* from a foreign-dyn expansion. 2) Specify that the form answered by expand-type-to-foreign must have the exact same semantics as those evaluated by calling translate-type-to-foreign, thereby meeting a congruence with CL compiler-macros. This means that it returns an ALLOC-PARAM as its second value, which is then passed to the gf free-type-translated-object in an unwind-protect. 3) Specify that the form answered by expand-type-to-foreign, when evaluated, answers a value that does not need to be freed. This is currently implemented by not providing a foreign-dyn expansion method, in the case of foreign-typedefs anyway. I prefer 1 to 2 because the point of the expand-* interface is to avoid gf calls and stuff. If you let users expect optimization in all cases without providing foreign-dyn expansions, they may not expect that the result may still contain an unwind-protect whose cleanup form calls a gf. I would rather say that the user can only expect perfect optimization by providing explicit to-foreign, from-foreign, and to-foreign-dyn expansions. This does not preclude providing a function to trivialize overriding the default behavior in favor of option 3, if the type-defining user so chooses. (defun simple-foreign-dyn-expansion (value var body typename) "Answer a form to be answered from `expand-to-foreign-dyn', where VALUE, VAR, BODY, and TYPENAME are the respective args to the aforementioned GF. The result will not be freed." `(let ((,var ,(expand-type-to-foreign value (parse-type typename)))) ,@body)) -- Stephen Compall http://scompall.nocandysw.com/blog
On 2006-mar-13, at 18:48, Stephen Compall wrote:
Surely we want free-translated-object to be called in this case.
It will be. The problem appears in the current rules for providing a fallback when expand-to-foreign-dyn is not provided. This is exemplified in the new tests: one answers T and the other answers NIL, when both should answer T or NIL.
Judging from the comment in misc-types.expand.6 I thought you did want it to return nil.
According to node "Optimizing Type Translators", several forms are guaranteed to use `expand-type-to-foreign' even when they want the semantics of foreign-dyn expansion, when such an explicit expansion is unavailable.
I think this is useful.
3) Specify that the form answered by expand-type-to-foreign, when evaluated, answers a value that does not need to be freed. This is currently implemented by not providing a foreign-dyn expansion method, in the case of foreign-typedefs anyway.
This was what I had in mind.
This does not preclude providing a function to trivialize overriding the default behavior in favor of option 3, if the type-defining user so chooses.
(defun simple-foreign-dyn-expansion (value var body typename) "Answer a form to be answered from `expand-to-foreign-dyn', where VALUE, VAR, BODY, and TYPENAME are the respective args to the aforementioned GF. The result will not be freed." `(let ((,var ,(expand-type-to-foreign value (parse-type typename)))) ,@body))
I guess that picking option 1 or 3 is a matter of picking the best default behavior. Isn't option 3 the most common case? -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Mon, 2006-03-13 at 19:28 +0000, Luís Oliveira wrote:
I guess that picking option 1 or 3 is a matter of picking the best default behavior. Isn't option 3 the most common case?
Yes, but it would require some additional definition from those who wanted to rely on translate-*. It would also require at least one of the tests misc-types.expand.{5,6} to match NIL instead of T. Consider that I have defined a string type as in your previous message. I have defined translate-to-foreign and free-translated-object on it so I can pass strings as arguments. If I do not define an expand-to-foreign-dyn as well for it: (eval-when (:compile-toplevel :load-toplevel :execute) (defmethod expand-to-foreign-dyn (value var body (type-name (eql ':string))) (freeing-foreign-dyn-expansion value var body type-name))) I risk something in different cases: * Where tests 5 and 6 are NIL: The string will not be freed. * Where test 5 is T, but 6 is NIL: If I define an expand-to-foreign method, even if it ever returns *runtime-translator-form*, the value will not be freed. This is the current behavior. * Where test 5 is NIL, but 6 is T: This is just confusing, but: the semantic requirement from option 2 is implicitly imposed, and the value will not be freed *unless* I define an expand-to-foreign method. * Where both tests are T: This is essentially option 1. -- Stephen Compall http://scompall.nocandysw.com/blog
On Tue, 2006-03-14 at 12:01 -0600, Stephen Compall wrote:
* Where both tests are T: This is essentially option 1.
This is currently available at http://scompall.nocandysw.com/farewell-rtt-1.darcs.patch , including the previous changes to test/misc-types.lisp. Wed Mar 15 12:48:04 CST 2006 Stephen Compall <scompall@nocandysw.com> * remove *runtime-translator-form* from the effective interface - Add default methods to expand-to-foreign-dyn, expand-to-foreign, and expand-from-foreign, and always use them as fallback, as the "fallback" for specializations that want to fail should be exactly the same as that when there are no specializations. - Use *fto-called* instead of .fto-called. in misc-types.lisp. - misc-types.expand.6 test: expect T instead of NIL. -- Stephen Compall http://scompall.nocandysw.com/blog
Stephen Compall <s11@member.fsf.org> writes:
Wed Mar 15 12:48:04 CST 2006 Stephen Compall <scompall@nocandysw.com> * remove *runtime-translator-form* from the effective interface
Ok, so this looks good. I think I like using (call-next-method) instead of the *runtime-translator-form* special. This patch fails misc-types.expand.{1,2} though since these expect expand-to-foreign-dyn to use expand-to-foreign. As I mentioned before, I kind of like that behavior. Can you convince me that it's a bad idea? :-) -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Thu, 2006-03-16 at 12:36 +0000, Luís Oliveira wrote:
This patch fails misc-types.expand.{1,2} though since these expect expand-to-foreign-dyn to use expand-to-foreign. As I mentioned before, I kind of like that behavior. Can you convince me that it's a bad idea? :-)
Well, translate-type-to-foreign is currently used as the fallback for expand-type-to-foreign, which is currently used as the fallback for expand-type-to-foreign-dyn (when an e-t-f method is defined, anyway, as demonstrated by tests 5 and 6). However, expand-type-to-foreign has different semantics from translate-type-to-foreign. I highlighted this semi-issue in the most recent patch by wrapping it in VALUES; after all, the second value, ALLOC-PARAM, is discarded in test 6. Since this behavior is always fine for cases where foreign-dyn expansion isn't even tried, like setting C variables or translating callback return values, but not for when you expect the normal free behavior from translate-type-to-foreign in something like test 6, I think it would be more intuitive for it to fall back directly to the translate-* functions. This is option 1 as described earlier, and implemented in the latest patch. The semantics for forms answered by expand-type-to-foreign and calls to translate-type-to-foreign could also be unified, so that ettf forms can also answer an ALLOC-PARAM and expect to be freed in foreign-dyn expansion. This is option 2, and less than ideal for some reason. One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3, the current behavior as demonstrated by tests 5 and 6. As this is just as arduous as requiring an explicit call to simple-foreign-dyn-expansion (the function presented earlier) or something if you want e-t-f semantics for *runtime-translator-form*, but less predictable in its behavior, I would rather it be changed. I should note that regardless, *runtime-translator-form* can still be eliminated in much the same way for any of the above. Here goes: regardless, *runtime-translator-form* can still be eliminated in much the same way for any of the above. -- Stephen Compall http://scompall.nocandysw.com/blog
Stephen Compall <s11@member.fsf.org> writes:
One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3 [...]
I think *runtime-translator-form* should be a translate-type-to-foreign call when expand-type-to-foreign is called directly and a m-v-b/t-t-t-f/unwind-protect/f-t-t-o thingie when it's called through expand-type-to-foreign-dyn. -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Tue, 2006-03-21 at 12:57 +0000, Luís Oliveira wrote:
Stephen Compall <s11@member.fsf.org> writes:
One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3 [...]
I think *runtime-translator-form* should be a translate-type-to-foreign call when expand-type-to-foreign is called directly and a m-v-b/t-t-t-f/unwind-protect/f-t-t-o thingie when it's called through expand-type-to-foreign-dyn.
This is available at http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch , including documentation updates. Latest of "Option 1" at http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch , also including documentation updates. Both include the same modification to %expand-type-to-foreign; see the docs in "opt3" for more details. -- Stephen Compall http://scompall.nocandysw.com/blog
Stephen Compall <s11@member.fsf.org> writes:
http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch
Many thanks for producing patches for both options! I'm inclined towards option 1 and IIUC you are inclined towards option 3, so... James, which one do you prefer? :-) -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Fri, 2006-03-31 at 17:11 +0100, Luís Oliveira wrote:
Stephen Compall <s11@member.fsf.org> writes:
http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch
Many thanks for producing patches for both options! I'm inclined towards option 1 and IIUC you are inclined towards option 3
Great! I actually prefer option 1 as well. -- Stephen Compall http://scompall.nocandysw.com/blog
On 2006-mar-31, at 19:28, Stephen Compall wrote:
Great! I actually prefer option 1 as well.
Ugh... sorry about that. I prefer option 3 and you prefer option 1. :-) -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
participants (2)
-
Luís Oliveira -
Stephen Compall