I was looking at the new type translator expansion interface and noticed that one would experience different behavior depending on method definitions. I'll use expand-to-foreign-dyn as an example:
If a translation method is available [for some defn thereof], as found by compute-applicable-methods, but that method answered *runtime-translator-form*, the final result of the expander would be something like
`(multiple-value-bind (,var ,param) (translate-type-to-foreign ,value ,type) (unwind-protect (progn ,@body) (free-type-translated-object ,var ,type ,param)))
Otherwise, both implementations of expand-type-to-foreign-dyn would short-circuit the generic function call and return
`(let ((,var ,(expand-type-to-foreign value type))) ,@body)
Though only if by similar heuristic they found that a one-way expander was available.
I've rearranged things here to combine the cases, eliminate the *runtime-translator-form* exportation, and make the code simpler. Currently, both cases are collapsed into the former expansion. However, this breaks some misc-types.expand tests, as it breaks this guarantee listed in that file: Ensure that macroexpansion-time translators are called where this is guaranteed (defcfun, defcvar, foreign-funcall and defcallback)
Now, expand-to-foreign is undocumented (:), so I am not sure what its semantics are supposed to be anyway. (In particular, I don't see whether the expand-type-* specializations for foreign-typedef are following the actual-type chain.) If it is precisely like a compiler macro, this can easily be fixed by changing translate-type-to-foreign to expand-type-to-foreign in the current expansion, and things will work nicely. However, I must be able to call free-type-translated-object on the results of the forward-expanded form for this to work. Can I rely on this guarantee, so that I can in return reoffer the aforementioned guarantee?
On Tue, 2006-02-28 at 23:58 -0600, Stephen Compall wrote:
If a translation method is available [for some defn thereof], as found by compute-applicable-methods, but that method answered *runtime-translator-form*, the final result of the expander would be something like
`(multiple-value-bind (,var ,param) (translate-type-to-foreign ,value ,type) (unwind-protect (progn ,@body) (free-type-translated-object ,var ,type ,param)))
Otherwise, both implementations of expand-type-to-foreign-dyn would short-circuit the generic function call and return
`(let ((,var ,(expand-type-to-foreign value type))) ,@body)
I fixed the macroexpansion at http://paste.lisp.org/display/17379 and added a couple tests demonstrating this behavior to misc-types.lisp in http://scompall.nocandysw.com/cffi/foreign-dyn-behavior-tests.darcs.patch
Wed Mar 1 13:41:22 CST 2006 Stephen Compall scompall@nocandysw.com * demonstrate differing behavior between foreign-dyn expansions
- change expand-type-to-foreign-dyn for foreign-typedef to not short-circuit expand-type-to-foreign when falling back (see paste 17379) - add tests showing that you can change the foreign-dyn expansion semantics by providing an expand-to-foreign method that falls back
On 2006-mar-01, at 19:53, Stephen Compall wrote:
Wed Mar 1 13:41:22 CST 2006 Stephen Compall scompall@nocandysw.com
- demonstrate differing behavior between foreign-dyn expansions
;; free-translated-object must not be called when there is an etf,
but
;; they answer *runtime-translator-form*
I think this isn't a good idea. Here's an example:
(defmethod expand-to-foreign-dyn (value var body (name (eql :string))) (if <string's size is appropriate for stack allocation> `(with-foreign-string (,var ,value) ,@body) *runtime-translator-form*))
Surely we want free-translated-object to be called in this case.
Or did I miss something?
On Mon, 2006-03-13 at 17:11 +0000, Luís Oliveira wrote:
I think this isn't a good idea. Here's an example:
(defmethod expand-to-foreign-dyn (value var body (name (eql :string))) (if <string's size is appropriate for stack allocation> `(with-foreign-string (,var ,value) ,@body) *runtime-translator-form*))
Surely we want free-translated-object to be called in this case.
It will be. The problem appears in the current rules for providing a fallback when expand-to-foreign-dyn is not provided. This is exemplified in the new tests: one answers T and the other answers NIL, when both should answer T or NIL.
According to node "Optimizing Type Translators", several forms are guaranteed to use `expand-type-to-foreign' even when they want the semantics of foreign-dyn expansion, when such an explicit expansion is unavailable.
I ran into this problem while eliminating *runtime-translator-form* from the interface. I have a patch that implements option 1; it can be trivially changed to implement either other option, however.
1) Remove the guarantee; fall back on the translate-* gfs when no foreign-dyn expansion is available. This is currently implemented by answering *runtime-translator-form* from a foreign-dyn expansion.
2) Specify that the form answered by expand-type-to-foreign must have the exact same semantics as those evaluated by calling translate-type-to-foreign, thereby meeting a congruence with CL compiler-macros. This means that it returns an ALLOC-PARAM as its second value, which is then passed to the gf free-type-translated-object in an unwind-protect.
3) Specify that the form answered by expand-type-to-foreign, when evaluated, answers a value that does not need to be freed. This is currently implemented by not providing a foreign-dyn expansion method, in the case of foreign-typedefs anyway.
I prefer 1 to 2 because the point of the expand-* interface is to avoid gf calls and stuff. If you let users expect optimization in all cases without providing foreign-dyn expansions, they may not expect that the result may still contain an unwind-protect whose cleanup form calls a gf. I would rather say that the user can only expect perfect optimization by providing explicit to-foreign, from-foreign, and to-foreign-dyn expansions.
This does not preclude providing a function to trivialize overriding the default behavior in favor of option 3, if the type-defining user so chooses.
(defun simple-foreign-dyn-expansion (value var body typename) "Answer a form to be answered from `expand-to-foreign-dyn', where VALUE, VAR, BODY, and TYPENAME are the respective args to the aforementioned GF. The result will not be freed." `(let ((,var ,(expand-type-to-foreign value (parse-type typename)))) ,@body))
On 2006-mar-13, at 18:48, Stephen Compall wrote:
Surely we want free-translated-object to be called in this case.
It will be. The problem appears in the current rules for providing a fallback when expand-to-foreign-dyn is not provided. This is exemplified in the new tests: one answers T and the other answers NIL, when both should answer T or NIL.
Judging from the comment in misc-types.expand.6 I thought you did want it to return nil.
According to node "Optimizing Type Translators", several forms are guaranteed to use `expand-type-to-foreign' even when they want the semantics of foreign-dyn expansion, when such an explicit expansion is unavailable.
I think this is useful.
- Specify that the form answered by expand-type-to-foreign, when
evaluated, answers a value that does not need to be freed. This is currently implemented by not providing a foreign-dyn expansion method, in the case of foreign-typedefs anyway.
This was what I had in mind.
This does not preclude providing a function to trivialize overriding the default behavior in favor of option 3, if the type-defining user so chooses.
(defun simple-foreign-dyn-expansion (value var body typename) "Answer a form to be answered from `expand-to-foreign-dyn', where VALUE, VAR, BODY, and TYPENAME are the respective args to the aforementioned GF. The result will not be freed." `(let ((,var ,(expand-type-to-foreign value (parse-type typename)))) ,@body))
I guess that picking option 1 or 3 is a matter of picking the best default behavior. Isn't option 3 the most common case?
On Mon, 2006-03-13 at 19:28 +0000, Luís Oliveira wrote:
I guess that picking option 1 or 3 is a matter of picking the best default behavior. Isn't option 3 the most common case?
Yes, but it would require some additional definition from those who wanted to rely on translate-*. It would also require at least one of the tests misc-types.expand.{5,6} to match NIL instead of T.
Consider that I have defined a string type as in your previous message. I have defined translate-to-foreign and free-translated-object on it so I can pass strings as arguments. If I do not define an expand-to-foreign-dyn as well for it:
(eval-when (:compile-toplevel :load-toplevel :execute) (defmethod expand-to-foreign-dyn (value var body (type-name (eql ':string))) (freeing-foreign-dyn-expansion value var body type-name)))
I risk something in different cases:
* Where tests 5 and 6 are NIL: The string will not be freed.
* Where test 5 is T, but 6 is NIL: If I define an expand-to-foreign method, even if it ever returns *runtime-translator-form*, the value will not be freed. This is the current behavior.
* Where test 5 is NIL, but 6 is T: This is just confusing, but: the semantic requirement from option 2 is implicitly imposed, and the value will not be freed *unless* I define an expand-to-foreign method.
* Where both tests are T: This is essentially option 1.
On Tue, 2006-03-14 at 12:01 -0600, Stephen Compall wrote:
- Where both tests are T: This is essentially option 1.
This is currently available at http://scompall.nocandysw.com/farewell-rtt-1.darcs.patch , including the previous changes to test/misc-types.lisp.
Wed Mar 15 12:48:04 CST 2006 Stephen Compall scompall@nocandysw.com * remove *runtime-translator-form* from the effective interface
- Add default methods to expand-to-foreign-dyn, expand-to-foreign, and expand-from-foreign, and always use them as fallback, as the "fallback" for specializations that want to fail should be exactly the same as that when there are no specializations. - Use *fto-called* instead of .fto-called. in misc-types.lisp. - misc-types.expand.6 test: expect T instead of NIL.
Stephen Compall s11@member.fsf.org writes:
Wed Mar 15 12:48:04 CST 2006 Stephen Compall scompall@nocandysw.com
- remove *runtime-translator-form* from the effective interface
Ok, so this looks good. I think I like using (call-next-method) instead of the *runtime-translator-form* special.
This patch fails misc-types.expand.{1,2} though since these expect expand-to-foreign-dyn to use expand-to-foreign. As I mentioned before, I kind of like that behavior. Can you convince me that it's a bad idea? :-)
On Thu, 2006-03-16 at 12:36 +0000, Luís Oliveira wrote:
This patch fails misc-types.expand.{1,2} though since these expect expand-to-foreign-dyn to use expand-to-foreign. As I mentioned before, I kind of like that behavior. Can you convince me that it's a bad idea? :-)
Well, translate-type-to-foreign is currently used as the fallback for expand-type-to-foreign, which is currently used as the fallback for expand-type-to-foreign-dyn (when an e-t-f method is defined, anyway, as demonstrated by tests 5 and 6).
However, expand-type-to-foreign has different semantics from translate-type-to-foreign. I highlighted this semi-issue in the most recent patch by wrapping it in VALUES; after all, the second value, ALLOC-PARAM, is discarded in test 6.
Since this behavior is always fine for cases where foreign-dyn expansion isn't even tried, like setting C variables or translating callback return values, but not for when you expect the normal free behavior from translate-type-to-foreign in something like test 6, I think it would be more intuitive for it to fall back directly to the translate-* functions. This is option 1 as described earlier, and implemented in the latest patch.
The semantics for forms answered by expand-type-to-foreign and calls to translate-type-to-foreign could also be unified, so that ettf forms can also answer an ALLOC-PARAM and expect to be freed in foreign-dyn expansion. This is option 2, and less than ideal for some reason.
One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3, the current behavior as demonstrated by tests 5 and 6. As this is just as arduous as requiring an explicit call to simple-foreign-dyn-expansion (the function presented earlier) or something if you want e-t-f semantics for *runtime-translator-form*, but less predictable in its behavior, I would rather it be changed.
I should note that regardless, *runtime-translator-form* can still be eliminated in much the same way for any of the above. Here goes: regardless, *runtime-translator-form* can still be eliminated in much the same way for any of the above.
Stephen Compall s11@member.fsf.org writes:
One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3 [...]
I think *runtime-translator-form* should be a translate-type-to-foreign call when expand-type-to-foreign is called directly and a m-v-b/t-t-t-f/unwind-protect/f-t-t-o thingie when it's called through expand-type-to-foreign-dyn.
On Tue, 2006-03-21 at 12:57 +0000, Luís Oliveira wrote:
Stephen Compall s11@member.fsf.org writes:
One can also specify in the interface documentation that by merely defining an e-t-f, even if it answers *runtime-translator-form*, you are short-circuiting the alloc-param and free mechanism for dynamic-extent expansions, and therefore must either not do so, or always be sure to define a foreign-dyn expansion as well. This is option 3 [...]
I think *runtime-translator-form* should be a translate-type-to-foreign call when expand-type-to-foreign is called directly and a m-v-b/t-t-t-f/unwind-protect/f-t-t-o thingie when it's called through expand-type-to-foreign-dyn.
This is available at http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch , including documentation updates.
Latest of "Option 1" at http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch , also including documentation updates.
Both include the same modification to %expand-type-to-foreign; see the docs in "opt3" for more details.
Stephen Compall s11@member.fsf.org writes:
http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch
Many thanks for producing patches for both options! I'm inclined towards option 1 and IIUC you are inclined towards option 3, so... James, which one do you prefer? :-)
On Fri, 2006-03-31 at 17:11 +0100, Luís Oliveira wrote:
Stephen Compall s11@member.fsf.org writes:
http://scompall.nocandysw.com/cffi/etfd-opt3.darcs.patch http://scompall.nocandysw.com/cffi/etfd-opt1.darcs.patch
Many thanks for producing patches for both options! I'm inclined towards option 1 and IIUC you are inclined towards option 3
Great! I actually prefer option 1 as well.
On 2006-mar-31, at 19:28, Stephen Compall wrote:
Great! I actually prefer option 1 as well.
Ugh... sorry about that. I prefer option 3 and you prefer option 1. :-)
-
Luís Oliveira -
Stephen Compall