Returning errno as a second value
Hi all, In Allegro CL, the only safe way to get errno after a foreign function call is to configure the foreign function definition to return it as a second value. CFFI doesn't deal with errno. There has been some previous discussion ( http://lists.common-lisp.net/pipermail/cffi-devel/2009-January/003017.html) to return errno safely, but doesn't seem like it went anywhere. So any library that tries to get errno is potentially broken in Allegro CL. I'm seeing this in practice with lisp-zmq, for example. I'd like to introduce a new option to defcfun and foreign-funcall called :errno. It would look like this: (foreign-funcall ("strlen" :errno t) :string "foo" :int), or (defcfun (strlen "strlen" :errno t) :int (s :string)). Calling (strlen) would return two values, the return value of the foreign call, and errno. In some Lisps, the only way to get errno is to make an additional foreign call. Then perhaps that call could be made by CFFI and returns as the second value. I've created a pull request (https://github.com/cffi/cffi/pull/28) with a very rudimentary implementation for Allegro CL. Since CFFI delegates the handling of errno to the implementation, perhaps we could preserve the status quo and not get bugged down in grandiose plans of portability. Thoughts and suggestions are welcome. Thanks, Felix
Felix Filozov <ffilozov@gmail.com> writes:
In Allegro CL, the only safe way to get errno after a foreign function call is to configure the foreign function definition to return it as a second value.
Why does that happen. Is it because interrupt handlers may call functions that overwrite errno?
CFFI doesn't deal with errno. There has been some previous discussion (http://lists.common-lisp.net/pipermail/cffi-devel/2009-January/003017. html) to return errno safely, but doesn't seem like it went anywhere.
Re-reading that was useful. Thanks for the pointer. To sum that discussion up: it'd nice to have cffi-sys:get-errno plus a cffi-sys:without-3rd-party-messing-of-errno so we can implement errno handling in the portable side of CFFI. Do you think that cffi-sys:without-3rd-party-messing-of-errno macro is feasible? If not, we'll have to place the abstraction at the %FOREIGN-FUNCALL level as you suggest.
So any library that tries to get errno is potentially broken in Allegro CL. I'm seeing this in practice with lisp-zmq, for example.
Out of curiosity, what's messing with errno in your case?
I'd like to introduce a new option to defcfun and foreign-funcall called :errno.
It would look like this: (foreign-funcall ("strlen" :errno t) :string "foo" :int), or (defcfun (strlen "strlen" :errno t) :int (s :string)).
Calling (strlen) would return two values, the return value of the foreign call, and errno.
In some Lisps, the only way to get errno is to make an additional foreign call. Then perhaps that call could be made by CFFI and returns as the second value.
That makes sense. Except for the second value bit because CFFI types can yield more than one value. Alternatives that pop up: 1. Return it as the first value. (Probably requires some multiple-value-list mangling which might not be acceptable performance-wise?) 2. Have the foreign function accept an extra errno argument that takes some structure than can be modified with the errno value. Not quite happy with either. Any other ideas?
Since CFFI delegates the handling of errno to the implementation, perhaps we could preserve the status quo and not get bugged down in grandiose plans of portability.
CFFI /is/ a grandiose excursion into portability. :-) Luís
I'm not sure what's overwriting errno. What sort of interrupts do you expect could be overwriting errno, and wouldn't they have to go through the same mechanism of writing to errno like any other thread? Looking into Allegro CL SMP's manual, there's a macro called with-delayed-interrupts<http://www.franz.com/support/documentation/9.0/doc/operators/excl/with-delayed-interrupts.htm>, which should prevent interrupts and GC from running. However, the wrong errno was still returned when I used this macro. I will ask Franz if such a macro exists. I started exploring your second suggestion. I created an errno defstruct (this is what the user passes), in the CFFI package, and I tried passing that structure to the CFFI-SYS layer to be modified, but then I realized that CFFI-SYS doesn't know anything about the CFFI package so it can't modify the structure. So I'm passing a function to CFFI-SYS, which gets called with the returned errno value as an argument. I don't know if I like this callback mechanism. Also, while reading through the code I noticed that there's some edge cases that would need to be handled. For example %defcfun may make a call to foreign-funcall if the argument types and the returns types are considered call-by-value. What does that mean? I also don't quite understand what entry-vec is, if that's something that can be ignored for now. By the way, I closed my pull request since this clearly would need more work. On Wed, Aug 7, 2013 at 7:44 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
In Allegro CL, the only safe way to get errno after a foreign function call is to configure the foreign function definition to return it as a second value.
Why does that happen. Is it because interrupt handlers may call functions that overwrite errno?
CFFI doesn't deal with errno. There has been some previous discussion (http://lists.common-lisp.net/pipermail/cffi-devel/2009-January/003017. html) to return errno safely, but doesn't seem like it went anywhere.
Re-reading that was useful. Thanks for the pointer.
To sum that discussion up: it'd nice to have cffi-sys:get-errno plus a cffi-sys:without-3rd-party-messing-of-errno so we can implement errno handling in the portable side of CFFI.
Do you think that cffi-sys:without-3rd-party-messing-of-errno macro is feasible? If not, we'll have to place the abstraction at the %FOREIGN-FUNCALL level as you suggest.
So any library that tries to get errno is potentially broken in Allegro CL. I'm seeing this in practice with lisp-zmq, for example.
Out of curiosity, what's messing with errno in your case?
I'd like to introduce a new option to defcfun and foreign-funcall called :errno.
It would look like this: (foreign-funcall ("strlen" :errno t) :string "foo" :int), or (defcfun (strlen "strlen" :errno t) :int (s :string)).
Calling (strlen) would return two values, the return value of the foreign call, and errno.
In some Lisps, the only way to get errno is to make an additional foreign call. Then perhaps that call could be made by CFFI and returns as the second value.
That makes sense. Except for the second value bit because CFFI types can yield more than one value. Alternatives that pop up:
1. Return it as the first value. (Probably requires some multiple-value-list mangling which might not be acceptable performance-wise?)
2. Have the foreign function accept an extra errno argument that takes some structure than can be modified with the errno value.
Not quite happy with either. Any other ideas?
Since CFFI delegates the handling of errno to the implementation, perhaps we could preserve the status quo and not get bugged down in grandiose plans of portability.
CFFI /is/ a grandiose excursion into portability. :-)
Luís
Franz support has said that setting :errno-value is the only guaranteed way to get errno. Macros such as without-interrupt, without-scheduling, and with-delayed-interrupts, are not sufficient. On Thu, Aug 8, 2013 at 5:53 PM, Felix Filozov <ffilozov@gmail.com> wrote:
I'm not sure what's overwriting errno. What sort of interrupts do you expect could be overwriting errno, and wouldn't they have to go through the same mechanism of writing to errno like any other thread?
Looking into Allegro CL SMP's manual, there's a macro called with-delayed-interrupts<http://www.franz.com/support/documentation/9.0/doc/operators/excl/with-delayed-interrupts.htm>, which should prevent interrupts and GC from running. However, the wrong errno was still returned when I used this macro. I will ask Franz if such a macro exists.
I started exploring your second suggestion. I created an errno defstruct (this is what the user passes), in the CFFI package, and I tried passing that structure to the CFFI-SYS layer to be modified, but then I realized that CFFI-SYS doesn't know anything about the CFFI package so it can't modify the structure. So I'm passing a function to CFFI-SYS, which gets called with the returned errno value as an argument. I don't know if I like this callback mechanism.
Also, while reading through the code I noticed that there's some edge cases that would need to be handled. For example %defcfun may make a call to foreign-funcall if the argument types and the returns types are considered call-by-value. What does that mean?
I also don't quite understand what entry-vec is, if that's something that can be ignored for now.
By the way, I closed my pull request since this clearly would need more work.
On Wed, Aug 7, 2013 at 7:44 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
In Allegro CL, the only safe way to get errno after a foreign function call is to configure the foreign function definition to return it as a second value.
Why does that happen. Is it because interrupt handlers may call functions that overwrite errno?
CFFI doesn't deal with errno. There has been some previous discussion (http://lists.common-lisp.net/pipermail/cffi-devel/2009-January/003017. html) to return errno safely, but doesn't seem like it went anywhere.
Re-reading that was useful. Thanks for the pointer.
To sum that discussion up: it'd nice to have cffi-sys:get-errno plus a cffi-sys:without-3rd-party-messing-of-errno so we can implement errno handling in the portable side of CFFI.
Do you think that cffi-sys:without-3rd-party-messing-of-errno macro is feasible? If not, we'll have to place the abstraction at the %FOREIGN-FUNCALL level as you suggest.
So any library that tries to get errno is potentially broken in Allegro CL. I'm seeing this in practice with lisp-zmq, for example.
Out of curiosity, what's messing with errno in your case?
I'd like to introduce a new option to defcfun and foreign-funcall called :errno.
It would look like this: (foreign-funcall ("strlen" :errno t) :string "foo" :int), or (defcfun (strlen "strlen" :errno t) :int (s :string)).
Calling (strlen) would return two values, the return value of the foreign call, and errno.
In some Lisps, the only way to get errno is to make an additional foreign call. Then perhaps that call could be made by CFFI and returns as the second value.
That makes sense. Except for the second value bit because CFFI types can yield more than one value. Alternatives that pop up:
1. Return it as the first value. (Probably requires some multiple-value-list mangling which might not be acceptable performance-wise?)
2. Have the foreign function accept an extra errno argument that takes some structure than can be modified with the errno value.
Not quite happy with either. Any other ideas?
Since CFFI delegates the handling of errno to the implementation, perhaps we could preserve the status quo and not get bugged down in grandiose plans of portability.
CFFI /is/ a grandiose excursion into portability. :-)
Luís
Felix Filozov <ffilozov@gmail.com> writes:
Franz support has said that setting :errno-value is the only guaranteed way to get errno. Macros such as without-interrupt, without-scheduling, and with-delayed-interrupts, are not sufficient.
Cool, thanks. It'd be nice to know what the failure modes are so we can write tests and bug other Lisp vendors about it. Luís
Felix Filozov <ffilozov@gmail.com> writes:
Looking into Allegro CL SMP's manual, there's a macro called with-delayed-interrupts, which should prevent interrupts and GC from running. However, the wrong errno was still returned when I used this macro. I will ask Franz if such a macro exists.
What's your test case like?
I started exploring your second suggestion. I created an errno defstruct (this is what the user passes), in the CFFI package, and I tried passing that structure to the CFFI-SYS layer to be modified, but then I realized that CFFI-SYS doesn't know anything about the CFFI package so it can't modify the structure. So I'm passing a function to CFFI-SYS, which gets called with the returned errno value as an argument. I don't know if I like this callback mechanism.
Seems a bit heavy handed, yeah. We can split cffi-sys in two files: a shared file (with the package define and perhaps that structure definition) and an implementation-specific file.
Also, while reading through the code I noticed that there's some edge cases that would need to be handled. For example %defcfun may make a call to foreign-funcall if the argument types and the returns types are considered call-by-value. What does that mean?
We've recently added support for passing structures by value. When that functionality is needed, we use libffi for invoking the foreign function. I have no idea how we can reliably get errno in that case.
I also don't quite understand what entry-vec is, if that's something that can be ignored for now.
I don't remember the details, but doc/allegro-internals.txt has some info about it. Cheers, -- Luís Oliveira http://kerno.org/~luis
The test case is calling a foreign function that creates a ZMQ socket, and calling it twice with the same arguments sets errno. But (with-delayed-interrupts (foreign call) (get errno foreign calL)) always returns 0. As Franz said, this should not be expected to work. On Thu, Aug 8, 2013 at 11:37 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Looking into Allegro CL SMP's manual, there's a macro called with-delayed-interrupts, which should prevent interrupts and GC from running. However, the wrong errno was still returned when I used this macro. I will ask Franz if such a macro exists.
What's your test case like?
I started exploring your second suggestion. I created an errno defstruct (this is what the user passes), in the CFFI package, and I tried passing that structure to the CFFI-SYS layer to be modified, but then I realized that CFFI-SYS doesn't know anything about the CFFI package so it can't modify the structure. So I'm passing a function to CFFI-SYS, which gets called with the returned errno value as an argument. I don't know if I like this callback mechanism.
Seems a bit heavy handed, yeah. We can split cffi-sys in two files: a shared file (with the package define and perhaps that structure definition) and an implementation-specific file.
Also, while reading through the code I noticed that there's some edge cases that would need to be handled. For example %defcfun may make a call to foreign-funcall if the argument types and the returns types are considered call-by-value. What does that mean?
We've recently added support for passing structures by value. When that functionality is needed, we use libffi for invoking the foreign function. I have no idea how we can reliably get errno in that case.
I also don't quite understand what entry-vec is, if that's something that can be ignored for now.
I don't remember the details, but doc/allegro-internals.txt has some info about it.
Cheers,
-- Luís Oliveira http://kerno.org/~luis
I've dug more into the code and came up with something a bit nicer than what I originally described. Feel free to look at the changes<https://github.com/ffilozov/cffi/commit/e35e949891a0fe5faeca2a0a44050edff99b9f3c> (comments and suggestions are always welcome). My basic assumption is that the underlying CFFI-SYS functions like %foreign-funcall and defcfun-helper-forms would return two values whenever defcfun or foreign-funcall is configured to return errno. The first value is the foreign call return value, and the second is errno. errno is then captured to be passed to the user's structure, and the foreign call return value is propagated further. Here's a sample of how it looks: CL-USER> (defparameter errno-object (cffi:make-errno :value 0)) ERRNO-OBJECT CL-USER> (cffi:defcfun ("socket" :errno t) :int (a :int) (b :int) (c :int)) Warning: :call-direct ignored because :error-value specified. SOCKET CL-USER> (socket errno-object -1 -1 -1) -1 CL-USER> errno-object #S(CFFI::ERRNO :VALUE 22) More: CL-USER> (defparameter errno-object (cffi:make-errno :value 0)) ERRNO-OBJECT CL-USER> (cffi:foreign-funcall ("getenv" :errno errno-object) :string "SHELL" :string+ptr) ("/bin/bash" 140735646882874) CL-USER> errno-object #S(CFFI::ERRNO :VALUE 0) Last one: CL-USER> (defparameter errno-object (cffi:make-errno :value 0)) ERRNO-OBJECT CL-USER> (cffi:defcfun ("printf" :errno t) :int (control :string) &rest) PRINTF CL-USER> (printf errno-object "%s %s" :string "hi" :string "bob") 6 CL-USER> errno-object #S(CFFI::ERRNO :VALUE 0) On Thu, Aug 8, 2013 at 11:37 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Looking into Allegro CL SMP's manual, there's a macro called with-delayed-interrupts, which should prevent interrupts and GC from running. However, the wrong errno was still returned when I used this macro. I will ask Franz if such a macro exists.
What's your test case like?
I started exploring your second suggestion. I created an errno defstruct (this is what the user passes), in the CFFI package, and I tried passing that structure to the CFFI-SYS layer to be modified, but then I realized that CFFI-SYS doesn't know anything about the CFFI package so it can't modify the structure. So I'm passing a function to CFFI-SYS, which gets called with the returned errno value as an argument. I don't know if I like this callback mechanism.
Seems a bit heavy handed, yeah. We can split cffi-sys in two files: a shared file (with the package define and perhaps that structure definition) and an implementation-specific file.
Also, while reading through the code I noticed that there's some edge cases that would need to be handled. For example %defcfun may make a call to foreign-funcall if the argument types and the returns types are considered call-by-value. What does that mean?
We've recently added support for passing structures by value. When that functionality is needed, we use libffi for invoking the foreign function. I have no idea how we can reliably get errno in that case.
I also don't quite understand what entry-vec is, if that's something that can be ignored for now.
I don't remember the details, but doc/allegro-internals.txt has some info about it.
Cheers,
-- Luís Oliveira http://kerno.org/~luis
On 08/07/2013 10:44 AM, Luís Oliveira wrote:
It would look like this: (foreign-funcall ("strlen" :errno t) :string "foo" :int), or (defcfun (strlen "strlen" :errno t) :int (s :string)).
Calling (strlen) would return two values, the return value of the foreign call, and errno.
In some Lisps, the only way to get errno is to make an additional foreign call. Then perhaps that call could be made by CFFI and returns as the second value. That makes sense. Except for the second value bit because CFFI types can yield more than one value. Alternatives that pop up:
1. Return it as the first value. (Probably requires some multiple-value-list mangling which might not be acceptable performance-wise?)
2. Have the foreign function accept an extra errno argument that takes some structure than can be modified with the errno value.
Not quite happy with either. Any other ideas?
Perhaps instead of returning "errno" as an additional value, we could modify an internal global thread-local variable with the value of errno when the function is flagged with ":errno t", and export (cffi:errno) to return the thread's saved errno value? That's how the underlying machinery works at the C level after all. Of course, then CFFI-SYS needs a portable mechanism for thread-locals... If I was binding a library that was user-hostile enough to use "errno" as its error reporting mechanism, I'd be tempted to wrap its function calls to pass the error code out with an explicit output parameter. I wonder if there's any way to automate that. James
Hi Felix, Errno is a particularly tricky beast. It is a thread-local "global" that is modified by a large number of system calls and library functions. Code must be careful to check it before another such call is made. I could easily imagine a stop-the-world GC accidentally modifying errno if errno is not checked "atomically" with the function call. Also errno is very lightweight. An extra foreign call to check it could easily dwarf the cost of the check. Errno is a "hidden" return parameter, much like the second position in a multiple-value return. Thus the Allegro semantics of returning errno as a second value seem reasonable to me. As you suggested, CFFI could provide a default errno implementation for CLs that have different native semantics. - Daniel
I worked out a basic implementation for a thread-local errno<https://github.com/ffilozov/cffi/compare/thread-local-errno>. Bordeaux-threads has been added as a dependency as well as two extra functions: cffi:set-errno and cffi:get-errno. This is a much cleaner interface than passing an object, and the generated function doesn't need to have an extra parameter. Sample: CL-USER> (cffi:defcfun ("socket" :errno t) :int (a :int) (b :int) (c :int)) Warning: :call-direct ignored because :error-value specified. SOCKET CL-USER> (socket -1 -1 -1) -1 CL-USER> (cffi:get-errno) 22 T The implementation uses locking, so its efficiency could improved. I'm interested in your suggestions on how to do that. On Thu, Aug 8, 2013 at 4:29 AM, Daniel Herring <dherring@tentpost.com>wrote:
Hi Felix,
Errno is a particularly tricky beast. It is a thread-local "global" that is modified by a large number of system calls and library functions. Code must be careful to check it before another such call is made. I could easily imagine a stop-the-world GC accidentally modifying errno if errno is not checked "atomically" with the function call.
Also errno is very lightweight. An extra foreign call to check it could easily dwarf the cost of the check. Errno is a "hidden" return parameter, much like the second position in a multiple-value return.
Thus the Allegro semantics of returning errno as a second value seem reasonable to me.
As you suggested, CFFI could provide a default errno implementation for CLs that have different native semantics.
- Daniel
Felix Filozov <ffilozov@gmail.com> writes:
The implementation uses locking, so its efficiency could improved. I'm interested in your suggestions on how to do that.
Indeed, this interface is much better from the user's point of view. I feel a bit uneasy about the implementation though. I found this message from Duane Rettig (<https://groups.google.com/forum/#!msg/comp.lang.lisp/7rhuvXaiR_g/AQ2-i6-74s4J>) that suggests that the nearest ACL gets to thread-local variables is by defining a special variable then adding it to the list of default bindings. How about an interface like this? (with-errno (some-foreign-function-with-the-errno-option-on) (get-errno)) or perhaps (with-errno (errno) (some-foreign-function-with-the-errno-option-on) errno) Cheers, -- Luís Oliveira http://kerno.org/~luis
If I understood Duane Rettig's response correctly, when a thread binds a special variable using a let, that binding is only visible for that particular thread. Is this how other Lisps behave? With that in mind, in order to make with-errno work, the foreign function call expression would have to be identified and bound with a let. So something like this: (defmacro with-errno (&body body) (let ((exprs-before-ffc ...) (ffc-expression ..) (exprs-after-ffc ..)) `,@exprs-before-ffc (let ((errno ,ffc-expression)) ,@(exprs-after-ffc)))) There's some complications such as, how to determine which expression is a foreign function call since defcfun can assign any arbitrary name. There's also the issue of what should the foreign function call return? It seems like it would have to return errno as the last value. To mitigate the first concern, perhaps a more explicit interface would help: (with-foreign-funcall-errno (result errno (foreign-funcall "strlen" ...)) ...) You mentioned that a foreign function call could return errno as the first value, how about returning it as the last? Another alternative is to optimize the way synchronization is done, so that we could improve the thread-local code I wrote before. On Mon, Aug 12, 2013 at 10:54 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
The implementation uses locking, so its efficiency could improved. I'm interested in your suggestions on how to do that.
Indeed, this interface is much better from the user's point of view. I feel a bit uneasy about the implementation though.
)
I found this message from Duane Rettig (< https://groups.google.com/forum/#!msg/comp.lang.lisp/7rhuvXaiR_g/AQ2-i6-74s4... that suggests that the nearest ACL gets to thread-local variables is by defining a special variable then adding it to the list of default bindings.
How about an interface like this?
(with-errno (some-foreign-function-with-the-errno-option-on) (get-errno))
or perhaps
(with-errno (errno) (some-foreign-function-with-the-errno-option-on) errno)
Cheers,
-- Luís Oliveira http://kerno.org/~luis
Felix Filozov <ffilozov@gmail.com> writes:
If I understood Duane Rettig's response correctly, when a thread binds a special variable using a let, that binding is only visible for that particular thread. Is this how other Lisps behave?
All CLs I'm aware of behave like that, yes.
With that in mind, in order to make with-errno work, the foreign function call expression would have to be identified and bound with a let.
Why is that? I was thinking that the interface could work as follows: 1. WITH-ERRNO establishes a dynamic binding (cffi-sys:*errno* for instance) 2. FOREIGN-FUNCALL does something like (when (boundp '*errno*) (setf *errno* <errno>)) 3. GET-ERRNO simply returns the value of *errno*
You mentioned that a foreign function call could return errno as the first value, how about returning it as the last?
That would be much simpler wouldn't it? I wonder if we can avoid consing, at least for the single return value case... If all else fails, simply documenting that grabbing errno entails discarding all but the first return value might be a solution.
Another alternative is to optimize the way synchronization is done, so that we could improve the thread-local code I wrote before.
I don't have any good ideas on that front. Luís
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO. Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value? On Wed, Aug 14, 2013 at 12:15 AM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
If I understood Duane Rettig's response correctly, when a thread binds a special variable using a let, that binding is only visible for that particular thread. Is this how other Lisps behave?
All CLs I'm aware of behave like that, yes.
With that in mind, in order to make with-errno work, the foreign function call expression would have to be identified and bound with a let.
Why is that? I was thinking that the interface could work as follows:
1. WITH-ERRNO establishes a dynamic binding (cffi-sys:*errno* for instance) 2. FOREIGN-FUNCALL does something like (when (boundp '*errno*) (setf *errno* <errno>)) 3. GET-ERRNO simply returns the value of *errno*
You mentioned that a foreign function call could return errno as the first value, how about returning it as the last?
That would be much simpler wouldn't it? I wonder if we can avoid consing, at least for the single return value case... If all else fails, simply documenting that grabbing errno entails discarding all but the first return value might be a solution.
Another alternative is to optimize the way synchronization is done, so that we could improve the thread-local code I wrote before.
I don't have any good ideas on that front.
Luís
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like: (multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) )) Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?) Luís
Ah, I understand why it has to be dynamic now. I'll start working on with-errno then. The consing problem is clearer now, but I wonder how much overhead it adds compared to the cost of making a foreign call. According to the Allegro CL documentation, a foreign call which returns errno is less efficient. For other Lisps, an additional foreign call would have to be made to get errno. Maybe consing is negligible? On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
I have a with-errno <https://github.com/ffilozov/cffi/tree/with-errno>implementation. On Wed, Aug 14, 2013 at 5:23 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Ah, I understand why it has to be dynamic now. I'll start working on with-errno then.
The consing problem is clearer now, but I wonder how much overhead it adds compared to the cost of making a foreign call. According to the Allegro CL documentation, a foreign call which returns errno is less efficient. For other Lisps, an additional foreign call would have to be made to get errno. Maybe consing is negligible?
On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
Hi Luís, I am hoping to resurrect this thread. Did you by any chance take a look at the pull request <https://github.com/cffi/cffi/pull/31> I made a few months ago relating to this feature? Thanks, Felix On Wed, Aug 14, 2013 at 6:26 PM, Felix Filozov <ffilozov@gmail.com> wrote:
I have a with-errno <https://github.com/ffilozov/cffi/tree/with-errno>implementation.
On Wed, Aug 14, 2013 at 5:23 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Ah, I understand why it has to be dynamic now. I'll start working on with-errno then.
The consing problem is clearer now, but I wonder how much overhead it adds compared to the cost of making a foreign call. According to the Allegro CL documentation, a foreign call which returns errno is less efficient. For other Lisps, an additional foreign call would have to be made to get errno. Maybe consing is negligible?
On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
On Wed, Aug 14, 2013 at 3:04 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Maybe we can even do this using lexical binding, since GET-ERRNO could be a macro that expands to a lexical variable introduced by WITH-ERRNO.
The issue there is that foreign-funcall will be setting errno.
Could you elaborate where there's a consing issue, if errno is returned as the last, or first, value?
So, the pseudo-code for the implementation in foreign-funcall would be something like:
(multiple-value-bind (return-value errno) (%foreign-funcal ...) (let ((translated-return-values (multiple-value-list (translate-from-foreign ...)))) (values-list (cons errno translated-return-values)) ;; or (values-list (nconc translated-return-values (list errno))) ))
Right now I'm inclined to discard all but the first translated return value. (This is your initial solution, isn't it?)
Luís
On Tue, 2014-01-28 at 15:33 +0100, Felix Filozov wrote:
Hi Luís, I am hoping to resurrect this thread. Did you by any chance take a look at the pull request I made a few months ago relating to this feature?
I strongly object to anything that slows down the common call path, which is the case here because capture-errno is always called when not using libffi. In addition, IMO this is a hairy-enough case that we should ask the implementors to add this feature and not do it ourselves. As for the interface, the simplest thing might be to add it as last value - not second because custom foreign types can return multiple values, e.g. string+ptr - but that will turn out to be slow because I don't think our current type system knows about arity of custom types and marshalling unknown numbers of multiple values will cons. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Tue, 2014-01-28 at 16:11 +0100, Stelian Ionescu wrote:
On Tue, 2014-01-28 at 15:33 +0100, Felix Filozov wrote:
Hi Luís, I am hoping to resurrect this thread. Did you by any chance take a look at the pull request I made a few months ago relating to this feature?
I strongly object to anything that slows down the common call path, which is the case here because capture-errno is always called when not using libffi.
In addition, IMO this is a hairy-enough case that we should ask the implementors to add this feature and not do it ourselves.
As for the interface, the simplest thing might be to add it as last value - not second because custom foreign types can return multiple values, e.g. string+ptr - but that will turn out to be slow because I don't think our current type system knows about arity of custom types and marshalling unknown numbers of multiple values will cons.
For example (cffi:defcfun foo :string+ptr (a :int)) on SBCL expands to (DEFUN FOO (A) (LET ((#:G1087 A)) (CFFI:TRANSLATE-FROM-FOREIGN (CFFI-SYS:%FOREIGN-FUNCALL "foo" (:INT #:G1087 :POINTER) :CONVENTION :CDECL :LIBRARY :DEFAULT) #<CFFI::FOREIGN-STRING+PTR-TYPE :UTF-8>))) Since translate-from-foreign is in a tail position we currently don't care about the number of values it returns, but with errno and knowledge of multiple values we must make it expand to this: (defun foo (a) (let ((#:g1087 a)) (multiple-value-bind (ret errno) (cffi-sys:%foreign-funcall "foo" (:int #:g1087 :pointer) :convention :cdecl :library :default) (multiple-value-bind (val0 val1) (cffi:translate-from-foreign ptr #<cffi::foreign-string+ptr-type :utf-8>))) (values val0 val1 errno))) -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
G'day Everyone, On 13/08/2013, at 6:54 AM, Luís Oliveira wrote:
How about an interface like this?
(with-errno (some-foreign-function-with-the-errno-option-on) (get-errno))
or perhaps
(with-errno (errno) (some-foreign-function-with-the-errno-option-on) errno)
Isn't this problem more general than just errno? For example, the functions WSAGetLastError() and GetLastError() on Windows. // On Posix based systems: ssize_t rv = read(fd, buffer, buffer_size); if (rv == -1) { printf("ERROR: %s\n", strerror(errno)); return FAILED; } // On Windows (Winsock): SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); if (s == INVALID_SOCKET) { int last_error = WSAGetLastError(); WCHAR buffer[1024]; FormatMessage(FORMAT_MESSAGE_FROM_STRING | FORMAT_MESSAGE_IGNORE_INSERTS, 0, last_error, 0, buffer, 1024, NULL); return FAILED; } // On Windows (Named Pipes) HANDLE h = CreateFileA(pipe_name, GENERIC_READ | GENERIC_WRITE, ... ); if (h == INVALID_HANDLE_ERROR) { DWORD last_error = GetLastError(); WCHAR buffer[1024]; FormatMessage(FORMAT_MESSAGE_FROM_STRING | FORMAT_MESSAGE_IGNORE_INSERTS, 0, last_error, 0, buffer, 1024, NULL); return FAILED; } This entire thread has been enlightening. Thanks Mark
With Allegro CL, there's an option you can pass (:error-value :os-specific) where it will give you errno or GetLastError() depending on what OS you're on. On Thu, Aug 15, 2013 at 12:48 PM, Mark Cox <markcox80@gmail.com> wrote:
G'day Everyone,
On 13/08/2013, at 6:54 AM, Luís Oliveira wrote:
How about an interface like this?
(with-errno (some-foreign-function-with-the-errno-option-on) (get-errno))
or perhaps
(with-errno (errno) (some-foreign-function-with-the-errno-option-on) errno)
Isn't this problem more general than just errno? For example, the functions WSAGetLastError() and GetLastError() on Windows.
// On Posix based systems: ssize_t rv = read(fd, buffer, buffer_size); if (rv == -1) { printf("ERROR: %s\n", strerror(errno)); return FAILED; }
// On Windows (Winsock): SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); if (s == INVALID_SOCKET) { int last_error = WSAGetLastError(); WCHAR buffer[1024]; FormatMessage(FORMAT_MESSAGE_FROM_STRING | FORMAT_MESSAGE_IGNORE_INSERTS, 0, last_error, 0, buffer, 1024, NULL); return FAILED; }
// On Windows (Named Pipes) HANDLE h = CreateFileA(pipe_name, GENERIC_READ | GENERIC_WRITE, ... ); if (h == INVALID_HANDLE_ERROR) { DWORD last_error = GetLastError(); WCHAR buffer[1024]; FormatMessage(FORMAT_MESSAGE_FROM_STRING | FORMAT_MESSAGE_IGNORE_INSERTS, 0, last_error, 0, buffer, 1024, NULL); return FAILED; }
This entire thread has been enlightening.
Thanks Mark
participants (7)
-
Daniel Herring -
Edgar Gonçalves -
Felix Filozov -
James Bielman -
Luís Oliveira -
Mark Cox -
Stelian Ionescu