Allowing GC in Allegro CL SMP during foreign function calls
Hi all, When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing. At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing. Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution. What do you think about extending the defcfun and foreign-call interfaces to allow for native options to be passed? name-and-options could be extended to accept a new argument, :native-args, for example. Then you'd be able to use defcfun as follows: (defcfun ("strlen" :native-args (:allow-gc :always)) :int (str :string)) Another option is to create a global variable that holds native arguments, and def-foreign-call would read this directly. Thanks, Felix
Having had a closer look at the code, I noticed that the with-foreign-pointer macro allocates in a static area if #+(version>= 8 1). Perhaps it's better to add additional calls to #+(version>= 8 1) in the %foreign-funcall macro, and the defcfun-helper-forms function, so that if #+(version>= 8 1) the appropriate options are set to allow for GC to run during foreign function execution. On Tue, Jul 30, 2013 at 5:33 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Hi all,
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
What do you think about extending the defcfun and foreign-call interfaces to allow for native options to be passed? name-and-options could be extended to accept a new argument, :native-args, for example. Then you'd be able to use defcfun as follows:
(defcfun ("strlen" :native-args (:allow-gc :always)) :int (str :string))
Another option is to create a global variable that holds native arguments, and def-foreign-call would read this directly.
Thanks, Felix
I've created a pull request https://github.com/cffi/cffi/pull/27. On Tue, Jul 30, 2013 at 11:23 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Having had a closer look at the code, I noticed that the with-foreign-pointer macro allocates in a static area if #+(version>= 8 1).
Perhaps it's better to add additional calls to #+(version>= 8 1) in the %foreign-funcall macro, and the defcfun-helper-forms function, so that if #+(version>= 8 1) the appropriate options are set to allow for GC to run during foreign function execution.
On Tue, Jul 30, 2013 at 5:33 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Hi all,
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
What do you think about extending the defcfun and foreign-call interfaces to allow for native options to be passed? name-and-options could be extended to accept a new argument, :native-args, for example. Then you'd be able to use defcfun as follows:
(defcfun ("strlen" :native-args (:allow-gc :always)) :int (str :string))
Another option is to create a global variable that holds native arguments, and def-foreign-call would read this directly.
Thanks, Felix
I moved the code to a different local branch, and I recreated the request https://github.com/cffi/cffi/pull/29. Hopefully it makes things easier. On Wed, Jul 31, 2013 at 10:22 AM, Felix Filozov <ffilozov@gmail.com> wrote:
I've created a pull request https://github.com/cffi/cffi/pull/27.
On Tue, Jul 30, 2013 at 11:23 PM, Felix Filozov <ffilozov@gmail.com>wrote:
Having had a closer look at the code, I noticed that the with-foreign-pointer macro allocates in a static area if #+(version>= 8 1).
Perhaps it's better to add additional calls to #+(version>= 8 1) in the %foreign-funcall macro, and the defcfun-helper-forms function, so that if #+(version>= 8 1) the appropriate options are set to allow for GC to run during foreign function execution.
On Tue, Jul 30, 2013 at 5:33 PM, Felix Filozov <ffilozov@gmail.com>wrote:
Hi all,
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
What do you think about extending the defcfun and foreign-call interfaces to allow for native options to be passed? name-and-options could be extended to accept a new argument, :native-args, for example. Then you'd be able to use defcfun as follows:
(defcfun ("strlen" :native-args (:allow-gc :always)) :int (str :string))
Another option is to create a global variable that holds native arguments, and def-foreign-call would read this directly.
Thanks, Felix
On Tue, Jul 30, 2013 at 10:23 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Having had a closer look at the code, I noticed that the with-foreign-pointer macro allocates in a static area if #+(version>= 8 1).
Why is this relevant for the decision whether to allow GC (and concurrent threads) during an foreign function call? -- Luís Oliveira http://kerno.org/~luis/
Releasing the heap is available only in some versions. Manuals for 8.1 and greater show this option, but the 7.0 manual doesn't (couldn't find much information about versions in between, if they even exist). On Fri, Aug 2, 2013 at 3:00 AM, Luís Oliveira <loliveira@common-lisp.net>wrote:
On Tue, Jul 30, 2013 at 10:23 PM, Felix Filozov <ffilozov@gmail.com> wrote:
Having had a closer look at the code, I noticed that the with-foreign-pointer macro allocates in a static area if #+(version>= 8 1).
Why is this relevant for the decision whether to allow GC (and concurrent threads) during an foreign function call?
-- Luís Oliveira http://kerno.org/~luis/
On Tue, Jul 30, 2013 at 4:33 PM, Felix Filozov <ffilozov@gmail.com> wrote:
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
I understand the advantages of overriding ACL's default. What are the drawbacks? Also, what do other Lisps do? We probably want to be as consistent as possible across the platforms we support.
What do you think about extending the defcfun and foreign-call interfaces to allow for native options to be passed? name-and-options could be extended to accept a new argument, :native-args, for example. Then you'd be able to use defcfun as follows:
(defcfun ("strlen" :native-args (:allow-gc :always)) :int (str :string))
Another option is to create a global variable that holds native arguments, and def-foreign-call would read this directly.
I have nothing against this sort of thing. An alternative for this particular case, though, would be to have an :allow-gc option to defcfun and possibly have a range values like t, nil and :maybe where the first two would signal an error if the underlying Lisp doesn't support that functionality. Cheers, -- Luís Oliveira http://kerno.org/~luis/
On Wed, 2013-07-31 at 09:55 +0100, Luís Oliveira wrote:
On Tue, Jul 30, 2013 at 4:33 PM, Felix Filozov <ffilozov@gmail.com> wrote:
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
I understand the advantages of overriding ACL's default. What are the drawbacks?
Also, what do other Lisps do? We probably want to be as consistent as possible across the platforms we support.
SBCL never disables GC globally, and I don't think that CCL, ECL or ABCL do that either. http://www.lispworks.com/documentation/lw61/FLI/html/fli-94.htm#pgfId-107027... has no mention of GC, so it would seem that among SMP Lisps Allegro is the only one to do sophisticated stuff. CMUCL and Clisp are currently single-threaded so GC is implicitly disabled during foreign calls. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib
On Wed, Jul 31, 2013 at 10:15 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
SBCL never disables GC globally, and I don't think that CCL, ECL or ABCL do that either. http://www.lispworks.com/documentation/lw61/FLI/html/fli-94.htm#pgfId-107027... has no mention of GC, so it would seem that among SMP Lisps Allegro is the only one to do sophisticated stuff. CMUCL and Clisp are currently single-threaded so GC is implicitly disabled during foreign calls.
Precise confirmation would be nice -- a test perhaps -- would be nice. Felix, do you feel like writing a test that checks whether GC can happen while a foreign function call is ongoing? If we can confirm that ACL is the outlier, then (a) I'm all for changing cffi-allegro's implementation to match the semantics of the other Lisps and (b) documenting these semantics in the manual. Cheers, -- Luís Oliveira http://kerno.org/~luis/
On Fri, 2 Aug 2013 01:54:20 +0100, Luís Oliveira said:
On Wed, Jul 31, 2013 at 10:15 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
SBCL never disables GC globally, and I don't think that CCL, ECL or ABCL do that either. http://www.lispworks.com/documentation/lw61/FLI/html/fli-94.htm#pgfId-107027... has no mention of GC, so it would seem that among SMP Lisps Allegro is the only one to do sophisticated stuff. CMUCL and Clisp are currently single-threaded so GC is implicitly disabled during foreign calls.
Precise confirmation would be nice -- a test perhaps -- would be nice.
I confirm it for LispWorks -- entering foreign code never prevents the GC from running in another Lisp thread if needed. -- Martin Simmons LispWorks Ltd http://www.lispworks.com/
Do you mean an Allegro CL test? I don't think that's necessary since the manual is clear on the behavior, and I've witnessed GC block in my own code.
From reading other posts in the thread, it does seem that Allegro CL is the outlier.
I could add a note in the manual that mentions that whether GC blocks during a foreign function call is implementation dependent. On Fri, Aug 2, 2013 at 2:54 AM, Luís Oliveira <loliveira@common-lisp.net>wrote:
On Wed, Jul 31, 2013 at 10:15 AM, Stelian Ionescu <sionescu@cddr.org> wrote:
SBCL never disables GC globally, and I don't think that CCL, ECL or ABCL do that either.
http://www.lispworks.com/documentation/lw61/FLI/html/fli-94.htm#pgfId-107027... no mention of GC, so it would seem that among SMP Lisps Allegro is the only one to do sophisticated stuff. CMUCL and Clisp are currently single-threaded so GC is implicitly disabled during foreign calls.
Precise confirmation would be nice -- a test perhaps -- would be nice. Felix, do you feel like writing a test that checks whether GC can happen while a foreign function call is ongoing?
If we can confirm that ACL is the outlier, then (a) I'm all for changing cffi-allegro's implementation to match the semantics of the other Lisps and (b) documenting these semantics in the manual.
Cheers,
-- Luís Oliveira http://kerno.org/~luis/
Felix Filozov <ffilozov@gmail.com> writes:
Do you mean an Allegro CL test? I don't think that's necessary since the manual is clear on the behavior, and I've witnessed GC block in my own code.
Well, manuals lie and bugs exist. :-) The CFFI test suite has found quite a few implementation bugs. On the other hand, we want to ensure that all supported implementations have the same brehaviour. It'd be nice to have.
From reading other posts in the thread, it does seem that Allegro CL is the outlier.
Yep.
I could add a note in the manual that mentions that whether GC blocks during a foreign function call is implementation dependent.
If we can ensure that the GC does /not/ block it'd be nicer. -- Luís Oliveira http://kerno.org/~luis
It seems that creating such a test would be a nontrivial undertaking. Does it have to be done for this patch to be accepted? If not, then I would rather create a Launchpad issue describing the need for such a test so that it can be worked on later. On Wed, Aug 7, 2013 at 7:03 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
Do you mean an Allegro CL test? I don't think that's necessary since the manual is clear on the behavior, and I've witnessed GC block in my own code.
Well, manuals lie and bugs exist. :-) The CFFI test suite has found quite a few implementation bugs.
On the other hand, we want to ensure that all supported implementations have the same brehaviour. It'd be nice to have.
From reading other posts in the thread, it does seem that Allegro CL is the outlier.
Yep.
I could add a note in the manual that mentions that whether GC blocks during a foreign function call is implementation dependent.
If we can ensure that the GC does /not/ block it'd be nicer.
-- Luís Oliveira http://kerno.org/~luis
Felix Filozov <ffilozov@gmail.com> writes:
It seems that creating such a test would be a nontrivial undertaking. Does it have to be done for this patch to be accepted? If not, then I would rather create a Launchpad issue describing the need for such a test so that it can be worked on later.
Maybe so. A Launchpad issue will do. I've applied your patch, thanks! Cheers, -- Luís Oliveira http://kerno.org/~luis
Thanks. Launchpad bug has been opened. On Mon, Aug 12, 2013 at 11:32 PM, Luís Oliveira <luismbo@gmail.com> wrote:
Felix Filozov <ffilozov@gmail.com> writes:
It seems that creating such a test would be a nontrivial undertaking. Does it have to be done for this patch to be accepted? If not, then I would rather create a Launchpad issue describing the need for such a test so that it can be worked on later.
Maybe so. A Launchpad issue will do. I've applied your patch, thanks!
Cheers,
-- Luís Oliveira http://kerno.org/~luis
Hi, Vladimir Tzankov is the thread expert of CLISP's experimental thread branch, and here's what he has to say on this topic: On Tue, Jul 30, 2013 at 4:33 PM, Felix Filozov <ffilozov@gmail.com> wrote:
When defining a foreign function using def-foreign-call, there's an option one could set called :allow-gc. Setting it to :always, allows the garbage collector to run while the foreign function is executing.
At the moment CFFI uses the default value for :allow-gc, which is :never. This prevents GC from running when a foreign function is executing.
Moreover, once GC blocks, other processes can't make progress. In my case, I need GC to run during foreign function execution.
Stelian Ionescu wrote:
SBCL never disables GC globally, and I don't think that CCL, ECL or ABCL do that either. [LispWorks] has no mention of GC, so it would seem that among SMP Lisps Allegro is the only one to do sophisticated stuff. CMUCL and Clisp are currently single-threaded so GC is implicitly disabled during foreign calls.
Luís Oliveira wrote:
If we can confirm that ACL is the outlier, then (a) I'm all for changing cffi-allegro's implementation to match the semantics of the other Lisps and (b) documenting these semantics in the manual.
Vladimir Tzankow wrote (posted with permission):
While there is no official CLISP release with threads support - threads are usable when built with --with-threads configure option. GC is stop-the-world one i.e. all threads are suspended at safe points while one of them is performing the GC. FFI calls are safe points as well which means that if a thread is in FFI call (blocked or whatever) the GC is allowed to run. The only complication is when a pointer to object in the heap has been passed to foreign code. In this case we "pin" the object before leaving lisp land. GC knows about pinned objects and do not move them around.
Pinning is used mostly for i/o (streams.d) where we pass pointers in heap to syscalls (read,write,etc). There is no Lisp interface to pin objects. Code called via FFI never gets direct pointer in the heap. Nevertheless GC is allowed while the control is in foreign call.
These days, it feels backward IMHO to disable GC by default. Regards, Jörg Höhle
Hi Jörg, long time no see! On Wed, Aug 21, 2013 at 6:22 PM, <Joerg-Cyril.Hoehle@t-systems.com> wrote:
Nevertheless GC is allowed while the control is in foreign call.
Thank you (and Vladimir) for your clarification! Cheers, -- Luís Oliveira http://kerno.org/~luis/
participants (6)
-
Felix Filozov -
Joerg-Cyril.Hoehle＠t-systems.com -
Luís Oliveira -
Luís Oliveira -
Martin Simmons -
Stelian Ionescu