"Hoehle, Joerg-Cyril" Joerg-Cyril.Hoehle@t-systems.com writes:

I'm curious to know how precisely SBCL handles the details.

As far as CLISP is concerned, the second example works (if converted to CLISP FFI syntax), but will leak malloc()'ed memory if it's never deallocated.

From memory: each callback gets allocated separately, and will never

be reclaimed (currently). Each alien-lambda closure gets its own callback.

That is, if your application does not provide for a means to receive the callback address from the alien world, so that the Lisp side can free it, then your application leaks memory.

Right.

This tells us how a lambda callback ought to be designed:

• can create one,
• be able to keep its handle on the Lisp side,
• to be able to destroy it
  • via garbage collection
  • and/or via stack semantics
  • from the C side(?)

CLISP's FFI does not directly fir the bill. You need a little wrapper code. I haven't looked at how CFFI fits these requirements.

I'm not sure I agree. I think the best we can do for callbacks (anonymous or not) is invalidate them: switch the C->Lisp tramp to point to an error-signalling Lisp function, after which the original callback trampoline can be freed (but then we need to lock the trampoline in order not to yank it from beneath another thread -- or do something really clever).

That way any C-function that still happens to have the address we gave it will "act reasonably", instead of landing at a random location -- possibly a new and totally different callback.

I would not like callbacks to ever be released by default, nor from the C-side.

Second, you say "expensive". What do you mean? o The allocation of memory for the callback structure?

The allocation and contruction of the callback trampoline. There are a couple of shortcuts that could be taken to make it significantly more efficient, but they have not been implemented yet.

Cheers,

-- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs."

"Hoehle, Joerg-Cyril" Joerg-Cyril.Hoehle@t-systems.com writes:

Nikodemus Siivola wrote:

...

I would not like callbacks to ever be released by default, nor from the C-side.

Of course, I haven't looked at every FFI code every Lispnik wrote for CLISP, but I believe the following is not too much invented: (defun some-often-called-function ... (flet ((ignore-it (&rest args) (declare (ignore args)) nil))) (with-C-signal-handler (SIGINT/SIGxyz #'ignore-it) some-code ...)))

What I mean is 0. Assume with-signal handler installs the Lisp function as signal handler

1. code involving lambda-callbacks may be called every so often
2. every such call creates a trampoline.
3. that application leaks memory like a sieve.

About #1: (loop (call-some-c (alien-lambda () (print "ok!")))) is fine. The callback can be reused every time (at least SBCL allocated only a single callback for this case). Only when it is a closure do we run into trouble.

In summary, the language & library designers must not assume anything about the pattern of use of a given construct, except for the worst case.

Of course, the example is poorly chosen. The semantics of with-* are that we should be able to invalidate the trampoline upon exit.

;-)

I don't know about CLISP, but for most implementations I'd assume that WITH-SIGNAL-HANDLER doesn't actually involve a callback in the "normal sense", but that default signal handlers that take care of the transport to the Lisp-land already exist, and that only installing a new lisp-side function is required.

But assuming WITH-CALLBACK there instead, yes, I agree that it should be invalidated.

...

I think the best we can do for callbacks (anonymous or not) is invalidate them: switch the C->Lisp tramp to point to an error-signalling Lisp function,

Your suggestion of changing the trampoline to point to an error signaling function is nice for debugging (like it's nice to catch duplicate calls to free()). It's not so nice for production use, as memory consumption would continuously grow...

Not really. Or almost not.

0. DEFINE-CALLBACK-FUNCTION should not leak memory on redefinitions. (Currently in SBCL it does, but that's a bug.) ALIEN-LAMBDA for non-closures should allocate only a single callback no matter how many times it is called.

1. What I have in mind is keeping a linkage-table for callbacks: what is passed to C is and address in the linkage-table, and the linkage in turn holds code to jump to the "real" callback. So the not-cleanly-reclaimable memory would only grow by 1-4 words per callback. SBCL doesn't currently do this, though.

If push comes to shove even the linkage-table addresses could be reused -- in which case invalidated callback slots in the table that get reused will end up pointing in the "wrong" (new) callback -- but then there is no leak, and at least the jump will never go to a totally random place, like a piece of free'd memory.

2. I'd like to see a real usage-pattern that causes the callback space requirements to grow without bounds if the table entries aren't reused: keep in mind that we're talking about C callbacks, which aren't really the sort of thing that gets _generated_ a lot, since in C they are just function pointers -- they may get switched around, but new ones don't usually appear out of the blue.

3. (SETF ALIEN-CALLBACK-FUNCTION) helps too: no need to allocate a new callback, just point it to a new lisp function.

Granted, (let ((counter 0)) (display-button (alien-lambda int () (incf counter)))) is a reasonable pattern that will leak memory if used carelesssly, but that's about the size of it, and since there are no closures in C the pattern is more likely to be something like (let ((a-counter (make-alien 'int))) (display-button (alien-lambda int ((counter (* int))) (incf (deref counter))) (addr a-counter))) as the C-side callbacks are likely to allow registering a pointer to data along with the callback -- in which case the callback can be same on every call to display-button, and user-code can manage A-COUNTER allocation in any way it pleases.

...

after which the original callback trampoline can be freed

I don't understand. Either the trampoline is free'd or it's not and can point to error signaling code instead of the original function.

Bad terminology on my part: by trampoline I mean a piece of code that nows how to massage the C-side arguments for Lisp, and where to jump after the arguments have been dealt with. This can be freed if the linkage-table is first updated to point to the "bad-callback" trampoline instead.

(One further reason I'm not wild about actually freeing the memory is threading: to GC a stale trampoline safely is "a bit tricky" unless we are willing to have a lock per tramp. This is also one of the reasons I haven't (yet) gone forward with the callback linkage-table for SBCL.)

This is already too long and too rambling, but in my mind the design tradeoff with alien-lambda is between flexibility, safety, and licence to shoot your foot off.

Flexibility: since it is trivially easy to support closures as callbacks, I don't see any reason not to.

Safety: (1) callbacks always pointing to sane locations (2) memory. Since C manages to make do without releasing its callback functions I think we should manage without releasing C-to-Lisp callbacks, so I consider #1 to be more important.

Shooting yourself in the foot: alternatives are (1) being careless and using alien-lambda closures till heap runs out (2) being careless and releasing something C still expects to use. Neither is nice, but #2 is possibly worse. Either way carelessness costs, but for #1 there is something we can do: we can try to minimize the cost of callback closures and we can eg. make it possible to disable them, so you can be easily enough sure that you don't leak memory (assuming you are not calling EVAL or COMPILE at runtime, of course.)

Finally, while I'm willing to support freeing callbacks in their entirety, I actually view it as a more of an development trick, since I feel it likelier that you spend a lot of memory on callbacks you don't end up using after all during development, then in production. %%NUKE-CALLBACK sounds ominous enough...

Then again, maybe I am just overly conservative -- actual use-cases will carry the day. ...or maybe I am overly adventurous, and ALIEN-LAMBDA should signal an error for closures, in which care there should be no real leaks.

Cheers,

-- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs."