Detecting at runtime when bindings go out of scope?
I have been reworking my Reppy Channels for the past week or so. Then I ran into this article from yore… http://erikdemaine.org/papers/CCC97/paper.ps <http://erikdemaine.org/papers/CCC97/paper.ps> The article prompted me to wonder about Lisp bindings. We all know how to write a macro to protect and discard resources, e.g., (defmacro with-channel (ch &body body) `(let ((,ch (make-channel))) (unwind-protect (progn ,@body) (discard-channel ch)))) And that works just fine. At the end of use for the ephemeral channel, we discard it, and hence scavenge up any threads that might have been waiting on it. Threads are a relatively precious system resource, and so scavenging is a useful thing to do. They won’t ever be garbage collected because, typically, the Lisp system keeps a list of running threads. Hence there will always be outstanding, if useless, references to them. But then I began wondering if there could be a way to automatically detect when a resource goes out of scope / context? Why should we have to write the WITH- macros? Memory is automatically reclaimed. Why couldn’t we have a method that could also automatically reclaims vital system resources? Of course if the channel were ever handed off to another thread and stored away somewhere, it shouldn’t be reclaimed. But I already have GC finalization to help with that case. What I’m wondering about is being able to declare an item, a special kind of Constructor parameter, which refers to a system resource that needs scavenging, such that I could forego the WITH- formalism? Maybe I’m asking for too much here… how would we ever detect the item being stored? We’d need reference counting, and I tend to dislike that approach very much.. Anyone ever seen a system for resource reclamation that is better than WITH- and UNWIND-PROTECT? - DM
On 08/25/2016 06:20 PM, David McClain wrote:
What I’m wondering about is being able to declare an item, a special kind of Constructor parameter, which refers to a system resource that needs scavenging, such that I could forego the WITH- formalism? Maybe I’m asking for too much here… how would we ever detect the item being stored? We’d need reference counting, and I tend to dislike that approach very much..
Anyone ever seen a system for resource reclamation that is better than WITH- and UNWIND-PROTECT?
- DM
If I'm understanding correctly, your ideal would be something like this: (let ((channel (make-channel ...))) (do-stuff channel) (do-other-stuff channel)) and at the end of the lexical scope, due to the fact that the channel object has something special done to it during construction, it automatically calls disposal routines of the channel right at that point? I think that's going to be tough, as well as ambiguous. You don't want every lexical binding that happens to bind a channel object to destroy it at the end of the scope, a la (let ((ch (channel-slot-accessor obj))) ...). You want only those which "locally create" an object (by some distinguishable means) to destroy it at the end. I personally think that would imply some form indicating a specific scope type that determines an object lifetime, which really does boil back down to with-* forms again. It is the scope itself that rules the object lifetime, not the other way around. Examples from other languages would be C++ RAII (which is a usage style, not exactly an explicit feature), and Python's context managers. I think the Python example would be the closest to a style that would fit a CL abstraction, and this is actually something I've thought about for quite some time. This is off the cuff, but abstracting out the with-* style seems to boil down pretty neatly: (defmacro scope (var creation-form &body body) `(let ((,var ,creation-form)) (unwind-protect (progn ,@body) (unscope ,var))) (defgeneric unscope (obj)) So along with the standard construction form of an object, you'd define an UNSCOPE method for an instance of it, and the abstract (scope ch (make-channel ...) ...body...) should Just Work™. It's still built on top of UNWIND-PROTECT, but you as the programmer need not worry about it yourself. -- David Holz Director, Grindwork Corporation http://www.grindwork.com/
HI David, Yes, I think you put your finger on it. I had this odd sensation in the back of my mind, that I had actually seen this behavior before. And then I remembered C++-land. Automatic destructors called at function exit — a sort of leaving scope operation. So yes, you understand correctly. And the question arises, is there anything down in the bowels of Lisp, some little known feature, that can indicate scope exit? Your scode / unscope fits pretty well, but so does when - unwind-protect. I don’t see a lot of difference between them, except that you have a generic de-scoping function. What I was hoping for was something during compile time that would be called on scope-exit of bound variables. Something we might hook into. [ BTW, after completing that article I referenced, I had a deep sense of relief that I don’t live in C/C++ land very much anymore. I truly appreciate whatever Lisp has to offer. ] - DM
On Aug 25, 2016, at 20:13, David Holz <david.holz@grindwork.com> wrote:
On 08/25/2016 06:20 PM, David McClain wrote:
What I’m wondering about is being able to declare an item, a special kind of Constructor parameter, which refers to a system resource that needs scavenging, such that I could forego the WITH- formalism? Maybe I’m asking for too much here… how would we ever detect the item being stored? We’d need reference counting, and I tend to dislike that approach very much..
Anyone ever seen a system for resource reclamation that is better than WITH- and UNWIND-PROTECT?
- DM
If I'm understanding correctly, your ideal would be something like this:
(let ((channel (make-channel ...))) (do-stuff channel) (do-other-stuff channel))
and at the end of the lexical scope, due to the fact that the channel object has something special done to it during construction, it automatically calls disposal routines of the channel right at that point?
I think that's going to be tough, as well as ambiguous. You don't want every lexical binding that happens to bind a channel object to destroy it at the end of the scope, a la (let ((ch (channel-slot-accessor obj))) ...). You want only those which "locally create" an object (by some distinguishable means) to destroy it at the end. I personally think that would imply some form indicating a specific scope type that determines an object lifetime, which really does boil back down to with-* forms again. It is the scope itself that rules the object lifetime, not the other way around.
Examples from other languages would be C++ RAII (which is a usage style, not exactly an explicit feature), and Python's context managers. I think the Python example would be the closest to a style that would fit a CL abstraction, and this is actually something I've thought about for quite some time.
This is off the cuff, but abstracting out the with-* style seems to boil down pretty neatly:
(defmacro scope (var creation-form &body body) `(let ((,var ,creation-form)) (unwind-protect (progn ,@body) (unscope ,var)))
(defgeneric unscope (obj))
So along with the standard construction form of an object, you'd define an UNSCOPE method for an instance of it, and the abstract (scope ch (make-channel ...) ...body...) should Just Work™. It's still built on top of UNWIND-PROTECT, but you as the programmer need not worry about it yourself.
-- David Holz Director, Grindwork Corporation http://www.grindwork.com/
Would finalisers do the trick? Not part of the language itself, but many implementations support them. Various caveats apply. In particular you cannot guarantee when (or I guess whether) any particular resource will be finalised; only that the GC will get around to most resources in the fullness of time. So this solution might not meet your needs. - nick
On 26 Aug 2016, at 06:38, David McClain <dbm@refined-audiometrics.com> wrote:
HI David,
Yes, I think you put your finger on it. I had this odd sensation in the back of my mind, that I had actually seen this behavior before. And then I remembered C++-land. Automatic destructors called at function exit — a sort of leaving scope operation.
So yes, you understand correctly. And the question arises, is there anything down in the bowels of Lisp, some little known feature, that can indicate scope exit?
Your scode / unscope fits pretty well, but so does when - unwind-protect. I don’t see a lot of difference between them, except that you have a generic de-scoping function.
What I was hoping for was something during compile time that would be called on scope-exit of bound variables. Something we might hook into.
[ BTW, after completing that article I referenced, I had a deep sense of relief that I don’t live in C/C++ land very much anymore. I truly appreciate whatever Lisp has to offer. ]
- DM
On Aug 25, 2016, at 20:13, David Holz <david.holz@grindwork.com> wrote:
On 08/25/2016 06:20 PM, David McClain wrote:
What I’m wondering about is being able to declare an item, a special kind of Constructor parameter, which refers to a system resource that needs scavenging, such that I could forego the WITH- formalism? Maybe I’m asking for too much here… how would we ever detect the item being stored? We’d need reference counting, and I tend to dislike that approach very much..
Anyone ever seen a system for resource reclamation that is better than WITH- and UNWIND-PROTECT?
- DM
If I'm understanding correctly, your ideal would be something like this:
(let ((channel (make-channel ...))) (do-stuff channel) (do-other-stuff channel))
and at the end of the lexical scope, due to the fact that the channel object has something special done to it during construction, it automatically calls disposal routines of the channel right at that point?
I think that's going to be tough, as well as ambiguous. You don't want every lexical binding that happens to bind a channel object to destroy it at the end of the scope, a la (let ((ch (channel-slot-accessor obj))) ...). You want only those which "locally create" an object (by some distinguishable means) to destroy it at the end. I personally think that would imply some form indicating a specific scope type that determines an object lifetime, which really does boil back down to with-* forms again. It is the scope itself that rules the object lifetime, not the other way around.
Examples from other languages would be C++ RAII (which is a usage style, not exactly an explicit feature), and Python's context managers. I think the Python example would be the closest to a style that would fit a CL abstraction, and this is actually something I've thought about for quite some time.
This is off the cuff, but abstracting out the with-* style seems to boil down pretty neatly:
(defmacro scope (var creation-form &body body) `(let ((,var ,creation-form)) (unwind-protect (progn ,@body) (unscope ,var)))
(defgeneric unscope (obj))
So along with the standard construction form of an object, you'd define an UNSCOPE method for an instance of it, and the abstract (scope ch (make-channel ...) ...body...) should Just Work™. It's still built on top of UNWIND-PROTECT, but you as the programmer need not worry about it yourself.
-- David Holz Director, Grindwork Corporation http://www.grindwork.com/
Yes, finalizers can sometimes work. I have those in place. But they are tricky. Suppose I hand off a local channel to another thread to utilize of communication between us. Now there are two copies of that channel pointer. So when my local use goes out of scope, there is still one copy in use, by that thread. And that thread is likely just hanging, waiting for something to come across that channel, that never will, since it just went out of scope from the sending side. Hence GC will never have the opportunity to finalize. Instead, you have to invent contorted references to the channel that can be neutralized by the sender when it goes out of scope. And I say contorted because these references have to be complex enough that the compiler won’t perform some stack optimization and produce an inadvertent secondary reference. So, for example, it isn’t sufficient to make a reference to a channel as (list ch) / (car ch). That seems to become unwrapped at the receiver side by things like handler-case, ignore-errors. Rather you have to use a functional closure like (lambda () ch) / funcall. And now we’ve lost the symmetry of use on each side of the channel. Not only that, but now we have to understand Lisp implementation details that we never needed to know before. And so we likely aren’t semantically portable. Secondly, as you mentioned, not all Lisp support finalization, or not very well. Lispworks does fine, but I find SBCL particularly weak in that you only get told about an object being scavenged after it has already happened. Hence you have to keep a display array and use indirect references in SBCL. What I recall from C++ world is that the automatic destructor calls at scope exit only happen on stack allocated objects, not pointers. Makes sense, since pointers allow for the possibility of alias pointers all over the place. Well, that’s exactly the situation we have in most Lisps too. In fact there can be no such thing as a stack allocated object, even if it really is allocated on the stack. So then we have to invent reference counting to be sure we don’t destroy a shared object pointer too soon. What I’m asking for really isn’t safe in Lisp. The best we can do, it seems, is what David proposes with his Scope macro, or my WITH- / UNWIND-PROTECT. - DM
On Aug 25, 2016, at 22:42, Nick Levine <nick@nicklevine.org> wrote:
Would finalisers do the trick?
Not part of the language itself, but many implementations support them.
Various caveats apply. In particular you cannot guarantee when (or I guess whether) any particular resource will be finalised; only that the GC will get around to most resources in the fullness of time. So this solution might not meet your needs.
- nick
>From my perspective, there are two orthogonal things going on here: 1. The idea of a "resource" with a well-defined protocol for allocation, initialization, deinitialization, and deallocation. Genera (and CLIM) had macrology for this: defresource to define a resource, using-resource to use it in a "safe" way such that all those things happened in order. 2. Having let behave like using-resource. It would be perfectly simple to write a let macro that shadows cl:let, which tracks allocation/initialization of resources, evaluates the body, and then calls the deinitializer/deallocator. How you implement "resources" is up to you. :-) On Fri, Aug 26, 2016 at 7:52 AM, David McClain <dbm@refined-audiometrics.com > wrote: > Yes, finalizers can sometimes work. I have those in place. But they are > tricky. Suppose I hand off a local channel to another thread to utilize of > communication between us. Now there are two copies of that channel pointer. > So when my local use goes out of scope, there is still one copy in use, by > that thread. And that thread is likely just hanging, waiting for something > to come across that channel, that never will, since it just went out of > scope from the sending side. Hence GC will never have the opportunity to > finalize. > > Instead, you have to invent contorted references to the channel that can > be neutralized by the sender when it goes out of scope. And I say > contorted because these references have to be complex enough that the > compiler won’t perform some stack optimization and produce an inadvertent > secondary reference. > > So, for example, it isn’t sufficient to make a reference to a channel as > (list ch) / (car ch). That seems to become unwrapped at the receiver side > by things like handler-case, ignore-errors. Rather you have to use a > functional closure like (lambda () ch) / funcall. And now we’ve lost the > symmetry of use on each side of the channel. > > Not only that, but now we have to understand Lisp implementation details > that we never needed to know before. And so we likely aren’t semantically > portable. > > Secondly, as you mentioned, not all Lisp support finalization, or not very > well. Lispworks does fine, but I find SBCL particularly weak in that you > only get told about an object being scavenged after it has already > happened. Hence you have to keep a display array and use indirect > references in SBCL. > > What I recall from C++ world is that the automatic destructor calls at > scope exit only happen on stack allocated objects, not pointers. Makes > sense, since pointers allow for the possibility of alias pointers all over > the place. Well, that’s exactly the situation we have in most Lisps too. In > fact there can be no such thing as a stack allocated object, even if it > really is allocated on the stack. > > So then we have to invent reference counting to be sure we don’t destroy a > shared object pointer too soon. > > What I’m asking for really isn’t safe in Lisp. The best we can do, it > seems, is what David proposes with his Scope macro, or my WITH- / > UNWIND-PROTECT. > > - DM > > > > On Aug 25, 2016, at 22:42, Nick Levine <nick@nicklevine.org> wrote: > > > > Would finalisers do the trick? > > > > Not part of the language itself, but many implementations support them. > > > > Various caveats apply. In particular you cannot guarantee when (or I > guess whether) any particular resource will be finalised; only that the GC > will get around to most resources in the fullness of time. So this solution > might not meet your needs. > > > > - nick > > > > >
I knew there would be some old-dog out there that had a few tricks up his sleeve. Sadly, I am post-Genera generation. Thanks for chiming in there Scott. Gives me a lot to chew on. - DM
On Aug 26, 2016, at 06:17, Scott McKay <swmckay@gmail.com> wrote:
From my perspective, there are two orthogonal things going on here: 1. The idea of a "resource" with a well-defined protocol for allocation, initialization, deinitialization, and deallocation. Genera (and CLIM) had macrology for this: defresource to define a resource, using-resource to use it in a "safe" way such that all those things happened in order. 2. Having let behave like using-resource.
It would be perfectly simple to write a let macro that shadows cl:let, which tracks allocation/initialization of resources, evaluates the body, and then calls the deinitializer/deallocator. How you implement "resources" is up to you. :-)
On Fri, Aug 26, 2016 at 7:52 AM, David McClain <dbm@refined-audiometrics.com <mailto:dbm@refined-audiometrics.com>> wrote: Yes, finalizers can sometimes work. I have those in place. But they are tricky. Suppose I hand off a local channel to another thread to utilize of communication between us. Now there are two copies of that channel pointer. So when my local use goes out of scope, there is still one copy in use, by that thread. And that thread is likely just hanging, waiting for something to come across that channel, that never will, since it just went out of scope from the sending side. Hence GC will never have the opportunity to finalize.
Instead, you have to invent contorted references to the channel that can be neutralized by the sender when it goes out of scope. And I say contorted because these references have to be complex enough that the compiler won’t perform some stack optimization and produce an inadvertent secondary reference.
So, for example, it isn’t sufficient to make a reference to a channel as (list ch) / (car ch). That seems to become unwrapped at the receiver side by things like handler-case, ignore-errors. Rather you have to use a functional closure like (lambda () ch) / funcall. And now we’ve lost the symmetry of use on each side of the channel.
Not only that, but now we have to understand Lisp implementation details that we never needed to know before. And so we likely aren’t semantically portable.
Secondly, as you mentioned, not all Lisp support finalization, or not very well. Lispworks does fine, but I find SBCL particularly weak in that you only get told about an object being scavenged after it has already happened. Hence you have to keep a display array and use indirect references in SBCL.
What I recall from C++ world is that the automatic destructor calls at scope exit only happen on stack allocated objects, not pointers. Makes sense, since pointers allow for the possibility of alias pointers all over the place. Well, that’s exactly the situation we have in most Lisps too. In fact there can be no such thing as a stack allocated object, even if it really is allocated on the stack.
So then we have to invent reference counting to be sure we don’t destroy a shared object pointer too soon.
What I’m asking for really isn’t safe in Lisp. The best we can do, it seems, is what David proposes with his Scope macro, or my WITH- / UNWIND-PROTECT.
- DM
On Aug 25, 2016, at 22:42, Nick Levine <nick@nicklevine.org <mailto:nick@nicklevine.org>> wrote:
Would finalisers do the trick?
Not part of the language itself, but many implementations support them.
Various caveats apply. In particular you cannot guarantee when (or I guess whether) any particular resource will be finalised; only that the GC will get around to most resources in the fullness of time. So this solution might not meet your needs.
- nick
Please, I prefer the term dinosaur. :-) Here's defresource from CLIM: - https://github.com/franzinc/clim2/blob/master/clim/defresource.lisp On Fri, Aug 26, 2016 at 9:26 AM, David McClain <dbm@refined-audiometrics.com
wrote:
I knew there would be some old-dog out there that had a few tricks up his sleeve. Sadly, I am post-Genera generation. Thanks for chiming in there Scott. Gives me a lot to chew on.
- DM
On Aug 26, 2016, at 06:17, Scott McKay <swmckay@gmail.com> wrote:
From my perspective, there are two orthogonal things going on here: 1. The idea of a "resource" with a well-defined protocol for allocation, initialization, deinitialization, and deallocation. Genera (and CLIM) had macrology for this: defresource to define a resource, using-resource to use it in a "safe" way such that all those things happened in order. 2. Having let behave like using-resource.
It would be perfectly simple to write a let macro that shadows cl:let, which tracks allocation/initialization of resources, evaluates the body, and then calls the deinitializer/deallocator. How you implement "resources" is up to you. :-)
On Fri, Aug 26, 2016 at 7:52 AM, David McClain < dbm@refined-audiometrics.com> wrote:
Yes, finalizers can sometimes work. I have those in place. But they are tricky. Suppose I hand off a local channel to another thread to utilize of communication between us. Now there are two copies of that channel pointer. So when my local use goes out of scope, there is still one copy in use, by that thread. And that thread is likely just hanging, waiting for something to come across that channel, that never will, since it just went out of scope from the sending side. Hence GC will never have the opportunity to finalize.
Instead, you have to invent contorted references to the channel that can be neutralized by the sender when it goes out of scope. And I say contorted because these references have to be complex enough that the compiler won’t perform some stack optimization and produce an inadvertent secondary reference.
So, for example, it isn’t sufficient to make a reference to a channel as (list ch) / (car ch). That seems to become unwrapped at the receiver side by things like handler-case, ignore-errors. Rather you have to use a functional closure like (lambda () ch) / funcall. And now we’ve lost the symmetry of use on each side of the channel.
Not only that, but now we have to understand Lisp implementation details that we never needed to know before. And so we likely aren’t semantically portable.
Secondly, as you mentioned, not all Lisp support finalization, or not very well. Lispworks does fine, but I find SBCL particularly weak in that you only get told about an object being scavenged after it has already happened. Hence you have to keep a display array and use indirect references in SBCL.
What I recall from C++ world is that the automatic destructor calls at scope exit only happen on stack allocated objects, not pointers. Makes sense, since pointers allow for the possibility of alias pointers all over the place. Well, that’s exactly the situation we have in most Lisps too. In fact there can be no such thing as a stack allocated object, even if it really is allocated on the stack.
So then we have to invent reference counting to be sure we don’t destroy a shared object pointer too soon.
What I’m asking for really isn’t safe in Lisp. The best we can do, it seems, is what David proposes with his Scope macro, or my WITH- / UNWIND-PROTECT.
- DM
On Aug 25, 2016, at 22:42, Nick Levine <nick@nicklevine.org> wrote:
Would finalisers do the trick?
Not part of the language itself, but many implementations support them.
Various caveats apply. In particular you cannot guarantee when (or I guess whether) any particular resource will be finalised; only that the GC will get around to most resources in the fullness of time. So this solution might not meet your needs.
- nick
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml
Thanks for both of those references…. Dinosaur indeed… I’m old enough too, that I need to avoid walking near the La Brea Tar Pits… - DM
On Aug 26, 2016, at 06:42, Hans Hübner <hans.huebner@gmail.com> wrote:
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml <https://hanshuebner.github.io/lmman/resour.xml>
Custodians from Racket looks like what is described https://docs.racket-lang.org/reference/eval-model.html#%28part._custodian-mo... On Fri, Aug 26, 2016 at 5:51 PM, David McClain <dbm@refined-audiometrics.com> wrote:
Thanks for both of those references…. Dinosaur indeed… I’m old enough too, that I need to avoid walking near the La Brea Tar Pits…
- DM
On Aug 26, 2016, at 06:42, Hans Hübner <hans.huebner@gmail.com> wrote:
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml
Hi Ala, Yes, I looked at the Custodian concept. Not quite what I have in mind. I implemented some Custodians in Lisp to see how they feel in the code. And as a last resort, that would certainly be a way to go. But the level of control is very coarse. I have the impression that Scheme does not have UNWIND-PROTECT. And in a situation where you have literally thousands of ephemeral resource allocations between points of custodial interaction, then the custodian tables consume huge amounts of useless garbage that I’d prefer to hand back to GC. So really, GC, Finalization, and the ever-favorite UNWIND-PROTECT offer much finer grained control. If you want to see the custodians, just give me a holler. They are free for the asking. Custodians.lisp - DM
On Aug 26, 2016, at 13:42, Ala'a Mohammad <amalawi@gmail.com> wrote:
Custodians from Racket looks like what is described https://docs.racket-lang.org/reference/eval-model.html#%28part._custodian-mo...
On Fri, Aug 26, 2016 at 5:51 PM, David McClain <dbm@refined-audiometrics.com> wrote:
Thanks for both of those references…. Dinosaur indeed… I’m old enough too, that I need to avoid walking near the La Brea Tar Pits…
- DM
On Aug 26, 2016, at 06:42, Hans Hübner <hans.huebner@gmail.com> wrote:
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml
… also… on the Custodian concept — they forcibly shut down threads. I have also looked closely at how that plays out in application code, and it isn’t a given that when a thread is hung waiting on a channel that dies, killing the thread is the right thing to do. What I ended up doing in my Reppy Channels is that when a channel is discarded, the executive runs through the waiting queues and releases each of the waiting threads with a rendezvous failure indication. Let the thread resume running, and let it decide whether that warrants dying or not. In many cases there are still vital system structures that need cleaning up before that thread dies. Think of commit / rollback transactional processing... So I don’t really like to abruptly terminate threads unless I must. In many ways, the old fashioned single-process Lisp was much easier to code. But the reality today is multi-thread on SMP processors, and I do love the speedup that you can gain from proper use of parallelism. - DM
On Aug 26, 2016, at 15:51, David McClain <dbm@refined-audiometrics.com> wrote:
Hi Ala,
Yes, I looked at the Custodian concept. Not quite what I have in mind. I implemented some Custodians in Lisp to see how they feel in the code. And as a last resort, that would certainly be a way to go. But the level of control is very coarse. I have the impression that Scheme does not have UNWIND-PROTECT.
And in a situation where you have literally thousands of ephemeral resource allocations between points of custodial interaction, then the custodian tables consume huge amounts of useless garbage that I’d prefer to hand back to GC. So really, GC, Finalization, and the ever-favorite UNWIND-PROTECT offer much finer grained control.
If you want to see the custodians, just give me a holler. They are free for the asking. Custodians.lisp
- DM
On Aug 26, 2016, at 13:42, Ala'a Mohammad <amalawi@gmail.com> wrote:
Custodians from Racket looks like what is described https://docs.racket-lang.org/reference/eval-model.html#%28part._custodian-mo...
On Fri, Aug 26, 2016 at 5:51 PM, David McClain <dbm@refined-audiometrics.com> wrote:
Thanks for both of those references…. Dinosaur indeed… I’m old enough too, that I need to avoid walking near the La Brea Tar Pits…
- DM
On Aug 26, 2016, at 06:42, Hans Hübner <hans.huebner@gmail.com> wrote:
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml
… I think the notion of killing hung threads on dead channels originated in CML and carried over into Scheme. Both of those older systems used cheap Green Threads, and that encouraged a programming style wherein anything that needs doing, but might never complete, just fire off onto a new green thread. In our case in Lisp with Native Threads, the threads are heavier objects and we need to minimize the gratuitous spawning of new threads. So my code errs on the side of loading up threads with a hefty amount of work whenever possible. Make them count, to amortize the cost in time and memory. There is a sweet spot, if you can find it, in the throughput / delay curve, around the knee, just before it goes asymptotic. At maximum throughput, the delays stretch to infinity for any one task. Too many Native Threads will push you toward that asymptote. - DM
On Aug 26, 2016, at 16:06, David McClain <dbm@refined-audiometrics.com> wrote:
… also… on the Custodian concept — they forcibly shut down threads. I have also looked closely at how that plays out in application code, and it isn’t a given that when a thread is hung waiting on a channel that dies, killing the thread is the right thing to do.
What I ended up doing in my Reppy Channels is that when a channel is discarded, the executive runs through the waiting queues and releases each of the waiting threads with a rendezvous failure indication. Let the thread resume running, and let it decide whether that warrants dying or not. In many cases there are still vital system structures that need cleaning up before that thread dies. Think of commit / rollback transactional processing...
So I don’t really like to abruptly terminate threads unless I must. In many ways, the old fashioned single-process Lisp was much easier to code. But the reality today is multi-thread on SMP processors, and I do love the speedup that you can gain from proper use of parallelism.
- DM
On Aug 26, 2016, at 15:51, David McClain <dbm@refined-audiometrics.com> wrote:
Hi Ala,
Yes, I looked at the Custodian concept. Not quite what I have in mind. I implemented some Custodians in Lisp to see how they feel in the code. And as a last resort, that would certainly be a way to go. But the level of control is very coarse. I have the impression that Scheme does not have UNWIND-PROTECT.
And in a situation where you have literally thousands of ephemeral resource allocations between points of custodial interaction, then the custodian tables consume huge amounts of useless garbage that I’d prefer to hand back to GC. So really, GC, Finalization, and the ever-favorite UNWIND-PROTECT offer much finer grained control.
If you want to see the custodians, just give me a holler. They are free for the asking. Custodians.lisp
- DM
On Aug 26, 2016, at 13:42, Ala'a Mohammad <amalawi@gmail.com> wrote:
Custodians from Racket looks like what is described https://docs.racket-lang.org/reference/eval-model.html#%28part._custodian-mo...
On Fri, Aug 26, 2016 at 5:51 PM, David McClain <dbm@refined-audiometrics.com> wrote:
Thanks for both of those references…. Dinosaur indeed… I’m old enough too, that I need to avoid walking near the La Brea Tar Pits…
- DM
On Aug 26, 2016, at 06:42, Hans Hübner <hans.huebner@gmail.com> wrote:
The Lisp Machine Manual has a nice chapter on resources that may be of interest: https://hanshuebner.github.io/lmman/resour.xml
participants (6)
-
Ala'a Mohammad -
David Holz -
David McClain -
Hans Hübner -
Nick Levine -
Scott McKay