The following question was raised during my development of an asynchronous library https://github.com/lokedhs/cl-rabbit-async I'm currently building for RabbitMQ.
In summary, the library allows you to create an object of type ASYNCH-CONNECTION, from which instances of ASYNC-CHANNEL can be retrieved. A connection holds a reference to all channels that it uses, and each channel holds a reference to its connection. The connection object has a pointer to a native CFFI object that for the underlying connection (my library is built on the RabbitMQ C API).
My question is: Should I use trivial-garbage to create a GC hook for the connection object so that if the user of the library forgets to close the connection, it will get closed eventually once it's GC'ed?
I can see arguments for both behaviours:
- It's a bad idea, since losing the reference to the connection object means that the program is broken, and silently cleaning up the underlying connection might hide the fact there is a bug (if the GC doesn't run often enough I might have hundreds or even thousands of lingering connections) - On the other hand, it might be a good idea since Lisp developers often use the REPL to experiment, so it's easy to accidentally lose a reference to an object during testing. Thus, using the GC hook will improve the stability of one's development environment.
To me, there is no strictly correct answer to the question, which is why I'm asking for suggestions from you guys.
Regards, Elias (loke on #lisp)
Hi!
On 2015-07-04 19:30:01+0800, Elias Mårtenson wrote:
My question is: Should I use trivial-garbage to create a GC hook for the connection object so that if the user of the library forgets to close the connection, it will get closed eventually once it's GC'ed?
My first impulse is to say “that is what garbage collection is there for”. It needs not matter whether the object resides in C or Lisp land or what kind of resource it is; if the system can identify that it is garbage, it should collect it.
The alternative would also mean that it is difficult for the system to recover from the mistake without restart.
Additionally, the API might benefit from a with-async-connection and a with-open-channel macro.
Yours aye
Svante
On Sat, Jul 4, 2015 at 8:35 AM, Svante v. Erichsen Svante.v.Erichsen@web.de wrote:
Hi!
On 2015-07-04 19:30:01+0800, Elias Mårtenson wrote:
My question is: Should I use trivial-garbage to create a GC hook for the connection object so that if the user of the library forgets to close the connection, it will get closed eventually once it's GC'ed?
My first impulse is to say “that is what garbage collection is there for”. It needs not matter whether the object resides in C or Lisp land or what kind of resource it is; if the system can identify that it is garbage, it should collect it.
The alternative would also mean that it is difficult for the system to recover from the mistake without restart.
Additionally, the API might benefit from a with-async-connection and a with-open-channel macro.
It might be a good idea to have a GC hook, but you might want your GC hook to log and/or count dropped connections, so you can later detect and fix the underlying issues. Belt and suspenders.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org It's not "privilege". It's *capital*. It's not forcefully taken away from others in a negative sum game. It's created, preserved and transmitted in a positive sum game.
Hi,
Did you consider adding a configuration variable that turns this feature on and off and can be used to set up environments differently for dev/test/prod?
--- Vsevolod Dyomkin +38-096-111-41-56 skype, twitter: vseloved
On Sat, Jul 4, 2015 at 4:30 AM, Elias Mårtenson lokedhs@gmail.com wrote:
The following question was raised during my development of an asynchronous library https://github.com/lokedhs/cl-rabbit-async I'm currently building for RabbitMQ.
In summary, the library allows you to create an object of type ASYNCH-CONNECTION, from which instances of ASYNC-CHANNEL can be retrieved. A connection holds a reference to all channels that it uses, and each channel holds a reference to its connection. The connection object has a pointer to a native CFFI object that for the underlying connection (my library is built on the RabbitMQ C API).
My question is: Should I use trivial-garbage to create a GC hook for the connection object so that if the user of the library forgets to close the connection, it will get closed eventually once it's GC'ed?
I can see arguments for both behaviours:
- It's a bad idea, since losing the reference to the connection object
means that the program is broken, and silently cleaning up the underlying connection might hide the fact there is a bug (if the GC doesn't run often enough I might have hundreds or even thousands of lingering connections)
- On the other hand, it might be a good idea since Lisp developers
often use the REPL to experiment, so it's easy to accidentally lose a reference to an object during testing. Thus, using the GC hook will improve the stability of one's development environment.
To me, there is no strictly correct answer to the question, which is why I'm asking for suggestions from you guys.
Regards, Elias (loke on #lisp)
On 5 July 2015 at 01:20, Vsevolod Dyomkin vseloved@gmail.com wrote:
Did you consider adding a configuration variable that turns this feature on
and off and can be used to set up environments differently for dev/test/prod?
No, I didn't. Mainly because I really don't think one option is better for dev and another for prod.
Based on your replies, it seem like most people would expect a Lisp library to simply *work* as best as possible rather than do exactly what it's told.
Therefore, I'm starting lean towards using a GC hook here, even though the complexity of the GC'ed object is much more complex (the connection having a processing thread, for example) than the only other time I've actually used this https://github.com/lokedhs/cl-gss/blob/master/src/cl-gss.lisp#L48 feature for effect (which only had a simple native object that needed to be freed).
Regards, Elias
A somewhat off-topic reply, but if you're working on a CL / RabbitMQ implementation then do feel free to raid my old CL-RABBIT sources for working parts, in case any of them turn out to be of use.
http://nicklevine.org/cl-rabbit/
- nick
On 5 July 2015 at 14:56, Nick Levine nick@nicklevine.org wrote:
A somewhat off-topic reply, but if you're working on a CL / RabbitMQ implementation then do feel free to raid my old CL-RABBIT sources for working parts, in case any of them turn out to be of use.
Currently I'm linking to rabbitmq-c which is the C API for RabbitMQ. Unfortunately this library is not entirely ideal as a base for implementing a multithreaded async library. Your library avoids this, and I have definitely been thinking about moving away from rabbitmq-c. If I choose to do that, I will definitely have a look at your code.
Regards, Elias
On 15-07-04 07:30 AM, Elias Mårtenson wrote:
The following question was raised during my development of an asynchronous library https://github.com/lokedhs/cl-rabbit-async I'm currently building for RabbitMQ.
Aside / rhetorical: Are you trying to achieve a certain level of "efficiency"? If so, read on.
I've been working on flow-based concepts for 25+ years (I use various languages, incl. CL, as "assemblers").
The FBP model is kinda-like Actors (or CSP), with the embellishment that siblings cannot "see" or know about one another - they must ask their common parent to distribute messages.
FBP essentially discards most of the O/S and uses a small handful of concepts, e.g. a scheduler, components, ports (queues), ready/wait queues.
A very small "kernel" to handle only this set of concepts - entirely eschewing the use of processes - can be easily built. If you can read C, then peruse https://github.com/guitarvydas/collate-fbp-classic for the most bare-bones implementation of these concepts I have come up with to date (the example is slightly more complicated than necessary, because the "Collate" problem (page 91 of Paul Morrison's FBP book) requires the use of bounded buffers). If you don't read C, and are interested, ask me.
pt
On Sun, Jul 5, 2015 at 10:48 AM, Paul Tarvydas paultarvydas@gmail.com wrote:
The FBP model is kinda-like Actors (or CSP), with the embellishment that siblings cannot "see" or know about one another - they must ask their common parent to distribute messages.
FBP essentially discards most of the O/S and uses a small handful of concepts, e.g. a scheduler, components, ports (queues), ready/wait queues.
A very small "kernel" to handle only this set of concepts - entirely eschewing the use of processes - can be easily built. If you can read C, then peruse https://github.com/guitarvydas/collate-fbp-classic for the most bare-bones implementation of these concepts I have come up with to date (the example is slightly more complicated than necessary, because the "Collate" problem (page 91 of Paul Morrison's FBP book) requires the use of bounded buffers). If you don't read C, and are interested, ask me.
Interesting. How does that compare to the Chemical Abstract Machine and/or Jean-Bernard Stefani's Kell Calculus?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The ancients stole all our ideas from us. — Mark Twain
On 5 July 2015 at 22:48, Paul Tarvydas paultarvydas@gmail.com wrote:
On 15-07-04 07:30 AM, Elias Mårtenson wrote:
The following question was raised during my development of an asynchronous library https://github.com/lokedhs/cl-rabbit-async I'm currently building for RabbitMQ.
Aside / rhetorical: Are you trying to achieve a certain level of "efficiency"? If so, read on.
No, I'm not. I'm not trying to build something using Erlang-style async-I/O. I'm merely referring to a library that is thread-safe and which can use a single connection to RabbitMQ asynchronously from multiple threads (rabbitmq-c normally enforces a synchronous workflow with all of its calls being blocking).
That said, I'l still reading on. :-)
-
Elias Mårtenson -
Faré -
Nick Levine -
Paul Tarvydas -
Svante v. Erichsen -
Vsevolod Dyomkin