Hi, just two quick question: 1. Is it possible to disable the threading model and have multiple thread to eval the code or have one ECL per thread? 2. Can the garbage collector be disabled at all and have the memory managed in a reference counting fashion?
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory: * I can not spawn new thread but I've got concurrent thread to execute ecl call. * A component that bring GC would probably not be accepted and I'm wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
Can you point me out where to look in the source code to understand those aspect of ECL?
Thanks in advance. f.
Hello,
Bix writes:
Hi, just two quick question:
- Is it possible to disable the threading model and have multiple thread
to eval the code or have one ECL per thread?
You can build ECL without the threading support. You have to pass "--enable-threads=no" to the configure script. You can eval the code from the different threads but it's unsafe (concurrency).
Best solution would be to create a dedicated thread which will manage ECL calls.
You can't run separate ECL images from the same application AFAIK. It is a nice feature proposal though.
- Can the garbage collector be disabled at all and have the memory
managed in a reference counting fashion?
GC manages only the objects created by ECL. The collector won't scan the data sectors and you have to notify it if you want your data to be collectable.
GC may be disabled by calling the function `GC_disable()'. Explicit call to the GC
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory:
- I can not spawn new thread but I've got concurrent thread to execute
ecl call.
As above: compile ECL without the threads support and manage it from the separate thread. Is that what you want?
- A component that bring GC would probably not be accepted and I'm
wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
ECL uses Boehm-Demers-Weiser Garbage Collector. More information about it might be found here: http://www.hboehm.info/gc/ .
Can you point me out where to look in the source code to understand those aspect of ECL?
Of course. Keep in mind though that it's inherited codebase and my information might be inaccurate (sorry about that).
src/c/alloc.d - obsolete in-house GC which doesn't work src/c/alloc_2.d - necessary bridging between bdwgc and ECL. `si_gc' runs calls the GC explicitly in a safe manner src/c/main.d - `cl_boot' function starts the lisp, ecl_option_values contains configurable options src/c/unixint.d - handling the signals. Documentation available here: https://common-lisp.net/project/ecl/manual/ch32.html
There is some documentation about GC here: https://common-lisp.net/project/ecl/manual/ch30.html#ext.memory.intro but it's rather scarce. Also a very important note on handling signals: https://common-lisp.net/project/ecl/manual/ch32s04.html
Thanks in advance. f.
Best regards, Daniel
Thanks for the info. Having a single thread that manage the request may be an option. I have some concern that it would not keep up with a high volume of requests.
Cheers Bix
----- Messaggio originale ----- Da: "Daniel Kochmański" daniel@turtleware.eu Inviato: 25.10.2015 13:05 A: "Bix" strabixbox@yahoo.com Cc: "ecl-devel@common-lisp.net" ecl-devel@common-lisp.net Oggetto: Re: GC and Threading
Hello,
Bix writes:
Hi, just two quick question:
- Is it possible to disable the threading model and have multiple thread
to eval the code or have one ECL per thread?
You can build ECL without the threading support. You have to pass "--enable-threads=no" to the configure script. You can eval the code from the different threads but it's unsafe (concurrency).
Best solution would be to create a dedicated thread which will manage ECL calls.
You can't run separate ECL images from the same application AFAIK. It is a nice feature proposal though.
- Can the garbage collector be disabled at all and have the memory
managed in a reference counting fashion?
GC manages only the objects created by ECL. The collector won't scan the data sectors and you have to notify it if you want your data to be collectable.
GC may be disabled by calling the function `GC_disable()'. Explicit call to the GC
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory:
- I can not spawn new thread but I've got concurrent thread to execute
ecl call.
As above: compile ECL without the threads support and manage it from the separate thread. Is that what you want?
- A component that bring GC would probably not be accepted and I'm
wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
ECL uses Boehm-Demers-Weiser Garbage Collector. More information about it might be found here: http://www.hboehm.info/gc/ .
Can you point me out where to look in the source code to understand those aspect of ECL?
Of course. Keep in mind though that it's inherited codebase and my information might be inaccurate (sorry about that).
src/c/alloc.d - obsolete in-house GC which doesn't work src/c/alloc_2.d - necessary bridging between bdwgc and ECL. `si_gc' runs calls the GC explicitly in a safe manner src/c/main.d - `cl_boot' function starts the lisp, ecl_option_values contains configurable options src/c/unixint.d - handling the signals. Documentation available here: https://common-lisp.net/project/ecl/manual/ch32.html
There is some documentation about GC here: https://common-lisp.net/project/ecl/manual/ch30.html#ext.memory.intro but it's rather scarce. Also a very important note on handling signals: https://common-lisp.net/project/ecl/manual/ch32s04.html
Thanks in advance. f.
Best regards, Daniel
On 25/10/15 09:28, Bix wrote:
Hi, just two quick question:
- Is it possible to disable the threading model and have multiple thread
to eval the code or have one ECL per thread?
Not, yes, probably not. Check: https://common-lisp.net/project/ecl/manual/ch33.html reading cl_boot should be instructive.
Being able to have several ECL instances per process could be interesting: you would have multiple heaps, managed in different threads, and some lisp data could be shared from one instance to another (with some interesting consequences on teh garbage collection occuring in each instance). There would also be some fun with signals, if having to deal with them between ecl and the host program wasn't enough fun already.
Or alternatively, if you manage a single heap, why would you want to have several instances?
- Can the garbage collector be disabled at all and have the memory
managed in a reference counting fashion?
Ecl uses Boehm GC, so it's rather modular, and I guess you could link in a stub library. You could add a dealloc function, and manage the memory manually. Good luck.
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory:
- I can not spawn new thread but I've got concurrent thread to execute
ecl call.
Ecl doesn't create any thread on its own. Only lisp programs may do so. Do not use libraries that create threads and you'll be ok.
- A component that bring GC would probably not be accepted
Then I would suggest to resign and find a better job. Because, yes, you can disable the garbage collector, but I wouldn't wish that on my worst enemy.
and I'm wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
BoehmGC doesn't seem to offer a lot of options there. It has the option to use a generational garbage collector.
Can you point me out where to look in the source code to understand those aspect of ECL?
src/bdwgc contains a copy of the Boehm-Demers-Weiser Garbage Collector src/bdwgc/include/gc.h shows the API you could implement or stub to disable it from the point of view of ecl. Notice that it's mostly GC_MALLOC, and there's a GC_FREE (never used in ecl, you you would have to provide a lisp function calling it to free the lisp objects).
(And don't count on reference counting, most lisp data structures are circular).
Pascal J. Bourguignon writes:
On 25/10/15 09:28, Bix wrote:
Hi, just two quick question:
- Is it possible to disable the threading model and have multiple thread
to eval the code or have one ECL per thread?
Not, yes, probably not. Check: https://common-lisp.net/project/ecl/manual/ch33.html reading cl_boot should be instructive.
Being able to have several ECL instances per process could be interesting: you would have multiple heaps, managed in different threads, and some lisp data could be shared from one instance to another (with some interesting consequences on teh garbage collection occuring in each instance). There would also be some fun with signals, if having to deal with them between ecl and the host program wasn't enough fun already.
Wondering if I can have multiple ECL instance that share nothing. Signal are shared by definition, isn't ?
Or alternatively, if you manage a single heap, why would you want to have several instances?
true I almost convinced about a single heap.
- Can the garbage collector be disabled at all and have the memory
managed in a reference counting fashion?
Ecl uses Boehm GC, so it's rather modular, and I guess you could link in a stub library. You could add a dealloc function, and manage the memory manually. Good luck.
Yep, manage memory manually is impratical and as you said circular structures are one of the real problem.
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory:
- I can not spawn new thread but I've got concurrent thread to execute
ecl call.
Ecl doesn't create any thread on its own. Only lisp programs may do so. Do not use libraries that create threads and you'll be ok.
I played a bit with the boot option but I was unable to embed ECL and not have at least one additional thread spaw by the GC or it was the SIGNAL_HANDLING_THREAD ? However I always end up with at least two thread ( in Linux ), am I doing something wrong ?
- A component that bring GC would probably not be accepted
Then I would suggest to resign and find a better job. Because, yes, you can disable the garbage collector, but I wouldn't wish that on my worst enemy.
and I'm wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
BoehmGC doesn't seem to offer a lot of options there. It has the option to use a generational garbage collector.
I tried to build with --enable-gengc but I'm unable to build, I think that is Issue #81. I'm getting SegFault on "ecl_min compile" I debugged the code and so far I just find out that this is a consistent failure at cl_boot when it is doing this : env->bindings_array = si_make_vector
in si_make_vector it is doing ecl_array_allocself(x);
and in ecl_array_allocself the sefault happen when we are doing the assignment
case ecl_aet_object: x->array.self.t = alloc_pointerfull_memory(d);
I verified that before calling alloc_pointerfull_memory the assignment work, but after the call the assignment raise the signal. I also verified that x is not collected as it's flagged( I'm still double checking ) and my debugger show that x value does not change and still point to the same memory address.
any cue?
I just noticed the cl_boot options allow INCREMENTAL_GC what the difference between that one and the autoconf option --enable-gengc
Can you point me out where to look in the source code to understand those aspect of ECL?
src/bdwgc contains a copy of the Boehm-Demers-Weiser Garbage Collector src/bdwgc/include/gc.h shows the API you could implement or stub to disable it from the point of view of ecl. Notice that it's mostly GC_MALLOC, and there's a GC_FREE (never used in ecl, you you would have to provide a lisp function calling it to free the lisp objects).
(And don't count on reference counting, most lisp data structures are circular).
Kindest Regards bix
On Fri, 06 Nov 2015 00:12:40 +0100 Bix strabixbox@yahoo.com wrote:
Yep, manage memory manually is impratical and as you said circular structures are one of the real problem.
With reference counting these should be a problem, although libgc is a mark-and-sweep collector which should detect when elements are no longer referenced more effectively.
I played a bit with the boot option but I was unable to embed ECL and not have at least one additional thread spaw by the GC or it was the SIGNAL_HANDLING_THREAD ? However I always end up with at least two thread ( in Linux ), am I doing something wrong ?
ECL indeed starts a thread to handle signals when it is built with threads support.
I tried to build with --enable-gengc but I'm unable to build, I think that is Issue #81.
Unfortunately I presonally have no experience using the generational variant, but others here might have used it. I think that boehm-gc itself only supports the option on a few select operating systems.
I just noticed the cl_boot options allow INCREMENTAL_GC what the difference between that one and the autoconf option --enable-gengc
Presumably --enable-gengc allows to build optional support, and INCREMENTAL_GC permits to enable said support for that instance.
I also noticed your question about share-nothing; this is what would occur if you used actual OS processes for the instances.
On Sun, 25 Oct 2015 09:28:03 +0100 Bix strabixbox@yahoo.com wrote:
just two quick question:
- Is it possible to disable the threading model and have multiple
thread to eval the code or have one ECL per thread? 2. Can the garbage collector be disabled at all and have the memory managed in a reference counting fashion?
I like to embed the ECL in my multithread application but I've got very restrictive constrain about thread and memory:
- I can not spawn new thread but I've got concurrent thread to execute
ecl call.
- A component that bring GC would probably not be accepted and I'm
wondering if there is an alternative to that in ECL, or at least have more control how/when GC will be run.
Can you point me out where to look in the source code to understand those aspect of ECL?
If your OS supports real processes, you could fork() some processes dedicated for ECL, such that ECL's heap and gc wouldn't interfere with your application. Alternatively, you could use separate ECL-based server daemons to which your application would connect. ECL could then also either work non-threaded or threaded in that, or those processes, depending on your needs.
However, you'd also then have to invoke ECL in an RPC manner from the main application. An example is emacs+SLIME connecting to ECL+swank, which uses a simple s-exp based exchange protocol. Other examples would be using JSON messages, SOAP/XMLRPC, HTTP/REST, FastCGI, CORBA, etc, and of course you could implement your own text or binary communication/memory-sharing protocol, using the most appropriate IPC method for your use-case...
On the other hand, if you need ECL to have direct access to objects within your application's heap, you'd currently have no choice, your application would indeed dynamically link boehm-gc, and any thread calling into ECL would have to be "registered" with ECL (threads started by ECL itself are automatically registered).
It's also possible that your application might itself benefit from the GC; the library also remains friendly to code using explicit freeing.
Matthew Mondor writes:
On Sun, 25 Oct 2015 09:28:03 +0100 Bix strabixbox@yahoo.com wrote:
just two quick question: Can you point me out where to look in the source code to understand those aspect of ECL?
If your OS supports real processes, you could fork() some processes dedicated for ECL, such that ECL's heap and gc wouldn't interfere with your application. Alternatively, you could use separate ECL-based server daemons to which your application would connect. ECL could then also either work non-threaded or threaded in that, or those processes, depending on your needs.
However, you'd also then have to invoke ECL in an RPC manner from the main application. An example is emacs+SLIME connecting to ECL+swank, which uses a simple s-exp based exchange protocol. Other examples would be using JSON messages, SOAP/XMLRPC, HTTP/REST, FastCGI, CORBA, etc, and of course you could implement your own text or binary communication/memory-sharing protocol, using the most appropriate IPC method for your use-case...
I'm thinking about that possibilities but I really like to embed the engine in the very same engine to share the structures.
On the other hand, if you need ECL to have direct access to objects within your application's heap, you'd currently have no choice, your application would indeed dynamically link boehm-gc, and any thread calling into ECL would have to be "registered" with ECL (threads started by ECL itself are automatically registered).
I'm exploring that interesting possibility. I don't like that the GC halt all the other thread but only the ones involved in the embedding and calling of ECL. So I studying if it possible to register and de-register a thread. Given that my threads does not share anything I hope that I can have the thread de-register himself when the ecl part is no more needed so everything created by that thread can be collected.
Kind of: 1. Register my thread 2. Execute some lisp on ecl 3. Collect/copy he results 4. De-register
It's also possible that your application might itself benefit from the GC; the library also remains friendly to code using explicit freeing.
Kindest Regards bix
-
Bix -
Bixbox
-
Daniel Kochmański -
Matthew Mondor -
Pascal J. Bourguignon