Current ML is ecl-devel@common-lisp.net, I'm forwarding this mail there.
Daniel Kochmański writes:
Faré writes:
[Adding ECL-list to this conversation, that I believe should be publicly archived]
Good idea.
On Fri, Oct 2, 2015 at 7:24 AM, Daniel Kochmański daniel@turtleware.eu wrote:
Hey,
when fixing problem with randomized names etc I have taken a closer look at the build process. You asked earlier how ECL manages to do monolithic builds.
Each module is initialized with a special function which registers the whole module and creates lisp objects. Before calling this function ECL isn't aware of the existance of it neither it has environment.
Module may be composed on other modules. Builder, when constructing initialization function for "top module" gathers all submodules (object files or libraries) and parses the binary file to find special tag which contains name of functions which initialize submodules. Top module in it's initialization functions calls (beside the other things) all submodule initialization functions. Then it concatanates all of them in .a, .fasl, .so, ... files
Top-module Sub-module-1 Sub-module-2 …
So it is guaranteed, that if this top-module will be embedded in the other one it's special tag will be found first and the correct initialization function will be called (which will initialize the futher submodules).
This description might be not 100% accurate but I believe it is how things work there. Reverse engineering might be misleading :-)
I've fixed bundles once again and ASDF passes tests. It's your call if you want to enable it by default.
Best regards, Daniel
-- Daniel Kochmański | Poznań, Poland ;; aka jackdaniel
"Be the change that you wish to see in the world." - Mahatma Gandhi
Thanks for these explanations.
Questions:
1- what is a "tag" in this context?
Tag is a string embedded in the C code with deterministic prefix followed by a name of the object function. For instance assuming initialization function has name lib_init_xyz, then it will look something like:
void lib_init_xyz (cl_object block) { ... Cblock->cblock.data_text = (const cl_object *)"@EcLtAg_lib:lib_init_xyz@"; ... }
When emiting code for top module we scan the binary object files for appropriate tag (in this example @EcLAg_lib) to find the inititalization function name which we embed in a source code as a normall call:
void lib_init_top_module (cl_object block) { ... Cblock->cblock.data_text = (const cl_object *)"@EcLtAg_lib:lib_init_top_module@"; ... lib_init_xyz(block); ... }
Note that now function names are randomized and this is mere a psudocode. For further details consult src/cmp/cmpmain.lsp.
2- once again, why not use the linker support for initialization? If behavior has to depend on whether ECL was initialized yet, the initialization could consider in calling ecl_register_init_function, which depending on whether ECL was booted, would either call the function and/or add it to a hook. See once again attached files on how to use linker support for initialization.
Fact that we might bundle many objects in one archive doesn't mean that they don't have dependencies on one another. *I think* that we have to strictly control initialization order. It's possible that I'm wrong here though.
Also do we want to *always* initialize *everything* what is linked? We may have compiled-in support for number of lisp libraries in one module just for conveniance and require them on demand depending on the application using our code.
I'm guessing here since it's a design decision made by someone else. Your proposition is surely worth investigating - it is like it is because it was that way when I approached the codebase.
3- How does MKCL do it?
MKCL creates these initialization functions but when it loads files it "slurps" some vector with objects - it doesn't call initialization functions. Don't know if function creation is rendundant/obsolete/inconsistent/correct - you have to ask MKCL maintainer for details. It looks to me that it works for files built with MKCL but breaks linking from the outer code with the libraries created by it (and by breakage I mean that compilation goes fine but such modules doesn't work). Once again I'm not 100% about that. MKCL code diverged a lot with this regards.
Regards, Daniel
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Suppose 2/3 Congress were incinerated, what would we lose to offset our gain of their salaries and the salaries of their parasites? — H. L. Mencken
2- once again, why not use the linker support for initialization? If behavior has to depend on whether ECL was initialized yet, the initialization could consider in calling ecl_register_init_function, which depending on whether ECL was booted, would either call the function and/or add it to a hook. See once again attached files on how to use linker support for initialization.
Fact that we might bundle many objects in one archive doesn't mean that they don't have dependencies on one another. *I think* that we have to strictly control initialization order. It's possible that I'm wrong here though.
It *is* possible to strictly control initialization order while using linker functions: either ensure that you link objects in the correct order (duh), and/or have your ensure_foo_initialized functions explicitly call each other.
Also do we want to *always* initialize *everything* what is linked? We may have compiled-in support for number of lisp libraries in one module just for conveniance and require them on demand depending on the application using our code.
There again, the linker-called code could "just" register the initialization function to be called later, if that's what you want.
I'm guessing here since it's a design decision made by someone else. Your proposition is surely worth investigating - it is like it is because it was that way when I approached the codebase.
Some decisions that might have made sense 30 years ago when C++ was a new thing and linkers didn't specially support constructors don't make sense 30 years later when linkers do.
I'll attach my proof-of-concept code, that didn't make it to the list.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I'd rather write programs that write programs than write programs — Dick Sites
Faré writes:
2- once again, why not use the linker support for initialization? If behavior has to depend on whether ECL was initialized yet, the initialization could consider in calling ecl_register_init_function, which depending on whether ECL was booted, would either call the function and/or add it to a hook. See once again attached files on how to use linker support for initialization.
Fact that we might bundle many objects in one archive doesn't mean that they don't have dependencies on one another. *I think* that we have to strictly control initialization order. It's possible that I'm wrong here though.
It *is* possible to strictly control initialization order while using linker functions: either ensure that you link objects in the correct order (duh), and/or have your ensure_foo_initialized functions explicitly call each other.
How will it be different from the current approach (except introducing some compiler dependnet extensions)?
Also do we want to *always* initialize *everything* what is linked? We may have compiled-in support for number of lisp libraries in one module just for conveniance and require them on demand depending on the application using our code.
There again, the linker-called code could "just" register the initialization function to be called later, if that's what you want.
As above.
I'm guessing here since it's a design decision made by someone else. Your proposition is surely worth investigating - it is like it is because it was that way when I approached the codebase.
Some decisions that might have made sense 30 years ago when C++ was a new thing and linkers didn't specially support constructors don't make sense 30 years later when linkers do.
Is there any clear benefit from using such constructors over the current approach[1] (except the pride of using modern approach)?
I'll attach my proof-of-concept code, that didn't make it to the list.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org I'd rather write programs that write programs than write programs — Dick Sites
[1] Given it is bugfree(tm)
On Sun, Oct 4, 2015 at 8:27 AM, Daniel Kochmański daniel@turtleware.eu wrote:
It *is* possible to strictly control initialization order while using linker functions: either ensure that you link objects in the correct order (duh), and/or have your ensure_foo_initialized functions explicitly call each other.
How will it be different from the current approach (except introducing some compiler dependnet extensions)?
In our day and age, all compilers and linkers support that out of the box.
Is there any clear benefit from using such constructors over the current approach[1] (except the pride of using modern approach)?
[1] Given it is bugfree(tm)
Indeed, your approach may be better, especially if the "random number" (which I hope is instead based on the sha256sum of the source and its dependencies) allows overlaying multiple versions of the code without clashes.
On the other hand, if you want to be more directly integrated with the C toolchain, it makes sense to use the C toolchain as intended.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org When your hammer is C++, everything begins to look like a thumb. — Steve Hoflich on comp.lang.c++
-
Daniel Kochmański -
Faré