Shared/static libraries in static-program-op context
Good morning cffi-devel, I have been recently sending a bunch of pull requests: - https://github.com/cffi/cffi/pull/162 - https://github.com/cffi/cffi/pull/157 Which are really the following issue: - CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared" libraries and fails because the path doesn't exist on the target system Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program. In order to make it possible to distribute this static-program-op output, we essentially need to close the foreign library (IOW unload it), or not load it to begin with, before dumping the image. That means tracking those. The 2nd PR I made, as sionescu pointed out, doesn't work because C-FILE code isn't aware of whether it's going to be used for static-program-op or not. So I can see 2 ways out of this: - Either we make FOREIGN-LIBRARY-TYPE settable from the outside (remember that CFFI-TOOLCHAIN is not in the CFFI package), which doesn't sound like the correct way to go to me? - Or we add a CLOSE-ON-DUMP option, which we can use when registering the library. (Btw, that means exposing the REGISTER-FOREIGN-LIBRARY symbol, but that sounds uncontroversial to me.) Both of the PRs I mentioned earlier would benefit this CLOSE-ON-DUMP option. Instead of having to manually go through the foreign libraries and guess which ones need to be closed before dumping, this option would be explicit. What do you think? Regards, -- Florian Margaine
On Sun, 2020-05-24 at 21:57 +0200, Florian Margaine wrote:
Good morning cffi-devel,
I have been recently sending a bunch of pull requests:
- https://github.com/cffi/cffi/pull/162 - https://github.com/cffi/cffi/pull/157
Which are really the following issue:
- CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared" libraries and fails because the path doesn't exist on the target system
Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program.
In order to make it possible to distribute this static-program-op output, we essentially need to close the foreign library (IOW unload it), or not load it to begin with, before dumping the image. That means tracking those. The 2nd PR I made, as sionescu pointed out, doesn't work because C- FILE code isn't aware of whether it's going to be used for static-program- op or not.So I can see 2 ways out of this:
- Either we make FOREIGN-LIBRARY-TYPE settable from the outside (remember that CFFI-TOOLCHAIN is not in the CFFI package), which doesn't sound like the correct way to go to me? - Or we add a CLOSE-ON-DUMP option, which we can use when registering the library. (Btw, that means exposing the REGISTER-FOREIGN-LIBRARY symbol, but that sounds uncontroversial to me.)
I don't think either of those are a good idea because they're not solving the problem at its core: 1) the wrapper shared objects are loaded by absolute path and 2) the filesystem layout differs between host and target machine To fix that you need to either * keep the .so as they are, and have static-program-op output a directory containing the output binary and all the wrappers, then fixup the wrapper paths so that the runtime finds the shared objects in their new location (might need to close them and have them re-opened on start). This is what most other dynamic language runtimes do (Perl, Python, etc...) * statically link the wrapper: make the wrapper op also generate a static archive (.a), then link to that in static-program-op -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Mon, May 25, 2020 at 1:19 AM Stelian Ionescu <sionescu@cddr.org> wrote:
On Sun, 2020-05-24 at 21:57 +0200, Florian Margaine wrote:
Good morning cffi-devel,
I have been recently sending a bunch of pull requests:
- https://github.com/cffi/cffi/pull/162 - https://github.com/cffi/cffi/pull/157
Which are really the following issue:
- CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared" libraries and fails because the path doesn't exist on the target system
Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program.
In order to make it possible to distribute this static-program-op output, we essentially need to close the foreign library (IOW unload it), or not load it to begin with, before dumping the image. That means tracking those. The 2nd PR I made, as sionescu pointed out, doesn't work because C- FILE code isn't aware of whether it's going to be used for static-program- op or not.So I can see 2 ways out of this:
- Either we make FOREIGN-LIBRARY-TYPE settable from the outside (remember that CFFI-TOOLCHAIN is not in the CFFI package), which doesn't sound like the correct way to go to me? - Or we add a CLOSE-ON-DUMP option, which we can use when registering the library. (Btw, that means exposing the REGISTER-FOREIGN-LIBRARY symbol, but that sounds uncontroversial to me.)
I don't think either of those are a good idea because they're not solving the problem at its core: 1) the wrapper shared objects are loaded by absolute path and 2) the filesystem layout differs between host and target machine
To fix that you need to either * keep the .so as they are, and have static-program-op output a directory containing the output binary and all the wrappers, then fixup the wrapper paths so that the runtime finds the shared objects in their new location (might need to close them and have them re-opened on start). This is what most other dynamic language runtimes do (Perl, Python, etc...) * statically link the wrapper: make the wrapper op also generate a static archive (.a), then link to that in static-program-op
The static link is already what static-program-op is doing. The problem is that the foreign library is _also_ loaded as a shared library, and is restored when the image restores itself. Which fails with the wrong paths.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
-- Florian Margaine
Hello Florian, On Sun, 24 May 2020 at 20:58, Florian Margaine <florian@margaine.com> wrote:
- CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared" libraries and fails because the path doesn't exist on the target system
Let me point out, that this is an issue for regular foreign libraries. Take cl-sqlite as an example: (define-foreign-library sqlite3-lib (:darwin (:default "libsqlite3")) (:unix (:or "libsqlite3.so.0" "libsqlite3.so")) (t (:or (:default "libsqlite3") (:default "sqlite3")))) On UNIX, (load-foreign-library 'sqlite3-lib) will attempt to load "lbsqlite3.so.0" first. If that fails, "libsqlite3.so" will be used instead. Whichever happens to succeed when building an image, is what will be retried when starting up the image. Things get worse if things like *foreign-library-directories* come into play. When that's used, the library will be loaded using an absolute path, and that's what will be retried at startup. The proper way to handle this is to close all foreign libraries before dumping an image, and reload them using load-foreign-library at startup. I think we should either (a) have load-foreign-library register these hooks or (b) somehow hook into image/runtime-op (not the static versions) to perform this unload/reload dance. We might have to be mindful that people shipping applications most likely do this dance themselves, so we might clash. For instance, we could offer this functionality as a convenient hook users can use in their dump and startup routines.
Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program.
For :c-file, when shipped dynamically, I think the best we can do is make sure it's reloaded by name instead of absolute path. That'd work on Windows, by placing the DLL next to the executable. Not sure what the proper way would be on UNIX. For a statically linked :c-file (or wrapper, or whatever), then we of course need to avoid reopening. I conclude that we /always/ want to close on dump. Should we just do that unconditionally? As for reloading, we'd still need to distinguish between what should and shouldn't be reloaded, so I'm leaning towards your CLOSE-ON-DUMP idea but call it RELOAD-ON-RESTART (or something along those lines). Why do you say it implies exposing REGISTER-FOREIGN-LIBRARY? Seems like a LOAD-FOREIGN-LIBRARY option, no? Cheers, -- Luís Oliveira http://kerno.org/~luis/
We might have to be mindful that people shipping applications most likely do this dance themselves, so we might clash.
if it can be done in a way that just works nicely, then i wouldn't care about any clashes. people who update CFFI can take care of updating their image load/save hooks, too. backwards compatibility encoded into the codebase is overrated in the era of DVCS'es and quicklisp. just my 0.02, -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “[A] Computer [programming] language is inherently a pun — [it] needs to be interpreted by both men & machines.” — Henry Baker
Le lun. 25 mai 2020 à 11:57, Luís Oliveira <luismbo@gmail.com> a écrit :
Hello Florian,
On Sun, 24 May 2020 at 20:58, Florian Margaine <florian@margaine.com> wrote:
- CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared" libraries and fails because the path doesn't exist on the target system
Let me point out, that this is an issue for regular foreign libraries. Take cl-sqlite as an example:
(define-foreign-library sqlite3-lib (:darwin (:default "libsqlite3")) (:unix (:or "libsqlite3.so.0" "libsqlite3.so")) (t (:or (:default "libsqlite3") (:default "sqlite3"))))
On UNIX, (load-foreign-library 'sqlite3-lib) will attempt to load "lbsqlite3.so.0" first. If that fails, "libsqlite3.so" will be used instead. Whichever happens to succeed when building an image, is what will be retried when starting up the image. Things get worse if things like *foreign-library-directories* come into play. When that's used, the library will be loaded using an absolute path, and that's what will be retried at startup.
The proper way to handle this is to close all foreign libraries before dumping an image, and reload them using load-foreign-library at startup.
I think we should either (a) have load-foreign-library register these hooks or (b) somehow hook into image/runtime-op (not the static versions) to perform this unload/reload dance.
We might have to be mindful that people shipping applications most likely do this dance themselves, so we might clash. For instance, we could offer this functionality as a convenient hook users can use in their dump and startup routines.
Maybe a new option in the ASDF integration? I'm a big fan of backwards compatibility.
Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program.
For :c-file, when shipped dynamically, I think the best we can do is make sure it's reloaded by name instead of absolute path. That'd work on Windows, by placing the DLL next to the executable. Not sure what the proper way would be on UNIX.
For a statically linked :c-file (or wrapper, or whatever), then we of course need to avoid reopening.
I conclude that we /always/ want to close on dump. Should we just do that unconditionally?
This actually plays really well with the recent canary addition to cffi, as the load will be skipped for static libraries.
As for reloading, we'd still need to distinguish between what should and shouldn't be reloaded, so I'm leaning towards your CLOSE-ON-DUMP idea but call it RELOAD-ON-RESTART (or something along those lines).
We don't actually need to, if we just use the canary mechanism, right? Why do you say it implies exposing REGISTER-FOREIGN-LIBRARY? Seems
like a LOAD-FOREIGN-LIBRARY option, no?
That was mostly for plumbing, but I'm not quite sure myself now.
Cheers,
-- Luís Oliveira http://kerno.org/~luis/
Regards, Florian
Le lun. 25 mai 2020 à 13:57, Florian Margaine <florian@margaine.com> a écrit :
Le lun. 25 mai 2020 à 11:57, Luís Oliveira <luismbo@gmail.com> a écrit :
Hello Florian,
- CFFI loads a static foreign library, either as :grovel-wrapper or :system - The image is dumped with the loaded library - When restored, the runtime will try to reload those "shared"
On Sun, 24 May 2020 at 20:58, Florian Margaine <florian@margaine.com> wrote: libraries and fails because the path doesn't exist on the target system
Let me point out, that this is an issue for regular foreign libraries. Take cl-sqlite as an example:
(define-foreign-library sqlite3-lib (:darwin (:default "libsqlite3")) (:unix (:or "libsqlite3.so.0" "libsqlite3.so")) (t (:or (:default "libsqlite3") (:default "sqlite3"))))
On UNIX, (load-foreign-library 'sqlite3-lib) will attempt to load "lbsqlite3.so.0" first. If that fails, "libsqlite3.so" will be used instead. Whichever happens to succeed when building an image, is what will be retried when starting up the image. Things get worse if things like *foreign-library-directories* come into play. When that's used, the library will be loaded using an absolute path, and that's what will be retried at startup.
The proper way to handle this is to close all foreign libraries before dumping an image, and reload them using load-foreign-library at startup.
I think we should either (a) have load-foreign-library register these hooks or (b) somehow hook into image/runtime-op (not the static versions) to perform this unload/reload dance.
We might have to be mindful that people shipping applications most likely do this dance themselves, so we might clash. For instance, we could offer this functionality as a convenient hook users can use in their dump and startup routines.
Maybe a new option in the ASDF integration? I'm a big fan of backwards compatibility.
Specifically for the 2nd one, CFFI provides a :c-file ASDF integration, which, when used with :static-program-op, will compile the C file into an object file (among others), load it as a foreign library, and statically link the object file with the program.
For :c-file, when shipped dynamically, I think the best we can do is make sure it's reloaded by name instead of absolute path. That'd work on Windows, by placing the DLL next to the executable. Not sure what the proper way would be on UNIX.
For a statically linked :c-file (or wrapper, or whatever), then we of course need to avoid reopening.
I conclude that we /always/ want to close on dump. Should we just do that unconditionally?
This actually plays really well with the recent canary addition to cffi, as the load will be skipped for static libraries.
Actually, the canary option implies that we know an existing ELF symbol in the foreign library, which is not possible in our case.
As for reloading, we'd still need to distinguish between what should and shouldn't be reloaded, so I'm leaning towards your CLOSE-ON-DUMP idea but call it RELOAD-ON-RESTART (or something along those lines).
We don't actually need to, if we just use the canary mechanism, right?
Still trying to think of a good way to do that. I'll likely come back with a PR.
Why do you say it implies exposing REGISTER-FOREIGN-LIBRARY? Seems
like a LOAD-FOREIGN-LIBRARY option, no?
That was mostly for plumbing, but I'm not quite sure myself now.
Cheers,
-- Luís Oliveira http://kerno.org/~luis/
Regards, Florian
participants (4)
-
Attila Lendvai -
Florian Margaine -
Luís Oliveira -
Stelian Ionescu