[cffi-devel] %load-foreign-library on SBCL
Hi, I have a question about the implementation of %load-foreign-library on SBCL. It seems that the call to load-shared-object remembers the absolute pathname to the library that is loaded unless :dont-save t is used. This causes portability issues for my application, which can't load from the same path when used on different systems. Is there any reason to choose this behavior? Of course, with :dont-save t, the user then has to call use-foreign-library again when starting up from the saved image, but I find this preferable to the crash I receive otherwise. An example patch is included below. Thanks. --- src/cffi-sbcl.lisp~ 2009-04-14 19:33:40.000000000 -0700 +++ src/cffi-sbcl.lisp 2009-04-03 15:28:37.000000000 -0700 @@ -324,7 +324,7 @@ (defun %load-foreign-library (name path) "Load a foreign library." (declare (ignore name)) - (load-shared-object path)) + (load-shared-object path :dont-save t)) ;;; SBCL 1.0.21.15 renamed SB-ALIEN::SHARED-OBJECT-FILE but introduced ;;; SB-ALIEN:UNLOAD-SHARED-OBJECT which we can use instead. -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Fri, Jun 12, 2009 at 9:24 PM, Elliott Slaughter<elliottslaughter@gmail.com> wrote:
I have a question about the implementation of %load-foreign-library on SBCL. It seems that the call to load-shared-object remembers the absolute pathname to the library that is loaded unless :dont-save t is used. This causes portability issues for my application, which can't load from the same path when used on different systems. Is there any reason to choose this behavior?
It'd say it's convenient to be able to save a core with FFI code and have it just work. Ideally, we would try to reload libraries using all of the alternatives specified through DEFINE-FOREIGN-LIBRARY, but right now SBCL will use just the one that did succeed, IIRC. So, if you succeed in loading "libfoo.so", SBCL will try to reload that, not an absolute pathname.
Of course, with :dont-save t, the user then has to call use-foreign-library again when starting up from the saved image, but I find this preferable to the crash I receive otherwise.
IIRC, you can close the foreign libraries before saving an image and reload them manually afterwards. Would that be an acceptable work-around? -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
On Sun, Jun 14, 2009 at 10:29 AM, Luís Oliveira <luismbo@gmail.com> wrote:
On Fri, Jun 12, 2009 at 9:24 PM, Elliott Slaughter<elliottslaughter@gmail.com> wrote:
Of course, with :dont-save t, the user then has to call use-foreign-library again when starting up from the saved image, but I find this preferable to the crash I receive otherwise.
IIRC, you can close the foreign libraries before saving an image and reload them manually afterwards. Would that be an acceptable work-around?
Let me see if it works; I'll get back to you after I've tried it. Thanks. -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Fri, Jun 12, 2009 at 01:24:34PM -0700 or thereabouts, Elliott Slaughter wrote:
I have a question about the implementation of %load-foreign-library on SBCL. It seems that the call to load-shared-object remembers the absolute pathname to the library that is loaded unless :dont-save t is used. This causes portability issues for my application, which can't load from the same path when used on different systems. Is there any reason to choose this behavior?
Of course, with :dont-save t, the user then has to call use-foreign-library again when starting up from the saved image, but I find this preferable to the crash I receive otherwise.
Hi Elliott, I know you were asking about SBCL, but I think I have encountered a similar problem with CMUCL. When I bound cffi:*foreign-library-directories* and then loaded a library with cffi:load-foreign-library it set up a system::*global-table* like this: ((#<System-Area-Pointer: #x0806C6E0> . "/absolute/path/to/libwhatever.so")) When you save this image and reload it later, system::*global-table* is processed by #'reinitialize-global-table. The problem is that this will fail if libwhatever.so no longer lives in that directory when the image is reloaded. Since I loaded a library I had just compiled before subsequently installing it to /usr/lib/, the solution was to change the value of system::*global-table* to this before saving the image: ((#<System-Area-Pointer: #x0806C6E0> . "libwhatever.so")) That way when the image was reloaded it would search the default library path which was what I needed. Hope this helps, Doug PS. reference: around CMUCL src/code/foreign.lisp: ... #+linkage-table (pushnew #'reinitialize-global-table ext:*after-save-initializations*) ...
On Sun, Jun 14, 2009 at 9:33 AM, Elliott Slaughter < elliottslaughter@gmail.com> wrote:
On Sun, Jun 14, 2009 at 10:29 AM, Luís Oliveira <luismbo@gmail.com> wrote:
On Fri, Jun 12, 2009 at 9:24 PM, Elliott Slaughter<elliottslaughter@gmail.com> wrote:
Of course, with :dont-save t, the user then has to call use-foreign-library again when starting up from the saved image, but I find this preferable to the crash I receive otherwise.
IIRC, you can close the foreign libraries before saving an image and reload them manually afterwards. Would that be an acceptable work-around?
Let me see if it works; I'll get back to you after I've tried it.
I'm believe I'm seeing this issue again, but none of my previous workarounds seem to be working. I am using SBCL 1.0.29 on Windows XP, and when I save and run an executable, I keep getting undefined alien errors. I have tried both the :dont-save t hack that I initially suggested, and have tried calling (close-foreign-library ...) on all of the open libraries before saving the executable, and nothing helps. Can you suggest a course of action? Thanks. Here is the backtrace from when I try to run the executable: This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" debugger invoked on a UNDEFINED-ALIEN-ERROR: Undefined alien: "SDL_Linked_Version" Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. (no restarts: If you didn't do this on purpose, please report it as a bug.) (SB-SYS:ENSURE-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "SDL_Linked_Version" #<unused argument>) 0] WARNING: Starting a select without a timeout while interrupts are disabled. ba 0: (SB-SYS:ENSURE-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "SDL_Linked_Version" #<unused argument>) 1: ((FLET SB-THREAD::WITH-RECURSIVE-SYSTEM-SPINLOCK-THUNK)) 2: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-SYSTEM-SPINLOCK]196)) 3: (SB-THREAD::CALL-WITH-RECURSIVE-SYSTEM-SPINLOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-SYSTEM-SPINLOCK-THUNK) {22FCDD}> #<unavailable argument>) 4: (SB-SYS:UPDATE-LINKAGE-TABLE) 5: (SB-IMPL::FOREIGN-REINIT) 6: (SB-IMPL::REINIT) 7: ((LABELS SB-IMPL::RESTART-LISP)) 8: ("foreign function: #x4120C4") 9: ("foreign function: #x40AF88") -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Mon, Dec 21, 2009 at 8:55 PM, Elliott Slaughter <elliottslaughter@gmail.com> wrote:
I'm believe I'm seeing this issue again, but none of my previous workarounds seem to be working. I am using SBCL 1.0.29 on Windows XP, and when I save and run an executable, I keep getting undefined alien errors. I have tried both the :dont-save t hack that I initially suggested, and have tried calling (close-foreign-library ...) on all of the open libraries before saving the executable, and nothing helps.
Can you come up with a minimal test case? Something using e.g. libtest.dll would be nice. Thanks, -- Luís Oliveira http://r42.eu/~luis/
On Mon, Dec 21, 2009 at 1:26 PM, Luís Oliveira <luismbo@gmail.com> wrote:
I'm believe I'm seeing this issue again, but none of my previous workarounds seem to be working. I am using SBCL 1.0.29 on Windows XP, and when I save and run an executable, I keep getting undefined alien errors. I have
On Mon, Dec 21, 2009 at 8:55 PM, Elliott Slaughter <elliottslaughter@gmail.com> wrote: tried
both the :dont-save t hack that I initially suggested, and have tried calling (close-foreign-library ...) on all of the open libraries before saving the executable, and nothing helps.
Can you come up with a minimal test case? Something using e.g. libtest.dll would be nice.
Sure. For this test, I used SDL.dll from http://www.libsdl.org/release/SDL-1.2.14-win32.zip . $ cat test.lisp (asdf:oos 'asdf:load-op :cffi) (cffi:define-foreign-library sdl (:windows "SDL.dll")) (cffi:use-foreign-library sdl) (cffi:defcfun ("SDL_Linked_Version" SDL-Linked-Version) :pointer) (defun main () (format t "does it work?~%") (quit)) (cffi:close-foreign-library 'sdl) (save-lisp-and-die #+windows "main.exe" #-windows "main" :toplevel #'main :executable t) $ sbcl --load test.lisp This is SBCL 1.0.29, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" ; loading system definition from C:\Bin\asdf\cffi\cffi.asd into ; #<PACKAGE "ASDF0"> ; registering #<SYSTEM CFFI {23FFE3B9}> as CFFI ; loading system definition from C:\Bin\asdf\babel\babel.asd into ; #<PACKAGE "ASDF0"> ; registering #<SYSTEM BABEL {242FC621}> as BABEL ; loading system definition from C:\Bin\asdf\alexandria\alexandria.asd into ; #<PACKAGE "ASDF0"> ; registering #<SYSTEM :ALEXANDRIA {2449AB61}> as ALEXANDRIA ; loading system definition from ; C:\Bin\asdf\trivial-features\trivial-features.asd into #<PACKAGE "ASDF0"> ; registering #<SYSTEM TRIVIAL-FEATURES {246A9BF9}> as TRIVIAL-FEATURES WARNING: Saving cores with alien definitions referring to non-static foreign symbols is unsupported on this platform: references to such foreign symbols from the restarted core will not work. You may be able to work around this limitation by reloading all foreign definitions and code using them in the restarted core, but no guarantees. Dynamic foreign symbols in this core: SDL_Linked_Version [undoing binding stack and other enclosing state... done] [saving current Lisp image into main.exe: writing 2888 bytes from the read-only space at 0x22000000 writing 1736 bytes from the static space at 0x22100000 writing 27688960 bytes from the dynamic space at 0x22300000 done] $ ./main.exe This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" debugger invoked on a UNDEFINED-ALIEN-ERROR: Undefined alien: "SDL_Linked_Version" Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. (no restarts: If you didn't do this on purpose, please report it as a bug.) (SB-SYS:ENSURE-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "SDL_Linked_Version" #<unused argument>) 0] WARNING: Starting a select without a timeout while interrupts are disabled. ba 0: (SB-SYS:ENSURE-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "SDL_Linked_Version" #<unused argument>) 1: ((FLET SB-THREAD::WITH-RECURSIVE-SYSTEM-SPINLOCK-THUNK)) 2: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-RECURSIVE-SYSTEM-SPINLOCK]196)) 3: (SB-THREAD::CALL-WITH-RECURSIVE-SYSTEM-SPINLOCK #<CLOSURE (FLET SB-THREAD::WITH-RECURSIVE-SYSTEM-SPINLOCK-THUNK) {22FCDD}> #<unavailable argument>) 4: (SB-SYS:UPDATE-LINKAGE-TABLE) 5: (SB-IMPL::FOREIGN-REINIT) 6: (SB-IMPL::REINIT) 7: ((LABELS SB-IMPL::RESTART-LISP)) 8: ("foreign function: #x4120C4") 9: ("foreign function: #x40AF88") 0] WARNING: Starting a select without a timeout while interrupts are disabled. (quit) -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Mon, Dec 21, 2009 at 4:52 PM, Elliott Slaughter < elliottslaughter@gmail.com> wrote:
On Mon, Dec 21, 2009 at 1:26 PM, Luís Oliveira <luismbo@gmail.com> wrote:
I'm believe I'm seeing this issue again, but none of my previous workarounds seem to be working. I am using SBCL 1.0.29 on Windows XP, and when I save and run an executable, I keep getting undefined alien errors. I have
On Mon, Dec 21, 2009 at 8:55 PM, Elliott Slaughter <elliottslaughter@gmail.com> wrote: tried
both the :dont-save t hack that I initially suggested, and have tried calling (close-foreign-library ...) on all of the open libraries before saving the executable, and nothing helps.
Can you come up with a minimal test case? Something using e.g. libtest.dll would be nice.
Sure.
For this test, I used SDL.dll from http://www.libsdl.org/release/SDL-1.2.14-win32.zip .
$ cat test.lisp (asdf:oos 'asdf:load-op :cffi) (cffi:define-foreign-library sdl (:windows "SDL.dll")) (cffi:use-foreign-library sdl) (cffi:defcfun ("SDL_Linked_Version" SDL-Linked-Version) :pointer) (defun main () (format t "does it work?~%") (quit)) (cffi:close-foreign-library 'sdl) (save-lisp-and-die #+windows "main.exe" #-windows "main" :toplevel #'main :executable t)
After what feels like beating my head against a wall for several hours, and getting contradictory results from various attempts to boils the failure down to a manageable size, I finally tried my original build recipe again, and it just worked. I'm not really sure what to say, other than that I'm really confused. I don't think the workarounds suggested by myself and others are necessary, so I think we can all forget about this, at least until this comes to haunt me again. Sorry about the noise. -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Tue, Dec 22, 2009 at 6:56 AM, Elliott Slaughter <elliottslaughter@gmail.com> wrote:
After what feels like beating my head against a wall for several hours, and getting contradictory results from various attempts to boils the failure down to a manageable size, I finally tried my original build recipe again, and it just worked. I'm not really sure what to say, other than that I'm really confused. I don't think the workarounds suggested by myself and others are necessary, so I think we can all forget about this, at least until this comes to haunt me again.
This is something that definitely needs to be improved in CFFI, at least documentation-wise. Care to share your build recipe for dealing with foreign libraries? -- Luís Oliveira http://r42.eu/~luis/
On Tue, Dec 22, 2009 at 2:15 AM, Luís Oliveira <luismbo@gmail.com> wrote:
On Tue, Dec 22, 2009 at 6:56 AM, Elliott Slaughter <elliottslaughter@gmail.com> wrote:
After what feels like beating my head against a wall for several hours, and getting contradictory results from various attempts to boils the failure down to a manageable size, I finally tried my original build recipe again, and it just worked. I'm not really sure what to say, other than that I'm really confused. I don't think the workarounds suggested by myself and others are necessary, so I think we can all forget about this, at least until this comes to haunt me again.
This is something that definitely needs to be improved in CFFI, at least documentation-wise. Care to share your build recipe for dealing with foreign libraries?
Well, there's not really much to it, at least when it works. Basically, I just (asdf:oos 'asdf:load-op *some-system*) which presumably has a dependency on lispbuilder or cl-opengl or something else that depends on a foreign library. After than, just save the executable, #+sbcl (sb-ext:save-lisp-and-die #-windows "main" #+windows "main.exe" :toplevel #'main :executable t) The only platform where I typically have this not work is on SBCL/Windows. I do have a little piece of initialization code to reload any foreign libraries, but obviously that didn't help me this last time. Sometimes I also put the dlls in the current directory, but again, I'm not sure whether that actually helps or not. Other than that, I've just got a makefile which builds installers for Windows, .app files for Mac, and tarballs for Linux. Anyway, if you are interested in looking at the entire script, see: http://blackthorn-engine.googlecode.com/hg/dist.lisp and the makefile (particularly the dist and install-* targets): http://blackthorn-engine.googlecode.com/hg/Makefile Hope that helps. -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
On Mon, Dec 21, 2009 at 1:26 PM, Luís Oliveira <luismbo@gmail.com> wrote:
I'm believe I'm seeing this issue again, but none of my previous workarounds seem to be working. I am using SBCL 1.0.29 on Windows XP, and when I save and run an executable, I keep getting undefined alien errors. I have
On Mon, Dec 21, 2009 at 8:55 PM, Elliott Slaughter <elliottslaughter@gmail.com> wrote: tried
both the :dont-save t hack that I initially suggested, and have tried calling (close-foreign-library ...) on all of the open libraries before saving the executable, and nothing helps.
Can you come up with a minimal test case? Something using e.g. libtest.dll would be nice.
I think I managed to reproduce this error, and this time, I have a minimal test case! When I put a dll on *foreign-library-directories* and build a Windows executable with SBCL, the generated executable cannot run unless the dll is present at exactly the same absolute path as it was during compile time. If I instead move the dll into the current working directory, then I don't have any problem with absolute paths being saved in my binary. First, the contents of break.lisp: (asdf:oos 'asdf:load-op :cffi) (use-package :cffi) (pushnew (truename #p"lib/") *foreign-library-directories* :test #'equal) (define-foreign-library sdl (:windows "SDL.dll")) (defun main () (use-foreign-library sdl) (format t "It works~%")) (main) (save-lisp-and-die "main.exe" :toplevel #'main :executable t) (quit) Second, I create two binaries. I demonstrate that the first executable is broken by moving the original lib directory after compiling, while the second executable works fine. $ ls * break.lisp bin: lib: SDL.dll $ sbcl --load break.lisp This is SBCL 1.0.29, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. [ ... ] It works [undoing binding stack and other enclosing state... done] [saving current Lisp image into main.exe: writing 2888 bytes from the read-only space at 0x22000000 writing 1736 bytes from the static space at 0x22100000 writing 27582464 bytes from the dynamic space at 0x22300000 done] $ mv main.exe bin/ $ cp lib/SDL.dll bin/ $ cd bin/ $ ./main.exe This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" It works $ mv ../lib/ ../lib2 $ ./main.exe This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" debugger invoked on a SIMPLE-ERROR: Error opening shared object "c:\\Users\\Elliott\\Programming\\CommonLisp\\brea k-cffi\\lib\\SDL.dll": 126. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [CONTINUE ] Skip this shared object and continue. 1: [RETRY ] Retry loading this shared object. 2: [CHANGE-PATHNAME] Specify a different pathname to load the shared object fr om. (SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"c:\\Users\\Elliott\\Programming\\CommonLisp\\break-cffi\\lib\\S DL.dll" :NAMESTRING "c:\\Users\\Elliott\\Programming\\CommonLisp\\break-cffi\\lib\\S DL.dll" :HANDLE NIL :DONT-SAVE NIL)) 0] (quit) $ cd .. $ mv lib2/ lib $ cp lib/SDL.dll . $ rm bin/main.exe $ ls * SDL.dll break.lisp bin: SDL.dll lib: SDL.dll $ sbcl --load break.lisp [ ... ] It works [undoing binding stack and other enclosing state... done] [saving current Lisp image into main.exe: writing 2888 bytes from the read-only space at 0x22000000 writing 1736 bytes from the static space at 0x22100000 writing 27578368 bytes from the dynamic space at 0x22300000 done] $ mv main.exe bin/ $ cd bin/ $ ./main.exe This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" It works $ mv ../lib/ ../lib2 $ ./main.exe This is experimental prerelease support for the Windows platform: use at your own risk. "Your Kitten of Death awaits!" It works -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
participants (3)
-
doug＠hcsw.org -
Elliott Slaughter -
Luís Oliveira