[cffi-devel] Re: Clisp, cffi and defcfuns in a saved image
Greetings! I'm not really sure if this is a CFFI or a CLISP issue, so I'm posting to both ml's. I'm trying to make Windows executables images, using Clisp 2.41 (built with readline, iconv and sigsegv). My images have cl projects loaded, like clsql or celltk. These projects relly on cffi to load functions from certain dll's (like odbc32.dll). E.g., I load clsql and save an image. There are two problems here: 1) Unlike AllegroCL or Lispworks, Clisp (and SBCL too, btw) requires me to precede all defcfuns related to a library with: #+clisp (cffi::use-foreign-library Tcl) (defcfun ("Tcl_FindExecutable" tcl-find-executable) :void (argv0 :string)) If I don't do this, I can't apply this function at run-time (I'm not even talking about saved images, yet). 2) When I load the saved image (using the code from above), I get an error when applying defcfun'ed functions, telling me the library wasn't found: ** - Continuable Error FFI::FOREIGN-CALL-OUT: no dynamic object named "SQLAllocHandle" in library :DEFAULT But then, if I tell clisp where the libraries are, with (cffi:define-foreign-library odbc (t (:default "odbc32"))) (cffi:use-foreign-library odbc) , I get this: *** - FFI::FOREIGN-CALL-OUT: #<INVALID FOREIGN-POINTER #x00000000> comes from a previous Lisp session and is invalid The only workaround I can think of is to make an init-function on my image that loads all these defcfuns (at run-time), but this is not very sane, if you are to use many external CL libraries! Thanks for the help, -- Edgar Gonçalves
Olá, On 12/02/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
1) Unlike AllegroCL or Lispworks, Clisp (and SBCL too, btw) requires me to precede all defcfuns related to a library with:
#+clisp (cffi::use-foreign-library Tcl) (defcfun ("Tcl_FindExecutable" tcl-find-executable) :void (argv0 :string))
That doesn't sound right. You should only need to call (use-foreign-library tcl) only once, some time before you load your bindings.
2) When I load the saved image (using the code from above), I get an error when applying defcfun'ed functions, telling me the library wasn't found:
Just tested with GNU CLISP 2.41 (2006-10-13) on OSX. Loaded cffi-test, saved an image (tried with ":executable t" too), restarted with the image, and ran (rt:do-tests) and it works. (Except some funcall.* tests fail at first, for some reason. I blame it on some RT weirdness.) AFAICT, CLISP correctly reloads the libraries at startup. Can you try this with cffi-test or otherwise provide some sort of test case? -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
Viva, On 2/12/07, Luís Oliveira <luismbo@gmail.com> wrote:
On 12/02/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
1) Unlike AllegroCL or Lispworks, Clisp (and SBCL too, btw) requires me to precede all defcfuns related to a library with:
#+clisp (cffi::use-foreign-library Tcl) (defcfun ("Tcl_FindExecutable" tcl-find-executable) :void (argv0 :string))
That doesn't sound right. You should only need to call (use-foreign-library tcl) only once, some time before you load your bindings.
Let me rephrase it a bit, to avoid misunderstandings. Yes, I call (use-foreign-library tcl) only once. The point is that I *need* to call it, whereas with ACL, for instance, I don't - the defcfuns just work for themselves.
2) When I load the saved image (using the code from above), I get an error
when applying defcfun'ed functions, telling me the library wasn't found:
Just tested with GNU CLISP 2.41 (2006-10-13) on OSX. Loaded cffi-test, saved an image (tried with ":executable t" too), restarted with the image, and ran (rt:do-tests) and it works. (Except some funcall.* tests fail at first, for some reason. I blame it on some RT weirdness.)
AFAICT, CLISP correctly reloads the libraries at startup. Can you try this with cffi-test or otherwise provide some sort of test case?
Actually, I haven't tried tests, but I can't seem to get past the library bindings.lisp. It seems to be a problem with the dll (I'm running Windows 2003), given that my cl.exe (version 14.00.50727.42 for 80x86 ) won't recognize the <stdint.h>, and that my cygwin (latest, just checked for updates) makes a dll that hangs bindings.fas while loading. As for a short test case, here's the smallest I can make that fires up my error. After loading clisp, and loading asdf, I eval: ,----- | (asdf:operate 'asdf:load-op :cffi) | | (cffi:define-foreign-library odbc | (:windows (:or "odbc32.dll")) | (t (:default "odbc32.dll"))) | | (cffi:defcfun ("SQLAllocHandle" sql-handle) :short) | | (sql-handle) `----- The last form brings me to the following error: WARNING: FFI::FOREIGN-LIBRARY-FUNCTION: no dynamic object named "SQLAllocHandle" in library :DEFAULT *** - FUNCALL: undefined function NIL with the relevant backtrace being: <11> #<SYSTEM-FUNCTION ERROR> <12> #<COMPILED-FUNCTION SYSTEM::CHECK-VALUE> <13> #<SYSTEM-FUNCTION FUNCALL> 1 EVAL frame for form (VALUES (FUNCALL (LOAD-TIME-VALUE (MULTIPLE-VALUE-BIND (CFFI-SYS::FF ERROR) (IGNORE-ERRORS (FFI::FOREIGN-LIBRARY-FUNCTION "SQLAllocHandle" (FFI::FOREIGN-LIBRARY :DEFAULT) NIL NIL (FFI:PARSE-C-TYPE '(FFI:C-FUNCTION (:ARGUMENTS) (:RETURN-TYPE FFI:SHORT) (:LANGUAGE :STDC))))) (OR CFFI-SYS::FF (WARN (FORMAT NIL "~?" (SIMPLE-CONDITION-FORMAT-CONTROL ERROR) (SIMPLE-CONDITION-FORMAT-ARGUMENTS ERROR)))))))) APPLY frame for call (SQL-HANDLE) <14> #<FUNCTION SQL-HANDLE NIL (DECLARE (SYSTEM::IN-DEFUN SQL-HANDLE)) (BLOCK SQL-HANDLE (VALUES (CFFI-SYS:%FOREIGN-FUNCALL "SQLAllocHandle" :SHORT)))> 0 EVAL frame for form (SQL-HANDLE) Hope this helps tracing out something, tell me if you need anything more. Thanks, Edgar Gonçalves
Viva, On 2/12/07, Luís Oliveira <luismbo@gmail.com> wrote:
On 12/02/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
1) Unlike AllegroCL or Lispworks, Clisp (and SBCL too, btw) requires me to precede all defcfuns related to a library with:
#+clisp (cffi::use-foreign-library Tcl) (defcfun ("Tcl_FindExecutable" tcl-find-executable) :void (argv0 :string))
That doesn't sound right. You should only need to call (use-foreign-library tcl) only once, some time before you load your bindings.
Let me rephrase it a bit, to avoid misunderstandings. Yes, I call (use-foreign-library tcl) only once. The point is that I *need* to call it, whereas with ACL, for instance, I don't - the defcfuns just work for themselves.
2) When I load the saved image (using the code from above), I get an error when applying defcfun'ed functions, telling me the library wasn't found:
Just tested with GNU CLISP 2.41 (2006-10-13) on OSX. Loaded cffi-test, saved an image (tried with ":executable t" too), restarted with the image, and ran (rt:do-tests) and it works. (Except some funcall.* tests fail at first, for some reason. I blame it on some RT weirdness.)
AFAICT, CLISP correctly reloads the libraries at startup. Can you try this with cffi-test or otherwise provide some sort of test case?
Actually, I haven't tried tests, but I can't seem to get past the library bindings.lisp. It seems to be a problem with the dll (I'm running Windows 2003), given that my cl.exe (version 14.00.50727.42 for 80x86 ) won't recognize the <stdint.h>, and that my cygwin (latest, just checked for updates) makes a dll that hangs bindings.fas while loading. As for a short test case, here's the smallest I can make that fires up my error. After loading clisp, and loading asdf, I eval: ,----- | (asdf:operate 'asdf:load-op :cffi) | | (cffi:define-foreign-library odbc | (:windows (:or "odbc32.dll")) | (t (:default "odbc32.dll"))) | | (cffi:defcfun ("SQLAllocHandle" sql-handle) :short) | | (sql-handle) `----- The last form brings me to the following error: WARNING: FFI::FOREIGN-LIBRARY-FUNCTION: no dynamic object named "SQLAllocHandle" in library :DEFAULT *** - FUNCALL: undefined function NIL with the relevant backtrace being: <11> #<SYSTEM-FUNCTION ERROR> <12> #<COMPILED-FUNCTION SYSTEM::CHECK-VALUE> <13> #<SYSTEM-FUNCTION FUNCALL> 1 EVAL frame for form (VALUES (FUNCALL (LOAD-TIME-VALUE (MULTIPLE-VALUE-BIND (CFFI-SYS::FF ERROR) (IGNORE-ERRORS (FFI::FOREIGN-LIBRARY-FUNCTION "SQLAllocHandle" (FFI::FOREIGN-LIBRARY :DEFAULT) NIL NIL (FFI:PARSE-C-TYPE '(FFI:C-FUNCTION (:ARGUMENTS) (:RETURN-TYPE FFI:SHORT) (:LANGUAGE :STDC))))) (OR CFFI-SYS::FF (WARN (FORMAT NIL "~?" (SIMPLE-CONDITION-FORMAT-CONTROL ERROR) (SIMPLE-CONDITION-FORMAT-ARGUMENTS ERROR)))))))) APPLY frame for call (SQL-HANDLE) <14> #<FUNCTION SQL-HANDLE NIL (DECLARE (SYSTEM::IN-DEFUN SQL-HANDLE)) (BLOCK SQL-HANDLE (VALUES (CFFI-SYS:%FOREIGN-FUNCALL "SQLAllocHandle" :SHORT)))> 0 EVAL frame for form (SQL-HANDLE) Hope this helps tracing out something, tell me if you need anything more. Thanks, Edgar Gonçalves
On 13/02/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
,----- | (asdf:operate 'asdf:load-op :cffi) | | (cffi:define-foreign-library odbc | (:windows (:or "odbc32.dll")) | (t (:default "odbc32.dll"))) | | (cffi:defcfun ("SQLAllocHandle" sql-handle) :short) | | (sql-handle) `-----
OK, yes, you need to load the library before the defcfun. Some lisps won't mind but others do. They'll all complain by the time the call to sql-handle is evaluated though. Why aren't you loading the library? What are you trying to accomplish? What am I missing? -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/
-- Edgar Gonçalves Instituto Superior Técnico, INESC-ID - Software Engineering Group, Portugal On 2/13/07, Luís Oliveira <luismbo@gmail.com> wrote:
On 13/02/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
,----- | (asdf:operate 'asdf:load-op :cffi) | | (cffi:define-foreign-library odbc | (:windows (:or "odbc32.dll")) | (t (:default "odbc32.dll"))) | | (cffi:defcfun ("SQLAllocHandle" sql-handle) :short) | | (sql-handle) `-----
OK, yes, you need to load the library before the defcfun. Some lisps won't mind but others do. They'll all complain by the time the call to sql-handle is evaluated though.
Why aren't you loading the library? What are you trying to accomplish? What am I missing?
I was trying to explain my first problem, but if it's assumed that "some lisps won't mind" while others do, I guess the problem rolls back to us, developers, to do the use-foreign-library. About my second issue, I've come to the conclusion that this simple example (below) works all-right, given that you call use-foreign-library after loading the image (before calling the foreign functions). This is what I find most annoying, since I have to scoop through all 3rd party libraries for used dll's. Any chance to make this work for clisp just like it does for (e.g.) ACL (i.e., without having to call use-foreign-library)? ,----- | (asdf:operate 'asdf:load-op :cffi) | | (cffi:define-foreign-library odbc | (:windows (:or "odbc32.dll")) | (t (:default "odbc32.dll"))) | | (cffi::use-foreign-library odbc) | | (cffi:defcfun ("SQLAllocHandle" sql-handle) :short) | | (format t "---------------Before loading image, dll call yelds: ~A!~%" (sql-handle)) | | (ext:saveinitmem "test.exe" | :init-function #'(lambda () | (cffi::use-foreign-library odbc) | (format t "---------------After loading image, dll call yelds: ~A!~%" (sql-handle)) | (ext:quit) | ) | :NORC t | :script t | :executable t | :quiet t) | (ext:quit) `----- I'll try again making it work with my clsql project, and I'll post back my success.
On 2/13/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
I'll try again making it work with my clsql project, and I'll post back my success.
Eh, guess not, after all. The problem seems to reside somewhere betwee clsql-odbc and CFFI's uffi-compat.lisp. The following is related to the initialization function of the saved image. It assumes the cl-sql library was already loaded AND used properly before the image was saved. What I do before using the foreign functions is to call (clsql-sys:database-type-load-foreign :odbc), and this calls uffi:load-foreign-library, defined in cffi's uffi-compat.lisp, with the odbc32.dll filename. This, by its turn, calls cffi:load-foreign-library, which is the same as to call cffi::use-foreign-library. So, even though the result should apparently be the same, I still get this: ** - Continuable Error FFI::FOREIGN-CALL-OUT: no dynamic object named "SQLAllocHandle" in library :DEFAULT But then again, if I manually call (cffi::use-foreign-library "odbc32.dll"), what I get is something different: *** - FFI::FOREIGN-CALL-OUT: #<INVALID FOREIGN-POINTER #x00000000> comes from a previous Lisp session and is invalid What am I missing here? -- Edgar Gonçalves
Ok, I've just realized that I've been mislocating the problem at hands. It seems that defcfun works just fine, and the issue I'm getting resides on the foreign pointers. I've made tests to cffi, cffi-uffi-compat, ffi (answering to Ken's "CLisp exe vs FFI" post), and all have worked for a simple function definition/call. So I delved again to clsql initial situation, and I've traced the error to a call to %new-environment-handle, that uses a null pointer as an argument to the ff call to SQLAllocHandle. I also noticed that the null pointer was indeed showing up at the error message, so I tried this simple test: ,----- | (asdf:operate 'asdf:load-op 'cffi) | (asdf:operate 'asdf:load-op 'cffi-uffi-compat) | (defvar my-null-ptr (ffi:unsigned-foreign-address 0)) | (format t "- Before loading image, null ptr is: ~A!~%" my-null-ptr) | (ext:saveinitmem "test.exe" | :init-function #'(lambda () | (format t "- After loading image, my-null-ptr yelds: ~A.~%" my-null-ptr) | (ext:quit)) | :NORC t | :script t | :executable t | :quiet t) `----- The output I get when I run this code is (after the initial load messages), ,----- | (...) | - Before loading image, null ptr is: #<FOREIGN-ADDRESS #x00000000>! | C:\dev\test>test | test | - After loading image, my-null-ptr yelds: #<INVALID FOREIGN-ADDRESS #x00000000>. `----- What makes a foreign address valid or invalid? Is there some way I can validate a simple null pointer? I could understand if a given non-null pointer in C would be hard to validate after restarting an image, but a null pointer is, and always will be, 0x0, right? -- Edgar Gonçalves Instituto Superior Técnico, INESC-ID - Software Engineering Group, Portugal On 2/13/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
On 2/13/07, Edgar Gonçalves <edgar.goncalves@gmail.com> wrote:
I'll try again making it work with my clsql project, and I'll post back my success.
Eh, guess not, after all. The problem seems to reside somewhere betwee clsql-odbc and CFFI's uffi-compat.lisp.
The following is related to the initialization function of the saved image. It assumes the cl-sql library was already loaded AND used properly before the image was saved.
What I do before using the foreign functions is to call (clsql-sys:database-type-load-foreign :odbc), and this calls uffi:load-foreign-library, defined in cffi's uffi-compat.lisp, with the odbc32.dll filename. This, by its turn, calls cffi:load-foreign-library, which is the same as to call cffi::use-foreign-library. So, even though the result should apparently be the same, I still get this:
** - Continuable Error FFI::FOREIGN-CALL-OUT: no dynamic object named "SQLAllocHandle" in library :DEFAULT
But then again, if I manually call (cffi::use-foreign-library "odbc32.dll"), what I get is something different:
*** - FFI::FOREIGN-CALL-OUT: #<INVALID FOREIGN-POINTER #x00000000> comes from a previous Lisp session and is invalid
What am I missing here?
-- Edgar Gonçalves
participants (2)
-
Edgar Gonçalves -
Luís Oliveira