January 2006 - cffi-devel - mailman3.common-lisp.net

[cffi-devel] Lisp-name conversion
by Surendra Singhi 09 Jan '06

09 Jan '06

Hi, Is there any way which allows one to control the way cffi does the conversion from c-names to lispy names? Is it possible to disable this feature or customize it? Thanks. -- Surendra Singhi http://www.public.asu.edu/~sksinghi/index.html ,---- | By all means marry; if you get a good wife, you'll be happy. If you | get a bad one, you'll become a philosopher. | -- Socrates `----

2 1

[cffi-devel] New patches: 7-Jan-2006
by loliveira＠common-lisp.net 07 Jan '06

07 Jan '06

Sat Jan 7 07:15:26 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * add :default expression to auto-append library suffixes M ./src/libraries.lisp +20 Sat Jan 7 07:14:52 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * allow t in library feature expression as default M ./src/libraries.lisp +2 Sat Jan 7 12:05:58 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * remove some finished items from the TODO list M ./TODO -9 Sat Jan 7 06:34:55 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * signal an error when trying to load undefined "logical" foreign libraries M ./src/libraries.lisp -1 +3 Fri Jan 6 06:03:25 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * implement new callback interface for corman M ./src/cffi-corman.lisp -1 +8 Fri Jan 6 06:01:25 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * implement new callback interface for ecl Untested: the ECL backend does not seem to build for me. M ./src/cffi-ecl.lisp -4 +25 Fri Jan 6 05:57:53 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * implement new callback interface for cmucl Unsolved problems still prevent CFFI from working in a saved image on CMU CL. M ./src/cffi-cmucl.lisp -4 +28 Fri Jan 6 04:42:17 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * implement new callback interface for lispworks M ./src/cffi-lispworks.lisp -8 +30 Fri Jan 6 04:08:39 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * fix callbacks in saved images in clisp, sbcl, allegro, and openmcl M ./src/cffi-allegro.lisp -9 +48 M ./src/cffi-clisp.lisp -19 +61 M ./src/cffi-openmcl.lisp -8 +32 M ./src/cffi-sbcl.lisp -10 +11 M ./src/functions.lisp -4 +5 Sat Jan 7 04:17:09 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * New types. Sized ints and aliases. - Defined the :[u]int{8,16,32,64} types by looking at the sizes of the built-in types. - Removed :[u]int{8,16,32,64} from cffi-clisp. - New typedefs: :uchar, :ushort, :uint, :ulong, :llong, and :ullong. - New test: misc-types.sized-ints. - Removed the related TODO items. M ./TODO -4 M ./src/cffi-clisp.lisp -8 M ./src/early-types.lisp +4 M ./src/types.lisp -24 +53 M ./tests/misc-types.lisp +7 Sat Jan 7 03:18:35 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Substitute missing ensure-string. M ./src/libraries.lisp -1 +3 Sat Jan 7 03:13:48 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Remove the TODO item related to load-foreign-library M ./TODO -4 +1 Sat Jan 7 03:11:26 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * %foreign-funcall-pointer for ECL M ./src/cffi-ecl.lisp +8 Sat Jan 7 03:10:01 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Remove old comments in cffi-clisp and cffi-sbcl M ./src/cffi-clisp.lisp -7 M ./src/cffi-sbcl.lisp -6 Sat Jan 7 03:08:05 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Foreign library changes - Fix loading of foreign libraries in cffi-allegro. Now tries to load whatever is passed to %load-foreign-library instead of refusing to load files with types that aren't in excl::*load-foreign-types*. - Changes to %close-foreign-library in CMUCL and SBCL. Use string= instead of string-equal to match libraries. - Lispworks: use :connection-style :immediate in %load-foreign-library. - New foreign library interface. M ./src/cffi-allegro.lisp -5 +8 M ./src/cffi-cmucl.lisp -1 +1 M ./src/cffi-lispworks.lisp -1 +1 M ./src/cffi-sbcl.lisp -1 +1 M ./src/libraries.lisp -14 +193 M ./src/package.lisp -2 +9 Sat Jan 7 02:56:44 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Pushing cffi-features symbols to *features* - Renamed :cffi/no-foreign-funcall and :cffi/no-long-long to cffi-features:foreign-funcall and cffi-features:long-long respectively (with the opposite meanings of course). - Also new features (in the cffi-features package): darwin, unix, windows, ppc32 and x86. More could be added. The cffi-sys backends are responsible for pushing these features. M ./cffi.asd +1 M ./src/cffi-allegro.lisp -5 +12 M ./src/cffi-clisp.lisp -6 +14 M ./src/cffi-cmucl.lisp -5 +14 M ./src/cffi-corman.lisp -3 +9 M ./src/cffi-ecl.lisp -4 +13 M ./src/cffi-lispworks.lisp -5 +12 M ./src/cffi-openmcl.lisp -2 +9 M ./src/cffi-sbcl.lisp -1 +11 A ./src/features.lisp M ./src/package.lisp -3 M ./src/types.lisp -3 +3 M ./tests/callbacks.lisp -3 +3 M ./tests/defcfun.lisp -1 +1 M ./tests/foreign-globals.lisp -3 +3 M ./tests/funcall.lisp -3 +3 M ./tests/struct.lisp -1 +1 Sat Jan 7 02:54:54 CET 2006 Luis Oliveira <loliveira(a)common-lisp.net> * New TODO item about CLISP fasl portability M ./TODO +9 An updated tarball of CFFI's source can be downloaded here: http://common-lisp.net/project/cffi/tarballs/cffi-060107.tar.gz

1 0

[cffi-devel] New foreign library interface
by Luís Oliveira 07 Jan '06

07 Jan '06

Hello, There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like: (define-foreign-library opengl (:darwin (:framework "OpenGL")) (:unix (:alternatives "libGL.so" "libGL.so.1" #p"/myhome/mylibGL.so")) (:windows "opengl32.dll") ;; and a hypothetical example of a particular platform ;; where the OpenGL library is split in two. ((:and :some-system :some-cpu) "libGL-support.lib" "libGL-main.lib")) The library is then used with the following macro: (use-foreign-library opengl) So here's what's going on when loading this opengl library: - First, the proper clause is selected in a cond-like fashion. The symbols in there are tested against symbols in *features* that belong to the cffi-features[1] package. The :and, :or and :not operators are supported. - Then, the rest of the elements in the clause are handled. There can be more than one. For example you could define GLUT and OpenGL in one define-foreign-library form and whatever support libraries those might need. - Each of the elements can be: a) A string, eg. "libGL.so". In this case it's passed to load-foreign-library directly. What this means in e.g. unix-like systems is that the library will be searched in: 1. the LD_LIBRARY_PATH environment variable. 2. the directories in /etc/ld.so.cache (or equivalent) 3. /usr/lib and /lib If that fails, it tries to find it in the directories inside cffi:*foreign-library-directories* which is similar to asdf:*central-registry* in the sense that you can push stuff like '*default-pathname-defaults* or '(user-homedir-pathname) and they'll be "evaluated". b) A pathname, in which case CFFI doesn't try to find it and simply passes its namestring to load-foreign-library. c) A list of the form (:alternatives ...). Here we try to load each of the alternatives in order until one is loaded successfully. If none of the alternatives are loaded we get an error. d) A pair of the form (:framework "name"). CFFI will try to load the "name" framework, looking at the directories in cffi:*darwin-framework-directories* which is similar to cffi:*foreign-library-directories*. This list contains the following paths by default: 1. ~/Library/Frameworks 2. /Library/Frameworks 3. /System/Library/Frameworks Obviously, the user can push more paths into this list. There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework. Also, maybe a functional interface to use-foreign-library could be useful? Comments? [1] We'll be removing :cffi/no-foreign-funcall, and pushing cffi-features:foreign-funcall instead. Also, we provide others like cffi-features:{windows,unix,darwin,ppc32,x86} so that users can rely on these across different lisps. -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt

3 9

[cffi-devel] user: cffi in saved core?
by Keith Irwin 07 Jan '06

07 Jan '06

Folks-- I'm the guy with the toy "cl-browser" app which embeds gecko in a gtk app. Some context: What I want to do is create a nice standalone version so people don't have to know implementation details and I therefor (if this goes forward) don't have to write documentation. What this means for me is that I'll create a shell script, save a core with the code in it, save a copy of the runtime, build an installable package, etc, etc, and then, in my script, do something like: /usr/lib/cl-browser/my-sbcl --core /usr/lib/cl-browser/my-core --eval '(browser:main)' So far so good. This kind of thing has worked well for me on other projects. The problem: Alas, but when I try this with sbcl 0.9.7 and cffi (12/30/2005), I get a segmentation fault. Is this because of some problems with CFFI, something inherent in SBCL and its memory model? Here's what happens. sbcl * (require 'browser) * (save-lisp-and-die "foo.core") % sbcl --core foo.core --eval '(browser:main)' ... my debug output printf stuff ... Segmentation fault And that's the end of the story. The browser appears momentarily, then goes away. As near as I can tell, this happens about the time I try connecting callbacks to signals defined in the glade file. On one test of tall this, I got: connecting glade signals <== my printf fatal error encountered in SBCL pid 8097(tid 3085362880): fault in heap page not marked as write-protected So, does this mean we can't save a core with CFFI in it? I know there are other ways I can simulate a standalone executable, but I was just wondering if we users should just take for granted we can't build a new core file (with sbcl, anyway), or is this just an ongoing problem yet to be fixed? Thanks, Keith

2 1

[cffi-devel] DEFINE-FOREIGN-LIBRARY Additions
by James Bielman 07 Jan '06

07 Jan '06

After talking with ignas in #lisp about using this interface for GTK, I came up with a patch to add the following features to DEFINE-FOREIGN-LIBRARY: - A CFFI feature expression of T can be used to define a default clause. - If a library name is of the format (:DEFAULT STRING) then LOAD-FOREIGN-LIBRARY will append an OS-specific library suffix (.so, dll, .dylib, etc) to STRING before trying to load it. (We will need to extend this list as we test on more operating systems...) There is no facility for specifying versions with :DEFAULT; if you need to load a specific version, you must give the full library name. If the library you want to link against is "libfoo.so" on Linux, "libfoo.dylib" on Darwin, and "libfoo.dll" on Windows, then the following should do the right thing: (define-foreign-library foo (t (:default "libfoo"))) I was reluctant to add any default behavior that tries to interpret the foreign library name and add suffixes if not present. I think explicitly requesting addition of the suffix keeps the design cleaner. Luís, I put a patch file up if you'd like to look it over: http://slacknet.com/~jamesjb/suffixes.patch James

2 1

[cffi-devel] New patches: 6-Jan-2006
by loliveira＠common-lisp.net 06 Jan '06

06 Jan '06

Fri Jan 6 09:29:57 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * add :size option to defcunion as with defcstruct M ./src/types.lisp -17 +19 Fri Jan 6 08:47:08 CET 2006 James Bielman <jamesjb(a)jamesjb.com> * add todo list entry about setting union size M ./TODO +1 An updated tarball of CFFI's source can be downloaded here: http://common-lisp.net/project/cffi/tarballs/cffi-060106.tar.gz

1 0

Re: [cffi-devel] vararg (was: CFFI/CLISP - 2 out of 101 total tes ts failed)
by Hoehle, Joerg-Cyril 06 Jan '06

06 Jan '06

Yaroslav Kavenchuk wrote: >> Oops, if use libintl_sprintf instead of sprintf - all works: >> Hmm. I do not know why sprintf function does not work in clisp/mingw. >if define sprintf as function from msvcrt.dll - all works: Oh boy! >Why there is no opportunity to specify from what library link function? You probably mean in CFFI? UFFI has :module and CLISP has :library >And if I shall link some libraries with identical names of functions - >what to do? "Implementation-dependent" and underspecified On many UNIX systems, the more recently opened libraries probably shadow older ones. Now, do you know in which order the libraries where opened? The general answer is "no". E.g. did you read somewhere in clisp impnotes in what order it's reopening libraries when restarting from an image? Do you know what libraries got loaded in what order, when you're just a library programmer and don't know what the user is doing (and what s/he loaded)? No, you don't. There are several reasons why I prefer programmers to state which library or module thex expect a function to come from. It helps do the right thing. (But the most important reason back in 1995 was that the Amiga had no introspection of shared libraries. You *had* to tell you want e.g. Close from *that* specific library, and it was a reasonably way to avoid conflicts). >I do not know what library contains bad sprintf function. Maybe you can figure it out using (foreign-address #<foreign-function>) and see if you can relate that to the addresses of the .dll loaded into the current process? Regards, Jorg Hohle

6 10

Re: [cffi-devel] protect C frames from Lisp non-local exits in ca llbacks
by Hoehle, Joerg-Cyril 06 Jan '06

06 Jan '06

Stephen Compall wrote: >case of undefined behavior: >(block a > (block b > (unwind-protect (return-from a 1) > (return-from b 2)))) >[1] http://www.lisp.org/HyperSpec/Issues/iss152.html Steele had a nice text about breathing, roaming and fishing crabs in CLtL1 and II. >I would like to hear from those [...] running CFFI This has nothing to do with CFFI. Only the underlying implementation knows the answer. And the general answer is: don't exit non-locally across callbacks! It can always lead to unforeseen trouble. Rationale: a) Consider jumping to a shared library written in say C++, which has its own idea of unwind-protect. If you throw out of this via Lisp, the C++ destructor will never get called. Different run-times don't mix. b) Consider code written in C. It expects a functional style, i.e. it was not written with non-local exits in mind. It may set some variables or worse, allocate resources, call some function (your Lisp callback) and reset that variable upon exit. If you throw out of this via Lisp, the variable will not be reset, the resource onot freed. Your application may cease to work as a consequence of this (been there, done that). And it will take you time to debug and understand why. Of course, if you throw out of C:qsort(), I wouldn't expect trouble. That's no counterproof of the above. But consider calling into .NET (I think Edit Weitz has a binding to .NET). Non-local exits get you immediately out of any specified behaviour. >As an aside: would it be better for the defcallback semantics to simply >say "don't exit non-locally through a C frame"? Exactly. It's the only sensible thing to do. Ideally, every callback should contain something similar to ignore-errors (e.g. X event loops). >It would probably make >callbacks (and perhaps callouts, for that matter) cheaper to call, I don't understand that implication. Regards, Jorg Hohle.

2 1

[cffi-devel] vararg (was: CFFI/CLISP - 2 out of 101 total tests f ailed)
by Hoehle, Joerg-Cyril 06 Jan '06

06 Jan '06

Yaroslav Kavenchuk <kavenchuk(a)jenty.by> writes in cffi-devel: >> clisp from CVS head, cffi-051229.tar.gz >> 2 out of 101 total tests failed: CFFI-TESTS::FUNCALL.VARARGS.DOUBLE, >> CFFI-TESTS::STRUCT.NAMES. >About the FUNCALL.VARARGS.DOUBLE test, it passes on Linux and OSX but >fails on Win32. Yaroslav, I'd be grateful if you could express the testcase in terms of CLISP FFI forms (not CFFI) and submit it to the SF CLISP bugtracker, mentioning varargs in the subject line. So the clisp project would have a bugtracker issue for varargs and a place where to post comments. The current bugtracker FFI issues (long long, sparc64 etc.) are distinct from the specific varargs issue. Still, I maintain that attempts to interface to varargs is going to lead to errors here and there. E.g. I found the following comment in ffcall/avcall.h: :#ifdef rs6000 (applies to MacOS AFAIK) :float/double can be passed in float registers, but we also push them to the :corresponding int registers in case of varargs. I.e. ffcall contains special provision to make varargs work with doubles on PowerPC. It's nice to have, but it need not be there on other systems. And I must retract my recommendation to James to attempt o interface to *v() functions (those that take a va_list*). That's a priori not easier than the other functions. I expect the ffcall package may make this job easier in the future, since basically it itself builds up a similar array internally, so it could just pass a pointer to it to the *v() functions, but that's just a supposition of mine. So far my opinion remains that general infrastructure for varargs is not present in ffcall (despite particular work-around for e.g. PowerPC above), and that alone can explain the mentioned test results: + success on Linux and PowerPC/OSX - failure on Win32 (cygwin? mingw32? MS-VC?) IIRC, someody reported Lispworks explicitly mentions varargs not being supported, and Duane Rettig from Franz reported issues in cll etc. Furthermore, I wonder how a varargs call would look from the Lisp side? While the C compiler knows the args to printf() and can push either int, pointer or double on the stack, what should Lisp code do? A) Do a TYPECASE ((integer character) :int-or-long) ; but what about longlong? (string :string-or-pointer) (float :double via (coerce x 'double-float)) on each argument, at run-time? B) Or, for each call, declare a particular usage, e.g. corresponding to sprintf("%d%s%lld%f") and have the programmer write (foreign-funcall :int :string :long-long :[double-]float) ? That's IMHO still no solution with unknwon (at compile-time) format strings or other such arguments. James Bielman particular problem of interfacing to objc_msgSend[v]() might be solved like this: a) it's only relevant to one architecture (PowerPC/OSX) b) it could restrict itself to either :pointer or :long (ignore :double). Then the the stack layout (and the platform ABI) could be so regular that building the array and calling objc_msgSend() might work across all Lisps. Regards, Jörg Höhle.

2 3

[cffi-devel] Callback Fix Branch
by James Bielman 06 Jan '06

06 Jan '06

Hi all, I've made some progress updating the callback interface for each backend to work with callback pointers in a manner that should be safe when saving/reloading images. The branch is up at: http://slacknet.com/~jamesjb/cffi-callback-fix which can be obtained via 'darcs get', or pulled into an existing repository, or browsed over the web. The only significant visible change is the addition of an exported %CALLBACK function to the CFFI-SYS layer, which returns a pointer for a callback defined by %DEFCALLBACK. CFFI:CALLBACK is implemented in terms of CFFI-SYS:%CALLBACK instead of using the symbol's property list. This should also fix some problems on (at least) OpenMCL where the native callback was defined with a gensym'ed name, which means redefinitions of CFFI callbacks would not actually redefine the native callback. In CMU CL, almost every test fails from a saved image, with or without these changes---I'm not sure what's up with that yet. Allegro, CLISP, OpenMCL, and LispWorks now pass all expected tests from a saved image. (In CLISP, you need to reload foreign libraries by hand after loading the saved image. We might want to add another initialization hook to reload libraries loaded by CFFI...) SBCL passes all tests from a saved image with the PURIFY workaround, and should work out of the box when the underlying callback bug is fixed. I can't get the ECL backend to build, and I don't have access to Corman, so they are untested. Jörg, in the CLISP implementation, I'm a little concerned about using FFI::EXEC-ON-STACK to convert the functions to callbacks, but I couldn't find another way to get the effect of FFI:WITH-C-VAR at runtime, given a parsed type. Comments? James

1 0