[cffi-devel] New foreign library interface
Hello, There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like: (define-foreign-library opengl (:darwin (:framework "OpenGL")) (:unix (:alternatives "libGL.so" "libGL.so.1" #p"/myhome/mylibGL.so")) (:windows "opengl32.dll") ;; and a hypothetical example of a particular platform ;; where the OpenGL library is split in two. ((:and :some-system :some-cpu) "libGL-support.lib" "libGL-main.lib")) The library is then used with the following macro: (use-foreign-library opengl) So here's what's going on when loading this opengl library: - First, the proper clause is selected in a cond-like fashion. The symbols in there are tested against symbols in *features* that belong to the cffi-features[1] package. The :and, :or and :not operators are supported. - Then, the rest of the elements in the clause are handled. There can be more than one. For example you could define GLUT and OpenGL in one define-foreign-library form and whatever support libraries those might need. - Each of the elements can be: a) A string, eg. "libGL.so". In this case it's passed to load-foreign-library directly. What this means in e.g. unix-like systems is that the library will be searched in: 1. the LD_LIBRARY_PATH environment variable. 2. the directories in /etc/ld.so.cache (or equivalent) 3. /usr/lib and /lib If that fails, it tries to find it in the directories inside cffi:*foreign-library-directories* which is similar to asdf:*central-registry* in the sense that you can push stuff like '*default-pathname-defaults* or '(user-homedir-pathname) and they'll be "evaluated". b) A pathname, in which case CFFI doesn't try to find it and simply passes its namestring to load-foreign-library. c) A list of the form (:alternatives ...). Here we try to load each of the alternatives in order until one is loaded successfully. If none of the alternatives are loaded we get an error. d) A pair of the form (:framework "name"). CFFI will try to load the "name" framework, looking at the directories in cffi:*darwin-framework-directories* which is similar to cffi:*foreign-library-directories*. This list contains the following paths by default: 1. ~/Library/Frameworks 2. /Library/Frameworks 3. /System/Library/Frameworks Obviously, the user can push more paths into this list. There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework. Also, maybe a functional interface to use-foreign-library could be useful? Comments? [1] We'll be removing :cffi/no-foreign-funcall, and pushing cffi-features:foreign-funcall instead. Also, we provide others like cffi-features:{windows,unix,darwin,ppc32,x86} so that users can rely on these across different lisps. -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
On Wed, 2006-01-04 at 21:08 +0000, Luís Oliveira wrote:
There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like:
(define-foreign-library opengl (:darwin (:framework "OpenGL")) (:unix (:alternatives "libGL.so" "libGL.so.1" #p"/myhome/mylibGL.so")) (:windows "opengl32.dll") ;; and a hypothetical example of a particular platform ;; where the OpenGL library is split in two. ((:and :some-system :some-cpu) "libGL-support.lib" "libGL-main.lib"))
The only thing I might like to change here is that :alternatives seems a little verbose to me. Would it be confusing to just re-use :or for this? Or just a list with no leading keyword symbol?
The library is then used with the following macro:
(use-foreign-library opengl)
Eventually we need to sit down and figure out exactly what the evaluation-time issues are on each Lisp. I know that currently, because of at least CMU CL, we need to load the foreign library at compile-time because the alien functions must be present to compile functions that call them. I was wondering while stuck in traffic last night if we might be able to use a LOAD-TIME-VALUE in the CMU CL version of FOREIGN-FUNCALL to shift this requirement to load-time. I don't remember if any other Lisps need the function at compile time...
b) A pathname, in which case CFFI doesn't try to find it and simply passes its namestring to load-foreign-library.
I'm not sure I understand why the behavior for a pathname would be different than that of a namestring. If the pathname was relative, wouldn't it make sense to perform the same search if LOAD-FOREIGN-LIBRARY can't find it?
There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework.
Based on my experience with the UFFI interface, I think exporting FIND-* functions will lead to trouble. I see there being two scenarios for loading libraries: 1.) We want to "link" against a library, like -lfoo in gcc would do. 2.) We want to load a specific shared library at runtime, like a C user of 'dlopen' would do. I think DEFINE-FOREIGN-LIBRARY is the right thing for scenario 1, where we don't really care where the library is, we just want to system to find it for us. If we export functions to turn system libraries into absolute pathnames, users will end up hardcoding those pathnames into their saved Lisp images as often happens when using UFFI. Besides, it's not clear to me how we can perform the same search that a 'dlopen' would do without either trying to duplicate it, or trying it load it. For scenario 2, if the user doesn't want to load the library from the standard locations, he will need a custom search strategy for locating the library anyway, so whatever function we supply isn't going to cut it, I think.
Also, maybe a functional interface to use-foreign-library could be useful?
How about making LOAD-FOREIGN-LIBRARY load a "logical" library defined with DEFINE-FOREIGN-LIBRARY if passed a symbol? James
On 2006-jan-05, at 09:28, James Bielman wrote:
The only thing I might like to change here is that :alternatives seems a little verbose to me. Would it be confusing to just re-use :or for this? Or just a list with no leading keyword symbol?
Yeah, your original suggestion was to use just a list, but I added a symbol to more easily differenciate it from the other possible list: (:framework foo). I think I'll go with the :OR suggestion, since it's probably more readable than just a list too.
b) A pathname, in which case CFFI doesn't try to find it and simply passes its namestring to load-foreign-library.
I'm not sure I understand why the behavior for a pathname would be different than that of a namestring. If the pathname was relative, wouldn't it make sense to perform the same search if LOAD-FOREIGN-LIBRARY can't find it?
Right, makes sense. [about find-foreign-library and friends]
Besides, it's not clear to me how we can perform the same search that a 'dlopen' would do without either trying to duplicate it, or trying it load it.
For scenario 2, if the user doesn't want to load the library from the standard locations, he will need a custom search strategy for locating the library anyway, so whatever function we supply isn't going to cut it, I think.
Oh yes, find-foreign-library only searches in the directories that are passed to it as an argument (usually cffi:*foreign-library- directories*) not in the places that ldopen would search. But I suppose that could help in such custom search strategy anyway? Well, find-foreign-library is probably a misnomer. It's more of a find-file-in-a-list-of-directories, really.
Also, maybe a functional interface to use-foreign-library could be useful?
How about making LOAD-FOREIGN-LIBRARY load a "logical" library defined with DEFINE-FOREIGN-LIBRARY if passed a symbol?
While we're at it, I suppose that it could take a (:framework "foo") pair as well as a list of alternatives: (:or "lib1.so" "lib2.so")? And have it search in cffi:*foreign-library-directories* too. Ie. load-foreign-library should handle its argument exactly the same way an "element" in define-foreign-library is handled. Sounds intuitive. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
Am 04.01.2006 um 22:08 schrieb Luís Oliveira:
Hello,
There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like:
(define-foreign-library opengl
Is opengl then interned as a symbol (in which package) ? Or a keyword ?
(:darwin (:framework "OpenGL")) (:unix (:alternatives "libGL.so" "libGL.so.1" #p"/myhome/mylibGL.so")) (:windows "opengl32.dll") ;; and a hypothetical example of a particular platform ;; where the OpenGL library is split in two. ((:and :some-system :some-cpu) "libGL-support.lib" "libGL- main.lib"))
I like this approach very much as it caters for every case I came across. I think I also would follow James Bielman's suggestion to drop the :alternatives think in favor for a :or approach. On Darwin, the :framework options is mandatory of course and should also handle bundles.
The library is then used with the following macro:
(use-foreign-library opengl)
So here's what's going on when loading this opengl library:
- First, the proper clause is selected in a cond-like fashion. The symbols in there are tested against symbols in *features* that belong to the cffi-features[1] package. The :and, :or and :not operators are supported.
- Then, the rest of the elements in the clause are handled. There can be more than one. For example you could define GLUT and OpenGL in one define-foreign-library form and whatever support libraries those might need.
- Each of the elements can be:
a) A string, eg. "libGL.so". In this case it's passed to load-foreign-library directly. What this means in e.g. unix- like systems is that the library will be searched in:
1. the LD_LIBRARY_PATH environment variable. 2. the directories in /etc/ld.so.cache (or equivalent) 3. /usr/lib and /lib
This is completely platform dependant (as you probably know ;-) So, if you want to handle all the cases in CFFI then there are at least the following cases in addition: HP-UX: - SHLIB_PATH env var - no cache file for the dynamic loader - same dirs, plus I'd recommend to include /usr/local/lib, /usr/ lib/X11 (this is true for all unix like systems) Mac OS X (Darwin): - DYLIB_LIBRARY_PATH env var - I don't know of any cache file... - same dirs General note on Darwin: It is required not to load any lib or framework twice. Also, order of loading matters. So, it should be defined that loading occurs left to right as stated in the define- foreign library call. SUN anyone ???
If that fails, it tries to find it in the directories inside cffi:*foreign-library-directories* which is similar to asdf:*central-registry* in the sense that you can push stuff like '*default-pathname-defaults* or '(user-homedir-pathname) and they'll be "evaluated".
b) A pathname, in which case CFFI doesn't try to find it and simply passes its namestring to load-foreign-library.
c) A list of the form (:alternatives ...). Here we try to load each of the alternatives in order until one is loaded successfully. If none of the alternatives are loaded we get an error.
d) A pair of the form (:framework "name"). CFFI will try to load the "name" framework, looking at the directories in cffi:*darwin-framework-directories* which is similar to cffi:*foreign-library-directories*. This list contains the following paths by default:
1. ~/Library/Frameworks 2. /Library/Frameworks 3. /System/Library/Frameworks
Obviously, the user can push more paths into this list.
There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework. Also, maybe a functional interface to use-foreign-library could be useful?
Comments?
What I am missing is the support for handling multiple versions of foreign libs. It should be possible to state which version to load if more than one version is installed or if a certain version is required. I know this no trivial task as the version naming of libs is not unified. Also, there has to some means of specifying what to do if the requested lib version is not available. Some fallback behaviour like :load-newest or :signal-error or :load-newest-if- higher-version ... Did anyone else come across this requirement? Cheers, Frank
On 2006-jan-06, at 20:13, Frank Goenninger - PRION Consulting wrote:
(define-foreign-library opengl
Is opengl then interned as a symbol (in which package) ? Or a keyword ?
Yes, the current package. You can use a keyword if you prefer. Probably better to use a normal symbol though.
This is completely platform dependant (as you probably know ;-) So, if you want to handle all the cases in CFFI then there are at least the following cases in addition:
HP-UX: - SHLIB_PATH env var [etc..]
We don't really search for the library ourselves. dlopen() takes care of that for us. It was just an example.
What I am missing is the support for handling multiple versions of foreign libs. It should be possible to state which version to load if more than one version is installed or if a certain version is required. I know this no trivial task as the version naming of libs is not unified. Also, there has to some means of specifying what to do if the requested lib version is not available. Some fallback behaviour like :load-newest or :signal-error or :load-newest-if- higher-version ...
Would using :OR be enough? (:or "libFoo.so.2.2" "libFoo.so.2.1" "libFoo.so.2" "libFoo.so") or something like that? Also, in that private e-mail you sent me you mention that USE-FOREIGN- LIBRARY should evaluate its argument. But, the purpose of this macro is to wrap LOAD-FOREIGN-LIBRARY in an EVAL-WHEN basically because of CMUCL which needs the libraries loaded at compile-time. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
Am 07.01.2006 um 02:20 schrieb Luís Oliveira:
On 2006-jan-06, at 20:13, Frank Goenninger - PRION Consulting wrote:
(define-foreign-library opengl
Is opengl then interned as a symbol (in which package) ? Or a keyword ?
Yes, the current package. You can use a keyword if you prefer. Probably better to use a normal symbol though.
Yes, I agree.
This is completely platform dependant (as you probably know ;-) So, if you want to handle all the cases in CFFI then there are at least the following cases in addition:
HP-UX: - SHLIB_PATH env var [etc..]
We don't really search for the library ourselves. dlopen() takes care of that for us. It was just an example.
Ah. So, if you want to make dlopen() handle all searching, then you'd have to extend the respective environment variable with user-supplied search paths. Which brings us to ? Yes, platform-dependent putenv() calls or whatever the Lisp implementation's way is to set an env var. Or do I misunderstand still somethimng here?
What I am missing is the support for handling multiple versions of foreign libs. It should be possible to state which version to load if more than one version is installed or if a certain version is required. I know this no trivial task as the version naming of libs is not unified. Also, there has to some means of specifying what to do if the requested lib version is not available. Some fallback behaviour like :load-newest or :signal-error or :load- newest-if-higher-version ...
Would using :OR be enough? (:or "libFoo.so.2.2" "libFoo.so.2.1" "libFoo.so.2" "libFoo.so") or something like that?
Yes, if left-to-right search order is maintained and this use case documented as an example for how to deal with versions ;-)
Also, in that private e-mail you sent me you mention that USE- FOREIGN-LIBRARY should evaluate its argument. But, the purpose of this macro is to wrap LOAD-FOREIGN-LIBRARY in an EVAL-WHEN basically because of CMUCL which needs the libraries loaded at compile-time.
Yes, I now also figured that it's OK not to evaluate because of that case. I wasn't aware that CMUCL needs this. Thanks for the reply! Cheers Frank
On 2006-jan-07, at 16:52, Frank Goenninger - PRION Consulting wrote:
We don't really search for the library ourselves. dlopen() takes care of that for us. It was just an example.
Ah. So, if you want to make dlopen() handle all searching, then you'd have to extend the respective environment variable with user- supplied search paths. Which brings us to ? Yes, platform-dependent putenv() calls or whatever the Lisp implementation's way is to set an env var. Or do I misunderstand still somethimng here?
Hmm, I'm not sure what you mean. What happens is that, if dlopen() can't find the library, we try finding it in one of the directories in the cffi:*foreign-library-directories* list. -- Luís Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
Am 07.01.2006 um 20:25 schrieb Luís Oliveira:
On 2006-jan-07, at 16:52, Frank Goenninger - PRION Consulting wrote:
We don't really search for the library ourselves. dlopen() takes care of that for us. It was just an example.
Ah. So, if you want to make dlopen() handle all searching, then you'd have to extend the respective environment variable with user- supplied search paths. Which brings us to ? Yes, platform- dependent putenv() calls or whatever the Lisp implementation's way is to set an env var. Or do I misunderstand still somethimng here?
Hmm, I'm not sure what you mean. What happens is that, if dlopen() can't find the library, we try finding it in one of the directories in the cffi:*foreign-library-directories* list.
I assumed you would do that. Now you've confirmed that. Thanks. dlopen() of course does not always return a real error if it couldn't find the lib (in some cases I had a return value of NULL which in some cases meant that there was an error and in some cases not) . I had cases (on HP-UX, where you use shl_load() because there's no dlopen() in this OS) where I had to examine errno (and on other OSes, where available, with dlerror() ...) to find out what really happened. Thanks again - having talked/written that much now I figure I should start supporting you and begin coding that stuff... Will look for the source code. Frank -- PRION Group - PLM and IT Solutions Frank Goenninger CEO & Managing Partner Consulting Fon: +49 711 797353-30 Fax: +49 711 797353-50 E-Mail: fgoenninger@prion.de PRION Consulting Services AG Nikolaus-Otto-Straße 25 D-70771 Leinfelden-Echterdingen, Germany Geschäftsführung: Vorsitzender der Vorstands: Frank Gönninger Vorsitzender des Aufsichtsrats: Christopher Frim Sitz der Gesellschaft: 70771 Leinfelden-Echterdingen, Deutschland Registergericht Nürtingen HRB 5892
Answering to myself - one more case of "you better read before writing"... See below. Am 07.01.2006 um 21:56 schrieb Frank Goenninger - PRION Consulting:
Am 07.01.2006 um 20:25 schrieb Luís Oliveira:
On 2006-jan-07, at 16:52, Frank Goenninger - PRION Consulting wrote:
We don't really search for the library ourselves. dlopen() takes care of that for us. It was just an example.
Ah. So, if you want to make dlopen() handle all searching, then you'd have to extend the respective environment variable with user-supplied search paths. Which brings us to ? Yes, platform- dependent putenv() calls or whatever the Lisp implementation's way is to set an env var. Or do I misunderstand still somethimng here?
Hmm, I'm not sure what you mean. What happens is that, if dlopen() can't find the library, we try finding it in one of the directories in the cffi:*foreign-library-directories* list.
I assumed you would do that. Now you've confirmed that. Thanks.
dlopen() of course does not always return a real error if it couldn't find the lib (in some cases I had a return value of NULL which in some cases meant that there was an error and in some cases not) . I had cases (on HP-UX, where you use shl_load() because there's no dlopen() in this OS) where I had to examine errno (and on other OSes, where available, with dlerror() ...) to find out what really happened.
Thanks again - having talked/written that much now I figure I should start supporting you and begin coding that stuff... Will look for the source code.
I figured by reading the different cffi-XXX.lisp files that you rely on the different implementations' functions to handle all these things. Ok, now that I know that I better shut up... I assumed from your messages that you really call dlopen() and such directly - on second thought, well, I see that is useless... Frank
Frank Goenninger - PRION Consulting <fgoenninger@prion.de> writes:
I figured by reading the different cffi-XXX.lisp files that you rely on the different implementations' functions to handle all these things. Ok, now that I know that I better shut up... I assumed from your messages that you really call dlopen() and such directly - on second thought, well, I see that is useless...
Oops. Indeed, I wasn't very clear, sorry. -- Luís Oliveira luismbo (@) gmail (.) com Equipa Portuguesa do Translation Project http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
participants (3)
-
Frank Goenninger - PRION Consulting -
James Bielman -
Luís Oliveira