On Wed, 2006-01-04 at 21:08 +0000, Luís Oliveira wrote:

There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like:

(define-foreign-library opengl (:darwin (:framework "OpenGL")) (:unix (:alternatives "libGL.so" "libGL.so.1" #p"/myhome/mylibGL.so")) (:windows "opengl32.dll") ;; and a hypothetical example of a particular platform ;; where the OpenGL library is split in two. ((:and :some-system :some-cpu) "libGL-support.lib" "libGL-main.lib"))

The only thing I might like to change here is that :alternatives seems a little verbose to me. Would it be confusing to just re-use :or for this? Or just a list with no leading keyword symbol?

The library is then used with the following macro:

(use-foreign-library opengl)

Eventually we need to sit down and figure out exactly what the evaluation-time issues are on each Lisp. I know that currently, because of at least CMU CL, we need to load the foreign library at compile-time because the alien functions must be present to compile functions that call them.

I was wondering while stuck in traffic last night if we might be able to use a LOAD-TIME-VALUE in the CMU CL version of FOREIGN-FUNCALL to shift this requirement to load-time. I don't remember if any other Lisps need the function at compile time...

b) A pathname, in which case CFFI doesn't try to find it and simply
   passes its namestring to load-foreign-library.

I'm not sure I understand why the behavior for a pathname would be different than that of a namestring. If the pathname was relative, wouldn't it make sense to perform the same search if LOAD-FOREIGN-LIBRARY can't find it?

There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework.

Based on my experience with the UFFI interface, I think exporting FIND-* functions will lead to trouble. I see there being two scenarios for loading libraries:

1.) We want to "link" against a library, like -lfoo in gcc would do.

2.) We want to load a specific shared library at runtime, like a C user of 'dlopen' would do.

I think DEFINE-FOREIGN-LIBRARY is the right thing for scenario 1, where we don't really care where the library is, we just want to system to find it for us. If we export functions to turn system libraries into absolute pathnames, users will end up hardcoding those pathnames into their saved Lisp images as often happens when using UFFI.

Besides, it's not clear to me how we can perform the same search that a 'dlopen' would do without either trying to duplicate it, or trying it load it.

For scenario 2, if the user doesn't want to load the library from the standard locations, he will need a custom search strategy for locating the library anyway, so whatever function we supply isn't going to cut it, I think.

Also, maybe a functional interface to use-foreign-library could be useful?

How about making LOAD-FOREIGN-LIBRARY load a "logical" library defined with DEFINE-FOREIGN-LIBRARY if passed a symbol?

James

Am 04.01.2006 um 22:08 schrieb Luís Oliveira:

Hello,

There's been some discussion on IRC about what CFFI's "smarter" foreign library interface should look like. I put together some code and here's what my attempt looks like:

(define-foreign-library opengl

Is opengl then interned as a symbol (in which package) ? Or a keyword ?

(:darwin  (:framework "OpenGL"))
(:unix    (:alternatives "libGL.so" "libGL.so.1"
                         #p"/myhome/mylibGL.so"))
(:windows "opengl32.dll")
;; and a hypothetical example of a particular platform
;; where the OpenGL library is split in two.
((:and :some-system :some-cpu) "libGL-support.lib" "libGL- 

main.lib"))

I like this approach very much as it caters for every case I came across. I think I also would follow James Bielman's suggestion to drop the :alternatives think in favor for a :or approach. On Darwin, the :framework options is mandatory of course and should also handle bundles.

The library is then used with the following macro:

(use-foreign-library opengl)

So here's what's going on when loading this opengl library:

• First, the proper clause is selected in a cond-like fashion. The symbols in there are tested against symbols in *features* that belong to the cffi-features[1] package. The :and, :or and :not operators are supported.

• Then, the rest of the elements in the clause are handled. There

can be more than one. For example you could define GLUT and OpenGL in one define-foreign-library form and whatever support libraries those might need.

• Each of the elements can be:

  a) A string, eg. "libGL.so". In this case it's passed to load-foreign-library directly. What this means in e.g. unix-

like systems is that the library will be searched in:

     1. the LD_LIBRARY_PATH environment variable.
     2. the directories in /etc/ld.so.cache (or equivalent)
     3. /usr/lib and /lib

This is completely platform dependant (as you probably know ;-) So, if you want to handle all the cases in CFFI then there are at least the following cases in addition:

HP-UX: - SHLIB_PATH env var - no cache file for the dynamic loader - same dirs, plus I'd recommend to include /usr/local/lib, /usr/ lib/X11 (this is true for all unix like systems)

Mac OS X (Darwin): - DYLIB_LIBRARY_PATH env var - I don't know of any cache file... - same dirs

General note on Darwin: It is required not to load any lib or framework twice. Also, order of loading matters. So, it should be defined that loading occurs left to right as stated in the define- foreign library call.

SUN anyone ???

   If that fails, it tries to find it in the directories inside
   cffi:*foreign-library-directories* which is similar to
   asdf:*central-registry* in the sense that you can push
   stuff like '*default-pathname-defaults* or
   '(user-homedir-pathname) and they'll be "evaluated".

b) A pathname, in which case CFFI doesn't try to find it and  

simply passes its namestring to load-foreign-library.

c) A list of the form (:alternatives ...). Here we try to load  

each of the alternatives in order until one is loaded successfully. If none of the alternatives are loaded we get an error.

d) A pair of the form (:framework "name"). CFFI will try to  

load the "name" framework, looking at the directories in cffi:*darwin-framework-directories* which is similar to cffi:*foreign-library-directories*. This list contains the following paths by default:

     1. ~/Library/Frameworks
     2. /Library/Frameworks
     3. /System/Library/Frameworks

   Obviously, the user can push more paths into this list.

There are some functions that I'm not sure are worth exporting: find-foreign-library, find-and-load-foreign-library and find-darwin-framework. Also, maybe a functional interface to use-foreign-library could be useful?

Comments?

What I am missing is the support for handling multiple versions of foreign libs. It should be possible to state which version to load if more than one version is installed or if a certain version is required. I know this no trivial task as the version naming of libs is not unified. Also, there has to some means of specifying what to do if the requested lib version is not available. Some fallback behaviour like :load-newest or :signal-error or :load-newest-if- higher-version ...

Did anyone else come across this requirement?

Cheers, Frank