Hello, CFFI folks. What are the implications of declaring two foreign functions, each foreign name being the same but which are implemented in two separate DLLs? The specific case I have in mind is the Win32 function DllGetVersion, which can return different meta-data for comctl32.dll vs. shell32.dll.
I am effectively doing the following right now:
(load-foreign-library "comctl32.dll")
(defcfun ("DllGetVersion" comctl-dll-get-version) HRESULT (info :pointer))
(load-foreign-library "shell32.dll")
(defcfun ("DllGetVersion" shell-dll-get-version) HRESULT (info :pointer))
Is there any caching of CFFI foreign function meta-data behind the scenes, perhaps keyed by the foreign function name? Such as might be causing identical results no matter which is called first, or which load-foreign-library occurs first?
Note: my WinXP install has several different versions of comctl32.dll, one of which matches the shell32.dll version, but I have at least one configuration where I know the result really is wrong (as if the result from the other DllGetVersion call was being cached).
I wrote:
Note: my WinXP install has several different versions of comctl32.dll, one of which matches the shell32.dll version, but I have at least one configuration where I know the result really is wrong (as if the result from the other DllGetVersion call was being cached).
To alleviate possible confusion, what I meant to write is that it's as if the other DllGetVersion was being called instead of the correct one.
On 17 Dec 2006, at 21:15, Jack Unrue wrote:
Is there any caching of CFFI foreign function meta-data behind the scenes, perhaps keyed by the foreign function name? Such as might be causing identical results no matter which is called first, or which load- foreign-library occurs first?
Note: my WinXP install has several different versions of comctl32.dll, one of which matches the shell32.dll version, but I have at least one configuration where I know the result really is wrong (as if the result from the other DllGetVersion call was being cached).
The issue here is that CFFI doesn't have any way to tell which of the DllGetVersion functions you want to call. Both DEFCFUNs are pointing at the same name. This is one of the things that UFFI still handles better (unless I just haven't been keeping up-to-date with CFFI).
In UFFI, rather than
(load-foreign-library "comctl32.dll")
You would say:
(load-foreign-library "comctl32.dll" :module "comctl") (load-foreign-library "shell32.dll" :module "shell")
then, when defining the functions:
(def-function ("DllGetVersion" comctl-dll-get-version) ((info :pointer)) :returning HRESULT :module "comctl")
(def-function ("DllGetVersion" shell-dll-get-version) ((info :pointer)) :returning HRESULT :module "shell")
you can see the extra ":module" parameter, which tells it which DLL contains the function you want to use. I'm pretty sure it's not possible to do anything similar from CFFI.
I thought this was being worked on, but I haven't been paying attention too much lately.
Gregory Martin Pfeil sellout42@mac.com writes:
The issue here is that CFFI doesn't have any way to tell which of the DllGetVersion functions you want to call. Both DEFCFUNs are pointing at the same name. This is one of the things that UFFI still handles better (unless I just haven't been keeping up-to-date with CFFI).
In UFFI, rather than
(load-foreign-library "comctl32.dll")
You would say:
(load-foreign-library "comctl32.dll" :module "comctl") (load-foreign-library "shell32.dll" :module "shell")
then, when defining the functions:
(def-function ("DllGetVersion" comctl-dll-get-version) ((info :pointer)) :returning HRESULT :module "comctl")
(def-function ("DllGetVersion" shell-dll-get-version) ((info :pointer)) :returning HRESULT :module "shell")
you can see the extra ":module" parameter, which tells it which DLL contains the function you want to use. I'm pretty sure it's not possible to do anything similar from CFFI.
I believe most/many/some lisp implementations ignore the :MODULE in UFFI.
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs."
Hi,
This is one of the things that UFFI still handles better.
Indeed.
I believe most/many/some lisp implementations ignore the :MODULE in UFFI.
Only those that only work on UNIX, where a name is mapped to one address in the process space (and shadowing by precedence (order of opening libraries) rules apply). Not so on MS-Windows, as Jack Unrue shows: each library may export a function of a given name, and they could be the same (even more so with COM and interfaces: all subscribers provide the function of the name mandated by the interface).
IIRC, Yaroslav Kavenchuk once had a case where calling the function in the wrong library caused a crash.
Some Lisp implementations, when porting from UNIX to MS-Windows, preserved their UNIX API and threw in ad hoc behaviour: e.g. given no :library, scan the open ones in some order. That's why even on MS-Windows, where naming the correct library can be of utter importance, that argument may not be required or even available. This approach is good enough for many programs to work.
I've said for years now that the :library/:module thing is important. Alas, if one has a UNIX-centric view, one fails to recognize the value of this parameter -- which reminds me of a comment from Kent Pitman in cll some years ago.
Jack, you must roll your own. Luckily, you can do so. Use CFFI as soon as you get the address (pointer) of the function in the library that is of interest to you. Given that pointer, CFFI provides enough glue to be able to construct (call) a foreign function out of it, e.g. foreign-funcall-pointer. Also, use CFFI to obtain that pointer (via GetProcAddress() if all else fails).
Or use #+ to write native def-call-out forms for several Lisps that support the library argument?
Regards, Jorg Hohle.
On 12/19/06, Hoehle, Joerg-Cyril Joerg-Cyril.Hoehle@t-systems.com wrote:
Jack, you must roll your own. Luckily, you can do so. Use CFFI as soon as you get the address (pointer) of the function in the library that is of interest to you. Given that pointer, CFFI provides enough glue to be able to construct (call) a foreign function out of it, e.g. foreign-funcall-pointer. Also, use CFFI to obtain that pointer (via GetProcAddress() if all else fails).
Yep, that's a good idea.
Or use #+ to write native def-call-out forms for several Lisps that support the library argument?
That's also a possibility.
Thanks everyone for your thoughts on this problem.
"Jack Unrue" jdunrue@gmail.com writes:
Hello, CFFI folks. What are the implications of declaring two foreign functions, each foreign name being the same but which are implemented in two separate DLLs? The specific case I have in mind is the Win32 function DllGetVersion, which can return different meta-data for comctl32.dll vs. shell32.dll.
(load-foreign-library "comctl32.dll")
(defcfun ("DllGetVersion" comctl-dll-get-version) HRESULT (info :pointer))
(load-foreign-library "shell32.dll")
(defcfun ("DllGetVersion" shell-dll-get-version) HRESULT (info :pointer))
For SBCL 1.0 at least both will be backed by the same foreign function, namely the first one to be loaded.
The Right Thing, I believe, would be something like
(load-foreign-library "foo.dll" :name :foo)
(defcfun ("FooFun" foo-fun :library :foo) ...)
which would not be too hard to support on the SBCL end of things at least.
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious." Lispnik: "Buddha is big, has hairy armpits, and laughs."
The Right Thing, I believe, would be something like
(load-foreign-library "foo.dll" :name :foo)
(defcfun ("FooFun" foo-fun :library :foo) ...)
maybe one could create a defcpackage and an in-cpackage that map directly defpackage and in-package with a hashtable behind the scenes mapping from packages to foreign libraries.
just some idea,
Hello,
On 12/18/06, Jack Unrue jdunrue@gmail.com wrote:
Hello, CFFI folks. What are the implications of declaring two foreign functions, each foreign name being the same but which are implemented in two separate DLLs?
I've added this to the TODO list. I've also thought a little about how to support this best, API-wise. Besides the obvious :LIBRARY (or some other name) to DEFCFUN and friends it'd be nice if CFFI could infer the "current library" somehow. An IN-FOREIGN-LIBRARY macro for instance, though that might break when doing incremental development with SLIME for instance.
DEFINE-FOREIGN-LIBRARY could record the library name's symbol-package and then DEFCFUN could look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
This kind of mechanism (either this last one, or an IN-FOREIGN-LIBRARY, or something else) would also be useful for library-wide settings such as stdcall vs. cdecl. That's actually the context in which I had thought about this.
I'm sure this kind of problem has been dealt with before in other libraries (IIRC, five-am has an IN-SUITE macro). Any ideas?
On Tue, 2006-12-19 at 19:53 +0000, Luís Oliveira wrote:
I've added this to the TODO list. I've also thought a little about how to support this best, API-wise. Besides the obvious :LIBRARY (or some other name) to DEFCFUN and friends it'd be nice if CFFI could infer the "current library" somehow. An IN-FOREIGN-LIBRARY macro for instance, though that might break when doing incremental development with SLIME for instance.
DEFINE-FOREIGN-LIBRARY could record the library name's symbol-package and then DEFCFUN could look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
This kind of mechanism (either this last one, or an IN-FOREIGN-LIBRARY, or something else) would also be useful for library-wide settings such as stdcall vs. cdecl. That's actually the context in which I had thought about this.
I'm sure this kind of problem has been dealt with before in other libraries (IIRC, five-am has an IN-SUITE macro). Any ideas?
I've thought about this a little before as well, and I think we also want to be able to handle the use case of "plugin" type shared libraries where we want to dynamically call the same function "FooInitModule" in a programatically loaded library.
I haven't had the time to think about what this should look like yet, but if we're going to solve half the problem, we should probably solve all of it IMHO.
James
On 12/19/06, Luís Oliveira luismbo@gmail.com wrote:
I've added this to the TODO list. I've also thought a little about how to support this best, API-wise. Besides the obvious :LIBRARY (or some other name) to DEFCFUN and friends it'd be nice if CFFI could infer the "current library" somehow. An IN-FOREIGN-LIBRARY macro for instance, though that might break when doing incremental development with SLIME for instance.
Yeah, I see what you're getting at. Otherwise, I guess one has to encode context into the Lisp name of each function, sort of like I did in my DllGetVersion example.
DEFINE-FOREIGN-LIBRARY could record the library name's symbol-package and then DEFCFUN could look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
I don't know about silly :-) but I would vote against that option if it were to be the only solution. Even though I can imagine use-cases where that might be reasonable, it's not a natural thing to do. C++ programmers are not required to declare a new namespace for each DLL that they use, for example.
This kind of mechanism (either this last one, or an IN-FOREIGN-LIBRARY, or something else) would also be useful for library-wide settings such as stdcall vs. cdecl. That's actually the context in which I had thought about this.
I'm sure this kind of problem has been dealt with before in other libraries (IIRC, five-am has an IN-SUITE macro). Any ideas?
I've only got real experience implementing one library in CL, so I can't help there.
P.S. -- Sorry Jack, for the duplicate message.
No problem :-)
Jack Unrue wrote:
On 12/19/06, Luís Oliveira luismbo@gmail.com wrote:
I've added this to the TODO list. I've also thought a little about how to support this best, API-wise. Besides the obvious :LIBRARY (or some other name) to DEFCFUN and friends it'd be nice if CFFI could infer the "current library" somehow. An IN-FOREIGN-LIBRARY macro for instance, though that might break when doing incremental development with SLIME for instance.
Yeah, I see what you're getting at. Otherwise, I guess one has to encode context into the Lisp name of each function, sort of like I did in my DllGetVersion example.
DEFINE-FOREIGN-LIBRARY could record the library name's symbol-package and then DEFCFUN could look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
I don't know about silly :-) but I would vote against that option if it were to be the only solution. Even though I can imagine use-cases where that might be reasonable, it's not a natural thing to do. C++ programmers are not required to declare a new namespace for each DLL that they use, for example.
This kind of mechanism (either this last one, or an IN-FOREIGN-LIBRARY, or something else) would also be useful for library-wide settings such as stdcall vs. cdecl. That's actually the context in which I had thought about this.
I'm sure this kind of problem has been dealt with before in other libraries (IIRC, five-am has an IN-SUITE macro). Any ideas?
The package solution doesn't seem that silly to me --- an additional namespace doesn't seem like a huge price to pay. An alternative encapsulation method would be to use CLOS and locate different entry points relative to an object that corresponds to the library. I guess the choice would depend on whether you prefer namespaces or objects.
On 12/19/06, Robert P. Goldman rpgoldman@sift.info wrote:
Jack Unrue wrote:
On 12/19/06, Luís Oliveira luismbo@gmail.com wrote:
I've added this to the TODO list. I've also thought a little about how to support this best, API-wise. Besides the obvious :LIBRARY (or some other name) to DEFCFUN and friends it'd be nice if CFFI could infer the "current library" somehow. An IN-FOREIGN-LIBRARY macro for instance, though that might break when doing incremental development with SLIME for instance.
Yeah, I see what you're getting at. Otherwise, I guess one has to encode context into the Lisp name of each function, sort of like I did in my DllGetVersion example.
DEFINE-FOREIGN-LIBRARY could record the library name's symbol-package and then DEFCFUN could look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
I don't know about silly :-) but I would vote against that option if it were to be the only solution. Even though I can imagine use-cases where that might be reasonable, it's not a natural thing to do. C++ programmers are not required to declare a new namespace for each DLL that they use, for example.
This kind of mechanism (either this last one, or an IN-FOREIGN-LIBRARY, or something else) would also be useful for library-wide settings such as stdcall vs. cdecl. That's actually the context in which I had thought about this.
I'm sure this kind of problem has been dealt with before in other libraries (IIRC, five-am has an IN-SUITE macro). Any ideas?
The package solution doesn't seem that silly to me --- an additional namespace doesn't seem like a huge price to pay. An alternative encapsulation method would be to use CLOS and locate different entry points relative to an object that corresponds to the library. I guess the choice would depend on whether you prefer namespaces or objects.
Unless I'm misunderstanding, it seems like what Luís suggested would require the call sites for DLL functions to be in the same package as where they are defined. This would go against what I think is a common idiom -- defining a system-level package for low-level FFI bindings and then higher-level abstractions in a separate package which call those bindings. I structure my library that way, for example.
Is there some combination of compiler macros and/or symbol macros that could be defined such that the appropriate DLL is loaded, the function pointer retrieved and then called, and then the DLL unloaded, at each call site? The reason I suggest that is that I see a similar approach being used in SWT (from Java) where they call DllGetVersion() from comctl32 and shell32, loading and unloading each DLL in turn. The function pointer is retrieved on-the-spot in both cases.
I wrote:
Is there some combination of compiler macros and/or symbol macros that could be defined such that the appropriate DLL is loaded, the function pointer retrieved and then called, and then the DLL unloaded, at each call site? The reason I suggest that is that I see a similar approach being used in SWT (from Java) where they call DllGetVersion() from comctl32 and shell32, loading and unloading each DLL in turn. The function pointer is retrieved on-the-spot in both cases.
...which is essentially what Jörg suggested, I'm just saying maybe some macrology could hide the details.
Jack Unrue wrote:
Is there some combination of compiler macros and/or symbol macros that could be defined such that the appropriate DLL is loaded, the function pointer retrieved and then called, and then the DLL
unloaded,
at each call site? The reason I suggest that is that I see a
similar approach
being used in SWT (from Java) where they call DllGetVersion() from comctl32 and shell32, loading and unloading each DLL in
turn. The function
pointer is retrieved on-the-spot in both cases.
...which is essentially what Jörg suggested, I'm just saying maybe some macrology could hide the details.
I was not that detailed. Quite to the contrary, I had in mind a system which would locate the address once, (e.g. when evaluating def-call-out), not each time the function is called (irk!). Loading and unloading a library is - a priori - an expensive operation. I desribed a scenario where only opened libraries are queried, presumably expensive enough.
But if you really want, you *can* do the above for each foreign funcall. Some foreign libraries do that already, internally (e.g. allocate resource that some library provides, do stuff, then release library).
Luis wrote:
look at *PACKAGE* to figure out which DLL to use. That might be a silly idea.
The idea of associating a package with a library seems IMHO silly indeed, or rather short-sighted and limiting. It does not take into consideration scenarios where libraries are used dynamically. + Not all definitions are literally written in a source file by a programmer. + Some libraries tend to hide the precise location of the symbol (e.g. GL or GLU or GLext or ...). If they want to hide it, don't introduce different packages in Lisp to unhide that. Also, there's no guarantee that a symbol's home package is where you think it is -- it could come from another package. E.g. AFAIK there's not guarantee that all ~1000 ANSI-CL symbols have their home package in (find-package "COMMON-LISP"), but that's another topic.
Also, I very much came to appreciate declarative style and referential transparency. Global variables like *package* just make simple things work, but complex run-times awful or unreliable. Not what I strive for.
Proceed to an experiment: Set *read-base* and *print-base* to something other than 10, then run your favourite applications. Do you expect failure?
Regards, Jörg Höhle.
On 12/20/06, Hoehle, Joerg-Cyril Joerg-Cyril.Hoehle@t-systems.com wrote:
Jack Unrue wrote:
...which is essentially what Jörg suggested, I'm just saying maybe some macrology could hide the details.
I was not that detailed. Quite to the contrary, I had in mind a system which would locate the address once, (e.g. when evaluating def-call-out), not each time the function is called (irk!).
Sorry for not better separating what you suggested and my performance-destroying idea (irk indeed :-)
Loading and unloading a library is - a priori - an expensive operation. I desribed a scenario where only opened libraries are queried, presumably expensive enough.
True, although certain libraries such as shell32 and comctl32 are likely already loaded by the implementation (I've checked LW and CLISP so far), so the additional load/unload operations on top of that are just reference count manipulation (I think, but if you have better info please correct me).
But if you really want, you *can* do the above for each foreign funcall. Some foreign libraries do that already, internally (e.g. allocate resource that some library provides, do stuff, then release library).
That's right.
-
Attila Lendvai -
Gregory Martin Pfeil -
Hoehle, Joerg-Cyril -
Jack Unrue -
James Bielman -
Luís Oliveira -
Nikodemus Siivola -
Robert P. Goldman