September 2016 - cffi-devel - mailman3.common-lisp.net

Would love some feedback on wip feature
by Chris Bagley 06 Oct '16

06 Oct '16

Hi folks, I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction. The changes are very simple, we allow users to specify a cache directory by using an optional :cache-dir argument in :grovel-file & :wrapper-file forms. CFFI will then look there first for the required files before trying to compile them itself. The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match. To handle this I am currently using my with-cached-reader-conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used. With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users. For example I took osicat and added a cache directory to the following lines: (:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo") The "foo" directory has the following contents bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-file All the system does is copy these files to the same directory the system would have put the compiled results in the standard system. My next addition is going to be a function the user can call that will do the following: - Work out if cffi-grovel has cached files for the grovel and wrapper results - If it didnt have those files it will copy the cachable files to a user specified location (or maybe just the cache directory) The idea of this is that it will allow the users to easily dontate their cache files to the project which hopefully means we can better coverage quickly Thoughts would be very welcome, I hope this finds you all well, Baggers [0] https://github.com/cbaggers/cffi/tree/feature-grovel-caching [1] https://github.com/cbaggers/with-cached-reader-conditionals

5 9

Groveller feature proposal: include-local
by Chris Bagley 27 Sep '16

27 Sep '16

Aim: Add include-local to the groveller. Behaviour: - uses #include "name.h" rather than #include <name.h> - expects the header file to be local to the wrapper/groveler specification file. This file is copied to the cache dir (with the generated c file) so it can be used in the build. Reason: This allows us to wrap single header libraries simply by making a wrapper spec containing > (include-local "some_header_lib.h") I'd love to hear some thoughts on this. I'm happy to get this done if it's approved. Regards Chris

3 5

need help debugging wierd dlopen failure
by Jim Newton 21 Sep '16

21 Sep '16

If someone could give me some clues about how to debug this it would be great. I'm trying to figure out why "it" fails to load a particular dylib under macOS. "it" meaning that I'm not sure who is actually failing, but I suspect it is actually a failure detected and reported by dlopen(3) itself, rather than something wrong in cffi. The problem seems to be that loading libgdk-x11-2.0.0.dylib triggers a complaint about a library I have loaded from a different place, but the error message indicates a location I have not loaded my LD_LIBRARY_PATH. Could it possibly be that libgdk-x11-2.0.0.dylib has a hard coded path in it pointing to /usr/local/lib ? If someone could give me some clues about how to debug this it would be great. I've set up my LD_LIBRARY_PATH so that it seems to be loading the correct libcairo.2.dylib from /usr/local/Cellar/cairo/1.12.16_1/lib/libcairo.2.dylib. The reason I think this is because both vmmap (unix command line) and cffi::list-foreign-libraries (lisp) give compatible results. sh> vmmap -w 16787 | grep dylib | sed -e 's/^.*SM=... //' | grep cairo /usr/local/Cellar/cairo/1.14.6_1/lib/libcairo.2.dylib sh> and CL-USER> (cffi::list-foreign-libraries) (... #<CFFI:FOREIGN-LIBRARY :LIBCAIRO #P"/usr/local/Cellar/cairo/1.12.16_1/lib/libcairo.dylib" (truename=#P"/usr/local/Cellar/cairo/1.12.16_1/lib/libcairo.2.dylib")>) CL-USER> However, when I attempt to load libgdk-x11-2.0.0.dylib it complains that it cannot find a particular symbol in /usr/local/lib/libcairo.dylib. Why is it complaining about /usr/local/lib/libcairo.dylib? Unable to load any of the alternatives: ("libgdk-x11-2.0.0.dylib" "libgdk-x11-2.0.dylib") Attempting to load GDK::GDK from lib "libgdk-x11-2.0.dylib" (%LOAD-FOREIGN-LIBRARY GDK /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib) Unable to load foreign library (GDK::GDK). Error opening shared object "/usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib": dlopen(/usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib, 10): Symbol not found: _cairo_xlib_surface_create Referenced from: /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib Expected in: /usr/local/lib/libcairo.dylib in /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib. Attempting to load GDK::GDK from lib "libgdk-x11-2.0.0.dylib" (%LOAD-FOREIGN-LIBRARY GDK /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib) Unable to load foreign library (GDK::GDK). Error opening shared object "/usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib": dlopen(/usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib, 10): Symbol not found: _cairo_xlib_surface_create Referenced from: /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib Expected in: /usr/local/lib/libcairo.dylib in /usr/local/Cellar/gtk+/2.24.24/lib/libgdk-x11-2.0.0.dylib. [Condition of type CFFI:LOAD-FOREIGN-LIBRARY-ERROR]

3 4

Re: How to load a dylib from a particular place
by Luís Oliveira 19 Sep '16

19 Sep '16

On Thu, Sep 15, 2016 at 4:31 PM, Jim Newton <jimka.issy(a)gmail.com> wrote: > I can try your suggestion, but according to the documentation this won't > work. The documentation says the list is only used if dload(3) fails the > first time. I know that dload(3) succeeds the first time by loading the > wrong version. Oh, I see. Missed that bit of information, sorry. Can you delete the bad version or tweak the define-foreign-library definition to point to the right one? Cheers, -- Luís Oliveira http://kerno.org/~luis/

2 2

How to load a dylib from a particular place
by Jim Newton 15 Sep '16

15 Sep '16

I sent this email earlier, but then realized I was not subscribed. So I have know way of knowing whether it got spam-filtered out. So here it is again. Sorry if this is a repeat question. The program I'm trying to load has several dependencies which I'm still trying to figure out. One problem (I think) is that it loading some of its .dylib libraries from the wrong place. I'd like to load them explicitly to help me debug the problem. What is the correct way to do this. Here's what I've tried (but it is not yet correct). I've added the line to my .sbclrc file. (cffi:load-foreign-library "/usr/local/Cellar/cairo/1. 12.16_1/lib/libcairo.dylib") This seems indeed to load the dylib, but later my system calls (CFFI::LOAD-FOREIGN-LIBRARY-PATH :LIBCAIRO "libcairo.dylib" NIL) which apparently does not realise :LIBCAIRO is already loaded, so it tries to load cairo again but from /usr/local/Cellar/cairo/1.14. 6_1/lib/libcairo.2.dylib Is there a way for me to tell cffi:load-foreign-library that loading 1.12.16_1 loads :LIBCAIRO ? I thought about calling cffi:define-foreign-library to associate :LIBCAIRO with the explicit path "/usr/local/Cellar/cairo/1.12.16_1/lib/libcairo.dylib", but I don't see how to do that. Can someone help? Thanks. Jim

2 1

Re: How to load a dylib from a particular place
by Jim Newton 15 Sep '16

15 Sep '16

When I read the documentation of DEFINE-FOREIGN-LIBRARY at https://common-lisp.net/project/cffi/manual/html_node/define_002dforeign_00… I don't see any way of defining a library and associating a full path. It really seems (and correct me if I'm wrong) that you can only specify paths in the case "if not found in system-global directories." But exactly what that means is not immediately clear. I assume it means that in all cases dload() will be called first, passing the library name, and if that fails, then cffi will attempt to find it elsewhere. What I'd like to do is circumvent dload() from being called because I know it will load the wrong file (wrong version). Am I reading the documentation wrong? Is there really a way to load the dylib from a specified directory and associate it with the lisp symbol for cffi's db? On Thu, Sep 15, 2016 at 11:24 AM, Jim Newton <jimka.issy(a)gmail.com> wrote: > The program I'm trying to load has several dependencies which I'm still > trying to figure out. > One problem (I think) is that it loading some of its .dylib libraries from > the wrong place. > I'd like to load them explicitly to help me debug the problem. > What is the correct way to do this. > > Here's what I've tried (but it is not yet correct). > I've added the line to my .sbclrc file. > > (cffi:load-foreign-library "/usr/local/Cellar/cairo/1. > 12.16_1/lib/libcairo.dylib") > > This seems indeed to load the dylib, but later my system > calls (CFFI::LOAD-FOREIGN-LIBRARY-PATH :LIBCAIRO "libcairo.dylib" NIL) > which apparently does not realise :LIBCAIRO is already loaded, so it > tries to load cairo again but from /usr/local/Cellar/cairo/1.14. > 6_1/lib/libcairo.2.dylib > > Is there a way for me to tell cffi:load-foreign-library that loading > 1.12.16_1 loads :LIBCAIRO ? > I thought about calling cffi:define-foreign-library to associate :LIBCAIRO > with the explicit path "/usr/local/Cellar/cairo/1.12.16_1/lib/libcairo.dylib", > but I don't see how to do that. > > Can someone help? > > Thanks. > Jim >

1 0

Re: Questions about cffi's c2ffi support
by Chris Bagley 12 Sep '16

12 Sep '16

Fantastic! Thanks for all your work on this feature. I love being able to jump to the definitions without have to dance with any macros once I'm there. I expect to be booked up for the next week or two but if I get time I'll look at expanding the grovel-caching branch I'm working on to cache the libffi type stuff. Cheers, Chris On 11 September 2016 at 22:39, Attila Lendvai <attila(a)lendvai.name> wrote: >>> When might I want to use this in place of cl-autowrap? >> >> cl-autowrap only uses the CFFI backends and replaces the surface API >> of CFFI with its own, and while it has interesting features, not >> everyone wants to depart from CFFI. >> >> in contrast to that cffi/c2ffi is a small thing that generates vanilla >> CFFI definitions. > > > this should really go into the doc, but meanwhile here are two > examples that use it: > > https://github.com/attila-lendvai/hu.dwim.sdl > https://github.com/attila-lendvai/hu.dwim.bluez > > -- > • attila lendvai > • PGP: 963F 5D5F 45C7 DFCD 0A39 > -- > “If you suffer it is because of you, if you feel blissful it is > because of you. Nobody else is responsible – only you and you alone. > You are your hell and your heaven too.” > — Osho (1931–1990)

1 0

Questions about cffi's c2ffi support
by Chris Bagley 11 Sep '16

11 Sep '16

Hi folks, I'm interested in the c2ffi portion of cffi and whether I can help. When might I want to use this in place of cl-autowrap? Are there open tasks for this feature? I saw the comments say that it is "not very extensive" what kinds of things are missing? Regards

2 2