February 2012 - cffi-devel - mailman3.common-lisp.net

[cffi-devel] how to treat expected failures in tests
by Anton Vodonosov 14 Mar '12

14 Mar '12

Hello. For my common lisp testing project I aggregate results of a library test suite into single value - ok/fail. I just tested ECL and have the following output from CFFI test suite: 4 out of 228 total tests failed: DEFCFUN.NOOP, CALLBACKS.BFF.1, STRING.ENCODING.UTF-16.BASIC, STRING.ENCODINGS.ALL.BASIC. No unexpected failures. What meaning do you put into the term "expected failure"? Does it mean the library is buggy, but these bugs are known? Or it means that some non-required features are absent, but the library in general OK? I am interested in both short answer - as a library author, how do you think CFFI test suite should be marked if only expected failures present - OK or FAIL? And also I am curious in this concrete example, what these 4 failures mean for CFFI on ECL? Best regards, - Anton

10 25

[cffi-devel] Patch for ABCL to dynamically load CFFI JNA requirement
by Mark Evenson 12 Mar '12

12 Mar '12

Please consider the referenced patch for inclusion in cffi-0.10.7. Comments towards improvments solicited. Improvements to ABCL CFFI implementation: o autoload the 'jna.jar' dependency if possible (requires abcl-1.1.0-dev) o initial implementation of callbacks Needs ABCL specific tests, but I can load CL+SSL with this patch. [1]: http://slack.net/~evenson/abcl/cffi/abcl-cffi-20120224a.diff Cheers, easyE -- "A screaming comes across the sky. It has happened before, but there is nothing to compare to it now."

2 2

[cffi-devel] Digitool MCL support for CFFI (was Re: [info-mcl] ? cffi)
by Chun Tian (binghe) 10 Mar '12

10 Mar '12

Hi, CFFI developers Digitool MCL was a old&famous Common Lisp implementation on PPC Mac OS. Recently it was ported into Intel Mac OS X by running in Rosetta (a PPC emulator by Apple). So there're people still using this platform even when Clozure CL is a better choice. Attach is two patches, one is a CFFI MCL support patch made by James Anderson, the other is a trivial patch for MCL support of trivial-features made by me. With these two patch, CFFI can now be loaded into Digitool MCL. This work seems work with James' own project which is based on CFFI. But CFFI-TESTS cannot run at this time, because ASDF:RUN-SHELL-COMMAND doesn't support MCL (I don't know how to fix this right now). IOlib also cannot work in MCL because Gray Streams wasn't supported by MCL yet ... However, I hope CFFI maintainer could merge this patch and let other MCL users help improving exist work. Chun Tian (binghe) 在 2010-9-28，06:41， james anderson 写道： > good morning; > > i have worked out enough of a cffi-mcl to be able to accomplish the immediate goal - to use the cl-zmq zeromq ffi binding.[1] > in case someone wants to carry it further, it's posted along with a mcl-compatible zmq version,[2] which eliminates un-ported dependencies. > > --- > [1] : http://repo.or.cz/w/cl-zmq.git > [2] : http://github.com/lisp/de.setf.zeromq > > On 2010-09-22, at 08:29 , Chun Tian (binghe) wrote: > >> I've read these documents. MCL 4.0's FFI seems based on the function "define-entry-point" which then depends on "ppc-ff-call", both of them are still available in MCL 5.x, and the related MCL source code is "lib/deftrap.lisp", which was last modified in 2005. >> >> However, I think "define-entry-point" is not the preferred way any more: >> >> ; Change log >> ; >> ; put back define-entry-point and things it depends on - though MCL doesn't use it users may >> ; ------ 5.2b2 >> >> I think CFFI people, if want to support MCL, should research "deftrap.lisp" and figure out the preferred way to do the work. But for me, a USOCKET maintainer, I'd like to use this interface directly to implement UDP networking, which was lack from OpenTransport. >> >> --binghe >> >> 在 2010-9-22，13:30， james anderson 写道： >> >>> there are two sources. >>> >>> if you go to ftp.clozure.com, you will find the 5.2 release and a copy of the documentation for 4.0. >>> i know of no revised documentation for 5.2. other than that, there is always the source. >>> i believe 5.2 has a very different calling mechanism then 4.0, as it uses a different os x facility to manage foreign libraries. >>> >>> the biggest '?' is turning off the gc for the dynamic extent of any foreign calls which pass lisp data. i do not know if that is possible in mcl, but it could be faked by moving the data to/from temporary external storage just to get it working. >>> two little question marks are the data type size calculations and the call-back protocol. >>> since the source is there, there's nothing stopping me from adapting the clozure-specific code to work in mcl. >>> i was just hoping to avoid having to wrap my head around those bits. >>> >>> >>> On 2010-09-22, at 04:45 , Chun Tian (binghe) wrote: >>> >>>> Never know how to do FFI in Digitool MCL ... Where can I find some API documents? >>>> >>>> --binghe >>>> >>>> 在 2010-9-22，05:36， james anderson 写道： >>>> >>>>> good morning; >>>>> >>>>> has anyone ported cffi to mcl? >>>>> >>>>> >>>>> _______________________________________________ >>>>> info-mcl mailing list >>>>> info-mcl(a)clozure.com >>>>> http://clozure.com/mailman/listinfo/info-mcl >>>> >>>> _______________________________________________ >>>> info-mcl mailing list >>>> info-mcl(a)clozure.com >>>> http://clozure.com/mailman/listinfo/info-mcl >>> >>> _______________________________________________ >>> info-mcl mailing list >>> info-mcl(a)clozure.com >>> http://clozure.com/mailman/listinfo/info-mcl >> >> _______________________________________________ >> info-mcl mailing list >> info-mcl(a)clozure.com >> http://clozure.com/mailman/listinfo/info-mcl > > _______________________________________________ > info-mcl mailing list > info-mcl(a)clozure.com > http://clozure.com/mailman/listinfo/info-mcl

2 3

Re: [cffi-devel] [cffi] FSBV (#2)
by Liam Healy 27 Feb '12

27 Feb '12

Ryan - I am forwarding this to CFFI-devel and Martin directly (though I'm pretty sure he's subscribed to cffi-devel). Can you please send future messages to cffi-devel? I'm not sure many people read the github mailing list; this issue should get wider visibility. Thanks. Liam On Mon, Feb 20, 2012 at 12:02 PM, Ryan Pavlik < reply+i-1614209-ba246666762196459413560690eb7d3a39c7c7ee-838019(a)reply.github.com > wrote: > I'm not sure what you mean. It doesn't really matter that a pointer is > aggregate or not. The goal is to get at the Nth member of an > array-of-something. In the case of scalar C types, you're getting the > value; in the case of structs it's far more useful to get a pointer, > because you probably only want a single value out of the struct, or to > **put a value into the struct**. (The `setf` form works for scalars in the > latter case, but not for "member-of-struct-of-index-N"... you most likely > want the pointer in these cases.) > > It also occurred to me after posting that there is no difference between > `(mem-aref ptr '(:pointer (:struct foo)) N)` and just simply `(mem-aref ptr > :pointer N)` ... both return a pointer value as if `ptr` is a `void > *ptr[k]`. > > A struct of more bytes works the same way: > > ``` > (cffi:defcstruct my-struct > (x :long) > (y :long) > (x :long) > (t :long)) > > ... > > Old-ref style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFC7FB8) > aref: #.(SB-SYS:INT-SAP #X7FFFEEFC7FD8) > New-ref style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFC7FB8) > aref: (T 0 Y 0 X 0) > New-ref with :pointer style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFC7FB8) > aref: #.(SB-SYS:INT-SAP #X00000000) > ``` > > Note that on my system, a pointer is 8 bytes, not 4. This is why I > initially found the problem, when trying to access an array of points > defined by 2 short; each member is 4 bytes, and it was giving offsets to > `sizeof(void*)`. > > --- > Reply to this email directly or view it on GitHub: > https://github.com/cffi/cffi/pull/2#issuecomment-4057418 >

3 2

Re: [cffi-devel] [cffi] FSBV (#2)
by Liam Healy 20 Feb '12

20 Feb '12

The new syntax is as follows. If you want the structure itself, use (:struct foo). If you want a pointer to it, use (:pointer (:struct foo)). The latter should be identical to the old style bare struct specification, except for the annoying deprecation warning, of course. The issues I can identify, with their resolutions: 1) Using mem-aref with the (:pointer (:struct ...)) spec gives the wrong pointer. I have fixed an error which should now return the correct pointer for an offset of 0. For an offset of 1, it returns the base pointer +8 bytes, which is not what the old style gives (+ 4 bytes), but it seems to me correct, as I understand the index to refer to the number of whole structures. Pull ee90bfd517 and try. 2) The plist form representing the structure is not desirable. You can have any CL representation of the structure you like; you need to define a method cffi:translate-from-foreign for your type class. You are getting the default, a plist translation, because no such method is defined. See for example how I translate complex numbers<http://repo.or.cz/w/antik.git/blob/1ee407c69525b84b441f8cf7b48ac590e78bd635…>to CL complex numbers. You can even return a pointer if you want, but this probably isn't the specification to use if you want the pointer. On Mon, Feb 13, 2012 at 10:22 AM, Ryan Pavlik < reply+i-1614209-ba246666762196459413560690eb7d3a39c7c7ee-838019(a)reply.github.com > wrote: > I've pulled the latest and it appears the semantics have changed for > mem-aref, but there is still no way to get the old behavior. Here is a > complete example, though it doesn't use the test system definitions because > actual foreign calls and definitions aren't really the problem: > > ``` > (asdf:load-system :cffi) > > (cffi:defcstruct my-struct > (x :short) > (y :short)) > > (defun test-old-ref () > (declare (notinline cffi:mem-ref cffi:mem-aref)) > (cffi:with-foreign-object (ptr '(:struct my-struct) 2) > (format t "~&Old-ref style:~%ptr : ~A~%aref: ~A~%" > ptr (cffi:mem-aref ptr 'my-struct 1)))) > > (defun test-new-ref () > (cffi:with-foreign-object (ptr '(:struct my-struct) 2) > (format t "~&New-ref style:~%ptr : ~A~%aref: ~A~%" > ptr > (cffi:mem-aref ptr '(:struct my-struct) 1)))) > > (defun test-new-ptr-ref () > (cffi:with-foreign-object (ptr '(:struct my-struct) 2) > (format t "~&New-ref with :pointer style:~%ptr : ~A~%aref: ~A~%" > ptr > (cffi:mem-aref ptr '(:pointer (:struct my-struct)) 1)))) > > (progn > (test-old-ref) > (test-new-ref) > (test-new-ptr-ref)) > ``` > > The output I get, sans style-warnings about bare structs: > > ``` > Old-ref style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFCFFF0) > aref: #.(SB-SYS:INT-SAP #X7FFFEEFCFFF4) > New-ref style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFCFFF0) > aref: (Y 0 X 0) > New-ref with :pointer style: > ptr : #.(SB-SYS:INT-SAP #X7FFFEEFCFFF0) > aref: #.(SB-SYS:INT-SAP #X00000000) > ``` > > Note that in the first example, with the original semantics, if you > mem-aref a pointer to an array of `my-struct`, you get a pointer to the > array element. In the new style, with `(:struct my-struct)`, you get the > values parsed into a list, which is not particularly useful; it conses, and > you almost certainly have to re-parse a possibly long list for a single > element. In the new style with `:pointer`, it appears to dereference the > Nth element in an *array of pointers to my-struct*, which is not at all > what we want. > > The latter differs from the behavior before I updated, which seemed to > return a *pointer* to the Nth element in an array-of-pointers. None of the > above are like the old behavior. > > --- > Reply to this email directly or view it on GitHub: > https://github.com/cffi/cffi/pull/2#issuecomment-3941718 >

2 2

[cffi-devel] pathnames and libraries
by JTK 16 Feb '12

16 Feb '12

When trying to install GSLL (the gnu scientific library lisp package) using quicklisp, I noticed that it failed because the library name was a pathname rather than a string. I fixed it by changing cffi's src/libraries.lisp as described below, putting pathnames into the etypecase. Is there some reason why this isn't done normally? John ;; from src/libraries.lisp (defun load-foreign-library-helper (name thing &optional search-path) (etypecase thing (string (load-foreign-library-path name thing search-path)) (pathname ;; #### MAKE IT HANDLE PATHNAMES (load-foreign-library-path name (namestring thing) search-path)) ;; JTK (cons (ecase (first thing) (:framework (load-darwin-framework name (second thing))) (:default (unless (stringp (second thing)) (fl-error "Argument to :DEFAULT must be a string.")) (let ((library-path (concatenate 'string (second thing) (default-library-suffix)))) (load-foreign-library-path name library-path search-path))) (:or (try-foreign-library-alternatives name (rest thing)))))))

2 1

Re: [cffi-devel] [cffi] FSBV (#2)
by Liam Healy 12 Feb '12

12 Feb '12

I have updated the example in tests/struct.lisp, (defcfun "alloc_pair" (:pointer (:struct struct-pair)) (a :int) (b :int)) and when I call alloc-pair, I get a pointer back: (alloc-pair 1 2) #.(SB-SYS:INT-SAP #X00668E70) I am not sure how you are seeing the problems you describe. Can you please send complete example(s) and describe what you see and what you expect? If you use the definitions already in the test system, that will make it a little bit easier. Thank you. On Wed, Feb 8, 2012 at 2:28 PM, Ryan Pavlik < reply+i-1614209-ba246666762196459413560690eb7d3a39c7c7ee-838019(a)reply.github.com > wrote: > I've been using this, and while it works great for by-value, there no > longer seems to be a way to iterate struct arrays and get pointers back. > This is bad, because I _don't_ always want a struct parsed. Parsing is > slow and conses a lot. > > However, neither `(mem-aref ptr (:pointer (:struct foo)) n)` nor > `(mem-aref ptr (:struct foo) n)` give the same thing as the function > `(mem-aref ptr 'foo n)` does. Additionally, the compiler-macro form sets > `*PARSE-BARE-STRUCTS-AS-POINTERS*`, so this results in incorrect iteration > as well. I know at least a few existing codebases depend on the old > behavior. > > For instance, I have a cstruct `POINT` which consists of two `:short`. > The `foreign-type-size` is 4. However, with the compiler macro: > > ``` > (with-foreign-object (ptr 'point 2) > (mem-aref ptr 'point 1)) ;; -> ptr+8 > ``` > > This is identical to using `(mem-aref ptr '(:pointer (:struct point)) 1)`, > and wrong. If the *function* gets called with the old parsing style, you > get `ptr+4`, which is the only way to do so beyond using `mem-ref` and > `foreign-type-size` manually. This breaks a lot of existing code. > > If I use `(mem-aref ptr '(:struct point) 1)`, I get values returned, not > pointers, which is no good if I need the pointer itself. > > --- > Reply to this email directly or view it on GitHub: > https://github.com/cffi/cffi/pull/2#issuecomment-3874222 >

1 0

Re: [cffi-devel] First stab at reviewing cffi-fsbv
by Liam Healy 11 Feb '12

11 Feb '12

Hi Luis, On Sun, Dec 11, 2011 at 8:36 PM, Luís Oliveira <luismbo(a)gmail.com> wrote: > Hello, > > I went through all the code and here are my review notes. (In org-mode > format.) I suggest we tackle them incrementally, not necessarily in > this order. Meanwhile, I'll be tackling the missing bits in the new > convert-into-foreign-memory. Should we be having this discussion in > cffi-devel btw? > Sure. I've copied the list for anyone who's interested. > * cffi-fsbv.asd > What happens on #-unix? > (:cffi-grovel-file "libffi" :pathname #+unix "libffi-unix") > Absolutely nothing. Someone named "CRLF0710" sent me an email claiming to have a libffi-win32 for the old standalone version of FSBV, and I wrote back saying I was interested but I've heard nothing. > * fsbv/package.lisp > 1. Regarding SIZET, I'd say we can define this type using a > keyword :SIZET. > Yes, if you think that's OK. That was me totally sneaking something in that GSLL needed that was awkward to generate there but that CFFI could generate easily. > 2. cffi-fsbv calls and hooks into so many of CFFI internals. Should > we just use the CFFI package? I mean, the code is full of ::s. > I'm fine with that. I don't think there'll be any symbol conflicts. > > * fsbv/libffi-unix.lisp > 1. OSX ships with libffi. Is (cc-flags "-I/opt/local/include/") > necessary? (I guess it doesn't hurt. Just curious. Perhaps it > didn't ship with libffi in the past?) > Ugh, OSX. It apparently does ship with libffi, except when it doesn't. There are so many ways to install system software, I just gave up. I'm all for your approach because it does simplify the code and make it match the other OSes; when the complaints come in to cffi-devel, you can answer them :-) > > 2. Aren't :uint and :ushort equivalent to: > (ctype ushort "unsigned short") > (ctype unsigned "unsigned")? > Um, I don't know? What are you advocating here? Removing something that I put in? :uint and :ushort don't ring a bell for me. > > 3. Do we really need to grovel this stuff? Grovelling requires a C > compiler which is particularly tricky on Windows. It'd be nice to > avoid that requirement if possible. Didn't see anything in here > that seemed to *require* grovelling. Did I miss something? > "Here" means libffi-unix? The constants at the end aren't necessary, but I think the rest is. In particular, FFI_DEFAULT_ABI (abi) and FFI_OK (status) are pretty critical, as is the struct ffi-type. I'm a little confused; I thought the point of cffi-grovel was to provide access to definitions in .h files, and that's what we need for ffi.h and ffi-target.h. How do you propose defining these without groveling? Just hardwiring into lisp and hoping no one ever changes those values? By the way I have FFI_STDCALL commented out and didn't try to mate it with CFFI's definition on Windows, perhaps CRLF0710 or another Windows user can figure that out. > * fsbv/build-in-type.lisp > 1. Could we use DEFCVAR here to declare the various ffi_type_* > globals? (I think it'd be slightly clearer.) > I kind of like the automatic way of doing it; manually means that every time something changes (new type added or old one removed) you have to remember to do it in two places. That's a recipe for breakage. > 2. Why do we have to cache pointers for FOREIGN-TYPE-ALIAS and > FOREIGN-ENUM? (But see my comments on fsbv/cstruct.lisp) > We probably don't. > 3. Why different LIBFFI-TYPE-POINTER for those two types? > FOREIGN-ENUM inherits from FOREIGN-TYPE-ALIAS. > Is it really different, or is it just getting the pointer from the underlying type? > * fsbv/cstruct.lisp > 1. slots-in-order should be moved near FOREIGN-STRUCT-TYPE. > 2. I think we should move libffi-type-pointer out of FOREIGN-TYPE > and centralize storage in an hash-table. > a) this simplifies implementation a little bit: it's weird that > pretty much all libffi-type-pointer methods have :AROUND > qualifiers. > b) makes it easy to avoid memory leaks when redefining/reloading > structure types. (Although this sort of leak is probably not a > huge issue, it's certainly inelegant.) > 3. The code that creates ffi_types for structs could use some > refactoring. :-) I'll give it a go. > Sure. > > * fsbv/functions.lisp > 1. I think this file can benefit from a little bit of refactoring. > 2. I *think* we can move the indirection logic from the translation > methods onto FFCALL-BODY-LIBFFI, but it's not yet clear how that > can happen. > Interesting thought, I'd like to hear your ideas. > > * fsbv/example.lisp > 1. Need to clear the dead code and move it to examples/. > examples.lisp? It's probably all dead code now, just get rid of the whole thing. > > * src/functions.lisp > 1. We can add a neat restart to *foreign-structures-by-value*. > Yes I think you mentioned this idea on the todo list. I expect the restart is "load cffi-libffi"? > > * src/structures.lisp > 1. As I've mentioned before, we should discuss how we can implement > this sort of functionality in a backwards-compatible way. > I'd prefer not defaulting to previous assumptions (that is, no translation of structures), if that's what backwards-compatible means. But I'm interested in how you'd approach this. * src/early-types.lisp > 1. missing expand-into-foreign-memory, and all the hooking up that > implies. Started working on that. > 2. I think the :indirect keyword is delegating the responsibility to > the wrong place. All the type translations (in CFFI or in other > libraries) would have to be updated accordingly! As mentioned in > the fsbv/functions.lisp notes, I think we can handle the > indirection there. (It might require a slightly different way of > hooking up ffcall-body-libffi with foreign-funcall.) > No doubt. I would like to avoid special-casing though, e.g. "here's an enum, got to indirect it". I think that kind of thing belongs in a method associated with the type. > * Nomenclature > I can forsee using libffi to deal with stuff like callbacks, long > long, varargs, etc. Should we rename cffi-fsbv to cffi-libffi? > Sure. Also, you mentioned that some implementations have their own call-by-value mechanism, so it makes more sense to rename it, since they would have "fsbv" but no libffi. > > * Wishlist > *** foreign-funcall-varargs and friends could use libffi > ... to do proper varargs. > *** defcallbacks with structures... > Yup. Liam > > -- > Luís Oliveira > http://r42.eu/~luis/ >

1 1

[cffi-devel] ELS 2012, Zadar, Croatia
by Marco Antoniotti 01 Feb '12

01 Feb '12

Apologies for the multiple postings. PAPER SUBMISSION DEADLINE EXTENDED European Lisp Symposium 2012, Zadar, Croatia, April 30th - May 1st, 2012 http://european-lisp-symposium.org The purpose of the European Lisp Symposium is to provide a forum for the discussion and dissemination of all aspects of design, implementation and application of any of the Lisp and Lisp-inspired dialects, including Common Lisp, Scheme, Emacs Lisp, AutoLisp, ISLISP, Dylan, Clojure, ACL2, ECMAScript, Racket, SKILL, and so on. We encourage everyone interested in Lisp to participate. The main theme of the 2012 European Lisp Conference is "Interoperability: Systems, Libraries, Workflows". Lisp based and functional-languages based systems have grown a variety of solutions to become more and more integrated with the wider world of Information and Communication Technologies in current use. There are several dimensions to the scope of the solutions proposed, ranging from "embedding" of interpreters in C-based systems, to the development of abstractions levels that facilitate the expression of complex context dependent tasks, to the construction of exchange formats handling libraries, to the construction of theorem-provers for the "Semantic Web". The European Lisp Symposium 2012 solicits the submission of papers with this specific theme in mind, alongside the more traditional tracks which have appeared in the past editions. We invite submissions in the following forms: Papers: Technical papers of up to 15 pages that describe original results or explain known ideas in new and elegant ways. Demonstrations: Abstracts of up to 4 pages for demonstrations of tools, libraries, and applications. Tutorials: Abstracts of up to 4 pages for in-depth presentations about topics of special interest for at least 90 minutes and up to 180 minutes. Lightning talks: Abstracts of up to one page for talks to last for no more than 5 minutes. All submissions should be formatted following the ACM SIGS guidelines and include ACM classification categories and terms. For more information on the submission guidelines and the ACM keywords, see: http://www.acm.org/sigs/publications/proceedings-templates and http://www.acm.org/about/class/1998. Important dates: February 15th 2012: submission deadline (extended deadline) March 7th 2012: acceptance results April 30th 2012: Conference opens Program Commitee. Chair: Marco Antoniotti, Università degli Studi di Milano Bicocca, Milan, ITALY Local organizers: Damir Cavar, Eastern Michigan University Franjo Pehar, University of Zadar Damir Kero, University of Zadar Members: Giuseppe Attardi, Università degli Studi di Pisa, Pisa, ITALY Pascal Costanza, Intel, Bruxelles, BELGIUM Marc Feeley, Université de Montreal, Montreal, CANADA Scott McKay, Google, U.S.A. Kent Pitman, U.S.A. Christophe Rhodes, Department of Computing, Goldsmiths, University of London, London, UNITED KINGDOM Robert Strandh, LABRI, Université de Bordeaux, Bordaux, FRANCE Didier Verna, EPITA / LRDE, FRANCE Taiichi Yuasa, Kyoto University, JAPAN -- Marco Antoniotti

1 0