- cffi-devel - mailman3.common-lisp.net

[cffi-devel] Type Translator Branch Update
by James Bielman 03 Jan '06

03 Jan '06

Hi all, I've updated the cffi-new-translators branch, improving the exported interface and fixing a few bugs. The branch is available via Darcs (or a browser to browse the source) at: http://slacknet.com/~jamesjb/cffi-new-translators The exported interface for defining type translators is by specializing the following generic functions: Generic Function: TRANSLATE-TO-FOREIGN value type-name Convert the Lisp object VALUE to a foreign object of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE. Methods on this generic function should EQL-specialize TYPE-NAME for the type to translate. Returns the translated foreign object and an optional second value that will be passed as the PARAM argument to FREE-TRANSLATED-OBJECT. Generic Function: TRANSLATE-FROM-FOREIGN value type-name Convert the foreign object VALUE of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE, to a Lisp object. Methods on this generic function should EQL-specialize TYPE-NAME for the type to translate. Generic Function: FREE-TRANSLATED-OBJECT value type-name param Perform necessary deallocation of the foreign object VALUE, of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE. Methods on this generic function should EQL-specialize TYPE-NAME for the type to deallocate. PARAM will contain the second value returned by TRANSLATE-TO-FOREIGN for this type, or NIL if no second value was present. If no deallocation is required, it is not necessary to specialize this method. Here's an example, a simplified version of the :boolean type: (defctype boolean :int) (defmethod translate-to-foreign (value (type (eql 'boolean))) (if value 1 0)) (defmethod translate-from-foreign (value (type (eql 'boolean))) (not (zerop value))) Also, look at examples/gettimeofday.lisp and translator-test.lisp for more examples. There is another, unexported, set of generic functions used internally for translating whole classes of types. Look at TRANSLATE-TYPE-{TO,FROM}-FOREIGN and FREE-TYPE-TRANSLATED-OBJECT for these (and methods defined on them for FOREIGN-ENUM, etc). With respect to the old :TO-C-DYNAMIC translators, I'm inclined to not worry about optimizing this case until there is a specific performance need for it. I think the ability to dispatch on the type of the value being translated is more useful than the optimization it was providing. Comments? James

1 1

Re: [cffi-devel] a thought on string encodings
by Hoehle, Joerg-Cyril 03 Jan '06

03 Jan '06

James Bielman wrote: >I'm working on (to begin with), a UTF8-STRING type >which converts Lisp strings to/from UTF-8 on Unicode Lisps. >(Does the CLISP FFI provide something like memcpy?) No. Some voice in me says I should rather implement the vector<->memory functions for CLISP instead of playing around with cffi, Iterate, reading interesting papers etc. when I have a little time. >So, I think I need that block interface we've talked about. Reviewing the proposal has been on my TODO list for a long time as well, I'm sorry. > (ffi:with-foreign-string (ptr chars bytes s :encoding charset:utf-8) > (let ((buf (foreign-alloc :unsigned-char :count bytes))) > (memcpy buf ptr bytes) I think I'd rather use ext:convert-string-to-bytes, then use my non-existent vector->memory function. Sometimes I feel uneasy about stack-allocating possibly huge strings (e.g. 1MB or more!). In the meantime, (let* ((bytes (ext:convert ...)) (len (length bytes)) (buf foreign-alloc len)) (setf (memory-as buf (parse-c-type `(c-array uint8 ,len))) bytes) should be among the fastest ones (CLISP can copy the (c-array uint8 N) type fast). And people keep saying the generational GC should be able to free the garbage vector easily. >However, I haven't been able to find an inverse for >FFI:WITH-FOREIGN-STRING. Indeed, I should ... (see above) and implement string-from-foreign and (foreign-string-length :encoding) as a means to interface to mblen(). >I'd like to be able to convert a pointer back >to a Lisp string without looping in bytecode to create a vector of >octets from the pointer. I suggest the converse of the above, via an array of (unsigned-byte 8). > I tried a >whole bunch of combinations of FFI:MEMORY-AS with FFI:C-ARRAY-PTR types >and got nothing but segfaults. Please report a bug, but possibly you just did not re-read recent impnotes closely enough. For instance, while the FFI now accepts non 1:1 encodings, it's use with c-array-ptr and c-array[-max] is mostly broken and some parts revert to a 1:1 encoding (see *foreign-8bit-encoding*). I would have left the 1:1 restriction and take more time to think about the problems. :-( The operators that explicitly take an :encoding are safe, like with-foreign-string. Also, custom:*foreign-encoding* is a symbol macro. Thus (let ((custom:*foreign-encoding* charset:foo))) won't work as expected, you need setq and unwind-protect. :-( > Is there something I can use to convert >the pointer to either a vector of octets (which I can pass to >EXT:CONVERT-STRING-FROM-BYTES, or to a Lisp string directly? (memory-as pointer (parse-c-type `(c-array uint8 ,len))) Looking at the clisp sources, (c-array character N) -> Lisp string seems ok as well. But e.g. don't use (c-array-max character N) with UTF-16! As you can see, there's room for improvement. Summary: ptr -> Lisp string: either ext:convert + memory-as uint8 or (let ((old *foreign-encoding*)) (unwind-protect (progn (setq *foreign-encoding* utf-8) (memory-as pointer (parse-c-type `(c-array character ,known-length))) )(setq *foreign-encoding* old))) ; looks scary :-( Do you need unknown-length as well? Lisp string -> ptr: ext:convert + memory-as uint8 as mentioned above. Regards, Jörg Höhle.

3 2

[cffi-devel] GtkMozEmbed: fixed
by Keith Irwin 03 Jan '06

03 Jan '06

Folks-- For some reason I'm not getting the emails I send, even though I've properly set the bit in the list software. Ah, well. Jamesb on #lisp solved the problem. Apparently, you need to set the profile path for some sites, or the embedded gecko widget tanks. The profile path is the place the browser saves all the cookie/login information. I've updated the code I linked to earlier in case anyone wants to play with it. Personally, I think CFFI made this all pretty easy, though kudos to the mozilla team for the library in the first place. Thankfully, this really wasn't a CFFI problem. BTW, using glade made GTK pretty painless. I just had to define the procedures I needed. I can see how, over time, a GTK interface (CLOS based) might get implemented, but with libglade, I almost think, why bother? Now, if I could just integrate slime/swank into GtkSourceView.... Keith

1 0

[cffi-devel] gtkembedmoz and https segfault: cffi related?
by Keith Irwin 02 Jan '06

02 Jan '06

Folks-- I've written a little toy app that embeds mozilla as a widget, allows for some minimal back/forward and location bar functionality. Using CFFI, of course. It works really well, except I get a segfault (I think) when I click or type in any link which leads to (or incorporates part of) an HTTPS site. Might this in any way be an CFFI problem? You can download the code at: http://zentrope.com/code/cl-browser.tgz Runs on ubuntu (as long as you've got libglade installed, see the README). If you load the app (browser:main) at the repl, then click the "sign-in" link on the google page, you'll get a segfault. I get the following stack trace when I do that: 0: (SB-KERNEL::MEMORY-FAULT-ERROR) 1: ("foreign function: call_into_lisp") 2: ("foreign function: post_signal_tramp") 3: ("foreign function: NSSRWLock_LockRead") 4: ("foreign function: SECMOD_GetReadLock") 5: ("foreign function: PK11_TokenExists") 6: ("foreign function: #xB1E6388C") 7: ("foreign function: #xB1E6EDFC") 8: ("foreign function: #xB1E71F4A") 9: ("foreign function: #xB1E74340") 10: ("foreign function: #xB1E755F2") 11: ("foreign function: #xB1E75684") 12: ("foreign function: #xB1E7898D") 13: ("foreign function: #xB1D5070E") 14: ("foreign function: PR_Write") 15: ("foreign function: #xB6689D83") 16: ("foreign function: #x8578028") 17: ("foreign function: #x8578040") If this is a CFFI problem, then, cool, I'm reporting it to the right place. If not, perhaps you know what it might be? Surely not a mozilla prob. Perhaps SBCL? Keith

1 0

Re: [cffi-devel] Providing access to errno in CFFI-SYS
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

James Bielman wrote: >> See clisp/modules/bindings/glibc/linux.lisp >I'm not an expert CLISP user---is this not built as a part of the >standard image? What's standard? ./configure --build? - as found on sourceforge file download? - as found in Debian? I don't know the choices myself, but clearly, that file is only available on systems with glibc (I've heard mention of people trying and compiling this file on *BSD). Furthermore, it's possibly not in the default build. For sure, it was not in the default build of old CLISP and only there with clisp --full, but it seems Sam Steingold recently changed defaults and it may be there in a default build and invocation of clisp on Linux. So indeed, the linux package is not in every clisp. clisp --version will list the C Modules. My typical build contains none of them, while Ubuntu/Debian clisp-2.35 contains i18n, syscalls, regexp, linux, clx. So for now you could perhaps try something along: (ignore-errors (or (when (find-package "LINUX") (read-from-string "linux:errno")) (def-call-out errno-location (:name "__errno_location")(:return-type (c-pointer int))(:library :default)(:language :stdc)) (def-c-var errno (:type ffi:int)(:library :default)) ; works on Linux!??! )) ... Or just use the last one with def-c-var errno, since the rest is overengineered for the day when clisp might support threads. Don't hold your breath. (Actually, I'm surprised that dlsym() found "errno" on my Linux box. I'd thought __errno_location() rules in glibc/i386. Go figure). Regards, Jörg Höhle.

1 0

[cffi-devel] Re: CFFI/CLISP - 2 out of 101 total tests failed
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

Luís Oliveira wrote: >About the FUNCALL.VARARGS.DOUBLE test, it passes on Linux and OSX but >fails on Win32. Probably a CLISP bug, Joerg? :-) Maybe, maybe not. My opinion is that interfacing to varargs is hairy and I hear mention of problems with them, when coming from any language (Python, perl, Haskell, .NET, xyz). I believe the tests are (potentially) incorrect. As I was quoting in "[cffi-devel] bits from a new tutorial" on 2005-12-23, http://gcc.gnu.org/ml/gcc/2004-06/msg01019.html "It's assuming that the calling conventions for a varargs function and a non-varargs function are the same, and that's not necessarily true on all platforms." I believe this could explain the observed behaviour. The test calls sprintf() as if it were declared void sprintf(void*,char*,double); This could cause the value to be passed directly in a floating point register, which would not surprise me. Whilst varargs are all passed on the stack. So far, I've no reason to believe there's a bug in CLISP, but try to convince me of the contrary. I believe interfacing to varargs is a problem and needs careful thought. In the meantime, I suggest to recommend not to use them. Individual tests may remain in testsuite so we are remembered of the problem. My current thought is that I'm going to suggest to the maintainer of the ffcall package (which happens to be Bruno Haible, CC'ed) to provide some mean to interface to such functions(*), or restrict it, warn about their use, or try to narrow down the problematic area to well documented cases. (*) would mean that the top-level description also would need to account for that, i.e. ffi:def-call-out or cffi:defcfun need means to describe them. It's possibly hairy. Regards, Jörg Höhle.

1 0

[cffi-devel] Re: CFFI/CLISP issues with saved images
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

Luís Oliveira wrote: >I was just testing CFFI/CLISP by loading the testsuite, dumping an >image, loading it and running the tests. CLISP crashes. There are two >problems: > >- The foreign library (in this case libtest.so) is not being >reloaded. [...]I recall CLISP's ffi:def-call-out taking care of this but Hmm, did you try an absolute path? I suspect the library was not found. I'd naively have expected this to work in the context of cffi as well as with direct uses of the FFI. >- The callbacks become NULL pointers, thus all callback.* tests fail. >Any suggestions for fixing that? Randomly placing load-time-value >forms in %defcallback didn't seem to work. :-) [replying without having thought much about this issue]. CLISP's image is not like an Emacs dump of memory. It doesn't preserve addresses. It doesn't preserve malloc()'ed stuff, which I believe callbacks are (or parts thereof). Actually, I would *not* expect any implementation to be able to reproduce the exact same address of anything when started from an image. CMUCL seems an exception, where there are additional constraints (like on Solaris, getting the linker to load a shared object file at a known location). Similarly, any foreign library that stored a callback-pointer *must* *not* expect such a pointer to be valid when restarting. That library may itself be at a different address when reloaded, how could the callback not change? IMHO, it might work in a scenario on Linux with naive tests where you exit and immediately restart, but what if you upgrade your system in between etc.? Instead, my mental model is that callbacks must be re-established dynamically when an application starts. -- which makes me think that the code for free'ing an old callback upon redefinition in cffi should add a ext:validp test before free'ing. I just added a check in my branch. That's what I meant in my mail "unnamed callback" from 2005-12-19, where I wrote: >(defun my-application > (let (x y) > ...code... > (register-callback 'close-window I'd be happy to discuss this further. Restarting from one's images is being neglected in the Lisp community these days IMHO. Smalltalk implementations seems to behave better. Regards, Jörg Höhle.

1 0

[cffi-devel] foreign-alloc is mostly untested
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

I just found out that foreign-alloc is a complex function (many keyword arguments) and it has no single testcase yet. Regards, Jörg Höhle.

1 0

Re: [cffi-devel] Providing access to errno in CFFI-SYS
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

James Bielman wrote: >While working through my fancy gettimeofday CFFI example, I've added a >type translator used as the return type for standard C functions that >return a negative integer on error and set 'errno'. A very useful type indeed. I'd like to hear about its design. Would a result of -1 raise a Lisp error? [ok, I'm checking the sources myself, "darcs pull" ...] I think you did TRT by throwing an error in that case. Yet possibly some people might object throwing errors out of "low-level" FFI code, as it requires them to be very careful about freeing all resources... May I suggest you defined a CLCS condition, so there's a) a known slot where to get the errno value (instead of pulling it out of a string) b) a known condition and a possibility to hook in with HANDLER-BIND etc.? I believe exporting an interface to errno from CFFI will avoid many headaches to users (writing unportable or otherwise broken code to get its value). Please go ahead and include this stuff in the main tree, not examples/. >(minusp value) I think you should specifically check for -1. Some functions could return a legal value such large that it overflows an int and becomes negative. Actually, I wonder whether the DWIM pattern of all these functions shouldn't have been: "it returns an unsigned int, except for (unsigned-int)(-1) which is taken to mean `error'". E.g. lseek() etc. (although I don't know whether Linux supports files of length >2GB, but lseek is just an example). BTW, I find the name syscall-result misleading. What about int-or-errno? (Or rather, to keep with my own advice: uint-or-errno)? Or, like the boolean generic type, write a (define-foreign-type maybe-errno (&optional (base-type int))? >Since errno is often a macro expanding to a function, I can't very well >define it as a foreign global. It is not, so it would be wrong to do so. errno is a per-thread value on all systems that support threading. In some versions of GLIBC, it's a #define wrapper around a function -- errno_location(). > (defun errno-value () (sb-alien:get-errno)) As soon as errno is there, people will ask for being able to reset it to 0. >haven't found anything in Allegro CL or CLISP (Jörg?) for this yet. See clisp/modules/bindings/glibc/linux.lisp as a source of inspiration & read the comments about errno there. It's hairy indeed. >(I know errno isn't _completely_ portable---at least one OS I use >regularly, Windows CE, doesn't have it at all [...] Interesting to know, thanks! Regards, Jörg Höhle

2 1

Re: [cffi-devel] Branch: New Type Translator Interface
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

James Bielman wrote: >I've done yet-another rewrite of the type translator >interface. It does >everything through generic functions at run-time, which allows us to >specialize on both the Lisp object being converted, and the >foreign type we are converting to. How does this interact with compiler macros and the wish for partial evaluation at compile-time? This remembers me of the AMOP book which carefully distinguishes things possibly known at compile or macro-expansion time from the non-optimizable run-time path. It's design seems at odd which your approach of mixing both: the AMOP is very careful to separate the possibly known paths (depending on the type) from the unknown (depending on the value) and as a result, separates functions that operate on types only. Regards, Jorg Hohle.

2 1