January 2006 - cffi-devel - mailman3.common-lisp.net

Re: [cffi-devel] a thought on string encodings
by Hoehle, Joerg-Cyril 03 Jan '06

03 Jan '06

Hi, James Bielman wrote: >I think CLISP has enough to implement these fairly efficiently, The devil's details lie in the many keyword arguments... For example, all :offsets means that the CLISP code must provide for possible addition, instead of directly using the pointer. But it's ok to leave it in. For example, I somewhat dislike LispWork's convert-from-foreign-string because it takes both :length and :null-terminated-p keys, which are not orthogonal. :length nil :null-terminated-p nil is an error. (Perhaps I should also dislike the ANSI-CL functions with :Test and :Test-not etc.?) Plenty of keyword arguments tend to make implementations clumsy. Also, they tend to hide the low-level basic functionality from the nice-to-have (cf. cffi:foreign-alloc). Most of your proposal seems in-line with ideas I had for APIs for CLISP, however the following stroke me: >Function: FOREIGN-STRING-TO-LISP pointer encoding &key start end >Convert octets from START to END (octet indices) I think a) you should leave :start and :end to Lisp vectors and stick to :offset with foreign pointers b) octets indices will be unnatural and thus error-prone for people mostly on MS-windows, constantly manipulating UTF-16 strings. I suggest you use :offset (in bytes, as usual) and :count (also in bytes/octets) instead. Unlike LispWork's function, these are othogonal. >Function: LISP-STRING-TO-FOREIGN string encoding &key start end >Function: LISP-STRING-OCTET-LENGTH string encoding &key start end :start + :end consistently provided for Lisp strings is good. >Function: FOREIGN-STRING-LENGTH pointer encoding >This should be smart enough to look for 8-bit vs 16-bit null >terminators, as appropriate for the encoding. TRT >I don't think it will support much beyond :ascii or :iso-8859-1 in >non-Unicode Lisps---I don't want to encumber CFFI with a bunch of >character code tables. I agree, but You don't have users in Asia yet, do you? In other words, you should provide for extensibility. One way would be to pass non-recognized encoding designators through to the underlying Lisp (which would make it difficult to find out the number of zero terminators for functions of yours that need it, and possibly others quirks waiting at the corner). >Function: LISP-STRING-TO-FOREIGN string encoding &key buffer >BUFFER must be large enough to accommodate the foreign >string---this can be queried with LISP-STRING-OCTET-LENGTH. This will lead to double work being done by mbslen() et al. on behalf of people calling first *-length, then *-to-foreign. While I see the need to be able to write directly into a buffer area, there ought to be better APIs than this one. Regards, Jorg Hohle.

1 0

Re: [cffi-devel] a thought on string encodings
by Hoehle, Joerg-Cyril 03 Jan '06

03 Jan '06

James Bielman wrote: >Now that this is working, I will start pulling the >implementation-specific code into an encoding-aware >%FOREIGN-STRING-TO-LISP function in CFFI-SYS. How are you going to represent the encodings? - use the implementation's objects? - introduce your own names? - restrict yourself to known encodings, e.g. UTF-8 (users on MS-Windows would appreciate UTF-16)? >#+clisp >(defcfun "strlen" :unsigned-int > (s :pointer)) Hmm, I lost track of whether that works on MS-Windows in a MS-VC build (as opposed to mingw/cygwin/unix-like). Yeah, I should really add this string stuff to clisp instead. Regards, Jorg Hohle.

3 3

Re: [cffi-devel] a thought on string encodings
by Hoehle, Joerg-Cyril 03 Jan '06

03 Jan '06

Yaroslav Kavenchuk wrote: >>Thus (let ((custom:*foreign-encoding* charset:foo))) won't work as >expected, you >It does ext:letf[*] Thanks, Yaroslav. I forgot about that one, which is odd since IIRC it's me who pressed Sam Steingold to explicitly mention this is the CLISP impnotes. -- Blush. Still I don't appreciate interfaces that need unwind-protect etc. or special variables to work. I very much prefer functional APIs. The other ones get scoping typically wrong. Consider th effect of (ext:letf ((custom:*foreign-encoding* charset:utf-16)) (convert-some-string) (foreign-funcall some-function-which calls back into Lisp)) Within the extent of the callback, the encoding would be set to an unusual value... Regards, Jorg.

2 1

[cffi-devel] Type Translator Branch Update
by James Bielman 03 Jan '06

03 Jan '06

Hi all, I've updated the cffi-new-translators branch, improving the exported interface and fixing a few bugs. The branch is available via Darcs (or a browser to browse the source) at: http://slacknet.com/~jamesjb/cffi-new-translators The exported interface for defining type translators is by specializing the following generic functions: Generic Function: TRANSLATE-TO-FOREIGN value type-name Convert the Lisp object VALUE to a foreign object of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE. Methods on this generic function should EQL-specialize TYPE-NAME for the type to translate. Returns the translated foreign object and an optional second value that will be passed as the PARAM argument to FREE-TRANSLATED-OBJECT. Generic Function: TRANSLATE-FROM-FOREIGN value type-name Convert the foreign object VALUE of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE, to a Lisp object. Methods on this generic function should EQL-specialize TYPE-NAME for the type to translate. Generic Function: FREE-TRANSLATED-OBJECT value type-name param Perform necessary deallocation of the foreign object VALUE, of the type named by TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE. Methods on this generic function should EQL-specialize TYPE-NAME for the type to deallocate. PARAM will contain the second value returned by TRANSLATE-TO-FOREIGN for this type, or NIL if no second value was present. If no deallocation is required, it is not necessary to specialize this method. Here's an example, a simplified version of the :boolean type: (defctype boolean :int) (defmethod translate-to-foreign (value (type (eql 'boolean))) (if value 1 0)) (defmethod translate-from-foreign (value (type (eql 'boolean))) (not (zerop value))) Also, look at examples/gettimeofday.lisp and translator-test.lisp for more examples. There is another, unexported, set of generic functions used internally for translating whole classes of types. Look at TRANSLATE-TYPE-{TO,FROM}-FOREIGN and FREE-TYPE-TRANSLATED-OBJECT for these (and methods defined on them for FOREIGN-ENUM, etc). With respect to the old :TO-C-DYNAMIC translators, I'm inclined to not worry about optimizing this case until there is a specific performance need for it. I think the ability to dispatch on the type of the value being translated is more useful than the optimization it was providing. Comments? James

1 1

Re: [cffi-devel] a thought on string encodings
by Hoehle, Joerg-Cyril 03 Jan '06

03 Jan '06

James Bielman wrote: >I'm working on (to begin with), a UTF8-STRING type >which converts Lisp strings to/from UTF-8 on Unicode Lisps. >(Does the CLISP FFI provide something like memcpy?) No. Some voice in me says I should rather implement the vector<->memory functions for CLISP instead of playing around with cffi, Iterate, reading interesting papers etc. when I have a little time. >So, I think I need that block interface we've talked about. Reviewing the proposal has been on my TODO list for a long time as well, I'm sorry. > (ffi:with-foreign-string (ptr chars bytes s :encoding charset:utf-8) > (let ((buf (foreign-alloc :unsigned-char :count bytes))) > (memcpy buf ptr bytes) I think I'd rather use ext:convert-string-to-bytes, then use my non-existent vector->memory function. Sometimes I feel uneasy about stack-allocating possibly huge strings (e.g. 1MB or more!). In the meantime, (let* ((bytes (ext:convert ...)) (len (length bytes)) (buf foreign-alloc len)) (setf (memory-as buf (parse-c-type `(c-array uint8 ,len))) bytes) should be among the fastest ones (CLISP can copy the (c-array uint8 N) type fast). And people keep saying the generational GC should be able to free the garbage vector easily. >However, I haven't been able to find an inverse for >FFI:WITH-FOREIGN-STRING. Indeed, I should ... (see above) and implement string-from-foreign and (foreign-string-length :encoding) as a means to interface to mblen(). >I'd like to be able to convert a pointer back >to a Lisp string without looping in bytecode to create a vector of >octets from the pointer. I suggest the converse of the above, via an array of (unsigned-byte 8). > I tried a >whole bunch of combinations of FFI:MEMORY-AS with FFI:C-ARRAY-PTR types >and got nothing but segfaults. Please report a bug, but possibly you just did not re-read recent impnotes closely enough. For instance, while the FFI now accepts non 1:1 encodings, it's use with c-array-ptr and c-array[-max] is mostly broken and some parts revert to a 1:1 encoding (see *foreign-8bit-encoding*). I would have left the 1:1 restriction and take more time to think about the problems. :-( The operators that explicitly take an :encoding are safe, like with-foreign-string. Also, custom:*foreign-encoding* is a symbol macro. Thus (let ((custom:*foreign-encoding* charset:foo))) won't work as expected, you need setq and unwind-protect. :-( > Is there something I can use to convert >the pointer to either a vector of octets (which I can pass to >EXT:CONVERT-STRING-FROM-BYTES, or to a Lisp string directly? (memory-as pointer (parse-c-type `(c-array uint8 ,len))) Looking at the clisp sources, (c-array character N) -> Lisp string seems ok as well. But e.g. don't use (c-array-max character N) with UTF-16! As you can see, there's room for improvement. Summary: ptr -> Lisp string: either ext:convert + memory-as uint8 or (let ((old *foreign-encoding*)) (unwind-protect (progn (setq *foreign-encoding* utf-8) (memory-as pointer (parse-c-type `(c-array character ,known-length))) )(setq *foreign-encoding* old))) ; looks scary :-( Do you need unknown-length as well? Lisp string -> ptr: ext:convert + memory-as uint8 as mentioned above. Regards, Jörg Höhle.

3 2

[cffi-devel] GtkMozEmbed: fixed
by Keith Irwin 03 Jan '06

03 Jan '06

Folks-- For some reason I'm not getting the emails I send, even though I've properly set the bit in the list software. Ah, well. Jamesb on #lisp solved the problem. Apparently, you need to set the profile path for some sites, or the embedded gecko widget tanks. The profile path is the place the browser saves all the cookie/login information. I've updated the code I linked to earlier in case anyone wants to play with it. Personally, I think CFFI made this all pretty easy, though kudos to the mozilla team for the library in the first place. Thankfully, this really wasn't a CFFI problem. BTW, using glade made GTK pretty painless. I just had to define the procedures I needed. I can see how, over time, a GTK interface (CLOS based) might get implemented, but with libglade, I almost think, why bother? Now, if I could just integrate slime/swank into GtkSourceView.... Keith

1 0

[cffi-devel] gtkembedmoz and https segfault: cffi related?
by Keith Irwin 02 Jan '06

02 Jan '06

Folks-- I've written a little toy app that embeds mozilla as a widget, allows for some minimal back/forward and location bar functionality. Using CFFI, of course. It works really well, except I get a segfault (I think) when I click or type in any link which leads to (or incorporates part of) an HTTPS site. Might this in any way be an CFFI problem? You can download the code at: http://zentrope.com/code/cl-browser.tgz Runs on ubuntu (as long as you've got libglade installed, see the README). If you load the app (browser:main) at the repl, then click the "sign-in" link on the google page, you'll get a segfault. I get the following stack trace when I do that: 0: (SB-KERNEL::MEMORY-FAULT-ERROR) 1: ("foreign function: call_into_lisp") 2: ("foreign function: post_signal_tramp") 3: ("foreign function: NSSRWLock_LockRead") 4: ("foreign function: SECMOD_GetReadLock") 5: ("foreign function: PK11_TokenExists") 6: ("foreign function: #xB1E6388C") 7: ("foreign function: #xB1E6EDFC") 8: ("foreign function: #xB1E71F4A") 9: ("foreign function: #xB1E74340") 10: ("foreign function: #xB1E755F2") 11: ("foreign function: #xB1E75684") 12: ("foreign function: #xB1E7898D") 13: ("foreign function: #xB1D5070E") 14: ("foreign function: PR_Write") 15: ("foreign function: #xB6689D83") 16: ("foreign function: #x8578028") 17: ("foreign function: #x8578040") If this is a CFFI problem, then, cool, I'm reporting it to the right place. If not, perhaps you know what it might be? Surely not a mozilla prob. Perhaps SBCL? Keith

1 0

Re: [cffi-devel] Providing access to errno in CFFI-SYS
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

James Bielman wrote: >> See clisp/modules/bindings/glibc/linux.lisp >I'm not an expert CLISP user---is this not built as a part of the >standard image? What's standard? ./configure --build? - as found on sourceforge file download? - as found in Debian? I don't know the choices myself, but clearly, that file is only available on systems with glibc (I've heard mention of people trying and compiling this file on *BSD). Furthermore, it's possibly not in the default build. For sure, it was not in the default build of old CLISP and only there with clisp --full, but it seems Sam Steingold recently changed defaults and it may be there in a default build and invocation of clisp on Linux. So indeed, the linux package is not in every clisp. clisp --version will list the C Modules. My typical build contains none of them, while Ubuntu/Debian clisp-2.35 contains i18n, syscalls, regexp, linux, clx. So for now you could perhaps try something along: (ignore-errors (or (when (find-package "LINUX") (read-from-string "linux:errno")) (def-call-out errno-location (:name "__errno_location")(:return-type (c-pointer int))(:library :default)(:language :stdc)) (def-c-var errno (:type ffi:int)(:library :default)) ; works on Linux!??! )) ... Or just use the last one with def-c-var errno, since the rest is overengineered for the day when clisp might support threads. Don't hold your breath. (Actually, I'm surprised that dlsym() found "errno" on my Linux box. I'd thought __errno_location() rules in glibc/i386. Go figure). Regards, Jörg Höhle.

1 0

[cffi-devel] Re: CFFI/CLISP - 2 out of 101 total tests failed
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

Luís Oliveira wrote: >About the FUNCALL.VARARGS.DOUBLE test, it passes on Linux and OSX but >fails on Win32. Probably a CLISP bug, Joerg? :-) Maybe, maybe not. My opinion is that interfacing to varargs is hairy and I hear mention of problems with them, when coming from any language (Python, perl, Haskell, .NET, xyz). I believe the tests are (potentially) incorrect. As I was quoting in "[cffi-devel] bits from a new tutorial" on 2005-12-23, http://gcc.gnu.org/ml/gcc/2004-06/msg01019.html "It's assuming that the calling conventions for a varargs function and a non-varargs function are the same, and that's not necessarily true on all platforms." I believe this could explain the observed behaviour. The test calls sprintf() as if it were declared void sprintf(void*,char*,double); This could cause the value to be passed directly in a floating point register, which would not surprise me. Whilst varargs are all passed on the stack. So far, I've no reason to believe there's a bug in CLISP, but try to convince me of the contrary. I believe interfacing to varargs is a problem and needs careful thought. In the meantime, I suggest to recommend not to use them. Individual tests may remain in testsuite so we are remembered of the problem. My current thought is that I'm going to suggest to the maintainer of the ffcall package (which happens to be Bruno Haible, CC'ed) to provide some mean to interface to such functions(*), or restrict it, warn about their use, or try to narrow down the problematic area to well documented cases. (*) would mean that the top-level description also would need to account for that, i.e. ffi:def-call-out or cffi:defcfun need means to describe them. It's possibly hairy. Regards, Jörg Höhle.

1 0

[cffi-devel] Re: CFFI/CLISP issues with saved images
by Hoehle, Joerg-Cyril 02 Jan '06

02 Jan '06

Luís Oliveira wrote: >I was just testing CFFI/CLISP by loading the testsuite, dumping an >image, loading it and running the tests. CLISP crashes. There are two >problems: > >- The foreign library (in this case libtest.so) is not being >reloaded. [...]I recall CLISP's ffi:def-call-out taking care of this but Hmm, did you try an absolute path? I suspect the library was not found. I'd naively have expected this to work in the context of cffi as well as with direct uses of the FFI. >- The callbacks become NULL pointers, thus all callback.* tests fail. >Any suggestions for fixing that? Randomly placing load-time-value >forms in %defcallback didn't seem to work. :-) [replying without having thought much about this issue]. CLISP's image is not like an Emacs dump of memory. It doesn't preserve addresses. It doesn't preserve malloc()'ed stuff, which I believe callbacks are (or parts thereof). Actually, I would *not* expect any implementation to be able to reproduce the exact same address of anything when started from an image. CMUCL seems an exception, where there are additional constraints (like on Solaris, getting the linker to load a shared object file at a known location). Similarly, any foreign library that stored a callback-pointer *must* *not* expect such a pointer to be valid when restarting. That library may itself be at a different address when reloaded, how could the callback not change? IMHO, it might work in a scenario on Linux with naive tests where you exit and immediately restart, but what if you upgrade your system in between etc.? Instead, my mental model is that callbacks must be re-established dynamically when an application starts. -- which makes me think that the code for free'ing an old callback upon redefinition in cffi should add a ext:validp test before free'ing. I just added a check in my branch. That's what I meant in my mail "unnamed callback" from 2005-12-19, where I wrote: >(defun my-application > (let (x y) > ...code... > (register-callback 'close-window I'd be happy to discuss this further. Restarting from one's images is being neglected in the Lisp community these days IMHO. Smalltalk implementations seems to behave better. Regards, Jörg Höhle.

1 0