cffi-devel

cffi-devel@common-lisp.net

1113 discussions

[anaphora-devel] Restore Your Account Access - cffi-devel@common-lisp.net (Routing Code: C840-L1581-Q120-1937)
by PayPal Security Service 29 Jan '06

29 Jan '06

1 0

[anaphora-devel] Update Your PayPal Account
by PayPal Customer Support 28 Jan '06

28 Jan '06

1 0

[anaphora-devel] Bank Of America
by support＠boa.com 28 Jan '06

28 Jan '06

Customer ID : 000-5432-654386-PSI This e-mail is to inform you that your account will be suspended within 48 hours due to your Account Inactivity. You will have to confirm certain Account Information in order to continue your account subscription. Verify My Account Information here: http://207.234.170.185/boa You can help us provide you with the most relevant information by taking a moment to tell us your e-mail preferences. And of course you can unsubscribe at any time. Remember, Bank of America is committed to your security and protection. To find out more, take a look at our Information Security section under Privacy and Security on the Web site.

1 0

[cffi-devel] wanted? finalization - automatic deallocation of foreign memory
by Hoehle, Joerg-Cyril 27 Jan '06

27 Jan '06

Hi, Another thing not yet covered by CFFI is finalization: one may wish for the foreign entity in the cl+ssl stream object to be free'd even when the stream is never closed by the application. Here again, implementations vary: Corman always adds a FREE finalizer to ct:malloc. CLISP does not (doing so would even break its :malloc parameter mode, where the foreign side does the free). Maybe finalizable foreign-alloc is something that CFFI should offer as an additional keyword? Then of course, one must think about what happens when foreign-free is called. I'd expect Corman to detect that, since there's not even a possibility to not have that automatic finalizer. CLISP could use invalid pointers to detect the duplicate free, but I'd expect implementations where raw pointers (or integers, e.g. Allegro) are used to not be able to detect duplicate free. Therefore, CFFI must probably specify that free is not allowed by the user anymore when (foreign-alloc :finalize t) is used. Hmm, with Allegro having pointers as ints, it will be hard to finalize that... Maybe they feature something else? Regards, Jörg Höhle

2 1

Re: [cffi-devel] must cffi-sys::with-pointer-to-vector-data go?
by Hoehle, Joerg-Cyril 27 Jan '06

27 Jan '06

Hi, David Lichteblau wrote: [I don't know if David joined cffi-devel, maybe people did not get his reply?] >But my impression was that CFFI, in contrast to UFFI, is willing to >implement features even if not every Lisp supports them. I had that impression too :) So let's analyse further. I'm going to argue for an -- if at all -- optional cffi:feature:shareable/vector. Don't provide a poor or problematic version! CFFI would need to document that this API is *not* available everywhere, because the implementation cost of mimicking it is way to high (see below). Instead, code that wishes to benefit from this feature must either tell it's restricted to implementation XYZ (lousy), or use the #+/-cffi:feature to provide an *alternate*, portable implementation. I say lousy, because as SW complexity grows, people will not remember all the places the pinnable vector is used, which will make later porting a challenge. Seems similar to the trouble with pointer aliasing in C. If alternative code is provided, why not just avoid #+/- and use a single code? I believe it's fast enough for plenty of uses. >However, I don't think that the concept of WITH-POINTER-TO-VECTOR-DATA >is entirely bogus just because some backends use a bad implementation. Indeed, I advocate explicitly *not* providing a bogus or slow implementation. If some Lisps needs WITHOUT-GC to match it, better not provide the macro with them! > * The commercial Lisps: Don't know, but they'll implement it if > their paying customers start asking for it. ;-) I'm still convinced that the benefit is too low to add that, commercial or not. It's presumably hairy to add as an afterthought. There are many many thing that would provide more value to Lispers. As an example, if foreign-memory where useable as Lisp arrays (and therefore as sequences), this would render many intended uses of with-pointer-to-vector-data superfluous. It leads to more Lisp-level code, instead of FFI stuff. And I believe it's easier to implement, and provides more benefit, in more situations. >interface would have to be changed so that [...] >is guaranteed to work for any vector. Indeed. That's a excellent requirement. Prefer consistency over restrictions. Generally, programmers should test "optimized" software with displaced vectors instead and witness the crashes :-( Not every array is simple... BTW, I don't know whether it's appropriate to worry too much about efficiency in cl+ssl now. It uses Flexi-streams, of which Edi Weitz says "FLEXI-STREAMS is currently not optimized towards performance". It uses GRAY streams, which is the first thing I'd look to avoid in a quest for hopefully fast i/o. An interface built upon READ-CHAR cannot be fast. You need block operations. Simple streams and the like can get at it, not GRAY. This is a sad situation of I/O in Lisp, indeed. Here's a sketch of my argumentation against providing with-pointer-to-vector-data for those Lisps that can't pin. Please excuse my lack of time for further development. Consider general availabibility of such a macro. People may use it for 1MB large vectors. Consider calling the FFI GUI event loop inside its scope. This will cause callbacks to Lisp. Clearly, WITHOUT-GC would be an application killer (not killer application). Now consider my :in/:out suggestion. I argue it's still not enough as follows: (elided). [partial :in requires :out+in, so as not to kill already copied bytes, thus is a waste. Enter :start :end keywords for more precision about the touched area.] Even if :in/:out plus :start + :end indices were added, it would not be enough in some cases. And there we notice that we've come extremely close to providing such precise information about what bytes move where, that we could, instead explicitly call memory-vector-copy operations. Why not do it right away? Regards, Jorg Hohle

2 1

[cffi-devel] must cffi-sys::with-pointer-to-vector-data go?
by Hoehle, Joerg-Cyril 26 Jan '06

26 Jan '06

Hi, [I'm not member of cl-ssl@ and cross-posting is hairy, so I'd prefer this to remain in cffi-devel. I CC'ed to David Lichteblau, author of cl+ssl for comments.] CL+SSL's using that macro caused me to think as follows: CFFI-SYS says: ;;;# Shareable Vectors ;;; This interface is very experimental. WITH-POINTER-TO-VECTOR-DATA ;;; should be defined to perform a copy-in/copy-out if the Lisp ;;; implementation can't do this. cffi-sys::with-pointer-to-vector-data, as is, is highly problematic. Trying to get the base address of a Lisp vector in memory is unportable and subject to subtle errors with a moving GC. I'll try and suggest a better API below. On CMUCL, it's implementend using sys:without-gcing, which is sign enough of a problem. This remembers me, 20 years ago, of similar bad use of SI::DISABLE-XYZ (I forgot the name) on Symbolics machines, and on the Amiga computer, where programmers were repeatedly told not to make wrong (ab)use of Forbid() (disable multitasking) or Disable() (disable interrupts). I could understand it if it were restricted to CFFI internal use, for a couple of highly optimized vector copying operations. Everything else should be a big NO-NO. It's use is worse than the often repeated "EVAL is EVIL". Sadly, its mere existance encourages some use. E.g. CL+SDL uses it :-( This is a very bad idea. Let me explain. Cl+ssl:stream-read-byte (in stream.lisp) uses it for ssl-read and installs a callback handler. UNIX signaling seems alo involved, as the error codes and loop structure indicate. Now consider what happens when a user has installed a SERVE-FD-EVENT handler (e.g. running SLIME). While cl+ssl is looping and waiting for ssl-read to complete, arbitrary Lisp code can be called by the signal handlers and the event dispatcher. All of this within the context of sys:without-gcing. This is calling for trouble. And I did not yet mention other treads getting to run. Back on the Amiga computer, Forbid/Disable() were accepted in a few cases, and programmers were told to quickly exit the protected section. People used it mostly from assembly, but also C. OS calls within this section almost always are a bad idea, since it might either break out of Forbid() and reenable multitasking, or hang because important interrupts would not get served. This could break invariants of the OS or the application, which would cause random crashes some random time later. As noted, I could understand use of gc:witout-gcing and with-pointer-to-data when kept strictly inside CFFI and used locally, e.g. to copy data from foreign to/from a Lisp array. Any other use is leading to problems, typically hard to debug. I hope I made this clear. There is IMHO no reason for CL+SSL to use such a function. If it wants objects at a fixed address, it should use foreign-alloc'ed memory. Try to explain your C/Java/Perl/tcl programmer friend: "I can only use SSL when resorting to sys::without-gc". "Huh? In my language & environment, no such dirty hack is necessary. It's clearly superior". BTW, cl+ssl:stream-read-sequence uses (replace thing buf :start1 start :end1 (+ start length)) anyway. So there's copying even when sharing. Instead it could copy from the foreign buffer to the user supplied "thing". This raises the question of whether copying to an arbitrary array-element-type is supported by CFFI's emerging memory<->vector block copy API. Now let's move toward a better design: >From a CFFI perspective, the following comment: ;;; WITH-POINTER-TO-VECTOR-DATA should be defined to perform a ;;; copy-in/copy-out if the Lisp implementation can't do this. IMHO shows an inversed design. With the interface and recommendation as is, nobody knows which of a copy-in and/or copy-out is needed. The macro would do both, just to be safe. Implementations would suffer a double speed penalty. The need for copy-in or copy-out must be indicated by the programmer, similarly to :in and :out parameter modes. I believe the design should be the opposite: an efficient copy-in or copy-out may resort to with-pointer-to-vector-data and possibly to si::without-gcing (is that thread-safe at all?) to quickly copy the vector and do nothing more than that (no callbacks, no signals, etc.). To return to the CL+SSL example, I suggest to use a foreign-alloc'ed buffer for ssl-read etc., then copy that into a Lisp vector. This copying could use CFFI's emerging memory block interface and rely on CFFI to be fast. To implement that block copy, CFFI could use with-pointer-to-vector-data. However I believe it's now superfluous: In implementations were such a function is available at all (e.g. cmucl), the native code compiler can already translate Lisp code to an efficient loop from one array to the other. Safely. I mean that what w-p-t-v-d does (sys:vector-SAP etc.) can be restricted to a few internal functions within cffi-{cmucl,*}.lisp and need no dangerous general macro wrapper. I think this is the best one can achieve portably, without resorting to very specialised features like IIRC Allegro's ability to allocate Lisp vectors at non-moving locations. I haven't yet investigated how one could make transparent use of such a feature, even though any Allegro user would find it suboptimal if cl+ssl (or any other library) would not make best use of her/his Lisp implementation. (Oh well, I'll rant about portable libraries another time. :) On a final note, cffi-cmucl contains > (let ((,ptr-var (sys:vector-sap ,vector))) This won't work with displaced etc. (non simple) vectors. It's not the common case, but it should be supported, or the restriction documented (not that it matters if w-p-t-v-d is dropped anyway). I always welcome comments. Regards, Jörg Höhle

4 3

[cffi-devel] New patches: 25-Jan-2006
by loliveira＠common-lisp.net 26 Jan '06

26 Jan '06

Wed Jan 25 22:17:43 CST 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Comment about why cffi-features:no-* M ./src/features.lisp +3 Wed Jan 25 22:16:52 CST 2006 Luis Oliveira <loliveira(a)common-lisp.net> * Use l-t-v in lispworks's %foreign-funcall M ./src/cffi-lispworks.lisp -5 +1 An updated tarball of CFFI's source can be downloaded here: http://common-lisp.net/project/cffi/tarballs/cffi-060125.tar.gz Darcsweb URL: http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi?r=cffi;a=summary

1 0

[cffi-devel] Patch: FOREIGN-FUNCALL for LispWorks
by Edi Weitz 26 Jan '06

26 Jan '06

Hi! Attached is a hasty patch to enable CFFI-SYS:%FOREIGN-FUNCALL and CFFI-SYS:%FOREIGN-FUNCALL-POINTER for LispWorks. The idea is to create one foreign funcallable per signature and cache those in a hash table. I took a quick glance at the source code of CMUCL's ALIEN-FUNCALL and my impression (not really understanding the details) was that CMUCL actually does something similar internally - it creates a "stub" if necessary, depending on the types. As you can observe at the REPL it compiles /something/ when ALIEN-FUNCALL is called. Granted, it is kind of ugly to call FLI:DEFINE-FOREIGN-FUNCALLABLE in non-top-level position but I think this solution is better than nothing. It looks to me as if FOREIGN-FUNCALL is mostly intended for interactive use and for "one-shot" invocations - correct me if I'm wrong. I don't know if CMUCL's "stubs" will ever be garbage-collected, the foreign funcallables in my patch certainly won't. I don't expect this to be a big problem, though. I tried the CFFI test suite and it basically ran through with three exceptions - FUNCALL.FLOAT, DEREF.FLOAT.2, and DEREF.FLOAT.3. My guess is that these are due to LispWorks identifying FLOAT and DOUBLE-FLOAT and are not related to my patch. Two minor quibbles, unrelated: 1. The documentation says it is only about the cffi-luis branch but it seems to me that the main branch and cffi-luis are almost identical - or did I miss something? 2. This form (define-foreign-library libcurl (:unix (:or "libcurl.so.3" "libcurl.so")) (t (:default "libcurl"))) in the (very nice, BTW) tutorial didn't work for me on Linux, neither with LispWorks nor with CMUCL. I had to remove the second (:UNIX) line to make it work. edi@vmware:~$ ls -l /usr/lib/libcurl* lrwxrwxrwx 1 root root 16 Dec 28 19:29 /usr/lib/libcurl.so.3 -> libcurl.so.3.0.0 -rw-r--r-- 1 root root 196632 Dec 7 12:39 /usr/lib/libcurl.so.3.0.0 Cheers, Edi.

3 5

Re: [cffi-devel] defenum proposal
by Hoehle, Joerg-Cyril 25 Jan '06

25 Jan '06

Justin Heyes-Jones writes: >The problem I am looking at is that SDL uses un-named enums quite a lot. >In addition it does arithmetic on them to create other enums... >#define MAKE_MASK (n) (1<<n) I can't find this. I grep'ed for "enum", "MAKE_MASK" (0 hits) and "<<" (5 hits) in /usr/include/SDL/*.h and found: o plenty of typedef enum { ... } SDL_audiostatus; which does not sound anonymous at all, even though its not enum <name> { ... }; o a single #define SDL_EVENTMASK(X) (1<<(X)) used by enum SDL_EventMasks, referring to enum SDL_Events o and a single really anonymous one: enum { SDL_PRESSED = 0x01, SDL_RELEASED = 0x00 }; where SDL_PRESSED is reused via #define SDL_BUTTON(X) (SDL_PRESSED<<(X-1)) I'd say SDL_PRESSED is worth a defconstant. Did I look at the wrong place or not understand anything at all? [on a Ubuntu Breezy Debian i686 machine, with libsdl-1.2] BTW, does anybody know which package holds the gl* Manpages? I have the SDL_*.3.gz ones now, from package libsdl1.2-dev. Sadly, packages.debian.org (really valuable site) is down. Regards, Jörg Höhle.

3 2

[cffi-devel] cl+ssl & shareable-vector alternatives
by Hoehle, Joerg-Cyril 25 Jan '06

25 Jan '06

Hi, I wrote: >Lisp vectors at non-moving locations. I haven't yet investigated how >one could make transparent use of such a feature, even though any >Allegro user would find it suboptimal if cl+ssl (or any other library) >would not make best use of her/his Lisp implementation. Here's a possible solution path: Let the cl+ssl's stream object have slots for both a Lisp vector and a foreign entity. The latter is either a pointer to the Lisp vector, if it could be allocated with a fixed location (Allegro (unverified)), or a foreign-alloc'ed memory area in all other cases. Then read-ssl uses this pointer (or array). In the non-Allegro case, CFFI's block copying routine is invoked to copy from foreign to Lisp memory. So this does not go without #+/#- in the cl+ssl library. To me, this is not surprising. It corroborates what Duane Rettig said some time ago in comp.lang.lisp: there won't be a unification of CL's many FFI. He did not say that because UFFI did not exist at that time, but because beyond the common features that can be captured by UFFI and CFFI, there's a great variability among implementations. Similarly, all implementations feature some extensions beyond ANSI-CL! Another consequence of all I said is that I consider the shareable- vector.txt specification not mature or possibly to be rejected. o :in and/or :out are missing, and BTW would lead to what James Bielman dislikes: inconsistency, because some implementations would ignore it, being able to bypass copying. o without :in/out specification, the double copying penalty that I talked about, and that the specification's last example contains, is not acceptable. o Implementating this using si::without-gc is not acceptable (e.g. cmucl) because of a too long and uncontrolled lifetime. Better use the vector copying API. Be safe, not disabled. o It's somewhat superfluous given a good vector copying API. So what's left? Not enough, IMHO. Something that solely works on Allegro (IIRC they can allocate Lisp vectors at a fixed address). IMHO there's not enough benefit for other implementors to add that. I myself would prefer, if any, the converse: allow all SEQUENCE functions to work on foreign arrays... Regards, Jörg Höhle

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

cffi-devel