Hello,

The cffi-newtypes branch[1] is getting huge (~30 patches), and since it seems nobody has complained much about the new type system, I'd like to push it into the main branch along with the encoding support and other minor features and bugfixes that have accumulated over last ~2 months. Any objections?

Then, after some remaining issues are fixed, I'd like to release 0.9.3.

So here's my TODO list for 0.9.3:

- http://article.gmane.org/gmane.lisp.cffi.devel/1033 Figure out whether any of those issues are critical. The only change I've made to the typesystem since I wrote that message was a new :class option to DEFSTRUCT;

- iron out some issues with the new encoding support (see below, this might take a while);

- finish the documentation of the new features;

- integrate cffi-grovel into the CFFI tree and document it.

There are some issues with the current encoding support though:

- James' original code had a CFFI-SYS:DEFAULT-ENCODING function which would use some implemention-specific way of determining what the default encoding should be. Every implementation had a different way of determining that, so I though that simply picking one ourselves (say, :utf-8) would be better.

Also, I changed that to be a special variable instead. That way *DEFAULT-FOREIGN-ENCODING* can be bound to something else and affect the behaviour of the :string type and other string operations that don't explicitely specify an encoding. (E.g., the behaviour of :STRING would be influenced at run-time by the value of *D-F-E* whereas (:STRING :ENCODING :UTF-8) would not.)

- Allegro's %lisp-string-into-foreign overflows. Allegro's EXCL:STRING-TO-NATIVE doesn't take a bufsize argument. Another problem is that Allegro's %lisp-string-octet-length isn't very effective otherwise it'd be easier to check for an overflow. Anyway, this is not the worst issue.

- Corman, SCL and ECL support is broken, not necessarily because of the new string stuff. Also, a recent 1.1 snapshot of OpenMCL with unicode support is necessary; is this a problem?

Last but definitely not least, there's a huge problem with error semantics. Everything works fine until you try to, e.g. convert a #\λ into iso-8859-1. Or #\ç into ascii.

Some Lisps treat ascii as a synonym for iso-8859-1. Some silently substitute inconvertible characters with #? (or #\Sub) while others will raise an exception. Of those that do raise exceptions, some provide a use-value substitution restart, others don't. To sum it up, it's horribly inconsistent.

Before forward porting James's cffi-encondings branch, we discussed the possibility of doing the enconding conversion in portable CL, like what flexi-streams does for instance. I'm beginning to reconsider this idea. One the one hand it would mean introducing a dependency for CFFI, on the other hand it would provide consistent semantics across the various Lisps (including useful semantics for lisps that don't support unicode) we support and simplify CFFI-SYS. Any thoughts?

[1] http://common-lisp.net/~loliveira/darcs/cffi-newtypes/