Hello,
I'd like to merge cffi+lotsastuff[1] with the main branch soon. People have started using that branch in several projects and that has generated unnecessary confusion. This branch includes the following new features:
- efficient external-format support for foreign strings - cffi-grovel - emulated long-long types - other minor stuff
The only problem is that this branch brings in 3 new dependencies: babel[2], trivial-features[3] and alexandria[4]. Is this a big problem? I hope it isn't and I'll do the merge soon unless there are any objections.
[1] http://common-lisp.net/~loliveira/darcs/cffi+lotsastuff/ [2] http://www.cliki.net/trivial-features [3] http://common-lisp.net/project/babel/ [4] http://common-lisp.net/project/alexandria/
Luís Oliveira wrote:
Hello,
I'd like to merge cffi+lotsastuff[1] with the main branch soon. The only problem is that this branch brings in 3 new dependencies: babel[2], trivial-features[3] and alexandria[4].
I think I remember seeing trivial-garbage in there a while back, but not now. Could this be rolled into CFFI?
- Luke
On Mon, Apr 7, 2008 at 12:56 AM, Luke J Crook luke@balooga.com wrote:
I think I remember seeing trivial-garbage in there a while back, but not now. Could this be rolled into CFFI?
Why? What do you mean?
Luís Oliveira wrote:
On Mon, Apr 7, 2008 at 12:56 AM, Luke J Crook luke@balooga.com wrote:
I think I remember seeing trivial-garbage in there a while back, but not now. Could this be rolled into CFFI?
Why? What do you mean?
I hadn't realized that finalize and cancel-finalization are now part of CFFI. I'm still using trivial-garbage for this. Was this added recently?
- Luke
On Mon, Apr 7, 2008 at 6:38 AM, Luke J Crook luke@balooga.com wrote:
I hadn't realized that finalize and cancel-finalization are now part of CFFI. I'm still using trivial-garbage for this. Was this added recently?
No, they were removed. They are still in the 0.9.2 tarball (gosh, that release is old...) which is why they're still mentioned in the on-line manual.
Luís Oliveira <luismbo <at> gmail.com> writes:
On Mon, Apr 7, 2008 at 6:38 AM, Luke J Crook <luke <at> balooga.com> wrote:
I hadn't realized that finalize and cancel-finalization are now part of CFFI. I'm still using trivial-garbage for this. Was this added recently?
No, they were removed. They are still in the 0.9.2 tarball (gosh, that release is old...) which is why they're still mentioned in the on-line manual.
Oh, pity. I thought this was going into CFFI. Is the tarball the same as the asdf-install version available on the Cliki? I became confused as I was using one of the latest SVN versions but had to go back to the official release as the TRANSLATE-TO-FOREIGN methods stopped working.
- Luke\
On Mon, 7 Apr 2008, Luis Oliveira wrote:
I'd like to merge cffi+lotsastuff[1] with the main branch soon. People have started using that branch in several projects and that has generated unnecessary confusion. This branch includes the following new features:
- efficient external-format support for foreign strings
- cffi-grovel
- emulated long-long types
- other minor stuff
The only problem is that this branch brings in 3 new dependencies: babel[2], trivial-features[3] and alexandria[4]. Is this a big problem? I hope it isn't and I'll do the merge soon unless there are any objections.
I'd prefer if you held off on this one.
In particular, trivial-features bothers me. Libraries probably shouldn't modify the features set by an implementation. I would feel much more comfortable if all names were prefixed to reduce conflict. e.g. set :tf-darwin and :tf-linux instead of :darwin and :linux
Other thoughts: Babel is used for external string support, right? Does it slow down the base, untranslated case? Could it be optional/disabled for those of us who don't want translation?
Is there a structural reason to merge grovel with cffi? If not, could they stay separate? Does that reduce the core CFFI dependencies? You probably integrated grovel for trivial-features... Could tf search for :cffi-grovel in *features* and implement a fallback if not found?
Emulated long-long? Sounds good.
Thanks, Daniel
I'd prefer if you held off on this one.
In particular, trivial-features bothers me. Libraries probably shouldn't modify the features set by an implementation. I would feel much more comfortable if all names were prefixed to reduce conflict. e.g. set :tf-darwin and :tf-linux instead of :darwin and :linux
fwiw, i don't see in what way would it be better if the otherwise obvious symbol names were prefixed. it would just make the user code less readable. if tf modifies *features* in a wrong way then it needs to be fixed.
Other thoughts: Babel is used for external string support, right? Does it slow down the base, untranslated case? Could it be optional/disabled for those of us who don't want translation?
there's no untranslated case, you always need to turn lisp strings into byte-arrays using some encoding before calling C (except if relying on simple-base-string being ascii only, which is not mandated by the standard). the lack of proper encoding support was an important missing feature of cffi.
Is there a structural reason to merge grovel with cffi? If not, could they stay separate? Does that reduce the core CFFI dependencies? You probably integrated grovel for trivial-features... Could tf search for :cffi-grovel in *features* and implement a fallback if not found?
i'm not following you on this... cffi-grovel is a standalone .asd already, it's just practical to keep it in the same repo as cffi itself because it tightly depends on cffi and it's rather small to be a separate project.
On Mon, 7 Apr 2008, Attila Lendvai wrote:
On Mon, 7 Apr 2008, Daniel Herring wrote:
I'd prefer if you held off on this one.
In particular, trivial-features bothers me. Libraries probably shouldn't modify the features set by an implementation. I would feel much more comfortable if all names were prefixed to reduce conflict. e.g. set :tf-darwin and :tf-linux instead of :darwin and :linux
fwiw, i don't see in what way would it be better if the otherwise obvious symbol names were prefixed. it would just make the user code less readable. if tf modifies *features* in a wrong way then it needs to be fixed.
Until we get a CDR/other generally accepted spec on how *features* should reflect the platform, there are no wrong settings in *features*. Since this is a global value, special care must be taken to not break code which has already been tuned for an implementation. It seems rude for a nested library to change common global settings.
Putting a prefix on all customized features - minimizes conflicts with implementation-specific flags - is explicit about intent; #+:unix says "I want unix". #+:tf-unix says "I want a tf-compliant unix".
Since we probably can't alter the spec, a prefix may be reasonable.
Is there a structural reason to merge grovel with cffi? If not, could they stay separate? Does that reduce the core CFFI dependencies? You probably integrated grovel for trivial-features... Could tf search for :cffi-grovel in *features* and implement a fallback if not found?
i'm not following you on this... cffi-grovel is a standalone .asd already, it's just practical to keep it in the same repo as cffi itself because it tightly depends on cffi and it's rather small to be a separate project.
This branch adds three new dependencies. Babel for string translation is understandable. Alexandria and trivial-features are somewhat less obvious. Without reading the darcs branch, I was trying to ask whether these dependencies are only due to the integration of cffi-grovel, or whether they were being used in the core cffi functions as well.
If the core cffi is introducing these dependencies, then I'm checking that we gain something tangible in return. If these two are just for cffi-grovel, but cffi can be used without grovel, then sure; add grovel to cffi, but please make it clear in the install docs which deps are needed for what. Something like "Babel is needed for CFFI; Alexandria and trivial-features are only used by cffi-grovel."
I don't want to hold things back (cue CFFI-0.9.2 vs debian-stable jokes); but I would like to avoid unnecessary dependencies.
Later, Daniel
On Tue, Apr 8, 2008 at 6:30 AM, dherring@tentpost.com wrote:
Until we get a CDR/other generally accepted spec on how *features* should reflect the platform, there are no wrong settings in *features*. Since this is a global value, special care must be taken to not break code which has already been tuned for an implementation. It seems rude for a nested library to change common global settings.
Again, I agree with you that it doesn't feel very good at first sight. But, can you give me an example how trivial-features would break existing code? AFAICT, implementations seem to introduce aliases for already existing keywords in *features* from time to time. That is what trivial-features does as well.
Putting a prefix on all customized features
- minimizes conflicts with implementation-specific flags
Can you give me an example?
- is explicit about intent; #+:unix says "I want unix".
#+:tf-unix says "I want a tf-compliant unix".
OTOH, as implementations start to agree with each other, one can simply drop the trivial-features dependency and not have to change any code.
Also, implementations already agree on 80%[1] of this stuff. So you can write #+unix and pull in trivial-features and make it work on that one Lisp that doesn't push :UNIX to features (ECL on Darwin) instead of having to write (or, most likely, forgetting to write) #+(and unix darwin) or #+(or unix (and ecl darwin)).
My rationale is that the above pragmatic advantages outweight the low probability of incompatibilities due to the rudeness of making global changes to *features*. I may be wrong, of course.
This branch adds three new dependencies. Babel for string translation is understandable. Alexandria and trivial-features are somewhat less obvious.
The new string code could be moved into a separate ASDF system with a different type and have only that depend on Babel. The Alexandria utilities being used could be copied and pasted into CFFI and Babel. The endianess feature detection code could be copied and pasted into Babel as well. And the cffi-features stuff can stay as it is, instead of being replaced by trivial-features. I'm not (yet?) convinced it's worth the trouble.
If the core cffi is introducing these dependencies, then I'm checking that we gain something tangible in return.
At first they were introduced by Babel (which can't use cffi-features unless we move CFFI's external-format code onto a different system), then it's just seemed natural to use them in CFFI as well.
[1] made-up figure.
On Tue, 8 Apr 2008, Lu?s Oliveira wrote:
On Tue, Apr 8, 2008 at 6:30 AM, dherring@tentpost.com wrote:
Until we get a CDR/other generally accepted spec on how *features* should reflect the platform, there are no wrong settings in *features*. Since this is a global value, special care must be taken to not break code which has already been tuned for an implementation. It seems rude for a nested library to change common global settings.
Again, I agree with you that it doesn't feel very good at first sight. But, can you give me an example how trivial-features would break existing code? AFAICT, implementations seem to introduce aliases for already existing keywords in *features* from time to time. That is what trivial-features does as well.
Putting a prefix on all customized features
- minimizes conflicts with implementation-specific flags
Can you give me an example?
Unfortunately nothing new. The SBCL :unix on windows is the worst I knew of. It might just be my overactive imagination.
The moral equivalent to *features* in C/C++ are #define macros. There, programmers learned through pain that every library should prefix its macros with a library-specific prefix; the common names are always taken for differen purposes by several libs, leading to bizarre errors depending on header #include order. For example, the Boost libraries define things like BOOST_NO_AUTO_PTR and BOOST_NO_EXCEPTIONS when they detect implementation abnormalities.
In lisp, it would be important that trivial-features always loads before any other library which uses *features*; or else those loading prior might get incompatible settings with those loading after.
Since *features* is generally less-heavily used than #defines, its less of a problem, but I'm still uneasy about this. I guess it just depends on the scope of what you're trying to achieve.
- is explicit about intent; #+:unix says "I want unix".
#+:tf-unix says "I want a tf-compliant unix".
OTOH, as implementations start to agree with each other, one can simply drop the trivial-features dependency and not have to change any code.
Devil's advocate then says, "CFFI doesn't require trivial-features; but if it starts acting funny, you might try loading tf and then recompiling cffi."
If the core cffi is introducing these dependencies, then I'm checking that we gain something tangible in return.
At first they were introduced by Babel (which can't use cffi-features unless we move CFFI's external-format code onto a different system), then it's just seemed natural to use them in CFFI as well.
Ahh. Thanks for the explanation. Babel sounds like a generally good thing, so the others are good by association.
Following the principle of put up or shut up, I withdraw my objection to merging this branch.
Later, Daniel
On Mon, Apr 7, 2008 at 5:34 AM, Daniel Herring dherring@tentpost.com wrote:
In particular, trivial-features bothers me. Libraries probably shouldn't modify the features set by an implementation. I would feel much more comfortable if all names were prefixed to reduce conflict. e.g. set :tf-darwin and :tf-linux instead of :darwin and :linux
In principle, I agree with you and I considered that option you suggest. However, I hope that at some point, most of the files in src/tf-*.lisp become empty as implementations start to agree with each other.
Can you give me a particular example of a feature pushed by trivial-features that you don't like?
#+(and sbcl win32) (setq *features* (remove :unix *features*))
This one *I* was quite unconfortable with, but I sent a patch to SBCL getting rid of :UNIX on windows and it was accepted a few months ago.
Other thoughts: Babel is used for external string support, right? Does it slow down the base, untranslated case? Could it be optional/disabled for those of us who don't want translation?
According to my quick tests, it's faster and conses less, even when using the default external-format (UTF-8) instead of ISO-8859-1 (which is what CFFI had, in effect).
For small strings (<30 chars) I can see some very small speed overhead (probably because of the hashtable lookups and one or two extra function calls). This overhead can be shaved off with a compiler macro, if necessary.
I like the idea of including cffi-grovel. I would very convenient to me if it were included.
The dependencies worry me. I certainly understand the desire to avoid duplication of code, but making dependencies makes it harder to install. For example, none of these three dependencies is present in Debian, yet cl-cffi is. I know you're not the Debian maintainer for cl-cffi, but how would this be managed? Difficulty of installation is a bigger barrier for those new to Lisp, and I expect that CFFI would be important for many newcomers who wish to continue to use software they used in the non-Lisp environment.
Liam
On Mon, Apr 7, 2008 at 3:44 PM, Liam Healy lnp@healy.washington.dc.us wrote:
The dependencies worry me. I certainly understand the desire to avoid duplication of code, but making dependencies makes it harder to install. For example, none of these three dependencies is present in Debian, yet cl-cffi is. I know you're not the Debian maintainer for cl-cffi, but how would this be managed?
Obviously the Debian maintainers will have to package the dependencies as well. That shouldn't be a problem.
Difficulty of installation is a bigger barrier for those new to Lisp, and I expect that CFFI would be important for many newcomers who wish to continue to use software they used in the non-Lisp environment.
If you use clbuild today and install CFFI through it, it will happily pull the cffi+lotsastuff branch along with its 3 dependencies. ASDF-Install would do the same thing. And we could include the dependencies in a tarball linked from the main page if that turned out to be an issue; I doubt it will.
The only issue I was worrying about is release management. Making sure the libraries play well with each other, and so on. That essentially means we shouldn't let so much time go by between releases (0.9.2 was released over a year and a half ago), but I don't control Alexandria releases.
A buildbot would help with that, a lot. Any volunteers? :-)
-
Attila Lendvai -
Daniel Herring -
dherring＠tentpost.com
-
Liam Healy -
Luke Crook -
Luke J Crook
-
Luís Oliveira