CFFI govels size_t into :sizet [Was: cffi 0.17.0 breaks some stuff]
I'm seeing a bunch of projects that don't seem to work with the latest CFFI release. For example, I get this for a number of projects:
Unhandled CFFI::UNDEFINED-FOREIGN-TYPE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1003DD5643}>: Unknown CFFI type :SIZET
That happens in gsll and some other projects.
the situation: cffi-libffi used to grovel size_t under the name :sizet, and it's used by a few projects, e.g. gsll, which got broken. i deleted :sizet in this cleanup: https://github.com/cffi/cffi/commit/4fbe5864552d6f7d8866745f371346174acea942 i've reinstated it temporarily with an OBSOLETE warning just now. reasons for the removal: - for me it seems pretty ad-hoc why this one definition is included and the other C stdlib definitions are not. - if we want to include and support size_t in the CFFI contract, then why in cffi-libffi? keep in mind that it requires groveling, which is quite a heavy dependency that CFFI proper doesn't require at this point. - why in the keyword package? (and it's another discussion why :int and other standard C definitions (not to be confused with stdlib.h) are in the keyword package, but i won't pursuit that argument at this point in time) maybe we want to open a new ASDF system for the C stdlib that would depend on the groveler and accommodate for C stdlib definitions like errno, size_t, etc? -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Nationalism of one kind or another was the cause of most of the genocide of the twentieth century. Flags are bits of colored cloth that governments use first to shrink-wrap people's brains and then as ceremonial shrouds to bury the dead.” — Arundhati Roy (1961–), 'Come September' speech
On Wed, Feb 24, 2016 at 7:10 PM, Attila Lendvai <attila@lendvai.name> wrote:
- for me it seems pretty ad-hoc why this one definition is included and the other C stdlib definitions are not.
- if we want to include and support size_t in the CFFI contract, then why in cffi-libffi? keep in mind that it requires groveling, which is quite a heavy dependency that CFFI proper doesn't require at this point.
I suppose it was convenient for Liam. I agree it doesn't belong there. Liam, is it too much of a hassle to move the size_t grovelling to gsll? (If it is, maybe we can figure out something else.)
- why in the keyword package? (and it's another discussion why :int and other standard C definitions (not to be confused with stdlib.h) are in the keyword package, but i won't pursuit that argument at this point in time)
Re the standard CFFI types: convenience.
maybe we want to open a new ASDF system for the C stdlib that would depend on the groveler and accommodate for C stdlib definitions like errno, size_t, etc?
Maybe. Osicat sort of plays that role right now. It has size_t, and it has a bunch of errno functionality. (I don't remember if those are exported or anything.) Should I make a bugfix release for Zach to test? -- Luís Oliveira http://kerno.org/~luis/
On Wed, Feb 24, 2016 at 7:40 PM, Zach Beane <xach@xach.com> wrote:
I can test master temporarily.
Does current master still have any Quicklisp issues? -- Luís Oliveira http://kerno.org/~luis/
Luís Oliveira <luismbo@gmail.com> writes:
On Wed, Feb 24, 2016 at 7:40 PM, Zach Beane <xach@xach.com> wrote:
I can test master temporarily.
Does current master still have any Quicklisp issues?
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition. Other than that, things look fine. Zach
On Thu, Mar 3, 2016 at 12:44 PM, Zach Beane <xach@xach.com> wrote:
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition.
That's unfortunate. Is there some static thing I can place in the version property that means "this is bleeding edge"? -- Luís Oliveira http://kerno.org/~luis/
Luís Oliveira <luismbo@gmail.com> writes:
On Thu, Mar 3, 2016 at 12:44 PM, Zach Beane <xach@xach.com> wrote:
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition.
That's unfortunate. Is there some static thing I can place in the version property that means "this is bleeding edge"?
I don't know of an option for that. I'd guess using some higher version number with a distinguishing suffix might help? ASDF is a stickler about its version formatting. Zach
On 3/3/16 Mar 3 -9:18 AM, Zach Beane wrote:
Luís Oliveira <luismbo@gmail.com> writes:
On Thu, Mar 3, 2016 at 12:44 PM, Zach Beane <xach@xach.com> wrote:
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition.
That's unfortunate. Is there some static thing I can place in the version property that means "this is bleeding edge"?
I don't know of an option for that. I'd guess using some higher version number with a distinguishing suffix might help? ASDF is a stickler about its version formatting.
Zach
I fear that will not do what you want:
(asdf:version-satisfies "3.1.2.rc" "3.1.2") NIL
The RC there for some reason is breaking the match: R> (asdf:version-satisfies "3.1.2" "3.1.2") T This smells to me like a bug in ASDF:VERSION-SATISFIES, but we won't be able to get a fix into ASDF in time to let this work for QL + CFFI. Is there some reason you can't just bump x.y.z to x.y.z.xx :version in your cffi.asd file and push that? Cheers, r
On 3/3/16 Mar 3 -9:18 AM, Zach Beane wrote:
Luís Oliveira <luismbo@gmail.com> writes:
On Thu, Mar 3, 2016 at 12:44 PM, Zach Beane <xach@xach.com> wrote:
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition.
That's unfortunate. Is there some static thing I can place in the version property that means "this is bleeding edge"?
I don't know of an option for that. I'd guess using some higher version number with a distinguishing suffix might help? ASDF is a stickler about its version formatting.
Zach
I fear that will not do what you want:
(asdf:version-satisfies "3.1.2.rc" "3.1.2") NIL
The RC there for some reason is breaking the match:
R> (asdf:version-satisfies "3.1.2" "3.1.2") T
asdf:version-satisfies expects only numbers, which sounds pretty reasonable to me. S> (asdf:version-satisfies "3.1.2.990" "3.1.2") T -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
"Robert P. Goldman" <rpgoldman@sift.info> writes:
On 3/3/16 Mar 3 -9:18 AM, Zach Beane wrote:
Luís Oliveira <luismbo@gmail.com> writes:
On Thu, Mar 3, 2016 at 12:44 PM, Zach Beane <xach@xach.com> wrote:
Nothing related to the code, but I found that a couple projects will not build because they specify versioned dependencies on CFFI, and master does not have a version in its system definition.
That's unfortunate. Is there some static thing I can place in the version property that means "this is bleeding edge"?
I don't know of an option for that. I'd guess using some higher version number with a distinguishing suffix might help? ASDF is a stickler about its version formatting.
Zach
I fear that will not do what you want:
(asdf:version-satisfies "3.1.2.rc" "3.1.2") NIL
The RC there for some reason is breaking the match:
R> (asdf:version-satisfies "3.1.2" "3.1.2") T
This smells to me like a bug in ASDF:VERSION-SATISFIES, but we won't be able to get a fix into ASDF in time to let this work for QL + CFFI.
Is there some reason you can't just bump x.y.z to x.y.z.xx :version in your cffi.asd file and push that?
Right - I mean a suffix like ".999" or something, not letters. Zach
Note that if one really doesn't like what VERSION-SATISFIES does, it's a method. You could specialize it on your own component (possibly even EQL-specify it on the CFFI system, or make a trivial subclass of SYSTEM for CFFI). Then, if you wanted to add, e.g., "rc", "alpha", or "trial", you could do that. Cheers, r
On 3/3/16 Mar 3 -10:21 AM, Robert P. Goldman wrote:
Note that if one really doesn't like what VERSION-SATISFIES does, it's a method. You could specialize it on your own component (possibly even EQL-specify it on the CFFI system, or make a trivial subclass of SYSTEM for CFFI).
Then, if you wanted to add, e.g., "rc", "alpha", or "trial", you could do that.
Cheers, r
Groan. It's a generic function, not a method. But you knew what I meant, right?
On Thu, Mar 3, 2016 at 4:03 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
I fear that will not do what you want:
(asdf:version-satisfies "3.1.2.rc" "3.1.2") NIL
The RC there for some reason is breaking the match:
R> (asdf:version-satisfies "3.1.2" "3.1.2") T
This smells to me like a bug in ASDF:VERSION-SATISFIES, but we won't be able to get a fix into ASDF in time to let this work for QL + CFFI.
Using "999" or something like that might be distinguishing enough that we don't need a suffix. Allowing suffixes of some sort sounds useful, but we don't need it. (Meanwhile, I've read your other replies.) (defmethod asdf:version-satisfies ((c (eql (asdf:find-system :cffi))) version) (declare (ignore version)) (or (null (asdf:component-version c)) (call-next-method))) How unwise is this? :-)
Is there some reason you can't just bump x.y.z to x.y.z.xx :version in your cffi.asd file and push that?
I don't like committing version bumps, so CFFI's current setup creates a tag and sets the system version on the release tarball only. -- Luís Oliveira http://kerno.org/~luis/
On 3/3/16 Mar 3 -10:28 AM, Luís Oliveira wrote:
On Thu, Mar 3, 2016 at 4:03 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
I fear that will not do what you want:
(asdf:version-satisfies "3.1.2.rc" "3.1.2") NIL
The RC there for some reason is breaking the match:
R> (asdf:version-satisfies "3.1.2" "3.1.2") T
This smells to me like a bug in ASDF:VERSION-SATISFIES, but we won't be able to get a fix into ASDF in time to let this work for QL + CFFI.
Using "999" or something like that might be distinguishing enough that we don't need a suffix. Allowing suffixes of some sort sounds useful, but we don't need it.
(Meanwhile, I've read your other replies.)
(defmethod asdf:version-satisfies ((c (eql (asdf:find-system :cffi))) version) (declare (ignore version)) (or (null (asdf:component-version c)) (call-next-method)))
How unwise is this? :-)
Actually, that sounds entirely reasonable. I'm not sure the IGNORE will work everywhere: you might change it to IGNORABLE. And add a comment! ;-)
Is there some reason you can't just bump x.y.z to x.y.z.xx :version in your cffi.asd file and push that?
I don't like committing version bumps, so CFFI's current setup creates a tag and sets the system version on the release tarball only.
On Thu, Mar 3, 2016 at 4:46 PM, Robert P. Goldman <rpgoldman@sift.info> wrote:
Actually, that sounds entirely reasonable. I'm not sure the IGNORE will work everywhere: you might change it to IGNORABLE. And add a comment! ;-)
Thanks for the code review. :-) Pushed <https://github.com/cffi/cffi/commit/e10bbcb688a50b8668237e908746dcc9f68913c4>. -- Luís Oliveira http://kerno.org/~luis/
On Wed, Feb 24, 2016 at 2:21 PM, Luís Oliveira <luismbo@gmail.com> wrote:
On Wed, Feb 24, 2016 at 7:10 PM, Attila Lendvai <attila@lendvai.name> wrote:
- for me it seems pretty ad-hoc why this one definition is included and the other C stdlib definitions are not.
- if we want to include and support size_t in the CFFI contract, then why in cffi-libffi? keep in mind that it requires groveling, which is quite a heavy dependency that CFFI proper doesn't require at this point.
I suppose it was convenient for Liam. I agree it doesn't belong there. Liam, is it too much of a hassle to move the size_t grovelling to gsll? (If it is, maybe we can figure out something else.)
It was in GSLL originally but I moved it to CFFI-libffi because I thought others would need it; why force people to reinvent it over and over; isn't that the point of having a library? The discussion we had on #lisp seems to bear me out on this. I agree with the grovel issue, and putting it in cffi-libffi was just a hack to get grovel, so maybe the right place for it is a new system like Attila suggests. I'm happy with whatever we end up calling the symbol, I just need to do a global search replace in GSLL (and so would the other systems that use it), so we need to stall a bit and sync up with Zach in the end. As Attila point out on #lisp, there are possibly other symbols that could go there, so it might well be that people will want to add on to it. I don't think it should go back into GSLL for the reason I stated.
- why in the keyword package? (and it's another discussion why :int and other standard C definitions (not to be confused with stdlib.h) are in the keyword package, but i won't pursuit that argument at this point in time)
Re the standard CFFI types: convenience.
maybe we want to open a new ASDF system for the C stdlib that would depend on the groveler and accommodate for C stdlib definitions like errno, size_t, etc?
Maybe. Osicat sort of plays that role right now. It has size_t, and it has a bunch of errno functionality. (I don't remember if those are exported or anything.)
Interesting, I didn't know that. I had a dependence (for filenames, I think) on osicat and I was convinced that UIOP is now the way to go, so I switched over and removed the osicat dependence. (This may not have been GSLL, but in one of my systems.) Do you think that the osicat maintainer would be willing to "contribute" the stdlib definitions to the proposed new system? I am fairly neutral on the keyword vs other package issue. I chose the keyword package because to me, size_t is like a built in, so it looks a little cleaner. But I don't really care, as I said, it's really matter of global search replace once we decide on a course of action. And sizet is used *a lot* in GSLL, so there are a lot of replaces to do.
Should I make a bugfix release for Zach to test?
-- Luís Oliveira http://kerno.org/~luis/
I think the temporary reversion is enough to get us through the moment, then once we've decided on the final form, all the changes should be made at the same time so the next QL update everything is consistent. Over the next couple days I will have no time to work on this, but on the weekend I should get some time. Thanks, Liam
Do you think that the osicat maintainer would be willing to "contribute" the stdlib definitions to the proposed new system?
well, i always forget about osicat, but osicat itself *is* supposed to be that system i was pondering about, except that it's also kinda posixly confused... (in-package #:osicat-posix) (ctype size "size_t") (ctype ssize "ssize_t") there should be a system for C stdlib.h, stddef.h, etc stuff, which has nothing to do with the underlying OS. of course this system will require groveling and thus on windows it will be a pain to use, but what isn't a pain on windows...? i'll look into setting up the skeleton of such a new CFFI system. my proposed name is CFFI/STDLIB, both for the asdf system and for its package, but suggestions are welcome. details on the header files: https://en.wikipedia.org/wiki/C_standard_library#Header_files which brings up another question that i can't really decide from a distance... should this new system be based on CFFI/C2FFI? or should it use hand written grovel files? it needs groveling for definitions that are implemented as C macros, and the C standard is very liberal about allowing random stuff to be a macro, without a corresponding dlsym... which suggest that a hand written grovel file is the way to go, but then c2ffi can emit stuff that doesn't require the groveller, and is also automated. plus i've been planning to extend it to also emit a grovel file to accommodate for stuff that need groveling (e.g. inline functions if nothing else). oh well... any insights? which side of this fork will lead to less pain? -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Children are not the people of tomorrow, but are people of today. They have a right to be taken seriously, and to be treated with tenderness and respect. They should be allowed to grow into whoever they were meant to be. 'The unknown person' inside of them is our hope for the future.” — Janusz Korczak (1879–1942), together with his 192 orphans, he was murdered by lost souls poisoned with nazi ideology -- or maybe just following orders.
Do you think that the osicat maintainer would be willing to "contribute" the stdlib definitions to the proposed new system?
well, i always forget about osicat, but osicat itself *is* supposed to be that system i was pondering about, except that it's also kinda posixly confused...
(in-package #:osicat-posix)
(ctype size "size_t") (ctype ssize "ssize_t")
there should be a system for C stdlib.h, stddef.h, etc stuff, which has nothing to do with the underlying OS. of course this system will require groveling and thus on windows it will be a pain to use, but what isn't a pain on windows...?
i'll look into setting up the skeleton of such a new CFFI system. my proposed name is CFFI/STDLIB, both for the asdf system and for its package, but suggestions are welcome.
details on the header files:
https://en.wikipedia.org/wiki/C_standard_library#Header_files
I've often thought of doing something like this, especially since given a platform, the ABI is well-known. The hard part is determining the platform of the host Lisp, i.e. http://wiki.osdev.org/Target_Triplet. If we can determine that, we could have a system consisting of hard-coded values, one platform per file, and have CFFI load that at runtime. It could replace not only the grovelling done in Osicat and IOlib for POSIX types, but also the use of the host FFI's variable-sized C types, e.g. instead of CFFI's :long -> sb-alien:long, we'd have :long -> :int32 -> host 32bit signed integer. Of course, we can add a hard compile-time dependency on gcc/clang and run cc -dumpmachine in a macro. I would totally do this, some users might not like it though. OTOH, it would be a quick hack that works 100% and can only be improved in the future. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
Do you think that the osicat maintainer would be willing to "contribute" the stdlib definitions to the proposed new system?
well, i always forget about osicat, but osicat itself *is* supposed to be that system i was pondering about, except that it's also kinda posixly confused...
(in-package #:osicat-posix)
(ctype size "size_t") (ctype ssize "ssize_t")
there should be a system for C stdlib.h, stddef.h, etc stuff, which has nothing to do with the underlying OS. of course this system will require groveling and thus on windows it will be a pain to use, but what isn't a pain on windows...?
i'll look into setting up the skeleton of such a new CFFI system. my proposed name is CFFI/STDLIB, both for the asdf system and for its package, but suggestions are welcome.
details on the header files:
https://en.wikipedia.org/wiki/C_standard_library#Header_files
I've often thought of doing something like this, especially since given a platform, the ABI is well-known. The hard part is determining the platform of the host Lisp, i.e. http://wiki.osdev.org/Target_Triplet. If we can determine that, we could have a system consisting of hard-coded values, one platform per file, and have CFFI load that at runtime. It could replace not only the grovelling done in Osicat and IOlib for POSIX types, but also the use of the host FFI's variable-sized C types, e.g. instead of CFFI's :long -> sb-alien:long, we'd have :long -> :int32 -> host 32bit signed integer.
And this hard-coded file would not have to be generated manually. It can be generated by c2ffi or a similar dedicated program(but for bootstrapping reasons it should probably written in C++), and every time somebody gets an error that their host platform is not supported, tell them to compile and run that program and send us the resulting file to include in the repository. We can easily take care ourselves of x86-pc-linux-gnu, x86_64-pc-linux-gnu, x86_64-pc-freebsd10, x86_64-apple-darwin and x86_64-pc-win32. That should cover 99% of our users. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
I've often thought of doing something like this, especially since given a platform, the ABI is well-known. The hard part is determining the platform of the host Lisp, i.e. http://wiki.osdev.org/Target_Triplet.
FTR, the equivalent of this in cffi/c2ffi is here: https://github.com/cffi/cffi/blob/master/src/c2ffi/c2ffi.lisp it also includes an final extra 'environment' part, probably LLVM wants that, too. yes, it's a kludge, but i think it's worth massaging it if the alternative is a hard dependency on a working CC.
And this hard-coded file would not have to be generated manually. It can be generated by c2ffi or a similar dedicated program(but for bootstrapping reasons it should probably written in C++),
you mean here situations where the new platform has no LLVM/clang support? c2ffi seems to be able to instruct clang to process the header files as if they were on any of its supported platforms, regardless of the host where it's currently running. i don't know how reliable that is, though. i'd assume it's reliable, it's not science fiction to solve this reliably.
and every time somebody gets an error that their host platform is not supported, tell them to compile and run that program and send us the resulting file to include in the repository. We can easily take care ourselves of x86-pc-linux-gnu, x86_64-pc-linux-gnu, x86_64-pc-freebsd10, x86_64-apple-darwin and x86_64-pc-win32. That should cover 99% of our users.
if i understand you correctly here, this is pretty much what cffi/c2ffi does. with the difference that instead of running a hand written C++ code, it just instructs c2ffi to tell LLVM to emit a json database of the contents of the requested header files on the new platform. this can happen on any host running c2ffi. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “A state of war only serves as an excuse for domestic tyranny.” — Aleksandr Solzhenitsyn (1918–2008), 'The Gulag Archipelago' (1973)
I've often thought of doing something like this, especially since given a platform, the ABI is well-known. The hard part is determining the platform of the host Lisp, i.e. http://wiki.osdev.org/Target_Triplet.
FTR, the equivalent of this in cffi/c2ffi is here:
https://github.com/cffi/cffi/blob/master/src/c2ffi/c2ffi.lisp
it also includes an final extra 'environment' part, probably LLVM wants that, too. yes, it's a kludge, but i think it's worth massaging it if the alternative is a hard dependency on a working CC.
And this hard-coded file would not have to be generated manually. It can be generated by c2ffi or a similar dedicated program(but for bootstrapping reasons it should probably written in C++),
you mean here situations where the new platform has no LLVM/clang support? c2ffi seems to be able to instruct clang to process the header files as if they were on any of its supported platforms, regardless of the host where it's currently running. i don't know how reliable that is, though. i'd assume it's reliable, it's not science fiction to solve this reliably.
Ok, I read the code and (should have done that before), and it seems that it doesn't use CFFI so it should be workable. The only thing missing is the ability to filter definitions(see the last reply).
and every time somebody gets an error that their host platform is not supported, tell them to compile and run that program and send us the resulting file to include in the repository. We can easily take care ourselves of x86-pc-linux-gnu, x86_64-pc-linux-gnu, x86_64-pc-freebsd10, x86_64-apple-darwin and x86_64-pc-win32. That should cover 99% of our users.
if i understand you correctly here, this is pretty much what cffi/c2ffi does. with the difference that instead of running a hand written C++ code, it just instructs c2ffi to tell LLVM to emit a json database of the contents of the requested header files on the new platform. this can happen on any host running c2ffi.
The issue with that is that system headers contain a lot of internal auxilliary definitions that can change from one version to the next of the libc, so we want to output only the standard types and nothing else, therefore dumping everything is not an option. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
The only thing missing is the ability to filter definitions(see the last reply).
[not sure what you mean by last reply] c2ffi itself cannot filter, but the cffi/c2ffi generator, that generates the lisp forms from the json database, does have extensive filtering, both blacklisting and whitelisting modes. -- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “The world is merely a kabuki pantomime of early childhood.” — Stefan Molyneux (1966–)
participants (6)
-
Attila Lendvai -
Liam Healy -
Luís Oliveira -
Robert P. Goldman -
Stelian Ionescu -
Zach Beane