My apologies for the delay in answering, but I was a bit busy last quarter.
No problem at all, thanks for such a thorough reply
This patch goes against both principles, so as far as I am concerned my vote is a NACK. I'm sorry if you've worked so long on this, but in my opinion this feature should not be a part of CFFI.
Again no problem, the concept has value to me (even if the current implementation could be improved) and this will exist in some form as a separate library if it doesn't belong here.
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
The answers to these will be longer than the questions I'm afraid. But you are right that I didn't provide enough in the PR. As I have done no rigorous survey of new lisp programmers as large so I can only speak to my experiences an those I has watched get started.
why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler
TLDR: Because I don't believe library consumers should have to change their environment to accommodate an implementation detail of the library's developer.
I make libraries for lisp programmers to use, and naturally I use C libraries to avoid duplicating good work. The people that use my libraries shouldn't have to know the innards of the library they use, they should only *have to* be concerned in how they interact with it and the benefits it gives them.
To me being a developer on some libraries doesn't preclude you from being a consumer of others.
I have seen newcomers who were both excited by lisp and willing to make the emacs switch, drop it because they kept running into C when they were trying to pull a projects they were interested in.
So the reason to provide the option for compiling on with C compilers and not on those without is, I expect, similar to why we don't require people to install c2ffi on every machine that uses a library that was made using it: Namely to let the consumers of the library just be consumers and let the developers of the library handle that.
not why you think this approach is the best way to solve them
I certainly don't want to make that claim. It's one potential solution to an issue and has been provided as a PR for review as I expect it would be improved with more brains on the problem.
You haven't stated what those issues are
The issue is that a library's consumer should only have to be interested in the features on the library, and not have to go making changes to their environment just to use it (caveat: where this can be avoided etc).
I started this feature after finding it a pain to use osicat as neither Windows or OSX ship with a C compiler installed. When the groveler failed the output was very unhelpful, and as I wasn't a developer on the groveler I had no idea what it was or how I should be interacting with it. I was just a consumer who saw value in osicat.
It seemed a shame to me that work, which could be valid on many people's machines (who share the same OS, Architecture, etc), was being duplicated on all of them. Maybe I'm missing the benefit's, but the potential benefits so me sounds like the argument for using gentoo over debian, and whilst I appreciate the benefits gentoo users have given. I still prefer my dependencies shipped compiled where possible.
- Duplicates ASDF functionality in incompatible ways - like the choice
of a mapping from source file name to build artifact name
Cool this sounds fixable, I have read the asdf manual but didn't see any guide on mappings. I copied the style already in place where I could.
If there are duplication of functionality then would you mind listing them? I would MUCH prefer to rely on ASDF features where they exist.
2) Undermines ASDF's internal invariants by copying files into the ASDF cache from a secondary cache and creates empty files in the ASDF cache
Where can I find details on these 'internal invariants'? This sounds like a problem with the currently implementation and not the concept of avoiding duplication of work (caching).
3) Creates cache keys using one of the most brittle mechanism in Common Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations.
The C library loaded is no longer valid. This is an issue without my change though as the wrapper specifications use reader conditionals already. If you change *features* and have already loaded a grovelled library whos contents were dependent on one of the features changed..then the assumption for that library are now invalid.
If this is an issue then I can raise a ticket for cffi-grovel, though I am not sure how to fix it.
- Makes things difficult for the creation of a binary package because
now that different developers can set different :cache-dir directories in the .asd files they control, a user must review and possibly manually edit all .asd files of its dependencies (think pgloader for example) in order to enforce the use of the same cache directory and to be able to find the build artifacts.
If at all, this code should be part of ASDF as a cache available for all extension types. There are 2 important rules in doing cross-OS development and deployment:
- Follow the conventions of the host OS, don't fight them
- Work with your build system
This one is very interesting to me but also has me confused.
I haven't seen anything in ASDF that can package the shared-objects with the generated binary, so (in my case) they would have to be shipped anyway (As OS package managers are, as a rule, out of date). Whether I'm pulling those from some system folder or from some cffi cache folder doesn't seem to make any difference. I'm assuming you aren't suggesting trawling through the asdf cache was the correct was of finding one's compiled dependencies?
:) I expect at this point you are shaking your head and that I have totally missed your point, I'm also have had no need for pgloader so I'm unaware of it's details. I'll ask a question instead.
Why would a user ever be changing the cache directory of a library they don't maintain? If the library is using this feature it's that the developer of that library saw value in shipping the artifacts of the grovel/wrapping.
I really would like to understand this as I attempt to ship games in lisp and dealing with C dependencies from projects using CFFI is by far the single biggest issue of shipping.
Once again, I make no claim that this version of this feature is the final best one, just that such a feature has value. This view was shared by Luis when I approached him online before starting this and is driven in no small part by the difficulties seen around shipping C libs in the lisp-games community.
Thanks go going over this and I really hope I can find some nugget in here that can be of use CFFI.
Warm Regards,
Chris
On 4 October 2016 at 21:35, Stelian Ionescu sionescu@cddr.org wrote:
My apologies for the delay in answering, but I was a bit busy last quarter.
On Sun, 2016-08-21 at 23:41 +0200, Chris Bagley wrote:
Hi folks,
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
The changes are very simple, we allow users to specify a cache directory by using an optional :cache-dir argument in :grovel-file & :wrapper-file forms. CFFI will then look there first for the required files before trying to compile them itself.
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
To handle this I am currently using my with-cached-reader -conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-fileAll the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
So this patch
- Duplicates ASDF functionality in incompatible ways - like the choice
of a mapping from source file name to build artifact name 2) Undermines ASDF's internal invariants by copying files into the ASDF cache from a secondary cache and creates empty files in the ASDF cache 3) Creates cache keys using one of the most brittle mechanism in Common Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations. 4) Makes things difficult for the creation of a binary package because now that different developers can set different :cache-dir directories in the .asd files they control, a user must review and possibly manually edit all .asd files of its dependencies (think pgloader for example) in order to enforce the use of the same cache directory and to be able to find the build artifacts.
If at all, this code should be part of ASDF as a cache available for all extension types. There are 2 important rules in doing cross-OS development and deployment:
- Follow the conventions of the host OS, don't fight them
- Work with your build system
This patch goes against both principles, so as far as I am concerned my vote is a NACK. I'm sorry if you've worked so long on this, but in my opinion this feature should not be a part of CFFI.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.