Hi folks,
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
The changes are very simple, we allow users to specify a cache directory by using an optional :cache-dir argument in :grovel-file & :wrapper-file forms. CFFI will then look there first for the required files before trying to compile them itself.
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
To handle this I am currently using my with-cached-reader-conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")
The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-file
All the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
My next addition is going to be a function the user can call that will do the following:
- Work out if cffi-grovel has cached files for the grovel and wrapper results - If it didnt have those files it will copy the cachable files to a user specified location (or maybe just the cache directory)
The idea of this is that it will allow the users to easily dontate their cache files to the project which hopefully means we can better coverage quickly
Thoughts would be very welcome, I hope this finds you all well,
Baggers
[0] https://github.com/cbaggers/cffi/tree/feature-grovel-caching [1] https://github.com/cbaggers/with-cached-reader-conditionals
Hello Chris,
On Sun, Aug 21, 2016 at 10:41 PM, Chris Bagley chris.bagley@gmail.com wrote:
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
I didn't review your commits thoroughly but it's looking good!
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
Interesting, that's definitely a difficulty I hadn't anticipated.
The identifier I had in mind was the usual cpu-vendor-os triplet (e.g., i686-pc-linux-gnu). Even if the grovel file contains no reader conditionals whatsoever, it'll depend on this triplet due to varying type sizes across platforms at the very least.
I was thinking there might be an extra dependency here, maybe, which is the foreign library version. But usually bindings will depend on binary compatibility with a specific major version anyway, so perhaps that is a non-issue. If something like that did turn out to be an issue, perhaps we could add some extensible mechanism for users to add extra info to the cache key.
To handle this I am currently using my with-cached-reader-conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
Looks neat. dirty-featurep is not super pretty but it seems like the way to go. (Or we could just warn people that #+ #- will have a possibly different behaviour than expected within grovel files and just use alexandria:featurep.) I'm leaning towards just including it directly into cffi-grovel at this point, but feel free to try and change my mind. :-)
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-file
In Osicat's case, the cpu-vendor-os triplet would do the job on its own, but double-checking which features were used seems sensible nevertheless.
Indeed, my experience with Windows pathnames makes me very cautios of using the filename like that for arbitrary feature names because (a) Windows is picky about what a valid pathname is and (b) it's got a 260 character limit for pathnames.
Perhaps we just need to record the result of each reader conditional, store those boolean results as increasingly significant bits in an integer, then base64-encode that and append it to the cpu-vendor-os triplet? (Then maybe just append that to the grovel file name without a need for a subdirectory.)
All the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
My next addition is going to be a function the user can call that will do the following:
- Work out if cffi-grovel has cached files for the grovel and wrapper results
- If it didnt have those files it will copy the cachable files to a
user specified location (or maybe just the cache directory)
Can we avoid the copying by just writing to and loading from the cache directory unconditionally?
Anyway, it's looking good. I think this has the potential to improve usability for Windows (and perhaps OS X) users quite a bit. :-)
HTH,
My apologies for the delay in answering, but I was a bit busy last quarter.
On Sun, 2016-08-21 at 23:41 +0200, Chris Bagley wrote:
Hi folks,
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
The changes are very simple, we allow users to specify a cache directory by using an optional :cache-dir argument in :grovel-file & :wrapper-file forms. CFFI will then look there first for the required files before trying to compile them itself.
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
To handle this I am currently using my with-cached-reader -conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-fileAll the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
So this patch
1) Duplicates ASDF functionality in incompatible ways - like the choice of a mapping from source file name to build artifact name 2) Undermines ASDF's internal invariants by copying files into the ASDF cache from a secondary cache and creates empty files in the ASDF cache 3) Creates cache keys using one of the most brittle mechanism in Common Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations. 4) Makes things difficult for the creation of a binary package because now that different developers can set different :cache-dir directories in the .asd files they control, a user must review and possibly manually edit all .asd files of its dependencies (think pgloader for example) in order to enforce the use of the same cache directory and to be able to find the build artifacts.
If at all, this code should be part of ASDF as a cache available for all extension types. There are 2 important rules in doing cross-OS development and deployment:
1) Follow the conventions of the host OS, don't fight them 2) Work with your build system
This patch goes against both principles, so as far as I am concerned my vote is a NACK. I'm sorry if you've worked so long on this, but in my opinion this feature should not be a part of CFFI.
My apologies for the delay in answering, but I was a bit busy last quarter.
No problem at all, thanks for such a thorough reply
This patch goes against both principles, so as far as I am concerned my vote is a NACK. I'm sorry if you've worked so long on this, but in my opinion this feature should not be a part of CFFI.
Again no problem, the concept has value to me (even if the current implementation could be improved) and this will exist in some form as a separate library if it doesn't belong here.
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
The answers to these will be longer than the questions I'm afraid. But you are right that I didn't provide enough in the PR. As I have done no rigorous survey of new lisp programmers as large so I can only speak to my experiences an those I has watched get started.
why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler
TLDR: Because I don't believe library consumers should have to change their environment to accommodate an implementation detail of the library's developer.
I make libraries for lisp programmers to use, and naturally I use C libraries to avoid duplicating good work. The people that use my libraries shouldn't have to know the innards of the library they use, they should only *have to* be concerned in how they interact with it and the benefits it gives them.
To me being a developer on some libraries doesn't preclude you from being a consumer of others.
I have seen newcomers who were both excited by lisp and willing to make the emacs switch, drop it because they kept running into C when they were trying to pull a projects they were interested in.
So the reason to provide the option for compiling on with C compilers and not on those without is, I expect, similar to why we don't require people to install c2ffi on every machine that uses a library that was made using it: Namely to let the consumers of the library just be consumers and let the developers of the library handle that.
not why you think this approach is the best way to solve them
I certainly don't want to make that claim. It's one potential solution to an issue and has been provided as a PR for review as I expect it would be improved with more brains on the problem.
You haven't stated what those issues are
The issue is that a library's consumer should only have to be interested in the features on the library, and not have to go making changes to their environment just to use it (caveat: where this can be avoided etc).
I started this feature after finding it a pain to use osicat as neither Windows or OSX ship with a C compiler installed. When the groveler failed the output was very unhelpful, and as I wasn't a developer on the groveler I had no idea what it was or how I should be interacting with it. I was just a consumer who saw value in osicat.
It seemed a shame to me that work, which could be valid on many people's machines (who share the same OS, Architecture, etc), was being duplicated on all of them. Maybe I'm missing the benefit's, but the potential benefits so me sounds like the argument for using gentoo over debian, and whilst I appreciate the benefits gentoo users have given. I still prefer my dependencies shipped compiled where possible.
- Duplicates ASDF functionality in incompatible ways - like the choice
of a mapping from source file name to build artifact name
Cool this sounds fixable, I have read the asdf manual but didn't see any guide on mappings. I copied the style already in place where I could.
If there are duplication of functionality then would you mind listing them? I would MUCH prefer to rely on ASDF features where they exist.
2) Undermines ASDF's internal invariants by copying files into the ASDF cache from a secondary cache and creates empty files in the ASDF cache
Where can I find details on these 'internal invariants'? This sounds like a problem with the currently implementation and not the concept of avoiding duplication of work (caching).
3) Creates cache keys using one of the most brittle mechanism in Common Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations.
The C library loaded is no longer valid. This is an issue without my change though as the wrapper specifications use reader conditionals already. If you change *features* and have already loaded a grovelled library whos contents were dependent on one of the features changed..then the assumption for that library are now invalid.
If this is an issue then I can raise a ticket for cffi-grovel, though I am not sure how to fix it.
- Makes things difficult for the creation of a binary package because
now that different developers can set different :cache-dir directories in the .asd files they control, a user must review and possibly manually edit all .asd files of its dependencies (think pgloader for example) in order to enforce the use of the same cache directory and to be able to find the build artifacts.
If at all, this code should be part of ASDF as a cache available for all extension types. There are 2 important rules in doing cross-OS development and deployment:
- Follow the conventions of the host OS, don't fight them
- Work with your build system
This one is very interesting to me but also has me confused.
I haven't seen anything in ASDF that can package the shared-objects with the generated binary, so (in my case) they would have to be shipped anyway (As OS package managers are, as a rule, out of date). Whether I'm pulling those from some system folder or from some cffi cache folder doesn't seem to make any difference. I'm assuming you aren't suggesting trawling through the asdf cache was the correct was of finding one's compiled dependencies?
:) I expect at this point you are shaking your head and that I have totally missed your point, I'm also have had no need for pgloader so I'm unaware of it's details. I'll ask a question instead.
Why would a user ever be changing the cache directory of a library they don't maintain? If the library is using this feature it's that the developer of that library saw value in shipping the artifacts of the grovel/wrapping.
I really would like to understand this as I attempt to ship games in lisp and dealing with C dependencies from projects using CFFI is by far the single biggest issue of shipping.
Once again, I make no claim that this version of this feature is the final best one, just that such a feature has value. This view was shared by Luis when I approached him online before starting this and is driven in no small part by the difficulties seen around shipping C libs in the lisp-games community.
Thanks go going over this and I really hope I can find some nugget in here that can be of use CFFI.
Warm Regards,
Chris
On 4 October 2016 at 21:35, Stelian Ionescu sionescu@cddr.org wrote:
My apologies for the delay in answering, but I was a bit busy last quarter.
On Sun, 2016-08-21 at 23:41 +0200, Chris Bagley wrote:
Hi folks,
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
The changes are very simple, we allow users to specify a cache directory by using an optional :cache-dir argument in :grovel-file & :wrapper-file forms. CFFI will then look there first for the required files before trying to compile them itself.
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
To handle this I am currently using my with-cached-reader -conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-fileAll the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
So this patch
- Duplicates ASDF functionality in incompatible ways - like the choice
of a mapping from source file name to build artifact name 2) Undermines ASDF's internal invariants by copying files into the ASDF cache from a secondary cache and creates empty files in the ASDF cache 3) Creates cache keys using one of the most brittle mechanism in Common Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations. 4) Makes things difficult for the creation of a binary package because now that different developers can set different :cache-dir directories in the .asd files they control, a user must review and possibly manually edit all .asd files of its dependencies (think pgloader for example) in order to enforce the use of the same cache directory and to be able to find the build artifacts.
If at all, this code should be part of ASDF as a cache available for all extension types. There are 2 important rules in doing cross-OS development and deployment:
- Follow the conventions of the host OS, don't fight them
- Work with your build system
This patch goes against both principles, so as far as I am concerned my vote is a NACK. I'm sorry if you've worked so long on this, but in my opinion this feature should not be a part of CFFI.
-- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur.
On Wed, Oct 5, 2016 at 3:26 AM, Chris Bagley chris.bagley@gmail.com wrote:
I started this feature after finding it a pain to use osicat as neither Windows or OSX ship with a C compiler installed. When the groveler failed the output was very unhelpful, and as I wasn't a developer on the groveler I had no idea what it was or how I should be interacting with it. I was just a consumer who saw value in osicat.
I like to distinguish 4 roles: 1- Author, who writes a given system. 2- Referrer, who uses the system from another system. 3- Integrator, who assembles a bunch of systems into an application. 4- (End) User, who uses the application and shouldn't even have to know it's in Lisp.
Only the Integrator should have to worry about matching together Lisp systems, Lisp implementations, C compilers, etc. For others, it should "just work". By default.
What problem are you trying to solve for people in which role? How is it not already solved by the ASDF cache?
If it's anyone but Integrators, then mucking with implementation identifiers is wrong-headed. If you're trying to write something for Integrators, then look at the ASDF bundle operations as a good start.
And if you're not satisfied with ASDF, may I suggest you give a look at bazelisp? Sure, it only works on Linux amd64 using SBCL at this point, but it might be easier to port for much better results than would be trying to abuse ASDF and try to go against the design of ASDF.
Where can I find details on these 'internal invariants'? This sounds like a problem with the currently implementation and not the concept of avoiding duplication of work (caching).
The invariants of ASDF are mainly that you should be using the ASDF cache and it should just work. If you're not using it the intended way, either: 1) There's a bug in ASDF, so go fix ASDF, or 2) There's a bug in your code, so go fix your code.
I really would like to understand this as I attempt to ship games in lisp and dealing with C dependencies from projects using CFFI is by far the single biggest issue of shipping.
That's an application delivery issue, for Integrators. For that, please use ASDF (monolithic) bundle operations, GDL famously does just that.
See also the cffi-toolchain static-link capabilities, that could use some love. (SBCL notably reverted my patch to generate self-contained static linked binaries, because it was broken on MacOS and Windows.)
Or use Bazel (which patches SBCL).
But please don't introduce towers of complexity that defeat the design of ASDF.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you're a quiet, law-abiding citizen most of the time but occasionally cut someone up and bury them in your backyard, you're a bad guy. — Paul Graham, "The Power of the Marginal"
Hi Faré, thanks for all the answers.
I'm starting to see where the mismatch lies between my thinking and what is needed. However like with Stelian's, this email is excellent and I'd like to go through it fully. I'll do it in an odd order though.
If it's anyone but Integrators, then mucking with implementation identifiers is wrong-headed … you're not using it the intended way, either:
- There's a bug in ASDF, so go fix ASDF, or
- There's a bug in your code, so go fix your code.
…
True, and I'm fine with the idea that my approach was poor, I would love a better one. The static link capabilities did look really interested but SBCL is on an older ASDF (I guess for the reasons you mentioned). Maybe that is part of ASDF I can start reading with the hope I can contribute at some point.
But please don't introduce towers of complexity that defeat the design of ASDF.
I can only ask for your trust that this isnt what I don't want more complexity or to defeat the design on ASDF (even if I achieved both in the current PR). I'm just looking to ship useful things for lispers to use, publicly making these kind of mistakes isn't a problem to me if it gets me closer to making something that is useful and works well with the existing tooling (ASDF being a huge part of that for me).
may I suggest you give a look at bazelisp
Thanks, I'll have a peek. But I really like ASDF and want to be useful on as many lisp implementations as is feasible. So I'd rather stay in this pond (and help if I can). I just need to understand more deeply where certain things should be done.
1- Author, who writes a given system. 2- Referrer, who uses the system from another system. 3- Integrator, who assembles a bunch of systems into an application. 4- (End) User, who uses the application and shouldn't even have to know it's in Lisp.
What problem are you trying to solve for people in which role? How is it not already solved by the ASDF cache?
I'm trying to allow 2 & 3 to consume the lisp library without having to worry about the implementation decisions 1 made (unless they want to).
Let's say I am 1 in this case for example, and that I'm making a GL based UI for people to use. I decide to wrap an existing GL UI system written in C and make a nice lisp interface on top. I would argue that the fact my implementation uses a C library is irrelevant to the consumer of my library. They wanted a GL UI.
Of course this shouldn't stop them from digging into this if they want to, but the standard experience should be as standard a lisp experience as possible. load-system and start being productive.
So my closing question is: If I want to be able to use shared-object files with my lisp libraries what is the correct asdf way for this to be done? Given that: - The shared-object file is not available through any package manager - people should be able to just quickload the project and start working - I need to support osx/linux/windows so I will be including a selection of .so/dll/dylib files. - Requiring every user to have a C compiler set up on their machine is not acceptable.
Note that I don't mind some people having to build it themselves as it's not feasible include precompiled objects for all platforms. But I want to cover a basic subset
Thanks again for the time you are taking to go through this.
Regards Chris
On 6 October 2016 at 01:34, Faré fahree@gmail.com wrote:
On Wed, Oct 5, 2016 at 3:26 AM, Chris Bagley chris.bagley@gmail.com wrote:
I started this feature after finding it a pain to use osicat as neither Windows or OSX ship with a C compiler installed. When the groveler failed the output was very unhelpful, and as I wasn't a developer on the groveler I had no idea what it was or how I should be interacting with it. I was just a consumer who saw value in osicat.
I like to distinguish 4 roles: 1- Author, who writes a given system. 2- Referrer, who uses the system from another system. 3- Integrator, who assembles a bunch of systems into an application. 4- (End) User, who uses the application and shouldn't even have to know it's in Lisp.
Only the Integrator should have to worry about matching together Lisp systems, Lisp implementations, C compilers, etc. For others, it should "just work". By default.
What problem are you trying to solve for people in which role? How is it not already solved by the ASDF cache?
If it's anyone but Integrators, then mucking with implementation identifiers is wrong-headed. If you're trying to write something for Integrators, then look at the ASDF bundle operations as a good start.
And if you're not satisfied with ASDF, may I suggest you give a look at bazelisp? Sure, it only works on Linux amd64 using SBCL at this point, but it might be easier to port for much better results than would be trying to abuse ASDF and try to go against the design of ASDF.
Where can I find details on these 'internal invariants'? This sounds like a problem with the currently implementation and not the concept of avoiding duplication of work (caching).
The invariants of ASDF are mainly that you should be using the ASDF cache and it should just work. If you're not using it the intended way, either:
- There's a bug in ASDF, so go fix ASDF, or
- There's a bug in your code, so go fix your code.
I really would like to understand this as I attempt to ship games in lisp and dealing with C dependencies from projects using CFFI is by far the single biggest issue of shipping.
That's an application delivery issue, for Integrators. For that, please use ASDF (monolithic) bundle operations, GDL famously does just that.
See also the cffi-toolchain static-link capabilities, that could use some love. (SBCL notably reverted my patch to generate self-contained static linked binaries, because it was broken on MacOS and Windows.)
Or use Bazel (which patches SBCL).
But please don't introduce towers of complexity that defeat the design of ASDF.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If you're a quiet, law-abiding citizen most of the time but occasionally cut someone up and bury them in your backyard, you're a bad guy. — Paul Graham, "The Power of the Marginal"
[...]
So my closing question is: If I want to be able to use shared-object files with my lisp libraries what is the correct asdf way for this to be done? Given that:
- The shared-object file is not available through any package manager
- people should be able to just quickload the project and start working
- I need to support osx/linux/windows so I will be including a
selection of .so/dll/dylib files.
- Requiring every user to have a C compiler set up on their machine is
not acceptable.
Requiring those who compile the app to have a C compiler is the reality of today's operating systems, whether you find it palatable or not. If you decide to fight against this you'll probably end up creating more problems than you solve, and for little gain.
Sorry if I missed some bit of info, but:
Am 06.10.2016 um 11:09 schrieb Stelian Ionescu sionescu@cddr.org:
[...]
So my closing question is: If I want to be able to use shared-object files with my lisp libraries what is the correct asdf way for this to be done? Given that:
- The shared-object file is not available through any package manager
- people should be able to just quickload the project and start working
- I need to support osx/linux/windows so I will be including a
selection of .so/dll/dylib files.
- Requiring every user to have a C compiler set up on their machine is
not acceptable.
Requiring those who compile the app to have a C compiler is the reality of today's operating systems, whether you find it palatable or not. If you decide to fight against this you'll probably end up creating more problems than you solve, and for little gain.
Hm - I’d just use dlopen, dlsym (or the like on Windows) to load the shared library and access functions etc. That’s what CFFI does - so why not use it. This doesn’t require a C compiler. You’d need CFFI though, not only ASDF. Or just mimic (read: reimplement) the CFFI functionality.
;; Frank
Indeed, I'm aware now that my implementation needs work (Messing with the asdf cache was certainly a bad move) but I do believe in the value of being able to ship shared objects (package managers still seem to be a thing).
I'll move this to it's own project so it doesn't cause further issues on cffi though.
On 6 October 2016 at 11:28, Frank Goenninger frgo@me.com wrote:
Sorry if I missed some bit of info, but:
Am 06.10.2016 um 11:09 schrieb Stelian Ionescu sionescu@cddr.org:
[...]
So my closing question is: If I want to be able to use shared-object files with my lisp libraries what is the correct asdf way for this to be done? Given that:
- The shared-object file is not available through any package manager
- people should be able to just quickload the project and start working
- I need to support osx/linux/windows so I will be including a
selection of .so/dll/dylib files.
- Requiring every user to have a C compiler set up on their machine is
not acceptable.
Requiring those who compile the app to have a C compiler is the reality of today's operating systems, whether you find it palatable or not. If you decide to fight against this you'll probably end up creating more problems than you solve, and for little gain.
Hm - I’d just use dlopen, dlsym (or the like on Windows) to load the shared library and access functions etc. That’s what CFFI does - so why not use it. This doesn’t require a C compiler. You’d need CFFI though, not only ASDF. Or just mimic (read: reimplement) the CFFI functionality.
;; Frank
On Tue, Oct 4, 2016 at 8:35 PM, Stelian Ionescu sionescu@cddr.org wrote:
You haven't stated what those issues are, not why you think this approach is the best way to solve them nor why do you think it's reasonable for a development machine to have a CL compiler but not a C compiler.
I think it's quite reasonable for a Lisp programmer to minimise interaction with the C toolkit, in general, but the main issue here is that on some operating systems it can be quite a pain to set up said toolkit, particularly on Windows (and to a lesser extent on OS X).
- Undermines ASDF's internal invariants by copying files into the ASDF
cache from a secondary cache and creates empty files in the ASDF cache
I'm not sure what empty files you're referring to, but the general idea is to replace the gcc+execution step with one that simply grabs a pre-cooked lisp file. How does that suddenly undermine any ASDF invariant?
- Creates cache keys using one of the most brittle mechanism in Common
Lisp (*features*). Try thinking what happens during interactive development when a developer changes *features* between compilations.
If the environment changes in such a way that affects a given grovel file, a recompilation will be triggered. Seems like a feature, really. :-)
Do you have an alternative suggestion that plays well with the ASDF/quicklisp environment?
Cheers,
-
Chris Bagley -
Faré -
Frank Goenninger -
Luís Oliveira -
Stelian Ionescu