Hello Chris,
On Sun, Aug 21, 2016 at 10:41 PM, Chris Bagley chris.bagley@gmail.com wrote:
I'm currently working on adding caching of .so and wrapper files to cffi. This is to handle issues arising from trying to use various libraries on machines without a C compiler. I'm hoping you can take a peek at my branch [0] and see if I'm going in the right direction.
I didn't review your commits thoroughly but it's looking good!
The tricky bit is that we recommend that people use the #+ & #- reader conditionals in the grovel and wrapper specifications, which means the cached results are only applicable if all the required feature-forms match.
Interesting, that's definitely a difficulty I hadn't anticipated.
The identifier I had in mind was the usual cpu-vendor-os triplet (e.g., i686-pc-linux-gnu). Even if the grovel file contains no reader conditionals whatsoever, it'll depend on this triplet due to varying type sizes across platforms at the very least.
I was thinking there might be an extra dependency here, maybe, which is the foreign library version. But usually bindings will depend on binary compatibility with a specific major version anyway, so perhaps that is a non-issue. If something like that did turn out to be an issue, perhaps we could add some extensible mechanism for users to add extra info to the cache key.
To handle this I am currently using my with-cached-reader-conditionals library [1] which, whilst small and standalone, could be replicated inside CFFI if required. The library modifies the readtable so that feature expression still work as before, but the also record the feature-forms used.
Looks neat. dirty-featurep is not super pretty but it seems like the way to go. (Or we could just warn people that #+ #- will have a possibly different behaviour than expected within grovel files and just use alexandria:featurep.) I'm leaning towards just including it directly into cffi-grovel at this point, but feel free to try and change my mind. :-)
With the captured feature information we can make a subdirectory inside the :cache-dir directory, whose name is based on the feature information. In my current experiment it does something simple & easy to read, however it would likely make directory names too long for windows users.
For example I took osicat and added a cache directory to the following lines:
(:grovel-file "basic-unixint" :cache-dir "foo") (:wrapper-file "wrappers" :soname "libosicat" :cache-dir "foo")The "foo" directory has the following contents
bsd_nil_darwin_nil_freebsd_nil_linux_t_openbsd_nil unixint.processed-grovel-file darwin_nil_linux_t_mips_nil_openbsd_nil_windows_nil basic-unixint.processed-grovel-file linux_t_windows_nil libosicat.so wrappers.processed-wrapper-file
In Osicat's case, the cpu-vendor-os triplet would do the job on its own, but double-checking which features were used seems sensible nevertheless.
Indeed, my experience with Windows pathnames makes me very cautios of using the filename like that for arbitrary feature names because (a) Windows is picky about what a valid pathname is and (b) it's got a 260 character limit for pathnames.
Perhaps we just need to record the result of each reader conditional, store those boolean results as increasingly significant bits in an integer, then base64-encode that and append it to the cpu-vendor-os triplet? (Then maybe just append that to the grovel file name without a need for a subdirectory.)
All the system does is copy these files to the same directory the system would have put the compiled results in the standard system.
My next addition is going to be a function the user can call that will do the following:
- Work out if cffi-grovel has cached files for the grovel and wrapper results
- If it didnt have those files it will copy the cachable files to a
user specified location (or maybe just the cache directory)
Can we avoid the copying by just writing to and loading from the cache directory unconditionally?
Anyway, it's looking good. I think this has the potential to improve usability for Windows (and perhaps OS X) users quite a bit. :-)
HTH,