if you issue the following (available in quicklisp):
(asdf:load-system :hu.dwim.zlib)
then for the first time it should generate a lisp file, which then gets compiled and loaded.
issuing it for the second time shouldn't do anything, but since some revisions it regenerates the lisp file every time.
the relevant code is somewhere around here:
https://github.com/cffi/cffi/blob/master/src/c2ffi/asdf.lisp#L135
i tried to trace the INPUT-FILES, OUTPUT-FILES, PERFORM, OPERATION-DONE-P methods, but i don't see anything wrong.
one unusual thing is that the output of GENERATE-LISP-OP goes into the src/ directory, not to the usual fasl output dir. same applies to GENERATE-SPEC-OP.
any ideas how to debug this? or what to look at?
On Tue, Jan 30, 2018 at 5:01 PM, Attila Lendvai attila@lendvai.name wrote:
if you issue the following (available in quicklisp):
(asdf:load-system :hu.dwim.zlib)
then for the first time it should generate a lisp file, which then gets compiled and loaded.
issuing it for the second time shouldn't do anything, but since some revisions it regenerates the lisp file every time.
the relevant code is somewhere around here:
https://github.com/cffi/cffi/blob/master/src/c2ffi/asdf.lisp#L135
i tried to trace the INPUT-FILES, OUTPUT-FILES, PERFORM, OPERATION-DONE-P methods, but i don't see anything wrong.
one unusual thing is that the output of GENERATE-LISP-OP goes into the src/ directory, not to the usual fasl output dir. same applies to GENERATE-SPEC-OP.
any ideas how to debug this? or what to look at?
I can reproduce the issue using asdf 3.3.1.3 from master (the tag hadn't been pushed, so I pushed it, but the code was already there).
(plan-actions (nth-value 1 (operate 'load-op :hu.dwim.zlib)))
starts with: ((#<CFFI/C2FFI::GENERATE-SPEC-OP > . #<CFFI/C2FFI:C2FFI-FILE "hu.dwim.zlib" "c2ffi-spec" "zlib.h">) (#<CFFI/C2FFI::GENERATE-LISP-OP > . #<CFFI/C2FFI:C2FFI-FILE "hu.dwim.zlib" "c2ffi-spec" "zlib.h">) (#<COMPILE-OP > . #<CFFI/C2FFI:C2FFI-FILE "hu.dwim.zlib" "c2ffi-spec" "zlib.h">) (#<LOAD-OP > . #<CFFI/C2FFI:C2FFI-FILE "hu.dwim.zlib" "c2ffi-spec" "zlib.h">)
So somehow the first action above is considered not done. I suppose it's a bug in c2ffi.
Looking at the c2ffi source, I don't see anything obviously wrong, though you shouldn't the (the (not null) (find-system ...)) since find-system will error out for you if the system is not found (to avoid error, add optional argument nil, as in find-class).
My theory is that that you're failing to re-generate the .spec when it's present, even though your dependencies say it will be re-generated. Therefore, ASDF decides that it's out-of-date and must be re-generated again the next time over, etc. In other words, you lie to ASDF, and ASDF punishes you right back.
To make ASDF happy, you might instead want to have two disjoint operation modes, one to generate the spec files, and one to use it. Normal use mode wouldn't try to generate the file, but instead bork with a useful error message telling you how to generate it and submit it upstream.
Another strategy to make ASDF happy would be to detect whether the file exists as part of input-files and output-files. If it exists, it's in input-files and not output-files; if not, it's in output-files and not input-files. There's something I don't like about this strategy, but it might work (somehow I can't swear it won't bite you back somehow at some point).
Good luck!
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org There is only one thing more harmful to society than an elected official forgetting the promises he made in order to get elected; that's when he doesn't forget them. — John McCarthy
My theory is that that you're failing to re-generate the .spec when it's present, even though your dependencies say it will be re-generated. Therefore, ASDF decides that it's out-of-date and must be re-generated again the next time over, etc. In other words, you lie to ASDF, and ASDF punishes you right back.
that was indeed a lurking bug. excellent remote debugging skills, thanks! :)
i pushed the fix. now after i touch'd the appropriate spec files, it doesn't want to run c2ffi anymore, but it's still regenerating the lisp file unconditionally.
i ran out of hacking steam for today. i'll look into it tomorrow, but further remote debugging magic is welcome of course... :)
i'm kinda lost for now. i have no plan of action, so to say... :)
One solution is to create a new file with the correct timestamp, that is either a copy of the existing spec file or a new generated one.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org For followers of most ideologies (openly religious or not), toleration is a concession of defeat. For libertarians, it is victory itself.
On Thu, Feb 1, 2018 at 7:54 PM, Attila Lendvai attila@lendvai.name wrote:
My theory is that that you're failing to re-generate the .spec when it's present, even though your dependencies say it will be re-generated. Therefore, ASDF decides that it's out-of-date and must be re-generated again the next time over, etc. In other words, you lie to ASDF, and ASDF punishes you right back.
that was indeed a lurking bug. excellent remote debugging skills, thanks! :)
i pushed the fix. now after i touch'd the appropriate spec files, it doesn't want to run c2ffi anymore, but it's still regenerating the lisp file unconditionally.
i ran out of hacking steam for today. i'll look into it tomorrow, but further remote debugging magic is welcome of course... :)
i'm kinda lost for now. i have no plan of action, so to say... :)
-- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- Liberty is a demand. Tyranny is submission.
One solution is to create a new file with the correct timestamp, that is either a copy of the existing spec file or a new generated one.
ok, here's an idea that i'd be grateful if i could bounce off of you:
what if i make the .spec generation a standalone operation that needs to be explicitly run by the library author? it's a bit more burden for the lib author because he needs to keep track of changes to the configuration and/or the .h file, but that would implicitly fix one of the bugs.
in fact i have already pushed this change:
https://github.com/cffi/cffi/commit/d59331b050fb31f3a3f9103f302890be176b77ce
and shall i do the same with the next stage, namely the .lisp file generation? for some reason i think it's better to keep that automatic and integrated into the normal flow of the build process, but i'm not fully convinced about that either.
if i did that, it would annull the last remaining issue, namely the current unlucky
(load-op ,(find-system "cffi/c2ffi-generator"))
dependency.
but the price would be high: either users would need to regenerate .lisp files by hand each time they pull changes to the .spec file (this is unacceptable), or all libs would need to check in also the generated .lisp files into their repos (this is less of a constraint/burden, but still...).
any thoughts?
: Attila
what if i make the .spec generation a standalone operation that needs to be explicitly run by the library author?
I think that's indeed the right thing to do.
it's a bit more burden for the lib author because he needs to keep track of changes to the configuration and/or the .h file, but that would implicitly fix one of the bugs.
He needs to, anyway, as he writes the wrappers for the library.
https://github.com/cffi/cffi/commit/d59331b050fb31f3a3f9103f302890be176b77ce
LGTM.
and shall i do the same with the next stage, namely the .lisp file generation? for some reason i think it's better to keep that automatic and integrated into the normal flow of the build process, but i'm not fully convinced about that either.
Generating the Lisp file in advance is good if it allows you to avoid loading some of the code at build time. It might take more space. Or you could save the Lisp files and drop the spec files.
if i did that, it would annull the last remaining issue, namely the current unlucky
(load-op ,(find-system "cffi/c2ffi-generator"))
dependency.
Indeed, that would be slightly nicer on ASDF.
but the price would be high: either users would need to regenerate .lisp files by hand each time they pull changes to the .spec file (this is unacceptable), or all libs would need to check in also the generated .lisp files into their repos (this is less of a constraint/burden, but still...).
any thoughts?
If you generate the Lisp files, do you need to check in the .spec files?
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org - We're all different. - I'm not!
If you generate the Lisp files, do you need to check in the .spec files?
there's a strong argument for keeping the spec files: if the cffi/c2ffi generator itself gets a bug fix/upgrade, then the lisp files would need to be regenerated for those fixes/upgrades to take effect, and that's only possible if the spec files are available.
which, BTW, currently happens automagically due to that unlucky component -> system dependency on the "cffi/c2ffi-generator" system.
thanks for the review!
-
Attila Lendvai -
Faré