dear hackers,
i cannot convince ASDF to only perform my custom operation if the output file doesn't exist; i.e. regardless of the modification times.
it's in cffi/c2ffi: foo.h -> foo.spec -> foo.lisp
when the spec is there, regardless of the mod time, i'd like to skip the spec generation task.
neither reading the manual, nor grepping around helped me to come up with the required magic.
my custom operation-done-p isn't even called if the file's mod time instructs ASDF to run the operation.
any hints are appreciated, including pointers to examples or tests.
- attila
PS: i sent the same mail from the wrong account, don't allow that through moderation please.
FTR, i have managed to solve it with this:
(defclass generate-lisp-op (selfward-operation) ((selfward-operation :initform '()))) ; we will specify it in our own COMPONENT-DEPENDS-ON
(defmethod component-depends-on ((op generate-lisp-op) (c c2ffi-file)) `((load-op ,(find-system "cffi/c2ffi-generator")) ;; Regenerating the spec file is a lot of headache, so we ignore ;; the file modification times, and only communicate the ;; dependency to ASDF if the spec file is missing. ,@(let ((spec-file (input-file op c))) ; TODO is it legal to call ASDF:INPUT-FILES here? (when (or (not (probe-file spec-file)) (zerop (with-input-from-file (s spec-file) (file-length s)))) `((generate-spec-op ,c)))) ,@(call-next-method)))
the commit: https://github.com/cffi/cffi/commit/46975c644aeb0a832cd170e953cc8ec3cbdc78df
- attila
PS: sorry for the double send, i have cleaned up my mail settings.
On 14 Apr 2021, at 18:38, Attila Lendvai wrote:
FTR, i have managed to solve it with this:
(defclass generate-lisp-op (selfward-operation) ((selfward-operation :initform '()))) ; we will specify it in our own COMPONENT-DEPENDS-ON
(defmethod component-depends-on ((op generate-lisp-op) (c c2ffi-file)) `((load-op ,(find-system "cffi/c2ffi-generator")) ;; Regenerating the spec file is a lot of headache, so we ignore ;; the file modification times, and only communicate the ;; dependency to ASDF if the spec file is missing. ,@(let ((spec-file (input-file op c))) ; TODO is it legal to call ASDF:INPUT-FILES here? (when (or (not (probe-file spec-file)) (zerop (with-input-from-file (s spec-file) (file-length s)))) `((generate-spec-op ,c)))) ,@(call-next-method)))
the commit: https://github.com/cffi/cffi/commit/46975c644aeb0a832cd170e953cc8ec3cbdc78df
I was wondering why it was important to avoid the build when the `.h` file is newer than the `.spec` file. This shouldn't happen very often, should it? Is it because:
1. The `.h` file is getting refreshed too often by something upstream of ASDF? E.g., is `make` being too eager about rewriting the `.h` file?
2. There is something wrong in the ASDF dependencies? Is it possible that there was an extraneous dependency on the `.spec` file that was causing it to mistakenly think it needed refreshing when some component upstream in the ASDF system was modified? E.g., was this in a `:serial` system so that other changes were causing ASDF to be too eager about re-generating the spec file?
I don't have any great ideas about fixing the problem if it's caused by something outside ASDF, but if the second explanation is the case, it would be worth investigating further so that you don't need to make this change (and can regenerate the spec file when the header file is updated without causing lots of wasted work). I hate to see this kind of spoofing, because it can so easily lead to confusing and hard-to-debug issues later on when the `.h` file really *does* change and ASDF ignores the change.
If it is something outside ASDF that's rewriting the header file when it's not necessary then you could do something more sophisticated and squirrel away a copy of the old `.h` file and use `diff` or `cmp` in `operation-done-p`.
Best, R
- attila
PS: sorry for the double send, i have cleaned up my mail settings.
Don't worry about the double send: you are now cleared to submit emails from either of your email accounts, so even if you send from the wrong account, it should turn up OK.
thanks for the ideas Robert!
I was wondering why it was important to avoid the build when the .h file is newer than the .spec file. This shouldn't happen very often, should it? Is it because:
the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files.
one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either.
The .h file is getting refreshed too often by something upstream of ASDF? E.g., is make being too eager about rewriting the .h file?
the problem is not with too often, but with anything greater than zero, e.g. a git checkout.
- attila
One way to avoid unneeded regeneration in the future without lying about the dependency would be to modify the script to: 1. Remember in the output the hash of the input. 2. In the regeneration script, have a shortcut that leaves the output unchanged if the input matches.
That won't help if you have to deal with branches back in time, but in those cases, you can use touch to reset the .h file to an old date.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The greatest lies always come in a thin coating of truth.
On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai attila.lendvai@gmail.com wrote:
thanks for the ideas Robert!
I was wondering why it was important to avoid the build when the .h file is newer than the .spec file. This shouldn't happen very often, should it? Is it because:
the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files.
one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either.
The .h file is getting refreshed too often by something upstream of ASDF? E.g., is make being too eager about rewriting the .h file?
the problem is not with too often, but with anything greater than zero, e.g. a git checkout.
- attila
On 17 Apr 2021, at 17:09, Faré wrote:
One way to avoid unneeded regeneration in the future without lying about the dependency would be to modify the script to:
- Remember in the output the hash of the input.
- In the regeneration script, have a shortcut that leaves the output
unchanged if the input matches.
That won't help if you have to deal with branches back in time, but in those cases, you can use touch to reset the .h file to an old date.
That's neat. If the hash is written to the file system (and into the Git repo), then it can be checked, and that would avoid a redo even after an initial checkout!
If you do this, Attila, please let us know -- it might be a useful capability to generalize and put in the ASDF contribs.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The greatest lies always come in a thin coating of truth.
On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai attila.lendvai@gmail.com wrote:
thanks for the ideas Robert!
I was wondering why it was important to avoid the build when the .h file is newer than the .spec file. This shouldn't happen very often, should it? Is it because:
the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files.
one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either.
The .h file is getting refreshed too often by something upstream of ASDF? E.g., is make being too eager about rewriting the .h file?
the problem is not with too often, but with anything greater than zero, e.g. a git checkout.
- attila
That would be a great extension to have for ASDF in general, but would have to be an extension, unless the maintainer is willing to put the implementation of a fast hash function in ASDF. If so, I nominate BLAKE3. It should take a few hundreds of lines of code—a relatively small amount by ASDF 3.3 standards. And if the entry function is not-inline, like all UIOP functions, it can be replaced by a very fast implementation when a suitable cryptographic library is loaded.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org He who says he will die for a cause will probably lie for it and may kill for it. — John McCarthy
On Sat, Apr 17, 2021 at 7:29 PM Robert Goldman rpgoldman@sift.info wrote:
On 17 Apr 2021, at 17:09, Faré wrote:
One way to avoid unneeded regeneration in the future without lying about the dependency would be to modify the script to:
- Remember in the output the hash of the input.
- In the regeneration script, have a shortcut that leaves the output
unchanged if the input matches.
That won't help if you have to deal with branches back in time, but in those cases, you can use touch to reset the .h file to an old date.
That's neat. If the hash is written to the file system (and into the Git repo), then it can be checked, and that would avoid a redo even after an initial checkout!
If you do this, Attila, please let us know -- it might be a useful capability to generalize and put in the ASDF contribs.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The greatest lies always come in a thin coating of truth.
On Sat, Apr 17, 2021 at 4:44 PM Attila Lendvai attila.lendvai@gmail.com wrote:
thanks for the ideas Robert!
I was wondering why it was important to avoid the build when the .h file is newer than the .spec file. This shouldn't happen very often, should it? Is it because:
the problem is much more down to earth: generating the spec file requires launching a libllvm based external tool called c2ffi. compiling and properly configuring this tool can sometimes consume a day, and the users of e.g. hu.dwim.sdl shouldn't ever need to do that. it's only needed when you want to capture newly added artifacts from the SDL.h files.
one way to avoid this could be to not communicate the dependency to ASDF, and make the .h -> .spec generation manual (this used to be the case). for some reason that feels lame, but not the end of the world either.
The .h file is getting refreshed too often by something upstream of ASDF? E.g., is make being too eager about rewriting the .h file?
the problem is not with too often, but with anything greater than zero, e.g. a git checkout.
- attila
-
Attila Lendvai -
Attila Lendvai -
Faré -
Robert Goldman