Hi,
I found a ASDF-related CFFI bug a couple of days ago. Can anyone think of a good way of fixing it?
A method in grovel/asdf.lisp adds output files of process-op to output files of compile-op:
;;; Declare the .o and .so files as compilation outputs, ;;; so they get picked up by bundle operations. #.(when (version<= "3.1.6" (asdf-version)) '(defmethod output-files ((op compile-op) (c wrapper-file)) (destructuring-bind (generated-lisp lib-file c-file o-file) (output-files 'process-op c) (declare (ignore generated-lisp c-file)) (multiple-value-bind (files translatedp) (call-next-method) (values (append files (list lib-file o-file)) translatedp)))))
As a result inputs and outputs of the ops look like this: process-op: input: wrapper-file output: bindings-file.lisp file.c FILE.O FILE.SO
compile-op: input: bindings-file.lisp output: bindings-file.fasl FILE.O FILE.SO
The problem is that process-op generates file.o and file.so before it generates bindings-file.lisp. Thus compile-op gets re-executed all the time, because its output files file.o and file.so are always older than its input file bindings-file.lisp. And an execution of the compile-op does not change anything, because it does not really generate .o and .so. One way to fix it would be to "touch" .o and .so to get the right order of file modification times, but there may be a better way.
CFFI bug report: https://bugs.launchpad.net/cffi/+bug/1889491
Thanks, Ilya
I'm not developing ASDF anymore (unless for hire) but I believe the CFFI toolchain has a new maintainer, who might be willing to devote cycles to that (or at least to merging a patch to CFFI).
Note that if the code that builds stuff and the code that tracks the dependencies disagree, the right solution is to make the dependencies match the actual build, not to make it lie better: make sure that files are attributed as the output-files of the action that creates it, and as the input-files of the other actions.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Due to circumstances beyond your control, you are master of your fate and captain of your soul.
On Fri, Jul 31, 2020 at 1:18 PM Ilya Perminov iperminov@gmail.com wrote:
Hi,
I found a ASDF-related CFFI bug a couple of days ago. Can anyone think of a good way of fixing it?
A method in grovel/asdf.lisp adds output files of process-op to output files of compile-op:
;;; Declare the .o and .so files as compilation outputs, ;;; so they get picked up by bundle operations. #.(when (version<= "3.1.6" (asdf-version)) '(defmethod output-files ((op compile-op) (c wrapper-file)) (destructuring-bind (generated-lisp lib-file c-file o-file) (output-files 'process-op c) (declare (ignore generated-lisp c-file)) (multiple-value-bind (files translatedp) (call-next-method) (values (append files (list lib-file o-file)) translatedp)))))
As a result inputs and outputs of the ops look like this: process-op: input: wrapper-file output: bindings-file.lisp file.c FILE.O FILE.SO
compile-op: input: bindings-file.lisp output: bindings-file.fasl FILE.O FILE.SO
The problem is that process-op generates file.o and file.so before it generates bindings-file.lisp. Thus compile-op gets re-executed all the time, because its output files file.o and file.so are always older than its input file bindings-file.lisp. And an execution of the compile-op does not change anything, because it does not really generate .o and .so. One way to fix it would be to "touch" .o and .so to get the right order of file modification times, but there may be a better way.
CFFI bug report: https://bugs.launchpad.net/cffi/+bug/1889491
Thanks, Ilya
It looks like cffi-wrapper-file does roughly the same thing as protobuf-source-file in https://github.com/brown/protobuf/blob/master/protobuf.asd It runs some program to generate a Lisp file, then compiles and/or loads the Lisp file. Perhaps the approach taken by the protobuf ASDF support can be adapted to CFFI.
On Sat, Aug 1, 2020 at 4:25 AM Faré fahree@gmail.com wrote:
I'm not developing ASDF anymore (unless for hire) but I believe the CFFI toolchain has a new maintainer, who might be willing to devote cycles to that (or at least to merging a patch to CFFI).
Note that if the code that builds stuff and the code that tracks the dependencies disagree, the right solution is to make the dependencies match the actual build, not to make it lie better: make sure that files are attributed as the output-files of the action that creates it, and as the input-files of the other actions.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Due to circumstances beyond your control, you are master of your fate and captain of your soul.
On Fri, Jul 31, 2020 at 1:18 PM Ilya Perminov iperminov@gmail.com wrote:
Hi,
I found a ASDF-related CFFI bug a couple of days ago. Can anyone think of a good way of fixing it?
A method in grovel/asdf.lisp adds output files of process-op to output files of compile-op:
;;; Declare the .o and .so files as compilation outputs, ;;; so they get picked up by bundle operations. #.(when (version<= "3.1.6" (asdf-version)) '(defmethod output-files ((op compile-op) (c wrapper-file)) (destructuring-bind (generated-lisp lib-file c-file o-file) (output-files 'process-op c) (declare (ignore generated-lisp c-file)) (multiple-value-bind (files translatedp) (call-next-method) (values (append files (list lib-file o-file)) translatedp)))))
As a result inputs and outputs of the ops look like this: process-op: input: wrapper-file output: bindings-file.lisp file.c FILE.O FILE.SO
compile-op: input: bindings-file.lisp output: bindings-file.fasl FILE.O FILE.SO
The problem is that process-op generates file.o and file.so before it generates bindings-file.lisp. Thus compile-op gets re-executed all the time, because its output files file.o and file.so are always older than its input file bindings-file.lisp. And an execution of the compile-op does not change anything, because it does not really generate .o and .so. One way to fix it would be to "touch" .o and .so to get the right order of file modification times, but there may be a better way.
CFFI bug report: https://bugs.launchpad.net/cffi/+bug/1889491
Thanks, Ilya
I think protobuf and CFFI structure their operations in a very similar way - process-op is analogous to proto-to-lisp, it takes a "specification" file and generates a lisp file and some other files. protobuf generates lisp(fasl) files only, so it does not need to do anything special to support bundle operations. CFFI's process-op generates some .o and .so files that a bundle operation may need. The current implementation adds .o and .so files to outputs of compile-op and it causes the problem I described. I do not know what methods need to be defined on process-op to make bundle operations to pick up its output files. From my very limited understanding of ASDF I do not think there is a way to do it. Method "component-depends-on ((o gather-operation) (s system))" determines input-files of a bundle-op. The method returns dependencies of one operation only (e.g. compile-op), but in case of CFFI's wrapper-file we need output files of two operations: process-op and compile-op.
On Tue, Aug 4, 2020 at 1:03 PM Ilya Perminov iperminov@gmail.com wrote:
I think protobuf and CFFI structure their operations in a very similar way - process-op is analogous to proto-to-lisp, it takes a "specification" file and generates a lisp file and some other files. protobuf generates lisp(fasl) files only, so it does not need to do anything special to support bundle operations. CFFI's process-op generates some .o and .so files that a bundle operation may need. The current implementation adds .o and .so files to outputs of compile-op and it causes the problem I described. I do not know what methods need to be defined on process-op to make bundle operations to pick up its output files. From my very limited understanding of ASDF I do not think there is a way to do it. Method "component-depends-on ((o gather-operation) (s system))" determines input-files of a bundle-op. The method returns dependencies of one operation only (e.g. compile-op), but in case of CFFI's wrapper-file we need output files of two operations: process-op and compile-op.
The intended design is that you only take the output of compile-op, and that process-op is implicitly involved as a dependency to compute said output, but that process-op's own outputs are not included. If process-op does the compilation and you want its outputs, you lose.
That said, I haven't looked at the code recently, nor do I intend to, being too busy trying to make my startup survive and feed my family.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org A word for the epoch of free software and universal publishing: voluntocracy n 1. governance by those who do the work. 2. the volunteers who do the work. — Aubrey Jaffer, http://swissnet.ai.mit.edu/~jaffer/
On 4 Aug 2020, at 12:02, Ilya Perminov wrote:
I think protobuf and CFFI structure their operations in a very similar way
- process-op is analogous to proto-to-lisp, it takes a "specification"
file and generates a lisp file and some other files. protobuf generates lisp(fasl) files only, so it does not need to do anything special to support bundle operations. CFFI's process-op generates some .o and .so files that a bundle operation may need. The current implementation adds .o and .so files to outputs of compile-op and it causes the problem I described. I do not know what methods need to be defined on process-op to make bundle operations to pick up its output files. From my very limited understanding of ASDF I do not think there is a way to do it. Method "component-depends-on ((o gather-operation) (s system))" determines input-files of a bundle-op. The method returns dependencies of one operation only (e.g. compile-op), but in case of CFFI's wrapper-file we need output files of two operations: process-op and compile-op.
I don't claim to understand this process, but wouldn't it be possible for you to make your own `input-files :around` method for `gather-operation` that would collect the outputs from the `process-op`'s and add them to what you want?
Here's the existing definition of what I think is the relevant `input-files` method:
``` (defmethod input-files ((o gather-operation) (c system)) (unless (eq (bundle-type o) :no-output-file) (direct-dependency-files o c :key 'output-files :test (pathname-type-equal-function (bundle-pathname-type (gather-type o)))))) ```
This invokes `map-direct-dependencies` which invokes the `component-depends-on` method for `gather-operation` on the system which ... I don't really understand, but I believe it's the `compile-op`'s.
I think it would be easiest to write your own method that collects up all of the `process-op` outputs, drops any `.lisp` files (which will be superseded by the `.fasl` files), and adds them to the return value of `call-next-method`.
If you do that, and drop the `.o` and `.so` files from the `output-files` of `compile-op`, I think that would get what you want: you would collect the `.o` and `.so` files from the `process-op`, and you wouldn't get ASDF trying to regenerate the files when it's not necessary.
That said, I can think of a simpler, and easier method, and that would be to override the `operation-done-p` method for the `compile-op` so that it knows that the `.o` and `.so` files are generated by the `compile-op`, and only pays attention to the relationship between `bindings-file.lisp` and `bindings-file.fasl`.
One thing you didn't say was what it means that the compile-op is done over and over -- is it only compiling the bindings-file, or is it doing something to the object files? I don't believe it should change the `.o` and `.so` files, because those are already compiled by `process-op`, right?
On Fri, Jul 31, 2020 at 1:18 PM Ilya Perminov iperminov@gmail.com wrote:
As a result inputs and outputs of the ops look like this: process-op: input: wrapper-file output: bindings-file.lisp file.c FILE.O FILE.SO
compile-op: input: bindings-file.lisp output: bindings-file.fasl FILE.O FILE.SO
Are these FILE.O and FILE.SO two different files with the same name, or the same file reported twice?
In BOTH cases, you'll want to have DIFFERENT NAMES for the two sets of outputs. If the files are the same, then either you should compile them only during the compile-op, or, if not possible, you should copy the output to the new file with a different name.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The world is a comedy to those that think; a tragedy to those that feel. — Horatio Walpole, 4th Earl of Orford (1717–1797)
The files are the same. compile-op does not touch them at all. They are just its fake output-files. Is it a good idea to compile a .c to .o/.so in a compile-op? Its doc string says ""Operation for compiling a Lisp file to a FASL".
On Tue, Aug 4, 2020 at 6:13 PM Ilya Perminov iperminov@gmail.com wrote:
The files are the same. compile-op does not touch them at all. They are just its fake output-files. Is it a good idea to compile a .c to .o/.so in a compile-op? Its doc string says ""Operation for compiling a Lisp file to a FASL".
I don't see why there's any issue compiling a .c to a .o in a compile-op, the operation is for compiling in general. Maybe the doc string can be amended to "Operation for compiling source files or preprocessed source files to object files, fasls, etc.".
If they are not otherwise needed at a previous step, I would compile the files during compile-op, as a defmethod perform :before (or :after) on the suitable class. If needed at a previous step, I would copy them to a different filename.
Apologies for likely being the author of this bug.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Social peace will come when the socialists will love the poor more than they hate the rich. — Not Golda Meir
Thanks for the explanation. process-op does not really need a generated .so file, only its name. I'll try moving compilation of .c files to compile-op and see if it causes any problems. It seems it will be an easy thing to do.
I finally got some time to work on this and found a complication - load-source-op logically depends on generated .so files. Currently process-op produces .so files and load-source-op depends on it. If I move .c to .so compilation from process-op to compile-op, load-source-op will have to depend on compile-op somehow, which is obviously wrong.
I suppose this is what the previous setup was trying to avoid, but the perform method on load-source-op could compile the .so file if needed.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Corollaries to the Law of Bitur-Camember: The political process destroys the value of all known resources that are up for grabs. The socialist process of systematically denying legitimacy to property rights applies the political process universally and destroys the value of all available resources.
On Thu, Aug 27, 2020 at 2:34 PM Ilya Perminov iperminov@gmail.com wrote:
I finally got some time to work on this and found a complication - load-source-op logically depends on generated .so files. Currently process-op produces .so files and load-source-op depends on it. If I move .c to .so compilation from process-op to compile-op, load-source-op will have to depend on compile-op somehow, which is obviously wrong.