It looks like cffi-wrapper-file does roughly the same thing as protobuf-source-file in https://github.com/brown/protobuf/blob/master/protobuf.asd It runs some program to generate a Lisp file, then compiles and/or loads the Lisp file. Perhaps the approach taken by the protobuf ASDF support can be adapted to CFFI.

On Sat, Aug 1, 2020 at 4:25 AM Faré fahree@gmail.com wrote:

I'm not developing ASDF anymore (unless for hire) but I believe the CFFI toolchain has a new maintainer, who might be willing to devote cycles to that (or at least to merging a patch to CFFI).

Note that if the code that builds stuff and the code that tracks the dependencies disagree, the right solution is to make the dependencies match the actual build, not to make it lie better: make sure that files are attributed as the output-files of the action that creates it, and as the input-files of the other actions.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Due to circumstances beyond your control, you are master of your fate and captain of your soul.

On Fri, Jul 31, 2020 at 1:18 PM Ilya Perminov iperminov@gmail.com wrote:

...

Hi,

I found a ASDF-related CFFI bug a couple of days ago. Can anyone think of a good way of fixing it?

A method in grovel/asdf.lisp adds output files of process-op to output files of compile-op:

;;; Declare the .o and .so files as compilation outputs, ;;; so they get picked up by bundle operations. #.(when (version<= "3.1.6" (asdf-version)) '(defmethod output-files ((op compile-op) (c wrapper-file)) (destructuring-bind (generated-lisp lib-file c-file o-file) (output-files 'process-op c) (declare (ignore generated-lisp c-file)) (multiple-value-bind (files translatedp) (call-next-method) (values (append files (list lib-file o-file)) translatedp)))))

As a result inputs and outputs of the ops look like this: process-op: input: wrapper-file output: bindings-file.lisp file.c FILE.O FILE.SO

compile-op: input: bindings-file.lisp output: bindings-file.fasl FILE.O FILE.SO

The problem is that process-op generates file.o and file.so before it generates bindings-file.lisp. Thus compile-op gets re-executed all the time, because its output files file.o and file.so are always older than its input file bindings-file.lisp. And an execution of the compile-op does not change anything, because it does not really generate .o and .so. One way to fix it would be to "touch" .o and .so to get the right order of file modification times, but there may be a better way.

CFFI bug report: https://bugs.launchpad.net/cffi/+bug/1889491

Thanks, Ilya

On Tue, Aug 4, 2020 at 1:03 PM Ilya Perminov iperminov@gmail.com wrote:

I think protobuf and CFFI structure their operations in a very similar way - process-op is analogous to proto-to-lisp, it takes a "specification" file and generates a lisp file and some other files. protobuf generates lisp(fasl) files only, so it does not need to do anything special to support bundle operations. CFFI's process-op generates some .o and .so files that a bundle operation may need. The current implementation adds .o and .so files to outputs of compile-op and it causes the problem I described. I do not know what methods need to be defined on process-op to make bundle operations to pick up its output files. From my very limited understanding of ASDF I do not think there is a way to do it. Method "component-depends-on ((o gather-operation) (s system))" determines input-files of a bundle-op. The method returns dependencies of one operation only (e.g. compile-op), but in case of CFFI's wrapper-file we need output files of two operations: process-op and compile-op.

The intended design is that you only take the output of compile-op, and that process-op is implicitly involved as a dependency to compute said output, but that process-op's own outputs are not included. If process-op does the compilation and you want its outputs, you lose.

That said, I haven't looked at the code recently, nor do I intend to, being too busy trying to make my startup survive and feed my family.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org A word for the epoch of free software and universal publishing: voluntocracy n 1. governance by those who do the work. 2. the volunteers who do the work. — Aubrey Jaffer, http://swissnet.ai.mit.edu/~jaffer/

On Tue, Aug 4, 2020 at 6:13 PM Ilya Perminov iperminov@gmail.com wrote:

The files are the same. compile-op does not touch them at all. They are just its fake output-files. Is it a good idea to compile a .c to .o/.so in a compile-op? Its doc string says ""Operation for compiling a Lisp file to a FASL".

I don't see why there's any issue compiling a .c to a .o in a compile-op, the operation is for compiling in general. Maybe the doc string can be amended to "Operation for compiling source files or preprocessed source files to object files, fasls, etc.".

If they are not otherwise needed at a previous step, I would compile the files during compile-op, as a defmethod perform :before (or :after) on the suitable class. If needed at a previous step, I would copy them to a different filename.

Apologies for likely being the author of this bug.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org Social peace will come when the socialists will love the poor more than they hate the rich. — Not Golda Meir

I finally got some time to work on this and found a complication - load-source-op logically depends on generated .so files. Currently process-op produces .so files and load-source-op depends on it. If I move .c to .so compilation from process-op to compile-op, load-source-op will have to depend on compile-op somehow, which is obviously wrong.