I haven't used CFFI in a while.

TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

---------- and here follows the elaborate email that led me to find the above solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI bindings from a json file, that is in turn autogenerated from the C source.

so,

1) C -> json 2) json -> lisp (CFFI definitions) 3) asdf compiles/loads the generated lisp file

1) requires a heavyweight infrastructure (a binary run by run-program), so there's support to do that lazily, and just distribute the json files generated once by the lib author.

2) is relatively lightweight, but it still requires loading a broader scope of lisp dependencies, so there's support for the lib author to run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change that i'll report in a separate thread.

On 30 Jan 2018, at 15:53, Attila Lendvai wrote:

...

I haven't used CFFI in a while.

TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

---

and here follows the elaborate email that led me to find the above solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI bindings from a json file, that is in turn autogenerated from the C source.

so,

1. C -> json

2. json -> lisp (CFFI definitions)

3. asdf compiles/loads the generated lisp file

4. requires a heavyweight infrastructure (a binary run by

run-program), so there's support to do that lazily, and just distribute the json files generated once by the lib author.

2. is relatively lightweight, but it still requires loading a broader

scope of lisp dependencies, so there's support for the lib author to run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change that i'll report in a separate thread.

Thinking out loud:

I'll have a bit more of a look. IIUC what you are saying is that there should be an OP that covers the C -> JSON translation, and that requires some infrastructure that you don't want to load unless it's necessary.

Then there's JSON -> lisp that has additional dependencies that are optional.

The problem I see is that, IIRC, ASDF creates a build plan that is *unconditional*, and then *executes* it conditionally, by calling `OPERATION-DONE-P`, and skipping unnecessary operations. It does this because earlier steps in the process can change what `OPERATION-DONE-P` would return for later stages in plan execution.

The problem with this is that you want what comes *later* in the plan (whether or not the lisp derived from JSON) to affect what comes *earlier* in the plan (whether or not the JSON to lisp translation library gets loaded).

I think I know what is the Right Answer to this, but it might be so much more work that you would rather just keep your current hack working....

I think the Right Thing is to realize that what ASDF does is not so much transform files, as to maintain the consistency of the running lisp image. Now, in no case does the running lisp image need *either* the JSON generation system *or* the JSON to lisp translation system. All the *running lisp image* needs to function correctly is an up-to-date lisp file produced by this pipeline. So....

The Right Thing is to kick that pipeline out of the current lisp process. Instead of making ASDF use its `LOAD-OP`, etc. to do this translation for you, you should just create an external application (which might be a lisp program) to do the JSON generation (if needed) and the JSON to lisp translation (if needed). That external program could well use ASDF to manage itself. But all that would be in your *main* ASDF system definition would be a `JSON-GEN-OP` and a `JSON-TO-LISP-OP`, each of which would be implemented by invoking an external program. As in, for example ``` (DEFMETHOD PERFORM ((OP JSON-GEN-OP) (c json-component)) (uiop:run-program ....)) ```

Once you think about what ASDF does and doesn't do, I think this makes perfect sense. But, of course, it might be a big pain to do so.

Best, R

You're welcome, but really the intuition is due to Drew McDermott, I think in his ILC 2005 paper, which I recommend. (He describes a system like ASDF, but finer-grained, so that even individual data structures can be updated).

This is key to understanding some of the ways ASDF is NOT like "make."

Best, R

Sent from my iPad

On Mar 21, 2018, at 06:25, Attila Lendvai attila@lendvai.name wrote:

...

I think the Right Thing is to realize that what ASDF does is not so much transform files, as to maintain the consistency of the running lisp image. Now, in no case does the running lisp image need either the JSON generation system or the JSON to lisp translation system. All the running lisp image needs to function correctly is an up-to-date lisp file produced by this pipeline. So....

this is a very useful insight, thanks for formulating it explicitly! it's a good candidate for an introductory bird's eye view section of the manual.

i already made the JSON generation a separate operation, and now i'm considering to do the same with the JSON -> lisp generation.

in a way it makes sense: the lib author tested the code that he saw. if the lisp generation gets changed/fixed (by a new patch to CFFI), it may even break the rest of the lib that may be running with some old assumptions... so, maybe it's better to just make both of those as a separate process that needs to be initiated by the lib author explicitly, and then checked into the git history.

...

Once you think about what ASDF does and doesn't do, I think this makes perfect sense. But, of course, it might be a big pain to do so.

yeah, it feels like a lot of pain. it would be nice if there was a fork-like API in ASDF for implementing such exec'd compilation, but then i guess ASDF itself has no clue which /foo/bin/ directory has the underlying lisp binary, and/or which lisp to launch at all, let alone how...

-- • attila lendvai • PGP: 963F 5D5F 45C7 DFCD 0A39 -- “Far from helping students to develop into mature, self-reliant, self-motivated individuals, schools seem to do everything they can to keep youngsters in a state of chronic, almost infantile, dependency. The pervasive atmosphere of distrust, together with rules covering the most minute aspects of existence, teach students every day that they are not people of worth, and certainly not individuals capable of regulating their own behavior.” — Alfie Kohn, 'Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A's, Praise, and Other Bribes'