On Mon, May 9, 2016 at 8:41 AM, Robert Goldman rpgoldman@sift.net wrote:

I am inclined to agree with Andreas. I don't see that CLEAN-OP is impossible for ASDF. I can see that there are some issues that require thought, and for some systems additional coding by the developer, but I don't see that as an insuperable barrier. After all, "make clean" is in some sense impossible (it's generally hand-coded by the programmer), but we use it every day.

Many things are possible. My point was that there was no one-size-fits-all, especially not as ASDF is currently designed.

NB: Bazel, that manages all its output in its cache, and doesn't allow operations to disable that cache for some outputs like ASDF does, has a bazel clean that does the notional equivalent of rm -rf ~/.cache/common-lisp/ Maybe the solution is to become more like bazel.

I accept Faré's points about there being challenges to write an ideal CLEAN-OP. But I don't agree that this means that we can't write a good enough CLEAN-OP. After all, make manages "make clean." Yes, it requires some effort from the programmer, but who cares? Is that a reason we have to throw up our hands and refuse to do anything? Here are some specific responses to particular issues.

On the other hand, while I don't agree that these make the CLEAN-OP impossible, I *do* agree that they are difficult issues. So... I'd be happy to see CLEAN-OP added, and I'd be happy to support this effort, *BUT* CLEAN-OP patches must be preceded -- by an informal specification that describes what an implementation should do, covering these cases.

Yup.

Now that ASDF is on everyone's critical path, we need to think a little before changing it, and not just bash out a hunk of code.

We could possibly make that an extension to ASDF so as not to have to modify and test ASDF, which is a heavy thing to do... but then, you'd need to build and load the extension to run it, which would itself dirty the cache. Of course, the same is true if you use ASDF from git, anyway.

Specific responses to Faré's issues:

1. clean-op is not defined in general. I believe that this means "do we

clean the system" or "do we clean the operation" [as in arnesi, whose CLEAN-OP has a FOR-OP]?

This seems similar to the notion that Faré already had, of providing a BUILD-OP that does "the default build action" for a system (e.g., maybe it builds the library? Maybe it does LOAD-OP? Maybe it does LOAD-OP and DOC-OP?).

So one solution would be to have an UNMAKE-OP that does the inverse of the build-op, and then have a CLEAN-OP that undoes specific operations (like the arnesi CLEAN-OP). Then the UNMAKE-OP might default to being the CLEAN-OP for COMPILE-OP, which would handle probably 90% of systems.

On the other hand, I'm pretty unhappy with the notion of the BUILD-OP, because it requires retraining every single user of ASDF to use BUILD-OP instead of LOAD-OP. That would be an enormous user interface fail. So this is not a step to be taken lightly.

Maybe instead of a CLEAN-OP, we could have an unoperate function, or a clean-system function. It makes sense to have a function that is not operate because we'll also want to clean timestamp cache entries for the operations that are undone, i.e. undo the mark-operation-done for every action the output-files of which have been cleaned. Similarly, you probably do NOT want to include a hypothetical CLEAN-OP in said timestamp cache, and the cache is particularly ill-suited for a CLEAN-OP with a :for-op, :op or :operations option, since the current cache crucially fails to index such options.

Or maybe we just want a clean-cache function. And maybe we also want to remove the feature that allows output-files to not relocate things to the cache.

2. I think we can simply make clean-op clean for the current implementation.

Sure. But what if some operations actually involve cross-compilation with other implementations?

3. I believe we should mirror the semantics of :FORCE T and :FORCE ALL

when we unmake. I.e., unbuild the dependencies only with :FORCE :ALL. :FORCE T would actually be a no-op.

Makes sense.

4. "How do you deal with

defsystem-depends-on and other dependencies not explicitly included in the plan?" DEFSYSTEM-DEPENDS-ON is already broken, and will be deprecated, except as a checkable declaration (see earlier postings on this list for a description of the issues). For the rest, I'd say same as above -- if there's something that won't appear in the build plan that one needs to clean, then the programmer must write an :IN-ORDER-TO for the CLEAN-OP so that ASDF can know what it needs to do.

Just calling the feature broken and having people do imperatively what the feature tries to do declaratively (i.e. loading a system from a .asd file) doesn't make the usage pattern go away. And this usage pattern means that just to *compute* the plan to clean a particular system, you'll be *loading* systems and dirtying that cache; and with the current and foreseeable ASDF (i.e. unless and until the feature is actually fixed rather than ignored), these loaded systems won't be in the dependency graph. Thus by trying to clean, you WILL make the cache actually dirtier than it was. Fixing that the D-S-D feature will also be instrumental in making :FORCE :ALL anything but a joke.

Finally, there's the issue of whether and if so how to handle secondary systems. Should they too be cleaned? If not, then :FORCE T might not be doing much useful when most of the build happens in secondary systems (especially when using package-inferred-systems).

5. "Are you sure there aren't bugs and

omissions in asdf or any extension whereby some side output file is omitted from the list?"

ow does make deal with a file that isn't listed in the rule for "make clean"? You miss it. Very sorry. Fix your system definition. Use (:in-order-to (clean-op ....))

Sure, bugs and bugs and need be fixed. But until they are, any clean function will not be as useful as advertised. And we probably have many such bugs lurking in extensions, since nobody tested this aspect.

6. "What about randomly-named temporary

files created during an operation that has been interrupted?" Maybe we are doing the Wrong Thing with these. If they are temporary files, maybe it's the wrong thing to put them in the cache and rename them? Maybe they should live in /tmp (or equivalent) and be moved from there? Then it's not our problem to fix anymore -- that's for tmpwatch.

If you add your own rules that make temporary files, you have to add your own rules that remove them.

No, no, you don't get it. ASDF itself already uses temporary files all over the place, not to mention extensions such as CFFI; and to solve a concurrency issue, at least ASDF has recently started to include a random portion in the temporary file names (so that two concurrent attempts don't clobber each other). So as to hopefully achieve atomic renaming of temporary files to output files, these temporary files do NOT reside in /tmp but in the very same directory as the future output file. So it is NOT sufficient or possible to rely on tmpwatch to cleanup after ASDF. If ASDF had some *less* configurable output translations, it could probably put all temporary files in a subdirectory of "the" output cache, and then use the equivalent of tmpwatch on that directory. But that's not the case at this point.

7. "What if load-op is not the only operation that matters, but

e.g. doc-op generates LaTeX and PDF outputs but fails to be tracked by clean-op? Are you going to use the CLOS MOP to generate a list of all possible operations? What if that other operation isn't loaded in the current image? What if operation options cause different files to be generated? Or features? What about all secondary systems? Are you going to generate a plan for them, too?"

See the answer to #1: ASDF is not going to solve this problem. Either the programmer solves it using definitions of PERFORM and IN-ORDER-TO, or it's not solved. So sad. But I don't see any reason to avoid the 90% solution because there isn't a 100% solution.

Fine, but I see a clean-op more like a 50% solutions 50% increased problem than a 90% solution. rm -rf ~/.cache/common-lisp and/or git clean -xfd are the 90% solutions.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The fear of death follows from the fear of life. A man who lives fully is prepared to die at any time. — Mark Twain