On Mon, Jun 28, 2010 at 4:14 PM, Faré <fahree@gmail.com> wrote:

Dear Vsevolod,

> Approximately after the release of ASDF 2 I've got interested in creating
> yet another system distribution management system on top of ASDF, and that
> made me investigate ASDF internals. Through that investigation, I've
> understood 2 things:
> * ASDF has support for much more, than is well known and in ordinary use by
> most Common Lisp programmers :)
> * still it's support for versioning (which is essential to distribution

> management) is lacking due to: [...]
>
Yes, ASDF is lacking in terms of support for versioning. And I admit I
am not convinced that extending that support is a good thing. Unless
you're going to reimplement and maintain something that's better than
existing toolchains, then there's no point at all in the exercise:
Lisp users are much better off using existing tools to manage versions
than roll their own half-assed mock up. Dpkg or RPM or Nix are very
well suited for managing the mix-and-match of many different versions
of packages. Git or SVN are very well suited for managing the exact
match of package combinations.

I'm not saying that distribution management is useless, or that you
shouldn't be coding anything, but rather that there are a lot of
interesting problems in the world, and a lot of things that could help
the Lisp community, and that it's not obvious that yet another
distribution management system is the one most worth of the *huge*
cost of doing it right. — And I speak from the experience of trying to
turn a simple one-sentence idea of an improvement upon ASDF into
software (XCVB).

I understand that and I'm not pushing the particular issue of a distribution management tool here. I'm more in the mood of trying to build something on top of ASDF with the smallest effort. Moreover this is not directly connected to versioning issue, or let's say, proper versioning may have other application besides what I have in mind.

If I may suggest a simpler, cheaper and more useful strategy for
helping with the version management of Lisp software, it would be to
automatically produce Debian, RPM or Nix packages from ASDF
specifications, and let those tools do the rest of the integration.

> Attached is the proposed patch to ASDF, that, in my view, solves the
> aforementioned versioning problems. It does the following:
> * Unifies internal version representation format in the form of a list of
> integers: (for example, ASDF 2.106 will have an internal version of '(2 106
> 0), while ASDF 2.005 — '(2 5 0)).

Why only three integers? While you're at it, why not accept the whole
range of Debian or RPM version comparisons?

Actually, the number of integers is not limited, it's just that this is the default practice and currently available predicates only take into account the first two. So the internal representation is a list of integers. But the user can specify versions in many different formats, like "1.2.3" or even 1.2 (float number). Moreover it's possible to build adapters for more representations by defining methods for the generic function PROPER-VERSION.

> A final note about versioning: I needed to use pre-reading of ASD files in
> order to know their versions without arising version conflicts. This works,
> except for the fact, that READ-EVAL is turned off in the process, so such
> version specifiers as:
> * :version #.*hunchentoot-version*
> * :version #.(with-open-file (vers (merge-pathnames "version.lisp-expr"
> *load-truename*)) (read vers))
> do not produce expected results and are treated as empty versions. There
> will be a need to make a note about that to library developers.

To clarify my point: there are cases of libraries, that use the read-eval syntax in version definitions, most notable of which are Edi Weitz's and Cyrus Harmon's. I agree here with Robert Goldman, that this is a little contrary to the declarative nature of DEFSYSTEM form. Still it is not a problem with neither the existing, nor the proposed variants. It's just that in the FIND-SYSTEM pipeline such version declarations (or rather evaluations) won't be counted (i.e. considered equal to wildcard version). But during the evaluation of DEFSYSTEM form itself the version would be properly evaluated. So this peculiarity will effect the new behavior, but not the old, that is why I don't consider it a problem: if someone would like to use the new behavior, he is free to adapt his code, otherwise he can leave it as is and no problems would arise.

>

If we ever go this way, it might be better to extend ASDF to accept
reading versions from file, either by READing an object, or by
READ-LINEing it.

> Besides, an incompatible change is introduced to FIND-SYSTEM. The ERROR-P
> optional argument is removed, so plain NIL is unconditionally returned, when
> system is not found, and VERSION and VERSION-P optional arguments are added
> instead.

This sounds very bad to me. What is the use case?

The use case is that we can now (oos 'load 'system :version "1.2.3") or write in defsystem :depends-on (system "1.2.3" :above). As all those user-facing facilities use FIND-SYSTEM as the underlying tool. Independently from adding VERSION and VERSION-P I would still argue to remove ERROR-P argument, as it adds unnecessary complexity. There's only one place in the original ASDF code (OPERATE), where it's expected and it is easily handled in the caller.

How does that affect
the semantics of find-system? Can there now be many instances of a
system in the source-registry?

The semantics is the following: only zero or one instance of a system with a particular name can be in memory (in *defined-systems*) — just like before. The current behavior is that if some system is loaded, there's no possibility to load the new version. The new behavior is that, if FIND-SYSTEM is forced to load the different version of the system, a continuable error is signaled, that informs the user of the situation and it's possible either to abort or continue. If the user continues, the new definition is loaded instead of the old.

Does find-system now have to grovel for
*all* these instances, load the .asd files to determine the version
and pick the most recent one by default? That's C R A Z Y.

Yeah, now SYSDEF-CENTRAL-REGISTRY-SEARCH et al. return a list of found ASD files instead of the first one. Then the most suitable is chosen by these criteria:
* if no version is specified it will be either the one already in memory, or the first one in the list
* if version is specified it will be the one in memory, if VERSION-SATISFies, or the first from a list, for which version satisfies

Note that the current interface of find-system seems to be copied from
that of find-class.

Hmm, I see. I still don't support such decision, but maybe I just not understand something...

The way I see things is that whoever sets up the source-registry
thereby specifies which versions of each piece of software is going to
be used, at which point version management can be taken completely out
of ASDF and into the hands of whoever specifies said source-registry.

Yeah, that's also a possible approach. It has it's advantages for sure.

If you want to check that what is configured is correct, you can check
versions. And e.g. POIU, XCVB or ASDF-DEPENDENCY-GROVEL do have such
consistency checks. But that's as much as you need.

> I have supplied a number of unit tests to the functions,
> that I have added. This is done with a very simple unit testing library of
> my own (a microframework so to say) MUTEST, that can be found at:
> http://github.com/vseloved/mutest/.
>

I'd rather ASDF has only one test framework, if possible one with few
dependencies if at all.

I agree. If there's one already chosen, I'll stick to it. as I've said, when I've started this work, I didn't find any tests, so I used my own approach. By the way MUTEST is dependency-free.

The current one kind of sucks but works so
far. I'm not against using a different one, but someone should then do
the conversion of existing tests.

> I have also done functional testing of the changes with the systems, that I
> have got installed already (around 70 libraries), and have verified, that
> there are no problems. At the same time I would not claim, that there will
> not be any problems with some system definitions, that use obscure ASDF
> features (although, I have acquainted myself with ASDF to the point, that I
> quite well understand most of the features as well as internals now). So I
> ask anyone interested in this set of changes to thoroughly test them.
>

I think that's a good strategy, and one many people have been
following lately. Maybe there's a way to automate and share these
things further.

> Finally, the patch can be applied by simply loading ASDF-VERSION.LISP on top
> of the loaded ASDF 2.106.
>
> I hope, this work proves useful and is integrated into ASDF. Anyway I'm
> willing to answer any questions and/or improve the implementation, if
> needed.
>

I admit this is not on top of my priority list. I'll try to review
your code in detail later this week. Ideally, a new ASDF maintainer
would step forward and replace me.

Thanks for your efforts,

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Most economic fallacies derive... from the tendency to assume that there
is a fixed pie, that one party can gain only at the expense of another.
— Milton Friedman