Depending on and generating files in system definitions
Hi. It seems to be a common pattern to maintain a project's version information as a string in a separate file, e.g. version.sexp, and use something like (:version (:read-file-form "version.sexp")) to get that information into the system definition. This seems like a good thing to do as long as version changes are rare and performed manually. Unless there is a better way for this simple case, no problems or questions so far. There are, however, situations in which this doesn't work. My use-case is a project which is maintained in a GIT repository and uses the following versioning scheme MAJOR.MINOR.NUMBER-OF-COMMITS-SINCE-RELEASE This is useful for things like bug reports or determining compatibility of input files with the current version of the software. The crucial part is maintaining NUMBER-OF-COMMITS-SINCE-RELEASE automatically. Previously, I implemented this behavior using some horrible hacks that I would rather not talk about. I started from scratch and came up with a potential solution (detailed description attached). Since this seems like a common problem, I would like to ask whether the outlined approach is the proper way to do this. If so, I can try to turn the description into something that could be added to the FAQ section of the ASDF manual. Thanks in advance and kind regards, Jan
On 6/28/16 Jun 28 -7:19 AM, Jan Moringen wrote:
Hi.
It seems to be a common pattern to maintain a project's version information as a string in a separate file, e.g. version.sexp, and use something like
(:version (:read-file-form "version.sexp"))
to get that information into the system definition. This seems like a good thing to do as long as version changes are rare and performed manually. Unless there is a better way for this simple case, no problems or questions so far.
There are, however, situations in which this doesn't work. My use-case is a project which is maintained in a GIT repository and uses the following versioning scheme
MAJOR.MINOR.NUMBER-OF-COMMITS-SINCE-RELEASE
This is useful for things like bug reports or determining compatibility of input files with the current version of the software. The crucial part is maintaining NUMBER-OF-COMMITS-SINCE-RELEASE automatically.
Previously, I implemented this behavior using some horrible hacks that I would rather not talk about. I started from scratch and came up with a potential solution (detailed description attached).
Since this seems like a common problem, I would like to ask whether the outlined approach is the proper way to do this. If so, I can try to turn the description into something that could be added to the FAQ section of the ASDF manual.
Thanks in advance and kind regards, Jan
I only had time for a quick look at this, but it did suggest a couple of questions: 1. Is it necessary to have rsb-version write the version-string.sexp file? Could you simply override the COMPONENT-VERSION generic function and compute it on the fly from git and version.sexpr? 2. How does this work across developers? Couldn't you end up in a situation where two developers have versions x.y.z and x.y.z' where z = z' but the two systems are different (because they are on different git repositories with differing histories)? In an ideal world, the version identifier should identify a unique state of the codebase, yes? I suppose one could get the version identifier from some shared history (like the remotes/origin branch). 3. Your rsb-protocol-config system seems quite reasonable. We have done things like this to, for example, invoke "make" to build required non-lisp code. Hope that helps, r
On 6/28/16 Jun 28 -1:57 PM, 73budden . wrote:
What about #. ?
One can, of course, use #. to set any part of a system definition. But it's worth thinking about how this could go awry. E.g., if you list all the files in the system definition, in the classic way, and you add a file, the system definition file will change. If you try to load the system again, ASDF will notice that the .asd file has changed, and reload with your new file. OTOH, if you create a set of components using #. and DIRECTORY, and the directory contents change, the asd file will not have changed, and ASDF will not notice anything, and you will get something wrong if you try to load the system again. TBH, I don't like these computed component list ideas because to do the right thing, in general, we'd need to have a lot of additional machinery to make ASDF correctly notice when the computed components should change. IMO, that's a lot of engineering to save someone from having to type (iter (for file in (directory "*.lisp")) (format t "(:file \"~a\") (pathname-name file))) and paste the results into the .asd file. Also, as has been pointed out previously, CL does not have glob patterns, so wildcard matching is very weak. To do better than "match all" or "match all of type X" you need to load a regexp library, and iterate over namestrings from a call to DIRECTORY. As I said, that's a lot of engineering to avoid typing a file manifest. I'm not ruling it out, but I'm not enthused. Of course, if someone wants to use #. that's fine -- it's not my problem if it goes wrong. But if we put it into ASDF and it goes wrong, it *is* my problem. Best, r
(replying to myself as I'm not subscribed to the ml and thus didn't get Robert's email, sorry). Thank you for the quick reply, Robert.
I only had time for a quick look at this, but it did suggest a couple of questions:
1. Is it necessary to have rsb-version write the version-string.sexp file? Could you simply override the COMPONENT-VERSION generic function and compute it on the fly from git and version.sexpr?
I omitted the fact that version.sexp can also contain the complete version specification in which case rsb-version writes version- string.sexp without consulting git. This mode is intended for installing or packaging the sources independently of the git repository. One use case for this ability is packaging the source with cached version information and copying it into the workspace of a different project in a continuous integration scenario. Also, shelling out to git for each COMPONENT-VERSION method call seems expensive to me (But maybe I'm just paranoid since my previous solution was super inefficient and wasteful).
2. How does this work across developers? Couldn't you end up in a situation where two developers have versions x.y.z and x.y.z' where z = z' but the two systems are different (because they are on different git repositories with differing histories)? In an ideal world, the version identifier should identify a unique state of the codebase, yes?
I suppose one could get the version identifier from some shared history (like the remotes/origin branch).
The requirement here is mostly that non-lisp-developers (or even non- developers) can build (and pull + re-build) the master branch and release branches of the project and automatically get a binary with detailed and accurate version information. I guess rsb-version could also writes a version-details.sexp which would include the commit id. But I cannot see how that would work as an ASDF system version (I seem to recall that net.didierverna.clon has something like that and ASDF seems to get upset about it).
3. Your rsb-protocol-config system seems quite reasonable. We have done things like this to, for example, invoke "make" to build required non-lisp code.
Good, thanks. Again, thanks for the quick reply and valuable feedback. Kind regards, Jan
participants (3)
-
73budden . -
Jan Moringen -
Robert Goldman