Originally, I was very concerned to minimize the overhead of dependencies for building and testing ASDF, and I still think this is important.
But I now conclude that trying to pull in the cl-scripting infrastructure was wrong, because it had the side effect of making the system used for testing be a dependency of the system under test, introducing unhealthy circularities.
Going forward, if you want to experiment further along these lines, I'd suggest an entirely separate project -- with its own test suite! -- that would build an executable capable of executing CL scripts. That executable would have to NOT require bleeding-edge ASDF capabilities, only simple core capabilities. It should be built with the ASDF that's delivered with implementations, not with bleeding-edge ASDF.
That executable could be built, saved, tested, and installed, and then could be used as the engine for ASDF building, testing, etc., if one liked. It could be a required build tool, the way today "make" is a required build tool.
To be honest, I think such an engine should first be tested with much less demanding problems than building and testing ASDF. Simple scripting tasks, say within the ambit of the /Perl Cookbook/, would be a reasonable starting point.
Even doing this, I think there's only a 50-50 chance the approach will work out. CL's numerous incapabilities where dealing with the host OS and filesystem will still stand in the way. Committing to a single implementation -- probably CCL or SBCL -- might solve this problem. A less cumbersome means of line-by-line processing of streams should be developed, and some means of using regexp literals should be added. Variable interpolation into strings should be provided.
Just some thoughts....
Best, r
On Sat, Feb 27, 2016 at 11:32 PM, Robert Goldman rpgoldman@sift.net wrote:
Originally, I was very concerned to minimize the overhead of dependencies for building and testing ASDF, and I still think this is important.
Agreed.
Well, building ASDF strictly speaking only ever necessitate Make, a shell and cat (and, recently, Make was made superfluous, and on Windows, only CMD.EXE and its builtin COPY command were needed). [And under ASDF 1 and 2, there was no building: we directly edited asdf.lisp]
Releasing ASDF and testing it, on the other hand, always relied on some scripting infrastructure. Back in the day, it was an unhealthy mix of shell and perl, sed, and more. Early while writing ASDF 3 (January 2013), I started replacing a lot of the release scripting by a Lisp script bin/asdf-builder that was so much nicer than either shell or perl at dealing with lists of files, lists of files and regexps, lists of (tarball and list of tarball contents), etc.; but testing remained written in a mix of make and shell, which worked even when asdf was badly broken.
In May 2014, I started the minimakefile branch, which replaced run-tests.sh by a lisp script, and was eventually merged into master in November 2015 after the 3.1.6 release. Lisp scripting allowed much better error reporting, with a summary of failures at the end, and accurate-by-construction messages for reproducing and debugging failures, as well as a few bug fixes. The downsides were that:
1- It makes a hard requirement for testing that recent enough versions of plenty of lisp libraries should be installed for testing, when they were previously only weakly required for testing or strongly required for releasing but not for testing. This was mostly solved when we adopted git submodules in August 2014.
2- The setup was quite fragile, particularly on platforms that I wasn't testing it on, or in certain failure scenarios that I hadn't anticipated or insufficiently tested. This caused many bugs that were only weaned by being found by Robert the hard way over these two years. This really sucks, and I apologize to Robert again for each of them.
3- It fails badly when asdf itself is so broken that it can't load asdf-tools. I only very recently (January 2016) modified the lisp scripting infrastructure to build an executable ./build/asdf-tools the first time they are called, hopefully with a stable working asdf. [This unfortunately won't work well if the same checkout is shared on multiple architectures (e.g. Windows vs Linux vs OS X, x64 vs x86 vs ARM, etc.). But I believe we somehow already decided not to support that.] Robert still reinstantiated the old Makefile (February 2016), though he didn't delete the lisp scripting code (thanks!) In retrospect, I should probably have kept the two working side-by-side in master, rather than developing it in a branch — it would probably both have made errors easier to find, and easier to cope with when found. I was wrong to expect an easy transition, which was the idea behind the branch.
But I now conclude that trying to pull in the cl-scripting infrastructure was wrong, because it had the side effect of making the system used for testing be a dependency of the system under test, introducing unhealthy circularities.
Yes, see point 3 above.
Going forward, if you want to experiment further along these lines, I'd suggest an entirely separate project -- with its own test suite! -- that would build an executable capable of executing CL scripts. That executable would have to NOT require bleeding-edge ASDF capabilities, only simple core capabilities. It should be built with the ASDF that's delivered with implementations, not with bleeding-edge ASDF.
Agreed. That's what asdf-tools might mutate into.
That executable could be built, saved, tested, and installed, and then could be used as the engine for ASDF building, testing, etc., if one liked. It could be a required build tool, the way today "make" is a required build tool.
That was the idea behind XCVB, though it was never as portable as ASDF is. Bazel also kind of does provide this, though it's not lightweight at all.
To be honest, I think such an engine should first be tested with much less demanding problems than building and testing ASDF. Simple scripting tasks, say within the ambit of the /Perl Cookbook/, would be a reasonable starting point.
Even doing this, I think there's only a 50-50 chance the approach will work out. CL's numerous incapabilities where dealing with the host OS and filesystem will still stand in the way. Committing to a single implementation -- probably CCL or SBCL -- might solve this problem. A less cumbersome means of line-by-line processing of streams should be developed, and some means of using regexp literals should be added. Variable interpolation into strings should be provided.
Just some thoughts....
Well, inferior-shell allows for variable interpolation into strings on top of the functionality provided by uiop:run-program. optima-ppcre makes regexp's quite usable on top of the functionality provided by cl-ppcre:register-group-bind, etc. uiop:run-program already provides a variety of line-by-line processing options, with an extension mechanism to support more.
But yes, to make it more usable by the masses, a documentation effort would be required. My goal so far was only to make sure the basic capability is ubiquitous, which it now is thanks to ASDF3's UIOP, whereas Quicklisp makes it easy to install extensions (thanks, Xach). We require more minerals.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org If being against something is a phobia, then being for is mania. Peace and understanding through slurs of mental illness. Homomania, islamomania, etc.
3- It fails badly when asdf itself is so broken that it can't load asdf-tools. I only very recently (January 2016) modified the lisp scripting infrastructure to build an executable ./build/asdf-tools the first time they are called, hopefully with a stable working asdf.
isn't this a standard bootstrapping issue?
and the usual way to resolve that is to use git to check out the latest stable version into a tmp dir, build the tool, and then use the freshly built tool to test the bleeding edge.
for reference gambit scheme has a sophisticated method set up for adding new features that are then used in the compiler implementation in version n+1.
On 2/29/16 Feb 29 -1:37 PM, Attila Lendvai wrote:
3- It fails badly when asdf itself is so broken that it can't load asdf-tools. I only very recently (January 2016) modified the lisp scripting infrastructure to build an executable ./build/asdf-tools the first time they are called, hopefully with a stable working asdf.
isn't this a standard bootstrapping issue?
and the usual way to resolve that is to use git to check out the latest stable version into a tmp dir, build the tool, and then use the freshly built tool to test the bleeding edge.
for reference gambit scheme has a sophisticated method set up for adding new features that are then used in the compiler implementation in version n+1.
This isn't really a standard bootstrapping issue, and I think it's better to break the vicious cycle by simply booting the scripting engine out into a separate repository.
Here's the (I believe simple) approach:
1. Set up a repository that provides a *general* CL-scripting capability.
2. Build that capability using only stable tools.
3. Provide in the ASDF repository code that simply uses the CL-scripting capability. I.e., just provide CL scripts there, and not the scripting engine.
At this point we would have broken the vicious cycle, and the CL scripting engine would play a role akin to perl, make, etc., in the ASDF development process.
Instead, right now we have things like ASDF requiring to test that it successfully be able to build a tool suite using a new ASDF capability.
Note that this is what I'm suggesting *in the abstract*. Right now, the CL scripting framework is not, IMO, ready for prime time. I'm not willing to wrestle with undocumented, untested code as a precondition for simply fixing ASDF bugs.
I'm not saying that it's bad that CL scripting is at a primitive state now -- that's a state all software has to go through. All I'm saying is that CL scripting has a lot of promise, but it's not the kind of tested tool I require for ASDF development, and it's not going to get there (IMO) for some years yet.
So I am backing slowly away from it....
On Mon, Feb 29, 2016 at 3:21 PM, Robert Goldman rpgoldman@sift.net wrote:
This isn't really a standard bootstrapping issue, and I think it's better to break the vicious cycle by simply booting the scripting engine out into a separate repository.
Indeed, one big issue is that the asdf under test shouldn't be used to compile asdf-tools. I believe what's currently provides a solution by building asdf-tools the first time with a hopefully stable asdf. Certainly, it all needs to be better documented / made more usable when upgrade is needed, etc.
Here's the (I believe simple) approach:
- Set up a repository that provides a *general* CL-scripting capability.
This is not going to happen for a while. Also, part of the appeal of CL scripting is the ability to use standard ASDF to load systems to extend the scripting infrastructure. If the scripting systems are preloaded and can't be upgraded, that's a problem. If they CAN be upgraded, using ASDF, we're back to the same problem. If they CANNOT be upgraded, that sucks. If they CAN be upgraded, but using something else than ASDF... then either that something else is better, and then we can drop ASDF but we have the same issue with that ASDF replacement; or it's not better and then it's yet another bad tool that makes CL scripting bad, too.
- Build that capability using only stable tools.
Nope. There is an essential bootstrapping issue, here. We can move it around, but it will remain.
- Provide in the ASDF repository code that simply uses the CL-scripting
capability. I.e., just provide CL scripts there, and not the scripting engine.
Well, first, a lot of the dependencies are used in the tests themselves. Also, a lot of the code is ASDF-specific, and yet is written in a modular way, which, in CL, means using ASDF. Even if somehow dependencies are magically included in the generic tool, we'll still want to use ASDF to load the asdf-specific code in asdf-tools.
At this point we would have broken the vicious cycle, and the CL scripting engine would play a role akin to perl, make, etc., in the ASDF development process.
No, we haven't broken the vicious circle. The only way to break the vicious circle for ASDF is if there's a non-ASDF tool being used to build the code, in which case THAT tool has the vicious circle. That said, if we adopt, e.g. Bazel as the standard way to build and test ASDF, then yes, we pushed back the bootstrapping issues onto Bazel.
Instead, right now we have things like ASDF requiring to test that it successfully be able to build a tool suite using a new ASDF capability.
Well, that's why I introduced an extra step, i.e. build_asdf_tools, which builds a ./build/asdf-tools binary, and that you should run with a stable asdf, so you can later use it to test an unstable asdf.
Note that this is what I'm suggesting *in the abstract*. Right now, the CL scripting framework is not, IMO, ready for prime time. I'm not willing to wrestle with undocumented, untested code as a precondition for simply fixing ASDF bugs.
I'm not saying that it's bad that CL scripting is at a primitive state now -- that's a state all software has to go through. All I'm saying is that CL scripting has a lot of promise, but it's not the kind of tested tool I require for ASDF development, and it's not going to get there (IMO) for some years yet.
asdf-tools depends on many systems, most of which are relatively old and stable and widely used in the community and/or well-tested, and not scripting-specific.
Three systems that I wrote may count as new, unstable, and scripting-related: inferior-shell, cl-launch, and cl-scripting.
inferior-shell, which provides a user-friendlier front-end to uiop:run-program, including string interpolation, rich fd redirections and piping, is moderately documented, and has a small test suite. I believe it has several users besides me, and the trivial parts used by asdf-tools are stable.
The cl-scripting system that handles don't-stop behavior akin to make -k on the other hand is relatively new and largely undocumented indeed. But it now has a test suite (that I added after you reported some bad bugs indeed), and the code is quite small and straightforward. ASDF is the only user so far (beside other personal scripts of mine) and I apologize for having things break on you, but I believe the bugs have now been fixed, and with tests added, and the parts used by ASDF should now be stable.
Finally, I believe cl-launch has several users besides me, and it has had an extensive test suite for many years. The test suite doesn't cover the relatively new cl-launch/dispatch code that asdf-tools relies upon, but that code is very small and simple, stable, and heavily used by myself (and by anyone else who might use cl-launch to produce multi-call binaries). There haven't been any reported problems with that code, so I presume it's stable for good.
So I am backing slowly away from it....
The cl-scripting functionality (now that it's debugged) is miles ahead of what make -k provided in terms of usability: it gives you a readable summary of all the failures that happened during a run of test-all-no-stop, with enough context to replay. Detailed instructions to replay the failures are also included in the log file — but the log file isn't vomited at the console as it is with the shell-based testing framework, so your testing status isn't lost in horrible testing output.
Yes, there have been issues in the past, and you've been bitten by a lot of them, especially as you've been testing on Windows and Mac which I don't have regular access to (but I could steal my wife's mac once in a while and install and configure a Windows VM at some point).
There are issues, notably bootstrapping issues, with using CL to test ASDF. I believe these issues are fixable, that most have been fixed, and that the result is better than depending on an unreadable hodge-podge of Make, shell, sed and perl, that also have to be installed and curated.
—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org The probability of the people in power being individuals who would dislike the possession and exercise of power is on a level with the probability that an extremely tender-hearted person would get the job of whipping-master on a slave plantation. — Frank H. Knight
-
Attila Lendvai -
Faré -
Robert Goldman