On Oct 16, 2020, at 06:21, Eric Timmons etimmons@mit.edu wrote:
[…]
I'm a huge fan of Gitlab CI and was getting tired of running all tests locally whenever I made a change, so I started a Gitlab CI configuration for ASDF at https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/146 and also modified it a bit more today. The recent modifications are marking the upgrade tests as `allow_failure: true` (since in a previous email it sounded like the previous Jenkins config didn't even run them), increasing some timeouts in tests (I don't know what happened, but some tests started failing under CCL when run on cl.net runners), reordering things such that all regression tests ran first, and addressing a bug in the latest SBCL that was triggered by the cl.net infrastructure.
If you want to take some radically different approach than I did, that's great! I just wanted to make sure everyone knew that an 80% solution exists and didn't waste their time coming up with something substantially similar. I think the biggest things left on !146 are deciding how often to run the tests (every push? only on MRs?) and adding more runners for licensed implementations or OSes.
Wow! This looks like much more than 80%, and its working today!
Speaking of OSes, my research group has a Mac mini we've been using for our own CI purposes. By no means is it a speed demon, but it's lightly used so I can set up another runner on it for ASDF.
As for the licensed implementations: I'm not sure of what the cl.net infrastructure looks like and how paranoid you want to be about protecting the license, but it seems the easiest way forward would be to add a new VM based runner with the implementation and license already installed in the VM, register that runner so that only the asdf/asdf project can use it, and tag it with the implementation name. We can then add a job that is triggered only if it's run in the asdf/asdf project and have it require that tag.
As I said in another message, this would mean that forks can't use that runner (and MR tests are, unfortunately, run in the CI context of the fork). If it's OK for master to occasionally break, the changes from forks could be merged directly to master and breakage dealt with after that. If you want master to never break because of licensed implementations, then someone with write bits on asdf/asdf would need to pull the branch from the fork into asdf/asdf first.
Either way, the person pulling into asdf/asdf would have to do a sanity check to make sure the license isn't being exfiltrated, but I don't think that's any change from the previous approach (unless you were previously running the tests without external network access!).
Thanks for the suggested way forward on Allegro/LispWorks. We (i.e. CLF) certainly want to be fairly thorough on protecting the licenses for the commercial implementations. If we do a good enough job with ASDF, we would like to try to provide them as a potential service for any gitlab.common-lisp.net project.