Thanks for reaching out , Dave. I hadn't realized that cl.net had the full gitlab CI hooked up to it. Are there any resource issues? There are a lot of projects hosted there.

That said, are there sufficient resources there if we start running a pipeline?

-- Robert P. Goldman

On October 14, 2020 at 07:21:44, Dave Cooper (david.cooper@genworks.com(mailto:david.cooper@genworks.com)) wrote:

Dear Robert,

Are you sure you want to use Jenkins? Gitlab has built-in CI. And the CLF has acquired Allegro CL licenses for a ASDF testing (graciously donated by Franz Inc), and we have a to-do item to attempt the same for LispWorks. If you’re willing, we would like to help set up ASDF testing pipelines as part of common-lisp.net(http://common-lisp.net) infrastructure.

If you have reasons of your own to set up a Jenkins setup and could use access to Allegro and maybe Lispworks licenses for that purpose, please reach out regarding that as well. I am not sure about any Jenkins experience among clf or cl.net(http://cl.net) volunteers, however.

Please advise,

Dave

On Tue, Oct 13, 2020 at 11:06 PM Robert Goldman <rpgoldman@sift.info(mailto:rpgoldman@sift.info)> wrote:

...

Does anyone have expertise configuring Jenkins to run against a remote Gitlab install?

If so, would you please reach out to me? I have finally restored some of my CI support for ASDF, but I don't know how to set up my Jenkins to run against merge requests on Gitlab (if that is even possible).

I have a backlog of patches that I would like to get evaluated and merged, but could use some help with the automation.

Thanks!

-- My Best,

Dave Cooper, david.cooper@gen.works genworks.com(http://genworks.com), gendl.org(http://gendl.org) +1 248-330-2979

https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/146 uses cl.net's shared Gitlab CI runners. It appears that the shared runner can only run one job at a time, so the pipeline that MR introduces takes close to an hour to run (and that's only testing with four implementations). I also suspect no pipelines from other projects were running at the time.

If the CLF or someone else can provide additional runners for ASDF's use, that would be a nice way to cut down on pipeline time and avoid impacting other projects.

However, as much as I love Gitlab CI, the "core" tier of it (what I /believe/ cl.net runs) does have some limitations that, given my limited Jenkins knowledge, a properly configured Jenkins setup wouldn't have.

First, the core tier is unable to test against the merged result of an MR. I don't think that one's a deal breaker as any CI is better than none at all, issues are unlikely if the merge has no conflicts, and any issues would be quickly found after the merge completed (and either fixed or rolled back). Alternatively, ASDF could enforce (via Gitlab) that MRs need to be rebased on to the target branch, effectively making the source branch equivalent to the merged result.

The tougher one to work around is caused by the forking model of contributing code. Forks are unable to use their parent's CI runners and the pipelines for MRs from forks are run in the CI context of the fork. So if ASDF has special runners for the licensed implementations they can't be used until the MR is merged. Additionally, the pipelines before merging will be slow as they'll be using the cl.net shared CI runners.

There are a couple of ways to deal with the second issue, but they all have their drawbacks. The best way would likely be to pull contributions from forks into a separate branch in the parent repo before merging to master. That would allow any ASDF specific runners to do their job, but would require some extra steps from the maintainers. (And contributors would just have to deal with their jobs running slowly before the merge). I'm sure a sufficiently motivated person could even write a Gitlab CI job that auto merged to master from this testing branch if the tests all pass.

-Eric

"Robert P. Goldman" rpgoldman@sift.net writes:

Thanks for reaching out , Dave. I hadn't realized that cl.net had the full gitlab CI hooked up to it. Are there any resource issues? There are a lot of projects hosted there.

That said, are there sufficient resources there if we start running a pipeline?

-- Robert P. Goldman

On October 14, 2020 at 07:21:44, Dave Cooper (david.cooper@genworks.com(mailto:david.cooper@genworks.com)) wrote:

...

Dear Robert,

Are you sure you want to use Jenkins? Gitlab has built-in CI. And the CLF has acquired Allegro CL licenses for a ASDF testing (graciously donated by Franz Inc), and we have a to-do item to attempt the same for LispWorks. If you’re willing, we would like to help set up ASDF testing pipelines as part of common-lisp.net(http://common-lisp.net) infrastructure.

If you have reasons of your own to set up a Jenkins setup and could use access to Allegro and maybe Lispworks licenses for that purpose, please reach out regarding that as well. I am not sure about any Jenkins experience among clf or cl.net(http://cl.net) volunteers, however.

Please advise,

Dave

On Tue, Oct 13, 2020 at 11:06 PM Robert Goldman <rpgoldman@sift.info(mailto:rpgoldman@sift.info)> wrote:

...

Does anyone have expertise configuring Jenkins to run against a remote Gitlab install?

If so, would you please reach out to me? I have finally restored some of my CI support for ASDF, but I don't know how to set up my Jenkins to run against merge requests on Gitlab (if that is even possible).

I have a backlog of patches that I would like to get evaluated and merged, but could use some help with the automation.

Thanks!

-- My Best,

Dave Cooper, david.cooper@gen.works genworks.com(http://genworks.com), gendl.org(http://gendl.org) +1 248-330-2979

I think I'm in the same boat as you, unfortunately. I know it's possible to integrate Jenkins and Gitlab, but I've never set it up myself. A brief search seems to indicate that the Gitlab sanctioned way of doing it is not available in Core (https://docs.gitlab.com/ee/integration/jenkins.html). It looks like there are Jenkins plugins to do it (https://plugins.jenkins.io/gitlab-plugin/), but I don't use Jenkins and have zero way of evaluating the quality.

If you decide to use Gitlab CI with ASDF specific runners for the licensed implementations, I'd recommend giving regular contributors Developer access to the ASDF repo. This would let the people that are most likely going to be impacted by slow test times and lack of licensed implementations the ability to use the ASDF runners by doing all their work in the parent repo. The master branch can then be protected (if it isn't already) and configured to not allow Developers to push to it, so that MRs are still required.

Infrequent or one off contributors would just have to deal with the slow test times and someone with Developer rights would have to merge to some temporary branch to test the licensed implementations before merging to master.

-Eric

"Robert Goldman" rpgoldman@sift.info writes:

Eric,

Thank you very much for this quite helpful, and in depth discussion of the issues. If Jenkins was to be preferable, would it be possible to have the Gitlab CI trigger an external Jenkins job? I believe that this is possible, because I am working on a large program (hosted on someone else's site) where the code is hosted on GitLab, but the tests are run on a separate Jenkins server, and the two are integrated using Gitlab's pipelines.

Best, R

On 14 Oct 2020, at 10:50, Eric Timmons wrote:

...

https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/146 uses cl.net's shared Gitlab CI runners. It appears that the shared runner can only run one job at a time, so the pipeline that MR introduces takes close to an hour to run (and that's only testing with four implementations). I also suspect no pipelines from other projects were running at the time.

If the CLF or someone else can provide additional runners for ASDF's use, that would be a nice way to cut down on pipeline time and avoid impacting other projects.

However, as much as I love Gitlab CI, the "core" tier of it (what I /believe/ cl.net runs) does have some limitations that, given my limited Jenkins knowledge, a properly configured Jenkins setup wouldn't have.

First, the core tier is unable to test against the merged result of an MR. I don't think that one's a deal breaker as any CI is better than none at all, issues are unlikely if the merge has no conflicts, and any issues would be quickly found after the merge completed (and either fixed or rolled back). Alternatively, ASDF could enforce (via Gitlab) that MRs need to be rebased on to the target branch, effectively making the source branch equivalent to the merged result.

The tougher one to work around is caused by the forking model of contributing code. Forks are unable to use their parent's CI runners and the pipelines for MRs from forks are run in the CI context of the fork. So if ASDF has special runners for the licensed implementations they can't be used until the MR is merged. Additionally, the pipelines before merging will be slow as they'll be using the cl.net shared CI runners.

There are a couple of ways to deal with the second issue, but they all have their drawbacks. The best way would likely be to pull contributions from forks into a separate branch in the parent repo before merging to master. That would allow any ASDF specific runners to do their job, but would require some extra steps from the maintainers. (And contributors would just have to deal with their jobs running slowly before the merge). I'm sure a sufficiently motivated person could even write a Gitlab CI job that auto merged to master from this testing branch if the tests all pass.

-Eric

"Robert P. Goldman" rpgoldman@sift.net writes:

...

Thanks for reaching out , Dave. I hadn't realized that cl.net had the full gitlab CI hooked up to it. Are there any resource issues? There are a lot of projects hosted there.

That said, are there sufficient resources there if we start running a pipeline?

-- Robert P. Goldman

On October 14, 2020 at 07:21:44, Dave Cooper (david.cooper@genworks.com(mailto:david.cooper@genworks.com)) wrote:

...

Dear Robert,

Are you sure you want to use Jenkins? Gitlab has built-in CI. And the CLF has acquired Allegro CL licenses for a ASDF testing (graciously donated by Franz Inc), and we have a to-do item to attempt the same for LispWorks. If you’re willing, we would like to help set up ASDF testing pipelines as part of common-lisp.net(http://common-lisp.net) infrastructure.

If you have reasons of your own to set up a Jenkins setup and could use access to Allegro and maybe Lispworks licenses for that purpose, please reach out regarding that as well. I am not sure about any Jenkins experience among clf or cl.net(http://cl.net) volunteers, however.

Please advise,

Dave

On Tue, Oct 13, 2020 at 11:06 PM Robert Goldman <rpgoldman@sift.info(mailto:rpgoldman@sift.info)> wrote:

...

Does anyone have expertise configuring Jenkins to run against a remote Gitlab install?

If so, would you please reach out to me? I have finally restored some of my CI support for ASDF, but I don't know how to set up my Jenkins to run against merge requests on Gitlab (if that is even possible).

I have a backlog of patches that I would like to get evaluated and merged, but could use some help with the automation.

Thanks!

-- My Best,

Dave Cooper, david.cooper@gen.works genworks.com(http://genworks.com), gendl.org(http://gendl.org) +1 248-330-2979

Erik Huelsmann ehuels@gmail.com writes:

Hi,

On Wed, Oct 14, 2020 at 5:51 PM Eric Timmons etimmons@mit.edu wrote:

...

https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/146 uses cl.net's shared Gitlab CI runners. It appears that the shared runner can only run one job at a time, so the pipeline that MR introduces takes close to an hour to run (and that's only testing with four implementations). I also suspect no pipelines from other projects were running at the time.

I've increased the number of jobs this runner can run concurrently to 3, because the VM currently has 3 cores configured. If we need to configure more, let me know and I'll have a look at it.

Awesome! That seems to have helped, just not as much as I had hoped (down to 45 minutes). I think at this point the issue is there's one job that takes a long time (ABCL upgrade tests @ 33 minutes) and it's one of the later jobs to run. I'll talk about that more in another reply, though.

...

If the CLF or someone else can provide additional runners for ASDF's use, that would be a nice way to cut down on pipeline time and avoid impacting other projects.

There is also a machine provisioned for cl-test-grid. It's idle often, just not when a new Quicklisp release has been released. If cl-test-grid could be driven by GitLab CI, it would be easy to share the resources this machine has to spare. In that case, I think 2 extra cores would be available for running tests. It would be even better if others would step up to donate their cores.

...

However, as much as I love Gitlab CI, the "core" tier of it (what I /believe/ cl.net runs) does have some limitations that, given my limited Jenkins knowledge, a properly configured Jenkins setup wouldn't have.

First, the core tier is unable to test against the merged result of an MR.

I'm not sure what you mean by this. Do you mean that the CI wouldn't run tests/builds on the master branch after merging an MR?

What I do know that works: The current common-lisp.net site is being built after each merge to the master branch of that repository and the artifacts of the build used to publish the site.

Sorry, I should have been more clear. I meant you can't test against the merged result before actually performing the merge (this feature: https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_r...). Again, I don't think it's critical to have, but it is super annoying if a test passes for the MR but then immediately breaks once it hits master.

...

I don't think that one's a deal breaker as any CI is better than none at all, issues are unlikely if the merge has no conflicts, and any issues would be quickly found after the merge completed (and either fixed or rolled back). Alternatively, ASDF could enforce (via Gitlab) that MRs need to be rebased on to the target branch, effectively making the source branch equivalent to the merged result.

The tougher one to work around is caused by the forking model of contributing code. Forks are unable to use their parent's CI runners and the pipelines for MRs from forks are run in the CI context of the fork. So if ASDF has special runners for the licensed implementations they can't be used until the MR is merged. Additionally, the pipelines before merging will be slow as they'll be using the cl.net shared CI runners.

If we can disable artifacts on the ASDF runners, we might be able to use a somewhat more liberal assignment of runners to project members: if the artifacts can't be extracted, the options for abuse will become more limited.

I haven't looked into this a lot, but I think it's *possible* to disable artifact uploads for certain runner configurations (basically any configuration that uses Docker images). It's nasty, though, and would require overriding the helper image used to set up and tear down the testing environment.

It's also not clear to me that blocking artifact extraction would be sufficient (assuming you're concerned about exfiltrating license info). It's been a while since I had an Allegro license, but IIRC, it's a relatively small plain text file that could just be printed into the job log or curl'ed away to some remote computer.

...

There are a couple of ways to deal with the second issue, but they all have their drawbacks. The best way would likely be to pull contributions from forks into a separate branch in the parent repo before merging to master. That would allow any ASDF specific runners to do their job, but would require some extra steps from the maintainers. (And contributors would just have to deal with their jobs running slowly before the merge). I'm sure a sufficiently motivated person could even write a Gitlab CI job that auto merged to master from this testing branch if the tests all pass.

That would work too. Let me know what more we can research or I can help with to help solve this issue.

...

-Eric

Regards,

Erik.

On Oct 14, 2020, at 18:58, Jason Miller jason@milr.com wrote:

I have very little experience with gitlab, but I do have have experience with configuring Jenkins, and the gitlab-plugin looks reasonably straightforward as these things go. If nobody else chimes-in I can probably get it setup.

I think in order to get ASDF CI back to running ASAP, we should host a special Jenkins instance on common-lisp.net. Once ASDF is back to a more stable development cycle with CI like it was working, we can refactor the knowledge for commerical instances into the more generic Gitlab runner. Also, we would be free to hand the ASDF Jenkins instance as much oompf as it needs to return results in an acceptable wall-clock time.

As such, I would like to take Jason up on his kind offer.

@jason: could you do a Jenkins instance as a Docker container? This would be the fastest route to getting something working in the CLF infrastructure.

It would be great to coordinate things on the Freenode #common-lisp.net channel. For synchronous response from me, I will be working with Dave Cooper there tomorrow, Thursday, October 15 at 0800 UTC; otherwise just leave a message with @Colleen, and we’ll get back to ya.

yers in CONS, Mark

On Oct 14, 2020, at 19:34, Robert Goldman rpgoldman@sift.info wrote:

That would be great. I have a Jenkins set up on a multi-core box at SIFT that for now runs only Allegro, SBCL and CCL. It's hard for me to make it accessible to anyone else, though, because Jenkins is such a security nightmare: we keep access only to our VPN.

Are all the artifacts necessary to run the Jenkins pipeline in the ASDF repository? I haven’t used Jenkins in something like a decade, so I have long forgotten anything I may have remembered, but will start figuring things out tomorrow.

Running on Allegro, SBCL, CCL, ECL, and ABCL should be quite doable. I have yet to hit LispWorks up for a license, but will make the appropiate polite request when I know more about how to keep the licensing secure for Allegro in the Jenkins instance.

We should be able to provide somewhat of a progress report tomorrow around this time as to how far we actually got to running the ASDF CI under Jenkins.

best, Mark

"Robert Goldman" rpgoldman@sift.info writes:

In mine, I simply set the environment variables `ASDF_TEST_LISPS` and the variables that point to the lisp implementations. Then I call `make test-all-no-upgrades-no-stop`.

Just to clarify, it sounds from this that you don't actually care if the upgrade tests are run? Do you ever want to run them via CI? As it currently stands they're the slowest jobs, so finding some way to reduce the number of times they're run would be a win.

-Eric

On 16 Oct 2020, at 13:31, Mark Evenson wrote:

The CLF advocates a merge of [!146][] as we then we can adjust runners as needed for capacity and capability. The default runners are Linux. Eric Timmons is setting up a macOS runner which can be used. Windows runners are further in the future.

Agreed. I have merged 146.

On 16 Oct 2020, at 13:13, Eric Timmons wrote:

"Robert Goldman" rpgoldman@sift.info writes:

...

In mine, I simply set the environment variables `ASDF_TEST_LISPS` and the variables that point to the lisp implementations. Then I call `make test-all-no-upgrades-no-stop`.

Just to clarify, it sounds from this that you don't actually care if the upgrade tests are run? Do you ever want to run them via CI? As it currently stands they're the slowest jobs, so finding some way to reduce the number of times they're run would be a win.

I run them rarely, and usually only close to a release. Or sometimes I run them just on SBCL, hoping that because SBCL tries to hold tight to the standard, it will be a good diagnostic tool.

But note that I typically run with an up-to-date version in my daily work, so I get a chance to see if what's shipping in ACL, SBCL, and CCL is going to cause problems with the upgrade (e.g., I'm not sure that CCL 11.1 is still upgradeable).

I think we could stick with the normal tests, and run upgrade rarely (maybe weekly at a time that's likely to see low load at cl.net?)

On 15 Oct 2020, at 12:19, Mark Evenson wrote:

@Robert: would it be satisfactory to have the output of the ASDF tests available in a non-Jenkins environment? Do we need to run the ADSF tests under OSs other than Linux, that is do you current test other environment likes macOS/Windows/FreeBSD?

I got started with Jenkins only because my company had it, and so I was familiar with it, but hadn't used the GitLab/GitHub frameworks. I'm not ready to configure GL/GH style CI myself, beyond crude bug fixing, but I am quite comfortable with it. Indeed, I believe it's simpler, because it can be merge request aware, but doing that with Jenkins requires additional work.

[One cranky request --- if we set this up in that way, please use `gitlab-ci.yml` or some other *visible* filename, instead of the dotted filenames that are hidden by default. I'm not sure why people think hiding critical configurations is a good idea, but I don't. Thanks!]

I currently run the ASDF tests on MacOS, but not using Jenkins, only doing it by hand (since that's what my laptop runs). So running MacOS automatically would be a nice-to-have, but is not critical.

I currently *don't* run the ASDF tests on Windows, but I should, since it's such a different platform. Dave Cooper has kindly offered me access to a Windows VM for testing, but I haven't gotten around to setting it up. It would be lovely if we could hook Windows testing into whatever CI framework we end up with.

On Oct 16, 2020, at 06:21, Eric Timmons etimmons@mit.edu wrote:

[…]

I'm a huge fan of Gitlab CI and was getting tired of running all tests locally whenever I made a change, so I started a Gitlab CI configuration for ASDF at https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/146 and also modified it a bit more today. The recent modifications are marking the upgrade tests as `allow_failure: true` (since in a previous email it sounded like the previous Jenkins config didn't even run them), increasing some timeouts in tests (I don't know what happened, but some tests started failing under CCL when run on cl.net runners), reordering things such that all regression tests ran first, and addressing a bug in the latest SBCL that was triggered by the cl.net infrastructure.

If you want to take some radically different approach than I did, that's great! I just wanted to make sure everyone knew that an 80% solution exists and didn't waste their time coming up with something substantially similar. I think the biggest things left on !146 are deciding how often to run the tests (every push? only on MRs?) and adding more runners for licensed implementations or OSes.

Wow! This looks like much more than 80%, and its working today!

Speaking of OSes, my research group has a Mac mini we've been using for our own CI purposes. By no means is it a speed demon, but it's lightly used so I can set up another runner on it for ASDF.

As for the licensed implementations: I'm not sure of what the cl.net infrastructure looks like and how paranoid you want to be about protecting the license, but it seems the easiest way forward would be to add a new VM based runner with the implementation and license already installed in the VM, register that runner so that only the asdf/asdf project can use it, and tag it with the implementation name. We can then add a job that is triggered only if it's run in the asdf/asdf project and have it require that tag.

As I said in another message, this would mean that forks can't use that runner (and MR tests are, unfortunately, run in the CI context of the fork). If it's OK for master to occasionally break, the changes from forks could be merged directly to master and breakage dealt with after that. If you want master to never break because of licensed implementations, then someone with write bits on asdf/asdf would need to pull the branch from the fork into asdf/asdf first.

Either way, the person pulling into asdf/asdf would have to do a sanity check to make sure the license isn't being exfiltrated, but I don't think that's any change from the previous approach (unless you were previously running the tests without external network access!).

Thanks for the suggested way forward on Allegro/LispWorks. We (i.e. CLF) certainly want to be fairly thorough on protecting the licenses for the commercial implementations. If we do a good enough job with ASDF, we would like to try to provide them as a potential service for any gitlab.common-lisp.net project.