- clo-devel - mailman3.common-lisp.net

Outage of gitlab.common-lisp.net
by Erik Huelsmann 17 Apr '22

17 Apr '22

Hi, common-lisp.net GitLab instance is set to maintenance mode since last night due to unexpected outage: around 19.20 CEST, phoe approached me on the common-lisp.net:matrix.org chat channel (a.k.a. #common-lisp.net:libera.chat) about the unavailability of the with-contexts project. Logging into the system, it quickly became apparent that one of the discs had filled up, causing this failure. After a bit of research, it became apparent that some 23GB (less than 10% of space on the volume) of disc space was taken by Prometheus. A tool we're not using, but which comes out of the box with GitLab. Disabling Prometheus and removing its files quickly freed up enough space for basic web requests to work again. To have some more room for various processes to operate with, I'm using a maximum of 80% fill-rate for the volumes in the VM. So, I went looking for more possibilities to clean out storage. At one point, I ended up GitLab's PostgreSQL directory, where there was a little more than a GB of storage to be won. Not a lot, but since I was cleaning anyway, it seemed like a good thing to look at. There were clearly old Pg clusters (various Pg10 and Pg11 clusters while we were running on 12). There also was a script called "delete_old_clusters.sh". It seemed better to use a script from the vendor than meddling with the database data myself, so I used it. HOWEVER: it immediately and without warning *removed the production database* (contrary to the expectation that it would remove the *old* clusters laying around)! Although this is rather unfortunate, series of events, I quickly recovered from the heart attack that followed; turned off as many services on the machine as possible and searched (a) for older database copies and (b) for dumped backups. Unfortunately, misforture never comes alone: as soon as I found the backup, I realized it's from Feburary 27th. The backup system that had been running without problems for *years* had stopped running after March 1st and none of the current maintainers noticed: since the backup procedure didn't generate an error, but was plainly not executed, there were no mails about backups failing. On top of that, it turns out that the system I have in place to report disk usage problems, wasn't delivering messages of the common-lisp.net disk overage either! I've restored the database backup from the 27th and we have the backup procedure running again. This means that anything stored in the database is back to the 27th. MRs, issues, etc. The *repositories* are fine and never were in danger! So far, I've waited to enable the service, because I've contacted the #gitlab:libera.chat channel to ask if anything can be done to assure consistency between the repositories and the database. So far, the channel has remained silent (not just to my question, but to any questions posed). I am thinking to restore access on monday night CEST, if no answer appears on the gitlab channel, or as much earlier as a usable answer will be provided. Let me close off this mail by offering my sincere apologies for failing the trust you have put in me and for any inconvenience this may have caused. Please report any inconsistencies you run into to admin at common-lisp.net so we can work on fixes. Additional controls are being worked on to prevent a similar situation in future: "ping" messages from the monitoring infrastructure and checks on the off-site backup system to check that the weekly full backup (and the daily incrementals) have been delivered. -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

3 2

Test
by Marco Antoniotti 15 Apr '22

15 Apr '22

... is this getting through? -- Marco Antoniotti, Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY

2 1

CL-PLOT repository disappeared
by Marco Antoniotti 12 Apr '22

12 Apr '22

Hi I cannot access anymore my repository CL-PLOT. It seems like it disappeared.... Can you check what happened? All the best Marco -- Marco Antoniotti, Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY

1 0

Download directory no longer accessible, breaking cmucl CI
by Raymond Toy 28 Mar '22

28 Mar '22

For a long time, we had release and snapshot tarballs at https://common-lisp.net/project/cmucl/downloads/. But this now no longer works and a 403 Forbidden error is returned. This also breaks cmucl's CI that pulls snapshot tarballs for running CI. According to https://gitlab.common-lisp.net/cmucl/cmucl/-/pipelines, it was working a week ago, but has now stopped working. I have no problem updating docs and CI to use a new path, but I don't know what it would be. (It still seems reachable via rsync, though.) -- Ray

1 0

Here we go again! 2FA vs Gitlab instructions.
by Marco Antoniotti 27 Mar '22

27 Mar '22

Hi I am trying to import a repo in Gitlab. I just created the repo using the web interface (CL-PLOT). This is what the instructions on the site say. cd existing_repo git remote rename origin old-origin git remote add origin git@common-lisp.net:mantoniotti/cl-plot.gitgit push -u origin --all git push -u origin --tags Once I try git push I get the dreaded message. $ git push -u origin --all git(a)common-lisp.net: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. Why? I.e., why doesn't git understand the 2FA I set up a long time ago? Why I cannot get my SSH keys to work? How can I change the remote spec in order not to have to go through this every other month? Thank you All the best Marco -- Marco Antoniotti, Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY

2 2

Please remove Elias Pipping from ASDF project
by Robert Goldman 04 Mar '22

04 Mar '22

Sadly, Elias (`epipping`) passed away about 5 years ago. He still gets emails that go to his mother, who finds them a little distressing. I cannot remove him from ASDF because he is listed as an Owner on the project. Probably it would be best to shut down his common-lisp.net account. Best, Robert

2 1

Gitlab authetication
by Marco Antoniotti 04 Feb '22

04 Feb '22

Hi I have a repository I have not touched in a while and now I cannot push to gitlab. I use Sourcetree. I have TFA on and a personal token. I also have ssh public keys uploaded, but I cannot find the right incantation to set things up. I understand there is something I must do with the personal token, but what exactly? The Gitlab instructions are a bit cryptic for me. Please advise. Thanks. -- Marco Antoniotti, Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY

2 6

Ownership of CDR project/repos
by Marco Antoniotti 21 Jan '22

21 Jan '22

Hi I am listed as a developer for the CDR project/repos ( https://gitlab.common-lisp.net/cdr/cdr.common-lisp.dev/-/project_members) Could you list me as owner or maintainer? Thanks Marco -- Marco Antoniotti, Professor tel. +39 - 02 64 48 79 01 DISCo, Università Milano Bicocca U14 2043 http://dcb.disco.unimib.it Viale Sarca 336 I-20126 Milan (MI) ITALY

1 0

Pages not deploying?
by Raymond Toy 19 Jan '22

19 Jan '22

Not sure what happened but the pages for cmucl-site aren't actually deploying. For example, the pipeline https://gitlab.common-lisp.net/cmucl/cmucl-site/-/pipelines/5701 says the pages were deployed. If you download the artifacts and look at public/doc/index.html, you'll see that there's an entry for Hemlock docs. But when I visit https://common-lisp.net/project/cmucl/doc/index.html, the entry for the Hemlock docs is not there. This used to work, but I'm not sure what's changed. Also, a side note. If you browse the archive and navigate to public/doc and click on index.html, the URL is http://cmucl.pages.example.com/-/cmucl-site/-/jobs/26695/artifacts/public/d…, which doesn't exist. What can I do to make this work? -- Ray

5 18

Simplifying hosting services (aka "breaking up the monolith")
by Erik Huelsmann 13 Jan '22

13 Jan '22

Hi, In 2021, common-lisp.net has had two types of service interruptions: (1) mailing lists were not functioning correctly [while regular mail traffic seemed to be] an (2) deployment of project sites wasn't working. The latter only came to light recently, but seems to have existed for some time. Since I can remember (probably since the creation of the service) has common-lisp.net hosted its project webpages on its main domain in a sub directory, which at times has been called /project/, or /projects/ or even /p/. This setup mixes project hosting with hosting for the main site itself and restricts the tooling we can use to host the projects' websites. The reason for the sites not deploying well is that I've implemented a workaround in the past to be able to deploy projects using GitLab Pages while our deployment model (deploying to the /project/ subdirectory on the main domain) is out of line with what GitLab Pages were designed for. At the same time, our configuration is running with extensive sets of rewrite rules to keep historic URLs "working" and redirected to (hopefully) existing current URLs, which also extremely complicates our setup. In order to simplify our setup (and eliminate the deployment problems we're experiencing) and at the same time add support for requests like those from Marco who wants to be able to deploy sites for multiple projects under the same umbrella, I've decided I want to move projects' sites to their own (sub)domains. In the past I thought this would need to be subdomains along the lines of https://<project>.project.common-lisp.net/. Although that's a deployment model that would work with GitLab Pages, I've decided against it. I'll move the hosted projects to: https://<project>.common-lisp.dev/ (I acquired common-lisp.dev for this purpose this morning.) The setup I'm proposing is that we have a good look at the tons of rewrite rules we have currently in place and clean up the rewrite rules that we don't need anymore. Then, we create rewrite rules for the current project namespaces at https://common-lisp.net/project/<project>/ to map to https://<project>.common-lisp.dev/. There are a few projects which aren't using GitLab Pages to deploy their websites yet, mostly because they have no active maintainers. These projects will keep being served from their current locations on the filesystem of the common-lisp.net host, but their content will be available through the new <project>.common-lisp.dev URL space. Further change proposals in order to separate the mail flow for the mailing lists from the regular mail flow (and thereby further reducing integration between components) are upcoming, but I'll need to address one thing at a time (due to time constraints). Are there any comments, remarks, additions, things you want me to take into account with respect to the above? -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

7 9