clo-devel September 2017

clo-devel@common-lisp.net

5 participants
3 discussions

GitLab.common-lisp.net unavailability: post-mortem
by Erik Huelsmann 01 Oct '17

01 Oct '17

Hi, Last weekend and up to Wednesday, gitlab.common-lisp.net had issues, returning 500 Internal Server Errors while cloning or pulling; additionally the gitlab subdomain was down completely on Sunday. This mail provides an analysis of what happened. There is some context to all this to be started with: common-lisp.net uses the so-called "omnibus" package to run its GitLab install; it's a batteries-included package provided by GitLab, meaning that everything down to OpenSSL, nginx and Ruby are included in the package and installed in a separate - not interfering with the system - location. This omnibus package also comes with its own configuration (script) in the form of a Chef recipe. While the package provides a default configuration which uses an Nginx reverse proxy and default ports for daemons to be accessed over TCP sockets, this default configuration doesn't quite wok on common-lisp.net due to the fact that we use Apache 2.4 as our web-visible reverse proxy. Apache 2.4 also serves a truckload of other services, such as lisppaste, trac, abcl.org, cliki.net, darcsweb.cgi, etc. Due to this entanglement of Apache, we can't just replace it with nginx. Also, due to the large number of reverse-proxied services, not all standard ports for GitLab's configuration are open. This isn't a problem, because GitLab offers the ability to configure site-local deviations from the defaults configuration as input for the Chef recipe. We have succesfully been running with a configuration like this since GitLab 7.(something). The current GitLab version is 10.0. In its evolution from version 7 to version 10, gitlab started out with "Unicorn" based rails workers (a standard Rails setup). As demand grew, a custom webserver was developed (gitlab-git-http-server) which addressed Unicorn time-outs with long running "git" processes (clones). In order to support the "simple" setup with just Unicorn, the unicorn and gitlab-git-http-servers were configured to run each on their own port. Around GitLab 8.2, gitlab-git-http-server was renamed to gitlab-workhorse and the configuration keys were renamed with it, although the old config keys were still respected. Our local override contained these gitlab-git-http-server config keys last Sunday due to ports already being taken by other services. As of version 10, the 'gitlab-git-http-server' configuration keys are no longer supported: the configuration *must* now be specified in terms of 'gitlab-workhorse' keys. Last Sunday, when I upgraded the system to the current version (10.0.2) in the morning, I missed this fact, which caused the system to remain unconfigured (and thus unavailable) until I received notification on #common-lisp.net of problems. The cause at that time was quickly determined and the 'gitlab-git-http-server' configuration keys were quickly removed and the system was redeployed and all seemed to work again, after changing the reverse proxy rules to point to gitlab's remaining open ports. On Monday I received more signals of problems; being on a conference with little to no Net access, Mark pitched in, but was unable to determine the cause. When I *did* have access, everything looked fine, so I didn't check any further. Then on Tuesday, I received more signals of problems, but being on the same conference without Net access, still, I wasn't able to do much. On Wednesday morning, with yet more reports of problems, it became apparent that I was checking the web frontend for availability, but that the people reporting issues were actually experiencing problems with clones/pulls/etc. So, the git-over-http component wasn't working. With the actual problem identified and reproduced, it was quickly apparent that due to the removal of the gitlab-git-http-server config keys, 'gitlab-workhorse' was no longer being configured and started. With a bit of trial-and-error, it also turned out that gitlab-workhorse has a default configuration to run over Unix domain sockets; a configuration supported by Nginx, but not by Apache. With the configuration corrected and the system reconfigured, problems were solved by Wednesday noon. In retrospect, the removal of these config keys was in the release notes, so I could have known. It was 22 <PgDown> clicks down, by which time I wasn't alert enough to realise the importance of the deprecation announcement. Regards, -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

2 1

RFC: Future of lisppaste
by Erik Huelsmann 16 Sep '17

16 Sep '17

Hi, In the early days of web popularity, a very nice irc-bot+paste engine was developed, allowing discussion on IRC to be very nicely combined with paste sharing to help the discussion. It was written in lisp and given the nice name of "lisppaste". While the idea was great to start out, in its grand days, the bot even joined over 60 channels(!), it has gradually seen increases in abuse; initially "just" to spam IRC channels (which caused the bot to be removed from all channels, loosing most of its usefulness), later with lots of inappropriate content being posted. Various measures were taken over time to reduce the attractiveness of posting to lisppaste: * disabling of the XML-RPC api (which integrated with emacs!) * removal from the IRC channels * addition of a captcha up to most recently even: * removal of the listing of posts While the last measure was hoped to take care of the last incentive to post to lisppaste, it seems that links to the inappropriate content are posted elsewhere -- eliminating the need to list the pastes. With the majority of pastes being irrelevant to lisppaste's purpose (supporting programmers discussing their code in IRC or other channels), because they're inappropriate content of some kind or another, my question to you all is: What should we do with lisppaste? Should we simply remove it from the web? Should we put it in read-only mode* (so as to maintain the archive of relevant pastes)? Or are there other options? E.g. someone who wants to adopt lisppaste and invest the time and energy to implement measures to fight the spam and implement measures to discourage or disable posting of inappropriate content? * If we put it in read-only mode, it's my opinion that the existing inappropriate content should still be cleaned. Any contributions to that effect would still be greatly appreciated. -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

3 6

Re: Maintenance of closure-html
by Erik Huelsmann 06 Sep '17

06 Sep '17

Hi David, Thanks for the quick response! Elias, looks like you got the job! Could you send us your SSH public key (through admin(a)common-lisp.net), please so I can create a user account for you to maintain the project website at common-lisp.net/project/closure-html ? Thanks! Regards, Erik. On Wed, May 3, 2017 at 7:50 PM, David Lichteblau <david(a)lichteblau.com> wrote: > (Cc: Elias) > > Quoting Erik Huelsmann (ehuels(a)gmail.com): > > You are listed as the current maintainer of closure-html. We (the > > common-lisp.net admins) have been approached by Elias Martenson who > wants > > to take over maintenance. > > He could not reach you (received no response?), so I'm trying to reach > out > > and see if you are ready to share or hand over maintenance. > > > > Please let us know what we can do. > > Sure, go ahead! Thanks for reaching out. > > For the future maintainer -- The canonical repo for closure-html used to > be, until now: > http://repo.or.cz/closure-html.git > > Let's instead move it to > https://github.com/bluelisp/closure-html > > where we have started to collect other projects that could use some > affection by other community members. You should have received an > invite to that team on github. > > We can inform Zach about the repo change once everything is in place. > > > d. > -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

3 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

clo-devel September 2017