common-lisp.net http stopped running
The apache2 instance seems to have stopped at 0010 this evening. Seems to be something about a bad SSL configuration due to a package containing empty certs being deleted Looking into it on the machine; sending email in case their are others working on this as well. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 15, 2023, at 09:06, Mark Evenson <evenson@panix.com> wrote:
The apache2 instance seems to have stopped at 0010 this evening.
Web services for common-lisp.net <http://common-lisp.net/> have been restored.
Seems to be something about a bad SSL configuration due to a package containing empty certs being deleted
This was a bogus diagnosis. From what I could make of the logs, the apache2 instance which runs all of our user-facing HTTP(S) traffic quit serving shortly after midnight UTC. The cause is unclear. I had to start/restart apache2 a couple times to clear out resources. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Apache itself is rather robust in my experience. I'm suspecting any of the backend services that could be tying Apache resources as part of the proxy process. Are we sure all the backends are timely serving the requested responses? And that nobody is flooding Apache with requests causing backend contention? On Wed, Nov 15, 2023, 09:32 Mark Evenson <evenson@panix.com> wrote:
On Nov 15, 2023, at 09:06, Mark Evenson <evenson@panix.com> wrote:
The apache2 instance seems to have stopped at 0010 this evening.
Web services for common-lisp.net <http://common-lisp.net/> have been restored.
Seems to be something about a bad SSL configuration due to a package containing empty certs being deleted
This was a bogus diagnosis.
From what I could make of the logs, the apache2 instance which runs all of our user-facing HTTP(S) traffic quit serving shortly after midnight UTC. The cause is unclear.
I had to start/restart apache2 a couple times to clear out resources.
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Unfortunately, this is nothing new and has happened on several occasions over the past weeks. I have no idea what could be causing this issue, but could it be related to reverse proxying paste.lisp.org which is no longer serving pages? Or, if the server got restarted at some point (I turned it off earlier as I thought this to be the main attributing factor), I'm pretty sure it is related due to failure to actually serve the requests? On Wed, Nov 15, 2023, 09:08 Mark Evenson <evenson@panix.com> wrote:
The apache2 instance seems to have stopped at 0010 this evening.
Seems to be something about a bad SSL configuration due to a package containing empty certs being deleted
Looking into it on the machine; sending email in case their are others working on this as well.
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 15, 2023, at 12:29, Erik Huelsmann <ehuels@gmail.com> wrote:
Unfortunately, this is nothing new and has happened on several occasions over the past weeks. I have no idea what could be causing this issue, but could it be related to reverse proxying paste.lisp.org which is no longer serving pages?
I missed the retirement of paste.lisp.org <http://paste.lisp.org/>: it was running read-only for quite a while. Was a decision made to shut it down permanently? If paste.lisp.org <http://paste.lisp.org/> will not be running, we should return a proper HTTP response. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Wed, Nov 15, 2023, 13:39 Mark Evenson <evenson@panix.com> wrote:
On Nov 15, 2023, at 12:29, Erik Huelsmann <ehuels@gmail.com> wrote:
Unfortunately, this is nothing new and has happened on several occasions over the past weeks. I have no idea what could be causing this issue, but could it be related to reverse proxying paste.lisp.org which is no longer serving pages?
I missed the retirement of paste.lisp.org <http://paste.lisp.org/>: it was running read-only for quite a while. Was a decision made to shut it down permanently?
No, that decision made itself: the ever since the bots have been kicked off Libera, the server has been failing to serve lisppaste as well. The only thing I did after months of failure (which I did note only in arrears; I wasn't actively looking at these failures happening), I took the sbcl out of its misery and just shut it down.
If paste.lisp.org <http://paste.lisp.org/> will not be running, we should return a proper HTTP response.
I/we did not get a single reaction over it failing for months. I understand your point but given the limited availability to work on the maintenance of the server as it is in combination with the lack of complaints, even on the regular libera channels *and* nobody to step up even to fix the situation with the bots, I'm not sure there is enough priority and capacity anywhere to work on it... So then there is no point? Things do get lost on the web... Ideally since it was serving read-only anyway, we transform the content to their read-only pages and host just the static content. But even that takes time...
-- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 15, 2023, at 13:55, Erik Huelsmann <ehuels@gmail.com> wrote:
On Wed, Nov 15, 2023, 13:39 Mark Evenson <evenson@panix.com> wrote:
On Nov 15, 2023, at 12:29, Erik Huelsmann <ehuels@gmail.com> wrote:
Unfortunately, this is nothing new and has happened on several occasions over the past weeks. I have no idea what could be causing this issue, but could it be related to reverse proxying paste.lisp.org which is no longer serving pages?
I missed the retirement of paste.lisp.org <http://paste.lisp.org/>: it was running read-only for quite a while. Was a decision made to shut it down permanently?
No, that decision made itself: the ever since the bots have been kicked off Libera, the server has been failing to serve lisppaste as well. The only thing I did after months of failure (which I did note only in arrears; I wasn't actively looking at these failures happening), I took the sbcl out of its misery and just shut it down.
I heard about the “libera.chat bans bots” ambiently over the last couple months. I just had the following conversation in #libera-bots <easye> Mornin' all. I am trying to get some up-to-date information about the status of bots being allowed on libera chat networks. Over the past couple months, I've heard ambiently that "bots are now banned", but according to <https://libera.chat/guides/faq#are-bots-allowed> bots *are* allowed. [06:14] <easye> Has the policy outlined in the libera faq ("bots allowed with permission from channel operator") been changed? [06:19] <moonmoon> hasn't [06:21] <moonmoon> whether a bot is allowed in a channel is up to the ops of that channel <easye> moonmoon: thank you for the clarification. [06:22] <moonmoon> so if you heard that bots are banned from a particular channel, it means that. not a network-wide thing So, it must be that the operator of #commonlisp that banned the bot? Who is that at this point? Anybody have insight into whatever drama is -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On 16.11.2023 07:28, Mark Evenson wrote:
So, it must be that the operator of #commonlisp that banned the bot? Who is that at this point?
I'm an operator of #commonlisp on Libera. 07:42 <phoe> akick #commonlisp list 07:42 -ChanServ(ChanServ@services.libera.chat)- AKICK list for #commonlisp: 07:42 -ChanServ(ChanServ@services.libera.chat)- 1: lucie22!*@* (spamming links) [setter: Bike, modified: 1y 28w 1d] 07:42 -ChanServ(ChanServ@services.libera.chat)- Total of 1 entries in #commonlisp's AKICK list. Also, /mode #commonlisp bq gives me 07:40 -!- mode/#commonlisp [+Ccntz] 07:40 -!- Channel #commonlisp created Wed May 19 17:02:46 2021 07:43 -!- 1 - #commonlisp: ban pjb!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 10509119 secs ago] 07:43 -!- 2 - #commonlisp: ban ogamita!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 10509123 secs ago] 07:43 -!- 3 - #commonlisp: ban *!*@2a01:4f8:161:52e2::2 [by Shinmera!~shinmera@shirakumo/shinmera, 10509132 secs ago] 07:43 -!- 4 - #commonlisp: ban *!*@91.150.188.137 [by Bike!~bike@pool-173-75-249-194.phlapa.fios.verizon.net, 24152585 secs ago] 07:43 -!- 5 - #commonlisp: ban paule320!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 28839074 secs ago] 07:43 -!- 6 - #commonlisp: ban lucie22!*@* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- 7 - #commonlisp: ban *!*boar@66.205.193.* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- 8 - #commonlisp: ban *!*roar@66.205.193.* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- #commonlisp End of Channel Ban List 07:43 -!- #commonlisp q End of Channel Quiet List Does the bot in question match any of these entries?
On Nov 16, 2023, at 07:44, Michał phoe Herda <phoe@disroot.org> wrote:
On 16.11.2023 07:28, Mark Evenson wrote:
So, it must be that the operator of #commonlisp that banned the bot? Who is that at this point?
I'm an operator of #commonlisp on Libera.
07:42 <phoe> akick #commonlisp list 07:42 -ChanServ(ChanServ@services.libera.chat)- AKICK list for #commonlisp: 07:42 -ChanServ(ChanServ@services.libera.chat)- 1: lucie22!*@* (spamming links) [setter: Bike, modified: 1y 28w 1d] 07:42 -ChanServ(ChanServ@services.libera.chat)- Total of 1 entries in #commonlisp's AKICK list.
Also, /mode #commonlisp bq gives me
07:40 -!- mode/#commonlisp [+Ccntz] 07:40 -!- Channel #commonlisp created Wed May 19 17:02:46 2021 07:43 -!- 1 - #commonlisp: ban pjb!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 10509119 secs ago] 07:43 -!- 2 - #commonlisp: ban ogamita!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 10509123 secs ago] 07:43 -!- 3 - #commonlisp: ban *!*@2a01:4f8:161:52e2::2 [by Shinmera!~shinmera@shirakumo/shinmera, 10509132 secs ago] 07:43 -!- 4 - #commonlisp: ban *!*@91.150.188.137 [by Bike!~bike@pool-173-75-249-194.phlapa.fios.verizon.net, 24152585 secs ago] 07:43 -!- 5 - #commonlisp: ban paule320!*@* [by Shinmera!~shinmera@shirakumo/shinmera, 28839074 secs ago] 07:43 -!- 6 - #commonlisp: ban lucie22!*@* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- 7 - #commonlisp: ban *!*boar@66.205.193.* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- 8 - #commonlisp: ban *!*roar@66.205.193.* [by helium.libera.chat, 37849598 secs ago] 07:43 -!- #commonlisp End of Channel Ban List 07:43 -!- #commonlisp q End of Channel Quiet List
Does the bot in question match any of these entries?
Thanks! No, the bot doesn’t match any of these that I know of. I am trying to figure out what to do with the old lisp pastebot. We had to shut down adding new pastes due to uncontrolled spam using it for things that the CLF couldn’t assume liability for hosting from what I remember. Thanks again, phoe for clearing things up. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Thu, Nov 16, 2023, 07:29 Mark Evenson <evenson@panix.com> wrote:
On Nov 15, 2023, at 13:55, Erik Huelsmann <ehuels@gmail.com> wrote:
On Wed, Nov 15, 2023, 13:39 Mark Evenson <evenson@panix.com> wrote:
On Nov 15, 2023, at 12:29, Erik Huelsmann <ehuels@gmail.com> wrote:
Unfortunately, this is nothing new and has happened on several occasions over the past weeks. I have no idea what could be causing this issue, but could it be related to reverse proxying paste.lisp.org which is no longer serving pages?
I missed the retirement of paste.lisp.org <http://paste.lisp.org/>: it was running read-only for quite a while. Was a decision made to shut it down permanently?
No, that decision made itself: the ever since the bots have been kicked off Libera, the server has been failing to serve lisppaste as well. The only thing I did after months of failure (which I did note only in arrears; I wasn't actively looking at these failures happening), I took the sbcl out of its misery and just shut it down.
I heard about the “libera.chat bans bots” ambiently over the last couple months.
I just had the following conversation in #libera-bots
<easye> Mornin' all. I am trying to get some up-to-date information about the status of bots being allowed on libera chat networks. Over the past couple months, I've heard ambiently that "bots are now banned", but according to <https://libera.chat/guides/faq#are-bots-allowed> bots *are* allowed. [06:14] <easye> Has the policy outlined in the libera faq ("bots allowed with
permission from channel operator") been changed? [06:19] <moonmoon> hasn't [06:21] <moonmoon> whether a bot is allowed in a channel is up to the ops of that
channel
<easye> moonmoon: thank you for the clarification. [06:22] <moonmoon> so if you heard that bots are banned from a particular channel, it means that. not a network-wide thing
So, it must be that the operator of #commonlisp that banned the bot? Who is that at this point?
Anybody have insight into whatever drama is
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network. When I found them, every connection attempt was answered by the IRC network with a message like "you're banned. Fix your setup." So I expect that this isn't an issue with the libera policy on bots in general neither with the ops of the channels the bots want to join. Thanks for looking into this! Regards, Erik
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting. Then the shell script restarted them...
On Nov 16, 2023, at 08:25, Philipp Marek <philipp@marek.priv.at> wrote:
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting.
Then the shell script restarted them...
Thanks for the information: more than enough for me to debug when I get the time. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
Hi Mark!
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting.
Then the shell script restarted them...
Thanks for the information: more than enough for me to debug when I get the time.
How about moving them to the new host, so that you can use the "newest" or at least _more_ current version of SBCL??
On Nov 16, 2023, at 08:30, Philipp Marek <philipp@marek.priv.at> wrote:
Hi Mark!
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network. Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting. Then the shell script restarted them... Thanks for the information: more than enough for me to debug when I get the time.
How about moving them to the new host, so that you can use the "newest" or at least _more_ current version of SBCL??
Yes, I will do that as well. It seems that mevenson has access to gitlab-future.common-lisp.net <http://gitlab-future.common-lisp.net/>. Is that the host we’re configuring? It will have more than just gitlab running on it, correct? I’ve been a little out of the loop on the move. Did anyone keep any notes on what needs to be done? I can also help with configuring postfix, -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
I think one of the more urgent bots is the one on #common-lisp.net which authenticates new signups for our gitlab. Dave Cooper ---- On Thu, 16 Nov 2023 02:27:32 -0500 Mark Evenson <evenson@panix.com> wrote ---
On Nov 16, 2023, at 08:25, Philipp Marek <mailto:philipp@marek.priv.at> wrote:
I can shed a bit of light on that: the bots were alledgedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting.
Then the shell script restarted them...
Thanks for the information: more than enough for me to debug when I get the time. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 16, 2023, at 08:25, Philipp Marek <philipp@marek.priv.at> wrote:
I can shed a bit of light on that: the bots were allegedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting.
Then the shell script restarted them...
After more effort than I would have liked, minion is again connecting to libera.chat, and providing registration URLs on #common-lisp.net, which is the priority Dave Cooper asked for. I “fixed” the instance on the current host “common-lisp.net” in lieu of setting up on the new host (“future.common-lisp.net”), as I think we need to firm up that hosts’ operational status (backup, etc.) before committing operational services to it. Some notes on getting the bots working with libera.chat: 1. A few source level [changes][]. There were uncommitted changes [uncategorized][] on the common-lisp.net filesystem that I saved, but mostly discarded except for the changes to the irc-bot “small definition” database. [changes]: <https://github.com/stassats/lisp-bots/pull/18> [uncategorized]: <https://github.com/easye/lisp-bots/commit/b2364eb2ef2c1a201ba13fe4f8e55477013f56b3> 2. Use a current sbcl. The sbcl is installed local to the ‘lisppaste’ user, so shouldn’t be affected by os package changes. 3. Use current Quicklisp dependencies. 4. Link override of necessary ASDF definitions into SBCL via use of <file://common-lisp.net/srv/lisppaste/common-lisp/>. 5. Use a local git clone of IOLIB source, linked to <file://common-lisp.net/src/lisppaste/lfp/lib/*/> via setting the LD_LIBRARY_PATH variable. Not currently sure why this local source clone works, whereas the Quicklisp dependency doesn’t even though they seem to be the same source. The IOLIB build/connection to libfixposix was the single most difficulty due to the “age” of the Debian installation on common-lisp.net. Note that the fixed bot is not currently lashed to work on startup, as I am still working on stabilizing things a bit more, but I at least wanted to report progress here. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
On Nov 16, 2023, at 08:25, Philipp Marek <philipp@marek.priv.at> wrote:
I can shed a bit of light on that: the bots were allegedly continuously reconnecting to the IRC network.
Yes, because they crashed every time - ISTR some IRC library wasn't compatible with an updated SBCL and we ran into some ERROR condition _after_ connecting.
Then the shell script restarted them...
After more effort than I would have liked, minion is again connecting to libera.chat, and providing registration URLs on #common-lisp.net, which is the priority Dave Cooper asked for. I “fixed” the instance on the current host “common-lisp.net” in lieu of setting up on the new host (“future.common-lisp.net”), as I think we need to firm up that hosts’ operational status (backup, etc.) before committing operational services to it. Some notes on getting the bots working with libera.chat: 1. A few source level [changes][]. There were uncommitted changes [uncategorized][] on the common-lisp.net filesystem that I saved, but mostly discarded except for the changes to the irc-bot “small definition” database. [changes]: <https://github.com/stassats/lisp-bots/pull/18> [uncategorized]: <https://github.com/easye/lisp-bots/commit/b2364eb2ef2c1a201ba13fe4f8e55477013f56b3> 2. Use a current sbcl. The sbcl is installed local to the ‘lisppaste’ user, so shouldn’t be affected by os package changes. 3. Use current Quicklisp dependencies. 4. Link override of necessary ASDF definitions into SBCL via use of <file://common-lisp.net/srv/lisppaste/common-lisp/>. 5. Use a local git clone of IOLIB source, linked to <file://common-lisp.net/src/lisppaste/lfp/lib/*/> via setting the LD_LIBRARY_PATH variable. Not currently sure why this local source clone works, whereas the Quicklisp dependency doesn’t even though they seem to be the same source. The IOLIB build/connection to libfixposix was the single most difficulty due to the “age” of the Debian installation on common-lisp.net. Note that the fixed bot is not currently lashed to work on startup, as I am still working on stabilizing things a bit more, but I at least wanted to report progress here. -- "A screaming comes across the sky. It has happened before but there is nothing to compare to it now."
participants (6)
-
David Cooper -
Erik Huelsmann -
evenson -
Mark Evenson
-
Michał "phoe" Herda -
Philipp Marek