Hi all,
We have a big opportunity for a few volunteers to help move clnet services to a new host. You will become part of our Website Working Committee.
Our old host has proven resistant to doing an in-place OS upgrade, so we have determined the way forward is to provision a fresh host (funding for that is another open matter, for a Fundraising committee – volunteers welcome for that as well).
This (standing up the new host) should be a fun project with a lot of learning and networking opportunities. Please write to me directly if interested.
Dave Cooper
Our old host has proven resistant to doing an in-place OS upgrade, so we have determined the way forward is to provision a fresh host (funding for that is another open matter, for a Fundraising committee – volunteers welcome for that as well).
I kind of hoped that the CL foundation has a bit of money left, enough to get another (virtual) machine (eg. Hetzner) to run in parallel for 1-2 months at most until the old one can be scratched?
This (standing up the new host) should be a fun project with a lot of learning and networking opportunities. Please write to me directly if interested.
Yeah, I'm still willing to help, even if I'm a bit more time-starved than a year ago.
In the meantime I've come to believe that it won't be easy to get a full weekend from all volunteers at the same time -- and even if we did, we'd just stomp on each others toes.
Can we define some common documentation area (perhaps using cryptpad.fr or a similar independent service) where we can create a list of tasks, edit/complete/document/check there in parallel, and start the migration of services to the new VM in the background?
I'm well aware that we need a bit of coordination for the final sync run[1], but quite a lot of configuration can be done independently.
[1] I hope that we can define stuff like /home/ as read-only on the old machine after a first rsync run -- but for gitlab we'll need some coordinated down-time, I guess.
Dave Cooper
---- On Thu, 05 Oct 2023 01:59:07 -0400 Philipp Marek philipp@marek.priv.at wrote ---
Our old host has proven resistant to doing an in-place OS upgrade, so
we have determined the way forward is to provision a fresh host
(funding for that is another open matter, for a Fundraising committee –
volunteers welcome for that as well).
I kind of hoped that the CL foundation has a bit of money left,
enough to get another (virtual) machine (eg. Hetzner) to run in parallel
for 1-2 months at most until the old one can be scratched?
Yes we do have enough in our coffers to bootstrap at least one box for up to at least one year, for example I see this at Hetzner:
AMD Ryzen™ 5 3600 CPU6 cores / 12 threads @ 3.6 GHz Generation: Matisse (Zen 2) RAM 64 GB DDR4 RAM Drives. 2 x 512 GB NVMe SSD
Would that be sufficient for our gitlab at least? Or for running a hypervisor with one VM for gitlab and maybe others for other services?
This (standing up the new host) should be a fun project with a lot of
learning and networking opportunities. Please write to me directly if
interested.
Yeah, I'm still willing to help, even if I'm a bit more time-starved
than a year ago.
In the meantime I've come to believe that it won't be easy to get a
full weekend from all volunteers at the same time -- and even if we did,
we'd just stomp on each others toes.
Can we define some common documentation area (perhaps using cryptpad.fr
or a similar independent service) where we can create a list of tasks,
edit/complete/document/check there in parallel, and start the migration
of services to the new VM in the background
That sounds like a good idea. I suppose our Dropbox would not be sufficient, as it doesn't gracefully handle simultaneous fine-grained updates to a document... ?
I'm well aware that we need a bit of coordination for the final sync
run[1],
but quite a lot of configuration can be done independently.
[1] I hope that we can define stuff like /home/ as read-only
on the old machine after a first rsync run -- but for gitlab we'll need
some coordinated down-time, I guess.
Dave
for example I see this at Hetzner:
AMD Ryzen™ 5 3600 CPU6 cores / 12 threads @ 3.6 GHz Generation: Matisse (Zen 2) RAM 64 GB DDR4 RAM Drives. 2 x 512 GB NVMe SSD
Would that be sufficient for our gitlab at least?
It should be enough CPU and RAM; if we want to run local RAID1 (which I'd like to have!), storage could become a bit low.
root@common-lisp:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 2G 0 disk └─sda1 8:1 0 243M 0 part /boot sdb 8:16 0 200G 0 disk /srv /home /mnt/btrfs /usr / sdc 8:32 0 250G 0 disk /var vda 254:0 0 32G 0 disk [SWAP]
Yeah, currently we have ~450GB in filesystems, _used_ is only 111GB + 149GB = 260GB, so there's a bit of reserve left.
Or for running a hypervisor with one VM for gitlab and maybe others for other services?
I'd suggest to use Docker instead - according to our measurements, a VM layer costs 15-25% of performance, and having everything available in one filesystem makes backups much easier.
We might need VMs for cross-builds anyway, though, if we want to support that at some time. (Or at least qemu-static-<arch> - might be easier to handle)
Can we define some common documentation area (perhaps using cryptpad.fr or a similar independent service) where we can create a list of tasks, edit/complete/document/check there in parallel, and start the migration of services to the new VM in the background
That sounds like a good idea. I suppose our Dropbox would not be sufficient, as it doesn't gracefully handle simultaneous fine-grained updates to a document...?
I'm imagining that the people might meet every now and then (perhaps half an hour 2x to 7x a week?), and having a document that can *simultaneusly* be modified by several people with changes being visible at once is a big benefit.
Another open point for me is general architecture.
I guess we want to go with a single server (no HA pair), which should be fine for our use case.
- But what about backups? Do we keep the old place, or do we restart them too?
- Do we want/need external monitoring? If we do, run it on the backup machine or keep that one as just a storage box?
Hi Philip,
Thank you for your analysis and proposals so far!
see my (probably dumb) questions below..
---- On Fri, 06 Oct 2023 02:28:19 -0400 Philipp Marek mailto:philipp@marek.priv.at wrote ---
for example I see this at Hetzner:
AMD Ryzen™ 5 3600
CPU6 cores / 12 threads @ 3.6 GHz
Generation: Matisse (Zen 2)
RAM 64 GB DDR4 RAM
Drives. 2 x 512 GB NVMe SSD
Would that be sufficient for our gitlab at least?
It should be enough CPU and RAM;
if we want to run local RAID1 (which I'd like to have!),
storage could become a bit low.
root@common-lisp:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 2G 0 disk
└─sda1 8:1 0 243M 0 part /boot
sdb 8:16 0 200G 0 disk /srv
/home
/mnt/btrfs
/usr
/
sdc 8:32 0 250G 0 disk /var
vda 254:0 0 32G 0 disk [SWAP]
Yeah, currently we have ~450GB in filesystems,
_used_ is only 111GB + 149GB = 260GB,
so there's a bit of reserve left.
Given the above, how much disk do you reckon we'd need to run local RAID1? And I assume it has to be two separate drives?
By the way, the above Hetzner rig is currently offered for EUR~45 per month.
Or for running a hypervisor with one VM for gitlab and maybe others
for other services?
I'd suggest to use Docker instead - according to our measurements,
a VM layer costs 15-25% of performance,
and having everything available in one filesystem makes backups much
easier.
I can imagine if we have all the services Dockerized then in principle, among other benefits, it should be easier to migrate to different physical hosts in the future -- just push and pull a Docker image. Well, not really -- there will be loads of data to go along with each Docker image and service, plus orchestration to be replicated on a new phycal host. But I think those are solved problems and overall made easier by containerizing stuff -- at least the local environment for each service isn't suddenly changing out from under it by moving to a new physical host with a newer or different Linux flavor.
How about for Erik H's stated goal of being able to partition and delegate maintenance of different services to different volunteers? In the case of VMs then each VM would literally have one volunteer in charge of it. Could we do similar with Docker deployment? In that case, I'd expect the volunteer in charge of each Container to become familiar (and practice doing) backup, migration, restore of their particular container. And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
And given the services we are aiming to offer, do you think we are talking about standard off-the-shelf Docker images (e.g. for gitlab, nginx, etc) or are we looking at authoring and maintaining our own custom images (i.e. our own Dockerfiles).
We might need VMs for cross-builds anyway, though,
if we want to support that at some time.
(Or at least qemu-static-<arch> - might be easier to handle)
I'm not sure what qemu-static-<arch> means.
What platforms would you have in mind to support for cross-builds?
I assume each platform would add to CPU/RAM/disk requirements. Query whether to size for that eventuality now or later.
Can we define some common documentation area (perhaps using
cryptpad.fr
or a similar independent service) where we can create a list of tasks,
edit/complete/document/check there in parallel, and start the
migration
of services to the new VM in the background
That sounds like a good idea.
I suppose our Dropbox would not be sufficient, as it doesn't
gracefully handle simultaneous fine-grained updates to a document...?
I'm imagining that the people might meet every now and then (perhaps
half an hour 2x to 7x a week?), and having a document that can
*simultaneusly* be modified by several people with changes being
visible at once is a big benefit.
Philip could you go ahead and set up one document for this migration effort, in a place you find handy? If cryptpad will do what we need for now then let it be cryptpad. The cryptpad would be not such much for discussion (this mailing list can play that role for now) but more as a live to-do list with up to date status (I don't know if that's formatted like an org-mode list or what).
Another open point for me is general architecture.
I guess we want to go with a single server (no HA pair),
which should be fine for our use case.
- But what about backups?
Do we keep the old place, or do we restart them too?
- Do we want/need external monitoring?
If we do, run it on the backup machine or keep that one as just
a storage box?
In addition to HA (High Availability) and external monitoring, I can imagine just a couple reasons we might want to maintain separate hosts:
1. To have a dedicated build host just for running gitlab-runner and executors for it -- so that heavy pipeline jobs wouldn't bog down our whole server. That brings up the question of do we still support shell logins into the base OS (as we do now) for users as well as administrators? And do we still enable the "shell" executor for gitlab-runner, if everything is supposed to be dockerized and we're trying to avoid running things on the bare base OS?
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
3. And we might want hosts of different architectures, even Mac & Windows.
On that topic, we've had an offer of a donation of an Intel nuc to use as a permanent host (specs unknown). I also know of at least one Mac Mini M1 which could be donated as a build host. The idea of collecting hardware sounds nice - but is there a viable way to co-locate, provision network resources, and physically care for such hardware which may come into the Foundation's possession?
Thanks again,
Dave
Yeah, currently we have ~450GB in filesystems, _used_ is only 111GB + 149GB = 260GB, so there's a bit of reserve left.
Given the above, how much disk do you reckon we'd need to run local RAID1? And I assume it has to be two separate drives?
Yes -- your quoted Hetzner machine has 2x512GB, so it's good enough for a RAID1 (mirroring) and our current needs.
I'd suggest to use Docker instead - according to our measurements, a VM layer costs 15-25% of performance, and having everything available in one filesystem makes backups much easier.
I can imagine if we have all the services Dockerized
Hmm, I personally wouldn't do that. All the basic stuff (GitLab, MailMan, postfix [instead of exim], git, sshd access, ...) is IMO easier to administrate in one OS -- even more so if it's something well-known like Debian stable.
With all services docker you'll have to individually check them for security updates and so on.
then in principle, among other benefits, it should be easier to migrate to different physical hosts in the future -- just push and pull a Docker image. Well, not really -- there will be loads of data to go along with each Docker image and service, plus orchestration to be replicated on a new phycal host. But I think those are solved problems and overall made easier by containerizing stuff -- at least the local environment for each service isn't suddenly changing out from under it by moving to a new physical host with a newer or different Linux flavor.
Yeah, well, in the previous job we switched OS from Ubuntu to Redhat with barely an interruption for the service-VMs:
http://web.archive.org/web/20160510131958/http://blogs.linbit.com/p/458/dist...
I'm not sure whether we want to have the complexity of an HA stack, though.
How about for Erik H's stated goal of being able to partition and delegate maintenance of different services to different volunteers? In the case of VMs then each VM would literally have one volunteer in charge of it. Could we do similar with Docker deployment?
I guess so.
In that case, I'd expect the volunteer in charge of each Container to become familiar (and practice doing) backup, migration, restore of their particular container.
More or less guaranteeing that at least 1 out of 10 has some kind of bug in the backup scripts, so we might lose data in a catastrophic event.
I'd prefer having _one_ backup mechanism for everything -- I quite like doing simple rsyncs to another host and doing btrfs snapshots on both sides.
[[ The current btrfs snapshot sending is bandwidth-optimal -- my fear is that when badly broken data gets sent this way that _both_ side's filesystems become unavailable. With rsync there's less coupling -- just the normal POSIX semantics. ]]
And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
Ouch. No, please let's avoid the complexity.
And given the services we are aiming to offer, do you think we are talking about standard off-the-shelf Docker images (e.g. for gitlab, nginx, etc) or are we looking at authoring and maintaining our own custom images (i.e. our own Dockerfiles).
Yeah, exactly - the complexity starts creeping in.
I guess I wasn't clear enough in my previous mail.
_My_ idea (wish, recommendation, call it what you like) is a normal UNIX server with some system services[1] -- only in a few places I'd vote for Docker, like Mailman 2 (which isn't needed anymore, Erik already converted all the important mailing lists, right?!!), or when doing builds that require different userspace (like compiling SBCL for FreeBSD, or Windows, or Ubuntu, or RedHat.)
Ad 1: When using systemd, we can use cgroups to isolate and limit their resource usage.
We might need VMs for cross-builds anyway, though, if we want to support that at some time. (Or at least qemu-static-<arch> - might be easier to handle)
I'm not sure what qemu-static-<arch> means.
https://github.com/multiarch/qemu-user-static qemu-user-static is to enable an execution of different multi-architecture containers by QEMU 1 and binfmt_misc
My idea was that we might want to support building (and running tests for) SBCL for other architectures like eg. ARM64 on the machine as well (in the GitLab CI/CD pipelines).
In the meantime it occurred to me that it might be simpler (though having an external dependency!) to ask for access to the gcc compile farm for that stuff.
What platforms would you have in mind to support for cross-builds?
Well, http://sbcl.org/platform-table.html lists
X86, AMD64, PPC, PPC64, PPC64le, SPARC, MIPSbe, MIPSle, ARMel, ARMhf, ARM64, RISC-V32, RISC-V64
but that might be a bit much ;)
I assume each platform would add to CPU/RAM/disk requirements. Query whether to size for that eventuality now or later.
Well, with qemu-user-static this would run "just" as a normal process (tree), like any other CI/CD job and in fact mostly started by GitLab pipelines -- and like anything in the gitlab pipeline it would/should have RAM/CPU limits configured.
Philip could you go ahead and set up one document for this migration effort, in a place you find handy? If cryptpad will do what we need for now then let it be cryptpad. The cryptpad would be not such much for discussion (this mailing list can play that role for now) but more as a live to-do list with up to date status (I don't know if that's formatted like an org-mode list or what).
Ack. I'll get back to this thread with a link when I have one.
In addition to HA (High Availability)
HA adds complexity - and so adds to support load. I'm not sure we need it.
Perhaps the CL foundation wants to vote on it, though.
and external monitoring, I can imagine just a couple reasons we might want to maintain separate hosts:
- To have a dedicated build host just for running gitlab-runner and executors for it -- so that heavy pipeline jobs wouldn't bog down our whole server.
An external host means additional dependencies - plus costs.
I'd rather have gitlab jobs run in a cgroup hierarchy that is restricted so that it _can't_ bog down the main services.
But a backup machine (which might also run monitoring of the main host) would be a requirement anyway.
That brings up the question of do we still support shell logins into the base OS (as we do now) for users as well as administrators?
I believe that is a nice feature - even more so if we provide a few foreign architecture filesystems that people can use for qemu-static emulation.
And do we still enable the "shell" executor for gitlab-runner, if everything is supposed to be dockerized and we're trying to avoid running things on the bare base OS?
Hmmm, that's a good question. Having _every_ gitlab job in a docker means that these become independent from our host OS -- so we can upgrade packages as needed (security fixes, feature updates!) without impacting any CI/CD pipeline.
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
Cost?
3. And we might want hosts of different architectures, even Mac & Windows.
How about the GCC build farm?
My idea for building Windows binaries for SBCL would be via using WINE -- that way this job is just another linux process hierarchy...
On that topic, we've had an offer of a donation of an Intel nuc to use as a permanent host (specs unknown). I also know of at least one Mac Mini M1 which could be donated as a build host. The idea of collecting hardware sounds nice - but is there a viable way to co-locate, provision network resources, and physically care for such hardware which may come into the Foundation's possession?
Yeah, that's a good question. Easier said than done!
If these machines get used to build public binaries we would have to provide physical security for them as well, which is non-trivial!
Ack. I'll get back to this thread with a link when I have one.
VERY ROUGH DRAFT please add any details you know!!!
https://cryptpad.fr/pad/#/2/pad/edit/IYetddW5zzchwJOR-ivI8e08/
Thanks Philipp,
---- On Sat, 07 Oct 2023 15:39:54 -0400 Philipp Marek mailto:philipp@marek.priv.at wrote ---
Ack.
I'll get back to this thread with a link when I have one.
VERY ROUGH DRAFT
please add any details you know!!!
https://cryptpad.fr/pad/#/2/pad/edit/IYetddW5zzchwJOR-ivI8e08/
See my entries in items #1 and #2 of the Switchover section. I went ahead and provisioned a machine at Hetzner, and made a login for Philip. Philip you should have the credentials needed to log in and put an ssh public key. Could you try logging in (again, ssh is on port 4022). At some point I'd like to disable password login. See if you think this host will suit our needs for the time being.
Let's try to make the setup process as repeatable as possible - in case this doesn't end up being the be-all, end-all machine.
Is there a way to get notified of updates to the cryptpad? And to see the edits over time. Should we each make a login account over there?
Thanks again,
Dave
See my entries in items #1 and #2 of the Switchover section. I went ahead and provisioned a machine at Hetzner, and made a login for Philip. Philip you should have the credentials needed to log in and put an ssh public key. Could you try logging in (again, ssh is on port 4022). At some point I'd like to disable password login.
I installed my pubkeys and changed the password.
See if you think this host will suit our needs for the time being.
From a very quick glance it looks good - though I'd like to change the partitioning and switch to btrfs.
Let's try to make the setup process as repeatable as possible - in case this doesn't end up being the be-all, end-all machine.
Understood. I'll write down what I do resp. use VCS for the configuration.
My next block with more than 5mins time will be Thursday evening/night, let's see how far I get then.
Is there a way to get notified of updates to the cryptpad?
No, I don't think so.
And to see the edits over time.
It has chat on the right, a content overview to the left, ... ah right, "File" → "History".
Should we each make a login account over there?
That's not required.
Would it be better/more comfortable to switch to a Google document?
Yeah, currently we have ~450GB in filesystems, _used_ is only 111GB + 149GB = 260GB, so there's a bit of reserve left.
Given the above, how much disk do you reckon we'd need to run local RAID1? And I assume it has to be two separate drives?
Yes -- your quoted Hetzner machine has 2x512GB, so it's good enough for a RAID1 (mirroring) and our current needs.
I'd suggest to use Docker instead - according to our measurements, a VM layer costs 15-25% of performance, and having everything available in one filesystem makes backups much easier.
I can imagine if we have all the services Dockerized
Hmm, I personally wouldn't do that. All the basic stuff (GitLab, MailMan, postfix [instead of exim], git, sshd access, ...) is IMO easier to administrate in one OS -- even more so if it's something well-known like Debian stable.
With all services docker you'll have to individually check them for security updates and so on.
Most likely not, because you'll only be using popular software that have well-maintained official images.
then in principle, among other benefits, it should be easier to migrate to different physical hosts in the future -- just push and pull a Docker image. Well, not really -- there will be loads of data to go along with each Docker image and service, plus orchestration to be replicated on a new phycal host. But I think those are solved problems and overall made easier by containerizing stuff -- at least the local environment for each service isn't suddenly changing out from under it by moving to a new physical host with a newer or different Linux flavor.
Yeah, well, in the previous job we switched OS from Ubuntu to Redhat with barely an interruption for the service-VMs:
http://web.archive.org/web/20160510131958/http://blogs.linbit.com/p/458/dist...
I'm not sure whether we want to have the complexity of an HA stack, though.
How about for Erik H's stated goal of being able to partition and delegate maintenance of different services to different volunteers? In the case of VMs then each VM would literally have one volunteer in charge of it. Could we do similar with Docker deployment?
I guess so.
In that case, I'd expect the volunteer in charge of each Container to become familiar (and practice doing) backup, migration, restore of their particular container.
More or less guaranteeing that at least 1 out of 10 has some kind of bug in the backup scripts, so we might lose data in a catastrophic event.
I'd prefer having _one_ backup mechanism for everything -- I quite like doing simple rsyncs to another host and doing btrfs snapshots on both sides.
[[ The current btrfs snapshot sending is bandwidth-optimal -- my fear is that when badly broken data gets sent this way that _both_ side's filesystems become unavailable. With rsync there's less coupling -- just the normal POSIX semantics. ]]
And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
Ouch. No, please let's avoid the complexity.
For your case, K8s is a reasonably easy way to have HA without many issues. You can go very cheap with 3 Hetzner hosts x 50 EUR per month, and use Ubuntu with MicroK8s on the hosts.
And given the services we are aiming to offer, do you think we are talking about standard off-the-shelf Docker images (e.g. for gitlab, nginx, etc) or are we looking at authoring and maintaining our own custom images (i.e. our own Dockerfiles).
Yeah, exactly - the complexity starts creeping in.
I guess I wasn't clear enough in my previous mail.
_My_ idea (wish, recommendation, call it what you like) is a normal UNIX server with some system services[1] -- only in a few places I'd vote for Docker, like Mailman 2 (which isn't needed anymore, Erik already converted all the important mailing lists, right?!!), or when doing builds that require different userspace (like compiling SBCL for FreeBSD, or Windows, or Ubuntu, or RedHat.)
Ad 1: When using systemd, we can use cgroups to isolate and limit their resource usage.
We might need VMs for cross-builds anyway, though, if we want to support that at some time. (Or at least qemu-static-<arch> - might be easier to handle)
I'm not sure what qemu-static-<arch> means.
https://github.com/multiarch/qemu-user-static qemu-user-static is to enable an execution of different multi-architecture containers by QEMU 1 and binfmt_misc
My idea was that we might want to support building (and running tests for) SBCL for other architectures like eg. ARM64 on the machine as well (in the GitLab CI/CD pipelines).
In the meantime it occurred to me that it might be simpler (though having an external dependency!) to ask for access to the gcc compile farm for that stuff.
What platforms would you have in mind to support for cross-builds?
Well, http://sbcl.org/platform-table.html lists
X86, AMD64, PPC, PPC64, PPC64le, SPARC, MIPSbe, MIPSle, ARMel, ARMhf, ARM64, RISC-V32, RISC-V64but that might be a bit much ;)
I assume each platform would add to CPU/RAM/disk requirements. Query whether to size for that eventuality now or later.
Well, with qemu-user-static this would run "just" as a normal process (tree), like any other CI/CD job and in fact mostly started by GitLab pipelines -- and like anything in the gitlab pipeline it would/should have RAM/CPU limits configured.
Philip could you go ahead and set up one document for this migration effort, in a place you find handy? If cryptpad will do what we need for now then let it be cryptpad. The cryptpad would be not such much for discussion (this mailing list can play that role for now) but more as a live to-do list with up to date status (I don't know if that's formatted like an org-mode list or what).
Ack. I'll get back to this thread with a link when I have one.
In addition to HA (High Availability)
HA adds complexity - and so adds to support load. I'm not sure we need it.
I'm not sure you can afford not to have som for of HA. common-lisp.net already has a reputation of poor reliability and continuing on the same path doesn't seem a very good idea. All the new lispers, and many of the old ones, have moved to other services (mostly Github) for good reasons.
Perhaps the CL foundation wants to vote on it, though.
and external monitoring, I can imagine just a couple reasons we might want to maintain separate hosts:
- To have a dedicated build host just for running gitlab-runner and executors for it -- so that heavy pipeline jobs wouldn't bog down our whole server.
An external host means additional dependencies - plus costs.
I'd rather have gitlab jobs run in a cgroup hierarchy that is restricted so that it _can't_ bog down the main services.
But a backup machine (which might also run monitoring of the main host) would be a requirement anyway.
That brings up the question of do we still support shell logins into the base OS (as we do now) for users as well as administrators?
I believe that is a nice feature - even more so if we provide a few foreign architecture filesystems that people can use for qemu-static emulation.
And do we still enable the "shell" executor for gitlab-runner, if everything is supposed to be dockerized and we're trying to avoid running things on the bare base OS?
Hmmm, that's a good question. Having _every_ gitlab job in a docker means that these become independent from our host OS -- so we can upgrade packages as needed (security fixes, feature updates!) without impacting any CI/CD pipeline.
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
I suggest switching to a lighter-weight alternative like Gitea or, even better, bailing out of source hosting altogether. It take a lot of work to provide a capable service, which volunteers can hardly provide.
Cost?
3. And we might want hosts of different architectures, even Mac & Windows.
How about the GCC build farm?
My idea for building Windows binaries for SBCL would be via using WINE
that way this job is just another linux process hierarchy...
On that topic, we've had an offer of a donation of an Intel nuc to use as a permanent host (specs unknown). I also know of at least one Mac Mini M1 which could be donated as a build host. The idea of collecting hardware sounds nice - but is there a viable way to co-locate, provision network resources, and physically care for such hardware which may come into the Foundation's possession?
Yeah, that's a good question. Easier said than done!
If these machines get used to build public binaries we would have to provide physical security for them as well, which is non-trivial!
With all services docker you'll have to individually check them for security updates and so on.
Most likely not, because you'll only be using popular software that have well-maintained official images.
Still someone needs to notice and restart the services to get the new images downloaded, right?
And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
Ouch. No, please let's avoid the complexity.
For your case, K8s is a reasonably easy way to have HA without many issues. You can go very cheap with 3 Hetzner hosts x 50 EUR per month, and use Ubuntu with MicroK8s on the hosts.
If Hetzner does all the K8s infrastructure and we pay just for the worker nodes, maybe it'll be easy enough.
Doing Kubernetes hosting is _way_ out of scope for c-l.net, IMO.
In addition to HA (High Availability)
HA adds complexity - and so adds to support load. I'm not sure we need it.
I'm not sure you can afford not to have som for of HA. common-lisp.net already has a reputation of poor reliability and continuing on the same path doesn't seem a very good idea. All the new lispers, and many of the old ones, have moved to other services (mostly Github) for good reasons.
I don't think this is because of reliability of the hosting service - more of convenience resp. familiarity when comparing Gitlab to GitHub.
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
I suggest switching to a lighter-weight alternative like Gitea or, even better, bailing out of source hosting altogether. It take a lot of work to provide a capable service, which volunteers can hardly provide.
If we decide not to host any Git UI (and no ticket tracker, no pipeline, etc.), I'll still vote to have git repositories - as a reference source/backup.
Via https://gitolite.com/gitolite/index.html should be easy enough.
With all services docker you'll have to individually check them for security updates and so on.
Most likely not, because you'll only be using popular software that have well-maintained official images.
Still someone needs to notice and restart the services to get the new images downloaded, right?
Same as with a .deb-installed Gitlab: automatic upgrades are not reliable.
And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
Ouch. No, please let's avoid the complexity.
For your case, K8s is a reasonably easy way to have HA without many issues. You can go very cheap with 3 Hetzner hosts x 50 EUR per month, and use Ubuntu with MicroK8s on the hosts.
If Hetzner does all the K8s infrastructure and we pay just for the worker nodes, maybe it'll be easy enough.
Doing Kubernetes hosting is _way_ out of scope for c-l.net, IMO.
Sure, there are various ways. My point is that HA, done by someone not necessarily the c-l.net staff, is nowadays necessary.
In addition to HA (High Availability)
HA adds complexity - and so adds to support load. I'm not sure we need it.
I'm not sure you can afford not to have som for of HA. common-lisp.net already has a reputation of poor reliability and continuing on the same path doesn't seem a very good idea. All the new lispers, and many of the old ones, have moved to other services (mostly Github) for good reasons.
I don't think this is because of reliability of the hosting service - more of convenience resp. familiarity when comparing Gitlab to GitHub.
I know for a fact (as was told to me personally), that there are people who left because of reliability issues and (initially) lack of features in regards to CI/CD. Consider this: many of those who used to use c-l.net are free software advocates and would prefer to avoid Github, but left anyway.
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
I suggest switching to a lighter-weight alternative like Gitea or, even better, bailing out of source hosting altogether. It take a lot of work to provide a capable service, which volunteers can hardly provide.
If we decide not to host any Git UI (and no ticket tracker, no pipeline, etc.), I'll still vote to have git repositories - as a reference source/backup.
Sure, you can keep a backup using a bare-bones service that doesn't do CI/CD or reviews, but I adivse joining a project like Codeberg for the code development part.
For me, the best way to help the community would be to maintain a Docker image that's kept up-to-date with all the CL implementations, and perhaps host the CI/CD runners usable from other environments (Github, Gitlab.com, Codeberg). Even better if you could convince Franz and LW allow free use of their Enterprise versions for open-source development.
Dear All,
The cl.net infrastructure is a good-faith best effort of volunteers, and as with any such effort, it is understandable that it will never be "good enough" for everyone.
I appreciate a lot of the sentiments in the discussion below, but as a practical matter, and in the interest of "changing one thing at a time," all we can hope to do for starters is to replicate what we have now, but on a clean, up-to-date server. Once we are sitting on top of cleanly installed, up to date infrastructure, we can then discuss making it more robust (with HA etc) and adding or removing particular services. I feel any such discussion before then is premature.
Dave Cooper
P.S. Up do date docker images with CL impls is indeed supposed to be part of our standard CI setup and Eric Timmons has been leading that effort. Anyone working on that part of the infrastructure please check with Tim before possibly taking on any repeat work in this regard.
---- On Tue, 10 Oct 2023 13:11:21 -0400 Stelian Ionescu sionescu@cddr.org wrote ---
With all services docker you'll have to individually check them for security updates and so on.
Most likely not, because you'll only be using popular software that have well-maintained official images.
Still someone needs to notice and restart the services to get the new images downloaded, right?
Same as with a .deb-installed Gitlab: automatic upgrades are not reliable.
And For coordination of several containers running our several services, do you think we could we could use something simple such as docker-compose or would we better resort to a "real" orchestration setup such as kubernetes?
Ouch. No, please let's avoid the complexity.
For your case, K8s is a reasonably easy way to have HA without many issues. You can go very cheap with 3 Hetzner hosts x 50 EUR per month, and use Ubuntu with MicroK8s on the hosts.
If Hetzner does all the K8s infrastructure and we pay just for the worker nodes, maybe it'll be easy enough.
Doing Kubernetes hosting is _way_ out of scope for c-l.net, IMO.
Sure, there are various ways. My point is that HA, done by someone not necessarily the c-l.net staff, is nowadays necessary.
In addition to HA (High Availability)
HA adds complexity - and so adds to support load. I'm not sure we need it.
I'm not sure you can afford not to have som for of HA. common-lisp.net already has a reputation of poor reliability and continuing on the same path doesn't seem a very good idea. All the new lispers, and many of the old ones, have moved to other services (mostly Github) for good reasons.
I don't think this is because of reliability of the hosting service - more of convenience resp. familiarity when comparing Gitlab to GitHub.
I know for a fact (as was told to me personally), that there are people who left because of reliability issues and (initially) lack of features in regards to CI/CD. Consider this: many of those who used to use c-l.net are free software advocates and would prefer to avoid Github, but left anyway.
2. Maybe for a dedicated gitlab host as well, because that program is so freaking heavy.
I suggest switching to a lighter-weight alternative like Gitea or, even better, bailing out of source hosting altogether. It take a lot of work to provide a capable service, which volunteers can hardly provide.
If we decide not to host any Git UI (and no ticket tracker, no pipeline, etc.), I'll still vote to have git repositories - as a reference source/backup.
Sure, you can keep a backup using a bare-bones service that doesn't do CI/CD or reviews, but I adivse joining a project like Codeberg for the code development part.
For me, the best way to help the community would be to maintain a Docker image that's kept up-to-date with all the CL implementations, and perhaps host the CI/CD runners usable from other environments (Github, Gitlab.com, Codeberg). Even better if you could convince Franz and LW allow free use of their Enterprise versions for open-source development.
A completely different question --
has any contacted GitLab (the organization) how much a hosted instance (SaaS) costs?
Ie. we point gitlab.common-lisp.net to them, have a few admin and lots of user logins, and they care about running the box.
That would leave us with a much smaller VM -- mailman, video conferencing, perhaps SSH, hosting of static pages (ie. getting the *.common-lisp.dev project pages synced)...
As all (?) of the gitlab content is public and open-source anyway, there's no real risk to privacy, is there? I think they might host it for free (Open Source, community, etc.), but even if not it might be beneficial if it reduces "our" (actually mostly Dave, nowadays, right?) work load...
Hi Dave,
While I can't commit to availability at this moment, I am very keen to see some improvements made and have some suggestions accordingly. Please let me know the best mechanism to share those.
—jon
On Wed, Oct 4, 2023 at 3:59 PM David Cooper david.cooper@genworks.com wrote:
Hi all,
We have a big opportunity for a few volunteers to help move clnet services to a new host. You will become part of our Website Working Committee.
Our old host has proven resistant to doing an in-place OS upgrade, so we have determined the way forward is to provision a fresh host (funding for that is another open matter, for a Fundraising committee – volunteers welcome for that as well).
This (standing up the new host) should be a fun project with a lot of learning and networking opportunities. Please write to me directly if interested.
Dave Cooper
-
David Cooper -
Jon Boone -
Philipp Marek -
Stelian Ionescu