Hi,
It's nearly time for Debian Stretch to be released. When that happens (and GitLab releases packages for it), we want to upgrade the system quickly afterwards.
In preparation, I'm working on trial runs for the base OS upgrade. As a consequence, performance may be degraded. To facilitate the upgrade (make it more quick), I reviewed all packages installed on the system with the purpose of removing unused ones.
I've removed quite a few packages indeed (from the current production system, that is): all packages related to the X11 (server) have been removed. So have most LaTeX and LuaTeX packages. The idea behind this step is that the access provided by Common-Lisp.net is merely provided to support uploading and maintaining the static html pages -- my assumption is that that has little relationship to being able to start a desktop environment or graphics environment.
If you miss packages that you depend on, don't hesitate to speak up. Please explain what you need them for and I'll make them available again.
While going over the upgrade process, I'm doing the so-many-th round of configuration cleanup: due to the fact that the host has been managed by multiple people at various times in the past, each with their own approaches to managing the configuration, the configuration can be much more standardized. On each round of maintenance, that's exactly what I've been doing. Configuration is more-and-more getting in line with "how Debian intends it". Let me use this e-mail as an opportunity to ask current and future (co)maintainers to stick to it; it's often a learning curve, but saves a lot of effort on upgrades later on.
There will be a pre-notification and some (planned) downtime for the system when the final cut-over arrives.
For those interested in how the migration strategy and dress rehearsals work, I'm going over these steps (wash, rinse, repeat -- in other words: over and over until it works):
1. Create LVM writeable snapshots of the VM's volumes 2. Create a new VM on top of these snapshots 3. Start up the new VM with different IP and hostname settings (patched Apache config too) 4. Change the (git-versioned) /etc directory to the new target distribution version branch 5. Run dist-upgrade (and a number of additional steps) 6. Fix the configuration issues found in the resulting installation and commit (to another server) 7. Ditch the copy-VM and go back to (1)
This procedure, although it's taking quite a bit of time to run the dist-upgrade, is turning out pretty effective in producing a reproducible upgrade procedure which can be repeated consistently once the production server needs to be upgraded.
Sounds good, thank you for all this effort!
On Sun, Jun 18, 2017 at 9:38 AM, Erik Huelsmann ehuels@gmail.com wrote:
Hi,
It's nearly time for Debian Stretch to be released. When that happens (and GitLab releases packages for it), we want to upgrade the system quickly afterwards.
In preparation, I'm working on trial runs for the base OS upgrade. As a consequence, performance may be degraded. To facilitate the upgrade (make it more quick), I reviewed all packages installed on the system with the purpose of removing unused ones.
I've removed quite a few packages indeed (from the current production system, that is): all packages related to the X11 (server) have been removed. So have most LaTeX and LuaTeX packages. The idea behind this step is that the access provided by Common-Lisp.net is merely provided to support uploading and maintaining the static html pages -- my assumption is that that has little relationship to being able to start a desktop environment or graphics environment.
If you miss packages that you depend on, don't hesitate to speak up. Please explain what you need them for and I'll make them available again.
While going over the upgrade process, I'm doing the so-many-th round of configuration cleanup: due to the fact that the host has been managed by multiple people at various times in the past, each with their own approaches to managing the configuration, the configuration can be much more standardized. On each round of maintenance, that's exactly what I've been doing. Configuration is more-and-more getting in line with "how Debian intends it". Let me use this e-mail as an opportunity to ask current and future (co)maintainers to stick to it; it's often a learning curve, but saves a lot of effort on upgrades later on.
There will be a pre-notification and some (planned) downtime for the system when the final cut-over arrives.
For those interested in how the migration strategy and dress rehearsals work, I'm going over these steps (wash, rinse, repeat -- in other words: over and over until it works):
- Create LVM writeable snapshots of the VM's volumes
- Create a new VM on top of these snapshots
- Start up the new VM with different IP and hostname settings (patched
Apache config too) 4. Change the (git-versioned) /etc directory to the new target distribution version branch 5. Run dist-upgrade (and a number of additional steps) 6. Fix the configuration issues found in the resulting installation and commit (to another server) 7. Ditch the copy-VM and go back to (1)
This procedure, although it's taking quite a bit of time to run the dist-upgrade, is turning out pretty effective in producing a reproducible upgrade procedure which can be repeated consistently once the production server needs to be upgraded.
-- Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.
On 6/18/17 15:38, Erik Huelsmann wrote: […]
I've removed quite a few packages indeed (from the current production system, that is): all packages related to the X11 (server) have been removed. So have most LaTeX and LuaTeX packages. The idea behind this step is that the access provided by Common-Lisp.net is merely provided to support uploading and maintaining the static html pages -- my assumption is that that has little relationship to being able to start a desktop environment or graphics environment.
If you miss packages that you depend on, don't hesitate to speak up. Please explain what you need them for and I'll make them available again.
[…]
I appreciate the effort to get to Debian Stretch, but common-lisp.net also functions as a shell host and therefore needs more than uploading and maintaining the static html pages. To that end, I have reinstalled system-wide screen and gcc to continue with my use of the host.
In the future, we should probably have a explicit whitelist of who is using what package. Is there somewhere that I can record the list of Debian packages on common-lisp.net that I am "actively" depending upon to prevent their removal in the future?
A question: do you intend to retain 'file://common-lisp.net/srv/' hierarchy, or will that be absorbed into the redoing of the 'https://gitlab.common-lisp.net/clo/server-config/' hierarchy.
It would be nice to update documentation on what systems we are using, and how. We have started to document things in the Trac wiki https://trac.common-lisp.net/clo/wiki/TitleIndex. Are we going to maintain this or migrate it to something else?
Hi Mark,
On Tue, Jun 20, 2017 at 9:11 AM, Mark Evenson evenson@panix.com wrote:
On 6/18/17 15:38, Erik Huelsmann wrote: […]
I've removed quite a few packages indeed (from the current production system, that is): all packages related to the X11 (server) have been removed. So have most LaTeX and LuaTeX packages. The idea behind this
step
is that the access provided by Common-Lisp.net is merely provided to support uploading and maintaining the static html pages -- my assumption
is
that that has little relationship to being able to start a desktop environment or graphics environment.
If you miss packages that you depend on, don't hesitate to speak up.
Please
explain what you need them for and I'll make them available again.
[…]
I appreciate the effort to get to Debian Stretch, but common-lisp.net also functions as a shell host
Indeed I have noticed some very-long running sessions on your account in the past. Looking at the page which lists services to our projects ( https://common-lisp.net/project-intro/), I think your use of the host is unsupported (as in: I can't find it as a listed service). We currently use shell sessions to run lisppaste and cliki, but I think that with the introduction of `systemd`, we can probably move those to be run as "user services" instead. Based on that, I was expecting the system not to have long-running shell sessions anymore.
and therefore needs more than uploading and maintaining the static html pages. To that end, I have reinstalled system-wide screen and gcc to continue with my use of the host.
Can I ask in what way the shell sessions you use need to be a generic service to support the common lisp community (and what role does GCC play in it)?
As for the 'screen' program specifically: it's my plan to replace it with tmux (which *is* installed).
In the future, we should probably have a explicit whitelist of who is
using what package. Is there somewhere that I can record the list of Debian packages on common-lisp.net that I am "actively" depending upon to prevent their removal in the future?
A question: do you intend to retain 'file://common-lisp.net/srv/' hierarchy, or will that be absorbed into the redoing of the 'https://gitlab.common-lisp.net/clo/server-config/' hierarchy.
The /srv hierarchy should remain as it is: it stores the data for the services that the host provides. The sever config you're pointing to (note: you've just published a URL for a project that's marked *private* for obvious reasons...) is only versioning the config of /etc. Some (most) config stored there is actually mostly there because it has been edited from the standard Debian config. In a number of cases, the Wheezy branch contains the edited version, but the Jessie and Stretch branches contain the Debian standard version, adding new configuration files ("the Debian way") to extend/enable/disable the standard Debian config.
It would be nice to update documentation on what systems we are using, and how. We have started to document things in the Trac wiki https://trac.common-lisp.net/clo/wiki/TitleIndex. Are we going to maintain this or migrate it to something else?
I wasn't aware of those pages (anymore?). However, given that there's one based on Debian 3.1(?!?), I think the pages need a serious overhaul ( https://trac.common-lisp.net/clo/wiki/InstalledPackages) Also, I think that specific page doesn't add much over the current listing of installed packages queried through Debian. If we want a page with installed packages, we probably want to map them to the services the host is supporting (if there's an explicit relationship). So that when upgrading, we can verify that those services are correctly installed again.
I don't feel an immediate need to move those pages elsewhere, although Trac's idea of generating a pre-seeded wiki with a lot of pages about its syntax isn't really resulting in a pleasantly structured wiki IMO. If we throw out the "standard" pages, I think we could extend it there. The clo/server-config project is private, so we'd be documenting for the current maintainers only. On the Trac wiki, it's all public (thus potentially documenting for our users as well), which I have no problem with, as long as the actual configuration isn't published.
On 6/20/17 11:54, Erik Huelsmann wrote:
Hi Mark,
On Tue, Jun 20, 2017 at 9:11 AM, Mark Evenson evenson@panix.com wrote:
On 6/18/17 15:38, Erik Huelsmann wrote: […]
I've removed quite a few packages indeed (from the current production system, that is): all packages related to the X11 (server) have been removed. So have most LaTeX and LuaTeX packages. The idea behind this
step
is that the access provided by Common-Lisp.net is merely provided to support uploading and maintaining the static html pages -- my assumption
is
that that has little relationship to being able to start a desktop environment or graphics environment.
If you miss packages that you depend on, don't hesitate to speak up.
Please
explain what you need them for and I'll make them available again.
[…]
I appreciate the effort to get to Debian Stretch, but common-lisp.net also functions as a shell host
Indeed I have noticed some very-long running sessions on your account in the past. Looking at the page which lists services to our projects ( https://common-lisp.net/project-intro/), I think your use of the host is unsupported (as in: I can't find it as a listed service). We currently use shell sessions to run lisppaste and cliki, but I think that with the introduction of `systemd`, we can probably move those to be run as "user services" instead. Based on that, I was expecting the system not to have long-running shell sessions anymore.
Well, I need a shell host to perform tasks for common-lisp.net maintainence. As such, I install tools I can't get from Debian in file://common-lisp.net/usr/home/mevenson/local/. screen I know, so please retain.
[…]
Can I ask in what way the shell sessions you use need to be a generic service to support the common lisp community (and what role does GCC play in it)?
Maintaining support via IRC channels, and being able to answer the occacsional support request in the same Emacs.
[…]
-
Dave Cooper -
Erik Huelsmann -
Mark Evenson