[clo-devel] 2011-04-15: cl-net migration progress
My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished. At the same time, I'm trying to figure out how to best migrate /etc/password- and /etc/group-. Anybody here done someting alike before? I mean, I can't just copy it over, because the system accounts shouldn't be overwritten. I suppose a 'grep -v' (for inverse matching) should get us quite far filtering out system (uid<500) accounts. Another issue I'm thinking about is that we're going to switch to subdomains for hosting our services. However, the benefits of such a change seem low if we don't switch to a structure where our data is also grouped per service. Currently everything is in /project, which is grouped around projects, requiring all services to run on a single server because they need to access specific directories - one in each project. I'm thinking we should reverse the design: grouping data per service and making /project a set of symbolic links into the service directories. That way, all that's required is that the links point to the right locations in the file system, possibly NFS mount points, if the service were to run on a different server. Again, I'm interested in your comments and ideas. Regards, Erik.
On Fri, 2011-04-15 at 22:21 +0200, Erik Huelsmann wrote:
My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished.
If you make a backup while the server is still accessible you'll lose data modified after the backup but before the restore and switch to the new server. Here's how I'd do it: 1a) Start a new SSH server on old.cl.net(with a different RSA key) on a port other than 22 and move the current keys to new.cl.net 1b) Add an iptables redirect of port 22 to new.cl.net but don't start it on new.cl.net until the migration is done and kick out all people logged on the server 1c) Shutdown all services that might try to access /home and /project (apache, git, cvs, svn, etc...) 2a) Move /home and /project to new.cl.net 2b) Export them via a network FS to old.cl.net 3) Start the services and hope they work If all goes well, all those services should be able to access the projects' data via network and we can start to migrate each service to new.cl.net whenever there's time BTW, do you think we should use something other than NFS, perhaps CIFS ?
At the same time, I'm trying to figure out how to best migrate /etc/password- and /etc/group-. Anybody here done someting alike before? I mean, I can't just copy it over, because the system accounts shouldn't be overwritten. I suppose a 'grep -v' (for inverse matching) should get us quite far filtering out system (uid<500) accounts.
Human users should maintain their passwd records(and implicitly, their uid). A little awk will do the trick
Another issue I'm thinking about is that we're going to switch to subdomains for hosting our services. However, the benefits of such a change seem low if we don't switch to a structure where our data is also grouped per service. Currently everything is in /project, which is grouped around projects, requiring all services to run on a single server because they need to access specific directories - one in each project. I'm thinking we should reverse the design: grouping data per service and making /project a set of symbolic links into the service directories. That way, all that's required is that the links point to the right locations in the file system, possibly NFS mount points, if the service were to run on a different server.
This is a good idea, especially since it requires very little additional work, just a little care not to break users' existing setups -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib
Hi Stelian, Thanks for thinking with me on how to do this! On Fri, Apr 15, 2011 at 11:44 PM, Stelian Ionescu <sionescu@cddr.org> wrote:
On Fri, 2011-04-15 at 22:21 +0200, Erik Huelsmann wrote:
My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished.
If you make a backup while the server is still accessible you'll lose data modified after the backup but before the restore and switch to the new server.
Absolutely agreed. So, copying the data from an lvm snapshot is not the only action. However, in the past, I've been able to reduce downtimes of servers to just minutes by copying the bulk of the data in advance, followed by bringing down the server, doing a quick rsync and bringing up the new server. It's the plan I intend to use this time aswell.
Here's how I'd do it: 1a) Start a new SSH server on old.cl.net(with a different RSA key) on a port other than 22 and move the current keys to new.cl.net 1b) Add an iptables redirect of port 22 to new.cl.net but don't start it on new.cl.net until the migration is done and kick out all people logged on the server 1c) Shutdown all services that might try to access /home and /project (apache, git, cvs, svn, etc...) 2a) Move /home and /project to new.cl.net 2b) Export them via a network FS to old.cl.net 3) Start the services and hope they work
My current status is that I copied 'hot' /project, /home and /custom. I'll be able to switch cl-net off tonight, do the remaining copies and turn old cl-net back on. That way, the data is on new-cl-net, but the user accounts are still on old-cl-net. This would allow us to migrate anonymous services such as Trac, subversion, cvs and repository browsing. Since the IP of the server will change, what do we gain by using the same SSH key? I mean, ssh will tell the user the IP has changed, won't it?
If all goes well, all those services should be able to access the projects' data via network and we can start to migrate each service to new.cl.net whenever there's time
BTW, do you think we should use something other than NFS, perhaps CIFS ?
Well, given the long history of *nix and NFS, I'd say they make a good combination, at least for a small setup as this: we're going to phase out the networked filesystem after the migration again; I'm not anticipating adding further nodes into a grid or something alike.
At the same time, I'm trying to figure out how to best migrate /etc/password- and /etc/group-. Anybody here done someting alike before? I mean, I can't just copy it over, because the system accounts shouldn't be overwritten. I suppose a 'grep -v' (for inverse matching) should get us quite far filtering out system (uid<500) accounts.
Human users should maintain their passwd records(and implicitly, their uid). A little awk will do the trick
Ah. Thanks for pointing me to awk. It's definitely not a tool which is top of mind for me. I've got a script which works for /etc/passwd and can trivially be adapted to /etc/groups. All that remains is ofcourse the migration of /etc/shadow and /etc/gshadow, which need to be filtered for /etc/passwd and /etc/groups lines being migrated. This should also be rather trivial.
Another issue I'm thinking about is that we're going to switch to subdomains for hosting our services. However, the benefits of such a change seem low if we don't switch to a structure where our data is also grouped per service. Currently everything is in /project, which is grouped around projects, requiring all services to run on a single server because they need to access specific directories - one in each project. I'm thinking we should reverse the design: grouping data per service and making /project a set of symbolic links into the service directories. That way, all that's required is that the links point to the right locations in the file system, possibly NFS mount points, if the service were to run on a different server.
This is a good idea, especially since it requires very little additional work, just a little care not to break users' existing setups
Right. However, short term steps (sorry, I started about it myself...) would be: * Migrate basic data, switching to NFS access [this I can do tonight] * Migrate the CRON jobs * Migrate the password and group files [after this step, people can log in and test to see if svn, cvs, hg, darcs and git really work] * Switch ssh access to the new box * Migrate anonymous access services (www, subversion [svn://, add http://], cvs, trac, git, others?] * Migrate mail service [Hans Huebner will take this up as soon as he's got time] Comments? Additions? Erik.
Hi Stelian,
My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished.
If you make a backup while the server is still accessible you'll lose data modified after the backup but before the restore and switch to the new server.
Absolutely agreed. So, copying the data from an lvm snapshot is not the only action. However, in the past, I've been able to reduce downtimes of servers to just minutes by copying the bulk of the data in advance, followed by bringing down the server, doing a quick rsync and bringing up the new server. It's the plan I intend to use this time aswell.
Well, this part (moving everything to NFS) is finished. lisppaste has been restarted, but is rebuilding its in-memory metadata database from disc, so it may take a while to come back up. Please report any issues. Bye, Erik.
On Sat, 16 Apr 2011, Erik Huelsmann wrote:
Hi Stelian,
My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished.
If you make a backup while the server is still accessible you'll lose data modified after the backup but before the restore and switch to the new server.
Absolutely agreed. So, copying the data from an lvm snapshot is not the only action. However, in the past, I've been able to reduce downtimes of servers to just minutes by copying the bulk of the data in advance, followed by bringing down the server, doing a quick rsync and bringing up the new server. It's the plan I intend to use this time aswell.
Well, this part (moving everything to NFS) is finished. lisppaste has been restarted, but is rebuilding its in-memory metadata database from disc, so it may take a while to come back up. Please report any issues.
Sorry for being slow to comment, but do we really want to be using NFS? NFS is slow, has a tendency to lock up, has a tendency to leave invincible .nfsXYZ files, etc. etc. http://www.jargon.net/jargonfile/n/NightmareFileSystem.html We use it a lot at work, but I always convince the sysadmins to let me have a local home directory that gets rsync'd to nfs at night. Every month or two dozens of people around me will be frozen while I work happily. Sometimes that latter aspect leaves me feeling stupid... If different subdomains are hosted on different servers, couldn't the relevant files be hosted on those servers as well? Is there a compelling need to have the SVN database on the same filesystem as user home directories, project webpages, and mailing lists? - Daniel
On Sun, Apr 17, 2011 at 3:37 AM, Daniel Herring <dherring@tentpost.com> wrote:
Sorry for being slow to comment, but do we really want to be using NFS?
NFS works very well most of the time and it is easy to administer. It might not be the best network file system that exists and it certainly can't be as dependable as a local file system, but then again, common-lisp.net is not a high availability environment, does not generate a lot of load and the NFS access will be local within a virtual machine host. -Hans
participants (4)
-
Daniel Herring -
Erik Huelsmann -
Hans Hübner -
Stelian Ionescu