- clo-devel - mailman3.common-lisp.net

[clo-devel] Migration status 2011-04-26
by Erik Huelsmann 27 Apr '11

27 Apr '11

Although last week's attempt to run cl-net off a network FS failed, this weekend a renewed attempt succeeded. This means we'll have completely current data on both old and new cl.net. This provides a solid basis to start migrating different services to the new machine. One of the things currently being moved is the ssh authenticated access. Other services to migrate soon are: * trac.common-lisp.net * paste.lisp.org / the bots * www.common-lisp.net / the webserver setup (with proxies and all) * the cron jobs * MTA/mailing lists * others Stelian and I are working through the steps and issues slowly on #tech.coop; if you want to discuss with us or want to know about progress or help out, come to the channel, hang around and join our efforts. Bye, Erik.

3 2

[clo-devel] clnet down?
by Marco Antoniotti 19 Apr '11

19 Apr '11

Hello is clnet down while you are doing the move? I cannot get to it. -- Marco Antoniotti

2 1

[clo-devel] File ownership: uid/gid design issue
by Erik Huelsmann 18 Apr '11

18 Apr '11

While browsing around, I noticed that we may have an issue with file ownership for Trac (but maybe wider): all files in a specific project under /project are owned by the project initiator with the group set to the unix group by the same name (all project files are world readable). This applies to all files, including Trac database. The problem arises when Trac (running as www-data under the http server) wants to modify the Trac database in the project. Since the user and group are set to values other than www-data, it can't write to the database file. In the specific case of Trac, this may probably be solved by running it against a postgresql backend instead of sqlite. However, possibly, other services [provided now or in the future] may require write access too. What's the usual way to solve this issue? Do people add the www-data user to all project groups? That doesn't seem right from the security perspective. Bye, Erik.

3 2

[clo-devel] 2011-04-15: cl-net migration progress
by Erik Huelsmann 17 Apr '11

17 Apr '11

My work on cl-net migration got stuck a little on tech.coop internal work this week. However, I'm fully back to the migration and I'm making copies of /project and /home right now. I suppose I'll be creating a copy of much inside /custom too, when the other two transfers have finished. At the same time, I'm trying to figure out how to best migrate /etc/password- and /etc/group-. Anybody here done someting alike before? I mean, I can't just copy it over, because the system accounts shouldn't be overwritten. I suppose a 'grep -v' (for inverse matching) should get us quite far filtering out system (uid<500) accounts. Another issue I'm thinking about is that we're going to switch to subdomains for hosting our services. However, the benefits of such a change seem low if we don't switch to a structure where our data is also grouped per service. Currently everything is in /project, which is grouped around projects, requiring all services to run on a single server because they need to access specific directories - one in each project. I'm thinking we should reverse the design: grouping data per service and making /project a set of symbolic links into the service directories. That way, all that's required is that the links point to the right locations in the file system, possibly NFS mount points, if the service were to run on a different server. Again, I'm interested in your comments and ideas. Regards, Erik.

4 5

[clo-devel] cl-net /custom/backup/old-projects directory?
by Erik Huelsmann 16 Apr '11

16 Apr '11

In the /custom/backup directory, there seem to be a number of old projects being archived. However, I doubt this is really useful: nobody can see that they ever existed in that location, so nobody can ask to be assigned maintainership of them. These are the projects I'm talking about: alpaca blitzblog clark clazy cl-darx cl-finance cl-me clotho cl-plus-j cl-torrent cl-trance gnucard hansa2 kmachine-cl monetdb-mapi ninep openAIR patg progintellect qrita xuriella Should we rather just delete these projects? Or reinstate them and create a page listing all inactive projects for which interested parties can ask to be assigned maintenance role/ commit access? Bye, Erik.

2 2

[clo-devel] cl-net upgrading update [2011-04-02]
by Erik Huelsmann 08 Apr '11

08 Apr '11

Since this mail, I've been quietly working on setting up the VPS we need anyway, while waiting for people to get learn about the move and hopefully join the mailing list. On Tue, Mar 29, 2011 at 11:35 PM, Erik Huelsmann <ehuels(a)gmail.com> wrote: > The server running common-lisp.net has been running in the current > configuration for a number of years now. Its Debian distribution is > still Etch, which has been moved to the archives June 20th last year. > As a consequence, we're locked out of even the most basic of > maintenance needs: security updates. > > Bottom line: common-lisp.net as it is today can't be maintained > anymore. We need to upgrade the system. > > We have two options for the system upgrade: > > 1. Copy the VPS and run dist-upgrade; then fix whatever falls over. > 2. Create a new VPS, and make a clean start, meaning that we identify > all services currently running on cl-net and we arrange for the same > or equivalent services to be running on the new system. > > > Stelian has indicated his preference is for option (2), as too much > has been customized or is installed in custom locations to make a > standard dist-upgrade work. I concur. Well, the new VPS is up and running. The technical configuration isn't quite to my liking, but we can definitely work with it. > Another remark from Stelian is that we could migrate one service at a > time, creating subdomains (svn.common-lisp.net, git.common-lisp.net, > trac...., etc) which might help future migrations. As far as the domains go: I've set up 'new.common-lisp.net' to point to the new VPS; all the other domains will come when we're ready to go live with the different services. > So, from where we stand now, we need to: > > 1) Identify all services, users and data we need to migrate > 2) Find people who want to help migrate the services > 3) Per component, identify a migration plan - we can start migrating > services with complete migration plans > 4) Test the new clean setups as well as the combination of the new > setup and the old data While at it, I've aptitude installed most of the services mentioned below. > Services that I know we're running [without looking at the box]: > > * Mail [MTA] - postfix, postfix-gld > * Web - apache2 > * Ticket tracker (RT - internal use) - otrs2 > * Mailing lists - mailman > * Version control - subversion, git, hg, darcs, cvs > * Repository browsing > - Subversion > - Git > - Darcs > - CVS (do we still want to support CVS ??) > * Trac - trac, trac-mercurial, trac-xmlrpc, trac-git Does anybody have experience with trac-mastertickets, trac-spamfilter, trac-email2trac or trac-accountmanager? How would they fit with the cl-net site setup? They all sound interesting. > * spam detection - spamassasin; although postfix integration still needs setting up > * Git deamon > * rsync (what for?) - rsync > * sshd - sshd > * ftpd --- Actually, no. The service is running but unused on current cl-net; I think we can use http for downloads and use scp for uploads. > * lisppaste (on sbcl-0.9.7??) -- to be done > * fast-cgi - libapache2-mod-fastcgi > * custom administration scripts -- to be done With the status above, we're getting closer to the point where we actually have to do something with the data and services on the current machine. Before we get that far, though, it would be nice if someone could configure the OTRS instance I set up on the new box for cl-net use. [I'll help, of course.] I've been thinking about the next steps. The easy steps would be: 1. copy /project data to the new server 2. nfs-mount /project on the old box (if we use NFSv3, both Subversion and CVS should be NFS safe, I understand) 3. migrate the anonymous read services for version control to the new box 4. copy and test the custom scripts, including installation of cron jobs etc 5. migrate trac.common-lisp.net to the new server and from there it requires a bit more thought to get things right. Remaining questions: * how to migrate the users password and other secure data * related: how to we bridge the gap between the time of the migration of the user account data (/etc/passwd and /etc/group) and the migration of the last service * which services need account (login) data [I know about Subversion commit, trac administration, cvs commit, website update; others?] Well, more questions and ideas how to proceed are *very* welcome. Bye, Erik.

2 4

[clo-devel] upgrading: how do you (want to) use cl.net? [detected: very large home directories]
by Erik Huelsmann 03 Apr '11

03 Apr '11

As part of the upgrading process, I'm looking at the disk sizes, memory, etc to set up for new-cl.net. One of the things I found is that 30 or 40 user accounts together occupy the nearly the full size allocated to /home (14gb out of 15gb allocated). Which raises the question: what do people do with their /home dirs and is /home really the best way to do that (or the best place to store it)? At the same time, I'm wondering if you have possibly valid use-cases for common-lisp.net which you might have tried in the past, but did not succeed at, because cl.net wasn't sufficiently set up. As I'm now determining dimensions, it'd be a good time to take requests into account. Bye, Erik.

4 6

[clo-devel] common-lisp.net needs upgrading
by Erik Huelsmann 31 Mar '11

31 Mar '11

The server running common-lisp.net has been running in the current configuration for a number of years now. Its Debian distribution is still Etch, which has been moved to the archives June 20th last year. As a consequence, we're locked out of even the most basic of maintenance needs: security updates. Bottom line: common-lisp.net as it is today can't be maintained anymore. We need to upgrade the system. We have two options for the system upgrade: 1. Copy the VPS and run dist-upgrade; then fix whatever falls over. 2. Create a new VPS, and make a clean start, meaning that we identify all services currently running on cl-net and we arrange for the same or equivalent services to be running on the new system. Stelian has indicated his preference is for option (2), as too much has been customized or is installed in custom locations to make a standard dist-upgrade work. I concur. Another remark from Stelian is that we could migrate one service at a time, creating subdomains (svn.common-lisp.net, git.common-lisp.net, trac...., etc) which might help future migrations. So, from where we stand now, we need to: 1) Identify all services, users and data we need to migrate 2) Find people who want to help migrate the services 3) Per component, identify a migration plan - we can start migrating services with complete migration plans 4) Test the new clean setups as well as the combination of the new setup and the old data If it's an option, I'd like the full migration to take 2 months at max from the day we start. I'd be fine with starting immediately and taking Monday April 4th as the reference day. Services that I know we're running [without looking at the box]: * Mail [MTA] * Web * Ticket tracker (RT - internal use) * Mailing lists * Subversion * Repository browsing - Subversion - Git - Darcs - CVS (do we still want to support CVS ??) * Trac * spam detection * Git deamon * rsync (what for?) * sshd * ftpd * lisppaste (on sbcl-0.9.7??) * fast-cgi * custom administration scripts Drew and I want to replace RT by OTRS which is also standard in Debian and we have good experiences with - I set one up recently for ourselves. So, I guess for now, there are two questions to be answered: 1. Do you know of any additional services that we need to migrate? 2. Who wants to help out and which services can you help migrating with? So far, I have volunteered to coordinate the effort. Bye, Erik.

5 5

[clo-devel] cl.net FTP not useful?
by Erik Huelsmann 31 Mar '11

31 Mar '11

In my investigations, I found a running ftp daemon. However, it seems this ftp daemon requires authentication (through a password). Everybody receives ssh access, meaning you can use sftp to upload to cl.net anyway. Because of the required authorization, this service doesn't seem to support a general public service. Do we really need to configure an ftp service on our new host? Bye, Erik.

2 1

[clo-devel] change of add-trac setup script
by Erik Huelsmann 15 Mar '11

15 Mar '11

Some time ago, I optimized Trac performance. One of the ways I achieved it is by having Apache serve static files instead of Trac itself. This requires an extra step (trac-admin deploy) when creating new Trac directories for projects. Today, after Raymond Toy reported issues with Trac projects set up for him, I changed the add-trac.sh script to include the extra step. Just for your information. Bye, Erik.

1 0