- clo-devel - mailman3.common-lisp.net

[clo-devel] cl-net upgrading update [2011-04-02]
by Erik Huelsmann 08 Apr '11

08 Apr '11

Since this mail, I've been quietly working on setting up the VPS we need anyway, while waiting for people to get learn about the move and hopefully join the mailing list. On Tue, Mar 29, 2011 at 11:35 PM, Erik Huelsmann <ehuels(a)gmail.com> wrote: > The server running common-lisp.net has been running in the current > configuration for a number of years now. Its Debian distribution is > still Etch, which has been moved to the archives June 20th last year. > As a consequence, we're locked out of even the most basic of > maintenance needs: security updates. > > Bottom line: common-lisp.net as it is today can't be maintained > anymore. We need to upgrade the system. > > We have two options for the system upgrade: > > 1. Copy the VPS and run dist-upgrade; then fix whatever falls over. > 2. Create a new VPS, and make a clean start, meaning that we identify > all services currently running on cl-net and we arrange for the same > or equivalent services to be running on the new system. > > > Stelian has indicated his preference is for option (2), as too much > has been customized or is installed in custom locations to make a > standard dist-upgrade work. I concur. Well, the new VPS is up and running. The technical configuration isn't quite to my liking, but we can definitely work with it. > Another remark from Stelian is that we could migrate one service at a > time, creating subdomains (svn.common-lisp.net, git.common-lisp.net, > trac...., etc) which might help future migrations. As far as the domains go: I've set up 'new.common-lisp.net' to point to the new VPS; all the other domains will come when we're ready to go live with the different services. > So, from where we stand now, we need to: > > 1) Identify all services, users and data we need to migrate > 2) Find people who want to help migrate the services > 3) Per component, identify a migration plan - we can start migrating > services with complete migration plans > 4) Test the new clean setups as well as the combination of the new > setup and the old data While at it, I've aptitude installed most of the services mentioned below. > Services that I know we're running [without looking at the box]: > > * Mail [MTA] - postfix, postfix-gld > * Web - apache2 > * Ticket tracker (RT - internal use) - otrs2 > * Mailing lists - mailman > * Version control - subversion, git, hg, darcs, cvs > * Repository browsing > - Subversion > - Git > - Darcs > - CVS (do we still want to support CVS ??) > * Trac - trac, trac-mercurial, trac-xmlrpc, trac-git Does anybody have experience with trac-mastertickets, trac-spamfilter, trac-email2trac or trac-accountmanager? How would they fit with the cl-net site setup? They all sound interesting. > * spam detection - spamassasin; although postfix integration still needs setting up > * Git deamon > * rsync (what for?) - rsync > * sshd - sshd > * ftpd --- Actually, no. The service is running but unused on current cl-net; I think we can use http for downloads and use scp for uploads. > * lisppaste (on sbcl-0.9.7??) -- to be done > * fast-cgi - libapache2-mod-fastcgi > * custom administration scripts -- to be done With the status above, we're getting closer to the point where we actually have to do something with the data and services on the current machine. Before we get that far, though, it would be nice if someone could configure the OTRS instance I set up on the new box for cl-net use. [I'll help, of course.] I've been thinking about the next steps. The easy steps would be: 1. copy /project data to the new server 2. nfs-mount /project on the old box (if we use NFSv3, both Subversion and CVS should be NFS safe, I understand) 3. migrate the anonymous read services for version control to the new box 4. copy and test the custom scripts, including installation of cron jobs etc 5. migrate trac.common-lisp.net to the new server and from there it requires a bit more thought to get things right. Remaining questions: * how to migrate the users password and other secure data * related: how to we bridge the gap between the time of the migration of the user account data (/etc/passwd and /etc/group) and the migration of the last service * which services need account (login) data [I know about Subversion commit, trac administration, cvs commit, website update; others?] Well, more questions and ideas how to proceed are *very* welcome. Bye, Erik.

2 4

[clo-devel] upgrading: how do you (want to) use cl.net? [detected: very large home directories]
by Erik Huelsmann 03 Apr '11

03 Apr '11

As part of the upgrading process, I'm looking at the disk sizes, memory, etc to set up for new-cl.net. One of the things I found is that 30 or 40 user accounts together occupy the nearly the full size allocated to /home (14gb out of 15gb allocated). Which raises the question: what do people do with their /home dirs and is /home really the best way to do that (or the best place to store it)? At the same time, I'm wondering if you have possibly valid use-cases for common-lisp.net which you might have tried in the past, but did not succeed at, because cl.net wasn't sufficiently set up. As I'm now determining dimensions, it'd be a good time to take requests into account. Bye, Erik.

4 6

[clo-devel] common-lisp.net needs upgrading
by Erik Huelsmann 31 Mar '11

31 Mar '11

The server running common-lisp.net has been running in the current configuration for a number of years now. Its Debian distribution is still Etch, which has been moved to the archives June 20th last year. As a consequence, we're locked out of even the most basic of maintenance needs: security updates. Bottom line: common-lisp.net as it is today can't be maintained anymore. We need to upgrade the system. We have two options for the system upgrade: 1. Copy the VPS and run dist-upgrade; then fix whatever falls over. 2. Create a new VPS, and make a clean start, meaning that we identify all services currently running on cl-net and we arrange for the same or equivalent services to be running on the new system. Stelian has indicated his preference is for option (2), as too much has been customized or is installed in custom locations to make a standard dist-upgrade work. I concur. Another remark from Stelian is that we could migrate one service at a time, creating subdomains (svn.common-lisp.net, git.common-lisp.net, trac...., etc) which might help future migrations. So, from where we stand now, we need to: 1) Identify all services, users and data we need to migrate 2) Find people who want to help migrate the services 3) Per component, identify a migration plan - we can start migrating services with complete migration plans 4) Test the new clean setups as well as the combination of the new setup and the old data If it's an option, I'd like the full migration to take 2 months at max from the day we start. I'd be fine with starting immediately and taking Monday April 4th as the reference day. Services that I know we're running [without looking at the box]: * Mail [MTA] * Web * Ticket tracker (RT - internal use) * Mailing lists * Subversion * Repository browsing - Subversion - Git - Darcs - CVS (do we still want to support CVS ??) * Trac * spam detection * Git deamon * rsync (what for?) * sshd * ftpd * lisppaste (on sbcl-0.9.7??) * fast-cgi * custom administration scripts Drew and I want to replace RT by OTRS which is also standard in Debian and we have good experiences with - I set one up recently for ourselves. So, I guess for now, there are two questions to be answered: 1. Do you know of any additional services that we need to migrate? 2. Who wants to help out and which services can you help migrating with? So far, I have volunteered to coordinate the effort. Bye, Erik.

5 5

[clo-devel] cl.net FTP not useful?
by Erik Huelsmann 31 Mar '11

31 Mar '11

In my investigations, I found a running ftp daemon. However, it seems this ftp daemon requires authentication (through a password). Everybody receives ssh access, meaning you can use sftp to upload to cl.net anyway. Because of the required authorization, this service doesn't seem to support a general public service. Do we really need to configure an ftp service on our new host? Bye, Erik.

2 1

[clo-devel] change of add-trac setup script
by Erik Huelsmann 15 Mar '11

15 Mar '11

Some time ago, I optimized Trac performance. One of the ways I achieved it is by having Apache serve static files instead of Trac itself. This requires an extra step (trac-admin deploy) when creating new Trac directories for projects. Today, after Raymond Toy reported issues with Trac projects set up for him, I changed the add-trac.sh script to include the extra step. Just for your information. Bye, Erik.

1 0

[clo-devel] Neglected projects
by "john fred connors" 19 Jan '11

19 Jan '11

Hi, I have two projects on common-lisp.net which I am admin of : cl-x86-asm and cl-screen. Its a long time since I looked at them. I'd like a password reset on them, if possible and the associated mailing lists cl-x86-asm-devel, cl-screen-announce, cl-screen-devel, and cl-x86-asm-announce. I'd like at least to tidy up the projects and leave them in a more usable state now that I am a more experienced CL programmer, to justify their inclusion in cl-net. Thanks in advance, Johh Connors.

2 1

[clo-devel] Mailman password for cl-muproc-announce & -devel?
by Klaus Harbo 17 Jan '11

17 Jan '11

I am the admin of the essentially non-active cl-muproc-announce and cl-muproc-devel mailing lists. There is now, however, a spam mail which desires my attention -- but I seem to have misplaced the mailman passwords for the two lists. Can anyone assist me in getting the passwords reset, please? I do have the password for login access to common-lisp.net. Thanks in advance! brgds Klaus Harbo

2 1

[clo-devel] /custom put into git
by Hans Hübner 30 Apr '10

30 Apr '10

I've turned /custom into a git repository as I sometimes change scripts and want you to be able to keep track of the changes. If someone feels so inclined, please add a commit mail hook to this list. -Hans

1 0

[clo-devel] Proposal for simplification of services
by Erik Huelsmann 05 Apr '10

05 Apr '10

Hi, Our Trac service seems to have stabilized now that we disallow search engines to walk through the full Subversion history of all projects. (Tickets are still indexed, if the engines will do it, but queries have been disabled too.) My proposal is to remove the WebSVN service from c-l.net: the program hasn't been maintained for ages and it provides only a subset of the functions provided by Trac. With kind regards, Erik.

3 2

[clo-devel] Finding out how to set up Trac on fastcgi
by Erik Huelsmann 11 Mar '10

11 Mar '10

Since we seem to have lots of issues with our Trac install (most people find it slow; some find its availability disappointing), I thought I'd look at the Trac website to find out how their dev team advises on these matters. Well, the conclusion is simple: we're using the 'tracd' method of running Trac and this is certainly the only way how *not* to do it. They advise CGI for small sites or FastCGI/WSGI for larger sites. I estimate us to be in the latter category, so I took my own VPS (which is in the former category, but also runs Trac) and configured it for FastCGI. The steps turn out to be quite simple. I volunteer to make the same changes on c-l.net (which drewc already set me up for), but I'm posting here for review. The last thing I'd want is to open up a security hole in the site. Steps are: * Install mod_fcgid (although I used mod_fastcgi, which is older and not recommended by the Trac team; it works though) * Add the script handlers and the script alias to the httpd.conf file within the virtual host section for trac.c-l.net: AddHandler fastcgi-script .fcgi ScriptAlias / /path/to/apache/cgi-bin/trac.fcgi/ * copy the 'web/' subdir of the installed Trac into the cgi-bin directory * copy in the cgi-bin directory "fcgi_frontend.py" to "trac.fcgi" * add to the trac.fcgi script - before the first 'import' statement: import os os.environ['PYTHON_EGG_CACHE'] = "/tmp/httpd-egg-cache" os.environ['TRAC_ENV_PARENT_DIR'] = "/path/to/parent/trac/project/dirs" * create /tmp/httpd-egg-cache with the user and group used to run httpd and '700' access settings * restart apache Comments? The effect of the change is this: by default, a single trac.fcgi application will be loaded - just as with our current situation. However, on high demand, additional instances will be spawned - contrary to the current situation, where a single tracd will try to handle all requests. When the load is over, the number of servers is scaled back to normal (1). Bye, Erik.

3 4