October 2018 - clo-devel - mailman3.common-lisp.net

Proposal: project pages deployment through GitLab
by Erik Huelsmann 18 Nov '20

18 Nov '20

Hi all, As you all have noticed, I've been working to clean up common-lisp.net and simplifying the administration process. One of the things we have done is that we implemented a deployment pipeline for the common-lisp.net main site using GitLab. It's been a great joy to work with it so far and made deployment of new site content easier than ever before. *My proposal is to set up a GitLab CI based deployment pipeline* for *all common-lisp.net <http://common-lisp.net> projects*. Meaning that I'm proposing to import the current project pages (/project/*/public_html) into GitLab repositories (<project-name>-site) with a gitlab-ci file which causes the content to be published. The approach above will mean simple import of the existing static content. However, after import, the static output can be replaced by different input and a static content generator, just like we did with the common-lisp.net site. Eager to hear your thoughts, Erik.

5 8

Re: Proposal: project pages deployment through GitLab
by Erik Huelsmann 01 Nov '18

01 Nov '18

Hi Vladimir, On Tue, Oct 30, 2018 at 8:15 AM Vladimir Sedach <vas(a)oneofus.la> wrote: > > While you're probably correct about the move from SourceForge to Trac, > > there's definitely not something like that going on in the current > > proposal: you use TRAMP to build your static assets and you will keep > doing > > that, even after deployment moves to GitLab. GitLab Pages is - from a > user > > perspective - nothing more than a pipeline which specifies where the > > generated output is. > > TRAMP is not a static site generator, it is an Emacs virtual > file system for editing files and running commands remotely, that > works over SSH. I do not use a site generator for c-l.net projects. > Ah. Ok. My bad. I assumed TRAMP was a site generator. > > That pipeline/CI job builds your static content using > > your own preferred static site generator (in case of c-l.net, that's > some > > amalgamation of cl-markdown, plump, cl-who, etc. and in your case that's > > TRAMP). Building in this context can be as simple as "take the pre-built > > pages stored in a Git repository and declare those to be the build > output". > > The additional bit that Pages does is to take the output from a CI build > > and deploy it on the Pages host. This last step simply means the pages > > become available as https://common-lisp.net/project/<some-project>/... > > My concern is that this requires you to use git, Yes. Another assumption: everybody keeps their (site/software) assets in source control/version control these days. Isn't your common-lisp.net site versioned? > and then requires you to use Gitlab Sure. That's where you send your version controlled assets anyway, right? On my part (as a maintainer of the site) this isn't much more effort since everything required is already in place (from my perspective). > (and it sounds like you also need to configure > Gitlab Pages to know about files in your git repository), Well, you'll be given that by me upon the initial creation/import of your site into GitLab. All you need to do (when you want to go without a static site builder) is to edit your pages and push them to your repo. all so that Gitlab can copy some files to > https://common-lisp.net/project/<some-project>/. That is a lot of > dependencies if you only want to copy some files. I understand what you're saying and I appreciate the feedback. I also understand you would be annoyed if the only way to deploy would be to go through GitLab. Would you go so far as to say that you would move your project pages elsewhere, though? (Not threatening or anything, really just trying to understand your situation.) As you have encountered already, this approach has problems. Yea. I would assume every option has problems. The existing way of working has problems as well in the sense that they put a maintenance effort on the common-lisp.net admin team and I can fix issues from my mobile phone when I need to deal with GitLab issues whereas I really need a computer with terminal (I know there are terminals for phones, but they don't work for me) to fix issues with any other part of the setup. Also, people expect certain applications to be installed when logins are provided. GitLab defines the service level pretty well and all that in a single installation package. > It is a nice option, but I do not think it should be the only option. > That's clear. Thanks a bunch for the feedback! I'll check what I can do to allow multiple deployment methods to co-exist. I'm eager to hear from others regarding this topic as well, though! Regards, Erik.

3 3

Does the "clo" project warrant its own project page (/project/clo/)?
by Erik Huelsmann 31 Oct '18

31 Oct '18

Hi, Right at this time the clo project has its own project page ( https://common-lisp.net/project/clo/) which lists an old version of the front page of the common-lisp.net domain server. Does the "clo" project warrant a page of its own? Or do we use the common-lisp.net "main site" to explain both the purpose of the site, document the services we run on it and refer people to the resources available to contribute? In other words: do I remove(archive) /project/clo/ or does it serve a (renewed) purpose? Regards, Erik.

2 1

Stefil site forwarding to non-existing dwim.hu page
by Erik Huelsmann 30 Oct '18

30 Oct '18

Hi Attila, In the past you have decided to move projects from common-lisp.net to dwim.hu -- which is fine of course. However, at least the stefil project page now refers to a non-existing dwim.hu page. As we'd like to keep our resources relevant, we'd like them to be current. Could you update the Stefil (and other projects?) page to update to a current resource holding the Stefil webpage and repository? If there currently is no such resource, we'd be happy for you to put the (obsoleted?) stuff back on common-lisp.net where we can host it for whoever is interested in the code. Regards, Erik.

1 0

Re: Proposal: project pages deployment through GitLab
by Erik Huelsmann 30 Oct '18

30 Oct '18

Hi Vladimir, Thanks for your reaction! On Mon, Oct 29, 2018 at 3:38 AM Vladimir Sedach <vas(a)oneofus.la> wrote: > [ .. snip .. ] > > This all seems like a very complicated way of hosting static files. I > think trying to do something as fundamental as that by forcing it > through a huge piece of software like Gitlab is a mistake. Six or > eight years from now, Gitlab is going to go out of fashion, the way > Trac has. If some people want to use Gitlab generated pages, that is > a valuable feature right now, but it is not going to be very future > proof. > > I use TRAMP and rsync to edit project pages and copy files and would > definitely prefer to have plain SSH and SCP. It was a lot easier to > transition project pages from HTML4 to XHTML to HTML5 with mobile > support than it would have been to move from SourceForge to Trac to > GitHub to Gitlab etc. Plain hosting also gives people the option of > using their preferred static site generator instead of Gitlab. > I see there's a bit of a misunderstanding going on and I apologize that I wasn't more explicit on what this change would mean. While you're probably correct about the move from SourceForge to Trac, there's definitely not something like that going on in the current proposal: you use TRAMP to build your static assets and you will keep doing that, even after deployment moves to GitLab. GitLab Pages is - from a user perspective - nothing more than a pipeline which specifies where the generated output is. That pipeline/CI job builds your static content using your own preferred static site generator (in case of c-l.net, that's some amalgamation of cl-markdown, plump, cl-who, etc. and in your case that's TRAMP). Building in this context can be as simple as "take the pre-built pages stored in a Git repository and declare those to be the build output". The additional bit that Pages does is to take the output from a CI build and deploy it on the Pages host. This last step simply means the pages become available as https://common-lisp.net/project/<some-project>/... In that sense, while GitLab Pages may not be the fashion of tomorrow, this isn't much of a problem because it's not dictating technology like Trac used to do and as such qualifies (to me) as plain hosting. Basically the only thing that changes is "plain (SSH) access" in the sense that GitLab does your deployment, not rsync or SCP. Is this how you understood GitLab Pages to work? If not, with this additional explanation, how do you feel about the proposal? Regards, Erik.

2 1

Projects without project page
by Erik Huelsmann 30 Oct '18

30 Oct '18

Hi, Working on cleanup this weekend, I found that roughly one quarter of all projects never replaced their auto-generated project page. These projects will likely (I have yet to find out) come in two groups: those which never got off the ground (no commits in source control) and those which simply didn't bother setting up a project page. I'm thinking to point the project page to the source repository (e.g. GitLab) when the project has sources, but doesn't have a project page. As for the projects which never spawned code, I'm wondering: should we simply remove those off the system (after contacting the project members, of course)? I mean, there's little Common Lisp value in them... Regards, -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

6 14

Re: Proposal: project pages deployment through GitLab
by Erik Huelsmann 29 Oct '18

29 Oct '18

Hi Vladimir, On Sun, Oct 28, 2018 at 6:32 AM Vladimir Sedach <vas(a)oneofus.la> wrote: > > Ok. Good. It's probably good to note that in my plans this type of > project > > page deployment will be the *only* supported way for project page > > deployment once introduced. (Given the little feedback I received last > time > > I pitched the idea, I thought it was a good idea to mention this fact.) > > > > I've analysed the content of some of the largest /project/*/public_html > > directories to prepare for this move. While we made a huge step in > reducing > > the content of public_html/ directories which wasn't really "site > content" > > (yet, rather Git or Darcs repositories), some of the largest ones are > using > > the public_html/ directory for binary artifact distribution (mostly > release > > binaries). I'll have a look at what we can do to support hosting that > kind > > of content (maybe with an artifact repository of some kind?) instead of > > depending on public_html/ for it. > > I'm not following. What exactly does "importing the current project > pages into GitLab repositories" entail, and why is that a problem for > release tarballs? Is it not just putting the static site files into > source control? > Yes, that's what it means. And for smaller numbers of tarballs that should be fine. However, there are projects which host all releases since 2003 on our site. Importing those into a Git repository would result in gigabyte-sized repositories. By themselves, having a Gigabyte sized repository isn't a problem, but having to download all releases since 2003 (3GB?) in order to make a site update isn't practical and something I want to put projects through. Once cloned, the project will even take double the space as the zip files are in the ref files as well as in the working directories of the git repository. Hence my idea that we need a means to manage those released files separately. > One other thing that should be preserved is hosting for user pages, > like https://common-lisp.net/~vsedach/ Ok. Would it be problematic for you if the user-space would be hosted/supported the same way as the project pages will? As in: your personal "site" can be dealt with through pushing to a git repository and the downloads managed through some means I yet need to figure out to deal with large(r) numbers of large files. I guess my question is: does your use-case get sufficiently covered by the solution I'm proposing? (And if not, can you provide more details so I can understand your use-case?) Regards, Erik.

2 1

Change of policy on GitLab account creation/login?
by Erik Huelsmann 24 Oct '18

24 Oct '18

Hi all, Almost 4 years have passed since the initial installation of GitLab on common-lisp.net. During that period, GitLab (the open source project) has seen tremendous growth both in user community and functionality. Our own GitLab instance has grown quite a bit as well. With almost 700 projects and nearly 500 users, it's quite a bit bigger now than the initial CVS migration which resulted in 379 projects and 395 users. (We did both the Git and Darcs migrations after that, which added to the growth.) Originally, we decided we wanted to create our own accounts and not support account-auto-creation (based on experience shared by GitLab.com) Today, GitLab lets itself be configured to auto-create accounts from GitHub using GitHub credentials. We have long supported the possibility of allowing accounts to be tied to Google and GitHub OAuth2 for the purpose of authentication. Additionally, we support 2FA these days. I'm wondering if we can be a bit more relaxed with account creation (allowing auto-creation), but be a bit more strict on account login: i.e. require 2FA. Do you have any opinion on the matter? -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

3 5

[missing implementations]
by Daniel Kochmański 14 Oct '18

14 Oct '18

Hey, new website is great. There are though a few suprises: - clisp and lispworks are not listed in Resources -> Common Lisp Implementations - lispworks is not listed on getting started download page, but allegro is (since two free implementations are listed, it would be justified to mention two most popular commercial ones) Best regards, Daniel

8 16

GitLab now has 'reply by e-mail' enabled
by Erik Huelsmann 14 Oct '18

14 Oct '18

Hi, Today I found some time to enable GitLab's 'reply by e-mail' functionality. When you receive a notification e-mail from our GitLab instance, you can now send a reply to that notification. If the notification was from an issue being updated, then your reply will update the issue with a follow-up commont. Regards, Erik -- Bye, Erik. http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.

2 2