Hi everybody,
As a follow-up to the changes that I proposed for common-lisp.net to be implemented over the course of 2015 [ https://mailman.common-lisp.net/pipermail/clo-devel/2015-February/000161.htm...], I've been looking at the more detailed steps to be executed to implement the proposal. Remember that the changes are targetting a more modern, but also a simplified (for users *and* admins) system. It's an investment, but one in simpler maintenance while improving user experience. Currently, I'm thinking that the steps (no timing yet) should be:
0. Install a Code Commenting plugin on Trac to make it match the GitLab code commenting capabilities 1. Finish my experiments regarding the GitLab setup / installation 2. Install GitLab on common-lisp.net (under the gitlab.common-lisp.net domain? or should we prefer git.common-lisp.net?) 3. Run the migration to create all users and groups in GitLab --> do we want all project which we know *don't* use Git to be created in GitLab too? 4. Import the CMUCL git repositories (using the script to be used for all projects) 5. Run a trial period of 2 months with Raymond Toy and CMUCL to iron out any unnoticed issues 6. Import the user's git repositories 7. Import all other projects with Git repositories 8. Turn off gitweb --> introduce rewrite rules to point to gitlab 9. Turn off git plugin for Trac --> introduce rewrite rules to point to gitlab 10. Turn off git-daemon (fully depend on https checkouts) 11. Convert all project darcs repositories to git --> Notify all darcs project owners before we do about a planning/timing 12. Import the converted darcs repositories 13. Convert users' darcs repositories --> Notify all darcs users before we do about a planning/timing 14. Import converted users' darcs repositories 15. Turn off darcsweb webbrowsing --> introduce rewrite rules to point to gitlab 16. Contact CVS repository owners to ask if they want to migrate to Subversion or Git --> Which do we default to in case of no answer? 17. Convert CVS repositories to Subversion (where requested / defaulted) 18. Convert CVS repositories to Git (where requested / defaulted) 19. Import git-converted CVS repositories 20. Turn off ViewVC (Subversion & CVS) web-browsing --> introduce rewrite rules to point to gitlab/Trac 21. Turn off cvsd and related cron jobs 22. Turn off svnserve (fully depend on https checkouts)
"Import git repository" here means "move into gitlab space", which means the user will *only* be able to access the repository through GitLab's access methods (HTTPS and SSH(git user)) . Before we go to that step, we'll probably need to announce this step to all users who have git repositories in their home directories. (So they can object to the move -- if they do, the repository won't be moved, but we *will* scrap gitweb and git-daemon...)
Every step which says "Import" or "Convert" entails the creation (and testing) of a scripted procedure, as far as I'm concerned. Some steps will necessarily be executed in sequence. However some parts may be executed in parallel (e.g. contacting users about preferences and developing the scripted procedure). I'm hoping that when I sign up for this task, others pick up the rest of the work that relates to cl-net, such as fixing the mailing list archives or getting OAuth working for both Trac and GitLab (assuming we want that; do we?).
The order above is explicitly selected to benefit as soon as possible from having GitLab up and running, with namespaces reserved (hence the full group and user creation right at the start), but possibly unused at first (due to later imports of projects and repositories). I'm not completely sure about steps 10 and 22; if we can make step 10 work, I'm inclined to research if we can keep step 22 as well.
So, judging by the number of steps to be executed, this may take all year, with one or two steps every month. I don't have a problem with that, though thought it'd be good to set expectations.
So, apart from the fact that it looks like an enormous amount of work, what are your comments? (Note that I see this as a consequence of years and years maintenance backlog; so, it's not something that I expect to be solved in a day or two.)
"Erik" == Erik Huelsmann ehuels@gmail.com writes:
Erik> Hi everybody,
Hmm. Is this some kind of Friday the 13th joke? :-)
These plans seem awfully ambitious, but I look forward to them.
Erik> 0. Install a Code Commenting plugin on Trac to make it match Erik> the GitLab code commenting capabilities
Is this step necessary? We haven't had one thus far, and it seems that gitlab is the way to go, so adding functionality to Trac is just more work.
Depending on the difficulty, I think I would rather just migrate all of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
In any case, I'm excited about these changes.
-- Ray
Hey guys,
I know I “disappeared" for a long time and I’m not an important one, but I have one suggestion here. If I’m wrong or you don’t like it, please don’t blame me:
We should move all responsibilities to Github (or sourceForge if SVN or CVS is still required by some projects), and then ONLY use common-lisp.net for project web site hosting (project web pages, and file downloads for users who don’t use quicklisp) , because the hostname (common-lisp.net) is still the best choice for Common Lisp related projects.
Regards,
Chun Tian (binghe)
Il giorno 13/feb/2015, alle ore 17:23, Raymond Toy toy.raymond@gmail.com ha scritto:
"Erik" == Erik Huelsmann ehuels@gmail.com writes:
Erik> Hi everybody,
Hmm. Is this some kind of Friday the 13th joke? :-)
These plans seem awfully ambitious, but I look forward to them.
Erik> 0. Install a Code Commenting plugin on Trac to make it match Erik> the GitLab code commenting capabilities
Is this step necessary? We haven't had one thus far, and it seems that gitlab is the way to go, so adding functionality to Trac is just more work.
Depending on the difficulty, I think I would rather just migrate all of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
In any case, I'm excited about these changes.
-- Ray
Clo-devel mailing list Clo-devel@common-lisp.net https://mailman.common-lisp.net/cgi-bin/mailman/listinfo/clo-devel
Hi Chun,
I know I “disappeared" for a long time
No problem. Sometimes Life just takes over :-)
and I’m not an important one,
Well, since we're not a huge community, I think we should value the opinion of everybody who takes the time to weigh in (send a response).
but I have one suggestion here. If I’m wrong or you don’t like it, please don’t blame me:
We should move all responsibilities to Github (or sourceForge if SVN or CVS is still required by some projects), and then ONLY use common-lisp.net for project web site hosting (project web pages, and file downloads for users who don’t use quicklisp) , because the hostname (common-lisp.net) is still the best choice for Common Lisp related projects.
Well, I do understand where your reaction comes from. However, we have a number of projects who are still very much active. *Only* doing project website hosting disregards the fact that one of our most widely acknowledged service being the mailing lists and especially the archives thereof.
For me personally, there's no problem in doing the work for the migration. It's one step less extreme than just moving everything out to other parties. But for me (and probably others) to be able to maintain the system, it's only doable when we rationalize the system's setup. I'm thinking that your proposed "throwing it all out" will marginalize the role of common-lisp.net which really would be too bad.
Even if other projects decide not to host directly on common-lisp.net themselves, we could be a mirror. Then, when project pages go blank (and they sometimes do), we have a copy of the code base and others can take over from there.
There may be other options and ideas too.
Hi Chun Tian,
Thanks for the feedback. My $0.02:
I think common-lisp.net should indeed support project websites with their repositories hosted on github/bitbucket/sourceforge, but if volunteer administrators (currently led by Erik H) are willing to support repository hosting directly on common-lisp.net then why would we not want that as well?
Already there are hundreds of projects hosted at common-lisp.net and it will be less disruptive for them to get consolidated under gitlab than force all of them to relocate to another site entirely.
A bit further down the road, I have hopes that common-lisp.net will be able to offer CL-specific services, for example:
o private-repository hosting (paid)
o source escrow services (paid)
o online automated build verifications with multiple CL implementations (including defined Quicklisp snapshots), connected with the two items above
o online automated regression testing (connected with above)
o support for CL-based web application deployment and hosting (paid for proprietary, free for open-source). Eventually all of common-lisp.net itself should be set up as a CL-based web application.
I would contend that such services can only be delivered effectively from a CL-focused perspective, not by melding into and depending everything on whatever generic, commodity "cloud" services are currently popular.
With a combination of recurring donations plus paid services like these, my hope is that common-lisp.net can continue to be a viable and self-sustaining service as well as focal point for CL-related resources.
I'll go on record and say that Genworks would be at least one paying customer for services such as what I've listed above.
Best Regards,
Dave
On Fri, Feb 13, 2015 at 3:07 PM, Chun Tian (binghe) binghe.lisp@gmail.com wrote:
Hey guys,
I know I “disappeared" for a long time and I’m not an important one, but I have one suggestion here. If I’m wrong or you don’t like it, please don’t blame me:
We should move all responsibilities to Github (or sourceForge if SVN or CVS is still required by some projects), and then ONLY use common-lisp.net for project web site hosting (project web pages, and file downloads for users who don’t use quicklisp) , because the hostname (common-lisp.net) is still the best choice for Common Lisp related projects.
Regards,
Chun Tian (binghe)
Il giorno 13/feb/2015, alle ore 17:23, Raymond Toy toy.raymond@gmail.com ha scritto:
> "Erik" == Erik Huelsmann ehuels@gmail.com writes:
Erik> Hi everybody,
Hmm. Is this some kind of Friday the 13th joke? :-)
These plans seem awfully ambitious, but I look forward to them.
Erik> 0. Install a Code Commenting plugin on Trac to make it match Erik> the GitLab code commenting capabilities
Is this step necessary? We haven't had one thus far, and it seems that gitlab is the way to go, so adding functionality to Trac is just more work.
Depending on the difficulty, I think I would rather just migrate all of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
In any case, I'm excited about these changes.
-- Ray
Clo-devel mailing list Clo-devel@common-lisp.net https://mailman.common-lisp.net/cgi-bin/mailman/listinfo/clo-devel
Clo-devel mailing list Clo-devel@common-lisp.net https://mailman.common-lisp.net/cgi-bin/mailman/listinfo/clo-devel
These plans seem awfully ambitious, but I look forward to them.
Ok. Well, we'll take it one step at a time; it's my expectation that each step by itself is manageable.
Depending on the difficulty, I think I would rather just migrate all
of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
Well, if you're up to writing CL code to do the migration, I think it's doable:
** Wiki - Raw documents can be dowloaded from Trac, so if you have a list of pages to download, you can simply run those through cURL/wget. (URL format: https://trac.common-lisp.net/cmucl/wiki/GitAndCmucl?format=txt) - it looks like [http://johnmacfarlane.net/pandoc/ PanDoc] is your way to go for the conversion of the wiki format - I'm working on a cl-gitlab library which can be used to connect to the gitlab API and upload wiki documents
Hmm. come to think of it, you probably want to migrate the wiki into a local repository and push them to the server as described here: https://gitlab.com/gitlab-org/omnibus-gitlab/wikis/git_access
** Tickets - Tickets can be downloaded in one big XML document (URL: https://trac.common-lisp.net/cmucl/login?referer=%2Fcmucl%2Freport%2F11%3Fas... ) - After extracting the above URL, cl-gitlab can be used to create the milestones used in the extract - cl-gitlab can be used to (re)create the tickets from trac, where I suggest to move these fields to the body text - Not all fields available on Trac are available in the GitLab tracker; i'd propose adding them to the ticket's body; specifically: Reporter and Version - Fields "Priority" and "Component" can be transformed into Labels (a kind of "tags") - Successive comments should be uploaded as comments/notes; this can be done using cl-gitlab as well.
Note that both the wiki and the tickets can use Wiki markup in Trac as well as GitLab, so every text body (comment, wiki page or ticket) probably needs to be run through pandoc.
If there are more projects that want/need such conversion, we'd probably best try to share any efforts automating the process.
Hi Erik,
I just noticed this elisp gitlab library the other day. Perhaps it will be helpful in producing cl-gitlab, though it doesn't seem to have any wiki support:
https://github.com/nlamirault/emacs-gitlab
Cheers, Andy
On 15 February 2015 at 02:02, Erik Huelsmann ehuels@gmail.com wrote:
These plans seem awfully ambitious, but I look forward to them.
Ok. Well, we'll take it one step at a time; it's my expectation that each step by itself is manageable.
Depending on the difficulty, I think I would rather just migrate all of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
Well, if you're up to writing CL code to do the migration, I think it's doable:
** Wiki
- Raw documents can be dowloaded from Trac, so if you have a list of pages
to download, you can simply run those through cURL/wget. (URL format: https://trac.common-lisp.net/cmucl/wiki/GitAndCmucl?format=txt)
- it looks like [http://johnmacfarlane.net/pandoc/ PanDoc] is your way to
go for the conversion of the wiki format
- I'm working on a cl-gitlab library which can be used to connect to the
gitlab API and upload wiki documents
Hmm. come to think of it, you probably want to migrate the wiki into a local repository and push them to the server as described here: https://gitlab.com/gitlab-org/omnibus-gitlab/wikis/git_access
** Tickets
- Tickets can be downloaded in one big XML document (URL:
https://trac.common-lisp.net/cmucl/login?referer=%2Fcmucl%2Freport%2F11%3Fas...)
- After extracting the above URL, cl-gitlab can be used to create the
milestones used in the extract
- cl-gitlab can be used to (re)create the tickets from trac, where I
suggest to move these fields to the body text
- Not all fields available on Trac are available in the GitLab tracker; i'd
propose adding them to the ticket's body; specifically: Reporter and Version
- Fields "Priority" and "Component" can be transformed into Labels (a kind
of "tags")
- Successive comments should be uploaded as comments/notes; this can be
done using cl-gitlab as well.
Note that both the wiki and the tickets can use Wiki markup in Trac as well as GitLab, so every text body (comment, wiki page or ticket) probably needs to be run through pandoc.
If there are more projects that want/need such conversion, we'd probably best try to share any efforts automating the process.
-- Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.
Clo-devel mailing list Clo-devel@common-lisp.net https://mailman.common-lisp.net/cgi-bin/mailman/listinfo/clo-devel
Hi Andrew,
Thanks for the pointer!
I've taken a bit of a different approach, I think, by enumerating explicitly which elements exist in every object and making those elements accessible through slots with lispy names. Admittedly, I haven't written a (nearly) complete API mapping yet, but with my approach, SLIME's auto-complete function should work. There's also a value<->mnemonic mapping for API values which should be marshalled as numbers but have nice constants defined on the receiving (Ruby) side. There's a value<->keyword mapping on the Common Lisp side.
When I feel sufficiently confident in the library -- such as that I'm able to do the things I want to do for this migration -- I'll publish the library in a project on common-lisp.net so everybody can enjoy integration with their favorate Common Lisp environment.
Regards,
Erik.
On Sun, Feb 15, 2015 at 6:54 AM, Andrew Kirkpatrick ubermonk@gmail.com wrote:
Hi Erik,
I just noticed this elisp gitlab library the other day. Perhaps it will be helpful in producing cl-gitlab, though it doesn't seem to have any wiki support:
https://github.com/nlamirault/emacs-gitlab
Cheers, Andy
On 15 February 2015 at 02:02, Erik Huelsmann ehuels@gmail.com wrote:
These plans seem awfully ambitious, but I look forward to them.
Ok. Well, we'll take it one step at a time; it's my expectation that each step by itself is manageable.
Depending on the difficulty, I think I would rather just migrate all of my Trac stuff to gitlab instead of trying to make Trac work with gitlab in some fashion. (Unless gitlab and Trac already have some kind of integration.)
Well, if you're up to writing CL code to do the migration, I think it's doable:
** Wiki
- Raw documents can be dowloaded from Trac, so if you have a list of
pages
to download, you can simply run those through cURL/wget. (URL format: https://trac.common-lisp.net/cmucl/wiki/GitAndCmucl?format=txt)
- it looks like [http://johnmacfarlane.net/pandoc/ PanDoc] is your way
to
go for the conversion of the wiki format
- I'm working on a cl-gitlab library which can be used to connect to the
gitlab API and upload wiki documents
Hmm. come to think of it, you probably want to migrate the wiki into a
local
repository and push them to the server as described here: https://gitlab.com/gitlab-org/omnibus-gitlab/wikis/git_access
** Tickets
- Tickets can be downloaded in one big XML document (URL:
https://trac.common-lisp.net/cmucl/login?referer=%2Fcmucl%2Freport%2F11%3Fas... )
- After extracting the above URL, cl-gitlab can be used to create the
milestones used in the extract
- cl-gitlab can be used to (re)create the tickets from trac, where I
suggest to move these fields to the body text
- Not all fields available on Trac are available in the GitLab tracker;
i'd
propose adding them to the ticket's body; specifically: Reporter and
Version
- Fields "Priority" and "Component" can be transformed into Labels (a
kind
of "tags")
- Successive comments should be uploaded as comments/notes; this can be
done using cl-gitlab as well.
Note that both the wiki and the tickets can use Wiki markup in Trac as
well
as GitLab, so every text body (comment, wiki page or ticket) probably
needs
to be run through pandoc.
If there are more projects that want/need such conversion, we'd probably best try to share any efforts automating the process.
-- Bye,
Erik.
http://efficito.com -- Hosted accounting and ERP. Robust and Flexible. No vendor lock-in.
Clo-devel mailing list Clo-devel@common-lisp.net https://mailman.common-lisp.net/cgi-bin/mailman/listinfo/clo-devel
-
Andrew Kirkpatrick -
Chun Tian (binghe) -
Dave Cooper -
Erik Huelsmann -
Raymond Toy