[GNUnet-developers] Discussion, and Help Wanted: Moving to Gitlab for Git, CI, and Issues

[Devan C. dvn]

Hello my fellow GNUnetians,

As some of you know, I have been pushing for and working on getting us
to CI/CD system based on Gitlab CI. This is pretty much ready to start
using and building out the pipelines for now.

In a previous thread[0] there was a decision to give Gitlab a try, as
more than just our CI system, and to migrate our repos from Gitolite to
our own Gitlab instance.
There was less agreement regarding the idea to move from MantisBT to
Gitlab's issue tracker, but in the end we decided to try importing all
of the >2k Mantis tickets into Gitlab.[1]

This email is a status update on the overall progress of moving to
Gitlab.

Current Gitlab status:

- Gitlab is running and accessible at `https://gitlab.gnunet.org`

- Registration is open. There are no guarantees on uptime, or even
  data retention (though I don't expect data to disappear).

- wldhx has kindly offered two "Gitlab Runners" for running CI jobs.
  These will be added as shared runners, to be used by any projects on
  the instance. This may be changed later to only be shared by projects
  under the GNUnet namespace.

 **TODO**

- Setup email. Used for registration, password resets,
  notifcations, and interaction (eg. issue threads).

- Currently run using containers with docker-compose. Will switch to
using systemd services to with the containers, removing docker-compose.

- Daily remote backups. Perhaps 'firefly.gnunet.org' could be the backup
  site, hmm?

- Change configured hostname (in Gitlab) to 'gitlab.gnunet.org'.

- Setup redirect from 'git.gnunet.org' -> 'gitlab.gnunet.org'
----

Current [MantisBT -> Gitlab Issues] status:

Exporting:
- Mantis can export to CSV and "Excell" XML
  - These do not contain comments (bugnotes). It looks like there might
    be a possibility to enable them via a configuration option[2]. Not
    sure who all has admin access, whom I could coordinate with. Maybe
    easier if I can get admin rights? Grothoff, what do you think?

Importing:
- I have found only a couple of scripts[3,4] for this. They are both out
  of date, for old versions of both softwares. I have tried both to no
  avail. [4] is the most promising; It's not so old.
  I would really appreciate any help working on this.

I suppose this means that we will continue to use Mantis, and disable
issues in Gitlab for now. Any protests or ideas?
----

Migrating from Gitolite:

For those whom are not aware, we currently use gitolite for all of the
lovely repos in our collection. We will need to copy all of the repos to
Gitlab, as well as duplicate permissions, and setup githooks.

 1. Create namespaces/groups on our Gitlab
 
 2. Clone repos. This can be done via the web interface "Import" option
 when creating a new repo, or the new remote can be added, and then
 pushed. The little script found here can help with getting all the
 repos from Gitolite[5]
 
 3. Setup redirects. eg.  https://gnunet.org/git ->
 https://gitlab.gnunet.org
 
 4. Manually replicate permissions. Will need a Gitolite admin's help
 on this.
 
 5. Setup githooks. We have quite a few githooks setup, so we will 
 need to recreate those.

After all of that is done, I think we should be ready to switch over
to Gitlab for at least the git management and CI/CD.
----

That brings us to the final update: The CI System...

- We have a couple of small runners (thanks wldhx).

- We have some very basic '.gitlab-ci.yml'[6] files, defining jobs.
  - I will begin expanding these in the coming weeks.

**TODO**

- As we build out a matrix of pipelines, we will need more resources.
'firefly.gnunet.org' is a logical option. In the past I've seen it
utilized heavily by experiments. As long as we are okay with dedicating
some CPU and RAM to runners, then I will begin setting them up.
 
- Setup Gitlab Container Registry [7] for storing our CI artifacts.

- Expand our '.gitlab-ci.yml' files to include e2e tests, builds for
  multiple architecures, and continuous delivery of packages for various
  package managers.
----

Wow, so that's a lot of text. A lot of people have been asking me about
the status of Gitlab, and if and how they can help with CI. I hope this
gives people a thorough update, and answers. I really believe Gitlab can
be a useful software suite, despite its shortcomings. My hopes are that
it will help increase the feedback loop between development and testing,
as well as make it easier and more welcoming for new contributors. 


Be well, and Happy Hacking!
- Devan


[0] https://lists.gnu.org/archive/html/gnunet-developers/2019-01/msg00071.html
[1] https://lists.gnu.org/archive/html/gnunet-developers/2019-01/msg00082.html
[2] 
http://www.mantisbt.org/docs/master/en-US/Admin_Guide/html-desktop/#admin.config.display
[3] https://gitlab.kitware.com/utils/mantis-to-gitlab 
[4] https://github.com/timwiel/mantis2gitlab 
[5] 
https://tutorials.technology/tutorials/89-show-all-repositories-of-a-git-server.html
[6] https://docs.gitlab.com/ce/ci/quick_start/
[7] https://docs.gitlab.com/ce/user/project/container_registry.html

signature.asc
Description: PGP signature