CI as criterion for new FSF forge

[Zack Weinberg]

Hi, I'm writing to ask whether continuous integration capabilities
have been considered as a factor in the evaluation of new "forge"
services for the FSF.  I read through
<https://libreplanet.org/wiki/FSF_2020_forge_evaluation> and did not
see anything about it.

I've been a contributor to GNU projects for many years, notably both
GCC and GNU libc, and recently I led the effort to make the first
release of Autoconf since 2012.  Autoconf is hosted on Savannah at
present, GCC and glibc on sourceware.org; none of these services have
built-in continuous integration.  Several corporate contributors to
GCC and glibc run their own CI and post test results to the mailing
lists, but there's no dashboard and no automatic testing of submitted
patches.  Autoconf presently has no CI at all.

The lack of CI was a major hindrance during the run-up to the new
Autoconf release--I found dozens of regressions from the previous
release, some of which had gone undetected for *years*, because of
inadequate testing.  The incomplete Autoconf test suite is partially
to blame here, but even so, it was easier for these bugs to slip in
because there was no requirement for the test suite to pass before
merging a patch, and no automated way for me to check for regressions
on the many platforms and architectures Autoconf supports.

CI is also a major selling point for the proprietary forge services
that the FSF's forge would be competing with.  For example, I am also
a co-maintainer of libxcrypt <https://github.com/besser82/libxcrypt>
which has replaced part of GNU libc on most Linux distributions.  As
you can see from the URL, it's currently hosted on Github, and it
relies on TravisCI and coverage.io for continuous integration and
related functions.  Moving to a Free forge would be a much easier
argument for me to make if equivalent CI services were available from
the new host.

Of the three software packages currently on the short list, I know
that sourcehut does support CI to some extent.  I intend to experiment
with its CI for Autoconf in the near future, using a mirror repository
I created on sourcehut's self-hosted instance,
<https://git.sr.ht/~zackw/mirror-autoconf>.  I don't know about either
of the others.

I realize CI requires significantly more resources to support than
most of the other aspects of a forge service, but it is also
significantly more valuable to maintainers than some of the other
practical concerns that are listed in the evaluation (e.g. the exact
protocol for patch submission).  It might be possible to share
resources with the GCC Compile Farm
<https://gcc.gnu.org/wiki/CompileFarm> or with other such existing
clusters.

Thanks,
zw