Azure infrastructure update

[Paolo Bonzini]

Hi all,

a small update on the infrastructure we have set up on Azure and the
expected costs. Remember that we have $10000/year credits from the
Microsoft open source program, therefore the actual cost to the
project is zero unless we exceed the threshold.

Historically, QEMU's infrastructure was hosted on virtual machines
sponsored by Rackspace's open source infrastructure program. When the
program was abruptly terminated, QEMU faced a cost of roughly
$1500/month, mostly due to bandwidth.

As an initial step to cut these costs, downloads were moved to Azure.
However, bandwidth costs remained high and in 2022 we exceeded the
credits from the sponsorship and we had to pay roughly $4000 to
Microsoft, in addition to roughly $2000 for VMs that were still hosted
on Rackspace. While not a definitive solution, this saved the project
an expense of over $10000.

Fortunately, the GNOME project stepped in and offered to host
downloads for QEMU on their CDN. This freed up all the Azure credits
for more interesting uses. In particular, Stefan and I moved the
Rackspace VMs over to Azure, after which the Rackspace bill went down
to zero.

This resulted in two VMs, both running CentOS Stream 9:
- a larger one (E2s instance type) for Patchew and wiki.qemu.org,
costing ~$1900/year between VMs and disks. The websites on this VM are
implemented as podman containers + a simple nginx front-end on ports
80/443.
- a smaller one (D2s instance type) one that proxies qemu.org and
git.qemu.org to gitlab and provides an SSH mirror of the QEMU
downloads, costing $1200/year between VMs and disks. This was a more
traditional monolithic setup.

We also have two virtual machines from OSUOSL (Oregon State University
Open Source Labs); one is unused and can be decommissioned; the other
(also running CentOS Stream 9) is running Patchew background jobs to
import patches and apply them.

Last April, Camilla Conte also added Kubernetes-based private runners
for QEMU CI to our Azure setup. Private runners avoid hitting the
GitLab limits on shared runners and shorten the time it takes to run
individual test jobs. This is because CI, thanks to its burst-y
nature, can use larger VMs than "pet" VMs such as the ones above.
Currently we are using 8 vCPU / 32 GB VMs for the Kubernetes nodes,
and each node is assigned 4 vCPUs.

Starting June 1, all pipelines running in qemu-project/qemu have been
using the private runners. Besides benefiting from the higher number
of vCPUs per job, this, it leaves the GitLab shared runner allowance
to Windows jobs as well as updates to qemu-web. It also made it
possible to estimate the cost of running Linux jobs on Azure at all
times, and to compare the costs with the credits that are made
available through the sponsorship.

Finally, earlier this month I noticed that the OSUOSL mirror for
download.qemu.org was not being updated. Therefore, I rebuilt the
qemu.org and git.qemu.org proxies as containers and moved them to the
same VM running Patchew, wiki.qemu.org and now the KVM Forum website
too. This made it possible to delete the second VM mentioned above. We
will re-evaluate how to provide the source for mirroring
download.qemu.org.

Our consumption of Azure credits was as follows:
* $2005 as of Jun 1, of which $371 used for the now-deleted D2s VM
* $2673 as of Jun 28, of which $457 used for the now-deleted D2s VM

Based on the credits consumed from Jun 1 to Jun 28, which should be
representative of normal resource use, I am estimating the Azure costs
as follows:

$6700 for this year, of which:
- $1650 for the E2s VM
- $450 for the now-deleted D2s VM
- $1600 for the Kubernetes compute nodes
- $2500 for AKS (Azure Kubernetes Service) including system nodes,
load balancing, monitoring and a few more itemized services(*)
- $500 for bandwidth and IP address allocation

$7800 starting next year, of which:
- $1900 for the E2s VM
- $2250 for the Kubernetes compute nodes
- $3100 for AKS-related services
- $550 for bandwidth and IP address allocation

This fits within the allowance of the Azure open source credits
program, while leaving some leeway in case of increased costs or
increased usage of the private runners. As a contingency plan in case
costs surge, we can always disable usage of the private runners and
revert to wider usage of shared runners.

That said, the cost for the compute nodes is not small. In particular,
at the last QEMU Summit we discussed the possibility of adopting a
merge request workflow for maintainer pull requests. These merge
requests would replace the pipelines that are run by committers as
part of merging trees, and therefore should not introduce excessive
costs. However, as things stand, in case of a more generalized
adoption of GitLab MRs(**) the QEMU project will *not* be able to
shoulder the cost of running our (pretty expensive) CI on private
runners for all merge requests.

Thanks,

Paolo

(*) not that we use any of this, but they are added automatically when
you set up AKS

(**) which was NOT considered at QEMU Summit