Re: Azure infrastructure update

[Paolo Bonzini]

On Wed, Jun 28, 2023 at 1:28 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > $6700 for this year, of which:
> > - $1650 for the E2s VM
> > - $450 for the now-deleted D2s VM
> > - $1600 for the Kubernetes compute nodes
> > - $2500 for AKS (Azure Kubernetes Service) including system nodes,
> > load balancing, monitoring and a few more itemized services(*)
> > - $500 for bandwidth and IP address allocation
> >
> > $7800 starting next year, of which:
> > - $1900 for the E2s VM
>
> Same size VM as last year, but more ? Is this is simply you
> anticipating possible price increases from Azure ?

No it's just ~11 vs. 12 months, because we didn't set it up from the first day.

> > - $2250 for the Kubernetes compute nodes
>
> IIUC, the $1600 from year this will cover about 7.5 months worth
> of usage (Jun -> Dec), which would imply more around $2500 for a
> full 12 months, possibly more if we add in peaks for soft freeze.
> IOW could conceivably be closer to $3k mark without much difficulty,
> especially if also start doing more pipelines for stable branches
> on a regular basis, now we have CI working properly for stable.

It also covers several weeks in May. In any case I've saved the broken
down data and will redo the estimate after 4 months (Jun 1->Sep 30,
covering both a soft freeze and a hard freeze).

> > - $3100 for AKS-related services
>
> Same question about anticipated prices ?

Same answer about 9-10 months vs. 12. :)

> > That said, the cost for the compute nodes is not small. In particular,
> > at the last QEMU Summit we discussed the possibility of adopting a
> > merge request workflow for maintainer pull requests. These merge
> > requests would replace the pipelines that are run by committers as
> > part of merging trees, and therefore should not introduce excessive
> > costs.
>
> Depending on how we setup the CI workflow, it might increase our
> usage, potentially double it quite easily.
>
> Right now, whomever is doing CI for merging pull requests is fully
> serializing CI pipelines, so what's tested 100% matches what is
> merged to master.
>
> With a merge request workflow it can be slightly different based
> on a couple of variables.
>
> When pushing a merge request to their fork, prior to opening the
> merge request, CI credits are burnt in their fork for every push,
> based on whatever is the HEAD of their branch. This might be behind
> current upstream 'master' by some amount.
>
> Typically when using merge requests though, you would change the
> gitlab CI workflow rules to trigger CI pipelines from merge request
> actions, instead of branch push actions.

Yes, that was my idea as well.

> If we do this, then when opening a merge request, an initial pipeline
> would be triggered.
>
> If-and-only-if the maintainer has "Developer" on gitlab.com/qemu-project,
> then that merge request initial pipeline will burn upstream CI credits.
>
> If they are not a "Developer", it will burn their own fork credits. If
> they don't have any credits left, then someone with "Developer" role
> will have to spawn a pipeline on their behalf, which will run in
> upstream context and burn upstream credits. The latter is tedious,
> so I think expectation is that anyone who submits pull requests would
> be expected to have 'Developer' role on qemu-project. We want that
> anyway really so we can tag maintainers in issues on gitlab too.

Agreed. Is there no option to have the "Developer" use his own credits?

> IOW, assume that any maintainer opening a merge req will be burning
> upstream CI credits on their merge request pipelines. [...]
> Merge requests are not serialized though. [...]
> To address this would require using GitLab's  "merge trains" feature.
>
> When merge trains are enabled, when someone hits the button to apply
> a merge request to master, an *additional* CI pipeline is started
> based on the exact content that will be applied to master. Crucially,
> as the name hints, the merge train pipelines  are serialized. IOW,
> if you request to apply 4 merge requests in quick succession a queue
> of pipelines will be created and run one after the other. If any
> pipeline fails, that MR is kicked out of the queue, and the
> following pipelines carry on.
>
> IOW, the merge trains feature replicates what we achieve with the
> serialized 'staging' branch.
>
> What you can see here though, is that every merge request will have
> at least 2 pipelines - one when the MR is opened, and one when it
> is applied to master - both consuming upstream CI credits.
>
> IOW, we potentially double our CI usage in this model if we don't
> make any changes to how CI pipelines are triggered. [...]
> If we can afford the CI credits, I'd keep things simple and
> just accept the increased CI burn, but with your figures above
> I fear we'd be too close to the limit to be relaxed about it.

Hmm, now that I think about it I'm not sure the merge request CI would
use private runners. Would it use the CI variables that are set in
settings/ci_cd? If not, the pipeline would not tag the jobs for
private runners, and therefore the merge request would use shared
runners (thus burning project minutes, but that's a different
problem).

> If we can't afford the double pipelines, then we would have
> to write our GitLab CI yml rules to exclude the initial
> pipeline, or just do a very minimalist "smoke test", and
> focus bulk of CI usage on teh merge train pipeline.
>
> This is all solvable in one way or another. We just need to
> figure out the right tradeoffs we want.
>
> >          However, as things stand, in case of a more generalized
> > adoption of GitLab MRs(**) the QEMU project will *not* be able to
> > shoulder the cost of running our (pretty expensive) CI on private
> > runners for all merge requests.
>
> With more generalized adoption of MR workflow for all contributions
> bear in mind that many of the contributors will NOT have the
> 'Developer' role on gitlab.com/qemu-project. Thus their merge
> requests pipelines would run in fork context and consume their own
> CI credits, unless a "Developer" had to manually trigger a pipeline
> on their behalf.

I would expect most MRs to come from Developers but yes, that's not a given.

Paolo