Re: Proposal for a regular upstream performance testing

[Lukáš Doktor]

Dne 30. 11. 20 v 14:25 Stefan Hajnoczi napsal(a):
> On Thu, Nov 26, 2020 at 09:10:14AM +0100, Lukáš Doktor wrote:
> > The problem with those is that we can not simply use travis/gitlab/... machines 
> > for running those tests, because we are measuring in-guest actual performance. 
> > We can't just stop the time when the machine decides to schedule another 
> > container/vm. I briefly checked the public bare-metal offerings like rackspace 
> > but these are most probably not sufficient either because (unless I'm wrong) 
> > they only give you a machine but it is not guaranteed that it will be the same 
> > machine the next time. If we are to compare the results we don't need just the 
> > same model, we really need the very same machine. Any change to the machine 
> > might lead to a significant difference (disk replacement, even firmware 
> > update...).
>
> Do you have a suggested bare metal setup?
>
> I think it's more complicated than having a single bare metal host. It
> could involve a network boot server, a network traffic generator machine
> for external network iperf testing, etc.

Yes. At this point I only test host->guest connection, but run-perf is prepared 
to test multi-host connection as well (tested with uperf, but dedicated traffic 
generator could be added as well). Another machine is promised but not yet on the 
way.

> What is the minimal environment needed for bare metal hosts?

Not sure what you mean by that. For provisioning I have a beaker plugin, other 
plugins can be added if needed. Even without beaker one can also provide an 
installed machine and skip the provisioning step. Runperf would then only apply 
the profiles (including fetching the VM images from public sources) and run the 
tests on them. Note that for certain profiles might need to reboot the machine 
and in such case the tested machine can not be the one running run-perf, other 
profiles can use the current machine but it's still not a very good idea as the 
additional overhead might spoil the results.

Note that for a very simple issue which do not require a special setup I am 
usually just running a custom VM on my laptop and use a Localhost profile on 
that VM, which basically results in testing that custom-setup VM's performance. 
It's dirty but very fast for the first-level check.

> Stefan

Regards,
Lukáš