Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi]

On Thu, Jun 01, 2023 at 06:23:21AM +0200, Lukáš Doktor wrote:
> Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> > On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> >>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>
> >>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest 
> >>>>>> poll handlers - as a performance improvement when using multiple 
> >>>>>> concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 
> >>>>>> (on a slow rotational disk).
> >>>>>>
> >>>>>>     
> >>>>>> https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>>>
> >>>>>> Based on the commit message I guess it's expected so take this just as 
> >>>>>> a record of an improvement.
> >>>>>
> >>>>> The commit was not intended to change performance and I'm not sure why
> >>>>> it happens!
> >>>>>
> >>>>
> >>>> It had and today the x86_64 pipeline finished which shows similar 
> >>>> improvement just not in read but rather in write instead and only for 4k 
> >>>> blocks (~40%). For 1024k blocks I can see it scoring a bit better 
> >>>> (~1.5%). Reads are too jittery to really tell anything on that machine. 
> >>>> Anyway I have not done any thorough testing, just a bisection with the 
> >>>> most significant setting.
> >>>>
> >>>> From around the same time I can see a NVMe regression in 4k writes, but 
> >>>> first bisection job showed nothing. I'll increase the range and try 
> >>>> again as each job since that day shows similar drop.
> >>>
> >>
> >> Hello Stefan, folks,
> >>
> >> the regression proved to be there and stably reproducible. With NVMe 4k 
> >> writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
> >>
> >>     
> >> https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
> >>
> >> The rest of the cases doesn't show any change at all. I can provide more 
> >> data if someone is interested.
> > 
> > Which commit caused the regression?
> 
> Hello Stefan,
> 
> the same one that caused the improvement on rotational disks: 
> 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

Thanks. This commit definitely needs more investigation. I didn't expect
it to have a noticeable effect.

I will take a look in July.

Stefan

signature.asc
Description: PGP signature