Re: [PATCH 7/9] tests/performance: Add nightly tests

[Alex Bennée]

Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:

> A nightly performance testing system to monitor any change in QEMU
> performance across seventeen different targets.
>
> The system includes eight different benchmarks to provide a variety
> of testing workloads.
>
> dijkstra_double:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are double values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> dijkstra_int32:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are int32 values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> matmult_double:
> Standard matrix multiplication of an n*n matrix of randomly generated
> double numbers from 0 to 100. The value of n is passed as an argument
> with the -n flag. The default value is 200.
>
> matmult_int32:
> Standard matrix multiplication of an n*n matrix of randomly generated
> integer numbers from 0 to 100. The value of n is passed as an
> argument with the -n flag. The default value is 200.
>
> qsort_double:
> Quick sort of an array of n randomly generated double numbers from 0
> to 1000. The value of n is passed as an argument with the -n flag.
> The default value is 300000.
>
> qsort_int32:
> Quick sort of an array of n randomly generated integer numbers from 0
> to 50000000. The value of n is passed as an argument with the -n
> flag.The default value is 300000.
>
> qsort_string:
> Quick sort of an array of 10000 randomly generated strings of size 8
> (including null terminating character). The sort process is repeated
> n number of times. The value of n is passed as an argument with the
> -n flag. The default value is 20.
>
> search_string:
> Search for the occurrence of a small string in a much larger random
> string (“needle in a hay”). The search process is repeated n number
> of times and each time, a different large random string (“hay”) is
> generated. The value of n can be specified using the -n flag. The
> default value is 20.
>
> Syntax:
>     nightly_tests_core.py [-h] [-r REF]
>     Optional arguments:
>         -h, --help            Show this help message and exit
>         -r REF, --reference REF
>                         Reference QEMU version - Default is v5.1.0
>     Example of usage:
>         nightly_tests_core.py -r v5.1.0 2>log.txt
>
> The following report includes detailed setup and execution details
> of the system:
> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
>  tests/performance/nightly-tests/README.md     | 243 +++++
>  .../source/dijkstra_double/dijkstra_double.c  | 194 ++++
>  .../source/dijkstra_int32/dijkstra_int32.c    | 192 ++++
>  .../source/matmult_double/matmult_double.c    | 123 +++
>  .../source/matmult_int32/matmult_int32.c      | 121 +++
>  .../source/qsort_double/qsort_double.c        | 104 ++
>  .../source/qsort_int32/qsort_int32.c          | 103 ++
>  .../source/qsort_string/qsort_string.c        | 122 +++
>  .../source/search_string/search_string.c      | 110 +++
>  .../scripts/nightly_tests_core.py             | 920 ++++++++++++++++++
>  .../scripts/run_nightly_tests.py              | 135 +++
>  .../nightly-tests/scripts/send_email.py       |  56 ++
>  12 files changed, 2423 insertions(+)
>  create mode 100644 tests/performance/nightly-tests/README.md
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
>  create mode 100644 
> tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
>  create mode 100644
> tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c

Perhaps we could compress these paths down to:

  tests/tcg/benchmarks/foo.c
  tests/tcg/benchmarks/bar.c

and then we can also ensure they are built using the existing TCG tests
cross compile framework.

>  create mode 100755 
> tests/performance/nightly-tests/scripts/nightly_tests_core.py
>  create mode 100755 
> tests/performance/nightly-tests/scripts/run_nightly_tests.py
>  create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
>
> diff --git a/tests/performance/nightly-tests/README.md 
> b/tests/performance/nightly-tests/README.md
> new file mode 100644
> index 0000000000..6db3b351b3
> --- /dev/null
> +++ b/tests/performance/nightly-tests/README.md
> @@ -0,0 +1,243 @@
> +### QEMU Nightly Tests
> +
> +**Required settings:**
> +
> +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> +
> +For more details on how the system works, please check the [eighth
> report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/)
> of the "TCG Continuos Benchmarking" series.

As external URLs are potentially unstable I think we want to distil the
details into a rst do in docs/devel/

<snip>

-- 
Alex Bennée