Re: [PATCH v3 24/30] gitlab: add acceptance testing to system builds

[Thomas Huth]

On 26/06/2020 20.13, Alex Bennée wrote:
> As part of migrating things from Travis to GitLab add the acceptance
> tests. To do this:
>
>    - rename system1 to system-ubuntu-main
>    - rename system2 to system-fedora-misc
>    - split into build/check/acceptance
>    - remove -j from check stages
>    - use artifacts to save build stage
>    - add post acceptance template and use
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20200622143204.12921-16-alex.bennee@linaro.org>
>
> ---
> v2
>    - updated with danp's docker changes
>    - use needs instead of dependancies
>    - touch all the build files to prevent rebuild
> ---
>   .gitlab-ci.yml | 66 +++++++++++++++++++++++++++++++++++++++++++++++---
>   .travis.yml    | 23 ------------------
>   2 files changed, 63 insertions(+), 26 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index a7abc55a5c6..5ae8130bd1a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,8 +1,12 @@
> +# Currently we have two build stages after our containers are built:
> +#  - build (for traditional build and test or first stage build)
> +#  - test (for test stages, using build artefacts from a build stage)
>   stages:
>     - containers
>     - containers-layer2
>     - containers-layer3
>     - build
> +  - test
>   
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
> @@ -24,26 +28,82 @@ include:
>           ../configure --enable-werror $CONFIGURE_ARGS ;
>         fi
>       - make -j"$JOBS"
> -    - make -j"$JOBS" $MAKE_CHECK_ARGS
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template: &native_test_job_definition
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    - make $MAKE_CHECK_ARGS
> +
> +.post_acceptance_template: &post_acceptance
> +  after_script:
> +    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in 
> r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> +    - du -chs $HOME/avocado/data/cache
>   
> -build-system1:
> +build:system-ubuntu-main:
>     <<: *native_build_job_definition
>     variables:
>       IMAGE: ubuntu2004
>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu 
> lm32-softmmu
>         moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu 
> ppc-softmmu
>         riscv64-softmmu sparc-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004
>       MAKE_CHECK_ARGS: check
>   
> -build-system2:
> +acceptance:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004
> +    MAKE_CHECK_ARGS: check-acceptance
> +
> +build:system-fedora-alt:
>     <<: *native_build_job_definition
>     variables:
>       IMAGE: fedora
>       TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
>         riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
>         xtensa-softmmu nios2-softmmu or1k-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-fedora-alt:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora
>       MAKE_CHECK_ARGS: check
>   
> +check:system-fedora-alt:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora
> +    MAKE_CHECK_ARGS: check-acceptance

Why is Ubuntu "-main" and "Fedora "-alt" ? ... that does not make sense 
to me.

We also might want to rework the list of targets. To speed up the 
testing, I originally omitted some targets like sh4eb-softmmu which did 
not seem very interesting, but now that we add more and more builds in 
parallel, we could maybe split the two system targets into three or even 
four instead, and then add these targets, too. It would also be nice to 
have some tests with "centos8" and a debian container, too.

And "rx-softmmu" is also still missing in the target list.

Ok, it's quite a bit of change that still needs to be done here ... 
maybe that's rather something for a separate patch later.

 Thomas