Re: [PATCH v3 24/30] gitlab: add acceptance testing to system builds

[Philippe Mathieu-Daudé]

Cc'ing Wainer

On 6/26/20 8:13 PM, Alex Bennée wrote:
> As part of migrating things from Travis to GitLab add the acceptance
> tests. To do this:
> 
>   - rename system1 to system-ubuntu-main
>   - rename system2 to system-fedora-misc
>   - split into build/check/acceptance
>   - remove -j from check stages
>   - use artifacts to save build stage
>   - add post acceptance template and use
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20200622143204.12921-16-alex.bennee@linaro.org>
> 
> ---
> v2
>   - updated with danp's docker changes
>   - use needs instead of dependancies
>   - touch all the build files to prevent rebuild
> ---
>  .gitlab-ci.yml | 66 +++++++++++++++++++++++++++++++++++++++++++++++---
>  .travis.yml    | 23 ------------------
>  2 files changed, 63 insertions(+), 26 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index a7abc55a5c6..5ae8130bd1a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,8 +1,12 @@
> +# Currently we have two build stages after our containers are built:
> +#  - build (for traditional build and test or first stage build)
> +#  - test (for test stages, using build artefacts from a build stage)
>  stages:
>    - containers
>    - containers-layer2
>    - containers-layer3
>    - build
> +  - test
>  
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
> @@ -24,26 +28,82 @@ include:
>          ../configure --enable-werror $CONFIGURE_ARGS ;
>        fi
>      - make -j"$JOBS"
> -    - make -j"$JOBS" $MAKE_CHECK_ARGS
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template: &native_test_job_definition
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    - make $MAKE_CHECK_ARGS
> +
> +.post_acceptance_template: &post_acceptance
> +  after_script:
> +    - python3 -c 'import json; r = 
> json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) 
> for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

Wainer, I guess remember you said you'd send a script to do that
instead, right?

> +    - du -chs $HOME/avocado/data/cache
>  
> -build-system1:
> +build:system-ubuntu-main:
>    <<: *native_build_job_definition
>    variables:
>      IMAGE: ubuntu2004
>      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu 
> lm32-softmmu
>        moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu 
> ppc-softmmu
>        riscv64-softmmu sparc-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004
>      MAKE_CHECK_ARGS: check
>  
> -build-system2:
> +acceptance:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004

(here I'd use ubuntu2004:acceptance)

> +    MAKE_CHECK_ARGS: check-acceptance
> +
> +build:system-fedora-alt:
>    <<: *native_build_job_definition
>    variables:
>      IMAGE: fedora
>      TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu 
> mips-softmmu
>        riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu 
> x86_64-softmmu
>        xtensa-softmmu nios2-softmmu or1k-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-fedora-alt:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora
>      MAKE_CHECK_ARGS: check
>  
> +check:system-fedora-alt:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora

(here I'd use fedora:acceptance)

> +    MAKE_CHECK_ARGS: check-acceptance
> +
>  build-disabled:
>    <<: *native_build_job_definition
>    variables:
> diff --git a/.travis.yml b/.travis.yml
> index 74158f741b1..c24dfbe377f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -289,29 +289,6 @@ jobs:
>        python: 3.6
>  
>  
> -    # Acceptance (Functional) tests
> -    - name: "GCC check-acceptance"
> -      dist: bionic
> -      env:
> -        - CONFIG="--enable-tools 
> --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sh4-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> -        - TEST_CMD="make check-acceptance"
> -        - CACHE_NAME="${TRAVIS_BRANCH}-linux-gcc-acceptance"
> -      after_script:
> -        - python3 -c 'import json; r = 
> json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) 
> for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> -        - du -chs $HOME/avocado/data/cache
> -      addons:
> -        apt:
> -          packages:
> -            - python3-pil
> -            - python3-pip
> -            - python3-numpy
> -            - python3-opencv
> -            - python3-venv
> -            - rpm2cpio
> -            - tesseract-ocr
> -            - tesseract-ocr-eng
> -
> -
>      # Using newer GCC with sanitizers
>      - name: "GCC9 with sanitizers (softmmu)"
>        addons:
> 

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>