Re: [RFC PATCH 2/2] gitlab-ci: Introduce allow_skipping_job_template

[Philippe Mathieu-Daudé]

On 12/14/20 2:32 PM, Thomas Huth wrote:
> On 11/12/2020 18.17, Philippe Mathieu-Daudé wrote:
>> It might be overkill (and frustrating) to rerun all the jobs when
>> iterating over a single code unit. As some users are interested
>> in only rerunning a specific set of jobs, allow them to use a
>> "manual" profile, when jobs are created in the pipeline but not
>> started. It is up to the user to manually start the jobs of
>> interest.
>>
>> To switch a repository to manual, set QEMU_DEFAULT_WORKFLOW_MANUAL
>> in the project UI, following [1].
>>
>> Once a repository is using the manual workflow, the user can still
>> automatically start the jobs by explicitly set the
>> QEMU_IGNORE_DEFAULT_WORKFLOW which disable the manual mode.
>>
>> Example while pushing a branch/tag from command line (see [2]):
>>
>>   $ git push -o ci.variable="QEMU_IGNORE_DEFAULT_WORKFLOW=1" \
>>       myrepo mybranch
>>
>> [1] 
>> https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui
>> [2] 
>> https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .gitlab-ci.d/crossbuilds.yml |  4 ++++
>>  .gitlab-ci.yml               | 18 ++++++++++++++++++
>>  2 files changed, 22 insertions(+)
>>
>> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
>> index 864cad7cec5..56b053cbb52 100644
>> --- a/.gitlab-ci.d/crossbuilds.yml
>> +++ b/.gitlab-ci.d/crossbuilds.yml
>> @@ -1,4 +1,5 @@
>>  .cross_system_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 80m
>> @@ -18,6 +19,7 @@
>>  # KVM), and set extra options (such disabling other accelerators) via the
>>  # $ACCEL_CONFIGURE_OPTS variable.
>>  .cross_accel_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 30m
>> @@ -30,6 +32,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_user_build_job:
>> +  extends: .allow_skipping_job
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    script:
>> @@ -40,6 +43,7 @@
>>      - make -j$(expr $(nproc) + 1) all check-build
>>  
>>  .cross_sanity_check_job:
>> +  extends: .allow_skipping_job
> 
> So "extends" is possible ....
> 
>>    stage: build
>>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>    timeout: 3m
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 98bff03b47b..26b85f08f35 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -13,7 +13,21 @@ include:
>>    - local: '/.gitlab-ci.d/containers.yml'
>>    - local: '/.gitlab-ci.d/crossbuilds.yml'
>>  
>> +.allow_skipping_job_template: &allow_skipping_job_definition
>> +  rules:
>> +    # Skip if the user set the QEMU_DEFAULT_WORKFLOW_MANUAL variable for its
>> +    # project. Can be overridden by setting the QEMU_IGNORE_DEFAULT_WORKFLOW
>> +    # variable.
>> +    - if: $CI_PIPELINE_SOURCE == "push" && $QEMU_DEFAULT_WORKFLOW_MANUAL && 
>> $QEMU_IGNORE_DEFAULT_WORKFLOW == null
>> +      when: manual
>> +      allow_failure: true
>> +    - when: always
>> +
>> +.allow_skipping_job:
>> +  <<: *allow_skipping_job_definition
> 
> ... before the anchestor has been defined? That's nice!

'extends' is indeed more powerful and less hackish.

> I'm basically fine with this patch, but can you please also add some
> sentence how to use this to docs/devel/testing.rst ?

Thanks, I later discovered this isn't complete, because if a children
job is in automatic state and a parent (dependency) is in manual, there
is some "dead lock": the pipeline waits until manual job is started.

To be complete, I have to ensure if a job is in manual state, all its
dependencies also are.

Then we are back to 'extends' VS 'anchors', as this doesn't work with
anchors, I have to convert all jobs to 'extends'. Good news is I already
did it, I simply need to rebase this series:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html

Thanks for reviewing,

Phil.