Zuul Runner

Warning

This is not authoritative documentation. These features are not currently available in Zuul. They may change significantly before final implementation, or may never be fully completed.

While Zuul can be deployed to reproduce a job locally, it is a complex enough system to setup. Zuul jobs being written in Ansible, we shouldn’t have to setup a Zookeeper, Nodepool and Zuul service to run a job locally.

To that end, the Zuul Project should create a command line utility to run a job locally using direct ansible-playbook commands execution. The scope includes two use cases:

  • Running a local build of a job that has already ran, for example to recreate a build that failed in the gate, through using either a zuul-info/inventory.yaml file, or using the –change-url command line argument.

  • Being able to run any job from any Zuul instance, tenant, project or pipeline regardless if it has run or not.

Zuul Job Execution Context

One of the key parts of making the Zuul Runner command line utility is to reproduce as close as possible the zuul service environment.

A Zuul job requires:

  • Test resources

  • Copies of the required projects

  • Ansible configuration

  • Decrypted copies of the secrets

Test Resources

The Zuul Runner shall require the user to provide test resources as an Ansible inventory, similarly to what Nodepool provides to the Zuul Executor. The Runner would enrich the inventory with the zuul vars.

For example, if a job needs two nodes, then the user provides a resource file like this:

all:
  hosts:
    controller:
      ansible_host: ip-node-1
      ansible_user: user-node-1
    worker:
      ansible_host: ip-node-2
      ansible_user: user-node-2

Required Projects

The Zuul Runner shall query an existing Zuul API to get the list of projects required to run a job. This is implemented as part of the topic:freeze_job changes to expose the executor gearman parameters.

The CLI would then perform the executor service task to clone and merge the required project locally.

Ansible Configuration

The CLI would also perform the executor service tasks to setup the execution context.

Playbooks

In some case, running all the job playbooks is not desirable, in this situation the CLI provides a way to select and filter unneeded playbook.

“zuul-runner –list-playbooks” and it would print out:

0: opendev.org/base-jobs/playbooks/pre.yaml
...
10: opendev.org/base-jobs/playbooks/post.yaml

To avoid running the playbook 10, the user would use:

  • “–no-playbook 10”

  • “–no-playbook -1”

  • “–playbook 1..9”

Alternatively, a matcher may be implemented to express:

  • “–skip ‘opendev.org/base-jobs/playbooks/post.yaml’”

Secrets

The Zuul Runner shall require the user to provide copies of any secrets required by the job.

Implementation

The process of exposing gearman parameter and refactoring the executor code to support local/direct usage already started here: https://review.opendev.org/#/q/topic:freeze_job+(status:open+OR+status:merged)

Zuul Runner CLI

Here is the proposed usage for the CLI:

usage: zuul-runner [-h] [-c CONFIG] [--version] [-v] [-e FILE] [-a API]
                   [-t TENANT] [-j JOB] [-P PIPELINE] [-p PROJECT] [-b BRANCH]
                   [-g GIT_DIR] [-D DEPENDS_ON]
                   {prepare-workspace,execute} ...

A helper script for running zuul jobs locally.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG             specify the config file
  --version             show zuul version
  -v, --verbose         verbose output
  -e FILE, --extra-vars FILE
                        global extra vars file
  -a API, --api API     the zuul server api to query against
  -t TENANT, --tenant TENANT
                        the zuul tenant name
  -j JOB, --job JOB     the zuul job name
  -P PIPELINE, --pipeline PIPELINE
                        the zuul pipeline name
  -p PROJECT, --project PROJECT
                        the zuul project name
  -b BRANCH, --branch BRANCH
                        the zuul project's branch name
  -g GIT_DIR, --git-dir GIT_DIR
                        the git merger dir
  -C CHANGE_URL, --change-url CHANGE_URL
                        reproduce job with speculative change content

commands:
  valid commands

  {prepare-workspace,execute}
    prepare-workspace   checks out all of the required playbooks and roles
                        into a given workspace and returns the order of
                        execution
    execute             prepare and execute a zuul jobs

And here is an example execution:

$ pip install --user zuul
$ zuul-runner --api https://zuul.openstack.org --project openstack/nova --job tempest-full-py3 execute
[...]
2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'PLAY RECAP *********************************************************************'
2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'instance-ip                : ok=9    changed=5    unreachable=0    failed=0'
2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'localhost                  : ok=12   changed=9    unreachable=0    failed=0'
2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b''
2019-05-07 06:08:01,218 DEBUG zuul.Runner - Ansible output terminated
2019-05-07 06:08:01,219 DEBUG zuul.Runner - Ansible cpu times: user=0.00, system=0.00, children_user=0.00, children_system=0.00
2019-05-07 06:08:01,219 DEBUG zuul.Runner - Ansible exit code: 0
2019-05-07 06:08:01,219 DEBUG zuul.Runner - Stopped disk job killer
2019-05-07 06:08:01,220 DEBUG zuul.Runner - Ansible complete, result RESULT_NORMAL code 0
2019-05-07 06:08:01,220 DEBUG zuul.ExecutorServer - Sent SIGTERM to SSH Agent, {'SSH_AUTH_SOCK': '/tmp/ssh-SYKgxg36XMBa/agent.18274', 'SSH_AGENT_PID': '18275'}
SUCCESS