Zuul Runner =========== .. warning:: This is not authoritative documentation. These features are not currently available in Zuul. They may change significantly before final implementation, or may never be fully completed. While Zuul can be deployed to reproduce a job locally, it is a complex enough system to setup. Zuul jobs being written in Ansible, we shouldn't have to setup a Zookeeper, Nodepool and Zuul service to run a job locally. To that end, the Zuul Project should create a command line utility to run a job locally using direct ansible-playbook commands execution. The scope includes two use cases: * Running a local build of a job that has already ran, for example to recreate a build that failed in the gate, through using either a `zuul-info/inventory.yaml` file, or using the `--change-url` command line argument. * Being able to run any job from any Zuul instance, tenant, project or pipeline regardless if it has run or not. Zuul Job Execution Context -------------------------- One of the key parts of making the Zuul Runner command line utility is to reproduce as close as possible the zuul service environment. A Zuul job requires: - Test resources - Copies of the required projects - Ansible configuration - Decrypted copies of the secrets Test Resources ~~~~~~~~~~~~~~ The Zuul Runner shall require the user to provide test resources as an Ansible inventory, similarly to what Nodepool provides to the Zuul Executor. The Runner would enrich the inventory with the zuul vars. For example, if a job needs two nodes, then the user provides a resource file like this: .. code-block:: yaml all: hosts: controller: ansible_host: ip-node-1 ansible_user: user-node-1 worker: ansible_host: ip-node-2 ansible_user: user-node-2 Required Projects ~~~~~~~~~~~~~~~~~ The Zuul Runner shall query an existing Zuul API to get the list of projects required to run a job. This is implemented as part of the `topic:freeze_job` changes to expose the executor gearman parameters. The CLI would then perform the executor service task to clone and merge the required project locally. Ansible Configuration ~~~~~~~~~~~~~~~~~~~~~ The CLI would also perform the executor service tasks to setup the execution context. Playbooks ~~~~~~~~~ In some case, running all the job playbooks is not desirable, in this situation the CLI provides a way to select and filter unneeded playbook. "zuul-runner --list-playbooks" and it would print out: .. code-block:: console 0: opendev.org/base-jobs/playbooks/pre.yaml ... 10: opendev.org/base-jobs/playbooks/post.yaml To avoid running the playbook 10, the user would use: * "--no-playbook 10" * "--no-playbook -1" * "--playbook 1..9" Alternatively, a matcher may be implemented to express: * "--skip 'opendev.org/base-jobs/playbooks/post.yaml'" Secrets ~~~~~~~ The Zuul Runner shall require the user to provide copies of any secrets required by the job. Implementation -------------- The process of exposing gearman parameter and refactoring the executor code to support local/direct usage already started here: https://review.opendev.org/#/q/topic:freeze_job+(status:open+OR+status:merged) Zuul Runner CLI --------------- Here is the proposed usage for the CLI: .. code-block:: console usage: zuul-runner [-h] [-c CONFIG] [--version] [-v] [-e FILE] [-a API] [-t TENANT] [-j JOB] [-P PIPELINE] [-p PROJECT] [-b BRANCH] [-g GIT_DIR] [-D DEPENDS_ON] {prepare-workspace,execute} ... A helper script for running zuul jobs locally. optional arguments: -h, --help show this help message and exit -c CONFIG specify the config file --version show zuul version -v, --verbose verbose output -e FILE, --extra-vars FILE global extra vars file -a API, --api API the zuul server api to query against -t TENANT, --tenant TENANT the zuul tenant name -j JOB, --job JOB the zuul job name -P PIPELINE, --pipeline PIPELINE the zuul pipeline name -p PROJECT, --project PROJECT the zuul project name -b BRANCH, --branch BRANCH the zuul project's branch name -g GIT_DIR, --git-dir GIT_DIR the git merger dir -C CHANGE_URL, --change-url CHANGE_URL reproduce job with speculative change content commands: valid commands {prepare-workspace,execute} prepare-workspace checks out all of the required playbooks and roles into a given workspace and returns the order of execution execute prepare and execute a zuul jobs And here is an example execution: .. code-block:: console $ pip install --user zuul $ zuul-runner --api https://zuul.openstack.org --project openstack/nova --job tempest-full-py3 execute [...] 2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'PLAY RECAP *********************************************************************' 2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'instance-ip : ok=9 changed=5 unreachable=0 failed=0' 2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'localhost : ok=12 changed=9 unreachable=0 failed=0' 2019-05-07 06:08:01,040 DEBUG zuul.Runner - Ansible output: b'' 2019-05-07 06:08:01,218 DEBUG zuul.Runner - Ansible output terminated 2019-05-07 06:08:01,219 DEBUG zuul.Runner - Ansible cpu times: user=0.00, system=0.00, children_user=0.00, children_system=0.00 2019-05-07 06:08:01,219 DEBUG zuul.Runner - Ansible exit code: 0 2019-05-07 06:08:01,219 DEBUG zuul.Runner - Stopped disk job killer 2019-05-07 06:08:01,220 DEBUG zuul.Runner - Ansible complete, result RESULT_NORMAL code 0 2019-05-07 06:08:01,220 DEBUG zuul.ExecutorServer - Sent SIGTERM to SSH Agent, {'SSH_AUTH_SOCK': '/tmp/ssh-SYKgxg36XMBa/agent.18274', 'SSH_AGENT_PID': '18275'} SUCCESS