Zuul Admin Client

Zuul includes a simple command line client that may be used to affect Zuul’s behavior while running.

Note

For operations related to normal workflow like enqueue, dequeue, autohold and promote, the zuul-client CLI should be used instead.

Configuration

The client uses the same zuul.conf file as the server, and will look for it in the same locations if not specified on the command line.

Usage

The general options that apply to all subcommands are:

usage: zuul-admin [-h] [-c CONFIG] [--version] [-v] [--auth-token AUTH_TOKEN]
                  [--zuul-url ZUUL_URL] [--insecure]
                  {autohold,autohold-delete,autohold-info,autohold-list,enqueue,enqueue-ref,dequeue,promote,show,tenant-conf-check,create-auth-token,import-keys,export-keys,copy-keys,delete-keys,delete-state,delete-pipeline-state,prune-database}
                  ...

Zuul CLI client.

options:
  -h, --help            show this help message and exit
  -c CONFIG             specify the config file
  --version             show zuul version
  -v                    verbose output
  --auth-token AUTH_TOKEN
                        Authentication Token, needed if using theREST API
  --zuul-url ZUUL_URL   Zuul API URL, needed if using the REST API without a
                        configuration file
  --insecure            Do not verify SSL connection to Zuul, when using the
                        REST API (Defaults to False)

commands:
  valid commands

  {autohold,autohold-delete,autohold-info,autohold-list,enqueue,enqueue-ref,dequeue,promote,show,tenant-conf-check,create-auth-token,import-keys,export-keys,copy-keys,delete-keys,delete-state,delete-pipeline-state,prune-database}
                        additional help
    autohold            [DEPRECATED - use zuul-client] hold nodes for failed
                        job
    autohold-delete     [DEPRECATED - use zuul-client] delete autohold request
    autohold-info       [DEPRECATED - use zuul-client] retrieve autohold
                        request detailed info
    autohold-list       [DEPRECATED - use zuul-client] list autohold requests
    enqueue             [DEPRECATED - use zuul-client] enqueue a change
    enqueue-ref         [DEPRECATED - use zuul-client] enqueue a ref
    dequeue             [DEPRECATED - use zuul-client] dequeue a buildset by
                        its change or ref
    promote             [DEPRECATED - use zuul-client] promote one or more
                        changes
    show                [DEPRECATED - use zuul-client] show current statuses
    tenant-conf-check   validate the tenant configuration
    create-auth-token   create an Authentication Token for the web API
    import-keys         import project keys to ZooKeeper
    export-keys         export project keys from ZooKeeper
    copy-keys           copy keys from one project to another
    delete-keys         delete project keys
    delete-state        delete ephemeral ZooKeeper state
    delete-pipeline-state
                        delete single pipeline ZooKeeper state
    prune-database      prune old database entries

The following subcommands are supported:

tenant-conf-check

usage: zuul-admin tenant-conf-check [-h]

options:
  -h, --help  show this help message and exit

Example:

zuul-admin tenant-conf-check

This command validates the tenant configuration schema. It exits ‘-1’ in case of errors detected.

create-auth-token

Note

This command is only available if an authenticator is configured in zuul.conf. Furthermore the authenticator’s configuration must include a signing secret.

usage: zuul-admin create-auth-token [-h] --auth-config AUTH_CONFIG --tenant
                                    TENANT --user USER
                                    [--expires-in EXPIRES_IN]

Create an Authentication Token for the administration web API

Create a bearer token that can be used to access Zuul's
administration web API. This is typically used to delegate
privileged actions such as enqueueing and autoholding to
third parties, scoped to a single tenant.
At least one authenticator must be configured with a secret
that can be used to sign the token.

options:
  -h, --help            show this help message and exit
  --auth-config AUTH_CONFIG
                        The authenticator to use. Must match an authenticator
                        defined in zuul's configuration file.
  --tenant TENANT       tenant name
  --user USER           The user's name. Used for traceability in logs.
  --expires-in EXPIRES_IN
                        Token validity duration in seconds (default: 600)

Example:

zuul-admin create-auth-token --auth-config zuul-operator --user alice --tenant tenantA --expires-in 1800

The return value is the value of the Authorization header the user must set when querying a protected endpoint on Zuul’s REST API.

Example:

bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwOi8vbWFuYWdlc2Yuc2ZyZG90ZXN0aW5zdGFuY2Uub3JnIiwienV1bC50ZW5hbnRzIjp7ImxvY2FsIjoiKiJ9LCJleHAiOjE1Mzc0MTcxOTguMzc3NTQ0fQ.DLbKx1J84wV4Vm7sv3zw9Bw9-WuIka7WkPQxGDAHz7s

export-keys

usage: zuul-admin export-keys [-h] path

Export project secret keys from ZooKeeper

This command exports project secret keys from ZooKeeper
and writes them to a file which is suitable for backing
up and later use with the import-keys command.

The key contents are still protected by the keystore
password and can not be used or decrypted without it.

positional arguments:
  path        key export file path

options:
  -h, --help  show this help message and exit

Example:

zuul-admin export-keys /var/backup/zuul-keys.json

import-keys

usage: zuul-admin import-keys [-h] [--force] path

Import previously exported project secret keys to ZooKeeper

Given a file with previously exported project keys, this
command will import them into ZooKeeper.  Existing keys
will not be overwritten; to overwrite keys, add the
--force flag.

positional arguments:
  path        key export file path

options:
  -h, --help  show this help message and exit
  --force     overwrite existing keys

Example:

zuul-admin import-keys /var/backup/zuul-keys.json

copy-keys

usage: zuul-admin copy-keys [-h]
                            src_connection src_project dest_connection
                            dest_project

Copy secret keys from one project to another

When projects are renamed, this command may be used to
copy the secret keys from the current name to the new name
in order to avoid service interruption.

positional arguments:
  src_connection   original connection name
  src_project      original project name
  dest_connection  new connection name
  dest_project     new project name

options:
  -h, --help       show this help message and exit

Example:

zuul-admin copy-keys gerrit old_project gerrit new_project

delete-keys

usage: zuul-admin delete-keys [-h] connection project

Delete the ssh and secrets keys for a project

positional arguments:
  connection  connection name
  project     project name

options:
  -h, --help  show this help message and exit

Example:

zuul-admin delete-keys gerrit old_project

delete-state

usage: zuul-admin delete-state [-h] [--keep-config-cache]

Delete all ephemeral state stored in ZooKeeper

Zuul stores a considerable amount of ephemeral state
information in ZooKeeper.  Generally it should be able to
detect and correct any errors, but if the state becomes
corrupted and it is unable to recover, this command may be
used to delete all ephemeral data from ZooKeeper and start
anew.

Do not run this command while any Zuul component is
running (perform a complete shutdown first).

This command will only remove ephemeral Zuul data from
ZooKeeper; it will not remove private keys or Nodepool
data.

options:
  -h, --help           show this help message and exit
  --keep-config-cache  keep config cache

Example:

zuul-admin delete-state

delete-pipeline-state

usage: zuul-admin delete-pipeline-state [-h] tenant pipeline

Delete a single pipeline state stored in ZooKeeper

In the unlikely event that a bug in Zuul or ZooKeeper data
corruption occurs in such a way that it only affects a
single pipeline, this command might be useful in
attempting to recover.

The circumstances under which this command will be able to
effect a recovery are very rare and even so it may not be
sufficient.  In general, if an error occurs it is better
to shut Zuul down and run "zuul delete-state".

This command will lock the specified tenant and
then completely delete the pipeline state.

positional arguments:
  tenant      tenant name
  pipeline    pipeline name

options:
  -h, --help  show this help message and exit

Example:

zuul-admin delete-pipeline-state tenant pipeline

prune-database

usage: zuul-admin prune-database [-h] [--before BEFORE]
                                 [--older-than OLDER_THAN]
                                 [--batch-size BATCH_SIZE]

Prune old database entries

This command will delete database entries older than the
specified cutoff (which can be specified as either an
absolute or relative time).

options:
  -h, --help            show this help message and exit
  --before BEFORE       absolute timestamp (e.g., "2022-01-31 12:00:00")
  --older-than OLDER_THAN
                        relative time (e.g., "24h" or "180d")
  --batch-size BATCH_SIZE
                        transaction batch size

Example:

zuul-admin prune-database --older-than 180d

Deprecated commands

The following commands are deprecated in the zuul-admin CLI, and thus may not be entirely supported in Zuul’s current version. They will be removed in a future release of Zuul. They can still be performed via the zuul-client CLI. Please refer to zuul-client’s documentation for more details.

In order to run these commands, the webclient section is required in the configuration file.

It is also possible to run the client without a configuration file, by using the --zuul-url option to specify the base URL of the Zuul web server.

Autohold

usage: zuul-admin autohold [-h] --tenant TENANT --project PROJECT --job JOB
                           [--change CHANGE] [--ref REF] --reason REASON
                           [--count COUNT]
                           [--node-hold-expiration NODE_HOLD_EXPIRATION]

options:
  -h, --help            show this help message and exit
  --tenant TENANT       tenant name
  --project PROJECT     project name
  --job JOB             job name
  --change CHANGE       specific change to hold nodes for
  --ref REF             git ref to hold nodes for
  --reason REASON       reason for the hold
  --count COUNT         number of job runs (default: 1)
  --node-hold-expiration NODE_HOLD_EXPIRATION
                        how long in seconds should the node set be in HOLD
                        status (default: scheduler's default_hold_expiration
                        value)

Example:

zuul-admin autohold --tenant openstack --project example_project --job example_job --reason "reason text" --count 1

Autohold Delete

usage: zuul-admin autohold-delete [-h] REQUEST_ID

positional arguments:
  REQUEST_ID  the hold request ID

options:
  -h, --help  show this help message and exit

Example:

zuul-admin autohold-delete --id 0000000123

Autohold Info

usage: zuul-admin autohold-info [-h] REQUEST_ID

positional arguments:
  REQUEST_ID  the hold request ID

options:
  -h, --help  show this help message and exit

Example:

zuul-admin autohold-info --id 0000000123

Autohold List

usage: zuul-admin autohold-list [-h] --tenant TENANT

options:
  -h, --help       show this help message and exit
  --tenant TENANT  tenant name

Example:

zuul-admin autohold-list --tenant openstack

Dequeue

usage: zuul-admin dequeue [-h] --tenant TENANT --pipeline PIPELINE --project
                          PROJECT [--change CHANGE] [--ref REF]

options:
  -h, --help           show this help message and exit
  --tenant TENANT      tenant name
  --pipeline PIPELINE  pipeline name
  --project PROJECT    project name
  --change CHANGE      change id
  --ref REF            ref name

Examples:

zuul-admin dequeue --tenant openstack --pipeline check --project example_project --change 5,1
zuul-admin dequeue --tenant openstack --pipeline periodic --project example_project --ref refs/heads/master

Enqueue

usage: zuul-admin enqueue [-h] --tenant TENANT [--trigger TRIGGER] --pipeline
                          PIPELINE --project PROJECT --change CHANGE

options:
  -h, --help           show this help message and exit
  --tenant TENANT      tenant name
  --trigger TRIGGER    trigger name (deprecated and ignored. Kept only for
                       backward compatibility)
  --pipeline PIPELINE  pipeline name
  --project PROJECT    project name
  --change CHANGE      change id

Example:

zuul-admin enqueue --tenant openstack --trigger gerrit --pipeline check --project example_project --change 12345,1

Note that the format of change id is <number>,<patchset>.

Enqueue-ref

usage: zuul-admin enqueue-ref [-h] --tenant TENANT [--trigger TRIGGER]
                              --pipeline PIPELINE --project PROJECT --ref REF
                              [--oldrev OLDREV] [--newrev NEWREV]

Submit a trigger event

Directly enqueue a trigger event.  This is usually used
to manually "replay" a trigger received from an external
source such as gerrit.

options:
  -h, --help           show this help message and exit
  --tenant TENANT      tenant name
  --trigger TRIGGER    trigger name
  --pipeline PIPELINE  pipeline name
  --project PROJECT    project name
  --ref REF            ref name
  --oldrev OLDREV      old revision
  --newrev NEWREV      new revision

This command is provided to manually simulate a trigger from an external source. It can be useful for testing or replaying a trigger that is difficult or impossible to recreate at the source. The arguments to enqueue-ref will vary depending on the source and type of trigger. Some familiarity with the arguments emitted by gerrit update hooks such as patchset-created and ref-updated is recommended. Some examples of common operations are provided below.

Manual enqueue examples

It is common to have a release pipeline that listens for new tags coming from gerrit and performs a range of code packaging jobs. If there is an unexpected issue in the release jobs, the same tag can not be recreated in gerrit and the user must either tag a new release or request a manual re-triggering of the jobs. To re-trigger the jobs, pass the failed tag as the ref argument and set newrev to the change associated with the tag in the project repository (i.e. what you see from git show X.Y.Z):

zuul-admin enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/example_project --ref refs/tags/X.Y.Z --newrev abc123..

The command can also be used asynchronosly trigger a job in a periodic pipeline that would usually be run at a specific time by the timer driver. For example, the following command would trigger the periodic jobs against the current master branch top-of-tree for a project:

zuul-admin enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master

Another common pipeline is a post queue listening for gerrit merge results. Triggering here is slightly more complicated as you wish to recreate the full ref-updated event from gerrit. For a new commit on master, the gerrit ref-updated trigger expresses “reset refs/heads/master for the project from oldrev to newrev” (newrev being the committed change). Thus to replay the event, you could git log in the project and take the current HEAD and the prior change, then enqueue the event:

NEW_REF=$(git rev-parse HEAD)
OLD_REF=$(git rev-parse HEAD~1)

zuul-admin enqueue-ref --tenant openstack --trigger gerrit --pipeline post --project openstack/example_project --ref refs/heads/master --newrev $NEW_REF --oldrev $OLD_REF

Note that zero values for oldrev and newrev can indicate branch creation and deletion; the source code is the best reference for these more advanced operations.

Promote

usage: zuul-admin promote [-h] --tenant TENANT --pipeline PIPELINE --changes
                          CHANGES [CHANGES ...]

options:
  -h, --help            show this help message and exit
  --tenant TENANT       tenant name
  --pipeline PIPELINE   pipeline name
  --changes CHANGES [CHANGES ...]
                        change ids

Example:

zuul-admin promote --tenant openstack --pipeline gate --changes 12345,1 13336,3

Note that the format of changes id is <number>,<patchset>.

The promote action is used to reorder the changes in a pipeline, by putting the provided changes at the top of the queue.

The most common use case for the promote action is the need to merge an urgent fix when the gate pipeline has several patches queued ahead. This is especially needed if there is concern that one or more changes ahead in the queue may fail, thus increasing the time to land for the fix; or concern that the fix may not pass validation if applied on top of the current patch queue in the gate.

Any items in a dependent pipeline which have had items ahead of them changed will have their jobs canceled and restarted based on the new ordering.

If items in independent pipelines are promoted, no jobs will be restarted, but their change queues within the pipeline will be re-ordered so that they will be processed first and their node request priorities will increase.