Release Notes

5.2.1

New Features

  • The queue.dependencies-by-topic setting is added for Zuul users who wish to emulate the Gerrit submitWholeTopic behavior for some projects without enabling it site-wide in Gerrit.

  • The promote administrative action now functions with all pipeline managers. Previously it would only have an impact on dependent pipelines, but it will now re-order change queues as well as changes within any type of pipeline.

Upgrade Notes

  • The promote administrative action will no longer restart jobs for changes which have not been re-ordered within their change queue.

Bug Fixes

  • Fixed an issue in the Gerrit driver introduced in 5.2.0 which could cause infinite loops while querying changes.

5.2.0

New Features

  • Changes in dependent queues are now automatically dequeued if they no longer meet the mergability requirements set by the upstream code review system.

  • Zuul now matches Gerrit’s behavior when change.submitWholeTopic is set in Gerrit.

    When this setting is enabled in Gerrit and a change is submitted (merged), all changes with the same topic are submitted simultaneously. Zuul will now query for changes which are set to be submitted together by Gerrit when enqueing them and treat them as if they are a set of circular dependencies.

    If the projects are not part of pipeline queues which are configured to allow circular dependencies, then Zuul will report failure on enqueue. Be sure that the submitWholeTopic setting in Gerrit and the allow-circular-dependencies setting in Zuul match.

    This functionality requires an HTTP connection to Gerrit. If only an SSH connection is available, then changes submitted together will be ignored.

  • The ability to configure Microsoft Login as an OpenID Connect authentication provider has been added.

  • Zuul now reports total resource usage statistics.

    The following statistic is emitted:

    zuul.nodepool.resources.total.tenant.{tenant}.{resource}

    Gauge with the currently used resources by tenant in total, i.e., all nodes belonging to a tenant, regardles of their state.

  • It is now possible for jobs to skip retries caused by failures in ‘pre-run’ by returning retry: false:

    - zuul_return:
    data:
      zuul:
        retry: false

Upgrade Notes

  • Zuul now reports total and in-use resource usage statics separately.

    To distinquish those, the following statistics have been renamed:

    • zuul.nodepool.resources.tenant.{tenant}.{resource} -> zuul.nodepool.resources.in_use.tenant.{tenant}.{resource}

    • zuul.nodepool.resources.project.{project}.{resource}: -> zuul.nodepool.resources.in_use.project.{tenant}.{resource}

Security Issues

  • A vulnerability which allowed the execution of untrusted code on the executor was fixed.

    Zuul restricts the Ansible modules and plugins which can be used in the untrusted security context (i.e., untrusted projects). It also prohibits running programs on the Zuul executor in the untrusted security context.

    Ansible 2.8 and later versions support referencing builtin modules using the ansible.builtin.<name> alias. Playbooks which use this mechanism can bypass Zuul’s security restrictions and run arbitrary local code or otherwise restricted modules.

    Zuul’s use of bubblewrap means that any commands executed via this vulnerability would still be contained within the restricted environment, meaning that they can not access files outside of the build directory or continue running longer than the playbook. But they may have been able to access files within the build directory but outside of the work/ directory, as well as potentially exploit any kernel or hypervisor privilege escalation vulnerabilities.

    The Zuul team now considers the restricted Ansible environment to be ineffective as a security mechanism and is developing plans to remove the restrictions and rely entirely on bubblewrap in the future. These changes will occur in a future release of Zuul (likely 6.0.0) and will be preceded by more details about the change.

5.1.0

New Features

  • Added support for adding and removing merge request labels in the GitLab driver, as well as triggering pipelines on label removal (label addition was already supported).

  • A new buildset result MERGE_FAILURE is stored in the build database in the case where a buildset is unable to be merged by the upstream code review system.

  • A new command zuul-scheduler tenant-reconfigure has been added. It allows an operator to perform a reconfiguration of a single tenant. This may be helpful in clearing up issues after connection problems with the code hosting system.

  • New monitoring metrics specific to zuul-web are available.

Upgrade Notes

  • The buildset result MERGER_FAILURE has been renamed to MERGE_CONFLICT, and the pipeline reporter configuration merge-failure has been renamed to merge-conflict.

    These are more descriptive of the most common errors actually reported, and so are expected to be less confusing to users. This is also in service of a future change to support a new buildset result MERGE_FAILURE which will indicate that the change was unable to be merged in the upstream repository.

    When upgrading, it is recommended to stop all schedulers briefly (i.e, when the first scheduler of the new version starts, there should be no schedulers running the old version). The new scheduler will perform a database migration when it starts and update all existing MERGER_FAILURE buildset results to MERGE_CONFLICT. If old schedulers are running, they may continue to add MERGER_FAILURE entries which will need to be manually updated in order to be visible in the web UI or rest API.

Deprecation Notes

  • The merge-failure and merge-failure-message pipeline configuration options have been renamed to merge-conflict and merge-conflict-message respectively. The old settings are retained for backwards compatibility, but will be removed in a later version. Please update your usage of them as soon as possible.

Bug Fixes

  • The labels pipeline requirement in the gitlab driver erroneously treated the labels as a boolean or but should have treated them as a boolean and (i.e., all listed labels are required). The behavior has been updated to match the documentation and other drivers.

5.0.0

Upgrade Notes

  • Zuul no longer uses gearman. It is no longer required to run a Gearman server, and the Zuul scheduler no longer does so. All gearman-related settings in zuul.conf files are now ignored and may be removed.

    Commands such as zuul enqueue which were previously executed via Gearman may still be used by creating an authentication token and adding a web section to zuul.conf.

  • Support for configuring the database as a connection in zuul.conf was deprecated in Zuul version 4.0 and has now been removed. Refer to Database how to configure the database now.

4.12.0

Upgrade Notes

  • Due to a change in the ZooKeeper data format, the following upgrade procedure is required:

    • Stop all Zuul components

    • Run zuul delete-state

    • Start all Zuul components

  • Zuul-web now requires access to the Zookeeper keystore. Ensure that the keystore.password option is included in zuul.conf on at least the scheduler, executor, and web servers (it may be included on all components if desired).

  • Zuul-web requires information for all defined connections. Previously, zuul-web may have started without all of the connections fully defined in its config file, or with some requirements (such as keys used for connecting to remote services) present. They are now required in order for zuul-web to start.

4.11.0

New Features

  • The new variable zuul.playbook_context as well as new variables under zuul.projects have been added to help debug or audit the playbooks and roles used by Ansible when running jobs.

    These variables describe the repo configuration used for the playbooks and roles of each Ansible execution. These repos may have a different state than the workspace repos.

  • The optional prometheus service now also includes endpoints for readiness and liveness checks. See Liveness Probes for details.

  • Allow an authorized user to dequeue changes from the Web UI’s status page.

  • Add authentication in the web UI. Zuul’s web UI can be configured to authenticate users against an Identity Provider supporting the OpenID Connect protocol.

Upgrade Notes

  • Users upgrading from previous versions of zuul will need to run zuul delete-state to delete Zuul’s ephemeral state from Zookeeper. The ephemeral state produced by older versions of zuul is not compatible with this release. This command should be run while all Zuul services are stopped to prevent writes to the Zookeeper database. More information on this command can be found in the documentation: https://zuul-ci.org/docs/zuul/reference/client.html#delete-state

  • The prometheus endpoint would previously serve metrics at any URI. It now only returns metrics on the / and /metrics URI. The latter is recommended.

4.10.4

Bug Fixes

  • A fix was made to mitigate an issue where a ZooKeeper disconnect could cause too many threads to be spawned resulting in errors.

4.10.3

Bug Fixes

  • A bug with the change cache cleanup routine which could have caused items to be stuck in pipelines without running jobs has been corrected.

  • Gerrit driver: Zuul error responses could overflow the default message length with large configurations leading to Gerrit rejecting the comment (and hence no notification from Zuul on the change). Such messages are now truncated to a safe length.

4.10.2

Bug Fixes

  • A bug which prevented change cache cleanup (and therefore caused ZooKeeper usage to grow without limits) was fixed.

4.10.1

New Features

  • Add support for squashing MR with gitlab.

  • On the description page of a buildset in the zuul web dashboard, a user can display a timeline of the builds making up the buildset.

Bug Fixes

  • Fix a bug in the cleanup routine for the new Zookeeper based change cache. Previously this caused the change cache to leak entries. Note the fixed routine will clean up the leaked entries automatically once Zuul is restarted on this new version.

  • Fix detection of the failed merge in gitlab driver.

4.10.0

Upgrade Notes

  • Nodepool 4.3.0 is now required. Zuul stores additional information in node records in preparation for supporting multiple schedulers.

  • The scheduler time database has been removed. This was stored in the scheduler state directory, typically /var/lib/zuul/times. The entire state directory on the scheduler is no longer used and may now be removed.

    Zuul now derives its estimated build duration times from the SQL database.

4.9.0

New Features

  • The zuul delete-state command may be used to delete all of the ephemeral state stored by Zuul in ZooKeeper. Normally Zuul is able to detect and correct errors on its own, but in case it is unable to, this may prove a useful utility for manual recovery.

  • The new commands zuul copy-keys and zuul delete-keys may be useful when renaming projects in order to move the keys from the old to the new project names without service interruption.

  • Add new statsd gauge metrics of current open node requests exported as zuul.nodepool.tenant.<tenant>.current_requests. This metric tracks the currently open node requests per tenant. It drills down the overall zuul.nodepool.current_requests metric.

Upgrade Notes

  • Zuul no longer reads or writes project private key files from the scheduler’s filesystem. In order to load existing keys into ZooKeeper, run version 4.6.0 of the scheduler at least once, if you haven’t already.

    A new command zuul export-keys has been added to export the encrypted keys from ZooKeeper onto the filesystem for backup. Likewise, zuul import-keys will load a previously-exported backup into ZooKeeper. It is recommended that you use these commands in system backup scripts.

4.8.1

Upgrade Notes

  • An error was found in a change related to Zuul’s internal configuration cache which could cause Zuul to use cached in-repo configuration files which no longer exist. If a zuul.yaml (or zuul.d/* or any related variant) file was deleted or renamed, Zuul would honor that change immediately, but would attempt to load both the old and new contents from its cache upon the next restart.

    This error was introduced in version 4.8.0.

    If upgrading from 4.8.0, run zuul-scheduler full-reconfigure in order to correctly update the cache.

Bug Fixes

  • Restored the job.success-message and job.failure-message functionality which was inadvertently removed.

4.8.0

New Features

  • Zuul now can route live log streams via finger gateways to make it possible to distribute executors over multiple datacenters without the possibility to directly contact every executor from within zuul-web. This is typically the case in a Kubernetes based deployment.

  • The finger gateway and executor log streaming system now supports TLS connections.

    Normally zuul-web makes a direct connection to an executor in order to stream logs. With this new option, that connection can be encrypted if it crosses an untrusted network.

    The ability to route log streaming connections through finger gateway servers was recently added; these will also use TLS if required.

    The finger gateway server can also be used by end-users; in that case it may need a TLS certificate to use if it is required to connect to an encrypted executor or finger gateway to stream logs. An option to disable using TLS when acting as a server is provided for this case, since there are no TLS-enable finger clients.

    See fingergw.tls_cert and related options to enable encrypted connections for all three components.

4.7.0

New Features

  • The executor now honors the executor.sigterm_method configuration file setting to determine whether SIGTERM should be equivalent to the graceful command (the default) or the stop command.

Upgrade Notes

  • The default behavior when an executor receives SIGTERM has been changed from immediately stopping jobs to gracefully waiting for them to finish. Set the executor.sigterm_method configuration file setting to stop to preserve the old behavior.

  • The zuul.scheduler.eventqueues.result gauge was removed

  • Zuul now requires at least Nodepool version of 4.2.0 due to an internal API change.

Deprecation Notes

  • The following attributes are now ignored:

    • The report-build-page tenant configuration setting.

    • The success-url job attribute.

    • The failure-url job attribute.

    The URL of the build page is now always reported for every build. This is now also true for in-progress builds, which provides for a more consistent user experience.

    Since the build page is always reported as the URL, the success and failure URL job attributes are no longer useful, so this functionality has also been removed.

    Zuul’s configuration syntax checker will continue to allow these settings for now (they are simply ignored) but this will be removed in version 5.0 and using them will be considered an error.

    To achieve a similar result, consider returning the URL as an artifact from the job via zuul_return. This will cause a link to appear in the “Artifacts” section of the build page.

4.6.0

New Features

  • If sensitive data must be returned from a job in order to be provided to dependent jobs, the secret_data attribute of zuul_return attribute may now be used instead of the normal data attribute. The data will be provided via the same mechanism as job secrets, where the data are not written to disk in the work directory. Care must still be taken to avoid displaying or storing sensitive data within the job. For example:

    tasks:
  - zuul_return:
      secret_data:
        password: foobar
      data:
        this_is: not secret

Security Issues

  • The following connection-related variables are no longer allowed to be set in job definitions, as they may be used to circumvent security measures:

    • ansible_connection

    • ansible_host

    • ansible_python_interpreter

    • ansible_shell_executable

    • ansible_user

    They may still be set using the corresponding settings in Nodepool.

  • The ability to use Ansible Jinja templates in Zuul job variables is partially restricted.

    It was found that the ability to use Jinja templates in Zuul job variables could be used to expose the contents of secrets. To remedy this, the values of Zuul job variables are frozen at the start of the job and these values are used for trusted playbooks and playbooks with secrets. The freezing action is taken without access to any secrets so they can not be exposed.

    This means that Zuul job variables which reference non-secret values that are known at the start of the job (including any zuul.* variable) will continue to work as expected. Job variables which reference secrets will not work (they will be undefined). In untrusted playbooks, job variables are still dynamically evaluated and can make use of values that are set after the start of the job.

    Additionally, job.extra-vars are no longer passed to Ansible using the “-e” command line options. They could be used to expose secrets because they take precedence over some internal playbook variables in some circumstances. Zuul’s extra-vars are now passed as normal inventory variables, however, they retain precedence over all other Zuul job variables (vars, host-vars, and group-vars) except secrets.

    Secrets are also now passed as inventory variables as well for the same reason. They have the highest precedence of all Zuul job variables. Their values are tagged with !unsafe so that Ansible will not evaluate them as Jinja expressions.

    If you are certain that a value contained within a secret is safe to evaluate as a Jinja expression, you may work around this limitation using the following construct in a playbook:

    - set_fact:
    unsafe_var_eval: "{{ hostvars['localhost'].secret.var }}"

    This will force an explicit evaluation of the variable. This is generally safe to do in a situation where a playbook is accessing a single secret by name, with no other secrets in scope. Do not use this capability with more than one secret that is not under the control of the project where the playbook is defined.

    Similarly, versions of all the original job variables tagged with !unsafe are available under the unsafe_vars variable hierarchy. For example, the job variable myvar would be available under unsafe_vars.myvar. It is not recommended to evaluate unsafe_vars expressions except in the most controlled of circumstances. They are almost impossible to render safely.

4.5.1

Bug Fixes

  • The zuul tenant-conf-check command no longer needs a ZooKeeper connection to validate the tenants configuration file is valid.

4.5.0

Prelude

This is a checkpoint release. This release contains some bugfixes and other user-invisible changes, but its main purpose is to serve as a checkpoint as we move more of Zuul’s functionality into ZooKeeper. This version has seen production use in OpenDev and we believe it to be stable and ready for wider use. Please do upgrade to it and report any issues to zuul-discuss. You may downgrade to the previous release if you do encounter any issues. Likewise, operators running the latest development builds may downgrade to this release in the case of issues.

4.4.0

New Features

  • A new prometheus_port option for the services can be used to start the prometheus python client and exposes metrics.

  • Jobs may now request multiple semaphores and they will not start until all semaphores are acquired.

    Use the new job.semaphores (plural) attribute to specify them.

    Note that the new attribute is additive when considering inheritance and job variants. That is to say that a job definition with a semaphores attribute will extend the list of semaphores supplied by its parent rather than overriding it (which is the behavior for the deprecated attribute).

Deprecation Notes

  • The job attribute job.semaphore (note the singular rather than plural form) job attribute is now deprecated. Use the plural form job.semaphores instead. As with most list items in Zuul configuration, it also accepts a single item without the wrapping list, so to convert existing jobs, simply change the spelling of the attribute, no change to the value is required.

    The singular form will be removed in Zuul 5.0.

4.3.0

New Features

  • The MQTT reporter now includes artifact information along with build results.

  • Jobs may now specify an alternate scheme to use when preparing repositories in the workspace. The default remains the same golang-style, but an alternate scheme called flat is now available. See job.workspace-scheme for more details.

  • Project secrets keys and SSH keys are now stored in Zookeeper. All private data will be encrypted at rest, which requires a new mandatory setting keystore.password in zuul.conf.

    For backup purposes the secrets keys and SSH keys will still exist on the local filesystem of the scheduler as before.

Upgrade Notes

  • The internal git repo caches maintained by the mergers and executors now use a new naming scheme in order to avoid collisions. When existing executors and mergers are restarted, they will remove their git repo caches and re-clone repos using the new scheme. Jobs may be slow to start until the caches are warmed again.

  • As project secrets keys and SSH keys are stored encrypted in Zookeeper the new keystore.password option in zuul.conf is required. Please add it to your configuration for both the scheduler and executor.

Deprecation Notes

  • Zuul now correctly handles the rerequested action on check run trigger definitions (check_run) literally. This has been requested which doesn’t match the GitHub api. The value requested is now deprecated and will be removed in a later release.

Bug Fixes

  • Fixed a bug where multiple connections of the same type would not filter trigger events coming from the wrong connection.

4.2.0

Prelude

This is a checkpoint release. This release contains some bugfixes and other user-invisible changes, but its main purpose is to serve as a checkpoint as we move more of Zuul’s functionality into ZooKeeper. This version has seen production use in OpenDev and we believe it to be stable and ready for wider use. Please do upgrade to it and report any issues to zuul-discuss. You may downgrade to the previous release if you do encounter any issues. Likewise, operators running the latest development builds may downgrade to this release in the case of issues.

4.1.0

New Features

  • Zuul now honors the shell-type configuration from nodepool and uses it to set ansible_shell_type, a setting that is required when connecting to Windows workers over ssh.

    For Linux workers, there is a long standing ansible issue with using non-default ansible_shell_type and become, so this feature is primarily targeting Windows workers.

  • Projects can now configure change queues to queue per branch. See queue for more information.

  • Circular dependencies are now optionally supported (but disabled by default). To permit dependency cycles to be enqueued, see the queue.allow-circular-dependencies option.

  • A job now can leave warning messages on the change using zuul_return. See Return Values for examples.

  • Zuul now supports the --validate-tenants switch which can be used to validate the complete configuration of the listed tenants.

Upgrade Notes

  • Two sets of statsd metrics are now reported for executors: zoned and unzoned. The existing statsd keys are now deprecated; new statsd keys are available for both zoned and unzoned executors. See zuul.executors for details.

Deprecation Notes

  • Shared queues should be configured per project now instead per pipeline. Specifying project.<pipeline>.queue is deprecated and will be removed in a future release.

Bug Fixes

  • If zoned executors were used with prior releases of Zuul, the reported executor statistics would only represent a single, unspecified zone. This has now been corrected.

4.0.0

Prelude

This is the first 4.x release of Zuul. It requires some deployment changes by operators which have been carefully planned in order to facilitate work on Zuul version 5, which will be the first version of Zuul where every component is fault tolerant and able to scale. If you read the release notes for the last 3.x release, you may have already made all of the required changes. If not, please do so before upgrading to version 4. Every required change in version 4 is optionally supported in 3.19, so it is safe to make these changes and then upgrade. Please read all of the notes below, especially in the “Upgrading” section for details. The primary additional requirements are:

  • TLS ZooKeeper connections

  • Network connectivity from all components to ZooKeeper

  • An SQL database

With these changes in place, it is anticipated that further upgrades to Zuul made in support of the scale-out-scheduler work will be done with minimal disruption in the course of normal releases between version 4 and 5.

New Features

  • The action “promote” is now available via the tenant-scoped REST API using the zuul cli.

  • Builds in the SQL reporter have a “held” attribute set to True if the build triggered a autohold request. Builds can be filtered by held status.

  • Subdirectories of zuul.d or .zuul.d can be ignored when reading configuration by placing a .zuul.ignore file inside them.

  • The find module is now allowed to run on the executor.

  • Changes can now be dequeued via the Github checks API. If a github reporter is configured to use the checks API, all running checks will provide a custom “Abort” action.

  • Zuul now respects GitHub review requirements when enqueuing into gate pipelines. This works for github.com and GitHub Enterprise starting with version 2.21.0.

  • Jobs may specify the new intermediate flag to note they may only be inherited by abstract jobs. This can be useful if building a job hierarchy where wish to limit where a base job is instantiated.

  • The builds/ and buildset/ API endpoints now include information about retried builds. They are called non-final as those are all builds that were retried at least once and thus weren’t visible to the user so far.

    The builds/ API can filter those retried builds and you might exclude them from API result by setting final=false in the API request.

  • An option to use the URL of the Zuul build page when reporting has been added. This feature requires that all the pipelines in the tenant have a SQL reporter configured, and at least one of tenant.web-root or web.root must be defined.

    See tenant.report-build-page.

  • REST API: authorizations: add a tenant-scoped endpoint at ‘/api/tenant/{tenant}/authorizations’. Calling the endpoint will return a list of admin tenants limited to the scoped tenant, if the user has admin privileges on it.

  • Host key checking is now disabled for a host in the generated Ansible inventory if host-key-checking is set to False in the corresponding nodepool config.

Upgrade Notes

  • Support for Ansible 2.7 has been removed.

  • If using a SQL reporter, the zuul_builds table will be updated with a new ‘held’ column. The zuul-scheduler and zuul-web services need to be restarted together for the change to take effect.

  • Previously the SqlReporter would record the job name in the database in place of the url if the url was empty. This has now been updated to store a null in the database for that case. Zuul will automatically run a database migration to correct old values.

  • The -d switch doesn’t enforce foreground anymore. It only enables debug logging. To start zuul in foreground please add the -f switch instead.

  • Support for running Zuul under Python 3.5 has been dropped.

  • As further integration with the web interface is planned, the web.root setting in zuul.conf is marked required and future releases may error if it is missing. Please add it to your configuration now. See tenant.web-root for additional information about whitelabel tenants.

  • The components zuul-scheduler and zuul-web now require database configuration. There is now only one supported database connection. If multiple sql connections have been configured only the first one will be used.

  • Signal based triggering of full-reconfiguration in the scheduler (deprecated since 3.3.0) has been removed. Use zuul-scheduler full-reconfigure instead.

  • The zookeeper section in zuul.conf is required for all components, and all components must now be able to connect to ZooKeeper. Additionally, TLS is now required for all ZooKeeper connections. See Encrypted Connections for more details.

Deprecation Notes

  • Defining database connections using sql connection is now deprecated. Refer to Database how to configure the database now.

  • REST API: authorizations: the /api/user/authorizations endpoint is deprecated in favor of the tenant-scoped endpoint. It will be removed next release.

Bug Fixes

  • Although the documentation states that the MQTT reporter reports the build’s log_url, this was only true as long as tenant.report-build-page was disabled. As soon as the setting was enabled, the MQTT reporter reported the url to the build’s result page in Zuul. As MQTT is meant to be consumed by machines, this broke use cases like log post processing.

    This was fixed so that the <mqtt schema>.buildset.builds.log_url now always contains the log url, while an additional field <mqtt schema>.buildset.builds.web_url contains the url to the build’s result page if tenant.report-build-page is enabled.

3.19.1

Security Issues

  • A long-standing security vulnerability regarding executing code on the executor was corrected.

    The Zuul executor was designed to prohibit executing command or shell tasks on the executor itself (i.e., localhost in Ansible) from untrusted playbooks. Tobias Henkel discovered that this check has been broken for some time, likely since June of 2018.

    The use of bubblewrap means that any commands executed via this vulnerability would still be contained within the restricted environment, meaning that they can not access files outside of the build directory or continue running longer than the job. However, by executing arbitrary commands, users may have been able to connect to unprotected internal network services.

    Because this bug is so long-standing, it is possible, even likely, that users may have accidentally come to rely on it. We discovered two jobs in the zuul-jobs library which did so: dco-license and promote-docker-image. The promote-docker-image job has been altered so it no longer needs to run a command on the executor. The dco-license job has been altered to run on a node. If you would prefer to run it on the exceutor, you can create a new job in a config-project that uses the validate-dco-license role.

    You may want to look for other jobs in your system which may be affected by this. To aid in that, we have created a script which will examine the job-output.json files created by previous builds and output any tasks it finds which are now (once again) prohibited. This script is available here:

Bug Fixes

  • The dependency on kazoo has been upgraded to 2.8.0 which has an important fix for using Zookeeper over TLS.

  • The Github access token URL has been updated in order to remove a deprecation warning.

3.19.0

Prelude

This is expected to be the last 3.x release of Zuul (barring any unanticipated bugfix releases).

The next release of Zuul, 4.0.0, will require the following deployment changes:

  • TLS ZooKeeper connections

  • Network connectivity from all components to ZooKeeper

  • A SQL database

These features are supported now, so it is important that you take this time to switch your ZooKeeper connection to use TLS, and configure a SQL connection. Even though only some Zuul components connect to ZooKeeper today, you should ensure that the [zookeeper] section is present in zuul.conf for all components, and that they have network connectivity to the ZooKeeper servers. Doing so will make the upgrade to 4.0.0 seamless.

These changes are in support of ongoing work to enable multiple scheduler processes for scaling and availability. After the 4.0.0 release, it is anticipated that further upgrades to Zuul made in support of this work will be done with minimal disruption in the course of normal releases.

New Features

  • Pipelines now provide a pipeline.dequeue reporter action so that reporters may run whenever an item is dequeued. The dequeue reporters will only apply if the item wasn’t a success or failure.

  • Support for generating dynamic Badges has been added.

  • The annotation levels for the file comments reported via Github checks API are now configurable in zuul_return. Each file comment entry can provide an optional level parameter [info|warning|error] that will be picked up by the Github reporter.

    For more details on how to provide file comments from Zuul, see the documentation of the Return Values.

  • Zuul now matches tag items against their containing branches. This makes it simpler to have release jobs in-tree. See job.branches for details.

  • The zuul-executor now also pauses all merger related tasks when it’s paused. Further the zuul-merger can also be paused by running zuul-merger pause.

  • The serial pipeline manager has been added. It is designed to handle serialized deployment pipelines where supercedent is unsuitable in the case that not all jobs run on every merge.

  • Add new timezone selector in web interface

  • Zuul now supports a TLS secured connection to ZooKeeper.

Upgrade Notes

  • Support for Ansible 2.6 has been removed.

  • Ansible 2.7 is now deprecated since it only receives security updates and will be end of life soon.

  • Zuul now defaults to Ansible 2.9 if no version is specified.

  • With GitHub the Zuul app now requires permissions to read/write check runs.

  • The default behavior of the tools/encrypt_secret.py helper script is now to strip incoming input of leading and trailing whitespace. A new --no-strip option has been added to support people with secrets that contain valid leading or trailing whitespace.

  • Please configure your Zuul to use TLS secured connection. Running Zuul with an unsecured connection to ZooKeeper is deprecated and will be unsupported in a future release. See zookeeper for details.

Security Issues

  • Zuul no longer adds environment variables starting with the ZUUL_ prefix to ansibles environment which could result in secrets being exposed.

Bug Fixes

  • In some cases (such as when an older version of kubectl is present on the executor, or the kubectl on the executor is from OpenShift), streaming logs from Kubernetes or OpenShift pods did not work. The log streaming system has been corrected to work with a wider range of kubectl versions and error logging has been improved.

3.18.0

New Features

  • The Github driver can now report file comments via Github checks API. If a github reporter is configured to use the checks API, file comments provided via zuul_return will automatically be reported to the pull request in Github.

    For more details on how to provide file comments from Zuul, see the documentation of the Return Values.

Upgrade Notes

  • Kubectl and socat must now be installed on Zuul executors if using Kubernetes or OpenShift pod resources from Nodepool. Additionally, Nodepool version 3.12.0 or later is required, and the “start-zuul-console” role from zuul-jobs should be run in the pre-playbook of your base job.

Deprecation Notes

  • Running the zuul processes with the -d switch to enforce running in foreground is now deprecated. Switch to use -f instead for this purpose. The -d switch will change to just enable debug logging in the future.

Security Issues

  • The add_host host-vars blacklist is no longer effective for trusted playbook.

Bug Fixes

  • Previously, no output from shell or command tasks on pods was placed in the job output; that has been corrected and streaming output is now available.

3.17.0

New Features

  • The Github driver now has a basic support for the Github checks API. To enable reporting build results via the checks API one can configure the the new pipeline.<reporter>.<github source>.check attribute on the Github reporter. It’s also possible to trigger on a requested or completed check_run.

    To be able to use the checks API, zuul must be authenticated as Github app. For more information about the necessary requirements, please see the GitHub driver documentation.

Security Issues

  • The add_host module attributes that can be used to bypass localhost command execution are now also blacklisted using extra-vars to prevent abuse through untrusted host_vars.

3.16.1

Bug Fixes

  • When using GitHub app authentication the fallback to anonymous access was broken for repositories not having installed the zuul app.

3.16.0

New Features

Upgrade Notes

  • The default for <gerrit connection>.auth_type has changed from digest to basic. Digest authentication has not been supported in Gerrit since version 2.15.

    If your Zuul connects to an older version of Gerrit via HTTP authentication, you may now need to explicitly set this value.

Deprecation Notes

  • Authentication: the JWT driver “RS256withJWKS” is deprecated in favor of the “OpenIDConnect” driver. The “OpenIDConnect” driver simplifies configuration for administrators and is better aligned with OIDC configuration discovery conventions.

3.15.0

Bug Fixes

  • Files matcher has been fixed to act like irrelevant files matcher when no files are present in the tested change, i.e. it now matches such empty changes instead of rejecting them.

3.14.0

New Features

  • Zuul now supports triggering a smart reconfiguration by using the command zuul-scheduler smart-reconfigure.

Upgrade Notes

  • Zuul no longer supports Ansible 2.5 for running jobs.

3.13.0

New Features

  • Zuul now supports Ansible 2.9 for running jobs.

Upgrade Notes

  • The default version of Ansible used by Zuul jobs has been changed to 2.8. See job.ansible-version for more information.

  • As a result of switching to Ansible 2.8, it is possible for the ansible_python_interpreter version of your nodeset to change to ‘auto’. This could result in your jobs now running under a different version of python than prior to this release. See Ansible and Python 3 for additional info.

  • Ansible 2.6 for Zuul jobs has been marked deprecated and will be removed in a future release.

3.12.0

New Features

Upgrade Notes

  • The Gerrit driver now has an additional option, pipeline.reporter.<gerrit reporter>.comment which may be used to suppress reporting job results as review comments. Due to the configuration syntax for Gerrit reporters, the word “comment” may no longer be used as a review label.

Other Notes

  • The –id option to both the autohold-delete and autohold-info zuul CLI commands has been removed.

3.11.1

Bug Fixes

  • Fixed a regression where changes with a failing job with soft dependencies would remain in the queue indefinitely.

3.11.0

New Features

  • Added the pipeline.enqueue reporter action so that reporters may be run when an item is enqueued into a pipeline.

  • Added the pipeline.no-jobs reporter action so that reporters may be run when an item is dequeued into a pipeline without having run any jobs.

  • A new zuul CLI command, autohold-info, has been added to retrieve full details for an existing autohold request.

  • Autohold requests are now stored in ZooKeeper, rather than in memory. As a result of this change, a new zuul CLI command, autohold-delete has been added to remove existing requests.

  • New scheduler options, max_hold_expiration and default_hold_expiration, are added to control how long nodes held for an autohold request remain available. The max_hold_expiration option defaults to 0, which means the held nodes will not automatically expire, and default_hold_expiration defaults to the value of max_hold_expiration.

Upgrade Notes

  • The Gerrit driver will now perform query actions over HTTP if an HTTP password is configured. In this situation, the SSH connection is now only used for receiving events. The HTTP password is not required, but it is recommended since it provides more reporting options (including line comments).

3.10.2

New Features

  • The maximum starting builds depending on cpu cores can be limited using executor.max_starting_builds configuration.

Security Issues

  • The synchronize rsh rsync_opts is now prohibited for security reasons as it could be used to circumvent executor host command execution restrictions.

Bug Fixes

  • Under specific conditions Zuul could fail to properly update git repos causing it to fall back to testing master. If jobs were meant to run against commits other than current master this lead to testing the wrong commit. Zuul now correctly updates repos and should test the correct commit in all cases.

    Specifically this could happen if a new branch was created off a commit in an existing branch. Then the first proposed change to this new branch would be tested against master instead.

3.10.1

Bug Fixes

  • Fixed Zuul dashboard not loading properly due to javascript dependencies missing.

3.10.0

New Features

  • Allow users to perform tenant-scoped, privileged actions either through zuul-web’s REST API or zuul’s client, based on the JWT standard. The users need a valid bearer token to perform such actions; the scope is set by matching conditions on tokens’ claims; these conditions can be defined in zuul’s tenant configuration file. Zuul supports token signing and validation using the HS256 or RS256 algorithms. External JWKS are also supported for token validation only. Current tenant-scoped actions are “autohold”, “enqueue” and “dequeue”.

  • Config projects may now add any job to any project’s pipelines, regardless of the setting of allowed-projets (including the implicit setting of allowed-projects on jobs with secrets in untrusted projects).

  • It is now possible to select the merge method GitHub uses to merge a Pull Request to the base branch.

  • A new option, tenant.untrusted-projects.<project>.extra-config-paths has been added to allow an admin to indicate that Zuul should load extra configuration data from a specific project in a tenant.

    This feature may be useful to allow a project that primarily holds shared jobs or roles to include additional in-repo configuration for its own testing (which may not be relevant to other users of the project).

  • The merge-mode squash-merge is now supported for Github.

  • The job.cleanup-run attribute has been added to enable a new cleanup phase to be performed after a job execution.

  • Support for the Pagure code review system has been added.

  • Pipelines may now indicate that they supercede other pipelines with the pipeline.supercedes attribute.

    When a change is enqueued in a pipeline which supercedes others, it will be removed from the other pipelines. For example, a gate pipeline may supercede a check pipeline so that test resources are not spent running near-duplicate jobs simultaneously.

Upgrade Notes

  • If unprotected branches are excluded on a project they now also get filtered out from jobs.

  • The default value for the executor.job_dir configuration setting has been changed from /tmp to /var/lib/zuul/builds. It is important for executor.job_dir and executor.git_dir to be located on the same filesystem, so this change increases the chances that they end up on the same filesystem in most default configurations.

    The builds subdirectory will be created if it does not exist, however, /var/lib/zuul at least must exist and be writable by the Zuul user.

  • Jobs with file matchers will now automatically match if the configuration of the job is changed. This means that the Zuul configuration file no longer needs to be included in the list of files to match in order for changes to job configuration to be self-testing.

    To keep the old behavior, set job.match-on-config-updates to False.

3.9.0

New Features

  • Zuul now supports ansible 2.8 for running jobs.

  • Zuul now reports resource usage statistics if they are provided by nodepool.

    The following statistics are emitted:

    • zuul.nodepool.resources.tenant.{tenant}.{resource}: Gauge with the currently used resources by tenant and counter with the summed usage by tenant. e.g. cpu seconds

    • zuul.nodepool.resources.project.{project}.{resource}: Gauge with the currently used resources by project and counter with the summed usage by project. e.g. cpu seconds

3.8.1

Bug Fixes

  • Fixed a memory leak introduced in 3.8.0.

3.8.0

New Features

  • The artifacts returned by parent jobs are now also available in child jobs of the same buildset.

Security Issues

  • Fixed a bug where config (trusted) layout updates could be used pre-merge as a dynamically loaded layout. This could happen if Zuul was running with config errors that originated from outside of the config (trusted) repo. A logic error allowed code to fall through and return the trusted layout in this case.

    Users should upgrade.

Bug Fixes

  • Jobs which use the job.requires attribute and fail to have their requirements met are now recorded as “FAILED” rather than “SKIPPED”. This can happen if an earlier job which is expected to produce artifacts fails to do so due to an error.

3.7.1

Bug Fixes

  • When using Ansible 2.7 the uri module crashed with a module failure.

3.7.0

New Features

  • The ansible_python_interpreter variable is now whitelisted for the add_host task.

  • Zuul now supports ansible 2.6 for running jobs.

  • Zuul now supports ansible 2.7 for running jobs.

  • Jobs may now specify which ansible version is used to run them. The ansible version to use can now be specified by job.ansible-version.

Upgrade Notes

  • The user value in the [fingergw] configuration section previously defaulted to zuul, but now is unset by default, which will cause fingergw not to drop privileges. It is recommended that this value be explicitly set to an unprivileged user.

  • In order to support several ansible versions installed simultaneously Zuul now handles them itself in virtual environments. By default Zuul installs the needed ansible versions on startup so there is no further user action required. However it is recommended to pre-install the ansible environments during installation by invoking zuul-manage-ansible.

  • The hosts value in the [zookeeper] configuration section previously defaulted to localhost:2181, but now is unset by default. This value is required and must be explicitly set (and the documentation has always indicated this).

3.6.1

Security Issues

  • The raw module had not been blocked for local tasks. This could be used to bypass protection and execute commands on the executor.

3.6.0

New Features

  • Artifacts may now include a metadata field for storing arbitrary metadata about the artifacts in the SQL database.

  • The job.run attribute now supports a single or list of playbooks.

  • Support for expressing artifact or other resource dependencies between jobs running on different changes with a dependency relationship (e.g., a container image built in one project and consumed in a second project) has been added via the job.provides and job.requires job attributes.

  • The job.dependencies attribute may now be used to express “soft” dependencies – that is, to indicate a job should run after another completes, but only if it runs at all. For example, a deployment job which should always run, but depends on a build job which only runs if the source code is changed.

Upgrade Notes

  • Service Workers are now disabled by default in the Javascript dashboard. Deployers who wish to enable them need to set REACT_APP_ENABLE_SERVICE_WORKER

  • Zuul recently added the job variable zuul.message. This can contain jinja tags which can cause problems accessing the zuul variable in the job. Because of this the message is now base64 encoded and any job evaluating this variable needs to be changed from {{ zuul.message }} to {{ zuul.message | b64decode }}.

3.5.0

New Features

  • The executor.min_avail_mem setting now takes cgroup limits into account. There is also a new metric zuul.executor.<executor>.pct_used_ram_cgroup available.

  • The job.secrets.pass-to-parent attribute has been added to allow secrets to be made available to playbooks in parent jobs (for example, to allow for jobs which are designed to use secrets, but leave it to child jobs to actually supply them). See also the Secret documentation.

  • The restriction on using known_hosts in playbooks has been lifted.

Upgrade Notes

  • The zuul_return module has been converted to an Ansible action plugin. Job playbooks no longer need to use delegate_to or a localhost only play with this module as action plugin by default run on localhost.

Security Issues

  • Jobs with secrets in untrusted projects now automatically set allowed-projects.

    It is possible to circumvent the use of allowed-projects in untrusted projects by creating a change which Depends-On a change which alters a project definition. This behavior may be unexpected, so documentation has been updated with warnings to avoid relying on it in sensitive cases.

    It may have been possible to expose a secret, or use resources protected by a secret, if a job using a secret was defined in an untrusted project on a system with an independent pre-merge post-review pipeline – that is, a pipeline with post-review set to true, manager set to independent, and which operated on changes before they merged.

    To prevent disclosure or use in this situation, allowed-projects is now automatically set to the current project when a secret is used in a job defined in an untrusted project, and it can not be overridden.

Bug Fixes

  • Untrusted playbooks no longer see ‘Adding hosts ssh with ansible_user to the inventory is prohibited’ when using the add_host Ansible task on localhost.

  • Fixed an issue where a trailing slash on the baseurl of a git driver connection config could cause an indefinite job hang.

3.4.0

New Features

  • Jobs may now return artifact URLs and they will be stored in the SQL database (if configured). See Returning artifact URLs for usage.

  • One or more zuul executors can now be added to a executor.zone. This is helpful if a cloud does not have any public IP addresses for nodepool nodes. Now you’ll be able to have a zuul executor on the same private network as your nodepool nodes.

  • The Ansible inventory file now includes nodepool.host_id variable if the node was launched using the OpenStack Nodepool driver.

  • A new scheduler option, scheduler.relative_priority, can be used to instruct Nodepool to fulfull requests from less-busy projects more quickly.

Upgrade Notes

  • The zuul.nodepool statistics have been moved under zuul.nodepool.requests to allow sub-stats to work correctly. For example zuul.nodepool.requested has become zuul.nodepool.requests.requested.total. The previously missing label and size counters are now available at zuul.nodepool.requests.<state>.<size|label>. For more info see the monitoring documentation.

  • The zuul-web service now requires access to ZooKeeper, as a result you may need to update your firewall allow access from the service. Additionally, zuul.conf should now contain a zookeeper configuration section.

3.3.1

New Features

  • A job using a semaphore now can configure if it should acquire the it before requesting resources or just before running.

Upgrade Notes

  • The acquiring behavior of jobs with semaphores has been changed. Up to now a job requested resources and aquired the semaphore just before it started to run. However this could lead to a high amount of resource waste. Instead jobs now acquire the semaphore before requesting the resources by default. This behavior can be overridden by jobs using job.semaphores.resources-first if some waste of resources is acceptable.

Security Issues

  • The add_host module options are restricted to a hostname, port, user and password. Previously, malicious options could be used to bypass protection and execute tasks on the executor. Only ssh and kubectl connection are authorized.

Bug Fixes

  • Jobs that encountered unreachable nodes are now correctly detected and retried.

3.3.0

New Features

  • A local project key file URI (eg. file:///path/to/key.pub) is now supported by the encrypt_secret.py tool. This allows encrypting secrets without directly accessing the Zuul web API to retrieve the project key.

  • Zuul now supports triggering a full reconfiguration by using the command zuul-scheduler full-reconfigure.

  • The Gerrit driver can now (optionally) report via HTTP instead of SSH. In the future, this will be used to report file and line comments (the SSH API only supports review messages).

  • Zuul now supports leaving file comments, though currently only when using the Gerrit driver. See the documentation for zuul_return for details.

  • A job now can pause itself using Return Values and let the child jobs run until they are finished. This can be used to serve some service which can be used by the child jobs.

  • An SSH keypair is now generated for every project and may be used in post-review jobs to access systems which trust that project.

  • The Zuul web dashboard has been rewritten in React.

  • The restriction on using add_host in playbooks has been lifted.

  • A new Build page in the web interface enable displaying a single Build information.

  • A new Notification Drawer and a ConfigErrors page in the web interface enable displaying the config-errors endpoint data.

  • A new Job page in the web interface enable browsing through job configuration.

  • Client certificate locations to be used by winrm connections can be configured now.

Upgrade Notes

  • The Zuul web dashboard is now a single index.html and static offload server needs new rewrite rules. Check the install instruction for backward compatible rewrite rules. Note that serving the web dashboard from a sub-directory requires the application to be rebuilt using the desired homepage location.

Deprecation Notes

  • Signal based triggering of a full reconfiguration via sending SIGHUP to the zuul-scheduler PID is deprecated. Use the command zuul-scheduler full-reconfigure now.

3.2.0

New Features

  • Zuul client got a new command ‘tenant-conf-check’. This command validates the schema of the tenant configuration and exits -1 if errors have been detected.

  • A new Ansible inventory variable zuul.child_jobs which is a list of the first level child jobs to be run after a job has finished successfully.

  • Jobs are now able to use the job.extra-vars which will use the –extra-vars flag when a job runs.

  • Project and project-templates may now create variables via a vars configuration entry. Jobs can access these at runtime in the same manner as job variables.

  • The supercedent pipeline manager has been added. It is designed to make post-merge artifact build pipelines more efficient.

  • The dequeue command has been added to the Zuul CLI. It allows operators to stop a given buildset at will.

  • It is now possible to use zuul_return to skip child jobs. You can use the zuul.child_jobs inventory variable to get a list of child jobs configured to run, then use zuul_return to modify the list. Any child job not in zuul_return zuul.child_jobs will be skipped. See Return Values for examples.

Bug Fixes

  • Project Templates are now branch-aware and behave more like project stanzas. If a template is defined on a branch, it will only apply to changes to that branch.

  • The timer trigger does not enqueue an event for every branch of every project anymore and it now only processes projects actually using the pipeline triggered by a timer.

3.1.0

New Features

  • The GitHub driver can determine the required status checks of pull requests which are needed for entering a gate pipeline. This eliminates the need to hard code required status checks in the gate pipeline and makes interoperation with other GitHub apps much more flexible.

  • Zuul is now ables to start with an invalid configuration. When reading configuration files from project repositories, if an issue is detected, Zuul will store the issue and skip the broken block of configuration. Issues are then reported in the scheduler log at the end of the configuration phase.

  • The json log now also contains the role name and the uuid similar to the task entry.

Upgrade Notes

  • Files (and irrelevant-files) matchers are now overridable. Zuul now uses only branch matchers to collect job variants. Once those variants are collected, they are combined, and the files and irrelevant-files attributes are inherited and overridden as any other job attribute. The final values are used to determine whether the job should ultimately run.

  • Zuul now uses Ansible 2.5.

Security Issues

  • Tobias Henkel (BMW Car IT GmbH) discovered a vulnerability which is fixed in this release. If nodes become offline during the build, the no_log attribute of a task is ignored. If the unreachable error occurred in a task used with a loop variable (e.g., with_items), the contents of the loop items would be printed in the console. This could lead to accidentally leaking credentials or secrets. MITRE has assigned CVE-2018-12557 to this vulnerability.

Bug Fixes

  • Untrusted playbooks no longer see ‘Executing local code is prohibited’ when using the zuul_return Ansible task.

3.0.3

New Features

  • The project.default-branch option is now documented. It has been supported since version 3.0.0, but was omitted from the documentation.

  • Project stanzas now support regex matching of project.name. This can be used to apply project pipelines to many projects at once.

Deprecation Notes

  • The merge-mode and default-branch attributes may no longer appear in a Project Template stanza.

Bug Fixes

  • Configuration loading for dynamic configuration changes (i.e., changes to zuul.yaml files) is now significantly more CPU and memory efficient, incurring only a slight penalty compared to normal changes.

3.0.2

New Features

Upgrade Notes

  • The fb-re2 python library is added as a dependency; this may required the installation of the re2 library and header files in order to build.

  • The build start and end times are now stored as UTC in the database. Since there is no localization done on zuul web side, the zuul api also exposes these times as UTC now.

Bug Fixes

  • Story 2001441 is fixed. Failure by one Zuul reporter will not short circuit the reporting of other reporters. This ensures as much information as possible is reported for each change even if some failures occur. Note that the build set status is changed to ‘ERROR’ after the first failed reporter.

  • The zuul-changes.py script has been adapted to the new zuul-web api routes.

3.0.1

New Features

  • Git repositories will have a origin remote with refs pointing to the previous change in the speculative state.

    This allows jobs to determine the commits that are part of a change, which was not possible before. The remote URL is set to a bogus value which won’t work with git commands that need to talk to the remote repository.

  • PostgreSQL is now officially supported as database backend. See sql connection on how to configure database connections.

Upgrade Notes

  • The alembic version table is fixed to being prefixed too. This is necessary when using table_prefix. However if you are already using table_prefix you will need to rename the table alembic_version to <prefix>alembic_version before starting Zuul. Otherwise zuul will try to create the tables again and fail. If you’re not using table_prefix you can safely ignore this.

Bug Fixes

  • Zuul role repository checkouts now honor job.override-checkout.

    Previously, when a Zuul role was specified for a job, Zuul would usually checkout the master branch, unless that repository appeared in the dependency chain for a patch. It will now follow the usual procedure for determining the branch to check out, including honoring job.override-checkout options.

    This may alter the behavior of currently existing jobs. Depending on circumstances, you may need to set job.override-checkout or copy roles to other branches of projects.