Container Roles

build-container-image

Build one or more container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

build-container-image: Build the images.
upload-container-image: Upload the images to a registry.
promote-container-image: Promote previously uploaded images.

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.
The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command Default: podman: The command to use when building the image (E.g., docker).

container_registry_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar

container_registry_credentials{}.[registry name] Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username: The registry username.

container_registry_credentials{}.[registry name]{}.password: The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

docker : API is access via username/password, does not require token.
quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images Type: list

A list of images to build. Each item in the list should have:

container_images[].context: The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename: The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry: The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository: The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path: Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args Type: list: Optional: a list of values to pass to the --build-arg parameter.

container_images[].target: Optional: the target for a multi-stage build.

container_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

container_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env Type: dict: A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method Default: tag: A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

build-docker-image

Build one or more docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

build-docker-image: Build the images.
upload-docker-image: Upload the images to Dockeer Hub.
promote-docker-image: Promote previously uploaded images.

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile Default: Dockerfile: The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry Default: '': The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username: The Docker Hub username.

docker_credentials{}.password: The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

docker_use_buildkit Default: false Type: bool: Use BuildKit when creating images.

docker_images Type: list

A list of images to build. Each item in the list should have:

docker_images[].context: The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile: The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository: The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path: Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args Type: list: Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target: Optional: the target for a multi-stage build.

docker_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

docker_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels Default: [] Type: list: A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch Default: [] Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

collect-container-logs

An ansible role to collect all container logs.

Role Variables

container_command Default: docker: Container run-time CLI command

deploy-openshift: Deploy openshift using oc cluster up.

ensure-docker

An ansible role to install docker and configure it to use mirrors if available.

Role Variables

mirror_fqdn Default: {{ zuul_site_mirror_fqdn }}: The base host for mirror servers.

docker_mirror: URL to override the generated docker hub mirror url based on ensure-docker.mirror_fqdn.

use_upstream_docker Default: True: By default this role adds repositories to install docker from upstream docker. Set this to False to use the docker that comes with the distro.

docker_use_buildset_registry Default: False: This role does not enable the usage of the buildset registry by default, this variable allows enabling the usage of the buildset registry after installing Docker.

docker_compose_install Default: False: This role does not install docker-compose by default but you can use this setting to install docker-compose as well.

docker_update_channel Default: stable: Which update channel to use for upstream docker. The two choices are stable, which is the default and updates quarterly, and edge which updates monthly.

docker_insecure_registries Default: undefined: Declare this with a list of insecure registries to define the registries which are allowed to communicate with HTTP only or HTTPS with no valid certificate.

docker_gpg_key Default: string: The raw content of the upstream docker gpg key, as found here https://download.docker.com/linux/fedora/gpg

docker_distro_packages Default: list: List of packages to be installed when use_upstream_docker is set to false. The package set is defined by default using distro specific variables. If the package set needs to be changed this option can be overridden as needed.

docker_upstream_distro_required_packages Default: list: List of packages to be installed when use_upstream_docker is set to true. The package set is defined by default using distro specific variables and contains a list of supporting packages required to be installed prior to installing docker-ce. If the package set needs to be changed this option can be overridden as needed.

docker_upstream_distro_remove_packages Default: list: List of packages to be removed before installing new ones. It is used for avoiding potential conflicts. For example it can remove docker package before trying to install docker-ce. The default value is distro specific.

docker_upstream_distro_packages Default: list: List of packages to be installed when use_upstream_docker is set to true. The package set is defined by default using distro specific variables. If the package set needs to be changed this option can be overridden as needed.

docker_download_fqdn Default: download.docker.com: Add default option to set the docker download fqdn.

docker_mirror_base_url Default: https://{{ docker_download_fqdn }}/linux/{ubuntu,centos,fedora}: By default this option sets the repository base url. This variable is based on ensure-docker.docker_download_fqdn. When this option is unset, the role will use distro specific variables which are loaded at the time of execution.

docker_userland_proxy Type: bool: Set to false to disable the docker userland proxy. This variable is useful when docker is causing routing problem, such as when a kubernetes deployment is unable to reach its own service.

ensure-kubernetes

An ansible role to install kubernetes.

Role Variables

ensure_kubernetes_type Default: minikube: The kubernetes distribution to use. Currently `minikube or `microk8s`. Note that `microk8s` is only implemented for Ubuntu Jammy distributions currently.

ensure_kubernetes_microk8s_channel Default: latest/stable: The snap channel to use for `microk8s`. See https://microk8s.io/docs/setting-snap-channel.

ensure_kubernetes_microk8s_addons Default: ['dns', 'storage']: The addons for microk8s`. See https://microk8s.io/docs/addons

install_kubernetes_with_cluster Default: True: If true, installs a Minikube cluster.

minikube_version Default: latest: The version of Minikube to install.

minikube_dns_resolvers Default: []: List of dns resolvers to configure in k8s. Use this to override the resolvers that are found by default.

kubernetes_runtime Default: docker: Which kubernetes runtime to use for minikube; values are docker, cri-o or podman. For any other values the ensure_kubernetes_minikube_* options will be used instead. Please note that only some combinations of profiles and distros might be valid.

ensure_kubernetes_minikube_addons Default: []: List of addons to configure in k8s. Use this to enable the addons.

ensure_kubernetes_minikube_driver Default: none: Which driver to use for minikube. The default is the none driver. See also kubernetes_runtime.

ensure_kubernetes_minikube_runtime Default: docker: Which kubernetes runtime to use for minikube. See also kubernetes_runtime.

ensure_kubernetes_minikube_provider Default: docker: Which container provider to use for minikube. See also kubernetes_runtime.

ensure_kubernetes_bin_path Default: /tmp: Where to install binaries for minikube. This is currently set to retain compatibility with existing users, but the intention is to move the install default to a more sane location in the future.

ensure_kubernetes_minikube_memory Default: no-limit: For the podman runtime, the podman container running the entire minikube instance can have a global memory limit applied. The default value sets no limit.

ensure_kubernetes_minikube_cpus Default: no-limit: For the podman runtime, the podman container running the entire minikube instance can have a global cpu limit applied. The default value sets no limit.

ensure-openshift

Setup openshift requirements and pull the container images. The deploy-openshift role can be used to start the services.

This role only works on CentOS.

Role Variables

origin_version Default: v3.11.0: The origin version.

ensure-podman

Install podman container manager

Role Variables

ensure_podman_validate Default: false: Used to enable validation of podman engine.

ensure-skopeo

Install skopeo

Role Variables

ensure_skopeo_install_from_upstream Default: false

Build and install skopeo from upstream. This installs a go development environment and build dependencies. The binary is installed into /usr/local/bin.

Currently only implemented for Ubuntu Jammy.

ensure_skopeo_install_from_upstream_version Default: v1.9.3: When building skopeo locally, the version tag to use.

ensure-quay-repo

This role primarily exists to create a new public repository in quay. This role can be used to create private repos as well, but repos are created by default in quay if you simply push to them.

Users of this role will need to generate an application token with create repository permissions. Additional permissions are not necessary.

When invoking this role you should set no_log: true on the include_role task to prevent disclosure of your token.

** Role Variables **

container_registry_credentials Type: dict

Required. This is expected to be a Zuul secret in dictionary form. For convenience this is in the same format as the container_registry_credentials variable used by the other container roles. Specify an api_token which is issued from an application assigned to an organisation. See https://docs.quay.io/api/ This application API token needs create permissions. If the registry name is quay.io we know that the registry type is quay. If you are running a private Quay installation you can manually set type to quay to force this behavior.

This role will only take action on image repos that map to container registries for which both an api_token is set and type can be determined to be quay.

You may also set api_url on the registry credentials if the API is not hosted at the root of the registry name. Most installations should be able to ignore this and use the default of https://{{ $name }}.

Example:

container_registry_credentials:
  quay.io:
    type: 'quay'
    api_token: 'abcd1234'
    api_url: 'https://quay.io'

container_images Type: list

Required. A list of dictionaries. This provides info about the image repositories to be created in a quay registry. For convenience this is in the same format as the container_images variable used by other container roles. Specify a registry (this should match up with your credentials to locate the api token), namespace, repo_shortname, repo_description, and visibility attributes. By default visibility will be public.

Example:

container_images:
  - registry: quay.io
    namespace: myquayorg
    repo_shortname: myimage
    repo_description: The best container image

pause-buildset-registry

Pause a buildset registry

Utility role to pause a job providing a buildset registry.

Role Variables

buildset_registry: Location of external buildset registry. If this is defined, the job will not pause.

promote-container-image

Promote one or more previously uploaded container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

build-container-image: Build the images.
upload-container-image: Upload the images to a registry.
promote-container-image: Promote previously uploaded images.

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.
The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command Default: podman: The command to use when building the image (E.g., docker).

container_registry_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar

container_registry_credentials{}.[registry name] Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username: The registry username.

container_registry_credentials{}.[registry name]{}.password: The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

docker : API is access via username/password, does not require token.
quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images Type: list

A list of images to build. Each item in the list should have:

container_images[].context: The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename: The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry: The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository: The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path: Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args Type: list: Optional: a list of values to pass to the --build-arg parameter.

container_images[].target: Optional: the target for a multi-stage build.

container_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

container_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env Type: dict: A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method Default: tag: A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

promote_container_image_method Default: tag Type: string: If tag (the default), then this role will update tags created by the upload-container-image role. Set to intermediate-registry to have this role copy an image created and pushed to an intermediate registry by the build-container-role. In that case, the variables below provide the extra information needed to perform the query.

promote_container_image_api: Only required for the intermediate-registry method. The Zuul API endpoint to use. Example: https://zuul.example.org/api/tenant/{{ zuul.tenant }}

promote_container_image_pipeline: Only required for the intermediate-registry method. The pipeline in which the previous build ran.

promote_container_image_job: Only required for the intermediate-registry method. The job of the previous build.

promote_container_image_query Default: change={{ zuul.change }}&patchset={{ zuul.patchset }}&pipeline={{ promote_container_image_pipeline }}&job_name={{ promote_container_image_job }}: Only required for the intermediate-registry method. The query to use to find the build. Normally the default is used.

promote-docker-image

Promote one or more previously uploaded docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

build-docker-image: Build the images.
upload-docker-image: Upload the images to Dockeer Hub.
promote-docker-image: Promote previously uploaded images.

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile Default: Dockerfile: The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry Default: '': The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username: The Docker Hub username.

docker_credentials{}.password: The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

docker_use_buildkit Default: false Type: bool: Use BuildKit when creating images.

docker_images Type: list

A list of images to build. Each item in the list should have:

docker_images[].context: The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile: The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository: The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path: Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args Type: list: Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target: Optional: the target for a multi-stage build.

docker_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

docker_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels Default: [] Type: list: A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch Default: [] Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

pull-from-intermediate-registry

Pull artifacts from the intermediate registry

This role will pull any artifacts built for changes ahead of this change which have been placed in an intermediate registry into the buildset registry for this buildset.

Run this in a trusted pre-playbook at the start of a job (which, in the case of multiple dependent jobs in a buildset, should be at the root of the job dependency graph).

This requires the run-buildset-registry role already applied. It also requires an externally managed “intermediate” registry operating for the use of Zuul, and it requires “skopeo” and “socat” to be installed on the Zuul executors.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host: The host (IP address) of the registry.

buildset_registry.port: The port on which the registry is listening.

buildset_registry.username: The username used to access the registry via HTTP basic auth.

buildset_registry.password: The password used to access the registry via HTTP basic auth.

buildset_registry.cert: The (self-signed) certificate used by the registry.

intermediate_registry

Information about the registry. This is expected to be provided as a secret.

intermediate_registry.host: The host (IP address) of the registry.

intermediate_registry.port: The port on which the registry is listening.

intermediate_registry.username: The username used to access the registry via HTTP basic auth.

intermediate_registry.password: The password used to access the registry via HTTP basic auth.

push-to-intermediate-registry

Push artifacts to the intermediate registry

This role will push any images built by build-docker-image or build-container-image into an intermediate registry.

Run this in a trusted post-playbook at the end of a job after the image build.

This requires the run-buildset-registry role already applied. It also requires an externally managed “intermediate” registry operating for the use of Zuul, and it requires “skopeo” and “socat” to be installed on the Zuul executors.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host: The host (IP address) of the registry.

buildset_registry.port: The port on which the registry is listening.

buildset_registry.username: The username used to access the registry via HTTP basic auth.

buildset_registry.password: The password used to access the registry via HTTP basic auth.

buildset_registry.cert: The (self-signed) certificate used by the registry.

intermediate_registry

Information about the registry. This is expected to be provided as a secret.

intermediate_registry.host: The host (IP address) of the registry.

intermediate_registry.port: The port on which the registry is listening.

intermediate_registry.username: The username used to access the registry via HTTP basic auth.

intermediate_registry.password: The password used to access the registry via HTTP basic auth.

docker_images Type: list

A list of images built. Each item in the list should have:

docker_images[].repository: The name of the target repository for the image.

docker_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image.

container_images Type: list

A list of images built. Each item in the list should have:

container_images[].repository: The name of the target repository for the image.

container_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image.

remove-registry-tag

Remove tags from registry

This role creates a generic interface for removing tags from a container registry. The OCI distribution API (implemented essentially all registries) does specify a tag deletion endpoint, but as at 2023-03 essentially no registries implement it. This means practically we must talk to the per-registry API directly to remove tags. The methods to delete tags are generally similar across registries, but differ slightly in endpoint names, etc.

This role can run in two modes; either removing a single specific tag, or it can run a cleanup process removing all tags that match a given prefix and have not been modified in a given amount of time.

For public registries this role should guess the API from the repository name. If you are running against a private registry, you will need to explicitly specify the API type and URL prefix to communicate to using arguments below.

Role Variables

remove_registry_tag_repository Type: string: Required. This must be the full repository; e.g. quay.io/organisation/image

container_registry_credentials Type: dict

Required. This is expected to be a Zuul secret in dictionary form. For convenience this is in the same format as the container_registry_credentials variable used by the other container roles. You must specify the correct variables for the registry you are communicating with:

quay.io : Specify an api_key which is issued from an application assigned to an organisation. See https://docs.quay.io/api/
docker.io : Username and password

Example:

container_registry_credentials:
  quay.io:
    api_token: 'abcd1234'
  docker.io:
    username: 'username'
    password: 'password'

remove_registry_tag_tag Type: string: Optional. If set, the specific tag to remove.

remove_registry_tag_regex Default: '^change_.*$|^{{ zuul.pipeline }}_.*$' Type: string: Optional. If remove-registry-tag.remove_registry_tag_tag is unset, any tags matching this regex and exceeding the age in remove-registry-tag.remove_registry_tag_age will be removed. The default is tags matching those created by the promote upload roles.

remove_registry_tag_age Default: 86400 Type: int: Optional. The age, in seconds, a tag that matches remove-registry-tag.remove_registry_tag_regex last-modified timestamp must exceed to be removed.

remove_registry_tag_api_type Type: string

Optional. By default the role will guess the API type from the repository name. However, if you need to override this choice specify one of:

quay
docker

remove_registry_tag_api_url Type: string: Optional. This role will use the default URL for the given registry API. If you need to override this choice, specify this variable.

run-buildset-registry

Runs a container registry for the use of this buildset.

This may be used for a single job running on a single node, or it may be used at the root of a job graph so that multiple jobs running for a single change can share the registry.

Role Variables

buildset_registry_root Default: {{ ansible_user_dir }}/buildset_registry: Path for the registry volumes.

buildset_registry_port Default: 5000: The port on which the registry should listen.

container_command Default: docker: The command to use to run the registry container (E.g., podman).

Return Values

buildset_registry

Information about the registry.

buildset_registry.host: The host (IP address) of the registry.

buildset_registry.port: The port on which the registry is listening.

buildset_registry.username: The username used to access the registry via HTTP basic auth.

buildset_registry.password: The password used to access the registry via HTTP basic auth.

buildset_registry.cert: The (self-signed) certificate used by the registry.

upload-container-image

Upload one or more container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

build-container-image: Build the images.
upload-container-image: Upload the images to a registry.
promote-container-image: Promote previously uploaded images.

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.
The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command Default: podman: The command to use when building the image (E.g., docker).

container_registry_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar

container_registry_credentials{}.[registry name] Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username: The registry username.

container_registry_credentials{}.[registry name]{}.password: The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

docker : API is access via username/password, does not require token.
quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images Type: list

A list of images to build. Each item in the list should have:

container_images[].context: The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename: The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry: The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository: The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path: Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args Type: list: Optional: a list of values to pass to the --build-arg parameter.

container_images[].target: Optional: the target for a multi-stage build.

container_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

container_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env Type: dict: A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method Default: tag: A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

upload_container_image_promote Default: true Type: bool: If true (the default), then this role will upload the image(s) to the container registry with special tags designed for use by the promote-container-image role. Set to false to use this role to directly upload images with the final tag (e.g., as part of an un-gated release job).

upload-docker-image

Upload one or more docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

build-docker-image: Build the images.
upload-docker-image: Upload the images to Dockeer Hub.
promote-docker-image: Promote previously uploaded images.

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir Default: {{ zuul.project.src_dir }}: The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile Default: Dockerfile: The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry Default: '': The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username: The Docker Hub username.

docker_credentials{}.password: The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"

docker_use_buildkit Default: false Type: bool: Use BuildKit when creating images.

docker_images Type: list

A list of images to build. Each item in the list should have:

docker_images[].context: The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile: The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository: The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path: Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args Type: list: Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target: Optional: the target for a multi-stage build.

docker_images[].tags Default: ['latest'] Type: list: A list of tags to be added to the image when promoted.

docker_images[].siblings Default: [] Type: list: A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels Default: [] Type: list: A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch Default: [] Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

upload_docker_image_promote Default: true Type: bool: If true (the default), then this role will upload the image(s) to Docker Hub with special tags designed for use by the promote-docker-image role. Set to false to use this role to directly upload images with the final tag (e.g., as part of an un-gated release job).

use-buildset-registry

Adds a buildset registry to the docker configuration.

Use this role on any host which should use the buildset registry.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host: The host (IP address) of the registry.

buildset_registry.port: The port on which the registry is listening.

buildset_registry.username: The username used to access the registry via HTTP basic auth.

buildset_registry.password: The password used to access the registry via HTTP basic auth.

buildset_registry.cert: The (self-signed) certificate used by the registry.

buildset_registry_docker_user Default: {{ ansible_user }}: The system user to configure to use the docker registry. The docker configuration file for this user will be updated. By default, the user Ansible is running as.

buildset_registry_namespaces Default: [ ('docker.io', 'https://...'), ('quay.io', ...), ('gcr.io', ...)]

The namespaces that the buildset registry supports. Each entry should be a tuple with the first elemnet being the registry host (usually the internet domain name) and the second being the URL the registry is hosted at.

The buildset registry will be consulted first for images in these namespaces. Any others will be fetched only from their upstream sources.

Add any local or third-party registries necessary here.

The default may change in the future as more general-purpose public registries become known.

buildset_registry_unqualified_registries Default: [ 'docker.io' ]: Some tools` like podman and cri-o are stricter when looking for unqualified registries. This sets up containerd (and cri-o) with a default unqualified search prefix, making it compatible with legacy behaviour. More detail at containers-registries.conf.

use-docker-mirror

Configure docker to use mirrors if available.

Role Variables

mirror_fqdn Default: {{ zuul_site_mirror_fqdn }}: The base host for mirror servers.

docker_mirror: URL to override the generated docker hub mirror url based on ensure-docker.mirror_fqdn.

docker_insecure_registries Default: undefined: Declare this with a list of insecure registries to define the registries which are allowed to communicate with HTTP only or HTTPS with no valid certificate.