Kubernetes

Zuul can use pods or namespaces from Kubernetes as a source for build nodes.

Connection Configuration

The supported options in zuul.conf connections are:

<kubernetes connection>

<kubernetes connection>.driver (required)

kubernetes: The connection must set driver=kubernetes for Kubernetes connections.

<kubernetes connection>.kubeconfig_file: A path to a kubeconfig file with credentials to access the Kubernetes cluster. If this is not present, Zuul will use the value of the KUBECONFIG environment variable. If neither this setting or the environment variable is present, then any credentials obtainable by the Kubernetes client library will be used.

<kubernetes connection>.context: If the kubeconfig file has more than one context available, this option may be used to specify which one Zuul should use for this connection.

<kubernetes connection>.rate Default: 2: The API rate limit (in requests per second) to use when performing Kubernetes API calls.

Provider Configuration

The kubernetes driver adds the following options to the provider and section configurations:

provider[kubernetes] Type: dict

The attributes available for configuring a Kubernetes provider are below.

provider[kubernetes].abstract Default: False Type: bool: Whether a section is intended to be inherited by another section or a provider. This setting is currently unused (but may be used in the future). If a section is used to provide common values to other sections, set this to true. Otherwise, the default of false indicates that the section should be referenced directly by providers.

provider[kubernetes].connection Type: str: The name of the connection to use. This attribute is only used by section objects and may not be changed via inheritance.

provider[kubernetes].flavor-defaults Type: dict

Attributes to be set as default values for any flavor used with this provider. Many attributes which may be set on an individual flavor may be set once in this section and used for all the flavors in this provider. Values set on individual flavors may still override the values set here.

provider[kubernetes].flavor-defaults.final Default: False

Whether the configuration of the flavor may be updated by values in flavor-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The flavor may not be updated or overidden.

False: The flavor may be updated or overidden.

allow-override: The flavor may not be updated by flavor-defaults but may be explicitly overidden by redefining it in a new ‘flavor’ entry.

provider[kubernetes].flavors Type: dict

A list of flavors associated with this provider.

provider[kubernetes].flavors.description Type: str: A textual description of the image for reference purposes.

provider[kubernetes].flavors.final Default: False

Whether the configuration of the flavor may be updated by values in flavor-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The flavor may not be updated or overidden.

False: The flavor may be updated or overidden.

allow-override: The flavor may not be updated by flavor-defaults but may be explicitly overidden by redefining it in a new ‘flavor’ entry.

provider[kubernetes].flavors.name Type: str: The name of the flavor. Used to refer to the flavor in Zuul configuration.

provider[kubernetes].image-defaults Type: dict

Attributes to be set as default values for any image used with this provider. Many attributes which may be set on an individual image may be set once in this section and used for all the images in this provider. Values set on individual images may still override the values set here.

provider[kubernetes].image-defaults.connection-port Type: int: The port that Zuul should use when connecting to the node. For most nodes this is not necessary. This defaults to 22 when connection-type is ‘ssh’ and 5986 when it is ‘winrm’.

provider[kubernetes].image-defaults.connection-type

The connection type that a consumer should use when connecting to the node.

winrm: A winrm connection.

ssh: An ssh connection.

provider[kubernetes].image-defaults.final Default: False

Whether the configuration of the label may be updated by values in label-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The label may not be updated or overidden.

False: The label may be updated or overidden.

allow-override: The label may not be updated by label-defaults but may be explicitly overidden by redefining it in a new ‘label’ entry.

provider[kubernetes].image-defaults.import-timeout Default: 300 Type: int: The limit on the amount of time a successful image import can take.

provider[kubernetes].image-defaults.python-path Type: str: The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter.

provider[kubernetes].image-defaults.shell-type Type: str

The shell type of the node’s default shell executable. Used by Zuul to set ansible_shell_type. This setting should only be used

For a windows image with the experimental connection-type ssh in which case cmd or powershell should be set and reflect the node’s DefaultShell configuration.
If the default shell is not Bourne compatible (sh), but instead e.g. csh or fish, and the user is aware that there is a long-standing issue with ansible_shell_type in combination with become.

provider[kubernetes].image-defaults.tags Type: dict: A dictionary of tags to add to uploaded images, and to nodes created from them. Avoid the use of zuul_ as a key prefix since Zuul uses this for internal values.

provider[kubernetes].image-defaults.upload-methods Default: ['copy', 'import', 'upload'] Type: list

An ordered list of methods to use when creating an image in the provider.

copy: Copy the image from another provider if available.

import: Import the image directly from its storage location.

upload: Download the image from its storage location and upload it to the provider.

provider[kubernetes].image-defaults.username Type: str: The username Zuul should use when connecting to the node.

provider[kubernetes].images Type: list: A list of images associated with this provider.

provider[kubernetes].images[cloud] Type: dict

These are the attributes available for a cloud image.

provider[kubernetes].images[cloud].connection-port Type: int: The port that Zuul should use when connecting to the node. For most nodes this is not necessary. This defaults to 22 when connection-type is ‘ssh’ and 5986 when it is ‘winrm’.

provider[kubernetes].images[cloud].connection-type

The connection type that a consumer should use when connecting to the node.

winrm: A winrm connection.

ssh: An ssh connection.

provider[kubernetes].images[cloud].description Type: str: A textual description of the image for reference purposes.

provider[kubernetes].images[cloud].final Default: False

Whether the configuration of the label may be updated by values in label-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The label may not be updated or overidden.

False: The label may be updated or overidden.

allow-override: The label may not be updated by label-defaults but may be explicitly overidden by redefining it in a new ‘label’ entry.

provider[kubernetes].images[cloud].import-timeout Default: 300 Type: int: The limit on the amount of time a successful image import can take.

provider[kubernetes].images[cloud].name Type: str: The name of the image. Used to refer to the image in Zuul configuration.

provider[kubernetes].images[cloud].python-path Type: str: The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter.

provider[kubernetes].images[cloud].shell-type Type: str

The shell type of the node’s default shell executable. Used by Zuul to set ansible_shell_type. This setting should only be used

For a windows image with the experimental connection-type ssh in which case cmd or powershell should be set and reflect the node’s DefaultShell configuration.
If the default shell is not Bourne compatible (sh), but instead e.g. csh or fish, and the user is aware that there is a long-standing issue with ansible_shell_type in combination with become.

provider[kubernetes].images[cloud].type

The type of image.

cloud: An existing image available in the cloud.

provider[kubernetes].images[cloud].username Type: str: The username Zuul should use when connecting to the node.

provider[kubernetes].images[zuul] Type: dict

These are the attributes available for a Zuul image.

provider[kubernetes].images[zuul].connection-port Type: int: The port that Zuul should use when connecting to the node. For most nodes this is not necessary. This defaults to 22 when connection-type is ‘ssh’ and 5986 when it is ‘winrm’.

provider[kubernetes].images[zuul].connection-type

The connection type that a consumer should use when connecting to the node.

winrm: A winrm connection.

ssh: An ssh connection.

provider[kubernetes].images[zuul].description Type: str: A textual description of the image for reference purposes.

provider[kubernetes].images[zuul].final Default: False

Whether the configuration of the label may be updated by values in label-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The label may not be updated or overidden.

False: The label may be updated or overidden.

allow-override: The label may not be updated by label-defaults but may be explicitly overidden by redefining it in a new ‘label’ entry.

provider[kubernetes].images[zuul].import-timeout Default: 300 Type: int: The limit on the amount of time a successful image import can take.

provider[kubernetes].images[zuul].name Type: str: The name of the image. Used to refer to the image in Zuul configuration.

provider[kubernetes].images[zuul].python-path Type: str: The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter.

provider[kubernetes].images[zuul].shell-type Type: str

The shell type of the node’s default shell executable. Used by Zuul to set ansible_shell_type. This setting should only be used

For a windows image with the experimental connection-type ssh in which case cmd or powershell should be set and reflect the node’s DefaultShell configuration.
If the default shell is not Bourne compatible (sh), but instead e.g. csh or fish, and the user is aware that there is a long-standing issue with ansible_shell_type in combination with become.

provider[kubernetes].images[zuul].tags Type: dict: A dictionary of tags to add to uploaded images, and to nodes created from them. Avoid the use of zuul_ as a key prefix since Zuul uses this for internal values.

provider[kubernetes].images[zuul].type

The type of image.

zuul: An image managed by Zuul.

provider[kubernetes].images[zuul].upload-methods Default: ['copy', 'import', 'upload'] Type: list

An ordered list of methods to use when creating an image in the provider.

copy: Copy the image from another provider if available.

import: Import the image directly from its storage location.

upload: Download the image from its storage location and upload it to the provider.

provider[kubernetes].images[zuul].username Type: str: The username Zuul should use when connecting to the node.

provider[kubernetes].label-defaults Type: dict

Attributes to be set as default values for any label used with this provider. Many attributes which may be set on an individual label may be set once in this section and used for all the labels in this provider. Values set on individual labels may still override the values set here.

provider[kubernetes].label-defaults.annotations Type: dict: A dictionary of additional values to be added to the pod metadata. The value of this field is added to the metadata.annotations field in Kubernetes. This field contains arbitrary key/value pairs that can be accessed by tools and libraries.

provider[kubernetes].label-defaults.boot-timeout Default: 300 Type: int: The time (in seconds) to wait for a node to boot.

provider[kubernetes].label-defaults.executor-zone Type: str: Specify that a Zuul executor in the specified zone is used to run jobs with nodes from this label.

provider[kubernetes].label-defaults.final Default: False

Whether the configuration of the label may be updated by values in label-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The label may not be updated or overidden.

False: The label may be updated or overidden.

allow-override: The label may not be updated by label-defaults but may be explicitly overidden by redefining it in a new ‘label’ entry.

provider[kubernetes].label-defaults.image-pull-secrets Type: dict

The imagePullSecrets needed to pull container images from a private registry. Because Zuul creates pods in a new namespace, and image pull secrets must exist in the namespace of the pods that use them, the referenced secrets will be copied into the temporary namespace that Zuul creates before creating the pod. The new secrets will have the same name as the old secrets.

Each entry is a dictionary with the following keys:

provider[kubernetes].label-defaults.image-pull-secrets.name Type: str: Identifier for this secret. The referenced secret must already exist under this name so that Nodepool may copy it. It will be copied into the new namespace with the same name, therefore, if multiple entries are provided, they must have distinct names.

provider[kubernetes].label-defaults.image-pull-secrets.namespace Default: default Type: str: The namespace of the existing secret to copy.

provider[kubernetes].label-defaults.kind

The Kubernetes driver supports two types of labels:

pod: Pod labels provide a dedicated namespace with a single pod and a service account that can exec and get the logs of the pod.

namespace: Namespace labels provide an empty namespace configured with a service account that can create pods, services, configmaps, etc.

provider[kubernetes].label-defaults.max-age Default: 0 Type: int: The time (in seconds) since creation that a node may be available for use. Ready nodes older than this time will be deleted.

provider[kubernetes].label-defaults.max-ready-age Default: 0 Type: int: The time (in seconds) an unassigned node should stay in ready state.

provider[kubernetes].label-defaults.min-retention-time Default: 0 Type: int

The time (in seconds) since an instance was launched, during which a node will not be deleted. For node resources with minimum billing times, this can be used to ensure that the instance is retained for at least the minimum billing interval.

This setting takes precedence over max-[ready-]age.

provider[kubernetes].label-defaults.reuse Default: False Type: bool: Should the node be reused (True) or deleted (False) after use.

provider[kubernetes].label-defaults.slots Default: 1 Type: int: How many jobs are permitted run on the same node simultaneously.

provider[kubernetes].label-defaults.snapshot-expiration Default: 604800 Type: int: The time (in seconds) until a snapshot expires.

provider[kubernetes].label-defaults.snapshot-timeout Default: 3600 Type: int: The time (in seconds) to wait for a snapshot to complete.

provider[kubernetes].label-defaults.tags Type: dict: A dictionary of tags to add to nodes. Avoid the use of zuul_ as a key prefix since Zuul uses this for internal values.

provider[kubernetes].labels Type: dict

A list of labels associated with this provider.

provider[kubernetes].labels.annotations Type: dict: A dictionary of additional values to be added to the pod metadata. The value of this field is added to the metadata.annotations field in Kubernetes. This field contains arbitrary key/value pairs that can be accessed by tools and libraries.

provider[kubernetes].labels.boot-timeout Default: 300 Type: int: The time (in seconds) to wait for a node to boot.

provider[kubernetes].labels.description Type: str: A textual description of the label for reference purposes.

provider[kubernetes].labels.executor-zone Type: str: Specify that a Zuul executor in the specified zone is used to run jobs with nodes from this label.

provider[kubernetes].labels.final Default: False

Whether the configuration of the label may be updated by values in label-defaults or overidden with a new definition by sections or providers lower in the hierarchy than the point at which the final attribute is applied.

True: The label may not be updated or overidden.

False: The label may be updated or overidden.

allow-override: The label may not be updated by label-defaults but may be explicitly overidden by redefining it in a new ‘label’ entry.

provider[kubernetes].labels.flavor Type: str: The flavor to use with this label.

provider[kubernetes].labels.image Type: str: The image to use with this label.

provider[kubernetes].labels.image-pull-secrets Type: dict

The imagePullSecrets needed to pull container images from a private registry. Because Zuul creates pods in a new namespace, and image pull secrets must exist in the namespace of the pods that use them, the referenced secrets will be copied into the temporary namespace that Zuul creates before creating the pod. The new secrets will have the same name as the old secrets.

Each entry is a dictionary with the following keys:

provider[kubernetes].labels.image-pull-secrets.name Type: str: Identifier for this secret. The referenced secret must already exist under this name so that Nodepool may copy it. It will be copied into the new namespace with the same name, therefore, if multiple entries are provided, they must have distinct names.

provider[kubernetes].labels.image-pull-secrets.namespace Default: default Type: str: The namespace of the existing secret to copy.

provider[kubernetes].labels.kind

The Kubernetes driver supports two types of labels:

pod: Pod labels provide a dedicated namespace with a single pod and a service account that can exec and get the logs of the pod.

namespace: Namespace labels provide an empty namespace configured with a service account that can create pods, services, configmaps, etc.

provider[kubernetes].labels.max-age Default: 0 Type: int: The time (in seconds) since creation that a node may be available for use. Ready nodes older than this time will be deleted.

provider[kubernetes].labels.max-ready-age Default: 0 Type: int: The time (in seconds) an unassigned node should stay in ready state.

provider[kubernetes].labels.min-retention-time Default: 0 Type: int

The time (in seconds) since an instance was launched, during which a node will not be deleted. For node resources with minimum billing times, this can be used to ensure that the instance is retained for at least the minimum billing interval.

This setting takes precedence over max-[ready-]age.

provider[kubernetes].labels.name Type: str: The name of the label. Used to refer to the label in Zuul configuration.

provider[kubernetes].labels.reuse Default: False Type: bool: Should the node be reused (True) or deleted (False) after use.

provider[kubernetes].labels.slots Default: 1 Type: int: How many jobs are permitted run on the same node simultaneously.

provider[kubernetes].labels.snapshot-expiration Default: 604800 Type: int: The time (in seconds) until a snapshot expires.

provider[kubernetes].labels.snapshot-timeout Default: 3600 Type: int: The time (in seconds) to wait for a snapshot to complete.

provider[kubernetes].labels.spec Type: dict

Zuul will supply the contents of this value verbatim to Kubernetes as the spec attribute of the Kubernetes Pod definition.

This attribute allows for the creation of arbitrary complex pod definitions but the user is responsible for ensuring that they are suitable. The first container in the pod is expected to be a long-running container that hosts a shell environment for running commands. The following minimal definition is recommended as a starting point:

labels:
  - name: custom-pod
    kind: pod
    spec:
      containers:
        - name: custom-pod
          image: ubuntu:jammy
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh", "-c"]
          args: ["sleep infinity"]

provider[kubernetes].labels.tags Type: dict: A dictionary of tags to add to nodes. Avoid the use of zuul_ as a key prefix since Zuul uses this for internal values.

provider[kubernetes].launch-attempts Default: 3 Type: int: The number of times to attempt launching a node before considering the request failed.

provider[kubernetes].launch-timeout Type: int: The time to wait from issuing the command to create a new node until the node is reporting as running. If the timeout is exceeded, the node launch is aborted and the node deleted.

provider[kubernetes].name Type: str: The name of the provider. Used to refer to the provider in Zuul configuration.

provider[kubernetes].parent Type: str: The name of the parent section from which to inherit. This attribute is only used by section objects. To indicate which section a provider should be attached to, use provider[kubernetes].section

provider[kubernetes].resource-limits Type: dict

Resource limits for this provider. Configure these values to cause Zuul to attempt to limit the resource usage. This can be used to limit Zuul’s usage to a level below the cloud quota.

provider[kubernetes].resource-limits.namespaces Type: int: The number of pods.

provider[kubernetes].resource-limits.pods Type: int: The number of pods.

provider[kubernetes].section Type: str: The name of the section from which to inherit.