Release Notes

4.3.0

New Features

  • Added the option to set quota on resources on a per-tenant basis (i.e. Zuul tenants).

    A new top-level config structure tenant-resource-limits has been added under which one can specify a number of tenants, each with max-servers, max-cores, and max-ram limits. These limits are valid globally, i.e., for all providers. This differs from currently existing provider and pool quotas, which only are considered for nodes of the same provider. This feature is optional and tenant quotas are ignored for any NodeRequests that do not deliver tenant information with them. Also no quota is evaluated for tenants that have no limits configured for them.

4.2.2

Bug Fixes

  • Fixes a regression in gathering public SSH host keys on slower nodes. We now wait until SSHd on the remote node is started before gathering host keys.

4.2.1

Bug Fixes

  • Fix Kubernetes in-cluster configuration loading if no local config is present. The previous release missed a fallback case which has been corrected.

4.2.0

Upgrade Notes

  • Nodepool 0.3.6 introduced an unintended behaviour change with openstacksdk statsd reporting. Due to a change in the way nodepool manages OpenStack API calls, all API related statistics created during interaction with clouds are now generated by openstacksdk and prefixed with openstack.api instead of being created by nodepool and prefixed with nodepool.provider.<cloud> as in prior versions. If you wish to revert to the prior behaviour, changes have been provided to openstacksdk to allow setting custom prefixes via the cloud configuration file; see statsd documentation

4.1.0

New Features

  • Multi-line log messages (such as tracebacks from image builds) are now prefixed in the same manner as single-line messages.

  • AWS EC2, GCE, Kubernetes, Openshift, Openshift Pods, OpenStack and Static drivers now support a shell-type config. Shell-type config is intended to enable setting of cmd or powershell as shell type for Windows workers with connection-type ssh.

    For Linux workers, there is a long standing ansible issue with using non-default shell type and become, so care should be taken if using it for such workers.

Upgrade Notes

  • The zuul-public-key configuration attribute in the providers Azure driver has been moved and renamed. Please move this setting to its new location at providers.[azure].cloud-images.key

4.0.0

Upgrade Notes

  • TLS is now required for ZooKeeper connections. TLS support has been optional since version 3.13. If you have not already enabled it, we recommend enabling it before upgrading to 4.0.

3.14.0

New Features

  • Configuration value can be set from the envirnonment variables using the %(NODEPOOL_env_name) syntax.

  • Basic support for specifying k8s/OpenShift nodeSelectors on Pod node labels. This allows to schedule a Pod on k8s nodes with specific labels, e.g., having certain capabilities.

  • Support for passing environment variables to k8s and OpenShift Pod build nodes has been added.

    It is not possible to set persistent env vars in containers on run time because there is no login shell available. Thus, we need to pass in any env vars during node launch. This allows to set, e.g., http_proxy variables. Environment variables can be defined on node labels as a list of dictionaries with name and value fields as per the k8s container YAML schema.

Upgrade Notes

  • The k8s and OpenShift providers do not longer set the workingDir attribute of their container specs to /tmp.

    For increased flexibility for the user, the working dir specified in the container images Dockerfile is used as the default in container nodes. Please note that this might often be the root dir (‘/’) if not specified otherwise by the respective Dockerfiles WORKDIR directive.

3.13.1

Bug Fixes

  • The dependency on kazoo has been upgraded to 2.8.0 which has an important fix for using Zookeeper over TLS.

3.13.0

New Features

  • The docker images published to zuul/nodepool, zuul/nodepol-launcher and zuul/nodepool-builder are now built as multi-arch images and support arm64 in addition to amd64.

  • Entries in the diskimages section can now specify a parent image to inherit configuration values from. You can also specify images for configuration use only as abstract to consolidate common values.

  • Setting build-log-retention to -1 will disable the removal of old build logs.

  • Support for encrypted connections to ZooKeeper has been added.

    Before enabling, ensure that both Zuul and Nodepool software versions support encrypted connections. See the Zuul release notes, documentation, and associated helper scripts for more information.

    Both Zuul and Nodepool may need to be restarted together with the new configuration.

Upgrade Notes

  • The metric nodepool.provider.<provider>.downPorts has been renamed to nodepool.provider.<provider>.leaked.ports

Bug Fixes

  • Zookeeper hosts specified as IPv6 literals will now be configured correctly.

3.12.0

New Features

  • Support for resources in Google Compute Engine (GCE) has been added.

  • Add optional ebs-optimized on ec2 instances.

  • Add optional tags on ec2 instances and use cloud-image label as Name.

3.11.0

New Features

  • It is now possible to specify if AWS nodes shall get a providers.[aws].pools.public-ip-address.

  • The AWS driver now supports custom providers.[aws].pools.labels.userdata when launching instances.

  • There is a new GET /ready endpoint that can be used as a readiness probe.

3.10.0

New Features

  • Nodepool now supports providers.[openstack].post-upload-hook to run a user supplied script after an image has been uploaded to a cloud but before it gets used.

Bug Fixes

  • Fixed compatibility issue with openstacksdk 0.37.0 and above.

  • Fixed kubernetes driver service account creation issue resulting in zuul job to fail with: MODULE FAILURE: error: You must be logged in to the server (Unauthorized)

3.9.0

New Features

  • A diskimage can specify the full path to the diskimage-builder command with the dib-cmd configuration parameter. The --fake parameter of nodepool-builder (only used by CI) has been removed and replaced with explicit calls in testing fixtures.

  • The default python-path is now auto (instead of /usr/bin/python2). With this, Zuul 3.11.1 and greater will set the ansible_python_interpreter to auto when using Ansible >=2.8 to use automated interpreter discovery. When using earlier Ansible, it will remain the old default of /usr/bin/python2.

    This will remove the need to override python-path explicitly for Python 3-only distributions, which should be detected correctly automatically.

Upgrade Notes

  • This release should only be run against Zuul 3.11.1 or greater. Earlier Zuul releases will not convert the new default python-path of auto to /usr/bin/python2 for Ansible <2.8, leading to a configuration error. It may be possible to use earlier Zuul releases if you you are only using Ansible >= 2.8, or explicitly set python-path for every image.

3.8.0

New Features

  • The Kubernetes driver now supports optionally loading cluster admin service account information from the standard in-cluster configuration paths if Nodepool itself is running in Kubernetes. If this method is used, installation of a kube/config file in the Nodepool launcher pod is no longer required.

3.7.1

Bug Fixes

  • Fix dependency issue with openshift python client, that would prevent nodepool-launcher from starting properly.

3.7.0

New Features

  • A new driver is available to support unprivileged Openshift cluster as a resources provider to enable pod creation within a developper project.

  • Provider labels for the OpenStack driver are now able to toggle providers.[openstack].pools.labels.host-key-checking. This overrides the host-key-checking value defined by providers.[openstack].pools.host-key-checking.

  • Provider labels for the OpenStack driver are now able to select which networks to be attached to. This overrides any networks defined by providers.[openstack].pools.networks.

3.6.0

Upgrade Notes

  • The diskimage-builder stats have been reworked to be more useful. The return code and duration is now stored in nodepool.dib_image-build.<diskimage_name>.status.<rc|duration>; previously this was split for each image format. This is unnecessary and confusing since the results will always be the same, since all formats are generated from the same diskimage-builder run. An additional gauge nodepool.dib_image_build.<diskimage_name>.status.last_build is added to make it easy to show relative time of builds in dashboards.

  • The TaskManager used by the OpenStack provider has been removed. The keystoneauth1 library underneath openstacksdk has grown support for rate limiting using a FairSemaphore instead of a pool of worker threads. This should reduce the overall thread count.

  • statsd key names have changed. Because of the removal of TaskManager statsd calls are being deferred to openstacksdk. Instead of keys of the form ComputeGetServers, the openstacksdk keys are of the form compute.GET.servers. They will always start with the normalized service-type, followed by the HTTP verb, followed by a . separated list of url segments. Any service version, project-id entries in the url or .json suffixes will be removed.

3.5.0

New Features

  • The new Amazon Web Services (AWS) EC2 Driver allows launching EC2 instances as nodes.

  • A new option (build-timeout) has been added to the builder diskimage configuration to control how long the builder should wait for image builds before giving up. The default is 8 hours.

  • A new driver is available to support Openshift cluster as a resources provider to enable project and pod request.

Known Issues

  • The AWS driver does not support quota management at this time.

  • The AWS driver does not support custom image building.

3.4.0

New Features

  • A new configuration option is available under the ‘pools’ attribute of an OpenStack provider. This config value, ‘node-attributes’, can contain a dictionary of arbitrary key-value pairs and will be stored with the node data within ZooKeeper.

Upgrade Notes

  • A change to the ZooKeeper schema to support a new DELETED node state will require a total shutdown of all Nodepool launchers before restarting any of them with this version.

Bug Fixes

  • Fixes a regression of missing task statistics with OpenstackSDK versions greater than 0.19.0.

3.3.1

New Features

  • Added a new routine to the OpenStack driver cleanup resources phase that will remove any ports reported to be in the DOWN state. Ports will have to be seen as DOWN for at least three minutes before they will be removed. The number of ports removed will be reported to statsd.

Bug Fixes

  • The nodes by label and state statistic gauges are now correctly reset to zero if no node of a label and state exists.

  • Task names are now consistently normalised to CamelCase without deliminators. Some statistics sent to statsd with - or _ characters will have changed keys, for example ComputePostOs-volumes_boot is now ComputePostOsVolumesBoot.

3.3.0

New Features

  • Two new metrics are now reported after each run of the diskimage builder: nodepool.builder.dib_image_build.<diskimage_name>.<ext>.rc will be set to the last result code of the diskimage builder. This metric can be used to set up alerting for failed disk image builds. nodepool.builder.dib_image_build.<diskimage_name>.<ext>.duration will receive the time it took to build the disk image.

  • The OpenStack driver now supports configuring instance properties on boot. These properties end up in the instance metadata and will be visible to the instance after boot. Use the instance-properties dict on provider pool label to set this per label type booted.

  • The static driver now updates labels and connection related attributes in Zookeeper at startup and on config change. Changing the name of a node will be handled via the registration/deregistration flow as before.

Bug Fixes

  • Bump minimum version of openstacksdk library to 0.17.2 to correct an issue causing a crash in OpenStack provider communication threads.

3.2.0

New Features

  • A new boolean pool variable ignore-provider-quota has been added to allow the provider quota to be ignored for a pool. Instead, nodepool only checks against the configured max values for the pool and the current usage based on stored data. This may be useful in circumstances where the provider is incorrectly calculating quota.

  • The detailed nodepool list outputs the node’s pool.

  • Diskimages env-vars can be set in the secure.conf file.

Upgrade Notes

  • A new node status (ABORTED) is added to the ZooKeeper data model. It is recommended that, during your nodepool upgrade, you shut down all launcher processes before restarting any of them. Running multiple launchers with mixed support of this new node status may cause unexpected errors to be reported in the logs.

Bug Fixes

  • For pre-existing cloud images (not managed by nodepool), referencing them by ID was failing since they could not be found with this data, only by name.

3.1.0

New Features

  • Nodepool now defaults to building qcow2 diskimages instead of failing if the diskimage doesn’t specify an image format and the diskimage isn’t used by any provider. This makes it more convenient to build images without uploading them to a cloud provider.

  • Added support for specifying security-groups for the nodes in openstack driver. Pool.security-groups takes list of SGs to attach to the server.

  • The static driver now pre-registers its nodes with ZooKeeper at startup and on configuration changes. A single node may be registered multiple times, based on the value of max-parallel-jobs.

Upgrade Notes

  • Nodepool can now support multiple node labels, although the OpenStack and static node drivers do not yet support specifying multiple labels, so this is not yet a user-visible change. This does, however, require shutting down all launcher processes before restarting them. Running multiple launchers with mixed support of multi-label will cause errors, so a full shutdown is required.

Bug Fixes

  • Fixed a bug where if a request handler is paused and an exception is thrown within the handler, the handler was not properly unpaused and the request remained in the list of active handlers.

3.0.1

New Features

  • The connection port can now be configured in the provider diskimages section.

  • Added support for configuring windows static nodes. A static node can now define a connection-type. The ssh-port option has been renamed to connection-port.

Deprecation Notes

  • ssh-port in static node config is deprecated. Please update config to use connection-port instead.