Two new nodepool commands, nodepool export-image-data and nodepool import-image-data have been added to back up the image data in ZooKeeper to a file in case the ZooKeeper cluster is lost.
Added the option to set quota on resources on a per-tenant basis (i.e. Zuul tenants).
A new top-level config structure
tenant-resource-limitshas been added under which one can specify a number of tenants, each with
max-ramlimits. These limits are valid globally, i.e., for all providers. This differs from currently existing provider and pool quotas, which only are considered for nodes of the same provider. This feature is optional and tenant quotas are ignored for any NodeRequests that do not deliver tenant information with them. Also no quota is evaluated for tenants that have no limits configured for them.
Fixes a regression in gathering public SSH host keys on slower nodes. We now wait until SSHd on the remote node is started before gathering host keys.
Fix Kubernetes in-cluster configuration loading if no local config is present. The previous release missed a fallback case which has been corrected.
Nodepool 0.3.6 introduced an unintended behaviour change with
statsdreporting. Due to a change in the way nodepool manages OpenStack API calls, all API related statistics created during interaction with clouds are now generated by
openstacksdkand prefixed with
openstack.apiinstead of being created by nodepool and prefixed with
nodepool.provider.<cloud>as in prior versions. If you wish to revert to the prior behaviour, changes have been provided to
openstacksdkto allow setting custom prefixes via the cloud configuration file; see statsd documentation
Multi-line log messages (such as tracebacks from image builds) are now prefixed in the same manner as single-line messages.
AWS EC2, GCE, Kubernetes, Openshift, Openshift Pods, OpenStack and Static drivers now support a shell-type config. Shell-type config is intended to enable setting of cmd or powershell as shell type for Windows workers with connection-type ssh.
For Linux workers, there is a long standing ansible issue with using non-default shell type and become, so care should be taken if using it for such workers.
zuul-public-keyconfiguration attribute in the
providersAzure driver has been moved and renamed. Please move this setting to its new location at
TLS is now required for ZooKeeper connections. TLS support has been optional since version 3.13. If you have not already enabled it, we recommend enabling it before upgrading to 4.0.
Configuration value can be set from the envirnonment variables using the %(NODEPOOL_env_name) syntax.
Basic support for specifying k8s/OpenShift nodeSelectors on Pod node labels. This allows to schedule a Pod on k8s nodes with specific labels, e.g., having certain capabilities.
Support for passing environment variables to k8s and OpenShift Pod build nodes has been added.
It is not possible to set persistent env vars in containers on run time because there is no login shell available. Thus, we need to pass in any env vars during node launch. This allows to set, e.g., http_proxy variables. Environment variables can be defined on node labels as a list of dictionaries with name and value fields as per the k8s container YAML schema.
The k8s and OpenShift providers do not longer set the workingDir attribute of their container specs to /tmp.
For increased flexibility for the user, the working dir specified in the container images Dockerfile is used as the default in container nodes. Please note that this might often be the root dir (‘/’) if not specified otherwise by the respective Dockerfiles WORKDIR directive.
The dependency on kazoo has been upgraded to 2.8.0 which has an important fix for using Zookeeper over TLS.
The docker images published to
zuul/nodepool-builderare now built as multi-arch images and support arm64 in addition to amd64.
Entries in the diskimages section can now specify a parent image to inherit configuration values from. You can also specify images for configuration use only as abstract to consolidate common values.
-1will disable the removal of old build logs.
Support for encrypted connections to ZooKeeper has been added.
Before enabling, ensure that both Zuul and Nodepool software versions support encrypted connections. See the Zuul release notes, documentation, and associated helper scripts for more information.
Both Zuul and Nodepool may need to be restarted together with the new configuration.
nodepool.provider.<provider>.downPortshas been renamed to
Zookeeper hosts specified as IPv6 literals will now be configured correctly.
Support for resources in Google Compute Engine (GCE) has been added.
Add optional ebs-optimized on ec2 instances.
Add optional tags on ec2 instances and use cloud-image label as Name.
It is now possible to specify if AWS nodes shall get a
The AWS driver now supports custom
providers.[aws].pools.labels.userdatawhen launching instances.
There is a new
GET /readyendpoint that can be used as a readiness probe.
Nodepool now supports
providers.[openstack].post-upload-hookto run a user supplied script after an image has been uploaded to a cloud but before it gets used.
Fixed compatibility issue with openstacksdk 0.37.0 and above.
Fixed kubernetes driver service account creation issue resulting in zuul job to fail with: MODULE FAILURE: error: You must be logged in to the server (Unauthorized)
diskimagecan specify the full path to the diskimage-builder command with the
dib-cmdconfiguration parameter. The
nodepool-builder(only used by CI) has been removed and replaced with explicit calls in testing fixtures.
/usr/bin/python2). With this, Zuul 3.11.1 and greater will set the
autowhen using Ansible >=2.8 to use automated interpreter discovery. When using earlier Ansible, it will remain the old default of
This will remove the need to override python-path explicitly for Python 3-only distributions, which should be detected correctly automatically.
This release should only be run against Zuul 3.11.1 or greater. Earlier Zuul releases will not convert the new default
/usr/bin/python2for Ansible <2.8, leading to a configuration error. It may be possible to use earlier Zuul releases if you you are only using Ansible >= 2.8, or explicitly set
python-pathfor every image.
The Kubernetes driver now supports optionally loading cluster admin service account information from the standard in-cluster configuration paths if Nodepool itself is running in Kubernetes. If this method is used, installation of a
kube/configfile in the Nodepool launcher pod is no longer required.
Fix dependency issue with openshift python client, that would prevent nodepool-launcher from starting properly.
A new driver is available to support unprivileged Openshift cluster as a resources provider to enable pod creation within a developper project.
Provider labels for the OpenStack driver are now able to toggle
providers.[openstack].pools.labels.host-key-checking. This overrides the host-key-checking value defined by
Provider labels for the OpenStack driver are now able to select which networks to be attached to. This overrides any networks defined by
The diskimage-builder stats have been reworked to be more useful. The return code and duration is now stored in
nodepool.dib_image-build.<diskimage_name>.status.<rc|duration>; previously this was split for each image format. This is unnecessary and confusing since the results will always be the same, since all formats are generated from the same diskimage-builder run. An additional gauge
nodepool.dib_image_build.<diskimage_name>.status.last_buildis added to make it easy to show relative time of builds in dashboards.
TaskManagerused by the OpenStack provider has been removed. The
openstacksdkhas grown support for rate limiting using a
FairSemaphoreinstead of a pool of worker threads. This should reduce the overall thread count.
statsd key names have changed. Because of the removal of
TaskManagerstatsd calls are being deferred to openstacksdk. Instead of keys of the form
ComputeGetServers, the openstacksdk keys are of the form
compute.GET.servers. They will always start with the normalized
service-type, followed by the HTTP verb, followed by a
.separated list of url segments. Any service version, project-id entries in the url or
.jsonsuffixes will be removed.
The new Amazon Web Services (AWS) EC2 Driver allows launching EC2 instances as nodes.
A new option (build-timeout) has been added to the builder diskimage configuration to control how long the builder should wait for image builds before giving up. The default is 8 hours.
A new driver is available to support Openshift cluster as a resources provider to enable project and pod request.
The AWS driver does not support quota management at this time.
The AWS driver does not support custom image building.
A new configuration option is available under the ‘pools’ attribute of an OpenStack provider. This config value, ‘node-attributes’, can contain a dictionary of arbitrary key-value pairs and will be stored with the node data within ZooKeeper.
A change to the ZooKeeper schema to support a new DELETED node state will require a total shutdown of all Nodepool launchers before restarting any of them with this version.
Fixes a regression of missing task statistics with OpenstackSDK versions greater than 0.19.0.
Added a new routine to the OpenStack driver cleanup resources phase that will remove any ports reported to be in the DOWN state. Ports will have to be seen as DOWN for at least three minutes before they will be removed. The number of ports removed will be reported to statsd.
The nodes by label and state statistic gauges are now correctly reset to zero if no node of a label and state exists.
Task names are now consistently normalised to CamelCase without deliminators. Some statistics sent to statsd with
_characters will have changed keys, for example
Two new metrics are now reported after each run of the diskimage builder: nodepool.builder.dib_image_build.<diskimage_name>.<ext>.rc will be set to the last result code of the diskimage builder. This metric can be used to set up alerting for failed disk image builds. nodepool.builder.dib_image_build.<diskimage_name>.<ext>.duration will receive the time it took to build the disk image.
The OpenStack driver now supports configuring instance properties on boot. These properties end up in the instance metadata and will be visible to the instance after boot. Use the
instance-propertiesdict on provider pool label to set this per label type booted.
The static driver now updates labels and connection related attributes in Zookeeper at startup and on config change. Changing the name of a node will be handled via the registration/deregistration flow as before.
Bump minimum version of openstacksdk library to 0.17.2 to correct an issue causing a crash in OpenStack provider communication threads.
A new boolean pool variable
ignore-provider-quotahas been added to allow the provider quota to be ignored for a pool. Instead, nodepool only checks against the configured max values for the pool and the current usage based on stored data. This may be useful in circumstances where the provider is incorrectly calculating quota.
The detailed nodepool list outputs the node’s pool.
Diskimages env-vars can be set in the secure.conf file.
A new node status (ABORTED) is added to the ZooKeeper data model. It is recommended that, during your nodepool upgrade, you shut down all launcher processes before restarting any of them. Running multiple launchers with mixed support of this new node status may cause unexpected errors to be reported in the logs.
For pre-existing cloud images (not managed by nodepool), referencing them by ID was failing since they could not be found with this data, only by name.
Nodepool now defaults to building qcow2 diskimages instead of failing if the diskimage doesn’t specify an image format and the diskimage isn’t used by any provider. This makes it more convenient to build images without uploading them to a cloud provider.
Added support for specifying security-groups for the nodes in openstack driver. Pool.security-groups takes list of SGs to attach to the server.
The static driver now pre-registers its nodes with ZooKeeper at startup and on configuration changes. A single node may be registered multiple times, based on the value of max-parallel-jobs.
Nodepool can now support multiple node labels, although the OpenStack and static node drivers do not yet support specifying multiple labels, so this is not yet a user-visible change. This does, however, require shutting down all launcher processes before restarting them. Running multiple launchers with mixed support of multi-label will cause errors, so a full shutdown is required.
Fixed a bug where if a request handler is paused and an exception is thrown within the handler, the handler was not properly unpaused and the request remained in the list of active handlers.
The connection port can now be configured in the provider diskimages section.
Added support for configuring windows static nodes. A static node can now define a
ssh-portoption has been renamed to
ssh-portin static node config is deprecated. Please update config to use