IBM VPC Driver

This driver is for use with the Virtual Private Cloud (VPC) service of IBM Cloud.

Preparation

Note the following caveats:

IBM Cloud VPC does not support IPv6 addressing.
IBM Cloud VPC does not supply quota information via its API (but if you know the values, you can supply them in the Nodepool configuration).
Per-instance floating IP addresses are required for in-bound connections from the Internet (but are not required for outbound-only access). As of the time of this writing, floating IP addresses are billed monthly and are not pro-rated. Because Nodepool allocates and deallocates a floating IP address for each instance, this means that every instance will incur at least a full-month’s charge for floating IP use even if it is used only briefly.
IBM Cloud does not provide a facility for atomically associating metadata with instances, which Nodepool generally relies upon for detecting and cleaning up leaked resources. To approximate this functionality, the ibmvpc driver will create all instances and associated resources within resource groups that it will expect to control. These groups begin with either the string np_ or npimages_. Do not create any resource groups matching these names, and do not create any resources within these groups. To do so runs the risk of Nodepool mistaking your resource for one that it controls and automatically deleting it.

Before using this driver, attend to the following pre-requisites:

Authentication

Several authentication methods are available. The one which will work in the widest circumstances is using an API key. The console can be used to create an API key.

The API key (or other authentication information) may be provided to Nodepool either through environment variables, or stored in a file. If a file is used, the content is exactly the same as the environment variable definition. To use an API key, the content would be:

VPC_AUTH_TYPE=iam
VPC_APIKEY=api-key-goes-here

Either set these environment variables when running the Nodepool process, or write them to a file and specify the path with the providers.[ibmvpc].credentials-file provider attribute.

Note

Several other authentication types are available; their configuration is outside the scope of this document. While nodepool-launcher should work with any auth type supported by the IBM Cloud Python SDK, nodepool-builder is only compatible with the iam auth type.

Image Building

Custom images in IBM cloud require the use of Cloud Object Storage which is not integrated with the VPC API. To use nodepool-builder, the following additional steps are required:

Create a Cloud Object Storage instance and grant access to the VPC service.

Create a bucket within the Cloud Object Storage instance.

Select an endpoint for use with Nodepool.

Note the name of the Cloud Object Storage instance, the endpoint, and the name of the bucket you created for use when configuring Nodepool.

Configuration

Selecting the ibmvpc driver adds the following options to the providers section of the configuration.

providers.[ibmvpc] Type: list

A provider’s resources are partitioned into groups called pools, and within a pool, the node types which are to be made available are listed.

Note

For documentation purposes the option names are prefixed providers.[ibmvpc] to disambiguate from other drivers, but [ibmvpc] is not required in the configuration (e.g. below providers.[ibmvpc].pools refers to the pools key in the providers section when the ibmvpc driver is selected).

Example:

providers:
  - name: ibmvpc-us-south
    driver: ibmvpc
    credentials-file: /path/to/creds.env
    vpc: nodepool
    region: us-south
    subnet: sn-nodepool
    cloud-images:
      - name: bionic
        username: zuul
        keys:
          - zuul-ssh-key
        image-name: ibm-ubuntu-20-04-3-minimal-amd64-1
    pools:
      - name: main
        zone: us-south-1
        max-servers: 10
        host-key-checking: false
        labels:
          - name: bionic
            cloud-image: bionic
            profile: cx2-2x4

providers.[ibmvpc].name (required): A unique name for this provider configuration. The name may not contain underscores.

providers.[ibmvpc].vpc (required): The name of the VPC for this provider.

providers.[ibmvpc].region (required): Name of the IBM Cloud region to interact with (e.g., us-south).

providers.[ibmvpc].zone: Name of the IBM Cloud region zone to interact with (e.g., us-south-1). This field may be omitted if providers.[ibmvpc].pools.zone is supplied.

providers.[ibmvpc].credentials-file: Path to the file containing the authentication information. Required unless the information is supplied via environment variables.

providers.[ibmvpc].subnet: The name of the network upon which to create VMs. This field may be omitted if providers.[ibmvpc].pools.subnet is supplied.

providers.[ibmvpc].public-ipv4 Default: false Type: bool: Whether to attach a public IPv4 address to instances.

Warning

As of the time of this writing, floating IP addresses are billed monthly and are not pro-rated. Enabling this option can lead to high usage charges.

providers.[ibmvpc].use-internal-ip Default: false Type: bool: If a public IP is attached but Nodepool should prefer the private IP, set this to true.

providers.[ibmvpc].host-key-checking Default: true Type: bool: Whether to validate SSH host keys. When true, this helps ensure that nodes are ready to receive SSH connections before they are supplied to the requestor. When set to false, nodepool-launcher will not attempt to ssh-keyscan nodes after they are booted. Disable this if nodepool-launcher and the nodes it launches are on different networks, where the launcher is unable to reach the nodes directly, or when using Nodepool with non-SSH node platforms. The default value is true.

providers.[ibmvpc].rate Default: 1.0 Type: float seconds: In seconds, amount to wait between operations on the provider.

providers.[ibmvpc].boot-timeout Default: 120 Type: int seconds: Once an instance is active, how long to try connecting to the image via SSH. If the timeout is exceeded, the node launch is aborted and the instance deleted.

providers.[ibmvpc].launch-timeout Default: 3600 Type: int seconds: The time to wait from issuing the command to create a new instance until that instance is reported as “active”. If the timeout is exceeded, the node launch is aborted and the instance deleted.

providers.[ibmvpc].launch-retries Default: 3 Type: int: The number of times to retry launching a server before considering the request failed.

providers.[ibmvpc].post-upload-hook Default: None Type: string

Filename of an optional script that can be called after an image has been uploaded to a provider but before it is taken into use. This is useful to perform last minute validation tests before an image is really used for build nodes. The script will be called as follows:

<SCRIPT> <PROVIDER> <EXTERNAL_IMAGE_ID> <LOCAL_IMAGE_FILENAME>

If the script returns with result code 0 it is treated as successful otherwise it is treated as failed and the image gets deleted.

providers.[ibmvpc].object-storage

In order to use nodepool-builder, the information in this attribute must be supplied. The builder will upload images to the specified bucket during the image creation process, and immediately remove them. Long-term storage is not required.

providers.[ibmvpc].object-storage.instance-name (required): The name of the Cloud Object Storage instance.

providers.[ibmvpc].object-storage.endpoint: The endpoint URL that Nodepool should use.

providers.[ibmvpc].object-storage.bucket-name: The bucket in which to store image files.

providers.[ibmvpc].quota

The IBM Cloud VPC API does not provide quota information, so Nodepool is unable to estimate how much quota is available before creating instances. This may lead to a high failure rate.

If you know the quota values, you may supply them here.

providers.[ibmvpc].quota.instances Type: int: The number of instances available in this region.

providers.[ibmvpc].quota.cores Type: int: The number of vCPU cores available in this region.

providers.[ibmvpc].quota.ram Type: int: The amount of RAM available in this region (in mebibytes).

providers.[ibmvpc].cloud-images Type: list

Each entry in this section must refer to an entry in the labels section.

cloud-images:
  - name: bionic
    username: zuul
    keys:
      - zuul-ssh-key
    image-name: ibm-ubuntu-20-04-3-minimal-amd64-1
  - name: stretch
    username: zuul
    keys:
      - zuul-ssh-key
    image-filter:
      operating-system:
        family: "Debian GNU/Linux"
        version: "9.x Stretch/Stable - Minimal Install"

Each entry is a dictionary with the following keys

providers.[ibmvpc].cloud-images.name (required) Type: string: Identifier to refer this cloud-image from labels section. Since this name appears elsewhere in the nodepool configuration file, you may want to use your own descriptive name here.

providers.[ibmvpc].cloud-images.username (required) Type: str: The username that should be used when connecting to the node.

providers.[ibmvpc].cloud-images.keys Type: list: A list of SSH keys (specified by name) to install on the node.

providers.[ibmvpc].cloud-images.connection-type Type: string: The connection type that a consumer should use when connecting to the node. For most diskimages this is not necessary. However when creating Windows images this could be winrm to enable access via ansible.

providers.[ibmvpc].cloud-images.connection-port Default: 22 / 5986 Type: int: The port that a consumer should use when connecting to the node. For most diskimages this is not necessary. This defaults to 22 for ssh and 5986 for winrm.

providers.[ibmvpc].cloud-images.python-path Default: auto Type: str: The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter on Ansible >=2.8, and default to /usr/bin/python2 for earlier versions.

providers.[ibmvpc].cloud-images.shell-type Type: str

The shell type of the node’s default shell executable. Used by Zuul to set ansible_shell_type. This setting should only be used

For a windows image with the experimental connection-type ssh in which case cmd or powershell should be set and reflect the node’s DefaultShell configuration.
If the default shell is not Bourne compatible (sh), but instead e.g. csh or fish, and the user is aware that there is a long-standing issue with ansible_shell_type in combination with become

providers.[ibmvpc].cloud-images.image-filter Type: dict

Specifies a private image to use via filters. Either this field, providers.[ibmvpc].cloud-images.image-name, providers.[ibmvpc].cloud-images.image-href, or providers.[ibmvpc].cloud-images.image-id must be provided.

If a filter is provided, Nodepool will list all of the images in the provider and reduce the list using the supplied filter. All items specified in the filter must match in order for an image to match. If more than one image matches, the images are sorted by creation time and the last one matches.

The following filters are available:

providers.[ibmvpc].cloud-images.image-filter.operating-system

This is a dictionary with any of the following keys:

providers.[ibmvpc].cloud-images.image-filter.operating-system.architecture Type: str: The architecture (e.g., amd64).

providers.[ibmvpc].cloud-images.image-filter.operating-system.dedicated-host-only Type: bool: Whether the image requires a dedicated host.

providers.[ibmvpc].cloud-images.image-filter.operating-system.display-name Type: str: The display name (e.g., Debian GNU/Linux 9.x Stretch/Stable - Minimal Install (amd64))

providers.[ibmvpc].cloud-images.image-filter.operating-system.family Type: str: The OS family (e.g., Debian GNU/Linux).

providers.[ibmvpc].cloud-images.image-filter.operating-system.href Type: str: The URL for the operating system record in IBM Cloud VPC (e.g., https://us-south.iaas.cloud.ibm.com/v1/operating_systems/debian-9-amd64).

providers.[ibmvpc].cloud-images.image-filter.operating-system.name Type: str: The name of the operating system record in IBM Cloud VPC (e.g., debian-9-amd64).

providers.[ibmvpc].cloud-images.image-filter.operating-system.vendor Type: str: The vendor (e.g., Debian).

providers.[ibmvpc].cloud-images.image-filter.operating-system.version Type: str: The version (e.g., 9.x Stretch/Stable - Minimal Install).

providers.[ibmvpc].cloud-images.image-filter.owner-type Type: str: Whether the image is provided by the cloud (provider) or the user (user).

providers.[ibmvpc].cloud-images.image-filter.status Type: str: The status of the image (e.g., available).

providers.[ibmvpc].cloud-images.image-id Type: str: Specifies a private image to use by ID. Either this field, providers.[ibmvpc].cloud-images.image-name, providers.[ibmvpc].cloud-images.image-href, or providers.[ibmvpc].cloud-images.image-filter must be provided.

providers.[ibmvpc].cloud-images.image-href Type: dict: Specifies a public image to use by href. Either this field, providers.[ibmvpc].cloud-images.image-name, providers.[ibmvpc].cloud-images.image-id, or providers.[ibmvpc].cloud-images.image-filter must be provided.

providers.[ibmvpc].cloud-images.image-name Type: dict: Specifies a public image to use by name. Either this field, providers.[ibmvpc].cloud-images.image-href, providers.[ibmvpc].cloud-images.image-id, or providers.[ibmvpc].cloud-images.image-filter must be provided.

providers.[ibmvpc].diskimages Type: list

Each entry in a provider’s diskimages section must correspond to an entry in diskimages. Such an entry indicates that the corresponding diskimage should be uploaded for use in this provider. Additionally, any nodes that are created using the uploaded image will have the associated attributes (such as flavor or metadata).

If an image is removed from this section, any previously uploaded images will be deleted from the provider.

Each entry is a dictionary with the following keys

providers.[ibmvpc].diskimages.name (required) Type: str: Identifier to refer this image from providers.[ibmvpc].pools.labels and diskimages sections.

providers.[ibmvpc].diskimages.operating-system (required) Type: str: The name of the IBM Cloud VPC operating-system record for this image. IBM Cloud VPC requires that custom images correspond with one of the pre-defined operating systems. If your exact OS isn’t available, select the closest technology match. E.g., debian-9-amd64.

providers.[ibmvpc].diskimages.pause Default: False Type: bool: When set to True, nodepool-builder will not upload the image to the provider.

providers.[ibmvpc].diskimages.key Type: str: The SSH public key that should be installed on the node.

providers.[ibmvpc].diskimages.connection-type Type: string: The connection type that a consumer should use when connecting to the node. For most diskimages this is not necessary. However when creating Windows images this could be winrm to enable access via ansible.

providers.[ibmvpc].diskimages.connection-port Default: 22 / 5986 Type: int: The port that a consumer should use when connecting to the node. For most diskimages this is not necessary. This defaults to 22 for ssh and 5986 for winrm.

providers.[ibmvpc].diskimages.python-path Default: auto Type: str: The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter on Ansible >=2.8, and default to /usr/bin/python2 for earlier versions.

providers.[ibmvpc].pools Type: list

A pool defines a group of resources from a provider. Each pool has a maximum number of nodes which can be launched from it, along with a number of cloud-related attributes used when launching nodes.

providers.[ibmvpc].pools.name (required): A unique name within the provider for this pool of resources. The name may not contain underscores.

providers.[ibmvpc].pools.priority Default: 100 Type: int

The priority of this provider pool (a lesser number is a higher priority). Nodepool launchers will yield requests to other provider pools with a higher priority as long as they are not paused. This means that in general, higher priority pools will reach quota first before lower priority pools begin to be used.

This setting may be specified at the provider level in order to apply to all pools within that provider, or it can be overridden here for a specific pool.

providers.[ibmvpc].pools.zone: Name of the IBM Cloud region zone to interact with (e.g., us-south-1). This field may be omitted if providers.[ibmvpc].zone is supplied. If both are supplied, this will override.

providers.[ibmvpc].pools.subnet: The name of the network upon which to create VMs. This field may be omitted if providers.[ibmvpc].subnet is supplied. If both are supplied, this will override.

providers.[ibmvpc].pools.node-attributes Type: dict: A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

providers.[ibmvpc].pools.public-ipv4 Default: false Type: bool: Whether to attach a public IPv4 address to instances.

Warning

As of the time of this writing, floating IP addresses are billed monthly and are not pro-rated. Enabling this option can lead to high usage charges.

providers.[ibmvpc].pools.use-internal-ip Default: false Type: bool: If a public IP is attached but Nodepool should prefer the private IP, set this to true.

providers.[ibmvpc].pools.host-key-checking Default: true Type: bool: Whether to validate SSH host keys. When true, this helps ensure that nodes are ready to receive SSH connections before they are supplied to the requestor. When set to false, nodepool-launcher will not attempt to ssh-keyscan nodes after they are booted. Disable this if nodepool-launcher and the nodes it launches are on different networks, where the launcher is unable to reach the nodes directly, or when using Nodepool with non-SSH node platforms. The default value is true.

providers.[ibmvpc].pools.labels Type: list

Each entry in a pool’s labels section indicates that the corresponding label is available for use in this pool. When creating nodes for a label, the flavor-related attributes in that label’s section will be used.

labels:
  - name: bionic
    cloud-image: bionic
    profile: cx2-2x4

Each entry is a dictionary with the following keys:

providers.[ibmvpc].pools.labels.name (required) Type: str: Identifier for this label.

providers.[ibmvpc].pools.labels.cloud-image (required) Type: str: Refers to the name of an externally managed image in the cloud that already exists on the provider. The value of cloud-image should match the name of a previously configured entry from the cloud-images section of the provider.

providers.[ibmvpc].pools.labels.diskimage (required) Type: str: Refers to provider’s diskimages, see providers.[ibmvpc].diskimages. Mutually exclusive with providers.[ibmvpc].pools.labels.cloud-image

providers.[ibmvpc].pools.labels.profile (required): The name of the IBM Cloud VPC hardware profile for the instance. E.g., cx2-2x4.

providers.[ibmvpc].pools.labels.user-data Default: None Type: str: The user data value for newly created VMs.