Log Roles

add-fileserver

Add a remote fileserver to the inventory so that content can be uploaded in subsequent tasks or roles.

Role Variables

fileserver

Complex argument which contains the information about the remote destination as well as the authentication information needed. It is expected that this argument comes from a Secret

Example:

- secret:
    name: site_logs
      data:
        fqdn: logs.example.org
        path: /srv/static/logs
        ssh_known_hosts: |
          logs.example.org ssh-rsa ...
        ssh_username: zuul
        ssh_private_key: !encrypted/pkcs1-oaep
          - ...

fileserver.fqdn: The FQDN of the remote host.

fileserver.path: The remote path. Content will be put into a directory below this path that matches zuul.project.short_name. The full path including the project short name will be added to the hostvars of the host as zuul_fileserver_project_path.

fileserver.ssh_known_hosts: String containing known host signature for the remote host.

fileserver.ssh_private_key: Contents of the ssh private key to use.

fileserver.ssh_username Default: ansible_user: Remote ssh user name to use.

fileserver_leading_path: This is an optional variable that will be inserted between the base of the path (provided by the path variable) and the zuul.project.short_name final path element.

ara-report

If ARA is enabled, generates a report or saves a copy of the ARA database.

Role Variables

ara_report_run Default: ``true``

Whether to run this role or not. Possible values:

true (always run)
false (never run)
failure (only run when there has been a failure)

ara_database_path Default: ``{{ zuul.executor.work_root }}/.ara/ansible.sqlite``: Absolute path where the ARA database is expected on the control node. This should be where the ansible-playbook execution had ARA save the host, task and result data if you provided a custom location through ARA_DATABASE or an ansible.cfg file.

ara_report_type Default: ``html``

Possible values:

html
database

html will have ARA generate and save a statically generated HTML report inside ara_report_path.

database will only save the raw ARA sqlite database inside ara_report_path. The database can then be downloaded by users or loaded dynamically by the ara-wsgi-sqlite middleware.

See the ARA documentation for details.

ara_compress_html Default: ``true``: When report_type is ‘html’, whether to compress the ARA HTML output or not.

Tip

Make sure the web server is configured to set the required mimetypes in order to serve gzipped content properly.

ara_report_path Default: ``{{ zuul.executor.log_root }}/ara``: When report_type is ‘html’ directory where the HTML report will be generated. When report_type is ‘database’, directory where the database is saved.

ara_report_executable Default: ``ara``: Path to ara executable.

ara_report_artifact_prefix Default: None: Prefix to add to the artifact URL reported to Zuul. If you copy results into a subdirectory, add this here. Note this should have a trailing /.

ensure-output-dirs

Ensure output directories are in place and are empty.

Role Variables

zuul_output_dir Default: {{ ansible_user_dir }}/zuul-output: Base directory for collecting job output.

fetch-output

Collect output from build nodes

This role collects logs, artifacts and docs from subdirs of the zuul_output_dir on the remote nodes to equivalent directories on the executor so that later parts of the system can publish the content to appropriate permanent locations.

Note

Log content for multi-node jobs will be put into subdirectories based on remote node name. It is expected that artifacts and docs produced be inherently unique regardless of which build node they were produced on, so all artifacts and docs are pulled back to the same artifacts and docs directory.

Role Variables

zuul_output_dir Default: {{ ansible_user_dir }}/zuul-output: Base directory for collecting job output.

fetch-output-openshift

Collect output from build pods

This role can be used instead of the fetch-output role when the synchronize module doesn’t work with kubectl connection.

This role requires the origin-client oc to be installed.

Role Variables

zuul_output_dir Default: {{ ansible_user_dir }}/zuul-output: Base directory for collecting job output.

openshift_pods Default: {{ zuul.resources }}: The dictionary of pod name, pod information to copy the sources to.

zuul_log_verbose Default: false: The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.

generate-zuul-manifest

Generate a Zuul manifest file for log uploading

This generates a manifest file in preparation for uploading along with logs. The Zuul web interface can fetch this file in order to display logs from a build.

Role Variables

generate_zuul_manifest_root Default: {{ zuul.executor.log_root }}: The root directory to index.

generate_zuul_manifest_filename Default: zuul-manifest.json: The name of the manifest file.

generate_zuul_manifest_output Default: {{ zuul.executor.log_root }}/{{ generate_zuul_manifest_filename }}: The path to the output manifest file.

generate_zuul_manifest_type Default: zuul_manifest: The artifact type to return to Zuul.

generate_zuul_manifest_index_links Default: False: If True, the Zuul dashboard will link to “index.html” for directory entries; if False, it will link to the bare directory.

htmlify-logs

HTMLify text logs

This makes an HTML version of every file in the executor log directory with a .txt or .txt.gz extension. If the original was gzipped, the HTML version will be as well.

local-log-download

Add a script for users to bulk download logs locally

This adds a script for users to bulk download all logs to their local system. It queries the Zuul API for the manifest and then copies all files locally from the log server.

Role Variables

local_log_download_api: The Zuul API endpoint to use. Example: https://zuul.example.org/api/tenant/{{ zuul.tenant }}

merge-output-to-logs

Put artifacts and docs into the executor log dir

Note

This role only works in a trusted context. It is intended to be used in the post playbook of a base job.

This role moves artifacts and docs into the logs dir when zuul.change is defined so that they can be uploaded to the log server for developer preview and validation.

Artifacts and docs are left in place when zuul.change is not defined so that normal publication jobs can publish them to final locations.

publish-artifacts-to-fileserver

Publish contents of {{ zuul.executor.work_root }}/artifacts/ dir using rsync over ssh to a remote fileserver that has previously been added to the inventory by add-fileserver.

Role Variables

add-fileserver sets the following variable in the hostvars of the hosts it adds, but it is documented for reference.

zuul_fileserver_project_path: The remote path. Content will be put into a directory below this path that matches zuul.project.short_name. The full path including the project short name will be added to the hostvars of the host as zuul_fileserver_project_path.

set-zuul-log-path-fact

Sets a fact named zuul_log_path from zuul variables

The resulting log path will be based on the zuul tenant name and build uuid. The url will then be prefixed by a portion of the build uuid. This prefix allows for partitioning in object storage systems. Constructing the url in this way isn’t very human readable but produces consistent url lengths which is important for database record keeping and avoiding unexpected problems with url lengths exceeding limits.

upload-logs

Upload logs to a static webserver

This uploads logs to a static server using SSH. The server must have been previously added to the inventory; this can be done with the add-fileserver role; see that role’s documentation for a description of the site_logs secret in this example post-run playbook:

- hosts: localhost
  roles:
    - role: add-fileserver
      fileserver: "{{ site_logs }}"

- hosts: "{{ site_logs.fqdn }}"
  gather_facts: False
  roles:
    - role: upload-logs
      zuul_log_url: "http://logs.example.org"

Role Variables

zuul_log_url: Base URL where logs are to be found.

zuul_logserver_root Default: /srv/static/logs: The root path to the logs on the logserver.

zuul_log_compress Default: false: When enabled, the console logs Zuul produces will be compressed before uploading. You may need additional configuration for your web server to view these files.

zuul_log_verbose Default: false: The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

upload-logs-azure

Upload logs to Azure Storage

Before using this role, create a storage account in Azure and obtain an Access key for the account. You may create a container within the account, or allow this role to create the container (or containers) for you.

Role Variables

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition Default: false: If set to true, then the first component of the log path will be removed from the object name and added to the container name, so that logs for different changes are distributed across a large number of containers.

zuul_log_container: If partitioning is not enabled, this is the name of the container which will be used. If partitioning is enabled, then this will be used as the prefix for the container name which will be separated from the partition name by an underscore. For example, “logs_42” would be the container name for partition 42.

zuul_log_container_public Default: true: If the container is created, should it be created with global read ACLs. If the container already exists, it will not be modified.

zuul_log_path Default: Generated by the role `set-zuul-log-path-fact`: Prepend this path to the object names when uploading.

zuul_log_create_indexes Default: true: Whether to create index.html files with directory indexes.

zuul_log_connection_string: The Access key connection string for the Azure storage account.

upload-logs-failover

Upload logs to swift/s3/azure/… with a failover mechanism.

This role calls the upload-logs-[target] roles according to the list of upload targets passed as role vars. If the current upload target fails, it proceeds with the next item in its list until one of the uploads succeeded or no more targets are available.

It expects the corresponding role vars of upload-logs-[target] to be configured in the list of upload targets zuul_log_targets. An example configuration would look like the following and would ideally be passed as a Zuul secret as it may hold passwords and access keys.

zuul_log_targets:
  - target: s3
    args:
      upload_logs_s3_endpoint: https://objectstore.example.com/
      zuul_log_bucket: log-bucket
      zuul_log_aws_access_key: foo
      zuul_log_aws_secret_key: bar
  - target: swift
    args:
      zuul_log_cloud_config:
        auth:
          username: foo
          password: bar
          user_domain_name: example
          auth_url: https://openstack.example.com:5000/v3/
      zuul_log_container: log-container

Requirements and dependencies of called upload-logs-[target] roles apply accordingly, e.g. boto3 for the s3 role. Please refer to their corresponding documentation.

Role Variables

zuul_log_targets Type: list: List of upload targets

zuul_log_targets.target Type: string: Choice of “azure”, “gcs”, “s3”, “swift”, and any other platform specific “upload-logs-*” role that might be availble.

zuul_log_targets.args Type: dict: Complex argument which holds the variables required by the respective “upload-logs-*” role specified by the “target” var. Please refer to their respective documentation.

upload-logs-gcs

Upload logs to Google Cloud Storage

Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets (though it will configure CORS policies).

This role requires the google-cloud-storage Python package to be installed in the Ansible environment on the Zuul executor. It uses Google Cloud Application Default Credentials.

Role Variables

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition Default: false: If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.

zuul_log_container

This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation.

zuul_log_path Default: Generated by the role `set-zuul-log-path-fact`: Prepend this path to the object names when uploading.

zuul_log_create_indexes Default: true: Whether to create index.html files with directory indexes.

zuul_log_credentials_file

This log upload role normally uses Google Cloud Application Default Credentials, however it can also operate in a mode where it uses a credential file written by gcp-authdaemon: https://opendev.org/zuul/gcp-authdaemon

To use this mode of operation, supply a path to the credentials file previously written by gcp-authdaemon.

Also supply upload-logs-gcs.zuul_log_project.

zuul_log_project: When using upload-logs-gcs.zuul_log_credentials_file, the name of the Google Cloud project of the log container must also be supplied.

upload-logs-ibm

Upload logs to IBM Cloud Storage

Before using this role, create a cloud object storage service instance and a service credential.

You may create a bucket within the instance, or allow this role to create the bucket (or buckets) for you.

Role Variables

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition Default: false: If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.

zuul_log_bucket: If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.

zuul_log_bucket_public Default: true: If the bucket is created, this indicates whether it should be created with global read ACLs. If the bucket already exists, it will not be modified.

zuul_log_bucket_location: If the bucket is created, this storage location will be used as the location constraint.

zuul_log_path Default: Generated by the role `set-zuul-log-path-fact`: Prepend this path to the object names when uploading.

zuul_log_create_indexes Default: true: Whether to create index.html files with directory indexes.

zuul_log_api_key: The API key that was created as part of the service credential. This is required.

zuul_log_instance_id: The instance id that appears in the service credential. This is required.

zuul_log_endpoint_url: The cloud storage endpoint. This is required.

upload-logs-s3

Upload logs to S3

Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets.

This role requires the boto3 Python package to be installed in the Ansible environment on the Zuul executor.

Role Variables

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition Default: false: If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.

zuul_log_bucket

This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation.

zuul_log_bucket_public Default: true: Set to false to make logs private.

zuul_log_path Default: Generated by the role `set-zuul-log-path-fact`: Prepend this path to the object names when uploading.

zuul_log_create_indexes Default: true: Whether to create index.html files with directory indexes.

zuul_log_aws_access_key: AWS access key to use.

zuul_log_aws_secret_key: AWS secret key for the AWS access key.

upload_logs_s3_endpoint: The endpoint to use when uploading logs to an s3 compatible service. By default this will be automatically constructed by boto but should be set when working with non-aws hosted s3 service.

upload-logs-swift

Upload logs to a swift container

This uploads logs to an OpenStack Object Store (Swift) container.

Role Variables

zuul_site_upload_logs Default: true: Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_cloud_config: Complex argument which contains the cloud configuration in os-cloud-config (clouds.yaml) format. It is expected that this argument comes from a Secret.

zuul_log_partition Default: false: If set to true, then the first component of the log path will be removed from the object name and added to the container name, so that logs for different changes are distributed across a large number of containers.

zuul_log_container Default: logs

This role will create containers which do not already exist. If partitioning is not enabled, this is the name of the container which will be used. If partitioning is enabled, then this will be used as the prefix for the container name which will be separated from the partition name by an underscore. For example, “logs_42” would be the container name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation if using shared object stores that require globally unique container names. For example if using a public cloud whose Swift API is provided by Ceph.

zuul_log_container_public Default: true: If the container is created, should it be created with global read ACLs. If the container already exists, it will not be modified.

zuul_log_delete_after Default: 15552000: Number of seconds to delete objects after upload. Default is 6 months (15552000 seconds) and if set to 0 X-Delete-After will not be set.

zuul_log_path Default: Generated by the role `set-zuul-log-path-fact`: Prepend this path to the object names when uploading.

zuul_log_create_indexes Default: true: Whether to create index.html files with directory indexes. If set to false, Swift containers can be marked with a Web-Listings=true property to activate Swift’s own directory indexing.