Log Roles

add-fileserver

Add a remote fileserver to the inventory so that content can be uploaded in subsequent tasks or roles.

Role Variables

fileserver

Complex argument which contains the information about the remote destination as well as the authentication information needed. It is expected that this argument comes from a Secret

Example:

- secret:
    name: site_logs
      data:
        fqdn: logs.example.org
        path: /srv/static/logs
        ssh_known_hosts: |
          logs.example.org ssh-rsa ...
        ssh_username: zuul
        ssh_private_key: !encrypted/pkcs1-oaep
          - ...
fileserver.fqdn

The FQDN of the remote host.

fileserver.path

The remote path. Content will be put into a directory below this path that matches zuul.project.short_name. The full path including the project short name will be added to the hostvars of the host as zuul_fileserver_project_path.

fileserver.ssh_known_hosts

String containing known host signature for the remote host.

fileserver.ssh_private_key

Contents of the ssh private key to use.

fileserver.ssh_username
Default: ansible_user

Remote ssh user name to use.

fileserver_leading_path

This is an optional variable that will be inserted between the base of the path (provided by the path variable) and the zuul.project.short_name final path element.

ara-report

If ARA is enabled, generates a report or saves a copy of the ARA database.

Role Variables

ara_report_run
Default: ``true``

Whether to run this role or not. Possible values:

  • true (always run)

  • false (never run)

  • failure (only run when there has been a failure)

ara_database_path
Default: ``{{ zuul.executor.work_root }}/.ara/ansible.sqlite``

Absolute path where the ARA database is expected on the control node. This should be where the ansible-playbook execution had ARA save the host, task and result data if you provided a custom location through ARA_DATABASE or an ansible.cfg file.

ara_report_type
Default: ``html``

Possible values:

  • html

  • database

html will have ARA generate and save a statically generated HTML report inside ara_report_path.

database will only save the raw ARA sqlite database inside ara_report_path. The database can then be downloaded by users or loaded dynamically by the ara-wsgi-sqlite middleware.

See the ARA documentation for details.

ara_compress_html
Default: ``true``

When report_type is ‘html’, whether to compress the ARA HTML output or not.

Tip

Make sure the web server is configured to set the required mimetypes in order to serve gzipped content properly.

ara_report_path
Default: ``{{ zuul.executor.log_root }}/ara``

When report_type is ‘html’ directory where the HTML report will be generated. When report_type is ‘database’, directory where the database is saved.

ara_report_executable
Default: ``ara``

Path to ara executable.

ara_report_artifact_prefix
Default: None

Prefix to add to the artifact URL reported to Zuul. If you copy results into a subdirectory, add this here. Note this should have a trailing /.

ensure-output-dirs

Ensure output directories are in place and are empty.

Role Variables

zuul_output_dir
Default: {{ ansible_user_dir }}/zuul-output

Base directory for collecting job output.

fetch-output

Collect output from build nodes

This role collects logs, artifacts and docs from subdirs of the zuul_output_dir on the remote nodes to equivalent directories on the executor so that later parts of the system can publish the content to appropriate permanent locations.

Note

Log content for multi-node jobs will be put into subdirectories based on remote node name. It is expected that artifacts and docs produced be inherently unique regardless of which build node they were produced on, so all artifacts and docs are pulled back to the same artifacts and docs directory.

Role Variables

zuul_output_dir
Default: {{ ansible_user_dir }}/zuul-output

Base directory for collecting job output.

fetch-output-openshift

Collect output from build pods

This role can be used instead of the fetch-output role when the synchronize module doesn’t work with kubectl connection.

This role requires the origin-client oc to be installed.

Role Variables

zuul_output_dir
Default: {{ ansible_user_dir }}/zuul-output

Base directory for collecting job output.

openshift_pods
Default: {{ zuul.resources }}

The dictionary of pod name, pod information to copy the sources to.

zuul_log_verbose
Default: false

The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.

generate-zuul-manifest

Generate a Zuul manifest file for log uploading

This generates a manifest file in preparation for uploading along with logs. The Zuul web interface can fetch this file in order to display logs from a build.

Role Variables

generate_zuul_manifest_root
Default: {{ zuul.executor.log_root }}

The root directory to index.

generate_zuul_manifest_filename
Default: zuul-manifest.json

The name of the manifest file.

generate_zuul_manifest_output
Default: {{ zuul.executor.log_root }}/{{ generate_zuul_manifest_filename }}

The path to the output manifest file.

generate_zuul_manifest_type
Default: zuul_manifest

The artifact type to return to Zuul.

If True, the Zuul dashboard will link to “index.html” for directory entries; if False, it will link to the bare directory.

htmlify-logs

HTMLify text logs

This makes an HTML version of every file in the executor log directory with a .txt or .txt.gz extension. If the original was gzipped, the HTML version will be as well.

local-log-download

Add a script for users to bulk download logs locally

This adds a script for users to bulk download all logs to their local system. It queries the Zuul API for the manifest and then copies all files locally from the log server.

Role Variables

local_log_download_api

The Zuul API endpoint to use. Example: https://zuul.example.org/api/tenant/{{ zuul.tenant }}

merge-output-to-logs

Put artifacts and docs into the executor log dir

Note

This role only works in a trusted context. It is intended to be used in the post playbook of a base job.

This role moves artifacts and docs into the logs dir when zuul.change is defined so that they can be uploaded to the log server for developer preview and validation.

Artifacts and docs are left in place when zuul.change is not defined so that normal publication jobs can publish them to final locations.

publish-artifacts-to-fileserver

Publish contents of {{ zuul.executor.work_root }}/artifacts/ dir using rsync over ssh to a remote fileserver that has previously been added to the inventory by add-fileserver.

Role Variables

add-fileserver sets the following variable in the hostvars of the hosts it adds, but it is documented for reference.

zuul_fileserver_project_path

The remote path. Content will be put into a directory below this path that matches zuul.project.short_name. The full path including the project short name will be added to the hostvars of the host as zuul_fileserver_project_path.

set-zuul-log-path-fact

Sets a fact named zuul_log_path from zuul variables

Role Variables

zuul_log_path_shard_build
Default: False
Type: bool

Flag to specify whether or not paths that include a three character prefix based on the build uuid should prefix the log path. This is particularly useful for object storage systems where we want to spread out the number of files per container.

upload-logs

Upload logs to a static webserver

This uploads logs to a static server using SSH. The server must have been previously added to the inventory; this can be done with the add-fileserver role; see that role’s documentation for a description of the site_logs secret in this example post-run playbook:

- hosts: localhost
  roles:
    - role: add-fileserver
      fileserver: "{{ site_logs }}"

- hosts: "{{ site_logs.fqdn }}"
  gather_facts: False
  roles:
    - role: upload-logs
      zuul_log_url: "http://logs.example.org"

Role Variables

zuul_log_url

Base URL where logs are to be found.

zuul_logserver_root
Default: /srv/static/logs

The root path to the logs on the logserver.

zuul_log_compress
Default: false

When enabled, the console logs Zuul produces will be compressed before uploading. You may need additional configuration for your web server to view these files.

zuul_log_verbose
Default: false

The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.

zuul_site_upload_logs
Default: true

Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_path_shard_build
Default: False

This var is consumed by set-zuul-log-path-fact which upload-logs calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.

More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.

upload-logs-azure

Upload logs to Azure Storage

Before using this role, create a storage account in Azure and obtain an Access key for the account. You may create a container within the account, or allow this role to create the container (or containers) for you.

Role Variables

zuul_site_upload_logs
Default: true

Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition
Default: false

If set to true, then the first component of the log path will be removed from the object name and added to the container name, so that logs for different changes are distributed across a large number of containers.

zuul_log_container

If partitioning is not enabled, this is the name of the container which will be used. If partitioning is enabled, then this will be used as the prefix for the container name which will be separated from the partition name by an underscore. For example, “logs_42” would be the container name for partition 42.

zuul_log_container_public
Default: true

If the container is created, should it be created with global read ACLs. If the container already exists, it will not be modified.

zuul_log_path
Default: Generated by the role `set-zuul-log-path-fact`

Prepend this path to the object names when uploading.

zuul_log_create_indexes
Default: true

Whether to create index.html files with directory indexes.

zuul_log_path_shard_build
Default: false

This var is consumed by set-zuul-log-path-fact which upload-logs-azure calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.

More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.

zuul_log_connection_string

The Access key connection string for the Azure storage account.

upload-logs-gcs

Upload logs to Google Cloud Storage

Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets (though it will configure CORS policies).

This role requires the google-cloud-storage Python package to be installed in the Ansible environment on the Zuul executor. It uses Google Cloud Application Default Credentials.

Role Variables

zuul_site_upload_logs
Default: true

Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition
Default: false

If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.

zuul_log_container

This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation.

zuul_log_path
Default: Generated by the role `set-zuul-log-path-fact`

Prepend this path to the object names when uploading.

zuul_log_create_indexes
Default: true

Whether to create index.html files with directory indexes.

zuul_log_path_shard_build
Default: false

This var is consumed by set-zuul-log-path-fact which upload-logs-gcs calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.

More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.

zuul_log_credentials_file

This log upload role normally uses Google Cloud Application Default Credentials, however it can also operate in a mode where it uses a credential file written by gcp-authdaemon: https://opendev.org/zuul/gcp-authdaemon

To use this mode of operation, supply a path to the credentials file previously written by gcp-authdaemon.

Also supply upload-logs-gcs.zuul_log_project.

zuul_log_project

When using upload-logs-gcs.zuul_log_credentials_file, the name of the Google Cloud project of the log container must also be supplied.

upload-logs-s3

Upload logs to S3

Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets.

This role requires the boto3 Python package to be installed in the Ansible environment on the Zuul executor.

Role Variables

zuul_site_upload_logs
Default: true

Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_partition
Default: false

If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.

zuul_log_bucket

This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation.

zuul_log_bucket_public
Default: true

Set to false to make logs private.

zuul_log_path
Default: Generated by the role `set-zuul-log-path-fact`

Prepend this path to the object names when uploading.

zuul_log_create_indexes
Default: true

Whether to create index.html files with directory indexes.

zuul_log_path_shard_build
Default: false

This var is consumed by set-zuul-log-path-fact which upload-logs-s3 calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.

More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.

zuul_log_aws_access_key

AWS access key to use.

zuul_log_aws_secret_key

AWS secret key for the AWS access key.

upload_logs_s3_endpoint

The endpoint to use when uploading logs to an s3 compatible service. By default this will be automatically constructed by boto but should be set when working with non-aws hosted s3 service.

upload-logs-swift

Upload logs to a swift container

This uploads logs to an OpenStack Object Store (Swift) container.

Role Variables

zuul_site_upload_logs
Default: true

Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.

Note

Intended to be set by admins via site-variables.

zuul_log_cloud_config

Complex argument which contains the cloud configuration in os-cloud-config (clouds.yaml) format. It is expected that this argument comes from a Secret.

zuul_log_partition
Default: false

If set to true, then the first component of the log path will be removed from the object name and added to the container name, so that logs for different changes are distributed across a large number of containers.

zuul_log_container
Default: logs

This role will create containers which do not already exist. If partitioning is not enabled, this is the name of the container which will be used. If partitioning is enabled, then this will be used as the prefix for the container name which will be separated from the partition name by an underscore. For example, “logs_42” would be the container name for partition 42.

Note that you will want to set this to a value that uniquely identifies your Zuul installation if using shared object stores that require globally unique container names. For example if using a public cloud whose Swift API is provided by Ceph.

zuul_log_container_public
Default: true

If the container is created, should it be created with global read ACLs. If the container already exists, it will not be modified.

zuul_log_delete_after
Default: 15552000

Number of seconds to delete objects after upload. Default is 6 months (15552000 seconds) and if set to 0 X-Delete-After will not be set.

zuul_log_path
Default: Generated by the role `set-zuul-log-path-fact`

Prepend this path to the object names when uploading.

zuul_log_create_indexes
Default: true

Whether to create index.html files with directory indexes. If set to false, Swift containers can be marked with a Web-Listings=true property to activate Swift’s own directory indexing.

zuul_log_path_shard_build
Default: False

This var is consumed by set-zuul-log-path-fact which upload-logs-swift calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.

More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.