Log Roles¶
-
add-fileserver
¶ Add a remote fileserver to the inventory so that content can be uploaded in subsequent tasks or roles.
Role Variables
-
fileserver
¶ Complex argument which contains the information about the remote destination as well as the authentication information needed. It is expected that this argument comes from a Secret
Example:
- secret: name: site_logs data: fqdn: logs.example.org path: /srv/static/logs ssh_known_hosts: | logs.example.org ssh-rsa ... ssh_username: zuul ssh_private_key: !encrypted/pkcs1-oaep - ...
-
fileserver.
fqdn
¶ The FQDN of the remote host.
-
fileserver.
path
¶ The remote path. Content will be put into a directory below this path that matches
zuul.project.short_name
. The full path including the project short name will be added to the hostvars of the host aszuul_fileserver_project_path
.
-
fileserver.
ssh_known_hosts
¶ String containing known host signature for the remote host.
-
fileserver.
ssh_private_key
¶ Contents of the ssh private key to use.
-
fileserver.
ssh_username
¶
Default:ansible_user
Remote ssh user name to use.
-
-
fileserver_leading_path
¶ This is an optional variable that will be inserted between the base of the path (provided by the path variable) and the
zuul.project.short_name
final path element.
-
-
ara-report
¶ If ARA is enabled, generates a report or saves a copy of the ARA database.
Role Variables
-
ara_report_run
¶
Default:``true``
Whether to run this role or not. Possible values:
true
(always run)false
(never run)failure
(only run when there has been a failure)
-
ara_database_path
¶
Default:``{{ zuul.executor.work_root }}/.ara/ansible.sqlite``
Absolute path where the ARA database is expected on the control node. This should be where the ansible-playbook execution had ARA save the host, task and result data if you provided a custom location through
ARA_DATABASE
or anansible.cfg
file.
-
ara_report_type
¶
Default:``html``
Possible values:
html
database
html
will have ARA generate and save a statically generated HTML report insideara_report_path
.database
will only save the raw ARA sqlite database insideara_report_path
. The database can then be downloaded by users or loaded dynamically by theara-wsgi-sqlite
middleware.See the ARA documentation for details.
-
ara_compress_html
¶
Default:``true``
When report_type is ‘html’, whether to compress the ARA HTML output or not.
Tip
Make sure the web server is configured to set the required mimetypes in order to serve gzipped content properly.
-
ara_report_path
¶
Default:``{{ zuul.executor.log_root }}/ara``
When report_type is ‘html’ directory where the HTML report will be generated. When report_type is ‘database’, directory where the database is saved.
-
ara_report_executable
¶
Default:``ara``
Path to ara executable.
-
ara_report_artifact_prefix
¶
Default:None
Prefix to add to the artifact URL reported to Zuul. If you copy results into a subdirectory, add this here. Note this should have a trailing
/
.
-
-
ensure-output-dirs
¶ Ensure output directories are in place and are empty.
Role Variables
-
zuul_output_dir
¶
Default:{{ ansible_user_dir }}/zuul-output
Base directory for collecting job output.
-
-
fetch-output
¶ Collect output from build nodes
This role collects logs, artifacts and docs from subdirs of the
zuul_output_dir
on the remote nodes to equivalent directories on the executor so that later parts of the system can publish the content to appropriate permanent locations.Note
Log content for multi-node jobs will be put into subdirectories based on remote node name. It is expected that artifacts and docs produced be inherently unique regardless of which build node they were produced on, so all artifacts and docs are pulled back to the same artifacts and docs directory.
Role Variables
-
zuul_output_dir
¶
Default:{{ ansible_user_dir }}/zuul-output
Base directory for collecting job output.
-
-
fetch-output-openshift
¶ Collect output from build pods
This role can be used instead of the fetch-output role when the synchronize module doesn’t work with kubectl connection.
This role requires the origin-client oc to be installed.
Role Variables
-
zuul_output_dir
¶
Default:{{ ansible_user_dir }}/zuul-output
Base directory for collecting job output.
-
openshift_pods
¶
Default:{{ zuul.resources }}
The dictionary of pod name, pod information to copy the sources to.
-
zuul_log_verbose
¶
Default:false
The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.
-
-
generate-zuul-manifest
¶ Generate a Zuul manifest file for log uploading
This generates a manifest file in preparation for uploading along with logs. The Zuul web interface can fetch this file in order to display logs from a build.
Role Variables
-
generate_zuul_manifest_root
¶
Default:{{ zuul.executor.log_root }}
The root directory to index.
-
generate_zuul_manifest_filename
¶
Default:zuul-manifest.json
The name of the manifest file.
-
generate_zuul_manifest_output
¶
Default:{{ zuul.executor.log_root }}/{{ generate_zuul_manifest_filename }}
The path to the output manifest file.
-
generate_zuul_manifest_type
¶
Default:zuul_manifest
The artifact type to return to Zuul.
-
generate_zuul_manifest_index_links
¶
Default:False
If True, the Zuul dashboard will link to “index.html” for directory entries; if False, it will link to the bare directory.
-
-
htmlify-logs
¶ HTMLify text logs
This makes an HTML version of every file in the executor log directory with a
.txt
or.txt.gz
extension. If the original was gzipped, the HTML version will be as well.
-
local-log-download
¶ Add a script for users to bulk download logs locally
This adds a script for users to bulk download all logs to their local system. It queries the Zuul API for the manifest and then copies all files locally from the log server.
Role Variables
-
local_log_download_api
¶ The Zuul API endpoint to use. Example:
https://zuul.example.org/api/tenant/{{ zuul.tenant }}
-
-
merge-output-to-logs
¶ Put artifacts and docs into the executor log dir
Note
This role only works in a trusted context. It is intended to be used in the post playbook of a base job.
This role moves artifacts and docs into the logs dir when
zuul.change
is defined so that they can be uploaded to the log server for developer preview and validation.Artifacts and docs are left in place when
zuul.change
is not defined so that normal publication jobs can publish them to final locations.
-
publish-artifacts-to-fileserver
¶ Publish contents of
{{ zuul.executor.work_root }}/artifacts/
dir using rsync over ssh to a remote fileserver that has previously been added to the inventory by add-fileserver.Role Variables
add-fileserver sets the following variable in the hostvars of the hosts it adds, but it is documented for reference.
-
zuul_fileserver_project_path
¶ The remote path. Content will be put into a directory below this path that matches
zuul.project.short_name
. The full path including the project short name will be added to the hostvars of the host aszuul_fileserver_project_path
.
-
-
set-zuul-log-path-fact
¶ Sets a fact named
zuul_log_path
from zuul variablesRole Variables
-
zuul_log_path_shard_build
¶
Default:False
Type: bool Flag to specify whether or not paths that include a three character prefix based on the build uuid should prefix the log path. This is particularly useful for object storage systems where we want to spread out the number of files per container.
-
-
upload-logs
¶ Upload logs to a static webserver
This uploads logs to a static server using SSH. The server must have been previously added to the inventory; this can be done with the add-fileserver role; see that role’s documentation for a description of the site_logs secret in this example post-run playbook:
- hosts: localhost roles: - role: add-fileserver fileserver: "{{ site_logs }}" - hosts: "{{ site_logs.fqdn }}" gather_facts: False roles: - role: upload-logs zuul_log_url: "http://logs.example.org"
Role Variables
-
zuul_log_url
¶ Base URL where logs are to be found.
-
zuul_logserver_root
¶
Default:/srv/static/logs
The root path to the logs on the logserver.
-
zuul_log_compress
¶
Default:false
When enabled, the console logs Zuul produces will be compressed before uploading. You may need additional configuration for your web server to view these files.
-
zuul_log_verbose
¶
Default:false
The synchronize task in this role outputs a lot of information. By default, no_log is set to avoid overwhelming a reader of the logs. Set this to true to disable that behavior if it becomes necessary to debug this role.
-
zuul_site_upload_logs
¶
Default:true
Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.
Note
Intended to be set by admins via site-variables.
-
zuul_log_path_shard_build
¶
Default:False
This var is consumed by set-zuul-log-path-fact which upload-logs calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.
More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.
-
-
upload-logs-gcs
¶ Upload logs to Google Cloud Storage
Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets (though it will configure CORS policies).
This role requires the
google-cloud-storage
Python package to be installed in the Ansible environment on the Zuul executor. It uses Google Cloud Application Default Credentials.Role Variables
-
zuul_site_upload_logs
¶
Default:true
Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.
Note
Intended to be set by admins via site-variables.
-
zuul_log_partition
¶
Default:false
If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.
-
zuul_log_container
¶ This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.
Note that you will want to set this to a value that uniquely identifies your Zuul installation.
-
zuul_log_path
¶
Default:Generated by the role `set-zuul-log-path-fact`
Prepend this path to the object names when uploading.
-
zuul_log_create_indexes
¶
Default:true
Whether to create index.html files with directory indexes.
-
zuul_log_path_shard_build
¶
Default:false
This var is consumed by set-zuul-log-path-fact which upload-logs-gcs calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.
More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.
-
zuul_log_credentials_file
¶ This log upload role normally uses Google Cloud Application Default Credentials, however it can also operate in a mode where it uses a credential file written by gcp-authdaemon: https://opendev.org/zuul/gcp-authdaemon
To use this mode of operation, supply a path to the credentials file previously written by gcp-authdaemon.
Also supply upload-logs-gcs.zuul_log_project.
-
zuul_log_project
¶ When using upload-logs-gcs.zuul_log_credentials_file, the name of the Google Cloud project of the log container must also be supplied.
-
-
upload-logs-s3
¶ Upload logs to S3
Before using this role, create at least one bucket and set up appropriate access controls or lifecycle events. This role will not automatically create buckets.
This role requires the
boto3
Python package to be installed in the Ansible environment on the Zuul executor.Role Variables
-
zuul_site_upload_logs
¶
Default:true
Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.
Note
Intended to be set by admins via site-variables.
-
zuul_log_partition
¶
Default:false
If set to true, then the first component of the log path will be removed from the object name and added to the bucket name, so that logs for different changes are distributed across a large number of buckets.
-
zuul_log_bucket
¶ This role will not create buckets which do not already exist. If partitioning is not enabled, this is the name of the bucket which will be used. If partitioning is enabled, then this will be used as the prefix for the bucket name which will be separated from the partition name by an underscore. For example, “logs_42” would be the bucket name for partition 42.
Note that you will want to set this to a value that uniquely identifies your Zuul installation.
-
zuul_log_bucket_public
¶
Default:true
Set to false to make logs private.
-
zuul_log_path
¶
Default:Generated by the role `set-zuul-log-path-fact`
Prepend this path to the object names when uploading.
-
zuul_log_create_indexes
¶
Default:true
Whether to create index.html files with directory indexes.
-
zuul_log_path_shard_build
¶
Default:false
This var is consumed by set-zuul-log-path-fact which upload-logs-s3 calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.
More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.
-
zuul_log_aws_access_key
¶ AWS access key to use.
-
zuul_log_aws_secret_key
¶ AWS secret key for the AWS access key.
-
upload_logs_s3_endpoint
¶ The endpoint to use when uploading logs to an s3 compatible service. By default this will be automatically constructed by boto but should be set when working with non-aws hosted s3 service.
-
-
upload-logs-swift
¶ Upload logs to a swift container
This uploads logs to an OpenStack Object Store (Swift) container.
Role Variables
-
zuul_site_upload_logs
¶
Default:true
Controls when logs are uploaded. true, the default, means always upload logs. false means never upload logs. ‘failure’ means to only upload logs when the job has failed.
Note
Intended to be set by admins via site-variables.
-
zuul_log_cloud_config
¶ Complex argument which contains the cloud configuration in os-cloud-config (clouds.yaml) format. It is expected that this argument comes from a Secret.
-
zuul_log_partition
¶
Default:false
If set to true, then the first component of the log path will be removed from the object name and added to the container name, so that logs for different changes are distributed across a large number of containers.
-
zuul_log_container
¶
Default:logs
This role will create containers which do not already exist. If partitioning is not enabled, this is the name of the container which will be used. If partitioning is enabled, then this will be used as the prefix for the container name which will be separated from the partition name by an underscore. For example, “logs_42” would be the container name for partition 42.
Note that you will want to set this to a value that uniquely identifies your Zuul installation if using shared object stores that require globally unique container names. For example if using a public cloud whose Swift API is provided by Ceph.
-
zuul_log_container_public
¶
Default:true
If the container is created, should it be created with global read ACLs. If the container already exists, it will not be modified.
-
zuul_log_delete_after
¶
Default:15552000
Number of seconds to delete objects after upload. Default is 6 months (15552000 seconds) and if set to 0 X-Delete-After will not be set.
-
zuul_log_path
¶
Default:Generated by the role `set-zuul-log-path-fact`
Prepend this path to the object names when uploading.
-
zuul_log_create_indexes
¶
Default:true
Whether to create index.html files with directory indexes. If set to false, Swift containers can be marked with a Web-Listings=true property to activate Swift’s own directory indexing.
-
zuul_log_path_shard_build
¶
Default:False
This var is consumed by set-zuul-log-path-fact which upload-logs-swift calls into. If you set this you will get log paths prefixed with the first three characters of the build uuid. This will improve log file sharding.
More details can be found at set-zuul-log-path-fact.zuul_log_path_shard_build.
-