Project Testing Interface
The following sections describe an example PTI (Project Testing Interface) implementation. The goal is to setup a consistent interface for driving tests and other necessary tasks to succesfully configure Project Gating within your organization.
Projects layout
A proper PTI needs at least two projects:
org-config: a config-project curated by administrators, and
org-jobs: a untrusted-project to hold common jobs.
The projects that are being tested or deployed are also untrusted-project, and for the purpose of this example we will use a couple of integrated projects named:
org-server
org-client
org-config
The config project needs careful scrutiny as it defines priviledged Zuul configurations that are shared by all your projects:
Pipeline triggers and requirements let you define when a change is tested and what are the conditions for merging code. Approval from core members or special labels to indicate a change is good to go are pipelines configuration.
The base job let you define how the test environment is validated before the actual job is executed. The base job also defines how and where the job artifacts are stored.
More importantly, a config-project may enforce a set of integration jobs to be executed on behalf of the other projects. A regular (untrusted-project) can only manage its own configuration, and as part of a PTI implementation, you want to ensure your projects’ changes undergo validation that are defined globally by your organization.
Because the nature of those configuration settings are priviledged, config-projects changes are only effective when merged.
org-jobs
Jobs definition content are not priviledged Zuul settings and jobs can be defined in a regular untrusted-project. As a matter of fact, it is recommended to define jobs outside of the config project so that job updates can be tested before being merged.
In this example, we are using a dedicated org-jobs project.
Projects content
In this example PTI, the organization requirements are a consistent code style and an integration test to validate org-client and org-server works according to a reference implementation.
In the org-jobs project, we define a couple of jobs:
# org-jobs/zuul.yaml
- job:
name: org-codestyle
parent: run-test-command
vars:
test_command: $code-style-tool $org-check-argument
# e.g.: linters --max-column 120
- job:
name: org-integration-test
run: integration-tests.yaml
required-projects:
- org-server
- org-client
The integration-tests.yaml playbook needs to implement an integration test that checks both the server and client code.
In the org-config project, we define a project template:
# org-config/zuul.d/pti.yaml
- project-template:
name: org-pti
queue: integrated
check:
jobs:
- org-codestyle
- org-integration-test
gate:
jobs:
- org-codestyle
- org-integration-test
Finaly, in the org-config project, we setup the PTI template on both projects:
# org-config/zuul.d/projects.yaml
- project:
name: org-server
templates:
- org-pti
- project:
name: org-client
templates:
- org-pti
Usage
With the above layout, the organization projects use a consistent testing interface. The org-client or org-server does not need extra settings, all new contribution shall pass the codestyle and integration-test as defined by the organization admin.
Project tests
Projects may add extra jobs on top of the PTI. For example, the org-client project can add a user interface test:
# org-client/.zuul.yaml
- job:
name: org-client-ui-validation
- project:
check:
jobs:
- org-client-ui-validation
gate:
jobs:
- org-client-ui-validation
In this example, new org-client change will run the PTI’s jobs as well as the org-client-ui-validation job.
Updating PTI test
Once the PTI is in place, if a project needs adjustment, it can proceed as follow:
First a change on org-jobs is proposed to modify a job. For example, update a codestyle check using such commit:
# org-jobs/change-url
Update codestyle to enforce CamelCase.
Then, without merging this proposal, it can be tested accross the projects using such commit:
# org-client/change-url
Validate new codestyle.
Depends-On: org-jobs/change-url
Lastly the org-jobs may be enriched with:
# org-jobs/change-url
Update codestyle to enforce CamelCase.
Needed-By: org-client/change-url
Note
Extra care is required when updating PTI jobs as they affects all the projects. Ideally, the org-jobs project would use a org-jobs-check to run PTI jobs change on every projects.
Cross project gating
The org-pti template is using the “integrated” queue to ensure projects change are gated by the zuul scheduler. Though, the jobs need extra care to properly test projects as they are prepared by Zuul. For example, the org-integration-test playbook need to ensure the client and server are installed from the zuul src_root.
This is called sibling installation, and it is a critical piece to ensure cross project gating.