Testing

Zuul provides an extensive framework for performing functional testing on the system from end-to-end with major external components replaced by fakes for ease of use and speed.

Test classes that subclass ZuulTestCase have access to a number of attributes useful for manipulating or inspecting the environment being simulated in the test:

tests.base.simple_layout(path, driver='gerrit')

Specify a layout file for use by a test method.

Parameters

  • path (str) – The path to the layout file.

  • driver (str) – The source driver to use, defaults to gerrit.

Some tests require only a very simple configuration. For those, establishing a complete config directory hierachy is too much work. In those cases, you can add a simple zuul.yaml file to the test fixtures directory (in fixtures/layouts/foo.yaml) and use this decorator to indicate the test method should use that rather than the tenant config file specified by the test class.

The decorator will cause that layout file to be added to a config-project called “common-config” and each “project” instance referenced in the layout file will have a git repo automatically initialized.

class tests.base.ZuulTestCase(*args, **kwargs)

A test case with a functioning Zuul.

The following class variables are used during test setup and can be overidden by subclasses but are effectively read-only once a test method starts running:

Variables

  • config_file (str) – This points to the main zuul config file within the fixtures directory. Subclasses may override this to obtain a different behavior.

  • tenant_config_file (str) – This is the tenant config file (which specifies from what git repos the configuration should be loaded). It defaults to the value specified in config_file but can be overidden by subclasses to obtain a different tenant/project layout while using the standard main configuration. See also the simple_layout() decorator.

  • tenant_config_script_file (str) – This is the tenant config script file. This attribute has the same meaning than tenant_config_file except that the tenant configuration is loaded from a script. When this attribute is set then tenant_config_file is ignored by the scheduler.

  • create_project_keys (bool) – Indicates whether Zuul should auto-generate keys for each project, or whether the test infrastructure should insert dummy keys to save time during startup. Defaults to False.

  • log_console_port (int) – The zuul_stream/zuul_console port.

The following are instance variables that are useful within test methods:

Variables

  • fake_<connection> (FakeGerritConnection) – A FakeGerritConnection will be instantiated for each connection present in the config file and stored here. For instance, fake_gerrit will hold the FakeGerritConnection object for a connection named gerrit.

  • gearman_server (FakeGearmanServer) – An instance of FakeGearmanServer which is the Gearman server that all of the Zuul components in this test use to communicate with each other.

  • prometheus_server (PrometheusServer) – An instance of :py:class: ~test.base.PrometheusServer` which is the Prometheus metrics endpoint.

  • executor_server (RecordingExecutorServer) – An instance of RecordingExecutorServer which is the Ansible execute server used to run jobs for this test.

  • builds (list) – A list of FakeBuild objects representing currently running builds. They are appended to the list in the order they are executed, and removed from this list upon completion.

  • history (list) – A list of BuildHistory objects representing completed builds. They are appended to the list in the order they complete.

addEvent(connection, event)

Inject a Fake (Gerrit) event.

This method accepts a JSON-encoded event and simulates Zuul having received it from Gerrit. It could (and should) eventually apply to any connection type, but is currently only used with Gerrit connections. The name of the connection is used to look up the corresponding server, and the event is simulated as having been received by all Zuul connections attached to that server. So if two Gerrit connections in Zuul are connected to the same Gerrit server, and you invoke this method specifying the name of one of them, the event will be received by both.

Note

“self.fake_gerrit.addEvent” calls should be migrated to this method.

Parameters

  • connection (str) – The name of the connection corresponding to the gerrit server.

  • event (str) – The JSON-encoded event.

assertBuilds(builds)

Assert that the running builds are as described.

The list of running builds is examined and must match exactly the list of builds described by the input.

Parameters

builds (list) – A list of dictionaries. Each item in the list must match the corresponding build in the build history, and each element of the dictionary must match the corresponding attribute of the build.

assertHistory(history, ordered=True)

Assert that the completed builds are as described.

The list of completed builds is examined and must match exactly the list of builds described by the input.

Parameters

  • history (list) – A list of dictionaries. Each item in the list must match the corresponding build in the build history, and each element of the dictionary must match the corresponding attribute of the build.

  • ordered (bool) – If true, the history must match the order supplied, if false, the builds are permitted to have arrived in any order.

assertReportedStat(key, value=None, kind=None, timeout=5)

Check statsd output

Check statsd return values. A value should specify a kind, however a kind may be specified without a value for a generic match. Leave both empy to just check for key presence.

Parameters

  • key (str) – The statsd key

  • value (str) – The expected value of the metric key

  • kind (str) –

    The expected type of the metric key For example

    • c counter

    • g gauge

    • ms timing

    • s set

  • timeout (int) – How long to wait for the stat to appear

Returns

The value

commitConfigUpdate(project_name, source_name)

Commit an update to zuul.yaml

This overwrites the zuul.yaml in the specificed project with the contents specified.

Parameters

  • project_name (str) – The name of the project containing zuul.yaml (e.g., common-config)

  • source_name (str) – The path to the file (underneath the test fixture directory) whose contents should be used to replace zuul.yaml.

getSortedBuilds()

Return the list of currently running builds sorted by name

getUpstreamRepos(projects)

Return upstream git repo objects for the listed projects

Parameters

projects (list) – A list of strings, each the canonical name of a project.

Returns

A dictionary of {name: repo} for every listed project.

Return type

dict

logState()

Log the current state of the system

newTenantConfig(source_name)

Use this to update the tenant config file in tests

This will update self.tenant_config_file to point to a temporary file for the duration of this particular test. The content of that file will be taken from FIXTURE_DIR/source_name

After the test the original value of self.tenant_config_file will be restored.

Parameters

source_name (str) – The path of the file under FIXTURE_DIR that will be used to populate the new tenant config file.

printHistory()

Log the build history.

This can be useful during tests to summarize what jobs have completed.

setUp()

Hook method for setting up the test fixture before exercising it.

class tests.base.FakeGerritConnection(driver, connection_name, connection_config, changes_db=None, upstream_root=None, poller_event=None, ref_watcher_event=None)

A Fake Gerrit connection for use in tests.

This subclasses GerritConnection to add the ability for tests to add changes to the fake Gerrit it represents.

addFakeChange(project, branch, subject, status='NEW', files=None, parent=None)

Add a change to the fake Gerrit.

getEventQueue()

Return the event queue for this connection.

Returns

A zuul.zk.event_queues.ConnectionEventQueue instance or None.

getWebController(zuul_web)

Return a cherrypy web controller to register with zuul-web.

Parameters

zuul_web (zuul.web.ZuulWeb) – Zuul Web instance.

Returns

A zuul.web.handler.BaseWebController instance.

maintainCache(relevant)

Make cache contain relevant changes.

This lets the user supply a list of change objects that are still in use. Anything in our cache that isn’t in the supplied list should be safe to remove from the cache.

toDict()

Return public information about the connection

validateWebConfig(config, connections)

Validate web config.

If there is a fatal error, the method should raise an exception.

Parameters

  • config – The parsed config object.

  • connections (zuul.lib.connections.ConnectionRegistry) – Registry of all configured connections.

class tests.base.FakeGearmanServer(use_ssl=False)

A Gearman server for use in tests.

Variables

hold_jobs_in_queue (bool) – If true, submitted jobs will be added to the queue but will not be distributed to workers until released. This attribute may be changed at any time and will take effect for subsequently enqueued jobs, but previously held jobs will still need to be explicitly released.

release(regex=None)

Release a held job.

Parameters

regex (str) – A regular expression which, if supplied, will cause only jobs with matching names to be released. If not supplied, all jobs will be released.

class tests.base.RecordingExecutorServer(*args, **kw)

An Ansible executor to be used in tests.

Variables

hold_jobs_in_build (bool) – If true, when jobs are executed they will report that they have started but then pause until released before reporting completion. This attribute may be changed at any time and will take effect for subsequently executed builds, but previously held builds will still need to be explicitly released.

failJob(name, change)

Instruct the executor to report matching builds as failures.

Parameters

  • name (str) – The name of the job to fail.

  • change (Change) – The FakeChange instance which should cause the job to fail. This job will also fail for changes depending on this change.

release(regex=None, change=None)

Release a held build.

Parameters

regex (str) – A regular expression which, if supplied, will cause only builds with matching names to be released. If not supplied, all builds will be released.

returnData(name, change, data)

Instruct the executor to return data for this build.

Parameters

  • name (str) – The name of the job to return data.

  • change (Change) – The FakeChange instance which should cause the job to return data.

  • data (dict) – The data to return

class tests.base.FakeBuild(executor_server, build_request, params)
getWorkspaceRepos(projects)

Return workspace git repo objects for the listed projects

Parameters

projects (list) – A list of strings, each the canonical name of a project.

Returns

A dictionary of {name: repo} for every listed project.

Return type

dict

hasChanges(*changes)

Return whether this build has certain changes in its git repos.

Parameters

changes (FakeChange) – One or more changes (varargs) that are expected to be present (in order) in the git repository of the active project.

Returns

Whether the build has the indicated changes.

Return type

bool

isWaiting()

Return whether this build is being held.

Returns

Whether the build is being held.

Return type

bool

release()

Release this build.

class tests.base.BuildHistory(**kw)