CI & Test Runs

Run a test suite and wait for the result

POST /api/cli/run

Executes a test suite and returns the outcome in one call, which is what a pipeline step needs. The response tells you whether everything passed, so a CI job can fail on allPassed alone. Pass a reportFormat to get the report inline instead of downloading it separately.

The call blocks until the run finishes. For a long suite, send progressRunId and poll the status endpoint instead of holding the connection open.

Parameters

Request body

Required.

Content type: application/json

Responses

200 — The run completed. Check allPassed for the verdict.

400 — The request was malformed or failed validation.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

409 — A request with this Idempotency-Key is still running, or its response was too large to keep. The code is IDEMPOTENCY_KEY_IN_PROGRESS or IDEMPOTENCY_RESPONSE_NOT_STORED, and in the second case the request is not run again.

422 — This Idempotency-Key was already used with a different request body, so use a new key. The code is IDEMPOTENCY_KEY_REUSED.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

List the test suites of a project for a CI pipeline

GET /api/cli/suites

Returns the test suites of a project so a pipeline can resolve a suite by name.

Parameters

Responses

200 — A page of test suites.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

List the environments of a project

GET /api/cli/environments

Returns the environments of a project, so a pipeline can pick the one whose variables the run should use. An environment marked isReadOnly rejects every write request sent to it, and the test cases concerned are reported as blocked rather than failed.

Parameters

Responses

200 — The environments of the project.

Array of:

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

Get the progress of a test run

GET /api/cli/runs/{id}/status

Returns how far a run has come, which lets a pipeline show progress instead of waiting silently. While the run is going, currentDurationMs counts up and completedAt stays null.

Parameters

Responses

200 — The current status of the run.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

Download the report of a finished run

GET /api/cli/runs/{id}/report

Returns the run report as a file. JUnit XML is the default because GitHub Actions, Azure Pipelines and Jenkins all read it natively.

The run has to be finished. Asking for the report of a pending or running test run is answered with 409, so poll the status endpoint first.

Parameters

Responses

200 — The report file.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

409 — The run is still pending or running, so there is no report yet.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

List the test suites of a project

GET /api/projects/{projectId}/test-suites

The project scoped equivalent of /api/cli/suites, which returns the same page of suites. Use whichever fits the path style of your client.

Parameters

Responses

200 — A page of test suites.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.

Execute a test suite

POST /api/projects/{projectId}/test-suites/{suiteId}/execute

Runs every test case in a suite in order, carrying extracted variables from one case into the next. This is the full result shape, so it also carries the per-case detail, the seed results and the performance metrics that /api/cli/run leaves out.

Set persistResults to false for a dry run that leaves no test run behind.

Parameters

Request body

Content type: application/json

Responses

200 — The run completed.

400 — The request was malformed or failed validation.

401 — The API key is missing, invalid, expired or revoked. A US organization calling

without X-RB-Region: us also lands here, because the request reached the wrong region.

403 — The key is valid but lacks the permission or the project scope for this call.

A scoped key is also refused on organization level operations by design.

404 — The resource does not exist, or it belongs to another organization or project.

Both cases answer the same way on purpose, so the API cannot be used to probe for foreign identifiers.

409 — A request with this Idempotency-Key is still running, or its response was too large to keep. The code is IDEMPOTENCY_KEY_IN_PROGRESS or IDEMPOTENCY_RESPONSE_NOT_STORED, and in the second case the request is not run again.

422 — This Idempotency-Key was already used with a different request body, so use a new key. The code is IDEMPOTENCY_KEY_REUSED.

429 — The rate limit was exceeded. The Retry-After header gives the seconds to wait.