Test Suites
Routebase's Test Runner sends automated requests against your live API and tracks their health over time — so you find out your contract drifted before your consumers do. Work is organized into test suites that group test cases; each case sends one request and asserts on the response.
The Test Runner page
Open Test Runner from the sidebar (requires the tests:read permission). The page has three panels:
| Panel | Contents |
|---|---|
| Left | The suite tree — collapsible Suites and Scenarios sections with a shared search box (Search suites & scenarios...), plus Suite Settings and project-wide Testing settings buttons at the bottom. |
| Center | The selected suite, with an Overview / Tests view toggle and the run toolbar. |
| Right | A collapsible panel with Results and History tabs (toggle with Show Results Panel / Hide Results Panel). |
Each suite in the tree carries a status dot summarizing its last run: green (passed), red (failed or error), yellow (running or pending), gray (never run or cancelled).
Creating suites and folders
Click the + button on the Suites header and choose New Suite or New Folder (requires tests:write).
- Enter a Name (e.g.
User API Tests). - Optionally add a Description and Tags — tags help you organize and filter suites.
- Click Create Suite.
Folders can be nested and support drag & drop, so you can group suites by service, team, or environment. Right-click a suite for Run, Rename, and Delete; folders offer Rename, New Subfolder, and Delete.
The number of test suites is limited by your plan. At the limit, the create button reads Limit Reached and Routebase prompts you to upgrade.
Suite overview
The Overview view is a dashboard for the selected suite:
- Pass Rate — color-coded percentage across recent runs, with a trend badge (Improving, Declining, or Stable).
- Total Runs — with a passed/failed/error breakdown.
- Average Duration and Last Run status.
- A Contract Tests card with linked-test and coverage stats — see Contract Testing.
These stats and the run history follow the environment selected in the toolbar. If a suite has runs but none against the current environment, the Overview shows "No runs for this environment" (with a count of the runs hidden by the filter) rather than an empty "No test runs yet".
Project-wide test trends (pass rate over 7/30/90 days) also appear on your dashboard.
Suite configuration
Click Suite Settings at the bottom of the sidebar to open the Suite Configuration sheet. It covers everything a suite needs beyond its cases:
| Section | What it does |
|---|---|
| Authentication | Credentials applied to every case in the suite (see below). |
| Seeds (pre / post-run) | Setup and cleanup sequences — see Test Data. |
| Reset snapshot | A known-good state restored before each run — see Test Data. |
| Schedules | Automated runs on a cron schedule — see Scheduling. |
| Webhooks (project-wide) | Notifications when runs complete — see Scheduling. |
| CI/CD | Ready-made pipeline snippets — see Scheduling. |
| Data Sets | Tabular data for data-driven runs — see Data-Driven Tests. |
A suite carries no base URL and no variables of its own. Both come from the environment a run is started against — its base URL is available to every request as {{baseUrl}} — and values that should differ from run to run come from a data set. Authentication is the one thing a suite can define for itself.
Authentication
Suite authentication has three modes: Inherit from Environment (default — use whatever the active environment defines), Custom Auth, and No Auth. Custom auth supports the full method catalog: Basic Auth, Bearer Token, API Key, OAuth 2.0, OAuth 1.0, JWT Bearer, Digest Auth, AWS Signature V4, Hawk Auth, and NTLM Auth. Credential fields accept {{VARIABLE_NAME}} placeholders, so secrets can live in environment variables instead of the suite. Environment-level auth itself is configured per environment — see Project Auth.
Project-wide testing settings
The Testing settings button at the bottom of the sidebar opens a settings page that applies to the whole project, independent of any one suite. Today it governs how the Schema Validation assertion treats null values across every test — see Contract Testing.
Test cases
A test case pairs one HTTP request with a set of assertions. Click Add Test Case in a suite:
- Enter a Name (e.g.
GET /users returns 200). - Choose the method —
GET,POST,PUT,PATCH,DELETE,HEAD, orOPTIONS— and the request URL (e.g.https://api.example.com/users). URLs must start withhttp://,https://, or a variable placeholder.
Cases run top to bottom within a suite; drag to reorder them. Expanding a case opens its editor with four tabs: Request, Assertions, Scripts, and Extract.
Right-click a case — on its card or on its row in the suite tree — for Rename, Duplicate, and Delete; both places also carry a more-actions button, and the whole menu is hidden without tests:write. In the tree you can rename in place: double-click the name, Enter saves, Esc discards. Duplicate copies the request (method, URL, headers, body, scripts), every assertion and every extraction into a new case named <name> (Copy) — the spec link is not carried over, so link the copy yourself if it should be a contract test.
Request
- URL — supports
{{...}}placeholders with inline highlighting and autocomplete. Typing{{offers the variables earlier cases in this suite extract (each with its origin, e.g. Step 1 · $.data.id), then environment variables, then fixtures;{{fixture.and{{data.drill into fixture paths and data-set columns. Selecting an environment resolves the placeholders. - Headers — name/value pairs.
- Body — with a Content-Type selector: JSON, Form URL Encoded, Multipart Form, Plain Text, or XML. The Variable button inserts a placeholder from three groups — Extracted variables (what earlier cases capture), Environment variables, and Fixtures. For a case linked to a spec endpoint, Generate Body fills the body from the endpoint's request-body example, or builds one from its schema when the spec carries no example.
- Dynamic values — the Dynamic menu inserts generated-at-runtime tokens:
| Token | Value |
|---|---|
{{$timestamp}} |
Unix timestamp in milliseconds |
{{$isoTimestamp}} |
ISO 8601 date/time string |
{{$uuid}} |
Random UUID v4 |
{{$randomInt}} |
Random number |
{{$randomEmail}} |
Random email address |
- Request settings — Timeout (ms) (default 30000), Follow Redirects, SSL Verification, and URL Encoding (Auto (default), Manual, or None).
Assertions
Assertions define what "passing" means. Each assertion has a type, an operator, an optional target, and an expected value:
| Type | Target | Typical use |
|---|---|---|
| Status Code | — | equals 200 |
| Body | — | contains "success" |
| Header | Header name | Content-Type contains application/json |
| Latency (ms) | — | < less than 500 |
| JSON Path | JSONPath (e.g. $.path.to.value) |
Assert on a specific field |
| Schema Validation | automatic | Validates the response against the linked endpoint's schema — see Contract Testing |
Operators: equals, not equals, contains, not contains, matches regex, exists, < less than**, **> greater than, <= less or equal**, **>= greater or equal. Each assertion has an on/off checkbox, so you can temporarily disable one without deleting it. Expected values accept {{...}} placeholders, including data-set columns.
When the case is linked to a spec endpoint, the JSON Path target field suggests paths from that endpoint's response schema: a Response fields dropdown listing each path with its type, filtered as you type, navigated with the arrow keys and accepted with Enter or Tab. Free text stays possible at all times, so expressions the schema can't enumerate still work — and without a link the field is a plain input. The Extract tab uses the same dropdown.
Scripts
The Scripts tab adds JavaScript that runs in a secure sandbox — a Pre-Request Script before the request is sent and a Post-Response Script after the response arrives. Scripts can set headers (request.setHeader(...)), read environment values, and store variables (rb.setVariable(...)) for later cases; the built-in API Reference panel documents the full scripting API. Use Library to insert a saved snippet and Save to add the current script to your library. Script output appears in the Console tab of the results.
Extract
The Extract tab captures values from the response into variables for subsequent test cases (and scenarios) — from the JSON body via JSONPath, from a response header, or the status code. Reference them later as {{variableName}}. The JSONPath field offers the same Response fields suggestions as a JSON Path assertion when the case is linked to an endpoint.
Importing test cases from your spec
Instead of writing cases by hand, click Import in the suite toolbar to generate them from an API spec (available on the Starter plan and above — on Free the button shows a lock and an upgrade prompt):
- Import from API Spec — pick a spec, optionally filter by version and method, and select endpoints (grouped by path prefix, with Select All / Deselect All).
- Preview Import — review the generated cases, rename them, and uncheck any you don't want.
- The import creates one case per endpoint with sensible assertions pre-filled — see Contract Testing for what gets generated.
Running tests
Pick the target environment in the toolbar's environment selector — it determines the base URL, variables, and authentication for the run. Switching environments shows which auth type is now active, and a warning appears if the selected environment has no authentication configured.
- Run a single case with the play button on its card.
- Run Selected — select more than one case and a bulk-action bar appears (N selected) with a Run Selected button. This is a filtered suite run, not a set of isolated single-case runs, so variables extracted by an earlier selected case still flow into a later one.
- Run All executes every case in the suite in order.
Runs require the tests:execute permission. On completion a toast summarizes the outcome (e.g. "All 12 test cases passed (843ms)").
Live feedback while a run is in flight
A suite run doesn't make you wait for the summary. Every case starts queued — dimmed, with a pulsing dot — exactly one case is running at a time, with a spinner ring and a highlighted border, and each card gets its result the moment that case finishes: 200 · 128 ms when it passed, 500 · 2/3 failed when it didn't.
Failed cases are tinted red. Passed ones deliberately are not tinted green — in a suite that is mostly green, tinting everything hides the one row that matters. If a run ends before reaching every case, the cases it never got to fall back to not yet run rather than spinning forever.
Functional vs. Performance mode
The toolbar's mode toggle switches between Functional (the default — one pass, assertion-focused) and Performance. The selected mode applies to every run trigger — a single case, Run Selected, and Run All alike. In Performance mode a Performance Config popover controls the load profile:
| Setting | Range |
|---|---|
| Iterations | 1–10,000 |
| Threads | 1–50 |
| Delay between iterations (ms) | 0–60,000 |
| On Error | Ignore — Continue on error, Stop — Skip to next iteration, or Abort — Cancel entire run |
Iterations run in parallel across threads; test cases within each iteration run sequentially. Results include throughput (req/s), P95 response time, error rate, and a Per-Test-Case Breakdown table (Avg / Min / Max / P95 / Err% / Count).
Reading results
Run overview (Run All / Run Selected)
After a Run All or Run Selected run, the Results tab opens on a live run-overview table — one row per test case, in execution order, that fills in as each case finishes:
- The case status, its name, and its duration.
- Vars in — the variables that resolved into the request, with secrets masked — and Extracted — the variables the case captured for later cases.
Below the table, a summary shows passed / failed / total time (a progress bar while the run is still going). Click any row to focus that case and load its full detail underneath.
Case statuses
A case ends a run in one of four states. Two of them are neither a pass nor a failure, because the case never got an answer:
| Status | What it means |
|---|---|
| Passed | The request went out and every enabled assertion held. |
| Failed | The request went out and something was wrong with the answer — a failed assertion, a schema mismatch, a transport error. |
| Pending | The case is a contract test and the environment does not serve that endpoint yet. Amber pill; does not fail the run — see Contract Testing. |
| Blocked | The target environment is read-only and this case writes, so nothing was sent. Muted shield instead of a red X — see below. |
Pending and blocked cases are counted separately and appear next to the total time (· 2 pending, · 1 blocked). Neither is ever folded into the failed count, and neither turns the run red — the completion toast counts the same way and names what it means, so a suite of eight reads and two blocked writes reports "8 passed · 2 blocked" rather than claiming all ten passed. A run in which nothing was sent — every case blocked — is reported as a neutral notice rather than a success: it confirmed nothing.
Blocked: writing against a read-only environment
An environment can be marked read-only on its detail sheet, under Contract → Read-only environment (the switch needs the Manage governance permission — a protection switch that everyone who may rename an environment can flip is not a protection switch). Environments in that state carry a Read-only badge in the environment list.
Only GET, HEAD and OPTIONS may be sent to such an environment. Every other method — including verbs Routebase doesn't know, which count as writes until proven otherwise — is refused before the URL is parsed and before any socket opens, so a blocked request leaves no trace on the target. The case is reported as Blocked with a message naming both the method and the environment: "POST is not allowed against the read-only environment “Production”. Only GET, HEAD and OPTIONS may be sent there."
One consequence is worth knowing before you flip the switch: an API that reads via POST — search endpoints, GraphQL, report queries — cannot be exercised against a read-only environment either. The guard goes by the method, not by what the endpoint does with it.
Case detail
The case detail — shown for a single-case run, or when you focus a row in the run overview — carries the case's latest outcome: status code, response size, and elapsed time, the per-assertion pass/fail table (expected vs. actual), and detail tabs:
| Tab | Contents |
|---|---|
| Response | The response body with syntax highlighting. |
| Headers | Response headers. |
| Cookies | Cookies set by the response. |
| Actual Request | Exactly what was sent — after variable substitution. |
| Console | Pre-request and post-response script output. |
A Timing Breakdown splits the request into DNS, Connect, TLS, First Byte, and Download phases — useful for separating network latency from server time.
Run history, export and sharing
The History tab lists past runs with filter tabs All, My Runs, and Shared, plus a compact trend chart. Click a run to load it into the Results tab.
- Export a run as JSON, an HTML Report, or JUnit XML (for CI systems).
- Share Test Report generates a link to a run report. Choose the access level — Public (anyone with the link) or Team — and an expiry (1 hour, 24 hours, 7 days, or 30 days). Active links are listed with copy, Extend by 24 hours, and Revoke link actions.
Permissions at a glance
| Action | Permission |
|---|---|
| View suites, cases, and results | tests:read |
| Create, edit, delete suites and cases | tests:write |
| Execute runs | tests:execute |
All built-in roles (Member, Admin, Owner) include these testing permissions; project-level team access still applies.
Related
- Test Scenarios — chain cases into multi-step workflows
- Contract Testing — validate responses against your spec
- Data-Driven Tests — run a case across many data rows
- Test Data — fixtures, seeds, and snapshots
- Scheduling — automated runs, webhooks, and CI/CD
- Environments — where your tests run