Testing

Contract Testing

Contract testing proves that your running API still matches the spec you designed — not just that it returns 200. Link a test case to an endpoint and Routebase validates the live response against that endpoint's documented schema, and warns you when the spec changes underneath an existing test.

Linking a test case to an endpoint

  1. Expand the test case and click Link to Spec.
  2. In the dialog, choose the API Specification, optionally filter by Version (optional) ("Filter endpoints by a specific version"), and pick the endpoint from the searchable list.
  3. Click Link Endpoint.

Linking does two things:

  • Routebase stores a snapshot of the endpoint's contract (path, method, request-body schema, and response schemas) together with the spec version it came from. The snapshot is the baseline for drift detection.
  • If the case has no schema check yet, a Schema Validation assertion is added automatically.

Linked cases show a badge with the endpoint's method and version; hovering reveals the full target, e.g. "Linked to GET /users (1.0.0)". Click Unlink to remove the link. Linking requires the tests:write permission.

A test case linked to a spec endpoint with the contract badge

The Schema Validation assertion

Schema Validation is one of the assertion types in the assertions table. Unlike other assertions it needs no target or expected value — the table shows it as auto / schema match, and its tooltip explains: "Validates response against linked endpoint schema." When the case runs, the actual response body is validated against the linked endpoint's documented response schema; a mismatch fails the assertion, and the failure details appear in the assertion results like any other assertion.

This is the heart of contract testing: a field that changed type, a required property that went missing, or an undocumented shape all fail the run — even if the status code is still 200.

Like any assertion, Schema Validation has an on/off checkbox, so you can keep the link and its drift detection while temporarily muting the schema check.

Pending: the endpoint isn't there yet

Design-first has a failure mode. Generate a suite from a fresh spec and it is red from its very first run, and stays red until the service catches up. People get used to red — and that is exactly when the first real red gets missed.

The two cases are machine-distinguishable, so Routebase distinguishes them:

What the environment did Result
Nobody answered — connection refused, timeout, DNS failure Pending — not implemented yet
It answered 404 on a path the contract says exists Pending — not implemented yet
It answered, but wrongly — a failed assertion, a body that doesn't match the schema Failed, red, exactly as before

A pending case gets its own amber Pending pill, its own counter next to the run summary, and — the point of the whole thing — it does not fail the run. Your new suite isn't red, it's pending. As endpoints land, cases move from pending to passed one by one, and the day something goes red it means something is actually wrong.

Two guards keep this from swallowing real failures:

  • Only contract tests qualify. A hand-written case with no spec link that fails is simply failing — Routebase has no business deciding what its author expected.
  • A test that asked for what it got has passed. A negative test asserting Status Code equals 404 is green, not pending.

There is a third route into pending, from the contract side rather than the response. When the version an environment pins does not describe the linked endpoint at all, a run there reports the case as pending — the same statement as a 404 on a contractual path, established before the request goes out. The contract badge says so up front: "This environment is pinned to 1.2.0, which does not describe GET /users. A run here reports this case as pending." That route is deliberately narrowed to cases carrying a Schema Validation assertion; without the narrowing, a case that only asserts status == 200 — green against a running service — would turn pending because of a stale pin and drop out of the passed count.

Pending has a sibling: a case is reported as blocked when the target environment is marked read-only and the case's method writes. Nothing was sent, so it is neither a pass nor a failure either, and it gets its own muted badge and counter. See Test Suites for the read-only switch and what it refuses.

Null handling

Imported specs often omit nullable on fields that can legitimately return null. Under strict validation, the Schema Validation assertion then fails on otherwise-valid responses — a null where the schema never marked the field nullable.

The Testing settings page (opened from the bottom of the Test Runner sidebar) sets how schema validation treats null, project-wide, under Schema validation → Null handling:

Mode Behavior
Strict Reject null in fields the schema does not mark nullable.
Allow null values Accept null anywhere, while still type-checking every present value against its declared type.

The setting applies to every test in the project. Switch to Allow null values when you validate against imported specs whose nullability isn't fully annotated — real contract violations still fail, without null noise drowning them out.

Generating contract tests from your spec

The Import button in the suite toolbar bulk-creates test cases from spec endpoints (see Test Suites for the wizard flow). Each imported case comes with assertions derived from the endpoint's contract:

Generated assertion Based on
Status Code equals the documented success code The endpoint's first 2xx response (or 200)
Body exists Whether the response defines content
Header Content-Type contains the documented media type The response's content type
JSON Path exists for required fields (e.g. $.id) Top-level required properties of the response schema (up to 5)
Latency (ms) < 5000 A baseline performance guard

Imported case names follow the endpoint (e.g. GET /users - List all users), and URLs use a {{baseUrl}} placeholder so the same case runs against any environment.

Schema drift detection

The snapshot taken at link time lets Routebase detect schema drift: the spec endpoint has changed since the test was linked. Whenever you open a linked test case, Routebase compares its snapshot against the latest published version of the endpoint. If they diverge, a banner appears at the top of the case editor — e.g. "Schema changed (linked: v1.0.0, current: v2.1.0)".

The banner lists the detected changes — path, method, request-body schema, or response schemas, each with old and new values — and, if the endpoint was removed entirely, flags that too. Comparison ignores internal identifiers, so only meaningful contract changes count as drift.

Resolving drift

  • Sync from Spec updates the test case to the current contract. The confirmation dialog spells out the consequences: the case's URL, method, and request body are updated to match the current spec endpoint, and if the body changes you get a side-by-side Current / New diff before confirming — manual edits to the body will be overwritten. Syncing re-pins the snapshot to the latest published version, which clears the drift.
  • Check against linked version — an optional toggle that compares the snapshot against the originally linked spec version instead of the latest one, useful for diagnosing when a change was introduced.

Drift never fails a run by itself — it's an early warning that your test and your documentation have parted ways, so you can decide whether the spec change was intentional before consumers notice.

Contract coverage on the suite dashboard

The suite's Overview view includes a Contract Tests card summarizing the suite's contract health:

  • Linked Tests — how many cases are linked to spec endpoints.
  • Coverage — how many distinct endpoints those links cover.
  • Schema Drift — a warning count when linked cases have drifted.

Use it to spot suites that assert plenty of status codes but validate few actual contracts.

  • Test Suites — cases, assertions, runs, and results
  • Endpoints — the operations your contract tests validate
  • Schemas — the structures responses are validated against
  • Versioning — how spec versions relate to linked tests
  • Schema Drift — continuous drift monitoring in the Monitoring module