Verifying implementation against docs

← GuidesVerifying implementation against docsSet up `jsonui-doc check` end to end: declare checks in your config, wire up an adapter for whichever backend framework you're using, run the check locally and in CI, and read the resulting HTML page. Includes copy-pasteable snippets for FastAPI, Spring, NestJS, and Rails.~10 min read

What OpenAPI diff catches (and what it doesn't)The `builtin:openapi-diff` checker compares two OpenAPI documents: the one your backend declares about itself, and the one committed under `docs/api/*.json`. It catches: paths present or absent in either direction (`missing_in_impl` / `missing_in_doc`), request-body and parameter shapes (required, types, names), 2xx response schemas (property existence, types, nullable, required), enum value-set differences, and schema-name mismatches (reported as warnings, not mismatches — the DTO codegen uses the docs name). It does NOT catch: the actual JSON that flows on the wire (only the impl's declared shape), auth flows, or backend business logic. Error responses sit in between, and the distinction is worth holding precisely: a 4xx or 5xx gets a doc-to-impl presence check — the code must be declared on both sides — and nothing else. The body at that position is never compared, so two sides can disagree completely about what a 404 looks like without a finding. Impl-only codes are not reported either, which is why a framework's automatic 422 stays quiet. A code you silence with `ignore_response_codes` is skipped entirely, including the schema-name warning that would otherwise be the only thing said about a position no comparison inspects. Before comparing, both sides go through the same normalizer, so the spelling differences between OpenAPI 3.1 (what FastAPI and friends emit) and 3.0 (what a hand-written `docs/api` uses) do not show up as drift: a `type: ["string", "null"]` array and a null branch in `anyOf` both fold into the 3.0 `nullable` flag, a `null` member inside `enum` does the same, `allOf` wrappers around a `$ref` collapse, and — since jsonui-cli 1.6.11 — `const: X` folds into `enum: [X]`, which is the only way 3.0 can spell a single-value constraint. The `const` folding happens before the enum rules, so any real difference surfaces under the existing `enum` comparison key rather than adding a sixth key. Two consequences worth knowing when you upgrade: a `Literal[X]` field that used to produce a false `impl lacks: ['X']` mismatch now matches (on one real backend this removed 37 of 43 mismatches, leaving the 6 genuine ones), and two 3.1 sides declaring *different* `const` values — previously read by no comparison at all, and therefore silently OK — now report an `enum` mismatch. If your `docs/api` is written in 3.1, expect that second case to surface findings on the first run: they are real differences that were invisible before, not new noise. One path convention to know: the doc side is always `docs/api/` under the project root — this checker does not consult the `api_directory` setting — so declare it in the project that actually holds `docs/api`. If your backend can emit OpenAPI, this checker works regardless of framework — that's the point of adapter-driven design.

Declaring checks in jui.config.jsonAdd a top-level `checks` array to `jui.config.json`. Each entry has a `name` (identifier used in `--list` and reports), a `type` (`builtin:openapi-diff`, `builtin:db-schema`, or `checker` for full-checker), a type-specific field (`impl_openapi_command` for OpenAPI diff), and an optional `timeout_seconds` (default 60). An openapi-diff entry can also tune severity per comparison key: `"downgrade_to_warning": ["format"]` keeps `format` findings visible but off the gate, while `"ignore_schema_keys": ["nullable"]` drops that comparison outright. Downgraded findings land in the new `warning` status, which is reported with full expected/actual detail but does not count as a mismatch — so it never changes the exit code. On one real backend (178 paths) downgrading `format` alone took mismatches from 637 to 101 while keeping all 22 enum differences as real mismatches: the point is to stop genuine drift from drowning in noise, not to hide it. Auth credentials go in environment variables or command-line arguments — never in the config file itself. The runner cds into the project root and executes commands with the declared timeout; nothing outside `checks` runs.

jui.config.json

1// jui.config.json
2{
3  "checks": [
4    {
5      "name": "api",
6      "type": "builtin:openapi-diff",
7      "impl_openapi_command": "python -m app.export_openapi",
8      "timeout_seconds": 60
9    }
10  ]
11}

FastAPI adapter scriptFastAPI already knows its own OpenAPI — `app.openapi()` returns it as a dict. The adapter is a 5-line script that dumps that dict to stdout, invoked from `impl_openapi_command`. Save it as `app/export_openapi.py`, add `python -m app.export_openapi` to your config, and you're done. The same pattern works for any Python framework that exposes an OpenAPI dict.

app/export_openapi.py

1# app/export_openapi.py
2import json, sys
3from app.main import app
4 
5sys.stdout.write(json.dumps(app.openapi(), indent=2))

Spring / NestJS / Rails adaptersEvery popular backend framework has an OpenAPI-emitting mode. Spring Boot with springdoc-openapi exposes `/v3/api-docs` as JSON — `curl -s http://localhost:8080/v3/api-docs` becomes the `impl_openapi_command`. NestJS with `@nestjs/swagger` builds the document via `SwaggerModule.createDocument(app, config)` — dump it to stdout with `console.log(JSON.stringify(...))`. Rails with rswag emits YAML, so pipe it through `yq -o=json`. The exact shell varies but the pattern is the same: one command that prints valid OpenAPI JSON to stdout.

impl_openapi_command examples

# Spring (springdoc-openapi)
curl -s http://localhost:8080/v3/api-docs
 
# NestJS (@nestjs/swagger)
node -e "import('./main.js').then(m => console.log(JSON.stringify(m.openapi())))"
 
# Rails (rswag emits YAML)
bundle exec rake rswag:specs:swaggerize && yq -o=json swagger/v1/swagger.yaml

Running locally and in CILocally: run `jsonui-doc check --list` first to preview what will execute (this prints the resolved commands without running them). Then `jsonui-doc check api` runs only the API checkers, `jsonui-doc check db:main` runs only the checker targeting the `main` database, and `jsonui-doc check` with no filter runs everything declared. Results land in `docs/api/.check-report.json` (or `docs/db/{name}/.check-report.json`) — add these to `.gitignore` so they don't get committed. In CI: install the checker straight from the repository — it is not published on PyPI — with `pip install "git+https://github.com/Tai-Kimura/jsonui-cli.git@v1.6.13#subdirectory=document_tools"`. Use v1.6.13 or later: earlier tags declared the sibling `test_tools` package with no revision of its own, so pinning the tag still left that dependency floating on the default branch — the install was not reproducible in the way the pin implies. Pin the tag rather than tracking a branch: if the version of the tool doing the measuring moves on its own, a new finding no longer tells you whether the implementation drifted or the checker did. Then expose connection info via environment variables (e.g. `JSONUI_CHECK_DB_URL_MAIN`) and call `jsonui-doc check` as an explicit step. The exit code tells the CI whether to gate the merge: 0 = OK, 1 = mismatch, 2 = execution error. One measurement caveat: `impl_openapi_command` imports your backend's working tree, so an uncommitted change is part of the input — check that the tree is clean before you trust a number you are going to report.

local shell

# Preview what would run without executing
jsonui-doc check --list
 
# Execute all API checkers
jsonui-doc check api
# exit 0 = OK   exit 1 = mismatch   exit 2 = execution error
 
# Combined: check then generate HTML (with implementation-contract page)
jsonui-doc generate html --with-checks

.github/workflows/docs-check.yml

# .github/workflows/docs-check.yml (excerpt)
- name: Verify docs against implementation
  env:
    JSONUI_CHECK_DB_URL_MAIN: ${{ secrets.DB_URL_MAIN }}
  run: jsonui-doc check

Reading the HTML reportAfter check runs, `jsonui-doc generate html` (or the combined `generate html --with-checks` sugar) reads the report and adds an 'Implementation contract' page to the site. The page lists every mismatch row with target (path or table.column), status (mismatch / missing_in_impl / missing_in_doc), confidence badge, expected vs actual, and a human message. Missing rows in the checker output that the impl added get flagged as `missing_in_doc`; missing rows in the impl that the docs still list get flagged as `missing_in_impl`. A stale banner appears if `input_hashes` in the report don't match the current docs — meaning docs changed after the last check ran.

.check-report.json (mismatch row)

1// docs/api/.check-report.json (excerpt of one mismatch row)
2{
3  "target": "GET /orders/{id}",
4  "status": "mismatch",
5  "confidence": "proof",
6  "expected": "200: { id, total, status }",
7  "actual":   "200: { id, total, status, refunded_at }",
8  "message": "impl declares extra field 'refunded_at' not in docs/api"
9}

When you need a full checkerAdapter type can't express: hitting a live authenticated API and verifying the real response body, running semantic-invariant checks (like 'this column is null only when the order is refunded'), or spot-checking sampled data. Those go to `type: 'checker'` — you write a command that does the comparison itself and writes a result JSON conforming to the `.check-report.json` schema. The command runs with the same timeout / cwd / env-var contract as adapters. `schemaVersion: 1` is required at the top of the output; malformed output causes exit 2. A common pattern is a Python script using httpx to poll a handful of endpoints on staging, tag each response `ok` / `mismatch`, and set `confidence: 'sampled'` — a report that says 'the wire shape looked right for these N requests' rather than 'the schemas are identical.'

jui.config.json (full-checker slot)

1// jui.config.json (full-checker declaration)
2{
3  "checks": [
4    {
5      "name": "api-live",
6      "type": "checker",
7      "command": ".jsonui/checks/live_api_check.py",
8      "timeout_seconds": 120
9    }
10  ]
11}

.jsonui/checks/live_api_check.py

1# .jsonui/checks/live_api_check.py (outline)
2import httpx, json, sys
3 
4results = []
5for path in ["/items", "/users"]:
6    r = httpx.get(f"https://staging.example.com{path}", timeout=10)
7    results.append({"target": f"GET {path}",
8                    "status": "ok" if r.status_code == 200 else "mismatch",
9                    "confidence": "sampled"})
10sys.stdout.write(json.dumps({"schemaVersion": 1, "results": results}))