Skip to content

Record API responses in tests for client compatibility checks - #7368

Merged
agners merged 2 commits into
mainfrom
client-compat-recorder
Oct 15, 2026
Merged

agners merged 2 commits into
mainfrom
client-compat-recorder

Conversation

@mdegat01

@mdegat01 mdegat01 commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Proposed change

This is step 2 of #7307: recording the responses Supervisor's API tests produce, so the aiohasupervisor compat checker (python-supervisor-client#431) can check that released client versions can still parse them.

tests/common.py adds api_record_middleware(), which is registered on the api_client, api_client_v2 and app_api_client_with_prefix fixtures. The other API client fixtures reuse those. When SUPERVISOR_API_RECORD_DIR is set, every response whose body is a Supervisor JSON envelope is appended as one line to <dir>/<PYTEST_XDIST_WORKER or "main">.jsonl:

{"method": "GET", "template": "/host/disks/{disk}/usage", "params": ["default"],
 "query": [["max_depth", "2"]], "status": 200, "body": {...},
 "test": "tests/api/test_host.py::test_disk_usage_api"}
  • template is the matched route's canonical path, including /v2 for v2 routes. params holds the route values in template order.
  • query holds the query string as [key, value] pairs. Query keys can change a response (e.g. core/stats?one_shot), and the checker matches on them.
  • Responses that aren't Supervisor envelopes, such as the Core API proxy passthrough, are skipped.
  • With the variable unset, the middleware returns right after the handler and writes nothing. When recording, the cost is about 100µs per response, which doesn't show up in suite run time.

The middleware docstring documents the record format, which is the contract the client's checker reads. tests/api/test_api_record.py covers v1 and v2 recording, and covers the variable being unset.

Step 3, the client-compat CI job that runs the checker for the client versions in use, is still to do, so this doesn't close #7307.

Expected failures when running the checker locally: if you record with this branch and run the checker from the latest client release (0.7.0), it reports two failures until the client PRs below are merged and released:

With both client PRs applied, a recording of tests/api from this branch checks clean: 1108 responses, 216 replayed through the client, 0 failures, 0 warnings. Supervisor's own test suite passes either way.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (which adds functionality to the supervisor)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Ruff (ruff format supervisor tests)
  • Tests have been added to verify that the new code works.

If API endpoints or add-on configuration are added/changed:

🤖 Generated with Claude Code

@home-assistant home-assistant Bot added cla-signed refactor A code change that neither fixes a bug nor adds a feature labels Oct 14, 2026
@mdegat01
mdegat01 force-pushed the client-compat-recorder branch from 723c1aa to be9bf0e Compare October 14, 2026 03:38
@agners
agners requested a balanced review from Copilot October 14, 2026 09:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fresh recording directories fail, and envelope-shaped Core proxy responses are incorrectly recorded.

2 open findings
What changed in this PR

Adds test-time API response recording for future aiohasupervisor compatibility checks.

Changes:

  • Adds JSONL response-recording middleware.
  • Registers recording across v1, v2, and app API clients.
  • Tests enabled and disabled recording behavior.
File Description
tests/​common.py Implements response recording.
tests/​conftest.py Registers middleware for the primary API client.
tests/​api/​conftest.py Registers middleware for additional API clients.
tests/​api/​test_api_record.py Tests recorder output and opt-in behavior.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/common.py Outdated
Comment thread tests/common.py Outdated
mdegat01 and others added 2 commits October 14, 2026 18:10
Add an aiohttp middleware to the API test clients that, when
SUPERVISOR_API_RECORD_DIR is set, appends each Supervisor JSON response
with its route template, params, query, status and test id to a JSONL
file per xdist worker. The client library's compat checker replays these
records to catch responses it cannot parse. It is a no-op when the
variable is unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create the record directory on first write instead of requiring callers
to create it. Skip the Core API proxy and ingress passthrough routes by
their route template, since Core and app responses can look like
Supervisor envelopes. Drop the JSON and envelope checks so unexpected
responses fail loudly in the compat check instead of being skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mdegat01
mdegat01 force-pushed the client-compat-recorder branch from c625faa to 6d465b2 Compare October 14, 2026 18:10
@mdegat01
mdegat01 requested review from agners and a balanced review from Copilot October 14, 2026 20:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The recorder contract, fixture integration, proxy exclusions, and enabled/disabled behavior are consistently implemented and tested.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

@agners agners left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@agners
agners merged commit c461b3a into main Oct 15, 2026
22 checks passed
@agners
agners deleted the client-compat-recorder branch October 15, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed refactor A code change that neither fixes a bug nor adds a feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Catch aiohasupervisor response-parsing breakage in Supervisor CI

3 participants