Add Schemathesis for checking APIs against their OpenAPI specs

[aaronvanderlip]

Schemathesis generates requests from an OpenAPI spec and reports where the responses disagree with it.

Updated DEV_SETUP.md with instructions for running schemathesis and configuring schemathesis.toml to run against the future commcare-hq spec.

[millerdev]

The comments and questions I posted on the previous PR also apply here.

[aaronvanderlip]

The comments and questions I posted on the previous PR also apply here.

How long does the command take to complete? If it's short, that's great. If it's long (minutes), we may want to look into optimizing it.

In the current configuration it is only testing the endpoints described in the spec and not performing any fuzz testing or testing undocumented HTTP methods. In that configuration, the cost of running it in a testing pipeline would be the network and app overhead per request X the number of endpoints to be tested. The tool also supports multithreading so a pool of workers could be setup. It is TBD what the overhead for write operations would be, but simple GET requests I would expect to resolve quickly.

Is it flaky? I assume not, but worth noting if it is.

I would expect the flake to come from the target app or network.

Running it in automated tests would allow us to test write requests.

Yes. This would require the spec generation tool output link information which it currently does not.

In an automated context, would it make sense to split it up to have one test per API request?

Probably not.

I am concerned that this may fall into disrepair/disuse if it's not run automatically. Not that it's not useful, but that it may be forgotten over time.

The goal for this PR is to install a tool that can be used while the tooling to generate the spec is being developed to help ensure the correctness of the final spec. The current configuration is lenient and those decisions are documented in schemathesis.toml.example. Once the spec is well formed, the settings can be modified to become more strict so that the tool is testing more than the happy path.

Longer term, the tool should run automatically to ensure there is no drift in the spec and the endpoints are operating as expected.

As a dependency watcher, noting that this adds one new dependency to the dev group, which is not a concern in itself. However, the subgraph of dependencies pulled in is large-ish, which adds to the burden of keeping dependencies up to date.

Schemathesis is a wrapper around one of those dependencies, https://hypothesis.readthedocs.io/en/latest, which might prove to have some use in testing outside of API code.

[millerdev]

Wondering if it would make sense to remove the new dependency from dev requirements "while the tooling to generate the spec is being developed?"

Suggested change

'schemathesis', # API conformance checks, see DEV_SETUP.md

Users who want to test it can run a command to install it locally:

uv pip install schemathesis

This install command could be added to the new section of DEV_SETUP.md

That way we can defer the dependency maintenance burden until later, presumably when API checks are automated.