Skip to content

feat(adapter): share governed adapter execution with Evidence and close the delivery gaps - #2543

Merged
jeremi merged 29 commits into
mainfrom
feat/adapter-engine-evidence-packs
Nov 3, 2026
Merged

jeremi merged 29 commits into
mainfrom
feat/adapter-engine-evidence-packs

Conversation

@jeremi

@jeremi jeremi commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Shares one governed Adapter execution engine between the Adapter facade, the ingestion job host and Registry Evidence, and closes the remaining Adapter delivery gaps before v0.45.0.

  • registry-adapter-engine holds route compilation, execution, components and replay for the facade, adapter-job and Evidence.
  • Evidence Sources acquire through reviewed Adapter packs (pack, route, connection, inputs); the tagged http-json/sqlite-extract form is retired and refused with a migration pointer. The upgrade rehearsal skips the Evidence leg across this boundary and says why.
  • Ingestion jobs: PostgreSQL, file, SFTP and RabbitMQ sources; bounded profile quantities; passive RabbitMQ topology by default; closed refusal codes that name the member and the fix; delivery to a BReg Action or a Coordinator workflow.
  • KoboToolbox and ODK Central ingestion packs; pack pin mismatches name the asset and both digests.
  • Changelogs record the breaking changes for the release notes.

Twenty-nine commits, all signed off: the delivery, the review fixes, three fixes from automated review, three fixes from merge-queue runs, eight test-stability fixes, and thirteen follow-ups that keep the rest of the repository in step. Five are test-only and follow the new Adapter pack layout and fixture format (the bregctl prepare-source test and committed example, the Docker-only BReg dev lifecycle tests, the Evidence client deployment test, the Casework Evidence task-grant journey) or take the served Adapter's port from its own listening line instead of a pre-picked port another process could take. One points the secret-scan allowlist at the moved mTLS test key and the new SFTP test keys and introduces the BReg short form in the docs changelog. One teaches the release-tooling test that the Rust workspace result now waits for the Adapter PostgreSQL lane. One stores the staged item time at the microsecond precision PostgreSQL keeps, so a Coordinator event redelivered from the stored row carries the same event time as the first hand-off. One lets the Evidence fixture tests fall back to the system temporary directory when no Cargo target directory is set. One aligns the Evidence bundle and runtime connection schemas with the runtime (endpoint bindings are identified lists checked by name, and the runtime response-size bound matches the code) and records the Adapter job's product-coded quantity bounds in the configuration conformance expected failures. One regenerates the identifier catalog for those schema digests. One gives the AMQPS fixture broker up to a minute to boot before the private-CA test calls it unavailable. One raises the identifier catalog job's time limit from 10 to 20 minutes, since its cold build already took over nine minutes on main and the larger workspace pushed it past ten.

The three fixes from automated review:

  • Job retry identities no longer hash the job version. The item store keeps no version beside a pending item, so a version bump between attempts would have given the retry a new BReg command key or Coordinator event ID and allowed a second write. Command keys and event IDs now derive from the job, the item key, its sequence and its digest, as the concept specifies.
  • A job that enables both the RabbitMQ and file sources no longer builds a file decoder for a broker poll; the rabbitmq-json profile is decoded by the broker path. The RabbitMQ lane of the job PostgreSQL gate now also carries the job-file feature to keep that build proven.
  • PostgreSQL source pages execute through a portal that fetches at most one row past the page bound. The engine pages with OFFSET and no LIMIT, so each page used to drain the rest of the result until the statement timeout.

The three fixes from merge-queue runs:

  • The BReg PostgreSQL runner exported the real Evidence proof's Python interpreter as a path relative to the repository root, which cargo could not spawn from the crate directory when CARGO_TARGET_DIR was unset. The path is now absolute, with a runner test for the unset case.
  • The Evidence tutorials now show the pack-form explain output (pack, route, posture, typed facts), the check line that names deployment inputs it could not use, and the smaller packaged example.
  • The prove tutorial quotes a fixture case in the pack form, which intercepts the endpoint and supplies the exchange.

The eight test-stability fixes are test-only and address timing failures seen in merge-queue runs:

  • The Adapter lease race test clears the activity snapshot before each read, since PostgreSQL keeps the first pg_stat_activity read for the whole transaction and a status GET on a newer connection stayed invisible.
  • The BReg native session-loss journey allows the extra pooled worker failures that terminating every session of a replica can cause, and the harness now names each cleanup failure instead of counting them.
  • caseworkctl provider-pair tests and the evidencectl fixtures tests wait out Linux ETXTBSY on a freshly written executable, with a bounded retry that fires on that error only.
  • Messaging served-runtime tests restart on fresh ports when a bind finds the address taken.
  • Adapter status-link replays compare everything except the Date header.
  • The Render serve tests refuse audit writes by shutting down a socket pair instead of dropping a pipe a sibling fork could hold open.
  • The RabbitMQ broker fixture bounds its docker and rabbitmqctl calls.

Local checks on the same tree: cargo fmt --check, cargo clippy --workspace --all-targets -D warnings and cargo deny check clean; Adapter facade PostgreSQL suite ok; Adapter job PostgreSQL suite 190 passed, 0 failed; release tooling tests 51 ok; BReg PostgreSQL runner tests 16 ok; Evidence tutorial dry run ok; configuration conformance (strict) clean for the Adapter and Evidence formats; Evidence check-contracts.sh ok; configuration convention lint ok. A known Render serving test flake is unrelated to this change.

…se the delivery gaps

- Split the governed execution engine into registry-adapter-engine so Evidence and the facade share it.
- Bound acquisition and disclosure, and document Evidence consumer guidance.
- Hold async write custody and resume bounded facade writes.
- Give the job host store init, scheduling, recovery and hold guidance.
- Add RabbitMQ, SFTP, SQLite extract and hook sources, plus direct BReg and Coordinator destinations.
- Fix fixture and example pins; express main's #2533 and #2538 fixes in the engine crate.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ter delivery

Applies the independent review fixes on top of the delivery, resolved
against main: job and pack refusals name the member and the fix, sources
check their secret references and topology at load, the job host's profile
quantities are bounded types, pin mismatches name the computed digest, and
the generated schemas and catalogs are regenerated.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…urce test and example

The BReg Evidence source export now writes its pack under
adapter-packs/<sourceId>/ with JSON schemas, and route inputs carry readable
<selector>--<field> names. The prepare-source test still looked up
schemas/registry-facts.yaml, and the committed breg/evidence-source-export
example still carried the hashed-input digests.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The Evidence source export now writes the registry lookup as an Adapter
pack, so the two Docker lifecycle tests read the route call, constant
headers and response schema from adapter-packs/<id>/pack.yaml and
schemas/response.json, and the unresolved problem from
protocols/registry-lookup.json, in place of the source's request block.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…nt test

The adult-status fixture moved to block YAML and an Adapter pack source
connection, so the deployment test's profile rewrites, package limits and
runtime document now match it. The SQLite time-budget test accepts either
deadline report, since the worker and the caller's wait share one deadline
and both map to the same public budget code.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The harness picked a port by binding port 0 and closing it, then counted
any answer on that port as readiness. Under parallel load another process
could take the port first: the adapter then failed to bind while the
harness talked to the stranger, and the next request was reset. The
adapter now binds port 0 and the harness reads the bound port from its
`listening` line, draining stderr on a thread so the pipe never fills.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T15:22:21.189453Z 1174446 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

… in the changelog

The Adapter engine move left the secret-scan allowlist for the mTLS test
key on the old path, and the SFTP source tests add throwaway OpenSSH keys
with no entry. Each key is pinned by its embedded public key and scoped to
its test file. The docs style check wants the BReg short form introduced
before the changelog uses it.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68a3800f60

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ce result

The Adapter PostgreSQL and recovery suites run as their own path-gated CI job, and the aggregate Rust workspace result now waits for it.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…grant journey

The adult-status fixture moved to block YAML and an Adapter pack source connection, so the journey's profile rewrites, runtime source connection and package limits now match it.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ores

A redelivery reads the event time back from its timestamptz row, which keeps
microseconds, while the first hand-off carried the clock reading at nanosecond
resolution. On Linux the two differed, so a destination saw a different event
time on redelivery. Truncate the staged time to microseconds before it is
written or handed off, so every hand-off of a sequence carries one instant.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Two fixture suites required CARGO_TARGET_DIR, which CI does not set. Use it
when present and the system temp directory otherwise, as the statement-source
suite already does.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Source connection endpoints are an id-unique list keyed by endpoint name, so
the bundle and runtime schemas no longer declare them a set, and the runtime
endpoint response bound matches the shared binding's 16 MiB ceiling. The
conformance corpus records the Adapter job's product-coded integer bounds as
an expected failure and the bundle's shifted positions.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…chemas

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ing it unavailable

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…er workspace

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@jeremi
jeremi enabled auto-merge November 3, 2026 11:17
…nges

A pending item keeps its sequence across a redeploy, but its BReg command key and Coordinator event id also hashed the job version. A retry after a version bump then looked like a new command, so BReg could execute an Action twice. Derive both from the job id, item key, sequence and digest only.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
With job-file and job-rabbitmq in one build, every RabbitMQ poll fell into the file decoder's refusal arm and stopped with adapter.job.read-bound before reading the broker. The RabbitMQ pass suite now runs with job-file enabled so that combination stays covered.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The engine pages with OFFSET and no LIMIT, so an unbounded execution
kept the server producing the rest of the result after the page was
full, holding the connection until the statement timeout. The read now
binds a portal and fetches at most one row past the page bound.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92e19fbcc7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremi
jeremi added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
Without CARGO_TARGET_DIR the PostgreSQL runner exported the interpreter as
target/breg-test-python/bin/python, relative to the repository root, while
cargo runs the proof from the crate directory, so spawning it failed with
NotFound. The runner now anchors a relative environment path to its own
working directory.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
explain now names the pack, route, posture and typed facts instead of the
retired http-json request, Functions and limits; check names deployment
inputs it could not use; and the packaged example carries fewer files
without per-source extract Functions.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@jeremi
jeremi added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
A fixture case now intercepts the pack endpoint and supplies the exchange it answers with; the tutorial still quoted the retired bare source response.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@jeremi
jeremi added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
…e waiter

The lease race test reads pg_stat_activity inside the transaction that
holds the row lock. PostgreSQL keeps the first read's snapshot for the
whole transaction, so a status GET served on a connection opened after
that read never appeared and the wait timed out.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Terminating every PostgreSQL session of the owning replica can fail each
pooled worker loop once more before the pool notices, including the timer
worker after the owner resumes, and the review worker. The journey still
requires the timer failure it provokes and proves recovery by the peer.

The harness now reports each cleanup failure instead of only counting
them, so a log-code mismatch names the codes it saw.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…air tests

On Linux a concurrent test's fork can inherit the writer descriptor of a
stub written moments ago, so exec of it fails with ETXTBSY and caseworkctl
reports io-failure instead of the peer's exit class.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tests

The PATH-fallback test execs a freshly copied evidencectl; a concurrent
fork holding the copy's writer descriptor makes the first exec fail on
Linux.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… finds the address taken

A pre-picked port is released before the runtime binds it, so a parallel
test can take it first and the start fails with AddrInUse.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The server stamps Date from the wall clock, so two reads that straddle a
second differ there and nowhere else.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…of closing a pipe

A concurrently forked child keeps a dropped pipe's reader alive until it
execs, so the audit write could still succeed and the render return 200.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
An unbounded docker exec once hung for 44 minutes on a healthy broker and
stalled the whole job suite.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@jeremi
jeremi added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit e5c9120 Nov 3, 2026
136 of 141 checks passed
@jeremi
jeremi deleted the feat/adapter-engine-evidence-packs branch November 3, 2026 16:49
@jeremi jeremi mentioned this pull request Nov 3, 2026
1 of 2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants