Repository navigation
feat(adapter): build the governed-ingestion components - #2533
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 266e352e12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
6244336 to
b885e3e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fd6322577
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9fd6322 to
a60cc51
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a60cc51128
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add the listener-free engine boundary, the AdapterJob and AdapterJobRuntimeConfig formats, the file and read-only postgres job sources, the registry-adapter-job host with its adapter-job operator executable, and the kobotoolbox and odk-central source packs with example job deployments, so a registry can ingest from collection tools under the same governed contract as the live adapter. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
a60cc51 to
1fed546
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fed5462c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…st key and adapter-job The mTLS test key moved under the engine boundary, and the ODK repeats fixture carries 40-hex instance ids as item keys; both allowlists are scoped to their files. The CLI catalog publishes adapter-job, so the docs reference lists it as a draft, source-build-only binary. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Recovery listed the open runs under the input digest and took the first. A run another announcement opened over the same input would settle the attempt against the wrong run. Recovery now accepts only a run that matches every announced member, announces again when none does, and stays uncertain when more than one does. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A single reference such as $100000000 sized the parameter list by its index before any check, so compiling it attempted a multi-gigabyte allocation. References above the 16-parameter bound, and negative ones, now refuse at compile. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The configuration conformance suite never staged the adapter/job and adapter/job-runtime formats, so their mutation cells never ran. Staging them showed that `adapterctl job check` folded every reader finding into one root refusal, that the job runtime reader lost positions and accepted unknown members inside `store`, and that `job check` lacked `--deny-warnings`. - the job compile refuses with the reader's own located findings for the pack, job, profile, project, and job runtime config, and carries the reader's warnings on the compiled program; - the job runtime document decodes as one typed mapping, so every member, the store block included, refuses with its code, pointer, and position; the store block type moves to the engine reader and the job host reuses it; - `job check` takes `--deny-warnings` and reports a `diagnostics` list on success like every other check; - the conformance harness stages the kobotoolbox job example for both formats and checks the mutated file in place. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c57ab4073
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- ingestion run recovery reads every page of the caller's open runs under the input digest before it decides, and stays uncertain when a later page holds a second match or the listing outruns a ten-page bound, so it never announces a duplicate run nor accepts an ambiguous one; - a prepared batch records the identity of the batch binding it was prepared under, and recovery answers binding-changed instead of rebuilding the items against a binding of another identity. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
What this builds
The governed-ingestion components of the Registry Adapter (CONCEPT section 17), on top of the live adapter now on main:
src/engine/): the listener-free engine the ingestion job runs and the facade links, in the shape of the futureregistry-adapter-enginecrate: exchange, route interpreter, EIP processors, components, egress and outbound auth, scripts, languages, error handling, resilience, pack loading and compilation, replay, and a public compile entry that takes in-memory artifacts under a named profile. No listener, OIDC, journal, database, or PostgreSQL client type appears inside the boundary; write custody is a seam the facade installs, and the postgres source compiles its query behind a host-supplied client seam. The dependency gate proves the boundary at import level until the crate split.adapter/job, kindAdapterJob): the item contract, scan and recovery declarations, the edit property with the rescan delay, field projection, interval triggers with the Camel option names, and the call budget, all checked and bounded at the reader. The page reader is exactly as strict as its declared type: undeclared members refuse; a JSON nullnextreads as the finite end, like an absent one, so a route that maps a missing upstream cursor straight intonextends the scan.adapter/job-runtime, kindAdapterJobRuntimeConfig, conventionallyjob-runtime.yaml): the endpoint bindings every mode shares, the job source bindings, and the job store. It states none of the facade runtime's members and has its own schema, editor mapping, and conformance entry.job-fileandjob-postgresfeatures:file(readLock none and changed, doneFileName, Ant include, hidden files refused, per-poll window after the cursor) with CSV, JSONL, and raw decoders;pollEnrichas the job-only read step; read-onlypostgresover a parsed, gap-free plain SELECT with a required ORDER BY, under READ ONLY with a statement bound.registry-adapter-job): the no-payload item table with a lease that fences every write by epoch (acknowledge and refuse lock the lease row and write in one transaction), the seven-step pass with at-least-once hand-off through the destination seam, rescan traversals that continue from their saved cursor and end at the finite end (or, for an ongoing scan, at the tail), the interval trigger loop with an anchored initial delay, and theadapter-joboperator executable (check, pass, status, inspect, rescan) published in the CLI catalog. The host serves HTTP and GraphQL sources; a job whose source needs a PostgreSQL or file client refuses at compile withadapter.job.component-unavailableuntil the host constructs those clients.adapterctl job checkandadapter-job checkoffline end to end.AdapterEndpointBindingscarries the members every mode shares (base URL, credentials, TLS trust, egress); the facade document is unchanged for operators.Every Camel ported behavior was compared side by side against the pinned camel-4.23.0 sources; the deliberate divergences are recorded in the port table and the module docs.
Review fixes in this revision
All review threads are answered inline. Beyond those, a further review pass fixed:
postgres:send as an upstream send, matching the engine, which redelivers any non-direct:to;caBundleFileis spelled camelCase in the job runtime schema;adapterctl job checkrequires--profileand--projecttogether and counts the files it checked;maxMessagesPerPollkeeps Camel's option name, recorded as a configuration-convention exception;check-contracts.shrunsregistry-adapteron its own so the suites for a build without the job features run.postgres-test) compile clean under clippy, and the statement-timeout test uses an orderable expression so PostgreSQL cancels it rather than refusing the plan.adapter-jobas a draft, source-build-only binary.adapterctl job checkfolded reader findings into one root refusal, so the job compile now reports each reader's located findings (code, pointer, position), the job runtime document decodes as one typed mapping that refuses unknown members insidestoretoo,job checktakes--deny-warnings(listed in the CLI catalog), and a passing check reports itsdiagnosticslist.Verification
Run locally on the final tree. CI also runs
check-contracts.sh,test-job-postgres.shand the configuration conformance suite; it does not runtest-postgres.shor clippy withpostgres-test, which is where the lints and the statement-timeout test fixed in this revision had slipped through.products/adapter/scripts/check-contracts.sh: pass (3,620 tests, includingregistry-adapteron its own and the job-file lane; schemas, fmt, clippy)products/adapter/scripts/test-postgres.sh: pass (1,976 tests)products/adapter/scripts/test-job-postgres.sh: pass (61 tests)cargo clippy -p registry-adapter -p registry-adapter-core -p registry-adapter-job -p registry-adapterctl -p registry-adapter-connectors --all-targets --all-features -- -D warnings: cleancargo test -p registry-adapterctl: pass (132 tests)cargo test -p registry-adapter --test job_pack_replay: pass (15 tests)cargo test -p registry-cli-docs: pass (28 tests);generate-cli-reference.mjs --check: passproducts/identifiers/scripts/check.sh: passcheck-config-conventions.py: 135 findings, all recorded, 0 stalecheck-port-table.py: 43 rows, 35 delivered with evidence, 8 pending (the job rows, pending until the increment that delivers them)check_dependency_direction.py: passeditor mapping tests: pass (33 tests)
configuration conformance, the
adapter/jobandadapter/job-runtimeformats: 128 cells pass, 0 fail, 58 not applicableconfiguration conformance, full
--strictrun: no adapter format fails; the remaining failures are the products whose binaries were not built in the local target directory (CI builds them)check-config-conformance.py --check-generated: holds for 12 productsI reviewed the submitted changes