Skip to content

Commit ba9a283

Browse files
committed
feat: add pg_converter orchestration
1 parent 239787b commit ba9a283

20 files changed

Lines changed: 1610 additions & 22 deletions

‎.github/workflows/publish.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ jobs:
5151
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/python" -m pip install dist/*.whl
5252
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/python" -m pip check
5353
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/pg-play" capabilities
54-
for command in pg-configurator pg-diag pg-perf-bench pg-stand pg-workload; do
54+
for command in pg-configurator pg-converter pg-diag pg-perf-bench pg-stand pg-workload; do
5555
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/${command}" --version
5656
done
5757
- name: Upload distributions for publishing

‎.github/workflows/tests.yml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ jobs:
2828
with:
2929
repository: O2eg/pg_configurator
3030
path: .ci-components/pg_configurator
31+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
32+
with:
33+
repository: O2eg/pg_converter
34+
path: .ci-components/pg_converter
3135
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
3236
with:
3337
repository: O2eg/pg_diag
@@ -51,6 +55,7 @@ jobs:
5155
run: |
5256
python -m pip install \
5357
--editable .ci-components/pg_configurator \
58+
--editable .ci-components/pg_converter \
5459
--editable .ci-components/pg_diag \
5560
--editable .ci-components/pg_perf_bench \
5661
--editable .ci-components/pg_stand \
@@ -71,6 +76,10 @@ jobs:
7176
with:
7277
repository: O2eg/pg_configurator
7378
path: .ci-components/pg_configurator
79+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
80+
with:
81+
repository: O2eg/pg_converter
82+
path: .ci-components/pg_converter
7483
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
7584
with:
7685
repository: O2eg/pg_diag
@@ -99,6 +108,7 @@ jobs:
99108
python -m venv "${RUNNER_TEMP}/pg-play-wheel-smoke"
100109
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/python" -m pip install \
101110
--editable .ci-components/pg_configurator \
111+
--editable .ci-components/pg_converter \
102112
--editable .ci-components/pg_diag \
103113
--editable .ci-components/pg_perf_bench \
104114
--editable .ci-components/pg_stand \
@@ -107,6 +117,6 @@ jobs:
107117
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/python" -m pip check
108118
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/pg-play" --version
109119
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/pg-play" capabilities
110-
for command in pg-configurator pg-diag pg-perf-bench pg-stand pg-workload; do
120+
for command in pg-configurator pg-converter pg-diag pg-perf-bench pg-stand pg-workload; do
111121
"${RUNNER_TEMP}/pg-play-wheel-smoke/bin/${command}" --version
112122
done

‎README.md‎

Lines changed: 79 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ experiments.
1414
collection as a successful complete run.
1515
- Plans and runs `pg_perf_bench` only against an explicitly selected disposable
1616
database, then validates and compares benchmark artifacts and TPS evidence.
17+
- Plans and runs reviewed `pg_converter` packets against an exact alias set
18+
through a detached, auditable, sequential workflow.
1719
- Installs every component while preserving their independent use.
1820

1921
The independently installable components are:
@@ -25,7 +27,9 @@ The independently installable components are:
2527
- [`pg_diag`](https://github.com/O2eg/pg_diag) — diagnostic JSON and HTML
2628
artifacts;
2729
- [`pg_configurator`](https://github.com/O2eg/pg_configurator) — version-aware
28-
PostgreSQL configuration candidates.
30+
PostgreSQL configuration candidates;
31+
- [`pg_converter`](https://github.com/O2eg/pg_converter) — reviewed SQL/Python
32+
packet deployment and migration operations;
2933
- [`pg_perf_bench`](https://github.com/O2eg/pg_perf_bench) — controlled pgbench
3034
execution and environment evidence.
3135

@@ -43,13 +47,13 @@ The independently installable components are:
4347
pg_stand ------------------------+
4448
|
4549
v
46-
PostgreSQL stand
47-
/ | \
48-
v v v
49-
pg_workload pg_perf_bench pg_diag
50-
| | |
51-
| benchmark report |
52-
+--------- run ----------+----> diagnostic report
50+
PostgreSQL stand
51+
/ / \ \
52+
v v v v
53+
pg_converter pg_workload pg_perf_bench pg_diag
54+
| | | |
55+
packet run | benchmark report |
56+
+----------+------ run -----------+----> diagnostic report
5357
|
5458
change one reviewed input
5559
|
@@ -62,7 +66,7 @@ The control layers are deliberately separate:
6266
agent skills workflow and interpretation rules
6367
|
6468
v
65-
pg-play-mcp twenty-six typed, high-level operations
69+
pg-play-mcp thirty-one typed, high-level operations
6670
|
6771
v
6872
pg_play core validation, planning, state, comparison
@@ -71,6 +75,7 @@ pg_play core validation, planning, state, comparison
7175
component adapters argv arrays + strict JSON envelopes
7276
|
7377
+---- pg_configurator
78+
+---- pg_converter
7479
+---- pg_stand
7580
+---- pg_workload
7681
+---- pg_diag
@@ -87,14 +92,15 @@ are hidden from its primary help and its normal human output is unchanged.
8792
python -m pip install pg-play
8893
```
8994

90-
This installs compatible versions of all five component distributions. They
95+
This installs compatible versions of all six component distributions. They
9196
remain available through their own commands:
9297

9398
```bash
9499
pg-stand --help
95100
pg-workload --help
96101
pg-diag --help
97102
pg-configurator --help
103+
pg-converter --help
98104
pg-perf-bench --help
99105
```
100106

@@ -208,7 +214,7 @@ The packaged JSON Schema is available as the MCP resource
208214
Secrets are the deliberate exception: `pg_workload` continues to accept
209215
passwords only through environment/passfile mechanisms.
210216

211-
All five components use the same hidden orchestration options:
217+
All six components use the same hidden orchestration options:
212218
`--machine`, `--request-id`, and `--component-capabilities`. Their advertised
213219
`machine_interface` object makes these names machine-verifiable.
214220

@@ -219,6 +225,11 @@ The `pg-play` CLI contains only complete experiment operations:
219225
| Command | Effect |
220226
| --- | --- |
221227
| `capabilities` | Read installed component contracts |
228+
| `plan-converter-run --project DIR --config FILE --packet NAME` | Resolve packet content and exact database aliases without connecting |
229+
| `start-converter-run PLAN.json --plan-hash HASH --out DIR --run-id ID` | Start an unchanged packet plan in a detached worker |
230+
| `converter-run-status RUN_DIR` | Read durable packet-run state and detect a lost worker |
231+
| `converter-run-events RUN_DIR [--after-sequence N]` | Read ordered packet-run events |
232+
| `cancel-converter-run RUN_DIR [--reason TEXT]` | Request cooperative packet cancellation |
222233
| `validate MANIFEST` | Validate the manifest and non-mutating component inputs |
223234
| `plan MANIFEST` | Calculate the current read-only plan and its hash |
224235
| `start MANIFEST --plan-hash HASH --run-id ID` | Start exactly that plan in a detached worker and return immediately |
@@ -262,6 +273,40 @@ for a verified failed, cancelled, or interrupted attempt; use a new id when the
262273
manifest or plan changes. `run` retains the old synchronous behavior for
263274
scripts that explicitly need it.
264275

276+
### Reviewed pg_converter packet runs
277+
278+
`pg_converter` packets are trusted deployment code and are intentionally
279+
separate from the general experiment manifest. Plan an exact packet, review
280+
its legacy tracker checksum, SHA-256 source-tree hash, Python-step flag, and
281+
resolved database aliases and password-free connection identities, then start
282+
the unchanged plan:
283+
284+
```bash
285+
pg-play plan-converter-run \
286+
--project /opt/pg_converter/current \
287+
--config /etc/pg_converter/pg_converter.conf \
288+
--packet release_42 \
289+
--database-selector 'prod_a,prod_b' \
290+
--placeholders-file /run/pg_converter/release_42.json \
291+
--timeout-seconds 3600 > converter-plan.json
292+
293+
pg-play start-converter-run converter-plan.json \
294+
--plan-hash sha256:... \
295+
--out /var/lib/pg_play/converter-runs \
296+
--run-id release-42
297+
```
298+
299+
The machine path always uses sequential database processing and never enables
300+
`--force`, `--wipe`, `--stop`, `--unlock`, template copying, or skip-on-cancel
301+
modes. The reviewed plan replaces only the interactive multi-database prompt;
302+
it does not bypass PG Converter checksum or ambiguous-outcome safeguards.
303+
Credentials stay in the protected configuration file. Placeholder and
304+
configuration-override values are accepted only through protected JSON files
305+
and are represented in the plan by names and hashes, not values.
306+
See the
307+
[pg_converter integration runbook](https://github.com/O2eg/pg_play/blob/main/docs/pg_converter-integration.md)
308+
for the durable directory, cancellation, and recovery contracts.
309+
265310
## MCP server
266311

267312
Start the stdio server with:
@@ -278,6 +323,11 @@ for concise MCP and Agent Skills configuration examples for Codex CLI, Claude
278323
Code, Hermes Agent, Kimi Code CLI, Gemini CLI, and OpenCode.
279324

280325
- `component_capabilities`
326+
- `plan_converter_run`
327+
- `start_converter_run`
328+
- `converter_run_status`
329+
- `converter_run_events`
330+
- `cancel_converter_run`
281331
- `plan_live_diagnostics`
282332
- `start_live_diagnostics`
283333
- `live_diagnostics_status`
@@ -314,11 +364,20 @@ and intentionally excludes the 2.x prerelease API.
314364
The durable schemas are also exposed as `pgplay://run-state-schema` and
315365
`pgplay://run-event-schema`.
316366

367+
For packet deployment, an agent must call `plan_converter_run`, show the exact
368+
packet/source hashes, target aliases, Python-step risk, and mutation flags to
369+
the user, and pass the unchanged plan and hash to `start_converter_run`.
370+
Observe it through `converter_run_status` and `converter_run_events`. A failed
371+
or interrupted packet is not automatically retried because non-transactional
372+
actions can have an ambiguous outcome.
373+
317374
## Agent skills
318375

319-
The wheel contains five optional workflow skills under `pg_play/skills/`:
376+
The wheel contains six optional workflow skills under `pg_play/skills/`:
320377

321378
- `run-postgres-experiment` — validate, plan, start, and monitor a new run;
379+
- `run-pg-converter-packet` — review exact packet sources and targets, start
380+
the immutable run, and preserve ambiguous outcomes for operator recovery;
322381
- `recover-postgres-experiment` — inspect, cancel, and safely resume an
323382
existing durable run;
324383
- `diagnose-live-postgres` — capture a bounded, read-only diagnostic window on
@@ -431,6 +490,8 @@ unknown step or changed core artifact blocks recovery.
431490
combined plan hash. `start` and `resume` enforce the same hash. Machine-mode
432491
`pg_perf_bench benchmark` independently
433492
verifies a content-sensitive benchmark plan hash before resetting its database.
493+
- `pg_converter run` independently verifies the reviewed component plan hash,
494+
including the exact alias set, packet source tree, and file-based inputs.
434495
- Subprocesses receive argument arrays with `shell=False`.
435496
- Password-bearing CLI arguments and password-bearing machine output are
436497
rejected.
@@ -456,22 +517,25 @@ read-only profiles and one bounded snapshots window per immutable capture id.
456517
Configuration review extracts the bounded CPU, RAM, filesystem, mount, and disk
457518
facts listed above. Generation or application of TuneD/systemd artifacts remains
458519
a roadmap item; it is not silently approximated by the current implementation.
520+
Reviewed `pg_converter` runs are a separate durable workflow and are not
521+
silently injected into every experiment manifest.
459522

460523
## Development
461524

462525
```bash
463526
python3 -m venv .venv
464527
.venv/bin/pip install -e ../pg_stand -e ../pg_workload -e ../pg_diag \
465-
-e ../pg_configurator -e ../pg_perf_bench
528+
-e ../pg_configurator -e ../pg_converter -e ../pg_perf_bench
466529
.venv/bin/pip install -e '.[dev]'
467530
.venv/bin/ruff check .
468531
.venv/bin/ruff format --check .
469532
.venv/bin/pytest
470533
```
471534

472535
For coordinated releases, publish the component distributions before tagging
473-
`pg_play`: first `pg_configurator`, `pg_diag`, `pg_stand`, and `pg_workload`,
474-
then `pg_perf_bench` (which depends on `pg_diag`), and finally `pg_play`.
536+
`pg_play`: first `pg_configurator`, `pg_converter`, `pg_diag`, `pg_stand`, and
537+
`pg_workload`, then `pg_perf_bench` (which depends on `pg_diag`), and finally
538+
`pg_play`.
475539
Ordinary branch CI checks out the component sources so a coordinated source
476540
change can be tested before those versions reach PyPI. Tagged publish jobs use
477541
the package index deliberately and therefore enforce this release order.

‎docs/pg_converter-integration.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# pg_converter packet runs through pg_play
2+
3+
`pg_play` exposes a bounded, typed workflow for reviewed `pg_converter`
4+
packets. It does not expose raw component arguments, SQL, Python source,
5+
tracker deletion, backend termination, or `--force` through MCP.
6+
7+
## Prerequisites
8+
9+
- `pg_converter` and `pg_play` are installed in the same environment;
10+
- the packet project is available on the MCP server host;
11+
- the INI configuration and optional JSON input files are readable only by the
12+
runtime account;
13+
- packet SQL/Python and its source-control revision were reviewed separately;
14+
- the selected role has only the PostgreSQL privileges the packet requires.
15+
16+
The MCP server uses local file references. It does not accept a DSN, password,
17+
packet body, SQL string, or Python source in a tool call.
18+
19+
## Plan
20+
21+
Call:
22+
23+
```text
24+
plan_converter_run(
25+
project_directory,
26+
config_file,
27+
packet_name,
28+
database_selector="ALL",
29+
placeholders_file=null,
30+
config_overrides_file=null,
31+
timeout_seconds=3600
32+
)
33+
```
34+
35+
The operation invokes only the non-connecting `pg_converter --plan` path.
36+
The result contains:
37+
38+
- `pg_converter/plan-v1` with the exact alias set and password-free connection
39+
identities;
40+
- the v1-compatible MD5 tracker checksum;
41+
- the SHA-256 packet-tree hash, including filenames and nested data;
42+
- step names and an explicit Python-step risk flag;
43+
- safe configuration and protected input-file hashes without credentials or
44+
values;
45+
- a `pg_play/converter-run-plan-v1` wrapper and final `plan_hash`.
46+
47+
Review at least the packet name/type, both hashes, database aliases,
48+
`execute_sql`, Python-step flag, tracker-initialization flag, and timeout.
49+
50+
## Start and observe
51+
52+
Pass the unchanged plan object and hash:
53+
54+
```text
55+
start_converter_run(plan, plan_hash, output_directory, run_id)
56+
```
57+
58+
`run_id` is immutable and may contain letters, digits, `_`, `-`, and `.`. The
59+
new run directory is created with mode `0700`; files use `0600`.
60+
61+
The call returns after starting a detached worker. Continue with:
62+
63+
```text
64+
converter_run_status(run_directory)
65+
converter_run_events(run_directory, after_sequence=0, limit=1000)
66+
```
67+
68+
Use the returned `last_sequence` as the next cursor. Terminal states are
69+
`succeeded`, `partial`, `failed`, and `cancelled`. A vanished worker becomes
70+
`interrupted`.
71+
72+
The run directory contains:
73+
74+
```text
75+
RUN_ID/
76+
|-- plan.json
77+
|-- state.json
78+
|-- events.jsonl
79+
|-- result.json
80+
|-- worker.log
81+
|-- active-process.json # only while the component is active
82+
`-- cancel.request.json # only after cancellation is requested
83+
```
84+
85+
`result.json` contains the common component envelope and compact
86+
`pg_converter/machine-result-v1`. Raw step result sets and connection strings
87+
are not copied into durable pg_play state. Machine execution forces SQL logging
88+
off. Export paths are also withheld because the legacy random-password export
89+
mode can embed its password in a filename; use the per-database export counts
90+
and inspect the protected packet directory as an operator.
91+
92+
## Cancellation and recovery
93+
94+
Call `cancel_converter_run(run_directory, reason)` only for an active run and
95+
provide a concise audit reason. `pg_play` records the request and terminates
96+
only the process whose PID start time, operating-system owner, and executable
97+
identity match the active-process record.
98+
99+
There is deliberately no automatic resume tool for packet runs. Transactional
100+
actions retain PG Converter's tracker guarantees, but a non-transactional or
101+
maintenance action can finish with an ambiguous outcome. After `failed`,
102+
`cancelled`, or `interrupted`:
103+
104+
1. inspect `state.json`, `events.jsonl`, `worker.log`, and target PostgreSQL;
105+
2. follow the PG Converter `outcome_unknown` and lock recovery runbooks;
106+
3. resolve target state explicitly;
107+
4. build and review a new plan before starting a new immutable run id.
108+
109+
MCP cancellation does not authorize `wipe`, `unlock`, `stop`, or `force`.
110+
Those remain direct operator actions.
111+
112+
## Agent Skill
113+
114+
The wheel packages `run-pg-converter-packet`. It restricts an agent to the
115+
five typed converter tools, requires explicit authorization after showing the
116+
resolved endpoints and source hashes, and hands partial, failed, cancelled, or
117+
interrupted outcomes to an operator instead of retrying them.

‎pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "pg-play"
7-
version = "0.2.1"
7+
version = "0.3.0"
88
description = "MCP orchestrator for reproducible PostgreSQL performance experiments"
99
readme = "README.md"
1010
requires-python = ">=3.10"
@@ -31,6 +31,7 @@ classifiers = [
3131
dependencies = [
3232
"mcp>=1.28,<2",
3333
"pg-configurator>=0.9.1,<1",
34+
"pg-converter>=2.0b2,<3",
3435
"pg-diag>=0.10.4,<1",
3536
"pg-perf-bench>=0.2,<1",
3637
"pg-stand>=0.2.1,<1",

0 commit comments

Comments
 (0)