|
| 1 | +# pg_maker and pg_play |
| 2 | + |
| 3 | +Install the optional integration from PyPI: |
| 4 | + |
| 5 | +```bash |
| 6 | +python -m pip install --upgrade 'pg-play[maker]' |
| 7 | +pg-play capabilities --component pg_maker |
| 8 | +``` |
| 9 | + |
| 10 | +For local pg_maker development, install its checkout into the same environment: |
| 11 | + |
| 12 | +```bash |
| 13 | +pg_play/.venv/bin/python -m pip install -e ./pg_maker |
| 14 | +``` |
| 15 | + |
| 16 | +## Adapter contract |
| 17 | + |
| 18 | +`PgPlayService.maker_*` and the corresponding MCP tools invoke the separate |
| 19 | +`pg-maker` CLI with argument vectors. pg_maker owns its build workers, Git trees, |
| 20 | +patch series and caches. Preparation, resolution, build and runtime creation return |
| 21 | +after starting a durable worker. Use `maker_status` and `maker_events` to observe |
| 22 | +it, and `maker_cancel` to cancel it, including after a client disconnect. |
| 23 | + |
| 24 | +Preparation and resolution accept `config_dir`, `features`, `build_mode`, |
| 25 | +`with_doc` and `doc_languages` in Python and MCP. CLI options are |
| 26 | +`--config-dir`, `--feature`, `--build-mode`, `--with-doc`/`--without-doc` |
| 27 | +and `--doc-language`. |
| 28 | + |
| 29 | +Development operations are `maker_dev_open`, `maker_dev_status`, |
| 30 | +`maker_dev_patch`, `maker_dev_explain`, `maker_dev_build`, `maker_dev_exec`, |
| 31 | +`maker_dev_pin` and `maker_dev_prune`. The CLI uses matching hyphenated names: |
| 32 | + |
| 33 | +```bash |
| 34 | +pg-play maker dev-open ./work --from-run /path/to/operations/RUN |
| 35 | +pg-play maker dev-patch ./work commit --scope component:pg-wait-sampling --id fix-build |
| 36 | +pg-play maker dev-explain ./work --component pg-wait-sampling |
| 37 | +pg-play maker dev-build ./work --component pg-wait-sampling |
| 38 | +pg-play maker dev-exec ./work --component pg-wait-sampling -- make -n |
| 39 | +pg-play maker dev-prune ./work --keep-last 2 --out ./cleanup.json |
| 40 | +pg-play maker dev-prune ./work --keep-last 2 --apply ./cleanup.json |
| 41 | +``` |
| 42 | + |
| 43 | +Build and debug commands return an operation directory immediately. Observe or |
| 44 | +cancel them through the regular maker operation methods. A new development |
| 45 | +iteration freezes a new lock; `resume` retains the previous inputs and plan. |
| 46 | +Debug commands run inside a private container copy of an iteration's workspace. |
| 47 | +Pruning preserves the current and pinned iterations and requires an unchanged |
| 48 | +cleanup plan. It does not run a global Docker prune. |
| 49 | + |
| 50 | +`maker_plan(lock, input_store?, goal="package", run_db_tests=False)` returns |
| 51 | +`pg_play/maker-plan-v1`: copies of the BuildLock and BuildPlan, parameters and an |
| 52 | +overall `plan_hash`. `maker_start(plan, plan_hash, operations, operation_id, |
| 53 | +cache_store?)` checks that hash, stores the reviewed inputs under |
| 54 | +`.pg-play-plans/OPERATION/`, passes the original maker plan hash and returns |
| 55 | +`operation_directory`. Reusing an operation ID with a different plan is rejected. |
| 56 | +Resume uses the maker build plan hash, or the input hash for preparation/resolution. |
| 57 | +The wrapper plan hash and the maker plan hash are distinct. |
| 58 | + |
| 59 | +`maker_artifacts` lists verified artifact references, manifests, verification records |
| 60 | +and reports. `maker_export` copies a finished operation to a new directory. |
| 61 | +Ordinary run cleanup is available through pg_maker's CLI/Python API; development |
| 62 | +sessions have the separate `maker_dev_prune` operation. |
| 63 | + |
| 64 | +The CLI exposes `pg-play maker validate|prepare|resolve|plan|start|status|events| |
| 65 | +cancel|resume|artifacts|export|runtime-plan|runtime-start`. After preparation and |
| 66 | +resolution: |
| 67 | + |
| 68 | +```bash |
| 69 | +pg-play maker plan build.lock.json --goal package > reviewed-maker.json |
| 70 | +pg-play maker start reviewed-maker.json --plan-hash SHA256_FROM_REVIEWED_PLAN \ |
| 71 | + --operations ./maker-operations --operation-id candidate |
| 72 | +pg-play maker status ./maker-operations/candidate |
| 73 | +pg-play maker artifacts ./maker-operations/candidate |
| 74 | +``` |
| 75 | + |
| 76 | +## Runtime image for pg_stand |
| 77 | + |
| 78 | +The initial profile targets Debian 13 / x86_64 with server, client and libpq DEBs |
| 79 | +under a custom `/opt/...` prefix. `maker_runtime_plan(artifacts, base_image, |
| 80 | +output)` records hashes of packages, the local runtime repository and entrypoint |
| 81 | +files. `base_image` must be a prepared local Debian 13 image identified by |
| 82 | +`sha256:...`, containing Python 3, gosu, openssh-server, PgBouncer, iproute2, |
| 83 | +procps, lshw, sysstat, passwd, util-linux, locales and certificates. Match PgBouncer |
| 84 | +to the installed pg_stand profile. A custom platform recipe can add these packages |
| 85 | +to `environments.test.packages`; use the resulting image ID from `maker_prepare`. |
| 86 | + |
| 87 | +Runtime construction is offline and does not run initdb or start PostgreSQL. It |
| 88 | +checks libraries with ldd. The package payload must contain pg_stat_statements, |
| 89 | +pg_buffercache, pg_wait_sampling, pg_stat_kcache and auto_explain under the same |
| 90 | +prefix. Create a separate stand feature with these diagnostic components. PGXS |
| 91 | +uses maker's internal headers and pg_config; headers and PGXS are excluded from |
| 92 | +the delivered packages. Extensions for a different PostgreSQL prefix do not |
| 93 | +satisfy this contract. |
| 94 | + |
| 95 | +Pinned starting recipes are in [examples/maker](../examples/maker). Copy the five |
| 96 | +component recipes into the maker project's `components/` directory and |
| 97 | +`my-feature-stand.yaml` into `config/features/`, then select the feature in |
| 98 | +`build.yaml`. Previous local validation compiled the recipes against PostgreSQL |
| 99 | +18.1 and checked InstallTree/DEB packaging and offline runtime construction with |
| 100 | +`goal=package`, without documentation or database tests. Control and SQL files |
| 101 | +were included. pg_stat_statements uses its native check target because upstream |
| 102 | +does not support installcheck. CREATE EXTENSION and native database suites remain |
| 103 | +separate acceptance checks. |
| 104 | + |
| 105 | +The profile reuses the installed pg_stand entrypoint and PgBouncer helper, hashing |
| 106 | +their contents. Updates require a new plan. The custom bootstrap uses maker's |
| 107 | +binaries, the postgres account, PGDATA/POSTGRES_* and pg_stand's WAL/log mounts. |
| 108 | +Configuration stays in PGDATA. The Unix socket directory is fixed at |
| 109 | +`/var/run/postgresql`; do not override it in stand parameters. |
| 110 | + |
| 111 | +The result records the immutable image ID, build/runtime plan hashes, artifact |
| 112 | +manifest hash, prefix, version and original qualification. Creating an image does |
| 113 | +not raise its qualification. Publishing or exporting that local Docker image is a |
| 114 | +separate operation. |
| 115 | + |
| 116 | +## Maker phase in an experiment |
| 117 | + |
| 118 | +Add an optional section to the experiment manifest: |
| 119 | + |
| 120 | +```yaml |
| 121 | +spec: |
| 122 | + maker: |
| 123 | + plan: reviewed-maker.json |
| 124 | + plan_hash: sha256:... # pg_play/maker-plan-v1 hash |
| 125 | + variant: my-feature-stand.debian13-x64.candidate |
| 126 | + runtime_base_image: sha256:... # prepared image with pg_stand utilities |
| 127 | + timeout_seconds: 7200 |
| 128 | +``` |
| 129 | +
|
| 130 | +Execution order is maker build, package verification, runtime image, pg_stand, then |
| 131 | +the existing benchmark/workload/diagnostic phases. Planning uses an image reference |
| 132 | +derived from the reviewed inputs. Execution checks the original configuration, |
| 133 | +substitutes the produced immutable image ID and records the resulting stand plan |
| 134 | +in `RUN/maker/stand-plan.json`. Other settings retain their reviewed values and |
| 135 | +`pull_policy=never`. The new image ID changes the desired-state hash. The original |
| 136 | +stand YAML is not overwritten. |
| 137 | + |
| 138 | +`RUN/maker/active.json` is written before launching a worker so cancellation can |
| 139 | +find it after coordinator loss. Resume reuses successful operations and explicitly |
| 140 | +resumes failed ones after checking the maker plan hash. A timeout cancels the |
| 141 | +current maker operation. `RUN/maker/provenance.json` and `state.maker` retain |
| 142 | +qualification, database-test status and the plan, package and image hashes. |
| 143 | +A successful build with `db_tests=skipped` remains `unverified`. |
| 144 | + |
| 145 | +For baseline/candidate comparisons without long database suites, create two |
| 146 | +features with the same diagnostic components, one with the ChangeSet and one |
| 147 | +without it. Use separate manifests and stand project/storage paths, run the |
| 148 | +experiments sequentially and release each active stand through teardown. Compare |
| 149 | +their reports with the existing report comparison commands. Maker's built-in |
| 150 | +baseline branch is included only when database tests are enabled. |
| 151 | + |
| 152 | +Full database/workload execution is a separate acceptance step. Unit/contract |
| 153 | +tests and runtime image construction do not establish all topology/SQL behavior. |
| 154 | + |
| 155 | +## Source-editing workflow |
| 156 | + |
| 157 | +1. Resolve freezes a source snapshot in InputStore. `maker_changes_workspace` |
| 158 | + creates a separate editable `source/` tree. Supply `lock` and `feature` to |
| 159 | + include common and preceding ChangeSets; `until=CHANGE_ID/OPERATION_ID` selects |
| 160 | + the point before that operation. Without a lock, the base is raw upstream. |
| 161 | +2. Edit that copy, use `maker_changes_export` to create a ChangeSet and diff, and |
| 162 | + check its structure/files with `maker_changes_validate`. ABI, data and WAL |
| 163 | + compatibility remain unknown until separately established. |
| 164 | +3. Add the ChangeSet to a feature, resolve again and review the new plan hash. |
| 165 | + Build, run the experiment and compare it with the baseline. |
| 166 | +4. Documentation goals are `doc-source`, `translations`, `translated-docs` and |
| 167 | + `docs`. Translation bundles are checked for source identity and completeness; |
| 168 | + model credentials and calls remain outside maker. |
| 169 | +5. Export results before cleanup. Database build tests require an explicit |
| 170 | + `run_db_tests` selection; a quick successful build does not qualify a DB release. |
0 commit comments