Skip to content

Commit 724da03

Browse files
committed
feat: release v0.6.0 with managed PostgreSQL schema resets
Add SQL-only managed mode and reset profile schemas inside an existing database. Preserve the session commit policy by default, isolate loader settings, and require physical replica replay before the workload. Document managed and Patroni comparisons, add regression coverage and a deployment JOIN scenario, and ship offline guides with verified links in wheel and source distributions.
1 parent 0e7048b commit 724da03

49 files changed

Lines changed: 2201 additions & 237 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/publish.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ jobs:
4949
run: python -m build
5050
- name: Check distribution metadata and README rendering
5151
run: python -m twine check dist/*
52+
- name: Check documentation in wheel and source distribution
53+
run: python tests/packaging/check_docs.py dist/*.whl dist/*.tar.gz
5254
- name: Smoke-test built wheel
5355
shell: bash
5456
run: |

‎INITIALIZATION.md‎

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,15 +14,31 @@ selects it. Arbitrary old commands remain supported without SQL rewriting.
1414
| `--init-batch-rows` | `100000` | Target rows per committed data batch |
1515
| `--init-table-mode` | `unlogged` | Stored tables become UNLOGGED before data, then LOGGED afterward; `logged` skips conversion |
1616
| `--init-fsync` | `off` | Temporarily disable fsync on the selected primary; `keep` preserves it |
17-
| `--init-synchronous-commit` | `off` | Session setting in every loader connection; alternatives: `local`, `keep` |
17+
| `--init-synchronous-commit` | `keep` | Preserve the session commit policy; explicit `off` or `local` opts into faster loading |
1818

1919
The same scheduler runs bounded data batches and later builds indexes, including
2020
unique indexes used for primary keys. It keeps at most `init-workers` jobs in
2121
flight, regardless of dataset size. A batch commits independently; the loader
2222
does not create one transaction for the entire dataset. Progress logs report
2323
committed rows and batches per task.
2424

25-
Order for each fresh database:
25+
Before each iteration, `--reset-mode database` (default) recreates the disposable
26+
database. `--reset-mode schema` resets the schemas in `LoadPlan.schemas` inside
27+
a pre-created dedicated database; it requires fast initialization and `--init-fsync keep`.
28+
It uses the target database for version checks, the controller connection, reset,
29+
loader and replay barrier. No connection to `postgres` is made in this mode.
30+
Use `--managed` to disable host access and server lifecycle operations explicitly;
31+
an optional `--managed-pg-info FILE` embeds provider metadata and also implies this flag.
32+
33+
Schema reset checks CREATE privilege, schema ownership, read-write status and
34+
UNLOGGED/LOGGED conversion before deleting existing schemas. It refuses `public`,
35+
system schemas and system databases. A session advisory lock is held through the
36+
workload to prevent another schema-mode run from resetting the same database.
37+
Direct connections and compatible session pooling are supported; transaction/statement pooling
38+
are not. See the [pool requirements](doc/managed_mode_usage.md#connection-pooling). Reset does not alter database ownership, ACLs or permanent role settings.
39+
Use a dedicated database: CASCADE also removes objects depending on the profile schemas.
40+
41+
Order for each fresh dataset:
2642

2743
1. Create the schema without indexes or foreign keys; make stored tables UNLOGGED.
2844
2. Run data tasks in bounded batches, respecting their declared dependencies.
@@ -31,8 +47,8 @@ Order for each fresh database:
3147
5. Build indexes in parallel using the same worker limit and scheduler.
3248
6. Attach constraints and validate foreign keys, serially to avoid lock-order deadlocks.
3349
7. Complete profile setup and run `VACUUM (FREEZE, ANALYZE)` on stored tables and
34-
materialized views, then `ANALYZE`.
35-
8. Restore fsync, checkpoint and synchronize files on the database host.
50+
materialized views, then `ANALYZE` the profile relations, including partitioned parents.
51+
8. Restore fsync, checkpoint and synchronize files on the database host if fsync was changed.
3652
9. Wait until directly connected physical replicas replay the initialization WAL.
3753
10. Capture **Before workload** sizes and start pgbench with the original commit policy.
3854

@@ -41,7 +57,22 @@ host command, and replica wait. It does not bound the sum of all data batches.
4157
For large datasets, increase it to cover the longest table rewrite/index build
4258
and the time replicas need to catch up. A timeout or failed phase aborts the run
4359
before pgbench. Failed batches are not automatically retried; the next run
44-
recreates the disposable database.
60+
resets the database or profile schemas according to `--reset-mode`.
61+
62+
The common loader sets `search_path` from `LoadPlan.schemas` through SQL after
63+
opening each asyncpg connection. Explicit loader commit overrides are applied and
64+
verified the same way. Both settings are restored before returning connections to
65+
a session pool, including after errors and cancellation. Advisory locks are
66+
explicitly released. Diagnostic sessions reset their read-only mode and timeouts
67+
before returning to the pool. pgbench receives `search_path` through its child
68+
`PGOPTIONS`, preserving other caller options. In schema mode, a read-only psql
69+
probe checks the same libpq options before any schema reset; read-only pgbench
70+
probes also check prepared-statement cleanup. Incompatible pools are rejected
71+
with a configuration error before the existing dataset is removed. Bundled fast
72+
profiles do not change database/role search_path.
73+
The workload does not inherit the loader's temporary synchronous_commit value.
74+
Legacy Pagila initialization retains its existing persistent search_path setup
75+
and requires database reset; arbitrary legacy commands do not support schema reset.
4576

4677
## Durability and replication
4778

@@ -78,9 +109,14 @@ without host identity require the original SQL address; if it has changed, the
78109
run stops and reports the journal path for recovery on the original server.
79110

80111
Patroni's synchronous mode, DCS configuration and `synchronous_standby_names`
81-
remain unchanged. `synchronous_commit=off` in loader sessions removes waits for
82-
synchronous replica acknowledgements. It does **not** eliminate the WAL needed
83-
to convert tables to LOGGED or build their indexes. UNLOGGED contents are not
112+
remain unchanged. By default, `--init-synchronous-commit keep` leaves each loader
113+
session's commit policy untouched. Explicit `--init-synchronous-commit off` removes
114+
waits for local WAL flush and synchronous replica acknowledgements during preparation.
115+
`--init-synchronous-commit local` keeps local WAL flush waits but removes synchronous
116+
replica acknowledgement waits. These options affect only the loader; pgbench uses
117+
its original commit policy, and the mandatory replay barrier runs with every option.
118+
Changing commit policy does not eliminate the WAL needed to convert tables to LOGGED
119+
or build their indexes. UNLOGGED contents are not
84120
sent to replicas during data generation; conversion rewrites each table and
85121
generates its WAL. Budget disk space for a full extra copy of the largest table,
86122
WAL and index-building temporary files. Conversion of a single table remains
@@ -98,7 +134,9 @@ including VACUUM, fsync restoration and CHECKPOINT. Every required replica must
98134
no WAL from data preparation may remain unapplied when pgbench starts.
99135
Cascaded replicas and logical subscribers are not covered by this
100136
primary-side physical replay barrier. A restricted SQL role needs visibility of
101-
replica statistics (`pg_read_all_stats` or `pg_monitor`) when replicas are connected.
137+
replica statistics (`pg_read_all_stats`, `pg_monitor` or equivalent provider privileges)
138+
when replicas are connected. WAL-function access is checked at preflight too;
139+
insufficient permissions stop the run before reset/loading.
102140

103141
Initialization options, per-phase durations, committed row/batch counts, final
104142
fsync and the replay barrier are included in JSON and HTML reports. Options

‎MANIFEST.in‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,4 @@
11
include INITIALIZATION.md
2+
recursive-include doc *.md
3+
include tests/benchmark/README.md
4+
include tests/packaging/check_docs.py

0 commit comments

Comments
 (0)