You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 724da03
Browse filesBrowse the repository at this point in the historyBrowse files
feat: release v0.6.0 with managed PostgreSQL schema resets
Add SQL-only managed mode and reset profile schemas inside an existing database. Preserve the session commit policy by default, isolate loader settings, and require physical replica replay before the workload.
Document managed and Patroni comparisons, add regression coverage and a deployment JOIN scenario, and ship offline guides with verified links in wheel and source distributions.
Copy file name to clipboardExpand all lines: INITIALIZATION.md
+47-9Lines changed: 47 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,15 +14,31 @@ selects it. Arbitrary old commands remain supported without SQL rewriting.
14
14
|`--init-batch-rows`|`100000`| Target rows per committed data batch |
15
15
|`--init-table-mode`|`unlogged`| Stored tables become UNLOGGED before data, then LOGGED afterward; `logged` skips conversion |
16
16
|`--init-fsync`|`off`| Temporarily disable fsync on the selected primary; `keep` preserves it |
17
-
|`--init-synchronous-commit`|`off`|Session setting in every loader connection; alternatives: `local`, `keep`|
17
+
|`--init-synchronous-commit`|`keep`|Preserve the session commit policy; explicit `off` or `local` opts into faster loading|
18
18
19
19
The same scheduler runs bounded data batches and later builds indexes, including
20
20
unique indexes used for primary keys. It keeps at most `init-workers` jobs in
21
21
flight, regardless of dataset size. A batch commits independently; the loader
22
22
does not create one transaction for the entire dataset. Progress logs report
23
23
committed rows and batches per task.
24
24
25
-
Order for each fresh database:
25
+
Before each iteration, `--reset-mode database` (default) recreates the disposable
26
+
database. `--reset-mode schema` resets the schemas in `LoadPlan.schemas` inside
27
+
a pre-created dedicated database; it requires fast initialization and `--init-fsync keep`.
28
+
It uses the target database for version checks, the controller connection, reset,
29
+
loader and replay barrier. No connection to `postgres` is made in this mode.
30
+
Use `--managed` to disable host access and server lifecycle operations explicitly;
31
+
an optional `--managed-pg-info FILE` embeds provider metadata and also implies this flag.
32
+
33
+
Schema reset checks CREATE privilege, schema ownership, read-write status and
34
+
UNLOGGED/LOGGED conversion before deleting existing schemas. It refuses `public`,
35
+
system schemas and system databases. A session advisory lock is held through the
36
+
workload to prevent another schema-mode run from resetting the same database.
37
+
Direct connections and compatible session pooling are supported; transaction/statement pooling
38
+
are not. See the [pool requirements](doc/managed_mode_usage.md#connection-pooling). Reset does not alter database ownership, ACLs or permanent role settings.
39
+
Use a dedicated database: CASCADE also removes objects depending on the profile schemas.
40
+
41
+
Order for each fresh dataset:
26
42
27
43
1. Create the schema without indexes or foreign keys; make stored tables UNLOGGED.
28
44
2. Run data tasks in bounded batches, respecting their declared dependencies.
@@ -31,8 +47,8 @@ Order for each fresh database:
31
47
5. Build indexes in parallel using the same worker limit and scheduler.
32
48
6. Attach constraints and validate foreign keys, serially to avoid lock-order deadlocks.
33
49
7. Complete profile setup and run `VACUUM (FREEZE, ANALYZE)` on stored tables and
34
-
materialized views, then `ANALYZE`.
35
-
8. Restore fsync, checkpoint and synchronize files on the database host.
50
+
materialized views, then `ANALYZE` the profile relations, including partitioned parents.
51
+
8. Restore fsync, checkpoint and synchronize files on the database host if fsync was changed.
36
52
9. Wait until directly connected physical replicas replay the initialization WAL.
37
53
10. Capture **Before workload** sizes and start pgbench with the original commit policy.
38
54
@@ -41,7 +57,22 @@ host command, and replica wait. It does not bound the sum of all data batches.
41
57
For large datasets, increase it to cover the longest table rewrite/index build
42
58
and the time replicas need to catch up. A timeout or failed phase aborts the run
43
59
before pgbench. Failed batches are not automatically retried; the next run
44
-
recreates the disposable database.
60
+
resets the database or profile schemas according to `--reset-mode`.
61
+
62
+
The common loader sets `search_path` from `LoadPlan.schemas` through SQL after
63
+
opening each asyncpg connection. Explicit loader commit overrides are applied and
64
+
verified the same way. Both settings are restored before returning connections to
65
+
a session pool, including after errors and cancellation. Advisory locks are
66
+
explicitly released. Diagnostic sessions reset their read-only mode and timeouts
67
+
before returning to the pool. pgbench receives `search_path` through its child
68
+
`PGOPTIONS`, preserving other caller options. In schema mode, a read-only psql
69
+
probe checks the same libpq options before any schema reset; read-only pgbench
70
+
probes also check prepared-statement cleanup. Incompatible pools are rejected
71
+
with a configuration error before the existing dataset is removed. Bundled fast
72
+
profiles do not change database/role search_path.
73
+
The workload does not inherit the loader's temporary synchronous_commit value.
74
+
Legacy Pagila initialization retains its existing persistent search_path setup
75
+
and requires database reset; arbitrary legacy commands do not support schema reset.
45
76
46
77
## Durability and replication
47
78
@@ -78,9 +109,14 @@ without host identity require the original SQL address; if it has changed, the
78
109
run stops and reports the journal path for recovery on the original server.
79
110
80
111
Patroni's synchronous mode, DCS configuration and `synchronous_standby_names`
81
-
remain unchanged. `synchronous_commit=off` in loader sessions removes waits for
82
-
synchronous replica acknowledgements. It does **not** eliminate the WAL needed
83
-
to convert tables to LOGGED or build their indexes. UNLOGGED contents are not
112
+
remain unchanged. By default, `--init-synchronous-commit keep` leaves each loader
0 commit comments