Skip to content

Commit 7dcfd41

Browse files
committed
feat: release v0.3.0 with managed PostgreSQL benchmarks
Add managed instance metadata, complete workload evidence, summary metric charts and JOIN support. Expand IMDb and Pagila profiles, add Pagila HTAP, and improve report labels, units and documentation.
1 parent 4a1db60 commit 7dcfd41

90 files changed

Lines changed: 10116 additions & 1066 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 235 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ selected target. The workload commands themselves run on the machine where
6666
| `local` | local machine | local machine |
6767
| `docker` | existing container | local machine through a published port |
6868
| `ssh` | remote host | local machine through an SSH local-forwarding port |
69+
| `--managed-pg-info FILE` | PostgreSQL protocol only; no host transport | local machine through the managed endpoint |
6970

7071
This separation keeps workload generation independent of target management and
7172
makes the measured client location explicit.
@@ -253,8 +254,9 @@ commands. It must be longer than the expected command duration.
253254
| `ARG_PG_DATABASE` | `--database` |
254255
| `ARG_PGBENCH_PATH` | newest local pgbench, or validated `--pgbench-path` |
255256
| `ARG_PSQL_PATH` | matching local psql, or validated `--psql-path` |
256-
| `ARG_WORKLOAD_PATH` | `--workload-path` |
257+
| `ARG_WORKLOAD_PATH` | bundled profile directory, or `--workload-path` |
257258
| `ARG_WORKLOAD_SCALE` | `--workload-scale` |
259+
| `ARG_WORKLOAD_DURATION_SECONDS` | `--workload-duration-seconds`, or the profile's `default_duration_seconds` |
258260
| `ARG_PGBENCH_CLIENTS` | current client-axis value |
259261
| `ARG_PGBENCH_TIME` | current duration-axis value |
260262

@@ -265,16 +267,228 @@ For a custom workload, use `--benchmark-type custom`, supply an existing
265267
`--workload-path`, and reference files below that path from the command
266268
templates.
267269

268-
For a packaged maximum-TPS workload, use `--workload-profile imdb` or
269-
`--workload-profile pagila`, `--workload-scale SCALE`, and a
270-
`--pgbench-clients` sweep. `pg-perf-bench profiles` lists the installed
271-
collection. Each profile supplies its schema, deterministic Python generator,
272-
typical SQL query set and command templates. It deliberately does not reuse
273-
`pg_workload`'s scheduler-specific `profile.yml`.
270+
### Bundled workload profiles
271+
272+
`pg-perf-bench profiles` lists the installed profiles, their default scale,
273+
duration and script count. Select one with `--workload-profile`; its schema,
274+
generator, setup and workload commands are supplied automatically.
275+
276+
| Profile | Workload and default script weights | Default duration |
277+
|---|---|---:|
278+
| [`imdb`](src/pg_perf_bench/workload_profiles/imdb/README.md) | 38 analytical scripts over 21 tables; equal weights | 120 s |
279+
| [`pagila`](src/pg_perf_bench/workload_profiles/pagila/README.md) | OLTP: select / insert / update / delete = 50 / 25 / 20 / 5 | 60 s |
280+
| [`pagila-htap`](src/pg_perf_bench/workload_profiles/pagila-htap/README.md) | The same OLTP scripts plus reporting: 50 / 25 / 20 / 5 / 5 | 60 s |
281+
282+
Weights describe selection of whole pgbench scripts, each of which can execute
283+
several SQL statements or transactions. The default HTAP reporting share is
284+
`5 / 105`, approximately 4.8 %. Its reported TPS includes all five scripts.
285+
286+
| Setting | How to configure it |
287+
|---|---|
288+
| Data volume | `--workload-scale SCALE`, default `1`; positive fractional values such as `0.25` are accepted. Generators retain minimum table sizes at small scales. |
289+
| Concurrent clients | `--pgbench-clients 1,2,4,8,16`; each value gets a freshly recreated database. Bundled commands also use one pgbench job per client. |
290+
| Measured window per point | `--workload-duration-seconds 120`; overrides the profile default for every client count. |
291+
| Command time limit | `--command-timeout 300`; allow enough time for initialization and for the workload window plus completion of in-flight queries. |
292+
293+
Bundled profiles require `--pgbench-clients`; `--pgbench-time` is rejected.
294+
They select benchmark type `custom` automatically. `--workload-path` cannot be
295+
combined with `--workload-profile`.
296+
297+
For example, run the HTAP profile on a dedicated local benchmark instance
298+
(adjust connection and PostgreSQL paths to your environment):
299+
300+
```bash
301+
PGPASSWORD=secret pg-perf-bench benchmark \
302+
--connection-type local \
303+
--allow-database-reset \
304+
--host 127.0.0.1 --port 5432 --user postgres \
305+
--database pg_perf_bench_test \
306+
--pg-data-path /var/lib/postgresql/18/main \
307+
--pg-bin-path /usr/lib/postgresql/18/bin \
308+
--workload-profile pagila-htap \
309+
--workload-scale 4 \
310+
--workload-duration-seconds 120 \
311+
--pgbench-clients 1,2,4,8,16 \
312+
--command-timeout 300 \
313+
--report-name pagila-htap-scale4
314+
```
315+
316+
Use `--workload-profile pagila` for the OLTP baseline, or `imdb` for analytical
317+
joins. Keep scale, client counts and duration constant when comparing runs.
318+
Scale controls row counts, not a target size in bytes: IMDb scale 1 generates
319+
100,000 titles and 100,000 people; Pagila scale 1 generates 1,000 films,
320+
600 customers and 16,000 rentals. Choose data volume relative to the cache
321+
being tested; the profile READMEs describe the datasets in more detail.
322+
323+
The schemas support PostgreSQL 10–18. Generators use deterministic `hashint8()`
324+
streams; IMDb uses separate streams for related identifiers and attributes to
325+
avoid correlated, degenerate joins. Pagila initializes indexes, identifier
326+
bounds and statistics, and sets `search_path` for the database and the benchmark
327+
role within it. No manual role configuration is needed for the supplied commands.
328+
Generated values are reproducible; service timestamps such as Pagila's
329+
`last_update` use the current time.
330+
331+
Bundled commands use `--random-seed=42`, and both Pagila variants use
332+
`-M prepared`. A fixed seed makes random choices repeatable with the same
333+
client configuration, but a timed run can complete a different number of
334+
scripts; it does not guarantee identical observed mix proportions or TPS.
335+
These profiles use `profile.json`, independently of `pg_workload`'s
336+
scheduler-specific `profile.yml`.
337+
338+
### Changing script weights or pgbench options
339+
340+
Keep `--workload-profile` and supply `--workload-command` to replace only its
341+
measured command. The packaged initialization still runs, and
342+
`ARG_WORKLOAD_PATH` still points to the packaged profile directory.
343+
`--init-command` can similarly replace the initialization command.
344+
345+
For example, append this option to the HTAP command above to change the
346+
reporting weight from 5 to 25, making its target share `25 / 125 = 20 %`:
347+
348+
```bash
349+
--workload-command 'ARG_PGBENCH_PATH --no-vacuum --random-seed=42 -M prepared -c ARG_PGBENCH_CLIENTS -j ARG_PGBENCH_CLIENTS -T ARG_WORKLOAD_DURATION_SECONDS -h ARG_PG_HOST -p ARG_PG_PORT -U ARG_PG_USER -f ARG_WORKLOAD_PATH/sql/01_select.sql@50 -f ARG_WORKLOAD_PATH/sql/02_insert.sql@25 -f ARG_WORKLOAD_PATH/sql/03_update.sql@20 -f ARG_WORKLOAD_PATH/sql/04_delete.sql@5 -f ARG_WORKLOAD_PATH/sql/05_reporting.sql@25 ARG_PG_DATABASE'
350+
```
351+
352+
The replacement is a complete command: include every script you want to run.
353+
Use `-f FILE@WEIGHT` for relative weights; omit a file to remove it from the mix.
354+
Set pgbench options such as `--random-seed`, `-M` and `-j` in this command;
355+
there are no separate profile CLI flags for them. Retain the client and duration
356+
placeholders when those values should follow the benchmark settings.
357+
358+
### Editing SQL or the data generator
359+
360+
Copy the complete profile directory to the workload-generator host and edit
361+
that copy. Using Python from the environment where `pg-perf-bench` is installed:
362+
363+
```bash
364+
python3 - <<'PY'
365+
from pathlib import Path
366+
from shutil import copytree
367+
import pg_perf_bench
368+
369+
source = Path(pg_perf_bench.__file__).parent / 'workload_profiles' / 'pagila-htap'
370+
copytree(source, 'workloads/pagila-local')
371+
PY
372+
```
373+
374+
Edit `generator.py` to change data distributions. Adjust the probabilities in
375+
`sql/02_insert.sql` to change the frequency of customer, catalogue and staff changes.
376+
Edit the SQL scripts for query behavior. The copied `profile.json` contains
377+
the command templates, including the script list and weights.
378+
379+
Use `--benchmark-type custom --workload-path` for the copy. A custom path does
380+
not automatically apply commands or defaults from `profile.json`: pass both command templates explicitly,
381+
and set the duration explicitly if they use `ARG_WORKLOAD_DURATION_SECONDS`.
382+
The following reads your edited templates and runs them:
383+
384+
```bash
385+
workload_init=$(python3 -c 'import json; print(json.load(open("workloads/pagila-local/profile.json"))["benchmark"]["init_command"])')
386+
workload_run=$(python3 -c 'import json; print(json.load(open("workloads/pagila-local/profile.json"))["benchmark"]["workload_command"])')
387+
388+
PGPASSWORD=secret pg-perf-bench benchmark \
389+
--connection-type local \
390+
--allow-database-reset \
391+
--host 127.0.0.1 --port 5432 --user postgres \
392+
--database pg_perf_bench_test \
393+
--pg-data-path /var/lib/postgresql/18/main \
394+
--pg-bin-path /usr/lib/postgresql/18/bin \
395+
--benchmark-type custom \
396+
--workload-path ./workloads/pagila-local \
397+
--workload-scale 4 \
398+
--workload-duration-seconds 120 \
399+
--pgbench-clients 1,2,4,8,16 \
400+
--init-command "$workload_init" \
401+
--workload-command "$workload_run" \
402+
--command-timeout 300 \
403+
--report-name pagila-local-scale4
404+
```
405+
406+
To validate a configured command without touching PostgreSQL, insert `plan`
407+
before `benchmark`: `pg-perf-bench plan benchmark ...`.
408+
409+
### Profile files and parameters in reports
410+
411+
Both JSON and HTML reports retain the effective initialization and workload
412+
commands for every iteration, together with scale, duration, client counts and
413+
CLI arguments. This includes overridden script weights, seed, jobs and other
414+
pgbench options. Passwords are redacted.
415+
416+
For bundled profiles, `workload_evidence.files` contains every file listed in
417+
the manifest and `profile.json` itself. For a local profile, the report also
418+
captures the other files under `--workload-path`, including JSON, YAML, TOML,
419+
shell scripts and configuration files without extensions. The local manifest's
420+
`files` entries supply file roles; they do not select commands or defaults.
421+
Git/Mercurial/Subversion metadata, `.venv`, `venv`, `__pycache__`, `.pyc` and
422+
`.pyo` are excluded from automatic traversal. Keep reports, generated datasets
423+
and unrelated files outside the profile directory.
424+
425+
Literal input paths passed to psql or pgbench with `-f FILE`, `-fFILE`,
426+
`--file FILE` or `--file=FILE` are captured too, including external SQL used by
427+
an overridden command and pgbench's `FILE@WEIGHT` form. External files are
428+
identified by absolute path in the report; prefer absolute paths in commands.
429+
Relative file arguments are resolved from the directory where the benchmark
430+
was launched. Dependencies opened inside SQL, Python or shell code should be
431+
kept in the local profile directory: arbitrary shell expansion and runtime
432+
dependency discovery are not performed.
433+
434+
Captured files must already exist, be UTF-8 text and be no larger than 5 MiB
435+
each; symlinks are rejected. Their content and SHA-256 are stored in the report
436+
and contribute to both definition and execution hashes. In HTML, open
437+
**Workload initialization and configuration** for the manifest, schema,
438+
generator and supporting files, and **pgbench workload** for workload SQL.
439+
440+
### Managed PostgreSQL
441+
442+
Pass `--managed-pg-info FILE` to benchmark an instance whose operating system
443+
and PostgreSQL service are controlled by a cloud provider. The option selects
444+
managed mode automatically; `--connection-type`, `--pg-data-path` and
445+
`--pg-bin-path` are not required. The local `pgbench` and `psql` clients still
446+
need to be installed.
447+
448+
The metadata file can have **any format**: JSON, YAML, plain text, PDF, an image
449+
or another binary format. Its format is not parsed or used to configure the
450+
connection. Include the provider, region, instance class, CPU/RAM, storage and
451+
relevant service settings in it. The complete file, its name, size and SHA-256
452+
are embedded as `managed_pg_info` in JSON and HTML. The file contents are displayed
453+
directly as `plain_text` under **Managed PostgreSQL instance**. Binary content is
454+
stored and displayed as Base64, with an explicit encoding note.
455+
The file hash also participates in the execution plan and environment identity.
456+
457+
For example:
458+
459+
```bash
460+
PGPASSWORD=secret PGSSLMODE=require pg-perf-bench benchmark \
461+
--managed-pg-info ./cloud-instance.yaml \
462+
--host db.example.cloud --port 5432 --user bench_owner \
463+
--database pg_perf_bench_test --allow-database-reset \
464+
--workload-profile pagila-htap --workload-scale 0.1 \
465+
--workload-duration-seconds 30 --pgbench-clients 1,2,4 \
466+
--command-timeout 120 --report-name managed-pagila
467+
```
468+
469+
Use the TLS settings required by your provider; `PGSSLMODE` and `PGSSLROOTCERT`
470+
apply to the database connections and local client tools.
471+
472+
Managed mode measures TPS, latency, transaction counts and client connection
473+
time, retains the complete workload evidence, and collects PostgreSQL version,
474+
settings and extensions through SQL using the supplied role. It recreates the
475+
dedicated benchmark database before each iteration, so the role needs
476+
`CREATEDB`, access to the `postgres` maintenance database and ownership of the
477+
benchmark database. `--allow-database-reset` remains mandatory.
478+
479+
The utility does not restart the server, flush filesystems, drop OS caches,
480+
install `postgresql.conf`, read server logs or run the OS sampler. Host facts,
481+
OS metrics, `PostgreSQL pg_config` and server logs explicitly show
482+
`No data. Managed PostgreSQL.` These expected limitations have status
483+
`unsupported`; actual SQL or workload failures remain errors. Managed mode
484+
cannot be combined with SSH/Docker transport, `--pg-custom-config` or
485+
`--drop-os-caches`. `--collect-pg-logs` retains the unavailable-data marker.
486+
Custom workload commands execute exactly as supplied and must themselves be
487+
compatible with the provider's permissions.
274488

275489
### Iteration lifecycle
276490

277-
For each axis value the backend:
491+
For each axis value in the regular mode, the backend:
278492

279493
1. verifies access to the PostgreSQL instance;
280494
2. drops the dedicated benchmark database;
@@ -382,10 +596,12 @@ A benchmark report contains:
382596
- a compatibility preflight containing the PostgreSQL server major, the newest
383597
local pgbench/psql versions and the supported server range 10–18;
384598
- raw initialization and workload evidence for every completed iteration;
385-
- parsed clients, duration, transaction count, average latency, initial
386-
connection time, and TPS;
599+
- parsed clients, duration, transaction count, average latency, latency standard
600+
deviation, failed/retried transaction percentages, initial connection time, and TPS;
387601
- an explicit `maximum_tps` point with its axis value and complete metrics;
388-
- a TPS chart for the selected axis;
602+
- separate charts for TPS, average transaction latency, transaction latency
603+
standard deviation, failed/retried transaction percentages and initial
604+
connection time, all using the selected client or duration axis;
389605
- all `pg_diag` OS charts collected during every measured iteration: CPU
390606
utilization/load, RAM usage/pressure, disk throughput/IOPS/utilization/latency,
391607
and network throughput/packets;
@@ -395,6 +611,14 @@ A benchmark report contains:
395611
- an optional PostgreSQL log archive reference backed by the report-local
396612
`db_logs/` directory.
397613

614+
Optional metrics are taken from the overall pgbench summary. Percentages retain
615+
the values reported by pgbench. Retry statistics require `--max-tries` other than
616+
1; latency standard deviation requires timing statistics (for example,
617+
`--progress`); `--connect` reports average connection time instead of initial
618+
connection time. Absent values remain `null`, appear as gaps, and display an
619+
explicit no-data message when an entire chart is unavailable. A reported zero
620+
remains a zero. Joined reports include a series per source for these charts.
621+
398622
The JSON and HTML files are written through temporary files and atomically
399623
renamed into place. Report names cannot contain path separators, `.`/`..`, or a
400624
NUL byte.

0 commit comments

Comments
 (0)