Skip to content

Commit bc3d64b

Browse files
JuliaEdomclaude
andcommitted
perf(verification): stop paying OpenBLAS for arithmetic nobody does (FB-17)
The Windows shard suite sat 5.3% under the host's 1 GiB per-process guard, and the audit read that as a few heavy test modules to split up. Measuring says otherwise: the peak was a flat per-process cost every shard paid once. import numpy 655.6 MiB import numpy, OPENBLAS_NUM_THREADS=1 109.5 MiB INSERT ... VALUES ('r', 't', 200) 86.0 MiB the same insert as VALUES (?, ?, ?) 703.1 MiB OpenBLAS commits per-core scratch buffers when it loads -- ~30 MiB x 18 cores -- and duckdb reaches numpy on its own, importing pandas/pyarrow to convert bound parameters. That is why every module writing to a DuckDB store looked heavy and the docs-ratchet modules the audit suspected did not. Splitting shards could never have helped: the cost does not divide. The runner now forces OPENBLAS_NUM_THREADS=1 and OMP_NUM_THREADS=1 into collection and every shard. Forced rather than defaulted, because a stray OMP_NUM_THREADS=8 in an operator's shell would put the cost back silently. No file under src/ and no test imports numpy, so nothing loses arithmetic. The budget drops 1000 -> 600 MiB, which is a ratchet again instead of a near-miss of the guard, and the docstring stops quoting a "default 900 MiB" that had outlived two changes. The failure message no longer says "split it further" -- provably wrong advice -- and names the per-file attribution command instead. The API image does not pay any of this: requirements-docker.lock ships neither numpy nor pandas, and the same parameterized insert with both made unimportable writes its row at 87.6 MiB. docs/operations/windows-verification.md records the measurements and a test fails if that lock ever gains them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmsELVj792npHTA1m13mEM
1 parent 5f1e4b0 commit bc3d64b

7 files changed

Lines changed: 300 additions & 11 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,6 @@
1010
## Verification Hosts
1111
- Keep local Windows Python processes below 1 GiB. Use narrow checks and rely on `D:\SystemState\PythonMemoryGuard\python-memory-guard.ps1`.
1212
- Run Docker-heavy verification, jobs likely to exceed 1 GiB, and macOS diagnostics through SSH alias `deproject-mac`; the Mac checkout is `/Users/julia/agentflow-docker-check`.
13+
- The sharded Windows suite pins `OPENBLAS_NUM_THREADS=1` and `OMP_NUM_THREADS=1` in every child process; unpinned, OpenBLAS alone accounts for ~550 MiB of the 1 GiB budget. Measurements and the production caveat: `docs/operations/windows-verification.md`.
1314
- Non-secret Mac context belongs in `AGENT_STATE.md`, `docs/SESSION_HANDOFF.md`, and `docs/operations/local-verification-matrix.md`.
1415
- Never print, commit, or copy SSH keys, passwords, tokens, recovery codes, or other secrets into project files, logs, PRs, or chat.

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,7 @@ python -m pytest tests/unit tests/integration tests/sdk -q
226226
# broad Windows no-Docker suite (audit F-07): sequential per-process shards
227227
# with a per-shard peak-memory budget under the host's 1 GiB process guard.
228228
# Do not run the monolithic pytest command above for this purpose on Windows.
229+
# What that budget is spent on: docs/operations/windows-verification.md
229230
python scripts/run_windows_unit_shards.py tests/unit
230231

231232
# benchmark and regression gate

‎docs/operations/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ are inputs to later work, not general-purpose procedures.
4444
| [API DuckDB persistence and recovery design](api-duckdb-persistence-recovery-design.md) | Current preservation/recovery authorization-boundary owner | Status is `CAPABILITY_REHEARSAL_REQUIRED`; it is not an approved operator runbook |
4545
| [CI-soak Compose foundation](ci-soak-compose-foundation.md) | Topology and historical implementation reference | Its runtime-status sequence is superseded; resume from the current CI-soak runbook above |
4646
| [OpenSSF security posture](openssf-security-posture.md) | Scope and interpretation of free supply-chain posture signals | Neither Scorecard nor self-certification is a penetration test or attestation |
47+
| [Windows verification memory](windows-verification.md) | What the sharded Windows suite's per-process budget is spent on, measured | Describes the test host; pinning the same variables in the API image is an owner decision that page does not take |
4748

4849
## Consumed and dated records
4950

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Windows verification memory
2+
3+
The Windows verification host kills any Python process that passes 1 GiB, which
4+
is why `tests/unit` runs as sequential shards instead of one pytest command
5+
(audit F-07). This page owns what that budget is actually spent on. It exists
6+
because the budget was raised against the wrong cause, and the next person to
7+
touch it should start from the measurement rather than from the shard size.
8+
9+
**Audience:** maintainers running or tuning the Windows no-Docker verification suite
10+
11+
**Prerequisites:** a local checkout with the project virtualenv installed; the numbers below were measured on the supported 18-core Windows host on 2026-09-07, except where a line attributes them otherwise, with duckdb 1.5.4, numpy 2.3.5, pandas 3.0.3 and pyarrow 24.0.0
12+
13+
## Where the memory goes
14+
15+
Nearly all of it is one flat, per-process cost, paid the first time anything in
16+
the process imports numpy:
17+
18+
| Measured with `scripts/run_windows_unit_shards.py <file>` | Peak |
19+
| --- | --- |
20+
| `import numpy` | 655.6 MiB |
21+
| `import numpy`, with `OPENBLAS_NUM_THREADS=1` | 109.5 MiB |
22+
| `import numpy, pandas, pyarrow`, with the thread variables pinned | 141.0 MiB |
23+
| `INSERT INTO t VALUES ('r', 't', 200)` on a DuckDB file | 86.0 MiB |
24+
| the same insert as `VALUES (?, ?, ?)` with bound parameters | 703.1 MiB |
25+
26+
Two mechanisms combine:
27+
28+
- **OpenBLAS commits per-core scratch buffers when it loads.** On this host that
29+
is roughly 30 MiB times 18 cores, and it is charged to the process whether or
30+
not any array arithmetic ever happens. Pinning the pools to a single thread
31+
removes it.
32+
- **DuckDB reaches numpy on its own.** Passing parameters to `execute()` makes
33+
the Python client import pandas and pyarrow — and therefore numpy — to convert
34+
the bound values. The same statement written with literals never loads them.
35+
Parameter binding is the correct way to write these queries, so this is a cost
36+
to bound, not to avoid.
37+
38+
That second point is why the heavy modules looked like a random assortment:
39+
every test that writes to a DuckDB store paid it, and nothing else did. Bisecting
40+
one 797 MiB test in `tests/unit/test_analytics_middleware.py` reached
41+
`ensure_analytics_table` at 91.6 MiB and the single session insert after it at
42+
810.8 MiB, which is the whole finding in one line.
43+
44+
## Why a smaller `--shard-size` never helped
45+
46+
The cost is per process and flat, so it does not divide. An earlier pass on the
47+
same host (2026-08-21) recorded a 14-file shard at 830 MiB next to a 147-test
48+
shard at 94 MiB; splitting either one changes nothing, because each resulting
49+
process still imports numpy once. Those numbers were read as "individual heavy
50+
test modules" and the budget was raised instead, which is how the suite arrived
51+
at a 969.8 MiB worst shard against a 1024 MiB guard — 5.3% of headroom.
52+
53+
## What the runner does now
54+
55+
`scripts/run_windows_unit_shards.py` forces `OPENBLAS_NUM_THREADS=1` and
56+
`OMP_NUM_THREADS=1` into every child process it starts, for collection as well
57+
as for each shard. The variables are forced rather than defaulted: an
58+
`OMP_NUM_THREADS=8` left over in an operator's shell would otherwise put the
59+
whole cost back without any visible sign.
60+
61+
Neither `src/` nor the unit suite computes with numpy — it is loaded only as
62+
DuckDB's conversion helper — so single-threaded BLAS has nothing to slow down.
63+
Per-module peaks with the pinning in place:
64+
65+
| Module | Before | After |
66+
| --- | --- | --- |
67+
| `tests/unit/test_analytics_middleware.py` | 805.6 MiB | 192.1 MiB |
68+
| `tests/unit/test_versioning.py` | 826.7 MiB | 312.3 MiB |
69+
| `tests/unit/test_usage_db_connection_reuse.py` | 696.0 MiB | 150.6 MiB |
70+
71+
`tests/unit/test_windows_shard_thread_pinning.py` holds the pinning in place and
72+
fails if a change drops it.
73+
74+
Across the whole suite — 3790 tests in 14 shards — the peaks now run 120.7 to
75+
400.4 MiB, against 733 to 969.8 MiB before. The budget is back to being a
76+
ratchet: `DEFAULT_MEMORY_BUDGET_MIB` is 600 MiB, half again the worst shard and
77+
well under the 1024 MiB guard, so a regression trips the runner rather than the
78+
host.
79+
80+
## The API image does not pay this
81+
82+
The cost is an artefact of the development environment, not of the product. No
83+
file under `src/` imports numpy or pandas — the only match there is the Flink
84+
jobs' requirements lock, a separate runtime — no test imports them either, and
85+
`requirements-docker.lock`, which `Dockerfile.api` installs with
86+
`--require-hashes`, contains neither. DuckDB only reaches for them when they
87+
are already installed, which in this repository means a developer checkout.
88+
89+
Re-running the same `INSERT OR REPLACE ... VALUES (?, ?, ?)` with numpy and
90+
pandas made unimportable, which is the shape of the API image, gives the answer
91+
directly: the row is written and the process peaks at 87.6 MiB, the same as the
92+
literal insert. So there is nothing to pin in the container, and nothing to
93+
carry over from the numbers above when sizing the API.
94+
95+
`tests/unit/test_windows_shard_thread_pinning.py` fails if numpy or pandas
96+
enters that lock, because the paragraph above would then need re-measuring
97+
rather than re-reading.
98+
99+
## Re-measuring
100+
101+
To attribute a shard that exceeds its budget, measure its files one at a time —
102+
the runner prints a peak for whatever target it is given:
103+
104+
```bash
105+
python scripts/run_windows_unit_shards.py tests/unit/test_versioning.py
106+
```
107+
108+
Compare a suspect line's cost by isolating it the same way, down to a single
109+
node ID. When a peak moves, check whether the process gained an import before
110+
concluding that a test got heavier.

‎scripts/run_windows_unit_shards.py‎

Lines changed: 59 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,11 @@
99
every file to exactly one shard (asserted — no silent gaps);
1010
- runs each shard sequentially in a fresh Python process, so no single
1111
process accumulates the whole suite's allocations;
12-
- records peak private memory per shard (Windows: ``PeakPagefileUsage`` via
13-
ctypes; POSIX: best effort) and fails if a shard exceeds the budget
14-
(default 900 MiB — headroom under the 1024 MiB guard);
12+
- pins each shard's BLAS thread pools to one thread (``SHARD_ENV``), which is
13+
where this suite's memory actually went before audit FB-17;
14+
- records peak private memory per shard (Windows: ``PeakProcessMemoryUsed``
15+
via ctypes; POSIX: best effort) and fails if a shard exceeds the budget
16+
(``DEFAULT_MEMORY_BUDGET_MIB`` below — headroom under the 1024 MiB guard);
1517
- cross-checks each shard's executed-test count against its collected
1618
node-ID count, so a shard that silently drops tests fails the run.
1719
@@ -21,25 +23,59 @@
2123
Usage:
2224
python scripts/run_windows_unit_shards.py # tests/unit
2325
python scripts/run_windows_unit_shards.py tests/unit tests/sdk
24-
python scripts/run_windows_unit_shards.py --shard-size 250 --memory-budget-mib 900
26+
python scripts/run_windows_unit_shards.py --shard-size 250 --memory-budget-mib 800
27+
python scripts/run_windows_unit_shards.py tests/unit/test_versioning.py # one module's peak
2528
"""
2629

2730
from __future__ import annotations
2831

2932
import argparse
33+
import os
3034
import re
3135
import subprocess
3236
import sys
3337
from collections import Counter
3438
from pathlib import Path
3539

3640
DEFAULT_SHARD_SIZE = 300
37-
# Measured on the supported Windows host (2026-08-21): per-shard peaks are
38-
# dominated by individual heavy test modules, not shard size — a 14-file
39-
# shard peaked at 830 MiB while its 147-test neighbour peaked at 94 MiB, and
40-
# the hottest shard reached 944 MiB. Splitting shards further cannot reduce a
41-
# single module's peak, so the budget sits just under the 1024 MiB guard.
42-
DEFAULT_MEMORY_BUDGET_MIB = 1000
41+
42+
# Audit FB-17. Peaks were never about how many tests a shard held: a 14-file
43+
# shard peaked at 830 MiB next to a 147-test shard at 94 MiB, so the earlier
44+
# reading — "individual heavy test modules" — was measured but misattributed.
45+
# Bisecting one 797 MiB test down to its imports found a flat, per-process
46+
# cost that every shard paid once, on the supported Windows host (18 cores,
47+
# numpy 2.3.5 / pandas 3.0.3 / pyarrow 24.0.0 / duckdb 1.5.4, 2026-09-07):
48+
#
49+
# import numpy 655.6 MiB
50+
# import numpy, with OPENBLAS_NUM_THREADS=1 109.5 MiB
51+
# numpy + pandas + pyarrow, both vars set 141.0 MiB
52+
#
53+
# OpenBLAS commits per-thread scratch buffers at load, one set per core, so
54+
# the arithmetic is ~30 MiB x 18 cores. No file under src/ or tests imports
55+
# numpy, so nothing here loses arithmetic by pinning; duckdb reaches it on
56+
# its own: passing
57+
# parameters to `execute()` imports pandas/pyarrow (and so numpy) to convert
58+
# them, which is why every module that writes to a DuckDB store looked heavy
59+
# while the same INSERT with literals stayed at 86 MiB.
60+
#
61+
# This is a development-environment cost only: duckdb imports pandas/numpy
62+
# when they happen to be installed, and requirements-docker.lock ships
63+
# neither. With both made unimportable — the shape of the API image — the
64+
# same INSERT writes its row and peaks at 87.6 MiB. See
65+
# docs/operations/windows-verification.md.
66+
SHARD_ENV = {"OPENBLAS_NUM_THREADS": "1", "OMP_NUM_THREADS": "1"}
67+
68+
# With SHARD_ENV applied the full suite (3790 tests, 14 shards, 2026-09-07)
69+
# peaks at 400.4 MiB, down from 969.8 MiB. This budget is therefore a ratchet
70+
# against regressions again, not a near-miss of the 1024 MiB guard. Raise it
71+
# only with a measurement that says why.
72+
DEFAULT_MEMORY_BUDGET_MIB = 600
73+
74+
BUDGET_FAILURE_HINT = (
75+
"attribute it with `python scripts/run_windows_unit_shards.py <file>` per file "
76+
"before changing this number — a smaller --shard-size cannot lower a per-process "
77+
"import cost, and SHARD_ENV is what keeps that cost small"
78+
)
4379

4480
# "== 12 passed, 3 skipped, 1 xfailed in 4.56s ==" -> {"passed": 12, ...}
4581
_SUMMARY_RE = re.compile(r"(\d+) (passed|failed|error(?:s)?|skipped|xfailed|xpassed|warnings?)")
@@ -52,6 +88,7 @@ def collect_node_ids(targets: list[str]) -> list[str]:
5288
capture_output=True,
5389
text=True,
5490
check=False,
91+
env=shard_environment(),
5592
)
5693
if completed.returncode not in (0, 5):
5794
sys.stderr.write(completed.stdout + completed.stderr)
@@ -168,13 +205,24 @@ class JobObjectExtendedLimitInformation(ctypes.Structure):
168205
return peak
169206

170207

208+
def shard_environment() -> dict[str, str]:
209+
"""The parent environment with SHARD_ENV forced on top.
210+
211+
Forced, not defaulted: an ``OMP_NUM_THREADS=8`` left in an operator's
212+
shell would otherwise put back the ~550 MiB the pinning exists to avoid,
213+
and it would do it silently.
214+
"""
215+
return {**os.environ, **SHARD_ENV}
216+
217+
171218
def run_shard(files: list[str]) -> tuple[int, str, float | None]:
172219
job = _WindowsJob() if sys.platform == "win32" else None
173220
process = subprocess.Popen( # noqa: S603
174221
[sys.executable, "-m", "pytest", "-q", *files],
175222
stdout=subprocess.PIPE,
176223
stderr=subprocess.STDOUT,
177224
text=True,
225+
env=shard_environment(),
178226
)
179227
assigned = job.assign(process) if job is not None else False
180228
output, _ = process.communicate()
@@ -217,7 +265,7 @@ def main(argv: list[str] | None = None) -> int:
217265
if peak_mib is not None and peak_mib > args.memory_budget_mib:
218266
failures.append(
219267
f"shard {index} peaked at {peak_mib:.1f} MiB "
220-
f"(budget {args.memory_budget_mib:.0f} MiB); split it further"
268+
f"(budget {args.memory_budget_mib:.0f} MiB); {BUDGET_FAILURE_HINT}"
221269
)
222270

223271
print(f"total executed: {executed_total}/{len(node_ids)}")

‎tests/unit/test_docs_operations_index.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
"docs/operations/api-duckdb-persistence-recovery-design.md",
3333
"docs/operations/ci-soak-compose-foundation.md",
3434
"docs/operations/openssf-security-posture.md",
35+
"docs/operations/windows-verification.md",
3536
}
3637

3738
CONSUMED_OR_DATED_RECORDS = {
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
"""Audit FB-17: the shard runner must pin BLAS threads in every child process.
2+
3+
Unpinned, OpenBLAS commits one set of scratch buffers per core at import
4+
(measured: 655.6 MiB for `import numpy` on an 18-core host, 109.5 MiB with
5+
OPENBLAS_NUM_THREADS=1). That flat cost is what pushed shards to within 5% of
6+
the host's 1 GiB per-process guard, so losing the pinning would quietly undo
7+
the headroom rather than fail loudly.
8+
"""
9+
10+
import re
11+
import subprocess
12+
from pathlib import Path
13+
14+
import pytest
15+
16+
from scripts import run_windows_unit_shards as runner
17+
18+
ROOT = Path(__file__).resolve().parents[2]
19+
20+
21+
def test_shard_env_pins_both_thread_pools_to_one():
22+
assert runner.SHARD_ENV == {"OPENBLAS_NUM_THREADS": "1", "OMP_NUM_THREADS": "1"}
23+
24+
25+
def test_shard_environment_overrides_an_inherited_thread_count(monkeypatch: pytest.MonkeyPatch):
26+
monkeypatch.setenv("OMP_NUM_THREADS", "8")
27+
monkeypatch.setenv("OPENBLAS_NUM_THREADS", "18")
28+
29+
environment = runner.shard_environment()
30+
31+
assert environment["OMP_NUM_THREADS"] == "1"
32+
assert environment["OPENBLAS_NUM_THREADS"] == "1"
33+
34+
35+
def test_shard_environment_keeps_the_rest_of_the_parent_environment(
36+
monkeypatch: pytest.MonkeyPatch,
37+
):
38+
monkeypatch.setenv("AGENTFLOW_MARKER", "kept")
39+
40+
assert runner.shard_environment()["AGENTFLOW_MARKER"] == "kept"
41+
42+
43+
def test_run_shard_passes_the_pinned_environment_to_pytest(monkeypatch: pytest.MonkeyPatch):
44+
captured: dict = {}
45+
46+
class _FakeProcess:
47+
returncode = 0
48+
49+
def communicate(self):
50+
return "1 passed in 0.01s", ""
51+
52+
class _NoJob:
53+
"""Stand in for the Job Object: a fake Popen has no handle to assign."""
54+
55+
def assign(self, process):
56+
return False
57+
58+
def _fake_popen(argv, **kwargs):
59+
captured["argv"] = argv
60+
captured["env"] = kwargs.get("env")
61+
return _FakeProcess()
62+
63+
monkeypatch.setattr(subprocess, "Popen", _fake_popen)
64+
monkeypatch.setattr(runner, "_WindowsJob", _NoJob)
65+
66+
returncode, output, _ = runner.run_shard(["tests/unit/test_example.py"])
67+
68+
assert returncode == 0
69+
assert "passed" in output
70+
assert captured["env"] is not None, "run_shard inherited the environment instead of pinning it"
71+
for name, value in runner.SHARD_ENV.items():
72+
assert captured["env"][name] == value
73+
74+
75+
def test_collect_node_ids_pins_the_environment_too(monkeypatch: pytest.MonkeyPatch):
76+
captured: dict = {}
77+
78+
def _fake_run(argv, **kwargs):
79+
captured["env"] = kwargs.get("env")
80+
return subprocess.CompletedProcess(argv, 0, "tests/unit/test_example.py::test_one\n", "")
81+
82+
monkeypatch.setattr(subprocess, "run", _fake_run)
83+
84+
runner.collect_node_ids(["tests/unit"])
85+
86+
assert captured["env"] is not None
87+
for name, value in runner.SHARD_ENV.items():
88+
assert captured["env"][name] == value
89+
90+
91+
def test_docstring_quotes_no_budget_that_disagrees_with_the_constant():
92+
"""A stale "default 900 MiB" in the docstring outlived two budget changes."""
93+
quoted = {int(match) for match in re.findall(r"(\d+) MiB", runner.__doc__ or "")}
94+
95+
assert quoted <= {1024, int(runner.DEFAULT_MEMORY_BUDGET_MIB)}, (
96+
f"docstring quotes {sorted(quoted)} MiB; the budget is "
97+
f"{runner.DEFAULT_MEMORY_BUDGET_MIB:.0f} MiB and 1024 is the host guard"
98+
)
99+
100+
101+
def test_the_api_image_still_ships_neither_numpy_nor_pandas():
102+
"""The runbook's "the API image does not pay this" rests on this lock.
103+
104+
DuckDB imports pandas (and so numpy, and so OpenBLAS) only when they are
105+
installed, so the measured 87.6 MiB for a parameterized insert in the image
106+
holds exactly as long as this stays true.
107+
"""
108+
lock = (ROOT / "requirements-docker.lock").read_text(encoding="utf-8")
109+
requirements = {
110+
line.split("==", 1)[0].strip().lower()
111+
for line in lock.splitlines()
112+
if "==" in line and not line.lstrip().startswith(("#", "--"))
113+
}
114+
115+
intruders = requirements & {"numpy", "pandas"}
116+
assert not intruders, (
117+
f"{sorted(intruders)} entered requirements-docker.lock; re-measure before "
118+
"trusting docs/operations/windows-verification.md on the API image"
119+
)
120+
121+
122+
def test_budget_failure_hint_does_not_advise_splitting_shards():
123+
hint = runner.BUDGET_FAILURE_HINT
124+
125+
assert "split it further" not in hint
126+
assert "run_windows_unit_shards.py <file>" in hint
127+
assert "SHARD_ENV" in hint

0 commit comments

Comments
 (0)