Skip to content

webchannel/benchmark: tweak reports #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pmodular merged 1 commit into from
Oct 14, 2026
Merged

webchannel/benchmark: tweak reports #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions ...ark/lens/2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/artifacts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,16 @@ Histogram PNG and SVG files are generated with matplotlib. Each metric shares 10
[Execution settings](execution-settings.json) record the requested parameters, user-reported connection conditions, and frozen script hashes.

[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.

## Benchmark elapsed time by delay

Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.

| M1 delay (ms) | Elapsed seconds |
|---:|---:|
| 0 | 189.730 |
| 50 | 124.139 |
| 100 | 127.648 |
| 150 | 127.642 |
| 200 | 133.710 |

12 changes: 0 additions & 12 deletions ...chmark/lens/2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,6 @@ The HTTP 400 rate uses all 50 attempted pairs at each delay as the denominator,
| 150 | 50 | 0 | 0.0% | 0 | 0 | 48 | 0 |
| 200 | 50 | 1 | 2.0% | 47 | 0 | 44 | 0 |

## Benchmark elapsed time by delay

Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.

| M1 delay (ms) | Elapsed seconds |
|---:|---:|
| 0 | 189.730 |
| 50 | 124.139 |
| 100 | 127.648 |
| 150 | 127.642 |
| 200 | 133.710 |

## M1 delay: 0 ms

### TTFA: ON minus OFF
Expand Down
12 changes: 12 additions & 0 deletions ...rk/lens/2026-10-14_21-13_omnient-delay-sweep-250pairs-relaunch_JST/artifacts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,16 @@ Histogram PNG and SVG files are generated with matplotlib. Each metric shares 10

[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.

## Benchmark elapsed time by delay

Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.

| M1 delay (ms) | Elapsed seconds |
|---:|---:|
| 0 | 2035.562 |
| 50 | 649.217 |
| 100 | 670.682 |
| 150 | 1127.495 |
| 200 | 686.210 |

[Report rendering settings](report-render-settings.json) record the rendering fix and generator hashes separately from frozen measurement settings. [Rendering verification](report-render-verification.json) confirms unchanged data, selection and bins, raw preservation, and deterministic plot regeneration.
12 changes: 0 additions & 12 deletions ...hmark/lens/2026-10-14_21-13_omnient-delay-sweep-250pairs-relaunch_JST/report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,6 @@ The HTTP 400 rate uses all 250 attempted pairs at each delay as the denominator,
| 150 | 250 | 6 | 2.4% | 0 | 2 | 70 | 0 |
| 200 | 250 | 6 | 2.4% | 121 | 1 | 66 | 1 |

## Benchmark elapsed time by delay

Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.

| M1 delay (ms) | Elapsed seconds |
|---:|---:|
| 0 | 2035.562 |
| 50 | 649.217 |
| 100 | 670.682 |
| 150 | 1127.495 |
| 200 | 686.210 |

## M1 delay: 0 ms

### TTFA: ON minus OFF
Expand Down
2 changes: 1 addition & 1 deletion webchannel/benchmark/lens/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ When changing or generating files under this directory, follow [README.md](READM
- For long sweeps, `run-resumable-delay-sweep.py RUN_DIR [EXISTING_RUNNER_PID]` uses the run's `execution-settings.json` to resume safely and invokes `finalize-delay-sweep.py` after all trials are recorded. Only the specific destination-availability/system-notification failure before XCTest starts and before any result or measurement JSON is eligible for automatic recovery. Retain diagnostic and startup evidence before resuming; stop on unknown failures, failed device queries, or three unsuccessful recovery attempts for the same unstarted trial. Do not infer that the iPhone was locked from Xcode's generic unlock message. Record all scripts' hashes before they execute.
- For historical sweeps that already have a 400-excluded report, the supplemental `summarize-delay-sweep.py --include-early-post-400` mode remains available to preserve that original report. New sweeps use the main inclusive report (`--early-post-400-policy include`), and saved metadata preserves each run's policy when regenerating it.

- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total and per-delay wall-clock duration in the report. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.
- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total wall-clock duration in `report.md` and per-delay wall-clock duration in `artifacts.md`. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.

- New sweeps use `executionProtocol: xcode-relaunch-batch`: one Xcode invocation per delay, one selected `testColdBinaryOmnientBatch` method, `-test-iterations 2*PAIRS -test-repetition-relaunch-enabled YES`, and a fresh app process per iteration. The persisted batch ordinal alternates OFF then ON; never group both settings in one repetition. Do not use retry or run-until-failure flags. Import originals by XCTest repetition number and verified ordinal/process identity with `import-relaunch-batch.py`; preserve shared stdout, original exports, and byte-range mappings. Stop for review on incomplete batches, preserve evidence, and never silently replace entered trials. The legacy recovery runners apply only to historical runs. Freeze runner, importer, generators, Swift sources, and compiled XCTest hashes before measurement.

Expand Down
2 changes: 1 addition & 1 deletion webchannel/benchmark/lens/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ For a long sweep with `execution-settings.json` containing the device ID, derive

After all requested trials are recorded, the workflow runs `finalize-delay-sweep.py`. This generates the main inclusive report, independently verifies trial settings, transport evidence, selection rules, paired differences, bin counts, source hashes, and local links, and regenerates the report to verify identical output and unchanged raw evidence. `completion.json` is written only after those checks pass. Human-readable workflow progress remains in its invocation log; the absence of `completion.json` means the workflow has not confirmed completion.

New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total and per-delay wall-clock elapsed seconds when available, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.
New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total wall-clock elapsed seconds when available; per-delay elapsed seconds belong in `artifacts.md`, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.

The [2026-10-14 batched relaunch sweep](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/report.md) records 50 pairs at each of 0, 50, 100, 150, and 200 ms. All 500 trials used distinct app process instances and verified fresh HTTP/2 TCP/TLS transport; 499 completed and one reached the measurement deadline. No startup connection error was observed. Total benchmark wall time was 703.073 seconds. [Audit](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/connection-reuse-audit.md) and [artifacts](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/artifacts.md) retain the evidence.

Expand Down
13 changes: 7 additions & 6 deletions webchannel/benchmark/lens/summarize-delay-sweep.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,12 +183,6 @@ def main():
"|---:|---:|---:|---:|---:|---:|---:|---:|"]
for delay, data in datasets.items():
lines.append(f"| {delay} | {args.pairs} | {data['earlyPost400Pairs']} | {data['earlyPost400Rate'] * 100:.1f}% | {data['noEarlyPostPairs']} | {data['unknownEarlyPostPairs']} | {data['acceptedPairs']} | {data['includedEarlyPost400Pairs']} |")
if timing and "elapsedSeconds" in timing:
lines.extend(["", "## Benchmark elapsed time by delay", "",
"Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.", "",
"| M1 delay (ms) | Elapsed seconds |", "|---:|---:|"])
for delay in delays:
lines.append(f"| {delay} | {timing['delays'][str(delay)]['elapsedSeconds']:.3f} |")
for delay in delays:
lines.extend(["", f"## M1 delay: {delay} ms", ""])
for metric in ("TTFA", "TTFD"):
Expand Down Expand Up @@ -222,6 +216,13 @@ def main():
artifacts.extend(["[Execution settings](execution-settings.json) record the requested parameters, user-reported connection conditions, and frozen script hashes.", ""])
if timing is not None:
artifacts.extend(["[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.", ""])
if timing and "elapsedSeconds" in timing:
artifacts.extend(["", "## Benchmark elapsed time by delay", "",
"Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.", "",
"| M1 delay (ms) | Elapsed seconds |", "|---:|---:|"])
for delay in delays:
artifacts.append(f"| {delay} | {timing['delays'][str(delay)]['elapsedSeconds']:.3f} |")
Comment on lines +223 to +224

@gemini-code-assist gemini-code-assist Bot Oct 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential KeyError exceptions when processing incomplete or manually modified timing.json files, use defensive .get() lookups with a fallback placeholder (e.g., "—") for missing delay timing data.

Suggested change
for delay in delays:
artifacts.append(f"| {delay} | {timing['delays'][str(delay)]['elapsedSeconds']:.3f} |")
for delay in delays:
delay_seconds = timing.get("delays", {}).get(str(delay), {}).get("elapsedSeconds")
val = f"{delay_seconds:.3f}" if delay_seconds is not None else "—"
artifacts.append(f"| {delay} | {val} |")

artifacts.append("")
if (root / "logs/device-baseline").exists():
artifacts.extend(["[Device baseline](logs/device-baseline/) records USB recognition, CoreDevice availability, and Mac power assertions before measurement. Failure snapshots are retained under each delay’s logs/device-diagnostics/ directory.", ""])
if (root / "report-render-settings.json").exists():
Expand Down
Loading