Skip to content

Commit 23f809d

Browse files
committed
webchannel/benchmark: tweak reports
1 parent 7ebe4e1 commit 23f809d

7 files changed

Lines changed: 34 additions & 32 deletions

File tree

‎webchannel/benchmark/lens/2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/artifacts.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,16 @@ Histogram PNG and SVG files are generated with matplotlib. Each metric shares 10
1919
[Execution settings](execution-settings.json) record the requested parameters, user-reported connection conditions, and frozen script hashes.
2020

2121
[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.
22+
23+
## Benchmark elapsed time by delay
24+
25+
Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.
26+
27+
| M1 delay (ms) | Elapsed seconds |
28+
|---:|---:|
29+
| 0 | 189.730 |
30+
| 50 | 124.139 |
31+
| 100 | 127.648 |
32+
| 150 | 127.642 |
33+
| 200 | 133.710 |
34+

‎webchannel/benchmark/lens/2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/report.md‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,6 @@ The HTTP 400 rate uses all 50 attempted pairs at each delay as the denominator,
4444
| 150 | 50 | 0 | 0.0% | 0 | 0 | 48 | 0 |
4545
| 200 | 50 | 1 | 2.0% | 47 | 0 | 44 | 0 |
4646

47-
## Benchmark elapsed time by delay
48-
49-
Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.
50-
51-
| M1 delay (ms) | Elapsed seconds |
52-
|---:|---:|
53-
| 0 | 189.730 |
54-
| 50 | 124.139 |
55-
| 100 | 127.648 |
56-
| 150 | 127.642 |
57-
| 200 | 133.710 |
58-
5947
## M1 delay: 0 ms
6048

6149
### TTFA: ON minus OFF

‎webchannel/benchmark/lens/2026-10-14_21-13_omnient-delay-sweep-250pairs-relaunch_JST/artifacts.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,16 @@ Histogram PNG and SVG files are generated with matplotlib. Each metric shares 10
2020

2121
[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.
2222

23+
## Benchmark elapsed time by delay
24+
25+
Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.
26+
27+
| M1 delay (ms) | Elapsed seconds |
28+
|---:|---:|
29+
| 0 | 2035.562 |
30+
| 50 | 649.217 |
31+
| 100 | 670.682 |
32+
| 150 | 1127.495 |
33+
| 200 | 686.210 |
34+
2335
[Report rendering settings](report-render-settings.json) record the rendering fix and generator hashes separately from frozen measurement settings. [Rendering verification](report-render-verification.json) confirms unchanged data, selection and bins, raw preservation, and deterministic plot regeneration.

‎webchannel/benchmark/lens/2026-10-14_21-13_omnient-delay-sweep-250pairs-relaunch_JST/report.md‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -44,18 +44,6 @@ The HTTP 400 rate uses all 250 attempted pairs at each delay as the denominator,
4444
| 150 | 250 | 6 | 2.4% | 0 | 2 | 70 | 0 |
4545
| 200 | 250 | 6 | 2.4% | 121 | 1 | 66 | 1 |
4646

47-
## Benchmark elapsed time by delay
48-
49-
Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.
50-
51-
| M1 delay (ms) | Elapsed seconds |
52-
|---:|---:|
53-
| 0 | 2035.562 |
54-
| 50 | 649.217 |
55-
| 100 | 670.682 |
56-
| 150 | 1127.495 |
57-
| 200 | 686.210 |
58-
5947
## M1 delay: 0 ms
6048

6149
### TTFA: ON minus OFF

‎webchannel/benchmark/lens/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ When changing or generating files under this directory, follow [README.md](READM
1717
- For long sweeps, `run-resumable-delay-sweep.py RUN_DIR [EXISTING_RUNNER_PID]` uses the run's `execution-settings.json` to resume safely and invokes `finalize-delay-sweep.py` after all trials are recorded. Only the specific destination-availability/system-notification failure before XCTest starts and before any result or measurement JSON is eligible for automatic recovery. Retain diagnostic and startup evidence before resuming; stop on unknown failures, failed device queries, or three unsuccessful recovery attempts for the same unstarted trial. Do not infer that the iPhone was locked from Xcode's generic unlock message. Record all scripts' hashes before they execute.
1818
- For historical sweeps that already have a 400-excluded report, the supplemental `summarize-delay-sweep.py --include-early-post-400` mode remains available to preserve that original report. New sweeps use the main inclusive report (`--early-post-400-policy include`), and saved metadata preserves each run's policy when regenerating it.
1919

20-
- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total and per-delay wall-clock duration in the report. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.
20+
- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total wall-clock duration in `report.md` and per-delay wall-clock duration in `artifacts.md`. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.
2121

2222
- New sweeps use `executionProtocol: xcode-relaunch-batch`: one Xcode invocation per delay, one selected `testColdBinaryOmnientBatch` method, `-test-iterations 2*PAIRS -test-repetition-relaunch-enabled YES`, and a fresh app process per iteration. The persisted batch ordinal alternates OFF then ON; never group both settings in one repetition. Do not use retry or run-until-failure flags. Import originals by XCTest repetition number and verified ordinal/process identity with `import-relaunch-batch.py`; preserve shared stdout, original exports, and byte-range mappings. Stop for review on incomplete batches, preserve evidence, and never silently replace entered trials. The legacy recovery runners apply only to historical runs. Freeze runner, importer, generators, Swift sources, and compiled XCTest hashes before measurement.
2323

‎webchannel/benchmark/lens/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ For a long sweep with `execution-settings.json` containing the device ID, derive
6060

6161
After all requested trials are recorded, the workflow runs `finalize-delay-sweep.py`. This generates the main inclusive report, independently verifies trial settings, transport evidence, selection rules, paired differences, bin counts, source hashes, and local links, and regenerates the report to verify identical output and unchanged raw evidence. `completion.json` is written only after those checks pass. Human-readable workflow progress remains in its invocation log; the absence of `completion.json` means the workflow has not confirmed completion.
6262

63-
New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total and per-delay wall-clock elapsed seconds when available, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.
63+
New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total wall-clock elapsed seconds when available; per-delay elapsed seconds belong in `artifacts.md`, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.
6464

6565
The [2026-10-14 batched relaunch sweep](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/report.md) records 50 pairs at each of 0, 50, 100, 150, and 200 ms. All 500 trials used distinct app process instances and verified fresh HTTP/2 TCP/TLS transport; 499 completed and one reached the measurement deadline. No startup connection error was observed. Total benchmark wall time was 703.073 seconds. [Audit](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/connection-reuse-audit.md) and [artifacts](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/artifacts.md) retain the evidence.
6666

‎webchannel/benchmark/lens/summarize-delay-sweep.py‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -183,12 +183,6 @@ def main():
183183
"|---:|---:|---:|---:|---:|---:|---:|---:|"]
184184
for delay, data in datasets.items():
185185
lines.append(f"| {delay} | {args.pairs} | {data['earlyPost400Pairs']} | {data['earlyPost400Rate'] * 100:.1f}% | {data['noEarlyPostPairs']} | {data['unknownEarlyPostPairs']} | {data['acceptedPairs']} | {data['includedEarlyPost400Pairs']} |")
186-
if timing and "elapsedSeconds" in timing:
187-
lines.extend(["", "## Benchmark elapsed time by delay", "",
188-
"Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.", "",
189-
"| M1 delay (ms) | Elapsed seconds |", "|---:|---:|"])
190-
for delay in delays:
191-
lines.append(f"| {delay} | {timing['delays'][str(delay)]['elapsedSeconds']:.3f} |")
192186
for delay in delays:
193187
lines.extend(["", f"## M1 delay: {delay} ms", ""])
194188
for metric in ("TTFA", "TTFD"):
@@ -222,6 +216,13 @@ def main():
222216
artifacts.extend(["[Execution settings](execution-settings.json) record the requested parameters, user-reported connection conditions, and frozen script hashes.", ""])
223217
if timing is not None:
224218
artifacts.extend(["[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.", ""])
219+
if timing and "elapsedSeconds" in timing:
220+
artifacts.extend(["", "## Benchmark elapsed time by delay", "",
221+
"Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.", "",
222+
"| M1 delay (ms) | Elapsed seconds |", "|---:|---:|"])
223+
for delay in delays:
224+
artifacts.append(f"| {delay} | {timing['delays'][str(delay)]['elapsedSeconds']:.3f} |")
225+
artifacts.append("")
225226
if (root / "logs/device-baseline").exists():
226227
artifacts.extend(["[Device baseline](logs/device-baseline/) records USB recognition, CoreDevice availability, and Mac power assertions before measurement. Failure snapshots are retained under each delay’s logs/device-diagnostics/ directory.", ""])
227228
if (root / "report-render-settings.json").exists():

0 commit comments

Comments
 (0)