You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 23f809d
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: webchannel/benchmark/lens/2026-10-14_21-13_omnient-delay-sweep-250pairs-relaunch_JST/artifacts.md
+12Lines changed: 12 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,4 +20,16 @@ Histogram PNG and SVG files are generated with matplotlib. Each metric shares 10
20
20
21
21
[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.
22
22
23
+
## Benchmark elapsed time by delay
24
+
25
+
Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.
26
+
27
+
| M1 delay (ms) | Elapsed seconds |
28
+
|---:|---:|
29
+
| 0 | 2035.562 |
30
+
| 50 | 649.217 |
31
+
| 100 | 670.682 |
32
+
| 150 | 1127.495 |
33
+
| 200 | 686.210 |
34
+
23
35
[Report rendering settings](report-render-settings.json) record the rendering fix and generator hashes separately from frozen measurement settings. [Rendering verification](report-render-verification.json) confirms unchanged data, selection and bins, raw preservation, and deterministic plot regeneration.
Copy file name to clipboardExpand all lines: webchannel/benchmark/lens/AGENTS.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ When changing or generating files under this directory, follow [README.md](READM
17
17
- For long sweeps, `run-resumable-delay-sweep.py RUN_DIR [EXISTING_RUNNER_PID]` uses the run's `execution-settings.json` to resume safely and invokes `finalize-delay-sweep.py` after all trials are recorded. Only the specific destination-availability/system-notification failure before XCTest starts and before any result or measurement JSON is eligible for automatic recovery. Retain diagnostic and startup evidence before resuming; stop on unknown failures, failed device queries, or three unsuccessful recovery attempts for the same unstarted trial. Do not infer that the iPhone was locked from Xcode's generic unlock message. Record all scripts' hashes before they execute.
18
18
- For historical sweeps that already have a 400-excluded report, the supplemental `summarize-delay-sweep.py --include-early-post-400` mode remains available to preserve that original report. New sweeps use the main inclusive report (`--early-post-400-policy include`), and saved metadata preserves each run's policy when regenerating it.
19
19
20
-
- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total and per-delay wall-clock duration in the report. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.
20
+
- When elapsed benchmark time is requested or recorded, retain full-precision runner timestamps in `timing.json` and include total wall-clock duration in `report.md`and per-delay wall-clock duration in `artifacts.md`. State that launch, measurement, attachment export, and recovery gaps are included, while build and report generation are excluded. This requested duration overrides the default no-summary-statistics rule. Do not fabricate timings for older runs. Retain a cancellation report and `interruption.json` for a user-interrupted experiment; do not resume it.
21
21
22
22
- New sweeps use `executionProtocol: xcode-relaunch-batch`: one Xcode invocation per delay, one selected `testColdBinaryOmnientBatch` method, `-test-iterations 2*PAIRS -test-repetition-relaunch-enabled YES`, and a fresh app process per iteration. The persisted batch ordinal alternates OFF then ON; never group both settings in one repetition. Do not use retry or run-until-failure flags. Import originals by XCTest repetition number and verified ordinal/process identity with `import-relaunch-batch.py`; preserve shared stdout, original exports, and byte-range mappings. Stop for review on incomplete batches, preserve evidence, and never silently replace entered trials. The legacy recovery runners apply only to historical runs. Freeze runner, importer, generators, Swift sources, and compiled XCTest hashes before measurement.
Copy file name to clipboardExpand all lines: webchannel/benchmark/lens/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -60,7 +60,7 @@ For a long sweep with `execution-settings.json` containing the device ID, derive
60
60
61
61
After all requested trials are recorded, the workflow runs `finalize-delay-sweep.py`. This generates the main inclusive report, independently verifies trial settings, transport evidence, selection rules, paired differences, bin counts, source hashes, and local links, and regenerates the report to verify identical output and unchanged raw evidence. `completion.json` is written only after those checks pass. Human-readable workflow progress remains in its invocation log; the absence of `completion.json` means the workflow has not confirmed completion.
62
62
63
-
New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total and per-delay wall-clock elapsed seconds when available, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.
63
+
New sweep invocations record `timing.json` using `record-sweep-timing.py`. Reports include total wall-clock elapsed seconds when available; per-delay elapsed seconds belong in `artifacts.md`, including process launch, measurement, attachment export, and recovery gaps. Build and report generation are excluded. Historical reports without timing records retain their original format. A user-cancelled experiment retains `interruption.json`; the resumable runner refuses to restart it.
64
64
65
65
The [2026-10-14 batched relaunch sweep](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/report.md) records 50 pairs at each of 0, 50, 100, 150, and 200 ms. All 500 trials used distinct app process instances and verified fresh HTTP/2 TCP/TLS transport; 499 completed and one reached the measurement deadline. No startup connection error was observed. Total benchmark wall time was 703.073 seconds. [Audit](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/connection-reuse-audit.md) and [artifacts](2026-10-14_16-41_omnient-delay-sweep-50pairs-relaunch_JST/artifacts.md) retain the evidence.
artifacts.extend(["[Execution settings](execution-settings.json) record the requested parameters, user-reported connection conditions, and frozen script hashes.", ""])
223
217
iftimingisnotNone:
224
218
artifacts.extend(["[Benchmark timing](timing.json) retains full-precision wall-clock timestamps and elapsed intervals; excludes build and report generation.", ""])
219
+
iftimingand"elapsedSeconds"intiming:
220
+
artifacts.extend(["", "## Benchmark elapsed time by delay", "",
221
+
"Each interval runs from the start of that delay's runner to completion of its last attachment export. Recovery gaps, if any, are included.", "",
artifacts.extend(["[Device baseline](logs/device-baseline/) records USB recognition, CoreDevice availability, and Mac power assertions before measurement. Failure snapshots are retained under each delay’s logs/device-diagnostics/ directory.", ""])
0 commit comments