Skip to content

Commit 73eef09

Browse files
committed
bench: compare master, compatibility fixes and rebased performance layer
1 parent e300f4e commit 73eef09

22 files changed

Lines changed: 37203 additions & 34 deletions
Lines changed: 33 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,33 @@
1-
phase,method,workload,mean_ns,standard_error_ns,confidence99_halfwidth_ns,samples,allocated_bytes
2-
Compilation,SystemCompile,Arithmetic,21057.400173611113,480.06713333032013,2596.146657766782,8,4447
3-
Compilation,FastCompile,Arithmetic,2136.472993116752,29.83925431808885,161.36718177467185,8,943
4-
Compilation,LightCompile,Arithmetic,1861.2931854812869,14.306173933474842,85.2478605594403,7,943
5-
Compilation,SystemCompile,Array,46886.166581483645,561.0764982423005,3034.2357860623824,8,5120
6-
Compilation,FastCompile,Array,2750.989670356446,24.45831453192665,145.74260011322139,7,1151
7-
Compilation,LightCompile,Array,2768.623869131542,29.66190207535884,160.40808168169767,8,1151
8-
Compilation,SystemCompile,Switch,29992.961711711712,233.24966158584692,1261.3867672017197,8,6182
9-
Compilation,FastCompile,Switch,2425.097851613464,42.57471854073646,230.23907610197057,8,1343
10-
Compilation,LightCompile,Switch,2286.3197476641903,10.490214069947784,56.729845278147806,8,1360
11-
Compilation,SystemCompile,CapturedConstant,34270.82298108068,315.7630793153843,1707.60963643542,8,4904
12-
Compilation,FastCompile,CapturedConstant,2065.5865263146625,23.705268182687245,128.19530538784718,8,1183
13-
Compilation,LightCompile,CapturedConstant,1975.462104590091,44.4838550235361,240.56346191163306,8,1183
14-
Construction,SystemTree,Arithmetic,440.8064759139369,1.9679978125448743,10.6427009657735,8,584
15-
Construction,LightTree,Arithmetic,178.25360727976607,0.9876112338033758,5.340885525790328,8,360
16-
Construction,SystemTree,Array,1058.828216831608,18.78200639040111,111.91852337604925,7,1192
17-
Construction,LightTree,Array,401.1927925192827,3.773679071513119,20.40761307909032,8,672
18-
Construction,SystemTree,Switch,1126.56504952069,10.951442884606463,65.25763494089202,7,1432
19-
Construction,LightTree,Switch,209.80873104222894,1.3395839536704643,7.2443126443456105,8,824
20-
Construction,SystemTree,CapturedConstant,508.16101190874906,4.123871927411091,22.301414902402737,8,528
21-
Construction,LightTree,CapturedConstant,237.72549601577592,0.8994772171331511,4.864267117805024,8,272
22-
Invocation,SystemInvoke,Arithmetic,1.1857331990510527,0.00393391632142369,0.021274157301601867,8,0
23-
Invocation,FastInvoke,Arithmetic,1.0989445437726775,0.00231112943902114,0.012498316489941222,8,0
24-
Invocation,LightInvoke,Arithmetic,1.0980764832439314,0.002266402297427581,0.012256437362840716,8,0
25-
Invocation,SystemInvoke,Array,1.0417457639223606,0.0023781178586783683,0.012860581993508882,8,0
26-
Invocation,FastInvoke,Array,1.2857391691880478,0.0006201253789979213,0.0036952131355322184,7,0
27-
Invocation,LightInvoke,Array,1.2984493339059076,0.004658099760368262,0.025190456260840082,8,0
28-
Invocation,SystemInvoke,Switch,1.0546706028009836,0.0016552381833178306,0.008951333634564273,8,0
29-
Invocation,FastInvoke,Switch,1.3091380151624001,0.0036995188509872305,0.022044752748883865,7,0
30-
Invocation,LightInvoke,Switch,1.510968037136868,0.01995102199199213,0.10789278304520543,8,0
31-
Invocation,SystemInvoke,CapturedConstant,1.0419648666720536,0.0032910366958241537,0.017797539813192806,8,0
32-
Invocation,FastInvoke,CapturedConstant,1.0429572324938534,0.0019130115577829154,0.010345341759920204,8,0
33-
Invocation,LightInvoke,CapturedConstant,1.0311589844091018,0.002251910363243133,0.01341871988491064,7,0
1+
phase,method,workload,mean_ns,standard_error_ns,confidence999_halfwidth_ns,samples,allocated_bytes
2+
Compilation,SystemCompile,Arithmetic,21057.400173611113,480.06713333032013,2596.146657766782,8,4447
3+
Compilation,FastCompile,Arithmetic,2136.472993116752,29.83925431808885,161.36718177467185,8,943
4+
Compilation,LightCompile,Arithmetic,1861.2931854812869,14.306173933474842,85.2478605594403,7,943
5+
Compilation,SystemCompile,Array,46886.166581483645,561.0764982423005,3034.2357860623824,8,5120
6+
Compilation,FastCompile,Array,2750.989670356446,24.45831453192665,145.74260011322139,7,1151
7+
Compilation,LightCompile,Array,2768.623869131542,29.66190207535884,160.40808168169767,8,1151
8+
Compilation,SystemCompile,Switch,29992.961711711712,233.24966158584692,1261.3867672017197,8,6182
9+
Compilation,FastCompile,Switch,2425.097851613464,42.57471854073646,230.23907610197057,8,1343
10+
Compilation,LightCompile,Switch,2286.3197476641903,10.490214069947784,56.729845278147806,8,1360
11+
Compilation,SystemCompile,CapturedConstant,34270.82298108068,315.7630793153843,1707.60963643542,8,4904
12+
Compilation,FastCompile,CapturedConstant,2065.5865263146625,23.705268182687245,128.19530538784718,8,1183
13+
Compilation,LightCompile,CapturedConstant,1975.462104590091,44.4838550235361,240.56346191163306,8,1183
14+
Construction,SystemTree,Arithmetic,440.8064759139369,1.9679978125448743,10.6427009657735,8,584
15+
Construction,LightTree,Arithmetic,178.25360727976607,0.9876112338033758,5.340885525790328,8,360
16+
Construction,SystemTree,Array,1058.828216831608,18.78200639040111,111.91852337604925,7,1192
17+
Construction,LightTree,Array,401.1927925192827,3.773679071513119,20.40761307909032,8,672
18+
Construction,SystemTree,Switch,1126.56504952069,10.951442884606463,65.25763494089202,7,1432
19+
Construction,LightTree,Switch,209.80873104222894,1.3395839536704643,7.2443126443456105,8,824
20+
Construction,SystemTree,CapturedConstant,508.16101190874906,4.123871927411091,22.301414902402737,8,528
21+
Construction,LightTree,CapturedConstant,237.72549601577592,0.8994772171331511,4.864267117805024,8,272
22+
Invocation,SystemInvoke,Arithmetic,1.1857331990510527,0.00393391632142369,0.021274157301601867,8,0
23+
Invocation,FastInvoke,Arithmetic,1.0989445437726775,0.00231112943902114,0.012498316489941222,8,0
24+
Invocation,LightInvoke,Arithmetic,1.0980764832439314,0.002266402297427581,0.012256437362840716,8,0
25+
Invocation,SystemInvoke,Array,1.0417457639223606,0.0023781178586783683,0.012860581993508882,8,0
26+
Invocation,FastInvoke,Array,1.2857391691880478,0.0006201253789979213,0.0036952131355322184,7,0
27+
Invocation,LightInvoke,Array,1.2984493339059076,0.004658099760368262,0.025190456260840082,8,0
28+
Invocation,SystemInvoke,Switch,1.0546706028009836,0.0016552381833178306,0.008951333634564273,8,0
29+
Invocation,FastInvoke,Switch,1.3091380151624001,0.0036995188509872305,0.022044752748883865,7,0
30+
Invocation,LightInvoke,Switch,1.510968037136868,0.01995102199199213,0.10789278304520543,8,0
31+
Invocation,SystemInvoke,CapturedConstant,1.0419648666720536,0.0032910366958241537,0.017797539813192806,8,0
32+
Invocation,FastInvoke,CapturedConstant,1.0429572324938534,0.0019130115577829154,0.010345341759920204,8,0
33+
Invocation,LightInvoke,CapturedConstant,1.0311589844091018,0.002251910363243133,0.01341871988491064,7,0
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
phase,method,workload,mean_ns,standard_error_ns,confidence999_halfwidth_ns,samples,allocated_bytes
2+
Compilation,SystemCompile,Arithmetic,15844.082357296715,117.14190604498756,698.0270838748723,7,4447
3+
Compilation,FastCompile,Arithmetic,1810.91773726065,30.040842258692937,179.0078565985373,7,944
4+
Compilation,LightCompile,Arithmetic,1591.1956593496586,29.792845279752683,161.11620714758004,8,944
5+
Compilation,SystemCompile,Array,42645.951429263565,175.57477483085358,949.4877553961624,8,5120
6+
Compilation,FastCompile,Array,2393.7356108293993,52.03943593546823,310.09343230745947,7,1151
7+
Compilation,LightCompile,Array,2185.8164648564903,37.15748954054916,221.4146494947734,7,1152
8+
Compilation,SystemCompile,Switch,27327.010866195436,202.3703170054131,1094.39489948025,8,6182
9+
Compilation,FastCompile,Switch,2286.794093559805,45.01491264595569,268.2355893464061,7,1343
10+
Compilation,LightCompile,Switch,2129.9730526867133,45.06550584541042,243.70896122278313,8,1360
11+
Compilation,SystemCompile,CapturedConstant,33563.45956486043,367.30287574658945,2188.6903151275565,7,4903
12+
Compilation,FastCompile,CapturedConstant,1817.31042777688,9.800219336368553,58.397705446864656,7,1183
13+
Compilation,LightCompile,CapturedConstant,1757.592018363402,9.947656127690006,53.79575566725131,8,1183
14+
Construction,SystemTree,Arithmetic,446.7495686599703,2.8599120400153017,17.04169010758502,7,584
15+
Construction,LightTree,Arithmetic,176.39085490854032,0.8008337624067482,4.772021172565207,7,360
16+
Construction,SystemTree,Array,996.792736595607,3.130061731290863,16.927006116669833,8,1192
17+
Construction,LightTree,Array,404.4762214847132,2.945943735290582,17.554337164486697,7,672
18+
Construction,SystemTree,Switch,1126.901258079409,31.6513087179164,171.16655908502597,8,1432
19+
Construction,LightTree,Switch,223.43289142023625,5.741280784497018,31.048171984972665,8,824
20+
Construction,SystemTree,CapturedConstant,539.7029327203401,14.629401962529194,79.11408712084024,8,528
21+
Construction,LightTree,CapturedConstant,242.382570351469,2.0556169724482802,11.11653508885029,8,272
22+
Invocation,SystemInvoke,Arithmetic,1.1907134338045033,0.0029981832636031018,0.016213822856770626,8,0
23+
Invocation,FastInvoke,Arithmetic,1.1009212499043197,0.0007561604413512465,0.0045058210647113866,7,0
24+
Invocation,LightInvoke,Arithmetic,1.1085954129484215,0.0018091742396718748,0.009783801742599426,8,0
25+
Invocation,SystemInvoke,Array,1.0707570448998405,0.0035985578343859822,0.021443144610664336,7,0
26+
Invocation,FastInvoke,Array,1.3038193604005623,0.0023612641365360586,0.014070338917411785,7,0
27+
Invocation,LightInvoke,Array,1.3035723313280687,0.002988970637560874,0.016164002057452692,8,0
28+
Invocation,SystemInvoke,Switch,1.055336958921157,0.0026761006616824456,0.014472037984516533,8,0
29+
Invocation,FastInvoke,Switch,1.3488832816288703,0.01258963691991997,0.06808327740624193,8,0
30+
Invocation,LightInvoke,Switch,1.4580039992194078,0.010160983525320704,0.05494940517149252,8,0
31+
Invocation,SystemInvoke,CapturedConstant,1.0591227805490677,0.005009427756522489,0.029850259115848318,7,0
32+
Invocation,FastInvoke,CapturedConstant,1.0598954502524944,0.003608426486662622,0.02150195009542222,7,0
33+
Invocation,LightInvoke,CapturedConstant,1.0776547983562088,0.007377067469176524,0.03989431420010333,8,0

‎test/FastExpressionCompiler.Performance/Measurements/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
# Historical screening results (original stack)
2+
3+
These measurements belong to the original revisions below. For current three-stage FEC results after rebasing onto #558, see [Rebase-20261024](Rebase-20261024/README.md). System rows in these historical runs are unchanged reference controls; their variation does not mean System was modified. The original array optimization is already present in #558 and is no longer a #559 production change.
4+
15
# Local performance work, 2026-10-18
26

37
Base: compatibility fixes `3f682768d6b23e445ae46595e456658899e371e4`.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* -text
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Three-stage comparison after rebasing #559 onto #558
2+
3+
## Exact revisions and retained scope
4+
5+
- Before #558: current upstream master without #558, `233ed5adaa9d9ceea67d71e153d0d2457f2cce00`. This includes merged #557/#562 and upstream SmallMap/conversion-helper repairs. It is not the historical original parent of #558; choosing this matched base isolates the current PR's incremental cost.
6+
- After #558: `aaa79fe73f0978e9be8150116d4175f04cb5c971`.
7+
- After #559: rebased production/test source `e300f4e2`.
8+
9+
Rebased the performance layer from `3f682768` onto current #558. The array-classification commit `7b6863bc` was dropped: #558 already uses IsSZArray on modern configured targets and retains its Framework fallback. The remaining production change is `d4a74798`, named LightExpression field lookup using GetDeclaredField with existing special-metadata handling. Native runner conflicts retain all #558 regressions and add the named-field/conditional oracles. No #558 branch rewrite or source change was made. The earlier conditional branch candidate remains rejected.
10+
11+
Historical measurement CSVs are preserved; dropped source does not erase historical evidence. The old Summarize confidence-column correction is retained. Historical System rows are unchanged reference controls, not changes to System caused by this PR. Current results below supersede historical timing claims for this rebased source.
12+
13+
## Compilation comparison
14+
15+
Ratios above 1 mean slower. Each value is the descriptive median of 41 workload-specific time ratios, not an aggregate application result. The ratio columns have independently computed medians, so they do not multiply exactly.
16+
17+
| Runtime/flavor | Before #558 | After #558 / before | After #559 / before | After #559 / #558 |
18+
| --- | ---: | ---: | ---: | ---: |
19+
| 8.0.28 standard | 1.000x | 1.0461x | 1.0890x | 1.0395x |
20+
| 8.0.28 light | 1.000x | 1.1690x | 1.2031x | 1.0441x |
21+
| 10.0.9 standard | 1.000x | 1.1164x | 1.2042x | 1.0375x |
22+
| 10.0.9 light | 1.000x | 1.0746x | 1.1041x | 1.0303x |
23+
24+
#558 still has measured compilation costs relative to current master: these medians are about 4.6–16.9% higher. #559 does not establish a compilation improvement: its medians are 3.0–4.4% higher than #558 in this run. All Standard production source files are byte-identical between #558/#559, so the Standard timing movement cannot be attributed to a Standard compiler code change. Light's compiler/emitter source is also identical; its expression factory implementation differs. These observations retain run-to-run/JIT/layout/host variation and do not establish a causal CPU regression or universal parity. See every workload and raw distribution rather than treating a median ratio as a guarantee.
25+
26+
Adaptive operation counts produced small fractional allocation residuals. A separate equal-count check (200 compilations/sample, 100 warmups, seven samples in each of two opposite-order rounds) confirms for **each runtime/flavor**: #558 allocates less in **14/41** workloads and the same in **27/41**, with **zero increases**; #559 matches #558 in **all 41**. Original adaptive residuals remain in the files rather than being rounded away. `allocation-confirmation.json` contains the equal-count values.
27+
28+
## Remaining optimization: Light tree construction
29+
30+
The named-field change affects construction, not a text parser. The broad construction pass caps at 20,000 operations; short fast samples were therefore confirmed separately with a higher cap (2,000,000) and a 30 ms calibration target. Both datasets remain available. Longer-run medians for the three selected affected shapes follow; unchanged constant/array and additional capture controls are retained in `construction-confirmation.json`.
31+
32+
| Light construction / runtime | After #558, ns | After #559, ns | B/op after #558 → #559 |
33+
| --- | ---: | ---: | ---: |
34+
| new_member / 8.0.28 | 127.0 | 108.2 | 304.01 → 272.01 |
35+
| capture_reference / 8.0.28 | 139.2 | 97.3 | 368.01 → 336.01 |
36+
| capture_mutable_reference_ctor / 8.0.28 | 226.6 | 178.3 | 704.03 → 672.03 |
37+
| new_member / 10.0.9 | 116.5 | 109.6 | 304.01 → 272.01 |
38+
| capture_reference / 10.0.9 | 97.5 | 92.1 | 368.02 → 336.01 |
39+
| capture_mutable_reference_ctor / 10.0.9 | 188.5 | 171.4 | 704.03 → 672.03 |
40+
41+
These affected workloads save approximately **32 B per lookup** on this host/runtime, rather than reusing the old report's 40 B figure. Observed time reductions are roughly 5.6–30.1% across the six entries; unchanged controls also move, so this is not a universal percentage claim. Other workload construction values, including slower/noisy ones, remain in the full data.
42+
43+
## Generated delegate execution
44+
45+
After #558 vs after #559, all **82 root delegate IL bodies** match for 41 workloads in both flavors on .NET 10. This checks root IL, not every nested delegate body or JIT-generated machine code. The compiler source is unchanged. No delegate runtime speedup is claimed. Broad runtime medians (after559/after558) range about 0.994–1.039x; the raw samples and each workload's allocation results are in the phase files. For factory workloads, `invoke` includes outer activation plus invocation of the resulting inner delegate, rather than only a scalar call. All generated delegates pass an untimed System behavior oracle for the tested inputs.
46+
47+
## System: separate unchanged reference
48+
49+
System was previously measured in multiple process runs as a timing-drift control. Those varying numbers did not represent patches to System; presenting them as optimization before/after columns was misleading. Here the main comparison has exactly three FEC revisions. System appears only as **one separately measured unchanged reference per runtime**, pooled over two reference process repetitions. It has no before/after delta, and no optimization claim is normalized to it. The standalone reference executable uses only System, without a FEC library dependency. There are no System construction rows in this reference. Sample compilation references:
50+
51+
| Unchanged System reference | Workload | Compile us |
52+
| --- | --- | ---: |
53+
| 8.0.28 | new_member | 17.118 |
54+
| 8.0.28 | array | 30.947 |
55+
| 8.0.28 | capture8 | 194.630 |
56+
| 8.0.28 | balanced256 | 246.676 |
57+
| 10.0.9 | new_member | 14.402 |
58+
| 10.0.9 | array | 29.845 |
59+
| 10.0.9 | capture8 | 154.196 |
60+
| 10.0.9 | balanced256 | 220.502 |
61+
62+
## Method and validation
63+
64+
Windows ARM64, .NET 8.0.28 / 10.0.9, SDK 10.0.301, Release, tiering and ReadyToRun disabled. Separate frozen source/binary snapshots, serial processes, forward/reverse FEC revision order, seven samples per phase per round. 41 common workloads cover scalars, arrays, switches, blocks, scopes/EH, invocation and captures. Construction, compilation of a prebuilt tree (including delegate creation), and warmed execution are separate. Light trees are constructed directly. Allocation uses the current thread's managed allocated-byte counter. There are 20,664 broad FEC observations plus 2,296 System-reference observations; 6,888 equal-count compilation observations and 1,176 longer construction observations are retained separately. Six non-common/new-feature benchmark shapes are listed in source-manifest.json and were not timed. They do not change the imported test exclusions. No fresh cold/first-invocation, Framework or x64 timing, retained-memory or native/JIT allocation claim is made.
65+
66+
Validated source `e300f4e2`: all production targets build; full .NET 10/.NET 8 ARM64 native suites pass (1,714 / 1,713 legacy cases plus registered regressions); Framework passes 1,049 TestX + 1,597 legacy cases. All 21 imported batches pass: **35,064 executed passes, zero executed failures/crashes, 69 existing exclusions unchanged**. The deferred deep-tree modes remain deferred. The retained BenchmarkDotNet project builds. The evidence-only commit changes no validated production/test code.
67+
68+
## Files
69+
70+
- `three-stage.csv`: absolute timings, allocations and three-stage ratios for every workload/phase/configuration.
71+
- `phase-medians.csv`: all FEC phase medians and the separate System references, including ranges/sample operation counts.
72+
- `allocation-confirmation.json`: equal-count compilation allocation proof.
73+
- `construction-confirmation.json`: longer construction controls.
74+
- `analysis.json`, `summary.csv`: full analysis; adaptive allocation counts are screening observations, superseded by equal-count confirmation for allocation conclusions.
75+
- `validation.json`: revisions, counters and source/assembly hashes; `il-comparison.json`: root delegate comparisons.
76+
- `source-manifest.json`, `binary-manifest.json`, `system-reference-manifest.json`: immutable snapshot identities.
77+
- `range-diff.txt`: old/rebased performance-layer mapping.
78+
- `evidence.zip` and `evidence-manifest.json`: complete sources, harnesses/scripts, project files, native/imported validation logs, raw measurements, confirmation data and manifests. Binaries are hashed and retained locally, not committed. Runners refuse to mix existing output with a new run; preserve/relocate archived output before replaying in a fresh directory. The initial System-reference build error is retained; the corrected reference project was then built and measured successfully.

0 commit comments

Comments
 (0)