Skip to content

Commit f45b95b

Browse files
Add Ghostty comparison benchmarks and speed up the builtin engine (#61)
## Summary This adds a benchmark that compares DT's default engine with Ghostty on identical input, then uses it to speed the default engine up. The default engine now matches or beats the Ghostty library on 8 of 13 workloads. It started at 0.01–0.4× on most of them. Full methodology, results and remaining gaps: `docs/performance-benchmarks.md`. ### Benchmarks - **New `vt` mode** in `tools/Devolutions.Terminal.Bench`. It feeds identical bytes, in one process, to three targets: - the builtin engine; - raw `libghostty-vt`; - DT's Ghostty engine. - **Before timing**, it checks that the final screens match. - **The verdict is conservative:** `faster` only when the builtin's slowest run beats Ghostty's fastest. - **13 deterministic workloads**, modeled on `ghostty-gen` and vtebench. `--frame-kb N` also takes a screen snapshot every N KiB, as the renderer would. - **`scripts/Invoke-TerminalBenchmarks.ps1`** runs one process per workload. - `-Baseline` fails on a builtin/Ghostty ratio drop of more than 15% on identical inputs. - `-GhosttyBench` also replays the inputs through upstream `ghostty-bench`. ### Builtin engine - **Packed cell storage** (`CellRow`/`PackedCell`): 16 bytes per cell with no garbage-collected references, down from about 40. Combining marks and hyperlinks move to per-line side arrays. The public `Cell` and `TermColor` types are unchanged. - **Logically blank cells are never written.** Reusing a scrollback line and erasing to the end of a line are O(1). - **Bulk paths**, each checked against the original character-by-character path: - printable ASCII (AVX2 where available); - well-formed UTF-8 runs; - plain `ESC [` sequences; - control characters such as CR and LF. - **Lookup tables** for character width and grapheme joining, covering planes 0 and 1. ### Ghostty engine - The screen is copied out of Ghostty only when something reads it, and only for the rows Ghostty reports as changed. Before, it was copied in full after every chunk of output. ## Results (one Windows x64 machine, builtin speed ÷ Ghostty speed in the same run) | Workload | Before | After | | --- | ---: | ---: | | ascii / ascii-lines / light-cells | 0.009x | 1.19–1.24x | | scrolling (`y\n`) | 0.015x | 1.33x | | dense-cells | 0.42x | 1.44x | | styled / osc | 0.30x / 0.29x | 2.91x / 4.59x | | mixed | 0.03x | 1.84x | | scroll-region / cursor-motion | 0.018x / 0.11x | 0.96x / 0.90x | | utf8 / cjk / emoji | 0.10x / — / — | 0.77x / 0.77x / 0.80x | Screens are identical on every workload except one `styled` row. There, the builtin engine drops a combining mark that follows a space, a pre-existing difference. ## Known gaps - **Non-ASCII text is still about 0.8×.** Closing it needs a dedicated SIMD UTF-8 decoder or 8-byte cells. - **Frame snapshots cost about 0.03 ms more** (0.11 ms per 120x30 frame), because each packed cell is unpacked into a public `Cell`. - **These are JIT numbers** from one Windows machine, not from the NativeAOT build. ## Tests - New tests: - `AsciiFastPathTests`, `CsiFastPathTests`, `CellRowModelTests`, `PackedCellStorageTests` (Core); - `GhosttyProjectionTests` (Ghostty). - Every new shortcut was also checked by deliberately breaking it and confirming a test fails. - Release build of `Devolutions.Terminal.slnx`: 0 warnings, 0 errors. - Core, Render, Control, Ghostty, Compatibility, Connection, Cli, App, UI, Broker and Settings tests all pass (3,394 tests; 11 skipped in Connection). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent dc1cd26 commit f45b95b

23 files changed

Lines changed: 4217 additions & 191 deletions

‎docs/ghostty-engine.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,13 @@ onto a dedicated thread with an explicit `maxStackSize`.
5353

5454
Ghostty owns VT parsing, modes, viewport state, resize/reflow, scrollback,
5555
terminal replies, titles, and bells. Its render-state API is projected into the
56-
shared immutable cell model consumed by the existing Skia renderer. This keeps
56+
shared immutable cell model consumed by the existing Skia renderer.
57+
58+
Projection is lazy. A feed only marks the projection stale. The next read of
59+
`Buffer`, the cursor or a snapshot (normally once per frame) re-projects the
60+
rows that Ghostty's render state reports dirty, then cleans that state. Title
61+
and working-directory changes are still raised during the feed. See
62+
[performance benchmarks](performance-benchmarks.md) for the measured effect. This keeps
5763
Avalonia tabs, panes, accessibility, clipboard, selection, search, and window
5864
behavior common across engines.
5965

‎docs/performance-benchmarks.md‎

Lines changed: 391 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/renderer-benchmarks.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ The `render` mode of `tools/Devolutions.Terminal.Bench` measures the CPU work of
44
preparing and rasterizing terminal frames. It is not a GPU or presentation
55
benchmark. The existing `engine` and `control` modes measure different parts
66
of the pipeline; `control` uses Avalonia headless and does **not** paint.
7+
For the engine-level comparison against Ghostty (`vt` mode), the full suite
8+
script and the "beats Ghostty" criteria, see
9+
[performance benchmarks](performance-benchmarks.md).
710

811
## Reproducible input
912

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
[CmdletBinding()]
2+
param(
3+
# vt: builtin vs Ghostty ingest; render: frame prep + CPU raster; control: PTY->UI coalescing.
4+
[string[]] $Suite = @("vt", "render"),
5+
6+
# Smaller corpora and fewer runs for a fast local signal (not for published numbers).
7+
[switch] $Quick,
8+
9+
[string] $OutputDirectory,
10+
11+
# Comma-separated vt workloads; "all" runs every workload.
12+
[string] $Workloads = "all",
13+
14+
# Read a frame (snapshot / render-state update) every N KiB fed; 0 measures pure ingest.
15+
[ValidateRange(0, 1048576)]
16+
[int] $FrameKb = 0,
17+
18+
# A previous run's vt.json; medians that drop by more than -RegressionPercent fail the run.
19+
[string] $Baseline,
20+
21+
[ValidateRange(1, 100)]
22+
[int] $RegressionPercent = 15,
23+
24+
# Upstream ghostty-bench built from the pinned commit (see docs/performance-benchmarks.md).
25+
# When given, every vt workload is also replayed through `+terminal-stream`.
26+
[string] $GhosttyBench,
27+
28+
# Fail when builtin is not faster than raw libghostty-vt on every workload.
29+
[switch] $RequireBeatGhostty
30+
)
31+
32+
Set-StrictMode -Version Latest
33+
$ErrorActionPreference = "Stop"
34+
35+
# `pwsh -File` passes "vt,render" as one string; accept both forms.
36+
$Suite = @($Suite | ForEach-Object { $_ -split "," } | ForEach-Object { $_.Trim() } | Where-Object { $_ })
37+
foreach ($name in $Suite) {
38+
if ($name -notin @("vt", "render", "control")) {
39+
throw "Unknown suite '$name' (expected vt, render or control)."
40+
}
41+
}
42+
43+
$repo = Split-Path -Parent $PSScriptRoot
44+
$project = Join-Path $repo "tools\Devolutions.Terminal.Bench"
45+
if (-not $OutputDirectory) {
46+
$OutputDirectory = Join-Path $repo ("artifacts\bench\" + (Get-Date -Format "yyyyMMdd-HHmmss"))
47+
}
48+
49+
New-Item -ItemType Directory -Force -Path $OutputDirectory | Out-Null
50+
$OutputDirectory = (Resolve-Path -LiteralPath $OutputDirectory).Path
51+
52+
$megabytes = if ($Quick) { 2 } else { 8 }
53+
$runs = if ($Quick) { 3 } else { 7 }
54+
55+
Write-Host "Building $project (Release)..."
56+
dotnet build $project -c Release --nologo -v quiet
57+
if ($LASTEXITCODE -ne 0) {
58+
throw "Benchmark build failed."
59+
}
60+
61+
$bench = Get-ChildItem -Path (Join-Path $project "bin\Release") -Recurse -File |
62+
Where-Object { $_.BaseName -eq "Devolutions.Terminal.Bench" -and $_.Extension -in @(".exe", "") } |
63+
Select-Object -First 1
64+
if (-not $bench) {
65+
$bench = Get-ChildItem -Path (Join-Path $project "bin\Release") -Recurse -Filter "Devolutions.Terminal.Bench.dll" |
66+
Select-Object -First 1
67+
}
68+
69+
function Invoke-Bench {
70+
param([string] $Name, [string[]] $Arguments)
71+
72+
$log = Join-Path $OutputDirectory "$Name.txt"
73+
Write-Host ""
74+
Write-Host "== $Name =="
75+
# Out-Host keeps the transcript out of the function's return value.
76+
if ($bench.Extension -eq ".dll") {
77+
& dotnet $bench.FullName @Arguments 2>&1 | Tee-Object -FilePath $log | Out-Host
78+
}
79+
else {
80+
& $bench.FullName @Arguments 2>&1 | Tee-Object -FilePath $log | Out-Host
81+
}
82+
83+
return $LASTEXITCODE
84+
}
85+
86+
$failures = [System.Collections.Generic.List[string]]::new()
87+
88+
if ($Suite -contains "vt") {
89+
$vtJson = Join-Path $OutputDirectory "vt.json"
90+
$names = if ($Workloads -eq "all") {
91+
if ($bench.Extension -eq ".dll") { & dotnet $bench.FullName vt --list } else { & $bench.FullName vt --list }
92+
}
93+
else {
94+
$Workloads -split "," | ForEach-Object { $_.Trim() } | Where-Object { $_ }
95+
}
96+
97+
# One process per workload: GC heap and JIT state left behind by an earlier
98+
# workload shifted later ratios by up to ~50% in a shared process, which made
99+
# runs with different -Workloads subsets incomparable.
100+
$merged = $null
101+
foreach ($name in $names) {
102+
$partial = Join-Path $OutputDirectory "vt-$name.json"
103+
$vtArgs = @("vt", "--mb", $megabytes, "--runs", $runs, "--frame-kb", $FrameKb, "--workloads", $name, "--json", $partial)
104+
if (-not $RequireBeatGhostty) {
105+
$vtArgs += "--no-gate"
106+
}
107+
108+
if ((Invoke-Bench -Name "vt-$name" -Arguments $vtArgs) -ne 0) {
109+
$failures.Add("builtin did not beat ghostty-vt on vt workload '$name'")
110+
}
111+
112+
$report = Get-Content -LiteralPath $partial -Raw | ConvertFrom-Json
113+
if ($merged) {
114+
$merged.workloads = @($merged.workloads) + @($report.workloads)
115+
}
116+
else {
117+
$merged = $report
118+
}
119+
}
120+
121+
$merged | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $vtJson
122+
123+
if ($GhosttyBench) {
124+
$corpora = Join-Path $OutputDirectory "corpora"
125+
Invoke-Bench -Name "vt-export" -Arguments @("vt", "--mb", $megabytes, "--workloads", $Workloads, "--export", $corpora) | Out-Null
126+
$upstream = foreach ($file in Get-ChildItem -LiteralPath $corpora -Filter "*.vt") {
127+
$times = foreach ($run in 1..$runs) {
128+
$watch = [System.Diagnostics.Stopwatch]::StartNew()
129+
& $GhosttyBench "+terminal-stream" "--data=$($file.FullName)" "--terminal-rows=30" "--terminal-cols=120" | Out-Null
130+
$watch.Stop()
131+
if ($LASTEXITCODE -ne 0) {
132+
throw "ghostty-bench failed on $($file.Name)."
133+
}
134+
135+
$watch.Elapsed.TotalSeconds
136+
}
137+
138+
$sorted = @($times | Sort-Object)
139+
$median = $sorted[[int][Math]::Floor($sorted.Count / 2)]
140+
[pscustomobject]@{
141+
Workload = $file.BaseName
142+
MedianMBps = [Math]::Round($file.Length / 1MB / $median, 1)
143+
Note = "process wall time, includes startup and file load"
144+
}
145+
}
146+
147+
$upstream | Format-Table -AutoSize | Out-String | Tee-Object -FilePath (Join-Path $OutputDirectory "ghostty-bench.txt")
148+
$upstream | ConvertTo-Json | Set-Content -LiteralPath (Join-Path $OutputDirectory "ghostty-bench.json")
149+
}
150+
151+
if ($Baseline) {
152+
$previous = Get-Content -LiteralPath $Baseline -Raw | ConvertFrom-Json
153+
$current = Get-Content -LiteralPath $vtJson -Raw | ConvertFrom-Json
154+
Write-Host ""
155+
Write-Host "== regression check vs $Baseline (threshold $RegressionPercent%) =="
156+
foreach ($workload in $current.workloads) {
157+
$before = $previous.workloads | Where-Object { $_.name -eq $workload.name } | Select-Object -First 1
158+
if (-not $before) {
159+
continue
160+
}
161+
162+
if ($before.sha256 -ne $workload.sha256) {
163+
Write-Host ("{0,-14} skipped: input differs from the baseline (use the same -Quick/-Workloads settings)" -f $workload.name)
164+
continue
165+
}
166+
167+
foreach ($target in $workload.targets) {
168+
$old = $before.targets | Where-Object { $_.target -eq $target.target } | Select-Object -First 1
169+
if ($old) {
170+
$change = ($target.medianMBps - $old.medianMBps) / $old.medianMBps * 100
171+
Write-Host ("{0,-14} {1,-10} {2,9:F1} -> {3,9:F1} MB/s ({4:+0.0;-0.0}%)" -f $workload.name, $target.target, $old.medianMBps, $target.medianMBps, $change)
172+
}
173+
}
174+
175+
# Absolute MB/s drifts with clocks and background load between sessions, so
176+
# gate on builtin relative to ghostty-vt measured in the same run: Ghostty is
177+
# the control, and only the engine under our control can regress.
178+
$oldRatio = $before.PSObject.Properties["builtinVsGhosttyVt"]
179+
$newRatio = $workload.PSObject.Properties["builtinVsGhosttyVt"]
180+
if ($oldRatio -and $newRatio -and $oldRatio.Value -gt 0) {
181+
$change = ($newRatio.Value - $oldRatio.Value) / $oldRatio.Value * 100
182+
$line = "{0,-14} builtin/ghostty-vt {1:F3} -> {2:F3} ({3:+0.0;-0.0}%)" -f $workload.name, $oldRatio.Value, $newRatio.Value, $change
183+
if ($change -lt -$RegressionPercent) {
184+
$failures.Add("regression: $line")
185+
Write-Host "$line REGRESSION"
186+
}
187+
else {
188+
Write-Host $line
189+
}
190+
}
191+
}
192+
}
193+
}
194+
195+
if ($Suite -contains "render") {
196+
$renderArgs = if ($Quick) {
197+
@("render", "--mb", $megabytes, "--runs", $runs, "--samples", "60", "--warmup", "2", "--frames", "64")
198+
}
199+
else {
200+
@("render", "--mb", $megabytes, "--runs", "5", "--samples", "128", "--warmup", "2", "--frames", "256")
201+
}
202+
203+
if ((Invoke-Bench -Name "render" -Arguments $renderArgs) -ne 0) {
204+
$failures.Add("render benchmark failed")
205+
}
206+
}
207+
208+
if ($Suite -contains "control") {
209+
if ((Invoke-Bench -Name "control" -Arguments @("control", "--mb", $megabytes, "--runs", $runs)) -ne 0) {
210+
$failures.Add("control benchmark failed")
211+
}
212+
}
213+
214+
Write-Host ""
215+
Write-Host "Results: $OutputDirectory"
216+
if ($failures.Count -gt 0) {
217+
$failures | ForEach-Object { Write-Warning $_ }
218+
exit 1
219+
}

0 commit comments

Comments
 (0)