Skip to content

Commit 6fecb0b

Browse files
committed
test: add protected live Solheim focus validation and verify follow-up submission
1 parent bb755c9 commit 6fecb0b

7 files changed

Lines changed: 396 additions & 21 deletions

File tree

‎.github/workflows/windows-chat-focus.yml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,20 @@ name: Windows chat focus E2E
22

33
on:
44
workflow_dispatch:
5+
inputs:
6+
live_solheim:
7+
description: Run real Solheim terminal/diff tests (requires environment approval)
8+
type: boolean
9+
default: false
10+
live_runner:
11+
description: Hosted Windows Server or an interactive self-hosted Windows 11 x64 runner
12+
type: choice
13+
options:
14+
- windows-2022
15+
- windows11-x64
16+
default: windows-2022
517
pull_request:
18+
types: [opened, synchronize, reopened, labeled]
619
paths:
720
- ".github/workflows/windows-chat-focus.yml"
821
- "jetbrains_plugin/scripts/ui-harness/**"
@@ -131,3 +144,77 @@ jobs:
131144
name: windows-chat-focus-${{ matrix.ide }}-${{ github.sha }}
132145
path: focus-evidence/
133146
if-no-files-found: error
147+
148+
live-solheim:
149+
name: Live Solheim terminal and diff focus (WebStorm 2026.2.1)
150+
needs: [build, windows-e2e]
151+
if: >-
152+
(github.event_name == 'workflow_dispatch' && inputs.live_solheim) ||
153+
(github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'live-solheim'))
154+
environment: solheim
155+
runs-on: ${{ inputs.live_runner == 'windows11-x64' && fromJSON('["self-hosted", "Windows", "X64", "windows11-x64"]') || 'windows-2022' }}
156+
timeout-minutes: 60
157+
defaults:
158+
run:
159+
shell: pwsh
160+
steps:
161+
- uses: actions/checkout@v5
162+
with:
163+
persist-credentials: false
164+
- uses: actions/setup-python@v6
165+
with:
166+
python-version: "3.12"
167+
- name: Validate relay authentication without provider calls
168+
run: python -m unittest discover -s jetbrains_plugin/scripts/ui-harness -p test_solheim_proxy.py -v
169+
- uses: actions/download-artifact@v4
170+
with:
171+
name: chat-focus-plugin
172+
path: focus-plugin
173+
- name: Download WebStorm 2026.2.1 with bundled JBR and JCEF
174+
run: |
175+
$ErrorActionPreference = 'Stop'
176+
New-Item -ItemType Directory -Force live-evidence | Out-Null
177+
$url = 'https://download.jetbrains.com/webstorm/WebStorm-2026.2.1.win.zip'
178+
$expected = '08c65a4c2fbc5c8ede2d95de83fddd1665327c970f52c750e6e6414265f834d7'
179+
Invoke-WebRequest $url -OutFile "$env:RUNNER_TEMP/webstorm-focus.zip"
180+
$actual = (Get-FileHash "$env:RUNNER_TEMP/webstorm-focus.zip" -Algorithm SHA256).Hash
181+
if ($actual -ne $expected) { throw 'WebStorm archive checksum mismatch' }
182+
"$url SHA256=$actual" | Set-Content live-evidence/ide-download.txt
183+
$idePath = Join-Path $env:RUNNER_TEMP ('webstorm-focus-' + [guid]::NewGuid().ToString('N'))
184+
Expand-Archive "$env:RUNNER_TEMP/webstorm-focus.zip" $idePath
185+
Remove-Item "$env:RUNNER_TEMP/webstorm-focus.zip"
186+
$launchers = @(Get-ChildItem $idePath -Filter webstorm64.exe -Recurse)
187+
if ($launchers.Count -ne 1) { throw 'Expected exactly one WebStorm launcher' }
188+
"FOCUS_IDE_DIR=$($launchers[0].Directory.Parent.FullName)" >> $env:GITHUB_ENV
189+
- name: Run three fresh live conversations without retrying failures
190+
env:
191+
SOLHEIM_API_KEY: ${{ secrets.SOLHEIM_API_KEY }}
192+
REQUIRE_WINDOWS11: ${{ inputs.live_runner == 'windows11-x64' }}
193+
run: |
194+
$plugins = @(Get-ChildItem focus-plugin -Filter *.zip)
195+
if ($plugins.Count -ne 1) { throw 'Expected exactly one plugin distribution' }
196+
# The runner removes the key from its child-process environment.
197+
# Retain it only in this PowerShell process for the next fresh profile.
198+
$liveKey = $env:SOLHEIM_API_KEY
199+
try {
200+
for ($iteration = 1; $iteration -le 3; $iteration++) {
201+
$env:SOLHEIM_API_KEY = $liveKey
202+
& ./jetbrains_plugin/scripts/ui-harness/run-windows.ps1 `
203+
-IdeDir $env:FOCUS_IDE_DIR -PluginZip $plugins[0].FullName `
204+
-Provider solheim -DeadlineSeconds 780 -RequireWindows11:($env:REQUIRE_WINDOWS11 -eq 'true') `
205+
-EvidenceDir "$env:GITHUB_WORKSPACE/live-evidence/run-$iteration"
206+
}
207+
} finally { $liveKey = $null; $env:SOLHEIM_API_KEY = $null }
208+
- name: Upload live focus evidence and tool coverage
209+
if: always()
210+
uses: actions/upload-artifact@v4
211+
with:
212+
name: live-solheim-webstorm-2026.2.1-${{ github.sha }}
213+
path: live-evidence/
214+
if-no-files-found: error
215+
- name: Remove downloaded IDE on self-hosted runners
216+
if: always() && inputs.live_runner == 'windows11-x64'
217+
run: |
218+
if ($env:FOCUS_IDE_DIR -and (Test-Path $env:FOCUS_IDE_DIR)) {
219+
Remove-Item $env:FOCUS_IDE_DIR -Recurse -Force
220+
}

‎docs/validation/chat-focus-preservation.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,3 +291,61 @@ turn but lost DOM snapshot delivery. That intermediate run also used the old
291291
stream observer. The passing run is evidence that the corrected scenario works,
292292
not a claim that startup/snapshot failures or the original product issue are
293293
resolved. Windows CI must pass without retrying or discarding failed samples.
294+
295+
## Protected live Windows validation prepared (2026-11-06)
296+
297+
The workflow now includes an opt-in `live-solheim` job behind Environment
298+
`solheim` approval. It tests WebStorm 2026.2.1 with the real Solheim provider,
299+
three fresh two-turn runs, terminal completion markers, required provider tool
300+
names, the actual edited fixture, and preserving-diff log evidence in addition
301+
to exact typing and sampled focus checks. The upstream key is supplied only to
302+
the loopback relay and is removed from the IDE's process environment.
303+
304+
This is prepared test infrastructure, **not a live-provider PASS**. Hosted
305+
Windows Server remains distinct from the reported Windows 11 environment.
306+
Manual dispatch can select an interactive self-hosted Windows 11 x64 runner;
307+
the runner then validates the OS/architecture. Evidence must be reviewed before
308+
closing #5. See the harness README for the environment secret and launch steps.
309+
310+
## Local live Solheim verification (2026-11-06)
311+
312+
**Accepted live PASS**, 93 seconds, on Linux Xvfb/Openbox with
313+
IU-253.28294.334 (JBR 21.0.8). Endpoint `https://api.solheim.ai/v1`, model
314+
`qwen3.8-27b`, real native tool calls through the loopback relay. The plugin is
315+
the artifact built by the passing PR CI run
316+
[42154063861](https://github.com/Zoo-Code-Org/Zoo-Code-Jetbrains/actions/runs/42154063861),
317+
ZIP SHA-256
318+
`edc9bd9634e852a4dbc24e80a8f2baa91320b49f05894c6b457e5c0be7affdb3`,
319+
with the current local harness driver. The smaller connectivity probe separately
320+
returned HTTP 200, streaming events, and an `attempt_completion` tool call.
321+
322+
Accepted run evidence: `/tmp/zoo-solheim-e2e.0Ox96IMb/evidence/p/`. All 18
323+
first-turn and five follow-up draft samples matched exactly, with textarea and
324+
browser-window focus preserved. Both turns completed with partial updates during
325+
the typing loops. IDE logs confirm all three requested terminal commands exited
326+
0, the actual file changed to `solheim_e2e`, and the diff opened with
327+
`preserveFocus=true` and completed. Observed provider responses returned HTTP
328+
200 and included `execute_command`, `read_file`, `apply_diff`, and
329+
`attempt_completion`. The temporary upstream key file was deleted after relay
330+
startup; the relay was stopped after the run. No upstream credential is recorded
331+
here or in test artifacts.
332+
333+
An earlier local run at `/tmp/zoo-solheim-e2e.ikw85vsS/evidence/p/` reported
334+
PASS after 138 seconds but is **rejected as insufficient coverage**: the intended
335+
follow-up prompt was not pasted, the preserved first-turn draft was submitted,
336+
and the third command never ran. Its raw PASS is retained, not rewritten. This
337+
exposed a harness defect: accepting a completed `say` message before the final
338+
completed-task ask and submitting the next turn without checking native paste.
339+
The driver now waits for the completed-task ask, uses normal native key delays,
340+
verifies exact follow-up input before Enter, and requires approval of all three
341+
expected commands. The accepted run above exercises those corrected checks.
342+
343+
These are temporary local evidence paths. This verifies the live-provider
344+
scenario on Linux, not the new Windows PowerShell runner, WebStorm, or Windows
345+
11. The protected Windows job still needs to run before #5 can be evaluated for
346+
closure on its reported environment.
347+
348+
The corrected driver also passed the credential-free local fixture E2E in
349+
88 seconds using the same downloaded plugin artifact. Both turns streamed,
350+
the exact follow-up prompt was verified, and draft/focus checks passed. Evidence:
351+
`/tmp/zoo-solheim-e2e.TxEsZADF/evidence/p/`.

0 commit comments

Comments
 (0)