Skip to content

Commit 0d3d0fb

Browse files
committed
test(windows): add red regressions for platform-specific failures
Adds Windows-only red tests and analysis for native Windows failures found during a Windows red-test campaign. This change contains no production fixes. - windows_non_ascii_repo_path_preserves_definitions (integration): byte-identical TypeScript fixtures indexed under non-ASCII parent paths (Latin-1 accents, Cyrillic, CJK, Greek) extract zero definitions and only File/Folder nodes (5 nodes / 4 edges) versus the ASCII baseline (12 nodes / 20 edges / 5 definitions). The pipeline source readers open files with fopen() on a UTF-8 path, which the Windows CRT interprets in the active ANSI code page; directory discovery already uses the wide API, so files are listed but never parsed. - windows_cli_non_ascii_repo_path_is_honored (integration): the documented `cli index_repository` entrypoint rejects a non-ASCII repo_path because main() does not read the wide command line, so argv arrives in the ANSI code page. Both reproduce at the product surface (real MCP process, real stdio, real SQLite DB), are deterministic, and pass on Linux/macOS. A PowerShell runner builds the binary and runs the suite; standard-library Python only. See tests/windows/RED_TEST_ANALYSIS.md for environment, commands, ruled-out seed areas, and suspected fix locations. Signed-off-by: Flipper <jacobphilipp@ymail.com>
1 parent 9770803 commit 0d3d0fb

6 files changed

Lines changed: 694 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,7 @@ soak-results/
6262

6363
# Local npm cache
6464
graph-ui/.npm-cache-local/
65+
66+
# Python bytecode from tests/windows/ harness
67+
__pycache__/
68+
*.pyc

‎scripts/test-windows.ps1‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
<#
2+
.SYNOPSIS
3+
Run the native-Windows red-test suite for codebase-memory-mcp.
4+
5+
.DESCRIPTION
6+
Builds the production binary (build/c/codebase-memory-mcp.exe) if it is not
7+
already present, then runs the deterministic Windows red tests under
8+
tests/windows/. These tests reproduce platform-specific failures at the
9+
product surface (real MCP process, real stdio, real SQLite DB).
10+
11+
The unit/invariant C suite is built and run via Makefile.cbm. On native
12+
Windows the MinGW/LLVM toolchain ships no libasan/libubsan, so the sanitizer
13+
flags must be disabled for the local build (SANITIZE=). Where the toolchain
14+
*does* provide AddressSanitizer/UBSan (Linux containers, WSL), prefer
15+
scripts/test.sh which keeps the sanitizers on.
16+
17+
.PARAMETER Binary
18+
Path to an existing codebase-memory-mcp.exe. If omitted, the script looks for
19+
build/c/codebase-memory-mcp.exe and builds it when missing.
20+
21+
.PARAMETER Make
22+
Path to GNU make (default: 'make' on PATH; MSYS2 ships it at
23+
C:\msys64\usr\bin\make.exe).
24+
25+
.EXAMPLE
26+
pwsh -File scripts/test-windows.ps1
27+
#>
28+
[CmdletBinding()]
29+
param(
30+
[string]$Binary,
31+
[string]$Make = "make"
32+
)
33+
34+
$ErrorActionPreference = "Stop"
35+
$repoRoot = Split-Path -Parent $PSScriptRoot
36+
Set-Location $repoRoot
37+
38+
$python = (Get-Command python -ErrorAction SilentlyContinue)
39+
if (-not $python) { $python = (Get-Command py -ErrorAction SilentlyContinue) }
40+
if (-not $python) { throw "Python 3 is required to run the Windows red tests." }
41+
$py = $python.Source
42+
43+
# A writable Windows temp dir that GNU make forwards to the native gcc. MSYS2
44+
# strips TMP/TEMP from the environment it hands native children, so pass them as
45+
# make command-line variables (make exports those to recipe processes).
46+
$tmp = $env:TEMP
47+
if (-not $tmp) { $tmp = "$env:USERPROFILE\AppData\Local\Temp" }
48+
49+
function Resolve-Binary {
50+
param([string]$Explicit)
51+
if ($Explicit) { return (Resolve-Path $Explicit).Path }
52+
$built = Join-Path $repoRoot "build\c\codebase-memory-mcp.exe"
53+
if (Test-Path $built) { return $built }
54+
Write-Host "Building production binary via Makefile.cbm ..." -ForegroundColor Cyan
55+
& $Make "-j" "-f" "Makefile.cbm" "cbm" "TMP=$tmp" "TEMP=$tmp" "TMPDIR=$tmp"
56+
if ($LASTEXITCODE -ne 0) { throw "build failed (exit $LASTEXITCODE)" }
57+
if (-not (Test-Path $built)) { throw "binary not produced at $built" }
58+
return $built
59+
}
60+
61+
$bin = Resolve-Binary -Explicit $Binary
62+
Write-Host "Binary: $bin" -ForegroundColor Green
63+
64+
$env:PYTHONUTF8 = "1" # ensure the harness encodes argv/stdio as UTF-8
65+
66+
$tests = @(
67+
"tests\windows\test_non_ascii_path.py",
68+
"tests\windows\test_cli_non_ascii_arg.py"
69+
)
70+
71+
$failed = @()
72+
foreach ($t in $tests) {
73+
Write-Host "`n=== $t ===" -ForegroundColor Cyan
74+
& $py $t $bin
75+
$code = $LASTEXITCODE
76+
if ($code -eq 0) {
77+
Write-Host "GREEN ($t)" -ForegroundColor Green
78+
} elseif ($code -eq 1) {
79+
Write-Host "RED ($t) - Windows-specific failure reproduced" -ForegroundColor Red
80+
$failed += $t
81+
} else {
82+
Write-Host "SETUP ERROR ($t) exit=$code" -ForegroundColor Yellow
83+
$failed += $t
84+
}
85+
}
86+
87+
Write-Host ""
88+
if ($failed.Count -gt 0) {
89+
Write-Host ("RED suite: {0}/{1} Windows red tests failed (expected until the " -f $failed.Count, $tests.Count) -ForegroundColor Red
90+
Write-Host "platform issues are fixed). See tests/windows/RED_TEST_ANALYSIS.md." -ForegroundColor Red
91+
exit 1
92+
}
93+
Write-Host "All Windows red tests are GREEN." -ForegroundColor Green
94+
exit 0

‎tests/windows/RED_TEST_ANALYSIS.md‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Windows Red-Test Analysis
2+
3+
Deterministic, Windows-only red tests found during a native-Windows red-test
4+
campaign. They reproduce platform-specific failures at the product surface and
5+
are intended as regression guards while the underlying issues are fixed in
6+
separate maintainer PRs. **This PR contains no production fixes.**
7+
8+
## Environment
9+
10+
- OS: Microsoft Windows 11 Pro, build 10.0.26200
11+
- Source build: MinGW-w64 GCC 15.2.0 (MSYS2), `make -f Makefile.cbm cbm`
12+
- Filesystem: NTFS, code page 65001 (UTF-8 console)
13+
- Shells/launchers exercised: PowerShell 5.1 (5.1.26100), `cmd.exe`,
14+
Git Bash (MSYS2), direct Win32 process launch, Python `subprocess.Popen`,
15+
Python stdio (line-delimited JSON-RPC) transport
16+
- CBM source commit under test: `b075f05`
17+
- Binary: `build/c/codebase-memory-mcp.exe` (production build)
18+
19+
### Sanitizer note
20+
21+
The MinGW/LLVM toolchain available on this machine ships **no** `libasan` /
22+
`libubsan`, so an AddressSanitizer/UBSan build is not possible natively (the plan
23+
anticipates this). The C unit/invariant suite (`build/c/test-runner`) was built
24+
with `SANITIZE=` and runs; the two red tests below are product-level integration
25+
tests that drive a real `codebase-memory-mcp.exe` over stdio. On a host where the
26+
toolchain provides sanitizers (Linux container, WSL), the same fixtures should be
27+
run through an ASan/UBSan binary via `scripts/test.sh`.
28+
29+
## How to run
30+
31+
```powershell
32+
# Builds build/c/codebase-memory-mcp.exe if missing, then runs the red suite.
33+
pwsh -File scripts/test-windows.ps1
34+
# or, against an installed/relocated binary:
35+
pwsh -File scripts/test-windows.ps1 -Binary "C:\path\to\codebase-memory-mcp.exe"
36+
```
37+
38+
Each test exits `0` (green / invariant holds), `1` (red / Windows failure
39+
reproduced), or `2` (environment/setup error). Standard-library Python 3 only.
40+
41+
---
42+
43+
## windows_non_ascii_repo_path_preserves_definitions
44+
45+
- Class: integration
46+
- Test: `tests/windows/test_non_ascii_path.py`
47+
- Related issues: #636, #357, #571 (naming), #530
48+
- Environment: Windows 11 26200, PowerShell 5.1 / Python stdio, NTFS, CP 65001
49+
- Fixture: byte-identical 2-file TypeScript repo (`src/math.ts`, `src/main.ts`),
50+
copied to an ASCII parent path and to four non-ASCII parent paths
51+
(Latin-1 accents `café`, Cyrillic `проект`, CJK `日本語`, Greek `Ωμέγα`)
52+
- Expected: each non-ASCII copy produces the same graph counts as the ASCII
53+
baseline (12 nodes / 20 edges / 5 definition nodes)
54+
- Actual: every non-ASCII copy produces **5 nodes / 4 edges / 0 definition
55+
nodes** — only `File`/`Folder` nodes; zero `Function`/`Class`/`Method`
56+
- Command: `python tests/windows/test_non_ascii_path.py build\c\codebase-memory-mcp.exe`
57+
- Minimal failure output:
58+
59+
```
60+
baseline (ASCII): nodes=12 edges=20 definitions=5
61+
[FAIL] non-ascii/latin1_accents nodes=5 edges=4 definitions=0 (baseline 12/20/5)
62+
[FAIL] non-ascii/cyrillic nodes=5 edges=4 definitions=0 (baseline 12/20/5)
63+
[FAIL] non-ascii/cjk nodes=5 edges=4 definitions=0 (baseline 12/20/5)
64+
[FAIL] non-ascii/greek nodes=5 edges=4 definitions=0 (baseline 12/20/5)
65+
```
66+
67+
- Suspected implementation area: the per-pass source readers
68+
`read_file()` in `src/pipeline/pass_definitions.c`, `pass_calls.c`,
69+
`pass_parallel.c`, `pass_semantic.c` (and the `k8s`/`lsp_cross`/`pkgmap`
70+
variants) open files with plain `fopen(path, "rb")`. On Windows `fopen`
71+
interprets the UTF-8 path in the active **ANSI code page**, so a path with
72+
non-ASCII bytes cannot be opened and the tree-sitter parser receives no bytes.
73+
Directory discovery already uses the wide API
74+
(`cbm_utf8_to_wide` + `FindFirstFileW` in `src/foundation/compat_fs.c`,
75+
`src/foundation/platform.c`), which is why `File`/`Folder` nodes still appear
76+
while all definitions vanish. Fix direction: route the pass-level reads through
77+
the wide layer (`cbm_utf8_to_wide` + `_wfopen`), or add a shared
78+
UTF-8-aware file reader and use it from every pass.
79+
80+
Verified with `_wfopen` vs `fopen` on a non-ASCII path: `fopen(utf8, "rb")`
81+
returns `NULL`, `_wfopen(cbm_utf8_to_wide(utf8), L"rb")` opens the same file.
82+
83+
This invariant holds on Linux/macOS (byte-transparent UTF-8 filesystem); the test
84+
turns green once the pass readers convert to wide.
85+
86+
---
87+
88+
## windows_cli_non_ascii_repo_path_is_honored
89+
90+
- Class: integration
91+
- Test: `tests/windows/test_cli_non_ascii_arg.py`
92+
- Related issues: #636, #423, #20
93+
- Environment: Windows 11 26200, `cli` argv path, NTFS, CP 65001
94+
- Fixture: a TypeScript repo under a non-ASCII directory (`café_日本語_repo`),
95+
created with the OS wide API so it genuinely exists; an ASCII control repo
96+
- Expected: `codebase-memory-mcp cli index_repository '{"repo_path":"<non-ascii>"}'`
97+
indexes the directory (ASCII control proves the CLI path works)
98+
- Actual: the ASCII control indexes; the non-ASCII invocation fails with
99+
`repo_path is required` (the mangled, now-invalid-UTF-8 JSON argument is
100+
rejected) and exits non-zero
101+
- Command: `python tests/windows/test_cli_non_ascii_arg.py build\c\codebase-memory-mcp.exe`
102+
- Minimal failure output:
103+
104+
```
105+
ASCII control: indexed OK
106+
non-ASCII argv: rc=1
107+
stderr: ... repo_path is required
108+
```
109+
110+
- Suspected implementation area: `int main(int argc, char **argv)` in
111+
`src/main.c` does not use `wmain` / `GetCommandLineW`, so on Windows the C
112+
runtime delivers `argv` in the ANSI code page. The non-ASCII bytes in the JSON
113+
argument are corrupted before `yyjson` parses them. Fix direction: read the
114+
wide command line on Windows (`GetCommandLineW` + `CommandLineToArgvW`, or a
115+
`wmain` entrypoint) and convert each argument to UTF-8.
116+
117+
Real MCP clients pass `repo_path` inside a JSON-RPC message over stdio (which is
118+
byte-clean), so this affects the documented `cli` entrypoint and the hook/install
119+
flows that shell out to it, not the stdio server path. Holds on Linux/macOS
120+
(argv is UTF-8 bytes).
121+
122+
---
123+
124+
## Seed areas revisited and ruled out (green on native Windows)
125+
126+
Each was reproduced as a concrete attempt against the production binary and
127+
behaved correctly — recorded as green and **not** included as a red test:
128+
129+
| Area | Seed | Result on Windows |
130+
|---|---|---|
131+
| stdio `initialize` returns before stdin EOF | #513, #635 | green |
132+
| `tools/list` non-empty; all 14 tools return valid JSON-RPC | #530 | green |
133+
| Client exit terminates the server process (no residual `.exe`) | #185, #406 | green |
134+
| `--help` / `--version` exit 0 in PowerShell, cmd, Git Bash | — | green |
135+
| `search_code` works without bash/GNU grep (PowerShell `Select-String`) | #422, #348 | green |
136+
| `.gitignore` and `.cbmignore` honored | #274 | green |
137+
| `detect_changes` reports real changed files across commits | #371, #137 | green |
138+
| `query_graph` shapes (counts, paths, labels) — no crash/disconnect | #627 | green |
139+
| Paths with spaces, `&`, `()`, `[]`, `#`, `%`, `!`, apostrophe | #272 | green |
140+
| Mixed slash/backslash and lower-case drive letters | #133 | green |
141+
| Non-UTF-8 (CP949) source file emits valid UTF-8 JSON; no crash | #511 | green |
142+
| Re-index is idempotent (counts stable, single project) | #140 | green |
143+
| Index never escapes the selected root | #331 | green |
144+
| Every JSON-RPC response decodes as strict UTF-8 | invariant | green |
145+
146+
## Observed but intentionally out of scope for this PR
147+
148+
- **Project-name collision for non-ASCII paths (#571/#20).** Two distinct repos
149+
(`проект`, `日本語`) under the same parent derive the *same* project name,
150+
because `cbm_project_name_from_path` (`src/pipeline/fqn.c`) maps every
151+
non-`[A-Za-z0-9._-]` byte to `-` and then trims. This is a real bug but it is
152+
**not Windows-specific** — `cbm_project_name_from_path` is platform-independent
153+
and collides identically on Linux. Per the campaign rules it is recorded here
154+
and left for a cross-platform PR.
155+
- **Paths longer than 260 characters.** This machine has
156+
`HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled = 0`, so
157+
paths over `MAX_PATH` are unreachable by every application, not just CBM.
158+
CBM could opt in via the `\\?\` prefix + wide APIs, but the failure is gated by
159+
a machine-wide policy rather than a clean CBM-only defect, so it is excluded.
160+
- **C `test-runner` failures on Windows.** The in-process C suite reports many
161+
extraction-count failures concentrated in `test_grammar_probe_*`,
162+
`test_node_creation_probe`, `test_edge_*`, `test_matrix_*`, and
163+
`test_integration.c` (e.g. `integ_index_has_files` finds 0 files even for an
164+
**ASCII** fixture). The production binary indexes those same ASCII/CRLF cases
165+
correctly (CRLF vs LF source files were verified to extract identically), so
166+
these look like in-process test-harness issues rather than user-facing product
167+
regressions. Distinguishing genuine Windows-only product regressions from
168+
fixture/harness sensitivity requires a Linux baseline of the same commit and is
169+
left as a follow-up; they are deliberately **not** converted into red tests
170+
here to avoid shipping undiagnosed assertions.
171+
172+
## Stop-condition coverage
173+
174+
- Shells/launchers covered: PowerShell 5.1, `cmd.exe`, Git Bash, direct Win32,
175+
Python `subprocess`, Python stdio JSON-RPC (>= 3 required).
176+
- Classes covered in the green streak: smoke, integration, unit (the passing
177+
`build/c/test-runner` cases), invariant.
178+
- Seed areas (Unicode paths, mapped-drive/UNC, stdio, `search_code`,
179+
install/update, watcher/ignore, query, memory/process lifecycle) were each
180+
revisited or explicitly ruled out above.

0 commit comments

Comments
 (0)