Repository navigation
Commit 42a2e2d
committed
Port upstream DeusData#2198: benchmark runtime isolation (upstream DeusData#1696)
Finish the merge of the benchmark harness isolation work onto the CLI-first
fork. The auto-merged upstream hunks left both scripts as a mix of fork and
upstream invocations (undefined INDEX_RC, an unused INDEX_ERR, MCP
`cli index_repository` calls).
benchmark-index.sh: run `codebase-memory-cli index REPO --mode full --json`
with stderr captured to INDEX_ERR and the exit status kept in INDEX_RC, so a
refused index fails the run, prints "--- index stderr ---" and the cause.
Dropped the now-unneeded hand-built JSON path (argv carries the path).
benchmark-search-graph.sh: index through `index REPO --mode full --json`
instead of the MCP `cli index_repository` request; stop unwrapping the MCP
content envelope; fix usage comment (arg is a repo path).
docs/EVALUATION_PLAN.md: example invocations name codebase-memory-cli.
Contract test adaptations (assertions on isolation, stderr surfacing, exit
status, handoff and stop behaviour unchanged):
- Probe fixture detects `index` (argv 1) instead of `cli index_repository`.
- The fixture logs the repo path (argv 2) rather than a JSON body; the
quoted/backslash-path check now asserts the path reaches the index command
as one intact argument, since the fork's CLI has no JSON request.
- "MCP session" comment wording changed to "graph session".1 parent 3a4fdb9 commit 42a2e2d
4 files changed
Lines changed: 24 additions & 30 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
283 | 283 | | |
284 | 284 | | |
285 | 285 | | |
286 | | - | |
| 286 | + | |
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
| |||
878 | 878 | | |
879 | 879 | | |
880 | 880 | | |
881 | | - | |
| 881 | + | |
882 | 882 | | |
883 | 883 | | |
884 | 884 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | 69 | | |
74 | 70 | | |
75 | 71 | | |
| |||
115 | 111 | | |
116 | 112 | | |
117 | 113 | | |
118 | | - | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
119 | 120 | | |
120 | 121 | | |
121 | 122 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | 33 | | |
38 | 34 | | |
39 | | - | |
40 | | - | |
41 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
42 | 38 | | |
43 | 39 | | |
44 | 40 | | |
45 | | - | |
46 | | - | |
47 | 41 | | |
48 | 42 | | |
49 | 43 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
40 | | - | |
| 39 | + | |
| 40 | + | |
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| |||
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | | - | |
| 120 | + | |
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
| |||
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
264 | 265 | | |
265 | 266 | | |
266 | 267 | | |
| |||
275 | 276 | | |
276 | 277 | | |
277 | 278 | | |
278 | | - | |
279 | | - | |
| 279 | + | |
280 | 280 | | |
281 | 281 | | |
282 | 282 | | |
| |||
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
289 | | - | |
290 | | - | |
291 | | - | |
| 289 | + | |
| 290 | + | |
292 | 291 | | |
293 | 292 | | |
294 | 293 | | |
| |||
0 commit comments