@@ -0,0 +1,6 @@
+.work/
+results/raw/
+results/runs/
+__pycache__/
+.DS_Store
+fair-e2e-context-bundle.tar.gz
@@ -0,0 +1,112 @@
+# Benchmark Protocol
+
+## Isolation
+
+Each tool receives a separate detached worktree at the same repository SHA.
+This prevents `.repobrain/` from entering CodeGraph's corpus and `.codegraph/`
+from entering RepoBrain's corpus. Third-party repositories are shallow clones.
+
+## Tracks
+
+### Unrestricted native track (latest-v2 default)
+
+- Both products receive independent worktrees at the same pinned source SHA.
+- RepoBrain uses its public full refresh and `rb-ask` workflow.
+- CodeGraph uses its installed CLI/integration as the primary discovery tool.
+- Both answer agents may inspect source files for verification, with no command
+  count or context-budget limit beyond the native tools' own limits.
+- A product must actually be invoked; a plain model-only answer is invalid.
+- Compare final evidence correctness, total end-to-end wall time, and Trae
+  token usage. Report monetary cost only when the runtime emits an amount.
+- Use a shared requested model and record any observed server-side reroute.
+
+This track models normal IDE use. It does not isolate pure retrieval quality.
+
+### Deterministic index track
+
+- RepoBrain: `RB_REFRESH_SCAN_ONLY=1 rb-refresh`
+- CodeGraph: `codegraph init`
+- No model is involved.
+- Run tools sequentially on an otherwise idle machine.
+
+### Source-free native retrieval track
+
+- RepoBrain: full host-runner refresh followed by `rb-ask --json`
+- CodeGraph: `codegraph explore`
+- A subagent may invoke only the assigned product command. It may not read,
+  grep, search, or inspect the target repository directly.
+- Results are scored against the answer atoms in `config/questions.json`.
+- RepoBrain results are eligible only when the public refresh command promotes
+  an active generation. A scan-only index, deleted staging generation, direct
+  source answer, or manually promoted generation is not a valid substitute.
+
+This track measures each product's native workflow, not pure retrieval parity.
+RepoBrain produces a final answer; CodeGraph returns source context and graph
+paths. Report these differences explicitly.
+
+### Ceiling retrieval track
+
+- Use the same focused corpus and answer atoms for both products.
+- RepoBrain may waive only the deterministic generic host-runner
+  `git_insights=partial` state.
+- Reject promotion if any other stage or module is degraded, or if the failure
+  list is non-empty.
+- Copy the promoted RepoBrain knowledge into a source-free workspace before
+  `rb-ask`, preventing direct source reads.
+- Label these results as ceiling behavior rather than unmodified current-main
+  behavior.
+
+### Controlled answer track
+
+For a stricter future run, feed both retrieval outputs into the same frozen
+model with source-reading tools disabled and an equal context-token budget.
+This pilot does not claim that controlled-model result.
+
+## Metrics
+
+### Build
+
+- Wall, user, and system time from `/usr/bin/time -lp`
+- Maximum resident set size
+- Final index directory size
+- Completion state and parser errors
+- Indexed files divided by Git tracked files
+
+### Retrieval
+
+- Expected-file recall
+- Expected-symbol recall
+- Citation validity: cited file and line exist at the pinned SHA
+- Unsupported-claim rate
+- Wall time
+- Output bytes
+- Input, cached-input, output, and reasoning tokens when emitted by the runtime
+- Monetary cost only when explicitly emitted; otherwise mark it unavailable
+
+### Incremental
+
+- Explicit update time: `rb-refresh --quick` versus `codegraph sync`
+- Time until changed facts are queryable
+- Stale-fact rate
+- Incremental result versus clean rebuild
+- Update amplification in each tool's native unit
+
+## Repetition
+
+The checked-in result is a pilot with one cold-build run and one retrieval run
+per question. A publishable benchmark should use:
+
+- three cold builds per tool and repository
+- ten retrieval latency runs per deterministic query
+- three answer runs per question
+- paired bootstrap confidence intervals for answer metrics
+
+## Failure Policy
+
+Timeouts, crashes, incomplete indexes, and missing answers remain in the
+dataset. Environment-specific blocks are reported separately and are not
+attributed to the product.
+
+If a tool cannot create the prerequisite queryable index through its public
+workflow, record the retrieval result as unavailable with the build failure.
+Do not assign zero answer-atom recall because no answer was produced.
@@ -0,0 +1,120 @@
+# RepoBrain vs CodeGraph Benchmark
+
+## Latest v2 multi-repository suite
+
+The v2 suite preserves the original VERL/Excalidraw pilot below and adds a
+manifest-driven comparison over pinned releases of Flask, Vite, ripgrep,
+Prometheus, and Grafana. Grafana is a stress track and must be reported
+separately from the four-repository main track.
+
+The suite pins RepoBrain and CodeGraph as well as every corpus revision. It
+builds each product from an independent worktree. The default
+`unrestricted_native` track models normal IDE use: both agents can read the
+same pinned source tree and use their product's native integration without a
+command budget. The optional `source_free` track isolates retrieval indexes.
+Every run is stored below
+`results/runs/<run-id>/<track>/<repository>/repeat-<n>/<product>/`.
+
+Start with the Flask smoke test:
+
+```bash
+bash benchmarks/codegraph-comparison/scripts/bootstrap.sh --repository flask
+
+benchmarks/codegraph-comparison/.work/v2/tools/repobrain-venv/bin/python \
+  benchmarks/codegraph-comparison/scripts/prepare_corpora.py \
+  --repository flask
+
+benchmarks/codegraph-comparison/.work/v2/tools/repobrain-venv/bin/python \
+  benchmarks/codegraph-comparison/scripts/run_benchmark_v2.py \
+  --run-id flask-unrestricted-smoke \
+  --track unrestricted_native \
+  --repository flask \
+  --limit 1 \
+  --workers 1 \
+  --restart
+
+benchmarks/codegraph-comparison/.work/v2/tools/repobrain-venv/bin/python \
+  benchmarks/codegraph-comparison/scripts/render_benchmark_v2_report.py \
+  benchmarks/codegraph-comparison/results/runs/flask-unrestricted-smoke/unrestricted_native/results.json \
+  benchmarks/codegraph-comparison/results/runs/flask-unrestricted-smoke/unrestricted_native/report.md
+```
+
+For a formal main-track run, prepare the four non-stress repositories and use
+`--repeat 3 --workers 1`. Run Grafana under a different run id. Compare final
+evidence correctness, total wall time, and Trae token usage. Monetary cost is
+reported only when Trae emits it; otherwise it remains explicitly unavailable.
+For `source_free`, a generated corpus manifest records that source files are
+absent from the query directory; this is auditable isolation, not an
+OS/container security boundary.
+
+The v2 entrypoints are:
+
+- `config/manifest-latest-v2.json`: tool and corpus pins
+- `config/questions-v2.json`: 23 repository-specific questions
+- `scripts/bootstrap.sh`: pinned tool/worktree setup
+- `scripts/prepare_corpora.py`: public index builds and source-free publication
+- `scripts/run_benchmark_v2.py`: repeated fair-E2E runner
+- `scripts/render_benchmark_v2_report.py`: multi-repository report
+
+## Legacy pilot
+
+The legacy files contain the original historical comparison of:
+
+- RepoBrain `main` at `b0ebc36b611305dcbdda7343fc8e0136fa7f2686`
+- CodeGraph `1.5.0`
+
+The primary corpora are:
+
+- `verl-project/verl` at `8f8b122195ecf43475679fb6ee0204ee33f387c5`
+- `excalidraw/excalidraw` at `4a6f4e7a32ce3d3360d4d209d14105acc144d8c7`
+
+OpenClaw was also attempted at
+`1e6844b71aa2299c082d23cf66dce38ff00b2211`. Its CodeGraph run is excluded
+because a corporate endpoint policy terminated Node processes operating on an
+OpenClaw checkout. The RepoBrain result is retained only as a scalability
+observation, not as a paired comparison.
+
+## Layout
+
+- `config/manifest.json`: pinned tool, repository, and machine versions
+- `config/questions.json`: shared pilot question set and answer atoms
+- `PROTOCOL.md`: comparison rules and metric definitions
+- `scripts/`: repeatable index and query commands
+- `results/`: normalized summaries and reports
+- `.work/`: ignored third-party checkouts, virtual environments, and indexes
+
+## Run
+
+From the repository root:
+
+```bash
+bash benchmarks/codegraph-comparison/scripts/bootstrap.sh
+bash benchmarks/codegraph-comparison/scripts/run_index_benchmark.sh
+bash benchmarks/codegraph-comparison/scripts/run_retrieval_benchmark.sh
+```
+
+The scripts do not install anything globally. CodeGraph telemetry and update
+checks are disabled for benchmark runs.
+
+The historical results separate behavior of their pinned RepoBrain commit from
+a ceiling track. That old commit marked deterministic host-runner Git insights
+`partial` and removed the unpromoted generation. Current RepoBrain fixed the
+public full-refresh path; do not mix the old report with latest-v2 results.
+
+The detailed 15-question accuracy table is in
+`results/expanded-question-report.md`.
+
+## Interpretation
+
+RepoBrain scan-only and CodeGraph index different semantic units. Their raw
+node and edge counts are therefore descriptive, not directly comparable.
+Compare:
+
+- cold build wall time and peak RSS
+- index disk size
+- indexed-file coverage relative to tracked source files
+- retrieval answer-atom recall and citation validity
+- output size and query latency
+- explicit incremental update time and freshness
+
+Do not collapse these dimensions into one score.
@@ -0,0 +1,55 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://repobrain.dev/schemas/codegraph-comparison-corpus-manifest-v1.json",
+  "title": "Source-free benchmark corpus provenance",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schema_version", "provenance_schema", "created_at", "benchmark_id", "repository", "product", "source_free"],
+  "properties": {
+    "schema_version": {"const": 1},
+    "provenance_schema": {"const": "config/corpus-manifest-v1.schema.json"},
+    "created_at": {"type": "string", "format": "date-time"},
+    "benchmark_id": {"type": "string", "minLength": 1},
+    "repository": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["id", "url", "revision", "release", "primary_language", "corpus", "tracked_tree_sha256"],
+      "properties": {
+        "id": {"type": "string", "minLength": 1},
+        "url": {"type": "string", "format": "uri"},
+        "revision": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
+        "release": {"type": "string", "minLength": 1},
+        "primary_language": {"type": "string", "minLength": 1},
+        "corpus": {"type": "object"},
+        "tracked_tree_sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"}
+      }
+    },
+    "product": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["name", "pin", "build_status", "build_seconds"],
+      "properties": {
+        "name": {"enum": ["repobrain", "codegraph"]},
+        "pin": {"type": "string", "minLength": 1},
+        "build_status": {"const": "success"},
+        "build_seconds": {"type": "number", "minimum": 0}
+      }
+    },
+    "active_generation": {"type": "object"},
+    "source_free": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["verified", "hard_filesystem_isolation", "allowed_top_level_entries", "note"],
+      "properties": {
+        "verified": {"const": true},
+        "hard_filesystem_isolation": {"type": "boolean"},
+        "allowed_top_level_entries": {
+          "type": "array",
+          "uniqueItems": true,
+          "items": {"type": "string", "minLength": 1}
+        },
+        "note": {"type": "string", "minLength": 1}
+      }
+    }
+  }
+}