Skip to content

Commit fe31893

Browse files
authored
Merge pull request DeusData#981 from DeusData/docs/eval-plan-draft-note-v2
docs(eval-plan): mark Appendix A chapters as drafts pending ground-truth regen
2 parents 1a8ff1a + 5758e9b commit fe31893

1 file changed

Lines changed: 14 additions & 4 deletions

File tree

‎docs/EVALUATION_PLAN.md‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1127,10 +1127,20 @@ The evaluation's most serious flaw is not operational — it is that the bespoke
11271127

11281128
# Appendix A — Per-language chapters (159)
11291129

1130-
> Each chapter is a *draft specification* (per §12): repo + 5 dimension-tagged questions
1131-
> (symmetric-authoring split), the per-type graph-stats block (all 32 edge types, zeros
1132-
> kept), and — for the 9 LSP languages — the cross-repo + semantic/similarity deep-dive.
1133-
> Final question symbols are confirmed grep-first against the pinned commit at execution.
1130+
> **⚠️ STATUS — these are DRAFT specifications, not the final question bank.** The chapters below
1131+
> were **LLM-drafted from model knowledge of each repo** to establish the format, dimension mapping,
1132+
> and deep-dive shape. Symbols that could not be confirmed carry a `[verify]` tag. They are **not yet
1133+
> ground-truth-derived** and **must be regenerated at execution time** per §3.1 / §12:
1134+
>
1135+
> - **Question *types* are already grounded** in the Sillito et al. taxonomy (§3.1) — that part is final.
1136+
> - **Question *targets* (the specific symbols/files) are NOT yet from ground truth.** At execution they
1137+
> are replaced by: **SWE-QA** items for the major languages, and for the rest, targets **seeded from
1138+
> independent LSP symbol/reference data + git history — never from the model**, then confirmed
1139+
> grep-first against the pinned commit (symmetric-authoring split, [CR-1]).
1140+
>
1141+
> In short: each chapter shows *what kind* of question each dimension asks and *how* it is evaluated
1142+
> (output, stats block with all 32 edge types, aggregation, LSP deep-dive). It does **not** yet contain
1143+
> the final, ground-truth-sourced question set — do not read the drafted symbol names as validated.
11341144

11351145
---
11361146

0 commit comments

Comments
 (0)