@@ -1127,10 +1127,20 @@ The evaluation's most serious flaw is not operational — it is that the bespoke
11271127
11281128# Appendix A — Per-language chapters (159)
11291129
1130- > Each chapter is a *draft specification* (per §12): repo + 5 dimension-tagged questions
1131- > (symmetric-authoring split), the per-type graph-stats block (all 32 edge types, zeros
1132- > kept), and — for the 9 LSP languages — the cross-repo + semantic/similarity deep-dive.
1133- > Final question symbols are confirmed grep-first against the pinned commit at execution.
1130+ > **⚠️ STATUS — these are DRAFT specifications, not the final question bank.** The chapters below
1131+ > were **LLM-drafted from model knowledge of each repo** to establish the format, dimension mapping,
1132+ > and deep-dive shape. Symbols that could not be confirmed carry a `[verify]` tag. They are **not yet
1133+ > ground-truth-derived** and **must be regenerated at execution time** per §3.1 / §12:
1134+ >
1135+ > - **Question *types* are already grounded** in the Sillito et al. taxonomy (§3.1) — that part is final.
1136+ > - **Question *targets* (the specific symbols/files) are NOT yet from ground truth.** At execution they
1137+ > are replaced by: **SWE-QA** items for the major languages, and for the rest, targets **seeded from
1138+ > independent LSP symbol/reference data + git history — never from the model**, then confirmed
1139+ > grep-first against the pinned commit (symmetric-authoring split, [CR-1]).
1140+ >
1141+ > In short: each chapter shows *what kind* of question each dimension asks and *how* it is evaluated
1142+ > (output, stats block with all 32 edge types, aggregation, LSP deep-dive). It does **not** yet contain
1143+ > the final, ground-truth-sourced question set — do not read the drafted symbol names as validated.
11341144
11351145---
11361146
0 commit comments