EPIC: scoring metric + baselines/ablations

[Santisoutoo]

Outcome

The measurement half of the benchmark: a trajectory-level procedure-compliance metric, baselines across at least two models, and ablations. This is what the paper reports.

Context and motivation

The other half of the research contribution. #19 provides scenarios and ground truth; this turns a recorded trajectory into a number that means something, and then produces the evidence.

The metric is the actual research risk. Procedure compliance is not accuracy — it has to say something principled about an agent that took the right actions in the wrong order, or that fixed the aircraft while ignoring ECAM entirely, or that did the first three QRH steps correctly and then stopped. Those are different failures and a single scalar will hide them. Designing that metric well is the contribution; the infrastructure exists to make it measurable.

Sub-tasks

• Define the compliance metric (ordering, partial credit, harmful-action penalties)
• Decide what a scalar hides and what must be reported alongside it
• Scoring runner over EPIC: scenario suite + procedure ground truth #19's recorded trajectories
• Baselines: >=2 models per CLAUDE.md
• Ablations (candidates below)
• Human or expert reference point, if feasible — a score with no ceiling is hard to interpret
• Results, plots, and reproduction instructions

Open questions

• Metric design. Edit distance to the reference trajectory? Per-step precision/recall? Weighted by safety-criticality? Each encodes a different claim about what matters.
• Harmful actions. An agent that shuts down the wrong engine should score below one that does nothing. Does the metric let it?
• Ablation axes. Tool descriptions (End-to-end demo: an LLM resolves an injected failure through the MCP tools #17 notes these are effectively prompt engineering), QRH access vs not, ECAM-only vs full state, reasoning on/off.
• Statistical power. Runs per scenario given LLM nondeterminism; how many scenarios are needed to claim a difference between models.
• Contamination. A320 QRH procedures are public and may be in pretraining. That is not disqualifying, but it needs addressing rather than ignoring — knowing the procedure and executing it under observation are different tasks.

Definition of done

A defensible metric, a reproducible scoring runner, baselines on >=2 models with ablations, and results written up.

Phase

Fase 5 — Benchmark