Unscorable samples silently leave the metric denominator, inflating headline scores

[AUTHENSOR]

Summary

lm_eval/evaluator.py evaluate() collects metrics like this:

metrics = task.process_results(doc, [req.filtered_resps[filter_key] for req in requests])
...
for metric, value in metrics.items():
    acc["raw_metrics"][(metric, filter_key)].append(value)

If a task's process_results returns an empty dict, or omits a metric key, for a given document (a natural way for a task author to say "could not score this one": refusal, unparseable answer, empty extraction), that document contributes to no metric and the aggregate is a mean over the surviving documents only. Nothing warns, nothing counts the dropped documents per metric, and n-samples.effective in the results is computed in _compute_task_aggregations as the length of the LAST metric's list (the code carries a TODO admitting this), so with heterogeneous keys even the reported effective count is the wrong number for the other metrics.

The score inflation is structural: whatever a model does to make itself unscorable is removed from the average rather than counted against it.

Steps to reproduce

PoC (end-to-end through lm_eval.evaluator.evaluate, dummy LM, identical outputs in both runs): poc_unscorable_denominator.py

Six documents; the dummy model refuses on 2 and answers 4. The same model outputs are scored twice, differing only in the task's process_results convention. Observed (PROVED, run twice, byte-identical output):

model behavior: 2/6 docs refused (unscorable), 4/6 answered
[A: return empty dict for unscorable] acc=1.0  n-samples={'original': 6, 'effective': 4}
[B: return 0 for unscorable] acc=0.6666666666666666  n-samples={'original': 6, 'effective': 6}
RESULT: identical outputs, headline acc differs: True (1.0 vs 0.6666666666666666)
VERDICT: VULNERABLE

Impact

Headline accuracy can be inflated by exactly the documents a model fails to answer in a scorable way. Two tasks measuring identical model behavior can differ by the scoring convention alone, and consumers of the results file have no per-metric dropped-count to detect it. Group aggregation compounds this: Group.aggregate weights subtasks by the reported sample_len, which inherits the same last-metric count. Any task family that maps refusals or parse failures to an empty result (a pattern the process_results contract makes natural) is affected.

Suggested fix

Track denominators explicitly per (metric, filter): count evaluated documents per metric, count documents for which the metric was omitted, and report both (for example n-samples: {original, effective, unscorable_per_metric}) plus a warning when effective < original for any metric. Fix sample_len to be per metric rather than the last metric's length. Longer term, make the could-not-score case an explicit typed value in process_results rather than key omission.

References

• PoC script: poc_unscorable_denominator.py (double-run REPRO: REPRO-E1-F4.sh, byte-identical)
• Pinned code: EleutherAI/lm-evaluation-harness @ 0f8479c, lm_eval/evaluator.py lines 636 to 669, lm_eval/evaluator_utils.py _compute_task_aggregations lines 173 to 219 (sample_len TODO at line 204), lm_eval/api/group.py aggregate() lines 242 to 292

[linhongyu510]

I would like to investigate this, but the scoring contract needs maintainer direction before implementation. A conservative first patch could preserve current aggregation values while adding an offline regression for heterogeneous metric keys, tracking effective counts per (metric, filter), and emitting a warning when a document omits an expected metric. That would make denominator shrinkage visible without silently treating a missing metric as zero or changing existing task scores. Should the first patch stay within that observability boundary, or do maintainers want omission to become an explicit scoring value/error instead? I will hold off on code until the intended contract is confirmed.

[arthi-arumugam-git]

I opened #4068 for one narrow piece of this, the n-samples.effective part, and deliberately left the scoring contract alone.

sample_len = len(items) sits inside the loop over metrics, so the reported count is whichever metric came last. With heterogeneous keys that makes it depend on dict ordering: the same raw metrics report 1 or 3 depending only on which key is first. #4068 takes the largest per-metric count instead, which is the number of documents that produced any metric at all, and is unchanged when every metric holds the same number of values.

That is only the reported count. It does not answer the larger question here, which is whether a document that drops out of a metric should be surfaced at all, and where. @linhongyu510 I think you are right that the aggregation semantics need maintainer direction before anyone changes them, and I have not touched them.