mmlu_flan_cot_zeroshot group-level aggregate never populates (metric name mismatch: acc vs exact_match)

[zeeshanhaque21]

Description

The mmlu_flan_cot_zeroshot group task's top-level result never populates a metric value, even though every subtask and subgroup underneath it scores correctly.

Repro

python -m lm_eval run --model local-chat-completions \
  --model_args model=default_model,base_url=http://127.0.0.1:8199/v1/chat/completions,num_concurrent=1,max_retries=1,tokenized_requests=False,max_gen_toks=256 \
  --tasks mmlu_flan_cot_zeroshot \
  --limit 2 --apply_chat_template --confirm_run_unsafe_code \
  --output_path out.json

(any generate_until-capable backend reproduces this; not specific to local-chat-completions)

Observed

In results, every leaf subtask (e.g. mmlu_flan_cot_zeroshot_abstract_algebra) reports both exact_match,strict-match and exact_match,flexible-extract. Every mid-level subgroup (mmlu_flan_cot_zeroshot::stem, ::other, ::social sciences, ::humanities) reports only alias/name/sample_len — no metric key at all. The top-level mmlu_flan_cot_zeroshot entry is the same: just alias/name/sample_len, no aggregate score.

"mmlu_flan_cot_zeroshot": {
    "alias": "mmlu (flan style, zeroshot cot)",
    "name": "mmlu_flan_cot_zeroshot",
    "sample_len": 114
}

Expected

The group aggregate should roll up to some exact_match,... value (matching what the subtasks actually emit), the same way mmlu (the multiple_choice variant) correctly aggregates acc,none to a top-level number.

Root cause guess

lm_eval/tasks/mmlu/flan_cot_zeroshot/_mmlu.yaml and _mmlu_flan_cot_zeroshot_template_yaml declare:

aggregate_metric_list:
  - metric: acc
    weight_by_size: True

at every group level, but the leaf task's metric_list only defines exact_match (via strict-match/flexible-extract filters), never acc. The aggregator appears to look for a metric named acc to roll up and silently finds nothing, so the group-level result stays empty instead of erroring or falling back to exact_match.

Environment

• lm_eval 0.4.12 (pip)
• Reproduced against a local MLX-backed OpenAI-compatible chat endpoint, but the task config itself is backend-independent.

Impact

Anyone relying on mmlu_flan_cot_zeroshot's single-number group score (e.g. for a leaderboard) gets nothing back, despite every underlying sample being scored and correct. Individual 57-subject scores are fine; only the roll-up is broken.