Response cache (`--use_cache`) is not keyed by model, so one cache silently serves model A's outputs while reporting them as model B's results

[AUTHENSOR]

Summary

CachingLM (lm_eval/api/model.py) stores and looks up cached responses under hash_args(attr, req.args), which is a SHA-256 of the request type plus the request arguments only:

def hash_args(attr: str, args: Iterable[Any]) -> str:
    dat = json.dumps([attr] + list(args))
    return hashlib.sha256(dat.encode("utf-8")).hexdigest()

Nothing about the model is part of the key: not the model name, not model_args, not the revision, not the backend. If a response cache db written while evaluating model A is later reused while evaluating model B (same tasks, same prompts), every request is a cache hit and model B is never run. The final results dict attributes the scores to model B. No warning is printed and no check exists anywhere in the read path.

This is easy to trigger in normal use. The --use_cache help text says "Path to cache model responses (skips repeated inference)" and the docs describe it as a generic response cache path, with no instruction to use one db per model. Reusing one cache file across a sweep of models over the same benchmark is the workflow the flag appears to offer.

Steps to reproduce

PoC (deterministic, dummy models, no inference): poc_cache_model_identity.py

1. Two dummy LM subclasses are defined. Model A returns loglikelihood (-8.25, False) and generation " i do not know". Model B returns (-0.10, True) and " Paris".
2. Model A is wrapped in CachingLM with cache db responses_rank0.db and answers one loglikelihood request and one generate_until request. The db now holds A's outputs.
3. Model B is wrapped with the same db and the same requests are submitted.
4. Observed (PROVED, run twice, byte-identical output):

[run 2] model B loglikelihood=(-8.25, False)  generation=' i do not know'
[run 2] model B was actually called: {'loglikelihood': 0, 'generate_until': 0}
RESULT: model B received model A's cached responses: True
RESULT: model B forward passes executed: 0
VERDICT: VULNERABLE

Model B answered with model A's numbers after zero forward passes, and the run's config block would name model B.

Impact

Silent cross-contamination of published scores. Any workflow that shares a response cache db between models (model sweeps, leaderboard re-runs, CI caching, one operator's cache path reused by another) reports model A's measurements as model B's, for both loglikelihood and generate_until request types. Because loglikelihood scores feed acc/acc_norm on multiple choice benchmarks, the contamination directly changes headline numbers. There is no error, no log line, and no way to detect the contamination from the results file after the fact.

Suggested fix

Include a model identity component in the cache key, for example hash_args(model_identifier + attr, args) where model_identifier is the resolved model name plus the full model_args string (and ideally backend name and harness version). On the read path, store the model identifier alongside each entry and emit a loud warning or refuse the hit when the requesting model differs from the writing model. At minimum, document that a cache db must not be shared across models and warn when a db is opened by a different pretrained than the one recorded.

References

• PoC script: poc_cache_model_identity.py (double-run REPRO: REPRO-E1-F1.sh, byte-identical)
• Pinned code: EleutherAI/lm-evaluation-harness @ 0f8479c, lm_eval/api/model.py lines 229 to 325 (hash_args, CachingLM)