Repository navigation

Issues

New issue

Streamlining lm-eval Architecture
#3083 · baberabb opened on Jun 23, 2025
1
Datasets with loading scripts - no longer supported
#3171 · baberabb opened on Jul 21, 2025
27

vLLM overwrites enable_thinking supplied through chat_template_args
EleutherAI/lm-evaluation-harness#4397 · opened on Oct 16, 2026
SGLang omits tokenizer cache identity and chat template metadata
EleutherAI/lm-evaluation-harness#4396 · opened on Oct 16, 2026
Request cache defaults inside site-packages; dill entries load with no integrity check
EleutherAI/lm-evaluation-harness#4393 · opened on Oct 15, 2026
Task selection executes module code at build time (groups fan out; include: is unconfined)
EleutherAI/lm-evaluation-harness#4392 · opened on Oct 15, 2026
trust_remote_code=False cannot veto hardcoded task-level remote code
EleutherAI/lm-evaluation-harness#4391 · opened on Oct 15, 2026
HFLM ignores bare xpu, hpu, and npu device names
EleutherAI/lm-evaluation-harness#4386 · opened on Oct 14, 2026
2
Neuron stop criteria match the prompt and truncate generation after one token
EleutherAI/lm-evaluation-harness#4385 · opened on Oct 14, 2026
Feature request - LongMemEval Oracle - Evidence Evaluation
EleutherAI/lm-evaluation-harness#4380 · opened on Oct 13, 2026
gsm8k "flexible-extract" can extract "$.", "$$" or "..." instead of the final number
EleutherAI/lm-evaluation-harness#4375 · opened on Oct 13, 2026
arc_challenge_chat and jsonschema_bench send literal "\n" to the model (single-quoted doc_to_text)
EleutherAI/lm-evaluation-harness#4370 · opened on Oct 12, 2026
CSV additive steering fails because SAE decoder bias is always included
EleutherAI/lm-evaluation-harness#4364 · opened on Oct 12, 2026
Explicit HF sampling without temperature injects an invalid zero temperature
EleutherAI/lm-evaluation-harness#4363 · opened on Oct 12, 2026
afrobench: 438 multiple-choice configs end doc_to_text in whitespace, so every choice gets a doubled separator
EleutherAI/lm-evaluation-harness#4361 · opened on Oct 11, 2026
2
Group member overrides collapse referenced group and tag leaves
EleutherAI/lm-evaluation-harness#4354 · opened on Oct 10, 2026
1
Direct Task predict-only mode records a metric function instead of a value
EleutherAI/lm-evaluation-harness#4353 · opened on Oct 10, 2026
1
mgsm_native_cot: strict-match regex misses the bn, ja and de exemplar answer format
EleutherAI/lm-evaluation-harness#4350 · opened on Oct 10, 2026
Direct Task objects receive a different task_name on every access
EleutherAI/lm-evaluation-harness#4335 · opened on Oct 8, 2026
1
Task.doc_iterator(samples=[]) unexpectedly evaluates every document
EleutherAI/lm-evaluation-harness#4334 · opened on Oct 8, 2026
Size-weighted group metrics use another metric's sample count
EleutherAI/lm-evaluation-harness#4322 · opened on Oct 6, 2026
1
Response cache misses for equivalent dictionaries with different key order
EleutherAI/lm-evaluation-harness#4321 · opened on Oct 6, 2026
1
[Bug] nanmean in metrics.py silently shrinks denominator without reporting excluded count
EleutherAI/lm-evaluation-harness#4319 · opened on Oct 5, 2026
3
GGUF generation returns None for malformed completion responses
EleutherAI/lm-evaluation-harness#4305 · opened on Oct 3, 2026
1
GGUF generation strips leading and trailing whitespace from model outputs
EleutherAI/lm-evaluation-harness#4304 · opened on Oct 3, 2026
1
bootstrap_stderr runs fewer than bootstrap_iters resamples when it isn't a multiple of 1000
EleutherAI/lm-evaluation-harness#4288 · opened on Oct 1, 2026
1
minerva_math / putnam_axiom: is_equiv scores every sympy-equivalent answer as wrong on Windows (signal.SIGALRM is missing)
EleutherAI/lm-evaluation-harness#4286 · opened on Oct 1, 2026
1

Navigation Menu

Repository navigation

Issues

Issues

vLLM overwrites enable_thinking supplied through chat_template_args

SGLang omits tokenizer cache identity and chat template metadata

Request cache defaults inside site-packages; dill entries load with no integrity check

Task selection executes module code at build time (groups fan out; include: is unconfined)

trust_remote_code=False cannot veto hardcoded task-level remote code

HFLM ignores bare xpu, hpu, and npu device names

Neuron stop criteria match the prompt and truncate generation after one token

Feature request - LongMemEval Oracle - Evidence Evaluation

gsm8k "flexible-extract" can extract "$.", "$$" or "..." instead of the final number

arc_challenge_chat and jsonschema_bench send literal "\n" to the model (single-quoted doc_to_text)

CSV additive steering fails because SAE decoder bias is always included

Explicit HF sampling without temperature injects an invalid zero temperature

afrobench: 438 multiple-choice configs end doc_to_text in whitespace, so every choice gets a doubled separator

Group member overrides collapse referenced group and tag leaves

Direct Task predict-only mode records a metric function instead of a value

mgsm_native_cot: strict-match regex misses the bn, ja and de exemplar answer format

Direct Task objects receive a different task_name on every access

Task.doc_iterator(samples=[]) unexpectedly evaluates every document

Size-weighted group metrics use another metric's sample count

Response cache misses for equivalent dictionaries with different key order

[Bug] nanmean in metrics.py silently shrinks denominator without reporting excluded count

GGUF generation returns None for malformed completion responses

GGUF generation strips leading and trailing whitespace from model outputs

bootstrap_stderr runs fewer than bootstrap_iters resamples when it isn't a multiple of 1000

minerva_math / putnam_axiom: `is_equiv` scores every sympy-equivalent answer as wrong on Windows (`signal.SIGALRM` is missing)