Standardize max-output-token and Support Automatic Calibration

[boshmaf]

Background

Currently, max-output-token is configured using a benchmark-dependent rule with two predefined values. The larger value is assigned to benchmarks that are expected to produce longer outputs, particularly those involving reasoning.

This works as a default, but it has two limitations:

1. Responses that hit the max-output-token limit may be silently truncated.
2. Some models may produce substantially more verbose responses and therefore require a larger token budget.

At the same time, max-output-token must remain benchmark-dependent, not model-dependent. Allowing this parameter to vary across models would introduce an evaluation-pipeline confound and undermine fair model comparisons.

Desired Behavior

1. Detect truncated responses

Detect when a model response is incomplete because generation reached max-output-token.

When this happens, the harness should raise an explicit error rather than silently accepting the truncated response.

The caller/developer is responsible for deciding how to handle the error, e.g., by:

• increasing max-output-token;
• changing the benchmark configuration; or
• catching and handling the error externally.

The harness should not automatically continue generation, since doing so would implicitly introduce model-dependent evaluation behavior.

2. Support automatic max-output-token calibration

Add an automatic calibration mode that can be enabled either:

• through a command-line argument; or
• programmatically by setting max-output-token = "auto".

In "auto" mode, the harness should run a predefined calibration set containing known prompts selected to produce increasingly long outputs for each benchmark.

The observed outputs should then be used to determine an appropriate max-output-token value for each benchmark.

The result of calibration should be a simple benchmark-to-token-limit mapping, e.g.:

MAX_OUTPUT_TOKENS = {
    "benchmark_a": 2048,
    "benchmark_b": 4096,
    "benchmark_c": 8192,
}

The calibration result should be easy for the developer to inspect, save, and reuse in subsequent evaluation runs.

3. Warn when calibration exceeds the default

If automatic calibration determines that one or more benchmarks require a larger max-output-token than the current default, log a clear message explaining that the newly calibrated values should be used for all models in the comparison.

For example:

Calibration determined that some benchmarks require a larger max-output-token than the default configuration. If this model is part of a multi-model evaluation, use the resulting benchmark-level mapping for all models to ensure identical generation settings and a fair comparison.

The harness should not automatically maintain different calibrated mappings for different models.

Configuration Invariant

max-output-token must be defined exclusively as a mapping:

benchmark -> max-output-token

It must never be:

benchmark + model -> max-output-token

A developer may use "auto" while testing a new model to discover that the existing benchmark-level limits are insufficient. If so, the developer can explicitly adopt the resulting larger limits as the new benchmark configuration and rerun the relevant evaluations.

Once adopted, however, the same mapping must be applied to all models being compared.

Rationale

The broader goal is to ensure that evaluation parameters are explicitly defined and standardized across all stages of the harness.

No pipeline parameter should implicitly depend on the model being evaluated. Otherwise, differences in evaluation settings may affect computed scores and rankings independently of actual model capability.

In particular, max-output-token can directly affect whether a model completes its answer. Making this value model-dependent could therefore give more verbose models a different evaluation pipeline from less verbose models.

The intended workflow is therefore:

1. Start with the standard benchmark-level max-output-token mapping.
2. Detect and fail explicitly when generation is truncated by the token limit.
3. Optionally use "auto" calibration to identify insufficient benchmark-level limits.
4. Review the calibrated mapping.
5. If larger limits are needed, explicitly adopt the new mapping.
6. Use that same mapping for every model included in the comparison.

Acceptance Criteria

• Detect when generation terminates because max-output-token was reached.
• Raise an explicit error for token-limit truncation.
• Do not automatically continue truncated generations.
• Support max-output-token = "auto" programmatically.
• Expose equivalent automatic calibration through the CLI.
• Maintain a predefined benchmark-specific calibration prompt/sample set.
• Produce a simple benchmark -> max-output-token mapping after calibration.
• Clearly log benchmarks whose calibrated limits exceed the defaults.
• Warn developers that an updated mapping must be applied consistently to all models in comparative evaluations.
• Ensure max-output-token configuration is never implicitly model-dependent.
• Allow developers to explicitly provide and reuse a calibrated benchmark-level mapping.