Proposal summary
ScoreResult already has an optional metadata field, but successful evaluation scores currently lose that metadata when they are persisted as feedback scores.
EvaluationEngine computes ScoreResult objects and then log_test_result_feedback_scores() converts them to BatchFeedbackScoreDict. The persisted shape carries the score name/value/reason/category, but not ScoreResult.metadata; the current FeedbackScoreMessage / batch feedback-score types also have no metadata field.
A concrete use case is evaluator provenance:
return ScoreResult(
name="correctness",
value=0.83,
metadata={
"evaluator_revision": "rubric-v3",
"config_fingerprint": "...",
},
)The local evaluation result can retain this information, but the durable feedback score becomes only correctness = 0.83.
Could evaluation-created feedback scores preserve safe scorer-supplied metadata (or provide an equivalent first-class evaluator revision/fingerprint field)? I don't think the API needs to prescribe the metadata vocabulary; preserving an opaque user-defined revision/fingerprint would already solve the provenance problem.
There is some useful prior art in #5178, which attempted to propagate ScoreResult.metadata through feedback-score payloads while working on evaluator-failure visibility. That PR was later closed as inactive/merge-conflicted rather than merged. The current request is narrower: durable provenance for successful evaluation scores.
Motivation
A score name identifies the quality dimension, not necessarily the exact scoring definition. Two results can both be named correctness and even use the same judge model while being produced by different rubric prompts, scoring code, thresholds, or configuration.
Without evaluator provenance on the persisted score, historical experiment comparisons can become ambiguous: a score change may come from the evaluated system, or from a changed evaluator definition.
The desired invariant is roughly:
persisted score = evaluated target + score name + score value + evaluator revision/provenance
This does not require storing raw rubrics, prompts, credentials, or other sensitive configuration. A caller-controlled stable revision or fingerprint is enough, and would also make re-scoring historical outputs auditable.
Proposal summary
ScoreResultalready has an optionalmetadatafield, but successful evaluation scores currently lose that metadata when they are persisted as feedback scores.EvaluationEnginecomputesScoreResultobjects and thenlog_test_result_feedback_scores()converts them toBatchFeedbackScoreDict. The persisted shape carries the score name/value/reason/category, but notScoreResult.metadata; the currentFeedbackScoreMessage/ batch feedback-score types also have no metadata field.A concrete use case is evaluator provenance:
The local evaluation result can retain this information, but the durable feedback score becomes only
correctness = 0.83.Could evaluation-created feedback scores preserve safe scorer-supplied metadata (or provide an equivalent first-class evaluator revision/fingerprint field)? I don't think the API needs to prescribe the metadata vocabulary; preserving an opaque user-defined revision/fingerprint would already solve the provenance problem.
There is some useful prior art in #5178, which attempted to propagate
ScoreResult.metadatathrough feedback-score payloads while working on evaluator-failure visibility. That PR was later closed as inactive/merge-conflicted rather than merged. The current request is narrower: durable provenance for successful evaluation scores.Motivation
A score name identifies the quality dimension, not necessarily the exact scoring definition. Two results can both be named
correctnessand even use the same judge model while being produced by different rubric prompts, scoring code, thresholds, or configuration.Without evaluator provenance on the persisted score, historical experiment comparisons can become ambiguous: a score change may come from the evaluated system, or from a changed evaluator definition.
The desired invariant is roughly:
persisted score = evaluated target + score name + score value + evaluator revision/provenanceThis does not require storing raw rubrics, prompts, credentials, or other sensitive configuration. A caller-controlled stable revision or fingerprint is enough, and would also make re-scoring historical outputs auditable.