Skip to content

[Feature] Export calibrated shadow comparison datasets with immutable provenance #3280

Description

@Xunzhuo

Goal

Export calibrated primary-versus-shadow comparison datasets with immutable provenance for reviewed Router Model experiments.

Proposed Workgroup

Router Models & Inference Runtime

Scope

  • Convert approved shadow observations into a versioned dataset manifest.
  • Preserve prompt/reference identity, primary and shadow model identities, output hashes, judge or label provenance, sampling policy, and redaction state.
  • Define deterministic inclusion, exclusion, deduplication, balancing, and train/eval split rules.
  • Validate schema, lineage, privacy policy, and artifact immutability before export.
  • Support pluggable storage destinations without committing datasets to the source repository.

Non-goals

  • Capturing live shadow traffic; tracked by the shadow dispatch issue.
  • Training or promoting a model.
  • Treating an unreviewed judge score as ground truth.

Completion signal

  • One approved shadow corpus exports reproducibly from the same inputs to the same manifest and split identities.
  • Every example is traceable to immutable, privacy-reviewed evidence.
  • Invalid lineage, redaction, or split leakage fails before publication.

Parent Epic: #2975

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-acceptanceAwaiting explicit Maintainer or Workgroup acceptance before implementation.wg/router-models-inference-runtimeOwned by the Router Models and Inference Runtime Workgroup.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions