Skip to content

evaluation-cli.md

Latest commit

 

History

History
93 lines (71 loc) · 4.05 KB

evaluation-cli.md

File metadata and controls

93 lines (71 loc) · 4.05 KB
Raw

Evaluation CLI

The source-only structured-thinking-eval distribution exposes the st-eval command and the equivalent python -m st_eval module entry point. Run commands from the repository root unless --root names another complete source tree.

Command overview

Command Purpose External models
st-eval validate Validate schemas and the canonical behavior/trigger catalog No
st-eval run Execute a configured behavior or trigger suite and write evidence Yes
st-eval validate-evidence Recompute and validate an existing evidence manifest No new generation

The global form is st-eval --root /absolute/repository <command> .... The --root option appears before the subcommand.

Validate the catalog

rtk uv run --frozen st-eval validate

Inputs are the five schemas under evals/ and the canonical evals/cases/catalog.json. Success reports the behavior- and trigger-case counts. A non-zero exit means a schema, case, routed source, or generated-contract validation failed; no evidence run should start until it is resolved.

Run an evaluation

Model-backed runs require absolute adapter commands in ST_EVAL_MODEL_CONFIG. All suites require an exact Plugin archive through ST_EVAL_PLUGIN_ARCHIVE; release runs also require a current protected judge-calibration manifest through the workflow contract.

ST_EVAL_MODEL_CONFIG=/absolute/model-config.json \
ST_EVAL_PLUGIN_ARCHIVE=/absolute/structured-thinking-0.1.0-rc.1.tar.gz \
rtk uv run --frozen st-eval run \
  --suite smoke \
  --runs-per-condition 3 \
  --output-dir artifacts/smoke \
  --seed 20260713

Valid suites are smoke, pr, nightly, release, and trigger. --runs-per-condition must match the lane policy; three smoke runs are diagnostic only. --seed makes run ordering reproducible, not model output deterministic. The output directory receives raw attempts, grades, judge records where required, and manifest.json.

A non-zero exit can mean evaluation failure, gating failure, invalid provenance, adapter failure, or incomplete infrastructure. Inspect the retained attempt and error records; retries do not erase failed attempts.

Validate evidence

rtk uv run --frozen st-eval validate-evidence \
  --manifest artifacts/smoke/manifest.json

This command validates the manifest schema and recomputes referenced hashes, attempt accounting, grades, judge agreement, request-ID uniqueness, cache attestations, archive/catalog/evaluator bindings, and summary values. It must run against the source/evaluator version identified by the evidence. Success means the retained bytes are internally consistent; it does not promote the run to release evidence.

Evidence outputs

Artifact Meaning
manifest.json Provenance, status, summary, and hash index
Attempt records Every requested execution, including infrastructure failures
Raw executor output Provider response before grading
Grade records Deterministic and semantic assertion results
Raw judge records Judge response and provenance for semantic assertions
Plugin archive and checksum Exact treatment bytes evaluated by protected lanes

Artifact names are manifest-authoritative; consumers must not infer completeness from a directory listing alone.

Evidence boundaries

Local mocks and smoke runs test plumbing. Public regression cases test known behavior. Neither is a private holdout or a Production claim. Nightly evidence detects drift but does not create a release. Only a protected release-candidate workflow can create RC evidence, and publication must reuse the exact accepted archive without new generation or rebuild.

Missing credentials, malformed JSON, timeout, non-zero adapter exit, duplicate request ID, cache hit, missing provenance, failed clean-control isolation, or unavailable judge calibration makes a run failed or incomplete—never passing. See the quality policy, benchmark policy, and evaluator adapter.