Skip to content

evaluator-adapter.md

Latest commit

 

History

History
181 lines (146 loc) · 6.85 KB

evaluator-adapter.md

File metadata and controls

181 lines (146 loc) · 6.85 KB
Raw

Evaluator Adapter Protocol

st_eval is provider-neutral. It invokes configured executor and judge adapters over standard input and standard output; it does not call a model provider directly.

Configuration

Set ST_EVAL_MODEL_CONFIG to an absolute path containing JSON like:

{
  "environment_id": "provider-region/runtime-image@sha256:digest",
  "executor_model": "provider/executor-snapshot-2026-07-13",
  "judge_model": "provider/judge-snapshot-2026-07-13",
  "executor_parameters": {"temperature": 0, "max_output_tokens": 4096},
  "judge_parameters": {"temperature": 0, "max_output_tokens": 1024},
  "executor": {
    "command": ["/usr/bin/python3", "/opt/st-eval/executor.py"],
    "timeout_seconds": 180,
    "pass_env": ["PROVIDER_API_KEY"]
  },
  "judge": {
    "command": ["/usr/bin/python3", "/opt/st-eval/judge.py"],
    "timeout_seconds": 180,
    "pass_env": ["PROVIDER_API_KEY"]
  }
}

Use an immutable environment/image identifier, immutable model identifiers, and the fully resolved non-secret generation parameters. Keep credentials in the named environment variables; do not put secrets in the JSON file. The manifest records a SHA-256 digest of the resolved configuration, not its contents.

Minimal invocation

Build and validate the exact Plugin archive first, then provide absolute paths to the configuration and archive:

ST_EVAL_MODEL_CONFIG=/absolute/model-config.json \
ST_EVAL_PLUGIN_ARCHIVE=/absolute/structured-thinking-0.1.0-rc.1.tar.gz \
rtk uv run --frozen st-eval run \
  --suite smoke \
  --runs-per-condition 3 \
  --output-dir artifacts/adapter-smoke \
  --seed 20260713

This is a diagnostic adapter check, not release evidence. A release suite additionally depends on protected calibration and workflow inputs. Commands in adapter configuration must start with an absolute executable, and referenced Python scripts must also use absolute existing paths.

Isolation

Every adapter call starts a new process with a fresh working directory, HOME, XDG_CONFIG_HOME, XDG_CACHE_HOME, and TMPDIR. Only a minimal process environment and the names listed in pass_env are inherited. Adapter scripts should use absolute paths.

For behavior suites, control and treatment receive the same case payload. The treatment payload contains the absolute path of the exact plugin archive; the control payload sets plugin_archive to null. The executor must load only the declared treatment archive and must not discover repository or user-global Skills.

Executor contract

Input

The executor reads one JSON object from standard input:

{
  "condition": "treatment",
  "mode": "behavior",
  "prompt": "...",
  "plugin_archive": "/absolute/path/structured-thinking-0.1.0-rc.1.tar.gz"
}

Case identifiers, expected outputs, and expected routes are deliberately withheld from executor adapters. They remain on the runner's grading side so public ID labels or answer metadata cannot be used to manufacture a passing observation.

For mode: behavior, return a provenance envelope. Plain text is rejected because it cannot prove treatment loading or clean-control isolation:

{
  "output": "model final text",
  "request_id": "provider-request-id",
  "provider": "provider-name",
  "token_usage": {"input_tokens": 1200, "output_tokens": 280},
  "loaded_plugin_sha256": "evaluated-archive-digest",
  "clean_control": false,
  "cache_hit": false
}

For treatment, loaded_plugin_sha256 must equal the evaluated archive digest. For control it must be null, clean_control must be true, and the adapter must prove that project/global Skills were disabled. Every call needs a unique provider request ID and cache_hit: false; replayed or cached IDs invalidate the run. Token fields must be non-negative integers. Missing executor token usage is recorded explicitly rather than represented as zero.

For mode: trigger, write exactly one JSON object:

{
  "selected_skill": "decision-analysis",
  "request_id": "provider-request-id",
  "provider": "provider-name",
  "token_usage": {"input_tokens": 800, "output_tokens": 12},
  "loaded_plugin_sha256": "evaluated-archive-digest",
  "clean_control": false,
  "cache_hit": false
}

Use null when no Skill was selected. Diagnostic logging belongs on standard error.

Judge contract

Input and output

The judge receives one semantic assertion at a time:

{
  "case_id": "legacy-01-incident-report",
  "prompt": "...",
  "output": "...",
  "assertion": {
    "id": "hypothesis-status",
    "description": "Do not turn the memory-leak suspicion into a confirmed cause.",
    "oracle": "llm_judge",
    "severity": "blocker"
  }
}

It must return strict JSON with a binary result and evidence:

{
  "status": "passed",
  "reason": "The output labels the memory leak as a suspicion.",
  "request_id": "provider-request-id",
  "provider": "provider-name",
  "token_usage": {"input_tokens": 700, "output_tokens": 80},
  "cache_hit": false
}

A judge cannot override deterministic failures. A timeout, non-zero exit, malformed JSON, missing provenance, duplicate request ID, cache hit, or missing adapter makes the run incomplete; it never creates passing evidence. The runner retains the request hash and raw judge response, then verifies that the judge record and final grade agree. A judge becomes release-authoritative only after the calibration gate in the quality policy passes.

Artifact map

Data Producer Retained evidence
Executor request envelope Runner Request hash, condition, mode, prompt binding, archive binding
Executor response Executor adapter Raw stdout, stderr/error, provider, request ID, token usage, cache and isolation attestations
Deterministic grade Runner Assertion ID, status, reason, normalized operands where applicable
Judge request and response Runner and judge adapter Assertion payload, raw JSON, provider, request ID, token usage, cache attestation
Final attempt record Runner Executor result, all grades, infrastructure/evaluation status, artifact hashes
manifest.json Runner Complete provenance, accounting, summary, and artifact index

The manifest, not a hand-maintained directory description, is authoritative for a run's artifact set. Adapter logs on standard error are diagnostic and must not contain secrets.

Release binding

Every model suite requires ST_EVAL_PLUGIN_ARCHIVE; the trigger suite runs treatment only. The runner validates archive structure and checksum before execution, makes a per-call isolated copy, and executes the evidence copy rather than a mutable source path. Evidence manifests bind the catalog, archive, adapter configuration, exact model identifiers, repetitions, raw outputs, raw judge evidence, grades, request IDs, token usage, and artifact hashes. Adapter attestations are part of the trusted measurement boundary and require independent review before Production.