reference.md

Reference

Command reference, runner templates, and troubleshooting for frontierharness-eval.

Runta CLI commands used

Purpose	Command
Create a clean runtime	runta run --name demo --cpus 4 --memory 8192 --disk-size-gib 50
Run a command inside it	runta exec demo -- sh -lc 'harbor --version'
Copy files in or out	runta cp local.txt demo:/work/ · runta cp demo:/work/jobs ./jobs
Store a provider key	runta secret set OPENAI_API_KEY --value-env OPENAI_API_KEY
Restrict network	runta egress set demo --mode allowlist --allow api.fireworks.ai
Freeze a golden checkpoint	runta checkpoint create demo fh-golden-v1
Restore a fresh runtime	runta checkpoint restore fh-golden-v1 fh-trial-01
List checkpoints	runta checkpoint ls
Inspect or clean up	runta inspect demo · runta rm demo

runta run requires both --cpus and --memory. --disk-size-gib defaults to 16, which is too small for the eval environment, so pass 50. Restoring a checkpoint always creates a new runtime with the checkpoint's capacity; it never mutates the checkpoint. Full docs: https://runta.com/docs/.

Use -- before the remote command in runta exec whenever arguments could be parsed as CLI options, and wrap multi-step commands in sh -lc '...'.

Model and providers

The benchmark fixes the model at Kimi K3, matching benchmark.json ("model": "Kimi K3"). Both scripts warn if the model is anything else. The provider is a choice: the published baselines used Fireworks ("model_provider": "Fireworks"), but another provider requires a matched control to establish comparable serving behavior. Select one with --provider; the presets live in scripts/providers.sh.

--provider	Model route	Credential	Egress host
fireworks (default)	fireworks_ai/accounts/fireworks/models/kimi-k3	FIREWORKS_API_KEY	api.fireworks.ai
moonshot	moonshot/kimi-k3	MOONSHOT_API_KEY	api.moonshot.ai
openrouter	openrouter/moonshotai/kimi-k3	OPENROUTER_API_KEY	openrouter.ai
together	together_ai/moonshotai/Kimi-K3	TOGETHER_API_KEY	api.together.xyz
custom	--model (required)	--secret-name (required)	--secret-host (required)

The provider prefix is the LiteLLM provider route, which is what Harbor, Pier, and mini-swe-agent expect. A harness that calls the provider directly through an OpenAI-compatible client wants the bare model id against that provider's base URL instead: accounts/fireworks/models/kimi-k3 at https://api.fireworks.ai/inference/v1, kimi-k3 at https://api.moonshot.ai/v1, and so on. Moonshot's China platform is https://api.moonshot.cn/v1 with a separate key; keys are not interchangeable across its regional platforms.

Use custom for a gateway, a regional endpoint, or a provider not listed here, and keep the model id recognisably Kimi K3 or the scripts will warn:

--provider custom \
  --model openai/kimi-k3 \
  --secret-name MY_GATEWAY_API_KEY \
  --secret-host gateway.example

Trial network access

The provision and trial scripts use the same host allowlist for both suites: the selected provider plus the hosts in scripts/providers.sh. These cover uv and its redirects, GitHub sources, Harbor's Supabase task registry, PyPI, npm, Ubuntu and Debian apt mirrors, and PyTorch packages. The verifier probe in PR #11 identified these dependencies. Allowing only the provider and uv hosts leaves Harbor's registry and verifier dependency installs blocked.

Runta's policy applies to the whole runtime, including agent containers. Agents therefore have package-registry and source access under this policy, subject to additional Harbor/Pier isolation; it is not a verifier-only exception. Every hostname must match an allowlist entry, including redirect targets. Related subdomains use wildcard patterns such as *.githubusercontent.com, *.supabase.co, and *.cloudfront.net; apex hosts like github.com stay listed because a wildcard does not match them. There is no provider-only fallback. Provisioning installs packages and clones sources before restricting egress; each trial pulls its selected image before applying the policy, so container registries need not stay open during execution. Confirm with:

runta egress describe fh-build

run.json.egress_policy records mode: "allowlist", scope: "runtime", and the sorted, deduplicated allowed_hosts generated by the same helper used to apply the policy. candidate.json and both reports carry this policy forward. A policy setup failure prevents agent launch and leaves the trial infra_invalid.

The published baselines' applied allowlist was not recorded. Package installation can change task outcomes, so new runs set methodology_comparable: false and remain unranked even with full task coverage. Run a matched control using the same network policy and environment before asserting comparability; record the supporting evidence before changing that flag. A matching pass count alone is insufficient.

Resuming with a different provider host, changed allowlist, or missing policy record is refused before touching a runtime. Use a new run id for the new policy; retain old evidence under its original conditions. For a retained runtime that still needs evidence collection, use the original script version and configuration to recover it.

Cost comparability

The cost calculation uses the frozen benchmark price table (scripts/pricing.json, version kimi-k3-2026-08-20): $3.00 fresh input and cache writes, $0.30 cache reads, and $15.00 output per million tokens. Recognized Kimi K3 routes use this common benchmark basis; these are not live provider bills. Unknown models need an explicit table, keyed by the exact model name:

node /path/to/scripts/normalize-results.mjs --run runs/my-run --pricing /path/to/pricing.json

Harbor/Pier agent_result and step_results provide inclusive input, cache read, cache write, and output totals. The bundled harness log parsers recover first-call cache reads. First-cold cost is token-priced cost plus first_turn_cached_tokens * (fresh_input_rate - cache_read_rate) / unit_tokens. Later calls retain their cache discount. The normalized cache rate is (cached_input_tokens - first_turn_cached_tokens) / input_tokens.

Reported billing stays in reported_cost_usd; it is only a fallback for cost_usd. Missing first-call details leave cost_first_cold_usd null. The normalizer also recalculates existing runs from retained job evidence without rewriting raw records. Multiple leaf results are ambiguous and remain unpriced, avoiding retry double counting.

Success-only mean and median normalized costs require 100% successful-task cost coverage. Effective cost per pass is the sum of known costs across all canonical cells (including invalid cells with known costs) divided by passes, with missing-cost coverage disclosed.

Runner templates

run-trials.sh picks a template from the task id prefix. Override with --cmd. Placeholders: {task}, {suite}, {harness}, {model}, {jobs}.

Terminal-Bench through Harbor (terminal-bench/*):

harbor run -d terminal-bench@2.0 -i {task} -a {harness} -m {model} --jobs-dir {jobs} --extra-docker-compose /work/runta-ca-overlay.yaml -r 2 -y

terminal-bench@2.0 is the registry name that holds this benchmark's 21 Terminal-Bench tasks. terminal-bench/terminal-bench@4.0.0 is a different 66-task corpus that shares none of these ids, so it silently evaluates nothing. Harbor 0.22 selects tasks with -i, not the removed --task-id.

DeepSWE through Pier (datacurve/*):

pier run -p /work/deep-swe/tasks/{task} --agent {harness} --model {model} --jobs-dir {jobs}

Verify the flag names against the installed versions before a full run — Harbor and Pier both change CLI surface between releases:

runta exec fh-build -- sh -lc 'harbor run --help | head -40'
runta exec fh-build -- sh -lc 'pier run --help | head -40'

Pier exists because Harbor blocks all outbound traffic on allow_internet = false tasks, including the agent's own LLM calls. Pier adds per-agent network allowlists, so the agent reaches its provider while the task environment stays isolated. That is why DeepSWE tasks go through Pier rather than Harbor.

If a harness is not a built-in agent for either runner, register it as a custom agent in the runner's agent registry inside --install-script, then pass its registered name as --harness. This includes services: use --harness-topology runtime-service for a service inside the Runta host, or external-service for a service on another host. Record its version, resource budget, network access, and per-task state reset. Custom registration is supported, but is not proof of equivalence to the container CLI baselines.

Running Harbor with Runta as the environment provider

The skill's default topology runs Harbor and Pier inside one Runta runtime, which is what makes a single golden checkpoint cover the whole stack. The alternative is to run Harbor locally and let Runta provide a fresh runtime per trial:

uv pip install 'runta-sdk[harbor]'
harbor run -d terminal-bench-sample@2.0 -a oracle -l 1 \
  --environment-import-path runta_harbor.environment:RuntaEnvironment \
  --ek mode=auto

This parallelizes better, but Harbor controls runtime creation, so trials no longer start from one shared golden checkpoint. Prefer it only for large sweeps where throughput matters more than identical cold starts, and say so in the report. Apply the same runtime quota and concurrency rules to Harbor-managed runtimes; bound concurrency before starting a sweep.

Known limitations of the provider: harbor run -e runta is not registered yet, so the import path is required; GPU, TPU, Windows, and mounted-resource tasks are unsupported; storage overrides are accepted but not applied.

Task metadata

Each tasks/<task>/task.toml in this repo carries the environment contract:

[metadata]
task_id = "anko-typed-variable-bindings"
repository_url = "https://github.com/mattn/anko"
base_commit_hash = "3f269a72ff69398b1250c584171f32d12c0d8085"
[agent]
network_mode = "no-network"
timeout_sec = 5400.0
[environment]
docker_image = "public.ecr.aws/..."
cpus = 2
memory_mb = 8192

Match cpus, memory_mb, and timeout_sec when running a task, otherwise results are not comparable to the baseline. The trial script pulls the selected image after each restore. Terminal-Bench uses these files; DeepSWE uses the pinned corpus metadata. --prepull-tasks tasks is optional and can produce an oversized checkpoint.

Trial record contract

run-trials.sh writes one trial.json per task. normalize-results.mjs reads only these fields, so any custom runner can produce them directly:

{
  "id": "terminal-bench/regex-log",
  "title": "regex-log",
  "suite": "terminal-bench",
  "status": "success",
  "success": true,
  "duration_seconds": 269,
  "cost_first_cold_usd": 0.0561,
  "turns": 3,
  "cache_hit_rate_normalized": 0.521,
  "exit_code": 0,
  "runtime": "fh-2026-09-02-terminal-bench-regex-log",
  "checkpoint": "fh-golden-myharness-v1"
}

status is one of success, failure, or infra_invalid. Only infra_invalid is excluded from scoring. Setup failures (restore, readiness, image pull, egress, credentials) and unconfirmed execution or evidence transfer are marked automatically. Verifier outcomes require model usage. A timeout after agent execution starts is failure; pre-execution timeouts and unproven crashes are infra_invalid. Valid outcomes are never retried by transport recovery. A pending record carries recovery: true, runtime, and the exact runner_command: re-running the same run reattaches to that runtime. A valid attempt is never replaced. Earlier setup failures are archived under runs/<run-id>/attempts/ before retrying. Use a new run id for a new experiment. Retained runtimes continue to consume resources until recovered or explicitly removed.

Reward extraction reads top-level resolved, is_resolved, reward, or passed, followed by the pinned runners' explicit verifier_result.rewards.reward field, from result-named JSON files first (result.json, results.json, eval.json, verifier.json), then other JSON files in sorted path order. It does not recursively walk nested objects, so a passing unit test cannot mark the trial as success. If a runner reports success differently, the extracted value will be wrong — spot-check the first trial:

jq . runs/<run-id>/trials/<task>/trial.json
find runs/<run-id>/trials/<task>/jobs -name '*.json' | head

Troubleshooting

No module named 'runta_harbor' — harbor is running from a Python environment without runta-sdk[harbor], usually a global uv tool install at ~/.local/bin/harbor. Use uvx --with "runta-sdk[harbor]" harbor run ....

RESOURCE_EXHAUSTED on create or restore — the tenant CPU or memory limit is reached. Stop provisioning and follow the runtime quota and concurrency rules. Inspect runta ps -a; a crashed run-trials.sh can leave runtimes behind, including attempts retained for evidence recovery. Recover those attempts first. Use runta rm <name> only for this evaluation's finished runtimes after verifying their evidence, or its build runtime after the checkpoint and checks succeed. Leave unrelated runtimes alone, and recheck capacity before retrying.

Agent cannot reach the model provider — the task is air-gapped. Confirm the runtime egress policy allows the provider host and that the key is a stub inside the runtime:

runta egress describe fh-build
runta exec fh-build -- sh -lc 'test "$FIREWORKS_API_KEY" = runta-secret-stub && echo stubbed'

Every task fails identically at turn zero — almost always harness configuration, not task difficulty. Restore the checkpoint once and run the harness by hand before spending a full sweep:

runta checkpoint restore fh-golden-myharness-v1 fh-debug
runta exec fh-debug -- sh -lc 'cd /work && <runner command with one task>'

Costs are null in the report — retain token totals and supported per-call harness logs, including first-call cached reads. Provider billing alone cannot establish first-cold cost. Custom harnesses need a parser in scripts/usage_details.py.

Corpus pin and Pi control

The published dataset label v1.1 in benchmark.json is not an upstream Git tag. The reproduction workflow pins 435ee89ec2f2e2289f33b0da4f992f0b7b7266b9. Do not silently substitute a tag or the latest branch. The resolved SHA is recorded in each manifest; --deep-swe-ref allows an explicitly documented override.

This commit has the selected task names, but is not byte-identical to the frozen public metadata: for example, HTTPX uses schema 1.3, a separate verifier, and a -v1.1 image tag where this repository records schema 1.1 and an unsuffixed image. The report discloses this difference. The publication dates alone do not establish which commands, upstream refs, or platform behavior produced the original baselines.

To assess reproduction, use a separate Pi control run with the same patched scripts, provider, task set, CPU, memory, disk, image policy, and runc setting as the candidate. metadata/harness-versions.json pins Pi to 0.84.2, identified as pi-responses. Install and register the matching Responses adapter in both runners before using that name; the public version record alone does not supply the adapter implementation or all of its configuration. Record that provenance in the install script and report.

After provisioning a Pi checkpoint with those settings, run:

bash "$FH/run-trials.sh" --checkpoint fh-golden-pi-control --harness pi-responses \
  --provider fireworks --run-id pi-control --tasks tasks --out runs
node "$FH/normalize-results.mjs" --run runs/pi-control --label "Pi control"

Use one task from each suite first to verify setup, then resume the same run id for all 30 so valid smoke attempts stay canonical. Compare all 30 task-level outcomes with the published Pi result (18/30), including missing or infra-invalid tasks and cost coverage. A matching total alone does not prove every environment difference is immaterial. This repair does not claim that the control has been run.

Checkpoint stays creating — large snapshots have been reported to stall around 10 GiB; this is an observed failure range, not a guaranteed platform limit. The full image set can reach roughly 22 GiB. Leave --prepull-tasks unset and pull per task. The script waits up to --checkpoint-timeout 900 for ready, fails clearly on error or timeout, and leaves the build runtime for inspection. Verify the eventual state before deleting it or requesting another checkpoint.

Pier verifier image build hangs for 1,800 seconds — Runta's injected init has been reported to prevent container exit. Provisioning restores only the known /usr/local/sbin/runc -> /opt/runta/runta-runc symlink to /usr/bin/runc before any sample or verifier runs. It records the resolved binary path. The existing Harbor CA overlay still supplies trust to task containers. --keep-runta-runc disables this workaround for platform validation; use the same choice for candidate and control.

runta exec disconnects or runta cp fails — the worker runs under nohup, setsid, and a lock, with a remote timeout and atomic completion record. Repeating launch after a lost acknowledgement cannot execute the harness again. The controller polls in short sessions, then copies a checksum-verified archive of jobs, runner log, completion record, and manifest. A nonzero copy exit with a matching checksum is accepted; a partial copy is retried up to three times. If recovery is still needed, the runtime is retained and the attempt is excluded until its evidence is collected. Re-run the same command; do not delete that runtime or start another harness attempt.

Local regression checks (no cloud or model spend): npm test.