Epic: Harbor-based coding evals to BlindBench review and training data

[nmogil]

Outcome

Mogil Bench becomes the Harbor-based execution and evidence layer for coding-agent evaluations, with local Docker and Daytona environments, full Pi trajectories, BlindBench human review, and an approval-gated path to open-weights fine-tuning data.

Architecture direction

• Harbor owns coding task/trial/environment/verifier execution.
• Mogil Bench owns benchmark packs, model/harness matrices, stable logical identity, evidence policy, and BlindBench workflow.
• BlindBench owns blinded human review and approved-data reuse.
• Daytona is an initial remote Harbor environment; local Docker is the first implementation target.
• Fireworks-compatible SFT/DPO export reuses BlindBench #287; training jobs remain separately approved.

Workstreams

• Adopt Harbor as the coding-eval execution foundation #7 — Harbor foundation and local Docker execution.
• Add Pi trajectory capture with ATIF and OTel projections #6 — Pi structured trajectory, ATIF v1.7 projection, workspace/verifier evidence.
• Add Daytona as a Harbor sandbox backend #8 — Daytona Harbor backend and provider conformance.
• Provider-parity matrix, lifecycle hardening, and BlindBench ingest compatibility (feat: add Daytona provider parity matrix #15, fix: reuse event loop across Harbor attempts #16, fix: align parity evidence with BlindBench ingest #17).
• Export deterministic BlindBench paired comparisons from parity runs #18 — deterministic BlindBench paired-comparison export.
• Make BlindBench uploads timeout-aware and diagnostically safe #19 — timeout-aware, diagnostically safe BlindBench uploads.
• Prepare Mogil Bench v0.1.0 public-alpha release #20 — v0.1.0 public-alpha release readiness.
• Render full Harbor agent spans and evidence in blind review nmogil/blind-bench#354 — full agent-span human review.
• Final product push: end-to-end BlindBench readiness for customer testing nmogil/blind-bench#287 — approval-gated Fireworks-compatible dataset export.

Product gate

The paid-comparison gate was met with fictional/public tasks: 18/18 provider-parity attempts were quality eligible, retained complete trajectories, passed separate hidden verification, produced no credential/canary leakage, and confirmed sandbox cleanup. Mogil Bench v0.1.0 was published as a verified GitHub public-alpha pre-release on 2026-07-15: https://github.com/mogilventures/mogil-bench/releases/tag/v0.1.0.

Non-goals

• BlindBench executing agents or owning model/runtime credentials.
• Training or deploying a model without explicit approval.
• Customer/private traces before data-boundary approval.
• Rebuilding Harbor environment orchestration inside Mogil Bench.

[nmogil]

Product principle: objective scoring plus expert human taste review

Mogil Bench should demonstrate how benchmark operators combine two distinct forms of evaluation:

1. Objective scorers/verifiers establish whether observable requirements were met, such as tests, artifacts, constraints, and deterministic outcome checks.
2. BlindBench human review evaluates qualities that automated scorers cannot reliably capture: taste, style, judgment, strategy, clarity, maintainability, unnecessary complexity, and whether the agent behaved like an expert would.

Expert reviewers may also identify missing steps or flawed approaches that a narrow deterministic verifier overlooks.

The two signals must remain separate in the data model and reporting. Human judgments complement scorer results; they must not be collapsed into one opaque score or used as a substitute for deterministic task verification.

Mogil Bench should be a public example of using BlindBench as the human feedback layer for LLM and AI-agent evaluations. A reviewable coding run therefore needs the full ordered agent span, tool calls/results, workspace changes, verifier evidence, and final output—not only a scorer result or final prose.

For approved training export, retain objective outcomes and human judgments as separate labeled fields so operators can construct SFT, preference, reward-model, or analysis datasets with explicit provenance and consent.

[nmogil]

Activation complete

PR: #11 — green (test, docker-smoke).

Three real Harbor 0.18.0/local-Docker/Pi 0.80.6 tasks ran successfully and produced strict quality-eligible full-span evidence. BlindBench accepted all three with complete counts and created this working blinded guest review:

https://app.blindbench.dev/review/verdict/68d3e2c450a627edf418493f795374c4

Production verification confirmed chronology, sanitized tool details, workspace changes/patch, verifier evidence, objective outcomes, final output, and human verdict controls. One independent human Strong verdict was submitted; review aggregate is now best=1, judgment_count=1, while objective verifier outcomes remain separately displayed/stored. The temporary Automation token was revoked and returns HTTP 401. Temporary containers, credentials, prompts, artifacts, Herdr workspaces, and implementation worktrees were removed.

[nmogil]

Remaining roadmap workstreams are now delivered as green PRs:

• Daytona Harbor backend/provider conformance: feat: add Harbor Daytona sandbox backend #12
• Approval-gated full-span Fireworks-compatible export: feat: gate full-span training exports nmogil/blind-bench#356

Both passed independent review and complete local/CI verification. These are PR-delivered, not merged or activated.

Still intentionally gated:

• live remote Daytona smoke requires credentials + immutable image digest
• no production training approval/export
• no Fireworks call or training job
• no merge/deployment without owner review