Mogil Bench v1: local benchmarks to BlindBench artifacts

.github/workflows/ci.yml

@@ -0,0 +1,32 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - run: python -m pip install --upgrade pip
+      - run: python -m pip install -e '.[dev]'
+      - run: pytest -q
+      - run: ruff check .
+      - run: mypy
+      - name: Credential-free artifact smoke test
+        run: |
+          mogil-bench pack validate packs/sample-v1.yaml
+          mogil-bench run packs/sample-v1.yaml --output-dir /tmp/mogil-bench-ci
+          mogil-bench artifact validate /tmp/mogil-bench-ci/blindbench.json

README.md

@@ -0,0 +1,93 @@
+# Mogil Bench
+
+Mogil Bench v1 is a local Python CLI for running versioned, real-work-like benchmark packs. It keeps Hermes/text and Pi/coding tasks distinct while exporting both as BlindBench `eval-record` v1 JSON and JSONL for blind human review.
+
+## Install
+
+Python 3.11+ is required.
+
+```bash
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install -e '.[dev]'
+```
+
+## Verified quick start
+
+Run from the repository root:
+
+```bash
+mogil-bench pack list packs
+mogil-bench pack validate packs/sample-v1.yaml
+mogil-bench run packs/sample-v1.yaml --output-dir /tmp/mogil-sample-run
+mogil-bench artifact validate /tmp/mogil-sample-run/blindbench.json
+mogil-bench artifact validate /tmp/mogil-sample-run/blindbench.jsonl
+mogil-bench export blindbench /tmp/mogil-sample-run
+```
+
+The sample uses a deterministic mock and needs no credentials. A deliberately guarded local command smoke test is separate:
+
+```bash
+mogil-bench pack validate packs/command-smoke-v1.yaml
+mogil-bench run packs/command-smoke-v1.yaml \
+  --output-dir /tmp/mogil-command-run --allow-commands
+```
+
+Remove an existing output directory before repeating these exact commands. Runs refuse to overwrite an existing directory.
+
+Each run contains:
+
+- `manifest.json`: pack identity/fingerprint and result index;
+- `results/*.json`: one raw execution result per task/configuration pair;
+- `blindbench.json`: `{ "records": [...] }` batch;
+- `blindbench.jsonl`: one `eval-record` v1 object per line.
+
+Record IDs are deterministic hashes of canonical task, configuration, pack revision, and fixture content. Timestamps do not affect IDs, so retries deduplicate in BlindBench. V1 records measured duration. It omits token counts and cost unless an adapter has authoritative values; mock word counts are not reported as tokens.
+
+## Pack format
+
+A pack has `version`, stable `id`, explicit `revision`, metadata, tasks, and configurations. Tasks identify a `lane` (`hermes-text` or `pi-coding`), category, prompt and/or relative fixture, privacy class, timeout, and optional command/verifier. Configurations identify provider, model, harness name/version/SDK, and adapter (`mock`, `command`, or `pi`).
+
+See [`packs/sample-v1.yaml`](packs/sample-v1.yaml), [`packs/command-smoke-v1.yaml`](packs/command-smoke-v1.yaml), and the non-quick-start [`packs/pi-template-v1.yaml`](packs/pi-template-v1.yaml). Fixture references cannot escape the pack directory. Update the pack revision whenever task intent changes; fixture bytes are also fingerprinted.
+
+### Trusted local Pi adapter
+
+The Pi template is deliberately not credential-free and contains placeholder provider/model/harness values. Review and copy it before use, then run explicitly:
+
+```bash
+mogil-bench run path/to/reviewed-pi-pack.yaml \
+  --output-dir /tmp/mogil-pi-run --allow-agents
+```
+
+The adapter resolves `pi` from `PATH`, or an absolute executable path named `pi` from `MOGIL_BENCH_PI_EXECUTABLE` (primarily for controlled testing). It invokes Pi directly with `--print`, `--no-session`, the configuration's `--provider` and `--model`, only the `read,write,edit` built-in tools, and a fixed benchmark system prompt that directs work to the temporary directory. It disables extensions, skills, prompt templates, context files, project approval, update checks, and telemetry. Known provider API-key variables and an explicitly set `PI_CODING_AGENT_DIR` are passed without being logged.
+
+V1 captures Pi's final stdout plus bounded stderr/status/duration; it does not normalize Pi's JSON event stream, tool-call trajectory, authoritative token usage, or provider cost. Those are follow-up work. Provider/model/harness fields in exported records are the same configuration values used for invocation.
+
+## Command safety
+
+Commands are deny-by-default and need **both** `allow_commands: true` in the pack and CLI `--allow-commands`. Pi runs use the separate `allow_agents: true` plus `--allow-agents` gate; one acknowledgement never enables the other. Execution uses argv arrays with `shell=False`, a fresh temporary work directory, fixture copies, a small environment allowlist, timeout, and bounded stdout/stderr. The generic command adapter still forbids Pi, Hermes, Git, common network clients, Docker, and kubectl. A failure, denial, timeout, or verifier failure is recorded and does not stop the rest of the matrix.
+
+These are guardrails, not a hardened OS sandbox: an allowed interpreter can run arbitrary code, and Pi's read/write/edit tools are not filesystem-confined by the operating system. Only acknowledge trusted packs and run them in an external sandbox when stronger isolation is required. Production-grade untrusted execution is out of scope.
+
+## BlindBench upload
+
+Upload is dry-run by default and validates both artifact and endpoint without making a request:
+
+```bash
+mogil-bench artifact upload /tmp/mogil-sample-run/blindbench.json \
+  --endpoint https://DEPLOYMENT.convex.site/ingest/v1/traces
+```
+
+A real upload additionally requires `BLINDBENCH_INGEST_TOKEN` and `--confirm`. Only HTTPS `*.convex.site/ingest/v1/traces` endpoints are accepted. The CLI never prints the token or record content and reports only response counts. It treats `invalid > 0`, `truncated: true`, a malformed counts response, or an imported-plus-deduped count that differs from the intended batch size as an upload failure. Tests make no network calls.
+
+Prompts and outputs are reviewer-visible and free text is not automatically scrubbed by BlindBench. Never benchmark secrets or customer data; set `privacy_class` accurately. Hidden verifier expectations are not exported.
+
+## Development
+
+```bash
+.venv/bin/pytest
+.venv/bin/ruff check .
+.venv/bin/mypy
+```
+
+See [architecture](docs/architecture.md) and [research](docs/research.md).

docs/architecture.md

@@ -0,0 +1,45 @@
+# Architecture
+
+## Ownership boundary
+
+Mogil Bench owns local pack parsing, fixture execution, result capture, and artifact creation. BlindBench owns artifact import, blind human judgment, and result reuse. V1 has no hosted service, scheduler, direct Hermes orchestration, live action, or LLM judge.
+
+## Flow
+
+```text
+versioned YAML pack + local fixtures
+              |
+       parse and validate
+              |
+ task x configuration matrix
+       |               |                  |
+ deterministic mock   guarded command   trusted local Pi
+       |               |                  |
+       per-result raw JSON
+              |
+ immutable-ish manifest (refuses overwrite)
+              |
+ BlindBench batch JSON + JSONL
+```
+
+`models.py` is the untrusted-input boundary for pack and export shapes. `packs.py` resolves fixtures within the pack directory and builds canonical SHA-256 fingerprints. `runner.py` owns three narrow adapters and continues across individual failures. `artifacts.py` maps both lanes to `eval-record` v1, validates JSON/JSONL, and owns the optional guarded upload. `cli.py` is a thin Typer shell.
+
+## Identity and leakage controls
+
+A pack fingerprint hashes canonical JSON of the validated pack plus every fixture file's bytes. A result ID hashes that fingerprint and the canonical task/configuration definitions; wall-clock time is excluded. Harness name/version/SDK, provider, and model remain explicit in every record.
+
+Representative fixtures should be recent, self-contained, and behaviorally verifiable. Private PR-derived fixtures can reduce public benchmark leakage, but v1 does not mine PRs. Before creating such a fixture, copy only the minimum sanitized snapshot into a pack, remove `.git`, do not expose repository history to the runner, keep held-out checks outside reviewer-visible prompt/output, and update the revision. The shipped fixtures are fictional/public.
+
+Verifier commands may determine `verification_passed`; their argv and expected text are not placed in BlindBench exports. Raw files contain outcomes, not hidden expected values. The pack itself remains benchmark-author-only material when it contains held-out checks.
+
+## Safety boundaries
+
+The command adapter requires pack opt-in plus operator acknowledgement, runs no shell, copies fixtures to a temporary directory, sanitizes environment variables, enforces timeout and output bounds, and denies known dangerous executable names.
+
+The Pi adapter has an independent pack/operator gate. It invokes the configured provider/model in ephemeral print mode from the fixture workdir, permits only Pi's read/write/edit built-ins, disables discovered extensions/skills/templates/context and project approval, and accepts an executable override only as an explicit absolute executable path. It passes only known provider credential variables and optional Pi config location; values are never logged. Pi can still access host paths through its tools, and provider inference requires network access. These controls limit accidents but are not process, filesystem, or network isolation; an external sandbox is required for untrusted packs.
+
+Because local adapters do not expose authoritative tokenizer/provider billing data, exports omit token counts and cost rather than inserting word-count estimates or zero-cost claims. Run directories are created with overwrite refusal. Individual errors become statuses (`failed`, `denied`, `timed_out`, or `verification_failed`) so a matrix still yields reviewable partial results.
+
+## Extension points
+
+V1 Pi integration captures the final print-mode output only. Follow-up can consume Pi JSON events to normalize tool calls, tool results, authoritative token usage, and provider-reported cost without changing pack identity or the BlindBench contract. Stronger external isolation and explicit network policy are also follow-up. A future Hermes adapter should return the same result fields; a full autonomous orchestration loop remains intentionally absent.

docs/research.md

@@ -0,0 +1,44 @@
+# Research notes and v1 decisions
+
+## Representative coding benchmarks
+
+Databricks describes an internal benchmark built from recent, representative pull requests in a multi-million-line private codebase, with human-authored self-contained tasks, held-out behavioral tests, sealed Git history, and comparison of model/harness combinations. It emphasizes end-to-end cost per solved task over token price alone and shows that harness choice affects quality and cost.
+
+Mogil Bench adopts explicit model+harness identity, pack/fixture revision hashes, behavioral verifier hooks, privacy-safe recent fixtures, no-history guidance, duration/token/cost fields, and partial matrix results. V1 deliberately does **not** mine pull requests or expose Git history.
+
+Source: [Benchmarking Coding Agents on Databricks' Multi-Million-Line Codebase](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase)
+
+## Small agent harness prior art
+
+[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) demonstrates the value of a small, composable harness and isolated task execution. Mogil Bench borrows the preference for narrow components and explicit environments, but does not vendor it or recreate an agent loop. V1 invokes deterministic mocks, explicitly acknowledged local argv commands, or an independently acknowledged one-shot Pi adapter.
+
+Pi's official [coding-agent CLI documentation](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent) documents print mode, ephemeral sessions, provider/model selection, resource-disabling flags, tool allowlists, and context-file controls. The v1 adapter uses those controls directly instead of wrapping Pi in a shell or recreating its loop.
+
+## Dataset / runner / scorer separation
+
+[Vellum Evals](https://github.com/vellum-ai/evals) is useful prior art for separating datasets, runners, and scoring. Similar evaluation systems reinforce keeping inputs/configurations independent from execution and evaluation. Mogil Bench represents dataset inputs as packs/fixtures and execution as adapters, then delegates subjective scoring to blind humans in BlindBench. It adds neither an LLM-as-judge nor a heavyweight eval framework.
+
+A broader framework review reinforced the same lightweight boundary:
+
+- [Harbor](https://github.com/harbor-framework/harbor) and its [Agent Trajectory Interchange Format (ATIF)](https://www.harborframework.com/docs/agents/trajectory-format) provide the strongest future shape for append-only agent/tool trajectories, linked observations, usage, errors, and artifact references. V1 does not take Harbor as a runtime dependency or claim ATIF compatibility; a projection is roadmap work.
+- [Inspect AI](https://inspect.aisi.org.uk/) supports a clean task/dataset/solver/scorer split, replayable logs, provider selection, and external agent bridges. Mogil Bench keeps the conceptual separation without coupling its native pack schema to Inspect.
+- [promptfoo](https://www.promptfoo.dev/docs/intro/) demonstrates declarative task × provider matrices and portable assertions. Mogil Bench adopts matrix semantics without requiring a Node runtime.
+- [SWE-bench](https://github.com/SWE-bench/SWE-bench) reinforces pinned fixtures, clean isolated verification, preserved logs, and gold/control validation. Its Docker/image machinery is intentionally outside v1.
+- [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) are useful for a future telemetry projection, but remain a developing telemetry vocabulary rather than a replay or human-review contract. Mogil Bench should pin any future convention version and keep prompts/tool bodies opt-in because they are sensitive.
+- [DeepEval](https://github.com/confident-ai/deepeval) and [OpenAI Evals](https://github.com/openai/evals) were not selected as foundations: judge-heavy/provider-centric assumptions add credentials and nondeterminism without improving the BlindBench human-review loop.
+
+These frameworks remain optional adapters or export targets. V1 keeps one small canonical pack/result model and no mandatory benchmark-platform dependency.
+
+Additional authoritative contract used: BlindBench's local `docs/native-ingest.md` (read from the sibling repository during implementation) defines `eval-record` version `"1"`, batching, deduplication IDs, privacy classes, endpoint constraints, and counts-only responses. Mogil Bench does not modify BlindBench.
+
+## Product decisions
+
+- Keep `hermes-text` and `pi-coding` as explicit lanes but normalize both to one artifact shape.
+- Ship fictional, credential-free fixtures and a deterministic mock for reproducible local use.
+- Keep the command adapter opt-in twice and visibly document that subprocess guardrails are not a hardened sandbox.
+- Export raw run data separately from reviewer artifacts; do not export verifier argv or expected values.
+- Make stable identity content-based and timestamps observational.
+- Omit tokens and cost when adapters lack authoritative provider usage; never substitute word counts or assumed zero cost.
+- Invoke Pi narrowly in ephemeral print mode with explicit provider/model identity and discovered resources disabled; capture final output now and defer JSON trajectory normalization.
+- Keep generic-command and Pi-agent opt-ins independent.
+- Keep upload optional, dry-run by default, endpoint-constrained, and token-from-environment only.

packs/command-smoke-v1.yaml

@@ -0,0 +1,29 @@
+version: "1"
+id: command-smoke-v1
+revision: "2026-07-11.1"
+name: Safe command adapter smoke pack
+description: Runs only a local Python argv command in an isolated temporary directory.
+allow_commands: true
+tasks:
+  - id: local-python-smoke
+    category: harness-smoke
+    lane: pi-coding
+    privacy_class: public
+    prompt: Execute the local deterministic smoke script and capture its result.
+    fixture: fixtures/smoke.py
+    timeout_seconds: 5
+    command: [python3, smoke.py]
+    verifier:
+      argv: [python3, smoke.py]
+      timeout_seconds: 5
+      expected_exit_code: 0
+      stdout_contains: SAFE_SMOKE_OK
+configurations:
+  - id: guarded-command
+    provider: local
+    model: local-command-smoke
+    adapter: command
+    harness:
+      name: mogil-bench-command
+      version: "0.1.0"
+      sdk: subprocess-argv

packs/fixtures/coding/calculator.py

@@ -0,0 +1,3 @@
+def clamp(value: int, lower: int, upper: int) -> int:
+    """Clamp value to the inclusive range, but this fixture contains a defect."""
+    return min(lower, max(value, upper))

packs/fixtures/coding/test_calculator.py

@@ -0,0 +1,5 @@
+from calculator import clamp
+
+
+def test_clamp_inside_range() -> None:
+    assert clamp(5, 0, 10) == 5

packs/fixtures/smoke.py

@@ -0,0 +1 @@
+print("SAFE_SMOKE_OK")

packs/fixtures/ticket.txt

@@ -0,0 +1,4 @@
+Customer: Example Books (fictional)
+Issue: The weekly inventory export shows duplicate rows after a timezone change.
+Impact: Reporting is delayed; no customer or payment data is involved.
+Request: Confirm investigation steps and provide a workaround.

packs/pi-template-v1.yaml

@@ -0,0 +1,29 @@
+version: "1"
+id: pi-local-template-v1
+revision: "2026-07-11.1"
+name: Trusted local Pi adapter template
+# TEMPLATE ONLY: choose a real provider/model/harness version before running.
+description: Explicit opt-in template for a one-shot Pi coding benchmark.
+allow_commands: false
+allow_agents: true
+tasks:
+  - id: fictional-clamp-fix
+    category: coding-fix
+    lane: pi-coding
+    privacy_class: public
+    fixture: fixtures/coding/calculator.py
+    prompt: Inspect calculator.py, fix the clamp defect in place, and report the change and verification.
+    timeout_seconds: 120
+    verifier:
+      argv: [python3, -c, "from calculator import clamp; assert clamp(5, 0, 10) == 5; assert clamp(-1, 0, 10) == 0; assert clamp(11, 0, 10) == 10"]
+      timeout_seconds: 10
+      expected_exit_code: 0
+configurations:
+  - id: replace-with-local-pi-config
+    provider: replace-me
+    model: replace-me
+    adapter: pi
+    harness:
+      name: pi-coding-agent
+      version: replace-with-local-pi-version
+      sdk: pi-print

packs/sample-v1.yaml

@@ -0,0 +1,30 @@
+version: "1"
+id: mogil-sample-v1
+revision: "2026-07-11.1"
+name: Mogil Bench representative sample
+description: Credential-free Hermes/text and Pi/coding examples.
+allow_commands: false
+tasks:
+  - id: support-summary
+    category: support-writing
+    lane: hermes-text
+    privacy_class: public
+    fixture: fixtures/ticket.txt
+    prompt: Summarize the fictional ticket in three bullets and draft a concise reply.
+    timeout_seconds: 10
+  - id: python-fix
+    category: coding-fix
+    lane: pi-coding
+    privacy_class: public
+    fixture: fixtures/coding
+    prompt: Diagnose the defect and describe a minimal patch plus behavioral verification.
+    timeout_seconds: 10
+configurations:
+  - id: deterministic-mock
+    provider: local
+    model: mogil-deterministic-mock-v1
+    adapter: mock
+    harness:
+      name: mogil-bench
+      version: "0.1.0"
+      sdk: mock