Mogil Bench v1: local benchmarks to BlindBench artifacts

[nmogil]

Summary

Builds Mogil Bench v1: a private local benchmark harness that runs versioned real-work-like packs and emits BlindBench-native eval-record v1 JSON/JSONL for blind human review.

Included

• typed Python CLI and strict YAML pack validation
• content-derived pack fingerprints and idempotent result IDs
• credential-free deterministic mock adapter
• guarded local command adapter with separate pack/operator opt-ins
• explicit local Pi adapter driven by configured provider/model, with reduced context/tool surface and separate --allow-agents gate
• isolated fixture copies, timeout, bounded output, sanitized environments, and partial-failure capture
• per-run manifest, raw result artifacts, BlindBench JSON batch, and JSONL
• guarded optional BlindBench upload: dry-run default, exact .convex.site endpoint validation, token from environment
• fictional sample, command smoke pack, and reviewed-copy-only Pi template
• research and architecture notes informed by Databricks' internal benchmark, mini-swe-agent, Vellum evals, and BlindBench's native ingest contract
• Python 3.11/3.12 GitHub Actions matrix

Safety boundary

This is not an OS/network sandbox. The shipped samples are fictional and safe, command and agent execution are deny-by-default, and the README requires trusted packs. V1 does not run live email/social/deploy actions, customer data, automatic fine-tuning, or an LLM judge.

The Pi adapter captures final stdout/status/duration. Full Pi tool-call trajectory normalization and authoritative usage/cost are follow-up work; v1 deliberately omits fabricated token/cost numbers.

Verification

Executed locally from the branch:

• .venv/bin/pytest -q → 14 passed
• .venv/bin/ruff check . → All checks passed!
• .venv/bin/mypy → Success: no issues found in 7 source files
• fresh temporary venv install + same checks → passed
• credential-free mock run → 2 succeeded results / 2 valid BlindBench records
• guarded command run → 1 succeeded result / 1 valid BlindBench record
• fake/no-network Pi tests verify opt-in, provider/model argv identity, and executable guardrails
• git diff --check → clean

A real paid Pi/provider call and confirmed BlindBench upload were intentionally not executed.

Durable notes

Obsidian: /root/obsidian/99_Hermes/Product/Mogil Bench/

Closes #1
Closes #2
Closes #3
Closes #4