Merge pull request #21 from mogilventures/feat/v0.1.0-publish-readiness

feat: prepare Mogil Bench v0.1.0 public alpha

Author: nmogil

.github/workflows/ci.yml

@@ -30,6 +30,27 @@ jobs:
           mogil-bench run packs/sample-v1.yaml --output-dir /tmp/mogil-bench-ci
           mogil-bench artifact validate /tmp/mogil-bench-ci/blindbench.json
 
+  distribution:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: python -m pip install --upgrade pip build
+      - run: python -m build
+      - name: Clean wheel install and credential-free CLI smoke
+        run: |
+          python -m venv /tmp/mogil-wheel-smoke
+          /tmp/mogil-wheel-smoke/bin/python -m pip install dist/*.whl
+          /tmp/mogil-wheel-smoke/bin/mogil-bench --help
+          /tmp/mogil-wheel-smoke/bin/mogil-bench pack validate packs/sample-v1.yaml
+          /tmp/mogil-wheel-smoke/bin/mogil-bench run packs/sample-v1.yaml \
+            --output-dir /tmp/mogil-wheel-sample
+          /tmp/mogil-wheel-smoke/bin/mogil-bench artifact validate \
+            /tmp/mogil-wheel-sample/blindbench.json
+
   docker-smoke:
     runs-on: ubuntu-latest
     steps:

CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - Unreleased
+
+Initial public-alpha release candidate.
+
+### Added
+
+- Versioned benchmark packs and deterministic local sample execution.
+- Harbor 0.18.0 Docker and optional Daytona 0.196.0 execution boundaries.
+- Pi 0.80.6 trajectory capture, strict evidence validation, checksummed run bundles, and reviewer-safe projections.
+- Guarded BlindBench legacy-artifact and strict-evidence upload commands.
+- Deterministic two-arm BlindBench paired-comparison CSV export.
+- Public contribution, security, release, and compatibility documentation.
+
+### Security
+
+- Deny-by-default command/agent execution, bounded evidence, strict identity/count reconciliation, secret-reference policies, and failure-safe artifact publication.
+
+[Unreleased]: https://github.com/mogilventures/mogil-bench/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/mogilventures/mogil-bench/releases/tag/v0.1.0

CONTRIBUTING.md

@@ -0,0 +1,34 @@
+# Contributing
+
+Mogil Bench is a public-alpha project. Small, reviewable changes that preserve its fail-closed evidence and execution boundaries are welcome.
+
+## Development setup
+
+Python 3.12 and Docker are required for the full suite.
+
+```bash
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install -e '.[dev]'
+```
+
+Before opening a pull request, run:
+
+```bash
+pytest -q -m 'not daytona_smoke'
+ruff check .
+mypy
+pytest -q -m docker_smoke tests/test_harbor_docker_smoke.py
+git diff --check
+```
+
+Tests must not require paid provider calls unless they use the existing explicit live gate. Add behavior-first tests for changes to validation, publication, uploads, or evidence handling. Do not add credentials, customer data, verifier canaries, generated run output, or mutable runtime references.
+
+## Pull requests
+
+- Link the issue and explain the observable behavior change.
+- Keep dependency and runtime-pin changes separate and explicitly approved.
+- Update user or release documentation when a command or operational boundary changes.
+- Do not weaken count reconciliation, identity checks, checksum checks, cleanup checks, or reviewer blinding to make a test pass.
+
+By contributing, you agree that your contribution is licensed under the repository's MIT license.

README.md

@@ -1,6 +1,19 @@
 # Mogil Bench
 
-Mogil Bench v1 is a local Python CLI for running versioned, real-work-like benchmark packs. It keeps Hermes/text and Pi/coding tasks distinct while exporting both as BlindBench `eval-record` v1 JSON and JSONL for blind human review.
+[![CI](https://github.com/mogilventures/mogil-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/mogilventures/mogil-bench/actions/workflows/ci.yml)
+[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![Status: public alpha](https://img.shields.io/badge/status-public_alpha-orange.svg)](CHANGELOG.md)
+
+Mogil Bench 0.1.0 is a **public alpha** CLI for reproducible coding-agent evaluation: versioned benchmark packs, Harbor execution, checksummed evidence, reviewer-safe BlindBench exports, and deterministic paired comparisons. Its core execution/evidence path is live-proven, but command and evidence contracts may still change before a stable release.
+
+## Status and boundaries
+
+- Supported: Python 3.12, `harbor==0.18.0`, optional `daytona==0.196.0`, and container Pi `0.80.6`.
+- Credential-free local sample and Docker fixture workflows are continuously tested.
+- Provider execution requires explicit pack and CLI gates; paid Daytona parity remains manually gated.
+- Local Docker is not VM-grade isolation. BlindBench campaign creation, training, PyPI publication, and backend/storage changes are outside this alpha.
+- Never use customer data or secrets in packs. See [`SECURITY.md`](SECURITY.md), [`CONTRIBUTING.md`](CONTRIBUTING.md), and the [release runbook](docs/release.md).
 
 ## Install
 
@@ -212,16 +225,38 @@ A completed run can be re-exported offline from its retained, checksummed per-at
 mogil-bench evidence re-export /tmp/mogil-daytona-provider-parity
 ```
 
-Create a **fresh BlindBench project** for a corrected re-export; do not upload it into a project containing rows from an earlier diagnostic import. Validate and dry-run the exact fresh-project uploads before adding `--confirm`:
+Turn that completed two-arm run into BlindBench's paired-comparison CSV without rerunning a provider. Candidate configuration IDs are selected explicitly on the command line; rows are ordered by immutable task ID and one-based attempt, and only a complete quality-eligible matrix with stable task revisions is accepted. For the shipped 3-task × 2-arm × 3-attempt profile this writes exactly nine cases. Re-exporting the same evidence produces identical bytes, and validation finishes before the destination is replaced.
+
+```bash
+mogil-bench export paired-comparison /tmp/mogil-daytona-provider-parity \
+  --candidate-a anthropic-direct \
+  --candidate-b openrouter-routed \
+  --output /tmp/mogil-daytona-provider-parity/paired-comparison.csv
+```
+
+`context`, `candidate_a`, and `candidate_b` come only from the blinded reviewer projection. The standard `candidate_a_model`, `candidate_b_model`, and harness columns retain owner-visible, provider-qualified route and harness provenance; Mogil emits no ignored custom owner columns and adds no arm labels to reviewer-visible text. Import the CSV through BlindBench's paired-comparison flow; campaign creation remains a manual BlindBench operation.
+
+A credential-free, fixture-backed reproduction of the nine-case contract is available from a repository checkout:
+
+```bash
+mogil-bench export paired-comparison tests/fixtures/completed-parity-run \
+  --candidate-a anthropic-direct \
+  --candidate-b openrouter-routed \
+  --output /tmp/mogil-fixture-comparison.csv
+python -c 'import csv; print(len(list(csv.DictReader(open("/tmp/mogil-fixture-comparison.csv")))))'
+# 9
+```
+
+Create a **fresh BlindBench project** for a corrected re-export; do not upload it into a project containing rows from an earlier diagnostic import. Validate and dry-run the exact fresh-project uploads before adding `--confirm`. The default upload timeout is 120 seconds (bounded to at most 600); set `--timeout 120` explicitly in recorded runbooks:
 
 ```bash
 mogil-bench evidence validate /tmp/mogil-daytona-provider-parity/mogil.harbor-evidence.jsonl
 mogil-bench artifact validate /tmp/mogil-daytona-provider-parity/blindbench.jsonl
 mogil-bench evidence upload /tmp/mogil-daytona-provider-parity/mogil.harbor-evidence.jsonl \
-  --endpoint https://BLINDBENCH_HOST/ingest/v1/eval-runs
+  --endpoint https://BLINDBENCH_HOST/ingest/v1/eval-runs --timeout 120
 BLINDBENCH_AUTOMATION_TOKEN='project-token' mogil-bench evidence upload \
   /tmp/mogil-daytona-provider-parity/mogil.harbor-evidence.jsonl \
-  --endpoint https://BLINDBENCH_HOST/ingest/v1/eval-runs --confirm
+  --endpoint https://BLINDBENCH_HOST/ingest/v1/eval-runs --timeout 120 --confirm
 ```
 
 The private envelope retains provider/model provenance and stable logical task/configuration identity. Its `reviewer` projection contains the shared task identity, blinded `isolated-sandbox` class, trajectory, objective outcomes, and bounded evidence—but no provider, model, vendor, configuration ID, credential, canary, absolute path, secret name, or secret value—so BlindBench can group same-task attempts without exposing the comparison arm to reviewers.
@@ -250,10 +285,10 @@ Upload is dry-run by default and validates both artifact and endpoint without ma
 
 ```bash
 mogil-bench artifact upload /tmp/mogil-sample-run/blindbench.json \
-  --endpoint https://DEPLOYMENT.convex.site/ingest/v1/traces
+  --endpoint https://DEPLOYMENT.convex.site/ingest/v1/traces --timeout 120
 ```
 
-A real upload additionally requires `BLINDBENCH_INGEST_TOKEN` and `--confirm`. Only HTTPS `*.convex.site/ingest/v1/traces` endpoints are accepted. The CLI never prints the token or record content and reports only response counts. It treats `invalid > 0`, `truncated: true`, a malformed counts response, or an imported-plus-deduped count that differs from the intended batch size as an upload failure. Tests make no network calls.
+A real upload additionally requires `BLINDBENCH_INGEST_TOKEN` and `--confirm`. Only HTTPS `*.convex.site/ingest/v1/traces` endpoints are accepted. Both upload paths default to a bounded 120-second timeout suitable for the canonical 18-record batch and accept `--timeout` values only through 600 seconds. A timeout means the outcome is unknown—the server may have completed—so Mogil never retries automatically. Check destination state, then resend the exact artifact if needed; deterministic IDs make that retry idempotent. HTTP errors report only status and a bounded, sanitized diagnostic. The CLI never prints the token or request content and reports only response counts. It treats `invalid > 0`, `truncated: true`, a malformed counts response, or an imported-plus-deduped count that differs from the intended batch size as an upload failure. Tests make no external network calls.
 
 Prompts and outputs are reviewer-visible and free text is not automatically scrubbed by legacy BlindBench exports. Never benchmark secrets or customer data; set `privacy_class` accurately. Hidden verifier expectations are not exported.
 
@@ -270,9 +305,9 @@ Upload is dry-run by default. The public endpoint must be HTTPS (HTTP is accepte
 
 ```bash
 mogil-bench evidence upload EVIDENCE.json \
-  --endpoint https://blindbench.example/ingest/v1/eval-runs
+  --endpoint https://blindbench.example/ingest/v1/eval-runs --timeout 120
 BLINDBENCH_AUTOMATION_TOKEN='project-token' mogil-bench evidence upload EVIDENCE.json \
-  --endpoint https://blindbench.example/ingest/v1/eval-runs --confirm
+  --endpoint https://blindbench.example/ingest/v1/eval-runs --timeout 120 --confirm
 ```
 
 The request body is a bounded batch of complete authoritative Pydantic artifacts, not reviewer projections or legacy trace records:
@@ -301,7 +336,7 @@ A successful consumer response uses exactly these completion counters (additiona
 }
 ```
 
-`complete` must equal the submitted `runs` count, `imported + deduped` must equal that same count, and `invalid` must be zero. A conflict or partial batch must not report a complete count. The token is a project Automation token and is never printed. Errors disclose only exception classes, never token, response body, prompts, or outputs.
+`complete` must equal the submitted `runs` count, `imported + deduped` must equal that same count, and `invalid` must be zero. A conflict or partial batch must not report a complete count. The token is a project Automation token and is never printed. HTTP errors disclose only status and a bounded, sanitized diagnostic—never authorization, request payloads, or unrestricted response bodies.
 
 ## Explicit deferrals
 

SECURITY.md

@@ -0,0 +1,15 @@
+# Security Policy
+
+## Supported versions
+
+Mogil Bench is a public alpha. Security fixes are applied to the latest code on `main`; no older release line is currently supported.
+
+## Reporting a vulnerability
+
+Do **not** open a public issue with vulnerability details, credentials, customer data, or benchmark evidence. Use the repository's **Security** tab to submit a private vulnerability report. If private reporting is unavailable, contact the repository owner, [@nmogil](https://github.com/nmogil), privately and ask for a secure reporting channel before sharing details.
+
+Include the affected version/commit, impact, minimal reproduction, and any suggested mitigation. Remove all live credentials and private evidence from the report. Maintainers will acknowledge receipt, coordinate validation and remediation privately, and publish an advisory when appropriate.
+
+## Operational boundaries
+
+Mogil Bench intentionally treats benchmark packs, provider credentials, retained trajectories, hidden verifiers, and upload destinations as sensitive boundaries. The local Docker backend is not a VM-grade isolation boundary. Follow the README's explicit execution gates and never place secret values in packs, command arguments, logs, fixtures, or repository files.

docs/release.md

@@ -0,0 +1,69 @@
+# v0.1.0 release runbook
+
+This is an operator checklist, not authorization to publish. A human maintainer must review the release candidate and explicitly approve tagging and the GitHub release. PyPI publication is deferred.
+
+## 1. Establish a clean release candidate
+
+- [ ] Confirm the implementation PR is reviewed, merged, and all required GitHub checks pass.
+- [ ] Start from an up-to-date, clean `main`; `git status --short` must be empty.
+- [ ] Confirm `pyproject.toml`, `CHANGELOG.md`, and the intended tag all say `0.1.0` / `v0.1.0`.
+- [ ] Confirm Python 3.12, Harbor 0.18.0, Daytona SDK 0.196.0, and container Pi 0.80.6 pins are unchanged.
+- [ ] Run `git diff --check`.
+
+## 2. Run the release gates
+
+From a fresh Python 3.12 environment:
+
+```bash
+python -m pip install -e '.[dev,daytona]'
+pytest -q -m 'not daytona_smoke'
+ruff check .
+mypy
+docker info
+pytest -q -m docker_smoke tests/test_harbor_docker_smoke.py
+```
+
+The Docker smoke must pass; a skip is not success. The paid/manual Daytona smoke is not a release requirement.
+
+## 3. Build and inspect distributions
+
+```bash
+rm -rf build dist
+python -m pip install build
+python -m build
+python -m zipfile --list dist/mogil_bench-0.1.0-py3-none-any.whl
+```
+
+- [ ] Both wheel and sdist exist.
+- [ ] The wheel includes `mogil_bench/py.typed` and no tests, credentials, run output, or customer data.
+- [ ] Package metadata and project URLs are accurate.
+
+Create a clean environment and exercise the installed wheel from the repository checkout:
+
+```bash
+python -m venv /tmp/mogil-v010-wheel
+/tmp/mogil-v010-wheel/bin/python -m pip install dist/*.whl
+/tmp/mogil-v010-wheel/bin/mogil-bench --help
+rm -rf /tmp/mogil-v010-sample
+/tmp/mogil-v010-wheel/bin/mogil-bench pack validate packs/sample-v1.yaml
+/tmp/mogil-v010-wheel/bin/mogil-bench run packs/sample-v1.yaml \
+  --output-dir /tmp/mogil-v010-sample
+/tmp/mogil-v010-wheel/bin/mogil-bench artifact validate \
+  /tmp/mogil-v010-sample/blindbench.json
+```
+
+## 4. Security and immutable-runtime review
+
+- [ ] Run the organization-approved secret scan over the checkout **and Git history**; investigate every finding without printing credential values.
+- [ ] Confirm fixtures and docs contain only fictional/public data.
+- [ ] Confirm `packs/daytona-provider-parity-v1.yaml` uses the reviewed immutable `image@sha256` reference.
+- [ ] Follow `docs/daytona-runtime-image.md` to verify that immutable runtime reference and its Python 3.12, `/bin/sh`, and Pi 0.80.6 contract. Do not rebuild or repoint it as part of this release.
+- [ ] Confirm the release commit is exactly the reviewed clean `main` commit.
+
+## 5. Publish after explicit approval
+
+- [ ] Create the annotated `v0.1.0` tag at the reviewed release commit and push it without rewriting history.
+- [ ] Create the GitHub release from that tag, mark it as a pre-release/public alpha, and use the `CHANGELOG.md` notes.
+- [ ] Attach the freshly built wheel and sdist and record their SHA-256 digests.
+- [ ] Recheck the tag, GitHub release assets, and immutable runtime reference from a fresh client.
+- [ ] Do not publish to PyPI until package-data policy and trusted publishing receive separate approval.

pyproject.toml

@@ -5,10 +5,23 @@ build-backend = "hatchling.build"
 [project]
 name = "mogil-bench"
 version = "0.1.0"
-description = "Safe local benchmark runner and BlindBench artifact exporter"
+description = "Safe Harbor-based coding evaluation and BlindBench evidence CLI"
 readme = "README.md"
 requires-python = ">=3.12,<3.13"
 license = {text = "MIT"}
+authors = [{name = "Mogil Ventures"}]
+maintainers = [{name = "Noah Mogil"}]
+keywords = ["benchmark", "coding-agents", "evaluation", "harbor"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.12",
+  "Typing :: Typed",
+]
 dependencies = [
   "harbor==0.18.0",
   "pydantic>=2.7,<3",
@@ -31,6 +44,12 @@ dev = [
 [project.scripts]
 mogil-bench = "mogil_bench.cli:app"
 
+[project.urls]
+Homepage = "https://github.com/mogilventures/mogil-bench"
+Repository = "https://github.com/mogilventures/mogil-bench"
+Issues = "https://github.com/mogilventures/mogil-bench/issues"
+Changelog = "https://github.com/mogilventures/mogil-bench/blob/main/CHANGELOG.md"
+
 [tool.hatch.build.targets.wheel]
 packages = ["src/mogil_bench"]