Skip to content

feat: prepare Mogil Bench v0.1.0 public alpha - #21

Merged
nmogil merged 1 commit into
mainfrom
feat/v0.1.0-publish-readiness
Jul 15, 2026
Merged

feat: prepare Mogil Bench v0.1.0 public alpha#21
nmogil merged 1 commit into
mainfrom
feat/v0.1.0-publish-readiness

Conversation

@nmogil

@nmogil nmogil commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Prepare Mogil Bench for a deliberate v0.1.0 public-alpha release by closing the reproducibility gap between completed provider-parity runs and BlindBench comparisons, hardening ambiguous upload failures, and adding release/package hygiene.

Closes #18
Closes #19
Advances #20

Parent epic: #9

What changed

Deterministic paired comparisons

  • add mogil-bench export paired-comparison
  • pair a completed two-arm run by immutable task identity and attempt number
  • require complete, contiguous, quality-eligible cells with stable task revision
  • emit only BlindBench's supported 12-column paired CSV contract
  • preserve provider-qualified model route and human-readable harness metadata in owner provenance fields
  • derive reviewer-visible context/output only from the reviewer projection
  • publish the destination through validated, rollback-safe replacement
  • add a checked-in 3×2×3 fixture whose deterministic export contains nine cases

Timeout-aware uploads

  • replace the hard-coded 30-second timeout with a bounded --timeout option
  • default to 120 seconds and reject non-finite, non-positive, or values above 600 seconds
  • report timeout outcomes as unknown instead of definitive failure
  • advise checking destination state before an idempotent exact-artifact retry
  • provide bounded, sanitized HTTP status diagnostics without authorization, request payload, path, or unrestricted response disclosure
  • preserve strict imported/deduped/invalid reconciliation and no automatic retry

Public-alpha release readiness

  • add package metadata, project URLs, Python/typing classifiers, and public-alpha status
  • add CHANGELOG.md, CONTRIBUTING.md, SECURITY.md, and docs/release.md
  • update README with status, boundaries, paired-export, and upload-recovery guidance
  • add a CI distribution job that builds wheel/sdist, clean-installs the wheel, runs --help, and executes the credential-free sample flow
  • preserve Python 3.12, Harbor 0.18.0, Daytona SDK 0.196.0, and container Pi 0.80.6 pins

Verification

  • pytest -q -m 'not daytona_smoke'140 passed, 1 deselected; mandatory Docker smoke passed; one upstream Harbor deprecation warning
  • ruff check . — passed
  • mypy — passed, 17 source files
  • git diff --check — passed
  • python -m build — built mogil_bench-0.1.0.tar.gz and mogil_bench-0.1.0-py3-none-any.whl
  • clean wheel install:
    • mogil-bench --help passed
    • credential-free sample produced and validated 2 records
    • paired fixture export matched expected bytes and contained 9 rows
  • real completed parity bundles exported to 9 rows with no reviewer-field provider/model/vendor/credential/canary/path leakage
  • BlindBench's current TypeScript consumer parsed both fixture and real CSV as 9 valid / 0 invalid, with exact route models:
    • anthropic/claude-sonnet-4-6
    • openrouter/anthropic/claude-sonnet-4.6
  • current checkout and Git history credential-signature scan: 0 Anthropic/OpenRouter/private-key/GitHub-token matches

Agent implementation and review

  • Core implementation: Herdr-managed Pi 0.80.7, explicitly pinned to openai-codex/gpt-5.6-sol with high thinking
  • Independent no-edit review: separate Herdr-managed Pi lane explicitly pinned to anthropic/claude-opus-4-5 with high thinking
  • Opus verdict: ready; no P0 or P1 findings
  • One supervisor correction round was applied before Opus review to use only supported CSV columns, provider-qualified route provenance, and immutable task-revision validation

Explicitly deferred / post-merge

This PR does not tag, publish a GitHub release, publish to PyPI, change repository settings, run paid models, alter BlindBench backend/UI, or perform training/export. After review and merge, issue #20's repository-settings/release checklist remains:

  • correct the GitHub description and add topics
  • protect main while keeping the solo/agent PR workflow viable
  • date the 0.1.0 changelog entry
  • create the annotated v0.1.0 tag and public-alpha GitHub release
  • continue deferring PyPI until package-data policy and trusted publishing are approved

@nmogil
nmogil merged commit 4aadcd2 into main Jul 15, 2026
3 checks passed
@nmogil
nmogil deleted the feat/v0.1.0-publish-readiness branch July 15, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make BlindBench uploads timeout-aware and diagnostically safe Export deterministic BlindBench paired comparisons from parity runs

1 participant