Skip to content

feat: add Daytona provider parity matrix - #15

Merged
nmogil merged 2 commits into
mainfrom
feat-daytona-provider-parity
Jul 12, 2026
Merged

feat: add Daytona provider parity matrix#15
nmogil merged 2 commits into
mainfrom
feat-daytona-provider-parity

Conversation

@nmogil

@nmogil nmogil commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a reproducible Harbor-managed Daytona provider-parity benchmark that compares Anthropic direct with OpenRouter routing to the same Claude Sonnet 4.6 model.

  • Adds a fixed public/synthetic matrix: 3 coding tasks × 2 provider paths × 3 attempts = 18 attempts.
  • Adds bounded --attempts support for Harbor-only runs with stable logical IDs and distinct deterministic attempt IDs.
  • Adds a manually gated run-daytona-parity command with reviewed pack fingerprint, runtime digest, policy, secret-inventory, output-count, evidence, blinding, and credential-retention checks.
  • Pins provider-specific secret references and exact network allowlists.
  • Preserves private provider/model provenance while rejecting it from reviewer-safe projections.
  • Documents secure Daytona secret creation and BlindBench upload workflow.

Provider boundary

Arm Pi provider/model Allowed host Daytona secret
Anthropic direct anthropic/claude-sonnet-4-6 api.anthropic.com mogil-anthropic-smoke
OpenRouter openrouter/anthropic/claude-sonnet-4.6 openrouter.ai mogil-openrouter-parity

Both model identifiers were verified against the immutable Pi 0.80.6 runtime catalog. OpenRouter remains an additional routing layer; this PR does not claim identical serving infrastructure.

Verification

Supervisor-run from the final diff:

  • pytest -q tests/test_daytona_parity.py tests/test_harbor_backend.py tests/test_pi_evidence.py — 47 passed
  • pytest -q -m 'not daytona_smoke' — 115 passed, 1 paid-live deselected, 1 upstream Harbor deprecation warning
  • ruff check . — passed
  • mypy — passed, 15 source files
  • Real Harbor Docker smoke included in the non-live suite — passed
  • git diff --check — passed
  • Daytona secret metadata — both required names present with exactly one expected host restriction; values were never read
  • No paid/live provider matrix was run

Safety

  • Public fictional tasks only.
  • No host-Pi or mock fallback.
  • Agent and verifier sandboxes remain separate.
  • Provider-effective policy and cleanup remain fail-closed.
  • No credential values are stored in the pack, repository, shell history, .harbor.env, or retained artifacts.
  • No BlindBench, training, Fireworks, or customer-data changes.

@nmogil
nmogil merged commit 143c940 into main Jul 12, 2026
2 checks passed
@nmogil
nmogil deleted the feat-daytona-provider-parity branch July 12, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant