This repository is a lightweight experimental artifact accompanying Takahashi (2026). It is designed to show that workflow-aware oversight metrics and claim-margin reporting can materially change how small coding-agent experiments should be interpreted.
The purpose is not to produce a benchmark leaderboard. The purpose is to make several workflow-level quantities visible:
- raw task success,
- hazard detection,
- retries,
- oversight cost,
- escalation load,
- and claim status after transport and audit-distortion budgets.
The repo therefore evaluates workflows, not base models in isolation.
- Task family: 12 small self-contained Python repair tasks.
- Protocol: the same prompt structure, task set, and oversight logic for all compared actual runs.
- Conditions:
ai_onlyautomated_oversightselective_escalation
- Actual runs included in this report:
| Model | Results directory |
|---|---|
| gemma3:1b | ../results/ollama_gemma3_1b_20260312_180932 |
| gemma3:4b | ../results/ollama_gemma3_4b_20260312_184007 |
| gemini-2.5-flash | ../results/gemini_2_5_flash_20260312_190453 |
The repo represents oversight channels symmetrically. Tests, heuristic policy checks, and the costly human-like review channel all share the same abstraction: name, type, cost, delay, trigger rule, pass/fail/abstain behavior, and detectable hazard classes. Human-like review is not treated as a privileged oracle.
The tasks are intentionally narrow. They are small Python edits such as off-by-one fixes, input-validation repairs, parsing corrections, and boundary-condition changes. This repo does not claim that these tasks capture broad deployment readiness or general coding competence.
| Model | AI-only Success | Automated Success | Selective Success | Automated Utility | Selective Utility | Automated Claim Status |
|---|---|---|---|---|---|---|
| gemma3:1b | 0.0 | 0.0 | 0.0 | -0.531 | -0.768 | fail-closed |
| gemma3:4b | 0.333 | 0.417 | 0.417 | 0.087 | -0.1 | fail-closed |
| gemini-2.5-flash | 0.75 | 0.833 | 0.833 | 0.732 | 0.632 | fail-closed |
The reporting rule is deliberately conservative:
total claim margin = estimation error + transport budget + audit-distortion budget
- Estimation error comes from the observed finite-sample comparison.
- Transport budget reflects that these are small synthetic Python tasks rather than deployment contexts.
- Audit-distortion budget reflects that the evaluation setup is narrow and visible.
The repo therefore distinguishes raw observed differences from what can actually be claimed.
The cross-model pattern is clear in the logged data.
gemma3:1bis metrology-positive but performance-negative in this setup. It produces useful workflow measurements, but no final task successes in the logged actual run.gemma3:4bshows mixed results. It improves raw success and protocol compliance, but the automated-oversight gain is small and the costly review path still hurts utility.gemini-2.5-flashshows the strongest raw performance and the cleanest protocol compliance. It also shows a small net-positive automated-oversight utility signal, but the associated claim still remains fail-closed under the current margin protocol.- Selective escalation is not justified in this lightweight setup. Across the actual runs, it either matches automated oversight on success while costing more, or performs worse on workflow utility.
Raw success is only part of the story.
- Stronger models made cheap automated oversight somewhat more useful, especially when response-format failures dropped.
- Oversight changed retries, cost, and review load even when it did not produce stronger claim status.
- As model capability increased, the residual failures shifted away from protocol and visible/public failures toward hidden semantic misses.
| Model | AI-only Parse Errors | AI-only Hazard Count | Automated Hazard Detection | Selective Hazard Detection |
|---|---|---|---|---|
| gemma3:1b | 3 | 17 | 0.55 | 0.6 |
| gemma3:4b | 1 | 11 | 0.591 | 0.643 |
| gemini-2.5-flash | 0 | 3 | 0.0 | 0.0 |
This is the central metrological point of the repo: raw success alone is insufficient, and stronger models do not automatically make stronger deployment claims once workflow costs and margins are included.
- small synthetic Python task set
- one lightweight protocol
- limited sample size
- no broad deployment claim
- no provider ranking claim
- no stability claim from a single logged run per model
- current oversight stack mainly captures visible/public hazards, not all hidden semantic failures
- repeat the same protocol across multiple logged trials before making stronger stability claims
- expand task coverage without collapsing back into raw-score-only reporting
- improve hidden-semantic auditing without treating the costly review channel as an oracle
- keep transport and audit-distortion budgets explicit when changing task families