Skip to content

Commit 6b9a7f4

Browse files
authored
Separate API and CLI execution routes (#55)
1 parent 52d88f8 commit 6b9a7f4

13 files changed

Lines changed: 166 additions & 72 deletions

File tree

‎.agents/skills/run-api-benchmark/SKILL.md‎

Lines changed: 28 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,19 @@ Do not dump every configuration question into one message.
3535

3636
3. Ask:
3737

38-
> Is this a local/test submission, or should the completed file be uploaded to the
39-
> official intake?
38+
> What should happen after the run?
39+
>
40+
> 1. Keep and validate the result locally without uploading.
41+
> 2. Upload an intake test that is scored privately but excluded from the leaderboard.
42+
> 3. Upload an official submission.
4043
41-
Default to local/test only when the caller explicitly delegates the choice. Do not ask
42-
for intake credentials unless upload is selected.
44+
Default to local-only only when the caller explicitly delegates the choice. Do not ask
45+
for intake credentials unless option 2 or 3 is selected.
4346

44-
4. Resolve `PR_REF` (default `v0.6.0`), `SUBMIT_LIMIT` (default 100), authoritative output
45-
path, and log path. Offer defaults and ask the caller whether to accept them rather than
46-
demanding paths one at a time.
47+
4. Resolve `PR_REF` (default `v0.6.0`), `SUBMIT_LIMIT` (default 100), and `STAMP` (default:
48+
the Makefile timestamp). Show the derived authoritative path
49+
`out/<stamp>/submission.json`; the trajectory is written alongside it. Do not ask for
50+
arbitrary host output or log paths when using `make run`.
4751

4852
## Configure safely
4953

@@ -55,6 +59,10 @@ guide the caller to set:
5559
- `API_BASE` and `MODEL_KWARGS` only when needed;
5660
- `SUBMIT_LIMIT` when non-default.
5761

62+
This route always uses the `mini-swe` backend. If an existing env file contains
63+
`AGENT_BACKEND`, remove it or require it to be `mini-swe`. Never select a coding-agent CLI
64+
with `AGENT_BACKEND` and never run one inside the container.
65+
5866
Do not expose secret values in command output or the final response. Do not add removed
5967
`AGENT_MODE`, `MAX_RULES`, or max-turn settings.
6068

@@ -63,24 +71,27 @@ Do not expose secret values in command output or the final response. Do not add
6371
1. Run `scripts/detect-engine.sh` and parse its `KEY=VALUE` output. Read
6472
[references/engines.md](references/engines.md) only if no engine is available or the RAM
6573
hint is low.
66-
2. Build with one consistent ref:
74+
2. Prepare one image at the selected ref. Prefer the published image:
6775

6876
```bash
69-
make runner-build PR_REF=v0.6.0
77+
make runner-pull PR_REF=v0.6.0
7078
```
7179

72-
For Podman, use the equivalent command from `references/engines.md`.
80+
Fall back to `make runner-build PR_REF=v0.6.0` only when the published image is
81+
unavailable or the caller explicitly wants a local build. For Podman, use the equivalent
82+
command from `references/engines.md`.
7383
3. Before the preflight's real API call, show the resolved model, backend `mini-swe`, API
74-
endpoint with secrets redacted, `PR_REF`, submit limit, output path, and log path. Ask for
75-
explicit confirmation.
84+
endpoint with secrets redacted, `PR_REF`, submit limit, `STAMP`, derived submission path,
85+
and upload goal. Ask for explicit confirmation.
7686
4. Run `make preflight`. It checks `pred`, rule sources, and one tiny LiteLLM call. Stop on
7787
any failure; never proceed to a full run after a failed preflight.
7888

7989
## Run and validate
8090

8191
After preflight passes, state that the full session can consume substantial time and API
8292
credits and ask for explicit confirmation to start it. Then run `make run` or the equivalent
83-
Podman command using the detector's `RUN_FLAGS`.
93+
Podman command using the detector's `RUN_FLAGS`. Pass `STAMP=<resolved-stamp>` when a fixed
94+
stamp was selected.
8495

8596
Confirm the authoritative `submission.json` exists. Always validate it:
8697

@@ -91,15 +102,15 @@ python -m benchmark.submit --predictions <submission.json> --dry-run
91102
Report `bugs_found`, `total_tokens_k`, submit attempts, any `run_error`, and absolute output
92103
and log paths. A `run_error` means partial salvage, not a clean zero-bug completion.
93104

94-
Upload only when the caller explicitly selected official intake and provided
105+
Upload only when the caller explicitly selected an intake upload and provided
95106
`PRB_SUBMIT_URL` plus `PRB_API_KEY` locally:
96107

97108
```bash
98-
python -m benchmark.submit --predictions <submission.json>
109+
python -m benchmark.submit --predictions <submission.json> # official
110+
python -m benchmark.submit --predictions <submission.json> --test # intake test
99111
```
100112

101-
Add `--test` only when the caller requests an end-to-end intake test excluded from the
102-
public board. Never upload merely because the run completed.
113+
Use `--test` only for option 2. Never upload merely because the run completed.
103114

104115
An exit code 137 means the engine needs more memory. Preserve partial outputs and read
105116
actual command errors before recommending changes.

‎.agents/skills/run-api-benchmark/references/engines.md‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,32 @@
11
# Getting a container engine, per platform
22

33
Read this when `detect-engine.sh` reports `ENGINE=none` (install one). The benchmark image
4-
compiles a Rust CLI, so the recurring failure everywhere is **build OOM (exit 137)** when the
5-
engine's Linux VM/host has < ~8GB RAM.
4+
is published on GHCR; pull it when available. A local build compiles a Rust CLI and can fail
5+
with **exit 137** when the engine's Linux VM/host has < ~8GB RAM.
6+
7+
## Prepare the image
8+
9+
With Docker or a Docker-compatible CLI:
10+
11+
```bash
12+
make runner-pull PR_REF=v0.6.0
13+
# Fallback only: make runner-build PR_REF=v0.6.0
14+
```
15+
16+
With rootless Podman, pull and tag the published amd64 image directly:
17+
18+
```bash
19+
podman pull --arch amd64 ghcr.io/codingthrust/problem-reductions-runner:v0.6.0
20+
podman tag ghcr.io/codingthrust/problem-reductions-runner:v0.6.0 \
21+
problem-reductions-runner:v0.6.0
22+
```
623

724
## macOS (Apple Silicon or Intel)
825

926
Any docker-compatible engine works; pick by situation:
1027

1128
- **Colima** — free, open source, scriptable. Best default when you don't know the licensing
12-
situation. **Its VM defaults to 2 CPU / 2GB RAM, which OOMs the Rust link** — always start it
13-
with more:
29+
situation. Its VM defaults to 2 CPU / 2GB RAM; allocate more for a local image build:
1430
```bash
1531
brew install colima docker
1632
colima start --cpu 4 --memory 8 --disk 100
@@ -39,8 +55,7 @@ Bind mounts + `--env-file` work natively and files come back owned by you (the V
3955
you (without it, files land as a high subuid you can't delete).
4056
- **`:z`** on the bind mount **only when SELinux is enforcing** (RHEL/Fedora) — relabels it so
4157
the container can write; omit on non-SELinux hosts.
42-
The Makefile hardcodes `docker`, so either `alias docker=podman` for the session or run the raw
43-
`podman build` / `podman run` commands.
58+
The Makefile hardcodes `docker`, so use the raw `podman` pull/build/run commands.
4459

4560
Build OOM: ensure the host (or its cgroup limit) has ≥8GB + swap; `CARGO_BUILD_JOBS=1` is already
4661
set in the Dockerfile.

‎.agents/skills/run-api-benchmark/references/provider-config.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
Use `submission.env.example` as the source of truth. Keep secrets in the gitignored
44
`submission.env`; never ask the caller to paste them into chat.
55

6+
This configuration is only for the containerized `mini-swe` API route. Do not select Codex,
7+
Claude Code, or another host CLI with `AGENT_BACKEND`.
8+
69
## Variables
710

811
| Variable | Purpose |

‎.agents/skills/run-benchmark/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,5 +27,9 @@ in plain text. Do not combine both routes in one run.
2727
- For **Model API**, invoke `$run-api-benchmark` and follow it completely.
2828
- For **Coding-agent CLI**, invoke `$run-cli-benchmark` and follow it completely.
2929

30+
The route also selects the runtime. Model API means mini-swe/LiteLLM in a container.
31+
Coding-agent CLI means an installed host process through `make run-local`; never place
32+
Codex, Claude Code, or another CLI harness inside the API container path.
33+
3034
The child skill owns all later questions, preflight, execution, validation, and optional
3135
upload. Do not duplicate those workflows here.

‎.agents/skills/run-cli-benchmark/SKILL.md‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -49,20 +49,23 @@ numbered stage at a time when its answer controls the next branch.
4949

5050
4. Ask:
5151

52-
> Is this a local/test submission, or should the completed file be uploaded to the
53-
> official intake?
52+
> What should happen after the run?
53+
>
54+
> 1. Keep and validate the result locally without uploading.
55+
> 2. Upload an intake test that is scored privately but excluded from the leaderboard.
56+
> 3. Upload an official submission.
5457
55-
Do not ask for intake credentials unless upload is selected.
58+
Do not ask for intake credentials unless option 2 or 3 is selected.
5659

5760
5. Resolve `PR_REF` (default `v0.6.0`), `SUBMIT_LIMIT` (default 100), and three separate,
5861
explicit paths: clone destination, authoritative submission JSON, and log directory.
5962
Offer concrete defaults and ask whether to accept them.
6063

6164
## Verify the selected harness
6265

63-
For Codex, require `codex` and a successful `codex login status`. For Claude Code, require
64-
`claude` and its authenticated login/token flow. For newly integrated agents, follow their
65-
adapter's documented auth probe.
66+
For Codex, require `codex` plus either a successful `codex login status` or a configured
67+
`OPENAI_API_KEY`. For Claude Code, require `claude` and its authenticated login/token flow.
68+
For newly integrated agents, follow their adapter's documented auth probe.
6669

6770
Also require:
6871

@@ -88,8 +91,13 @@ Before the first real model call, show:
8891

8992
State that the run can consume substantial time or credits, then get explicit confirmation.
9093

91-
Run through `make run-local` with the selected backend, or invoke
92-
`benchmark.run_submission` directly with the same explicit values. For example:
94+
This route runs the selected CLI directly on the host. Select it only with
95+
`LOCAL_BACKEND=<backend-id>`. Do not set `AGENT_BACKEND`, invoke `make run`/`make preflight`,
96+
or start Docker/Podman for a CLI run.
97+
98+
Run through `make run-local` with the selected backend. If invoking
99+
`benchmark.run_submission` directly, pass `--host-cli` with the same explicit values. For
100+
example:
93101

94102
```bash
95103
make run-local \
@@ -115,11 +123,12 @@ python -m benchmark.submit --predictions <submission.json> --dry-run
115123
Report `bugs_found`, `total_tokens_k`, submit attempts, any `run_error`, CLI warnings, and
116124
absolute output/log paths. Preserve partial results and logs on failure.
117125

118-
Upload only when the caller explicitly selected official intake and configured
126+
Upload only when the caller explicitly selected an intake upload and configured
119127
`PRB_SUBMIT_URL` plus `PRB_API_KEY` locally:
120128

121129
```bash
122-
python -m benchmark.submit --predictions <submission.json>
130+
python -m benchmark.submit --predictions <submission.json> # official
131+
python -m benchmark.submit --predictions <submission.json> --test # intake test
123132
```
124133

125-
Never upload merely because the run completed.
134+
Use `--test` only for option 2. Never upload merely because the run completed.

‎.agents/skills/run-cli-benchmark/references/cli-config.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
Derive support from `benchmark.run_submission.BACKENDS` and `_run_backend()` at runtime.
44
Documentation may lag code.
55

6+
Run these adapters on the host through `make run-local LOCAL_BACKEND=<backend-id>`.
7+
Do not set `AGENT_BACKEND` or use a container command to select a CLI harness.
8+
69
## Built-in adapters
710

811
| Backend ID | Harness | Authentication | Model syntax |

‎CONTRIBUTING.md‎

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -34,16 +34,13 @@ The runner image bundles the `pred` binary, the agent stack (mini-swe-agent + Li
3434
and the problem-reductions source pinned at `v0.6.0`. Any LiteLLM-routable provider key
3535
works.
3636

37-
The **target library version is not hardcoded** — it tracks the benchmark. The single knob
38-
is the `PR_REF` build arg (a tag or commit of problem-reductions); the image bakes the
39-
commit and `pred` version it actually built into itself, and the runner records/verifies
40-
against those. Bump `PR_REF` and rebuild for each benchmark round.
37+
The **target library version is not hardcoded** — it tracks the benchmark. `PR_REF` selects
38+
a tag or commit of problem-reductions; the image records that commit and its matching
39+
`pred`. Pull the published image for that ref, or build it locally when unavailable:
4140

4241
```bash
43-
# Build once (compiles pred from source — takes a few minutes).
44-
# --build-arg PR_REF=<tag-or-commit> selects the library version (default in the Dockerfile):
45-
docker build -f docker/Dockerfile --target runner \
46-
--build-arg PR_REF=v0.6.0 -t problem-reductions-runner:v0.6.0 .
42+
make runner-pull PR_REF=v0.6.0
43+
# Fallback: make runner-build PR_REF=v0.6.0
4744
```
4845

4946
### Configure and run
@@ -59,8 +56,8 @@ docker run --rm --env-file submission.env -v "$PWD/out:/out" \
5956
# → ./out/submission.json (or just: make run)
6057
```
6158

62-
`MODEL_NAME` and a provider API key are required for the mini-swe path. Everything else in
63-
the template has a sane default — uncomment only what you need. The knobs, by tier:
59+
`MODEL_NAME` and any credentials required by the selected provider are required for the
60+
mini-swe path. Everything else has a sane default — uncomment only what you need:
6461

6562
| Tier | Vars | When |
6663
|---|---|---|
@@ -148,6 +145,10 @@ sandbox; Claude uses `claude -p`. Both receive the same benchmark prompts and ag
148145
exactly one whole-repository session and stop themselves; no turn count is passed to either
149146
CLI.
150147

148+
The CLI backend always runs on the host through `make run-local` and is selected with
149+
`LOCAL_BACKEND`. Do not set `AGENT_BACKEND` or use Docker for Codex, Claude Code, or another
150+
coding-agent CLI.
151+
151152
The runner gives every backend a writable scratch workspace. The `submit` CLI exchanges
152153
atomic request/response files there while the attempt budget and verified ledger remain in
153154
runner memory, avoiding sandbox-blocked sockets or localhost networking. The prompt begins
@@ -162,7 +163,8 @@ mutated automatically. Submission JSON and live/final logs go to the separate, r
162163

163164
The CLI backend is intentionally less hermetic: it uses the host binary, authentication,
164165
Python environment, and `pred`. The runner still verifies the pinned `pred` version and
165-
records the target commit; use the API backend for an official reproducible batch.
166+
records the target commit; use the API backend when container-level reproducibility is
167+
required.
166168

167169
#### Add another coding-agent CLI
168170

@@ -174,8 +176,13 @@ must produce `verdict: reliable`.
174176

175177
## 2. Submit it (CLI upload)
176178

177-
Submission is a **CLI upload** — no web form, and the file never enters git. Get the
178-
endpoint URL + a token from the maintainer, then:
179+
Submission is a **CLI upload** — no web form, and the file never enters git. Choose one:
180+
181+
- validate and keep the result locally with `--dry-run`;
182+
- upload an intake test with `--test` (privately scored, excluded from the leaderboard);
183+
- upload an official submission with neither flag.
184+
185+
For either upload, get the endpoint URL and token from the maintainer, then:
179186

180187
```bash
181188
export PRB_SUBMIT_URL=<intake endpoint> # from the maintainer

‎Makefile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ preflight:
7474
## whole-repo trajectory lands alongside it). Each run gets its own timestamped dir so
7575
## successive runs never overwrite each other (override with STAMP=... to reproduce).
7676
## This RUNS the benchmark locally; it does NOT submit — submitting is a separate step
77-
## (open a GitHub PR adding the file, see CONTRIBUTING.md). Config lives in submission.env
77+
## (upload it with benchmark.submit, see CONTRIBUTING.md). Config lives in submission.env
7878
## (copy submission.env.example); run `make preflight` first to validate it.
7979
run:
8080
@if [ ! -f "$(ENV_FILE)" ]; then \
@@ -97,7 +97,7 @@ run-local:
9797
--repo-ref "$(PR_REF)" \
9898
--repo-url "$(REPO_URL)" \
9999
--output "$(abspath $(LOCAL_OUTPUT))" \
100-
--trajectory-dir "$(abspath $(LOCAL_LOG_DIR))" $(LOCAL_ARGS)
100+
--trajectory-dir "$(abspath $(LOCAL_LOG_DIR))" --host-cli $(LOCAL_ARGS)
101101
@echo "Wrote $(LOCAL_OUTPUT); logs are in $(LOCAL_LOG_DIR)."
102102

103103
## Score all submissions in SUBS_DIR with the zero-trust backend (needs pred). Writes scored
@@ -145,7 +145,7 @@ help:
145145
@echo " runner-build Build the dockerized submission runner image"
146146
@echo " runner-pull Pull the prebuilt runner image from GHCR (fast runner-build alternative)"
147147
@echo " preflight Validate submission.env (1 tiny real call) before a full run"
148-
@echo " run Run the benchmark via Docker → out/submission.json (not upload)"
148+
@echo " run Run the API backend via Docker → out/<stamp>/submission.json (not upload)"
149149
@echo " run-local Clone/verify a repo and run a local headless CLI agent"
150150
@echo " score-local Score SUBS_DIR submissions with the backend"
151151
@echo " board Build site/results.json from site/results/*.json (aggregate)"

‎README.md‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -74,10 +74,8 @@ printed only in the agent's final response do not count.
7474

7575
## How to run
7676

77-
Both backends require:
78-
79-
- `pred` binary in PATH (pinned commit `aa2d1a1` of problem-reductions)
80-
- Python 3.12 with dependencies: `pip install -r benchmark/requirements.txt`
77+
The API image contains `pred`, Python, dependencies, and the target source. The CLI backend
78+
instead uses those tools from the host.
8179

8280
```bash
8381
# Run all unit tests (no API key needed) — this exercises the backend wiring
@@ -89,8 +87,8 @@ make verify-calibration
8987

9088
### Model API backend
9189

92-
Configure a provider key in `submission.env`, then run the containerized mini-swe/LiteLLM
93-
backend:
90+
Install Docker, configure the model API in `submission.env`, then run mini-swe/LiteLLM in
91+
the container:
9492

9593
```bash
9694
cp submission.env.example submission.env
@@ -101,8 +99,8 @@ make run
10199

102100
### Coding-agent CLI backend
103101

104-
Install and authenticate a supported CLI, set `MODEL_NAME` in `submission.env`, and run it
105-
directly on the host:
102+
Install Python 3.12, the benchmark dependencies, the pinned `pred`, and a supported CLI.
103+
Authenticate the CLI, set `MODEL_NAME` in `submission.env`, and run it directly on the host:
106104

107105
```bash
108106
cp submission.env.example submission.env
@@ -129,7 +127,7 @@ Key `make` targets:
129127
| `make verify-calibration` | Test verifier against the fixtures (accept + reject paths) |
130128
| `make verify-judgment` | Pred-free sanity tests (docs, CI, trajectory) |
131129
| `make preflight` | Validate the API backend with one tiny real call before a full run |
132-
| `make run` | Run the API backend in Docker → `out/submission.json` (does not upload) |
130+
| `make run` | Run the API backend in Docker → `out/<stamp>/submission.json` (does not upload) |
133131
| `make run-local` | Run a coding-agent CLI on the host → the same output schema |
134132
| `make score-local` | Score submissions with the zero-trust backend |
135133

‎benchmark/run_submission.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,10 +330,20 @@ def main(argv: list[str] | None = None) -> None:
330330
"provider (default). claude-code: headless `claude -p`; auth via "
331331
"Claude login/token. codex: headless `codex exec`; auth via Codex "
332332
"login or OPENAI_API_KEY.")
333+
parser.add_argument("--host-cli", action="store_true",
334+
help="Confirm a coding-agent backend is running directly on the host "
335+
"(set by make run-local; never use in the API container route)")
333336
args = parser.parse_args(argv)
334337

335338
if not args.model:
336339
parser.error("--model (or env MODEL_NAME) is required")
340+
if args.backend != "mini-swe" and not args.host_cli:
341+
parser.error(
342+
"container/API runs require backend 'mini-swe'; run coding-agent CLIs on "
343+
"the host with `make run-local LOCAL_BACKEND=<backend-id>`"
344+
)
345+
if args.host_cli and args.backend == "mini-swe":
346+
parser.error("--host-cli requires a coding-agent backend, not 'mini-swe'")
337347

338348
library_commit = None
339349
if args.repo_ref:

0 commit comments

Comments
 (0)