@@ -35,15 +35,19 @@ Do not dump every configuration question into one message.
3535
36363 . Ask:
3737
38- > Is this a local/test submission, or should the completed file be uploaded to the
39- > official intake?
38+ > What should happen after the run?
39+ >
40+ > 1 . Keep and validate the result locally without uploading.
41+ > 2 . Upload an intake test that is scored privately but excluded from the leaderboard.
42+ > 3 . Upload an official submission.
4043
41- Default to local/test only when the caller explicitly delegates the choice. Do not ask
42- for intake credentials unless upload is selected.
44+ Default to local-only only when the caller explicitly delegates the choice. Do not ask
45+ for intake credentials unless option 2 or 3 is selected.
4346
44- 4 . Resolve ` PR_REF ` (default ` v0.6.0 ` ), ` SUBMIT_LIMIT ` (default 100), authoritative output
45- path, and log path. Offer defaults and ask the caller whether to accept them rather than
46- demanding paths one at a time.
47+ 4 . Resolve ` PR_REF ` (default ` v0.6.0 ` ), ` SUBMIT_LIMIT ` (default 100), and ` STAMP ` (default:
48+ the Makefile timestamp). Show the derived authoritative path
49+ ` out/<stamp>/submission.json ` ; the trajectory is written alongside it. Do not ask for
50+ arbitrary host output or log paths when using ` make run ` .
4751
4852## Configure safely
4953
@@ -55,6 +59,10 @@ guide the caller to set:
5559- ` API_BASE ` and ` MODEL_KWARGS ` only when needed;
5660- ` SUBMIT_LIMIT ` when non-default.
5761
62+ This route always uses the ` mini-swe ` backend. If an existing env file contains
63+ ` AGENT_BACKEND ` , remove it or require it to be ` mini-swe ` . Never select a coding-agent CLI
64+ with ` AGENT_BACKEND ` and never run one inside the container.
65+
5866Do not expose secret values in command output or the final response. Do not add removed
5967` AGENT_MODE ` , ` MAX_RULES ` , or max-turn settings.
6068
@@ -63,24 +71,27 @@ Do not expose secret values in command output or the final response. Do not add
63711 . Run ` scripts/detect-engine.sh ` and parse its ` KEY=VALUE ` output. Read
6472 [ references/engines.md] ( references/engines.md ) only if no engine is available or the RAM
6573 hint is low.
66- 2 . Build with one consistent ref:
74+ 2 . Prepare one image at the selected ref. Prefer the published image :
6775
6876 ``` bash
69- make runner-build PR_REF=v0.6.0
77+ make runner-pull PR_REF=v0.6.0
7078 ```
7179
72- For Podman, use the equivalent command from ` references/engines.md ` .
80+ Fall back to ` make runner-build PR_REF=v0.6.0 ` only when the published image is
81+ unavailable or the caller explicitly wants a local build. For Podman, use the equivalent
82+ command from ` references/engines.md ` .
73833 . Before the preflight's real API call, show the resolved model, backend ` mini-swe ` , API
74- endpoint with secrets redacted, ` PR_REF ` , submit limit, output path, and log path. Ask for
75- explicit confirmation.
84+ endpoint with secrets redacted, ` PR_REF ` , submit limit, ` STAMP ` , derived submission path,
85+ and upload goal. Ask for explicit confirmation.
76864 . Run ` make preflight ` . It checks ` pred ` , rule sources, and one tiny LiteLLM call. Stop on
7787 any failure; never proceed to a full run after a failed preflight.
7888
7989## Run and validate
8090
8191After preflight passes, state that the full session can consume substantial time and API
8292credits and ask for explicit confirmation to start it. Then run ` make run ` or the equivalent
83- Podman command using the detector's ` RUN_FLAGS ` .
93+ Podman command using the detector's ` RUN_FLAGS ` . Pass ` STAMP=<resolved-stamp> ` when a fixed
94+ stamp was selected.
8495
8596Confirm the authoritative ` submission.json ` exists. Always validate it:
8697
@@ -91,15 +102,15 @@ python -m benchmark.submit --predictions <submission.json> --dry-run
91102Report ` bugs_found ` , ` total_tokens_k ` , submit attempts, any ` run_error ` , and absolute output
92103and log paths. A ` run_error ` means partial salvage, not a clean zero-bug completion.
93104
94- Upload only when the caller explicitly selected official intake and provided
105+ Upload only when the caller explicitly selected an intake upload and provided
95106` PRB_SUBMIT_URL ` plus ` PRB_API_KEY ` locally:
96107
97108``` bash
98- python -m benchmark.submit --predictions < submission.json>
109+ python -m benchmark.submit --predictions < submission.json> # official
110+ python -m benchmark.submit --predictions < submission.json> --test # intake test
99111```
100112
101- Add ` --test ` only when the caller requests an end-to-end intake test excluded from the
102- public board. Never upload merely because the run completed.
113+ Use ` --test ` only for option 2. Never upload merely because the run completed.
103114
104115An exit code 137 means the engine needs more memory. Preserve partial outputs and read
105116actual command errors before recommending changes.
0 commit comments