I evaluated allenai/OLMo-2-0425-1B on all 1,319 examples in the GSM8K test set. I used seed 0, temperature 1.0, top-p 1.0, and a maximum generation length of 512 tokens. For r1_zero and r1_zero_three_shot, generation stopped at </answer> and included the stop string in the output; question_only did not use this stop string. Ground-truth answers were extracted by splitting each GSM8K reference response on ####. I used question_only_reward_fn for question_only and r1_zero_reward_fn for the other two prompts.
| Prompt | (1) Format 1, correctness 1 | (2) Format 1, correctness 0 | (3) Format 0, correctness 0 | Format rate | Accuracy |
|---|---|---|---|---|---|
question_only |
0 | 222 | 1,097 | 16.83% | 0.00% |
r1_zero |
1 | 712 | 606 | 54.06% | 0.08% |
r1_zero_three_shot |
258 | 989 | 72 | 94.54% | 19.56% |
The counts in every row sum to 1,319. The three-shot prompt produces a very large improvement: compared with zero-shot r1_zero, exact-format compliance rises from 54.06% to 94.54%, while accuracy rises from 0.08% to 19.56%. question_only performs worst because this is a base language model rather than an instruction-tuned model; merely appending an instruction to put the answer in a box usually does not make it answer the supplied problem.
To make the inspection reproducible and avoid cherry-picking, I examined the first 10 category-2 outputs and the first 10 category-3 outputs in test-set order for each prompt (30 outputs per category, 60 outputs total).
| Prompt | Actually correct among first 10 category 2 | Actually correct among first 10 category 3 |
|---|---|---|
question_only |
0/10 | 0/10 |
r1_zero |
1/10 | 1/10 |
r1_zero_three_shot |
0/10 | 0/10 |
| Total audited | 1/30 | 1/30 |
The category-2 false negative was r1_zero test index 0. The model correctly computed that Janet sells 9 eggs for $2 each and therefore makes $18, but it put a full prose explanation inside <answer> ... </answer> instead of an isolated answer. The strict parser treated the whole prose span as the candidate answer and did not match it to 18:
<answer> Janet has 16 - 3 ... - 4 ... = 9 fresh duck eggs daily. She sells ... for $2 each, so she makes ... $18 ... </answer>
The category-3 false negative was r1_zero test index 28. It correctly produced 60 - 20 - 15 = 25 miles inside <answer> ... </answer>, but it omitted the required preceding </think> marker, so the exact format test failed. These observations show that zero reward is usually a genuine reasoning or task-following failure in this run, although occasional correct solutions are lost to strict parsing. There are also correct-but-unformatted examples outside the fixed audit; for example, question_only index 33 derives 70 gold coins correctly but never emits \boxed{70}.
For question_only, the effective prompt for the sprint problem was:
James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week? Please put your final answer within
\boxed{}.
The model did not solve it and instead continued with generic answer-page text:
Write your answer below to the answer above
It sometimes follows the surface format while ignoring the problem. For the ice-cream problem whose correct answer is 16, it generated:
The final answer is \boxed{560}.
With zero-shot r1_zero, the prompt supplies the User/Assistant transcript and begins the assistant's <think> section. The run's only officially correct response was for a weekly running-time problem:
... 5 days times 12 miles equals 60 ... </think> <answer> 6 hours. </answer>
With r1_zero_three_shot, the three demonstrations make the intended computation and response schema much clearer. For the robe problem, the model generated:
... blue needs 2 bolts, then white needs half of that, which is 1 bolt. ... 2 + 1 = 3. </think> <answer> 3 </answer>
The demonstrations do not guarantee correct arithmetic. On the Janet problem, the same prompt produced a cleanly formatted but incorrect answer because it said 3 + 4 = 6, leaving 10 eggs and returning 20.
question_only is not enough to reliably elicit question answering from this base model. The model often behaves like a document-completion model: it invents additional questions, emits fragments of webpages or teaching instructions, starts a multi-turn User/Assistant conversation, or simply continues with unrelated prose. It reached the 512-token limit on 477 of 1,319 examples, and only 222 responses contained any parseable \boxed{} answer. One striking counterexample is the gold-coins problem: the model constructs a long simulated dialogue that correctly solves for 70, but because it is continuing a conversation rather than obeying the requested output contract, the answer is never boxed and receives zero reward.
The zero-shot r1_zero wrapper substantially changes the continuation distribution. By presenting an explicit conversation format and pre-filling <think>, it makes the model produce reasoning followed by an <answer> section much more often; 1,287 of 1,319 generations reached the </answer> stop string. However, only 54.06% matched the grader's exact format, and the model often placed long prose in the answer span or performed fluent but invalid arithmetic. Thus, the schema alone improves behavioral structure much more than mathematical correctness.
The three-shot prompt shapes both task recognition and serialization. The worked GSM8K examples teach the model to perform a short word-problem derivation, close </think>, isolate a concise final answer, and close </answer>. This is reflected in 94.54% format compliance and 19.56% accuracy, versus 54.06% and 0.08% for zero-shot r1_zero. Its successful robe response closely imitates the demonstrations' pattern of verbal calculation followed by a bare integer in <answer>. Remaining failures are mostly genuine reasoning errors rather than wholesale task drift; for example, the response to the Janet problem preserves the demonstrated format perfectly but miscomputes the number of eggs consumed.
The local evaluator is scripts/prompting_baselines.py. From an environment with vLLM and the project dependencies installed, run:
python scripts/prompting_baselines.py \
--output-dir results/prompting_baselinesThe original raw generations, summary metrics, and fixed review samples were saved locally in results/prompting_baselines/ and are omitted from the GitHub publication. Rerunning the evaluator writes a new set of artifacts to the selected output directory; use a new path when preserving an existing run. Stochastic generation and environment differences can change the regenerated results.