@@ -54,16 +54,22 @@ resolution.
5454
5555 * - ``ARENA_INFERENCE_ENDPOINT ``
5656 - Accessibility
57- - Default model
57+ - Model
5858 - API key variable
5959 - Pass rate
6060 - Mean runtime
6161 * - ``public `` (default)
6262 - Public (free)
63- - ``openai/gpt-oss-120b ``
63+ - ``deepseek-ai/deepseek-v4-pro-0813 ``
6464 - ``NVIDIA_API_KEY ``
65- - 13/15 (86.7%)
66- - 22.28 s
65+ - 15/15 (100%)
66+ - 150.66 s
67+ * - ``public `` (comparison)
68+ - Public (free)
69+ - ``nvidia/nemotron-3-ultra-550b-a55b ``
70+ - ``NVIDIA_API_KEY ``
71+ - 9/15 (60%)
72+ - 149.86 s
6773 * - ``internal ``
6874 - NVIDIA internal
6975 - ``openai/openai/gpt-5.6-terra ``
@@ -80,8 +86,14 @@ resolution.
8086.. note ::
8187 The benchmark ran each of five documented prompts three times. Pass rate is the fraction of
8288 generated specs that matched the expected structure; runtime is the mean end-to-end
83- ``generate_spec `` runtime. These results are snapshots rather than guarantees: model output is
84- non-deterministic, and service load affects runtime.
89+ ``generate_spec `` runtime. The two public models were measured at commit ``97c92a1f2 ``.
90+ These results are snapshots rather than guarantees: model output is non-deterministic, and
91+ service load affects runtime.
92+
93+ In this snapshot, DeepSeek produced the expected structure for all 15 runs. Nemotron produced the
94+ expected structure for all nine tabletop runs but failed all six kitchen runs, primarily by
95+ omitting background object references and spatial relations. Their mean runtimes were nearly the
96+ same, so DeepSeek is the public default based on spec quality rather than speed.
8597
8698Reviewing the Generated Spec
8799----------------------------
0 commit comments