Skip to content

Commit c72243d

Browse files
committed
Switch public inference model to DeepSeek
Signed-off-by: Xinjie Yao <xyao@nvidia.com>
1 parent 1128ec4 commit c72243d

2 files changed

Lines changed: 19 additions & 7 deletions

File tree

docs/pages/concepts/agentic_environment_generation/model_selection.rst

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,16 +54,22 @@ resolution.
5454

5555
* - ``ARENA_INFERENCE_ENDPOINT``
5656
- Accessibility
57-
- Default model
57+
- Model
5858
- API key variable
5959
- Pass rate
6060
- Mean runtime
6161
* - ``public`` (default)
6262
- Public (free)
63-
- ``openai/gpt-oss-120b``
63+
- ``deepseek-ai/deepseek-v4-pro-0813``
6464
- ``NVIDIA_API_KEY``
65-
- 13/15 (86.7%)
66-
- 22.28 s
65+
- 15/15 (100%)
66+
- 150.66 s
67+
* - ``public`` (comparison)
68+
- Public (free)
69+
- ``nvidia/nemotron-3-ultra-550b-a55b``
70+
- ``NVIDIA_API_KEY``
71+
- 9/15 (60%)
72+
- 149.86 s
6773
* - ``internal``
6874
- NVIDIA internal
6975
- ``openai/openai/gpt-5.6-terra``
@@ -80,8 +86,14 @@ resolution.
8086
.. note::
8187
The benchmark ran each of five documented prompts three times. Pass rate is the fraction of
8288
generated specs that matched the expected structure; runtime is the mean end-to-end
83-
``generate_spec`` runtime. These results are snapshots rather than guarantees: model output is
84-
non-deterministic, and service load affects runtime.
89+
``generate_spec`` runtime. The two public models were measured at commit ``97c92a1f2``.
90+
These results are snapshots rather than guarantees: model output is non-deterministic, and
91+
service load affects runtime.
92+
93+
In this snapshot, DeepSeek produced the expected structure for all 15 runs. Nemotron produced the
94+
expected structure for all nine tabletop runs but failed all six kitchen runs, primarily by
95+
omitting background object references and spatial relations. Their mean runtimes were nearly the
96+
same, so DeepSeek is the public default based on spec quality rather than speed.
8597

8698
Reviewing the Generated Spec
8799
----------------------------

isaaclab_arena/agentic_environment_generation/inference_backend.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ class InferenceEndpoint:
5151
name="public",
5252
base_url="https://integrate.api.nvidia.com/v1",
5353
# If you cannot access the model in your region, try "nvidia/nemotron-3-super-120b-a12b".
54-
model="openai/gpt-oss-120b",
54+
model="deepseek-ai/deepseek-v4-pro-0813",
5555
api_key_env_var="NVIDIA_API_KEY",
5656
)
5757
"""Publicly reachable build.nvidia.com endpoint, reached with an NGC API key."""

0 commit comments

Comments
 (0)