Skip to content

Commit 3bc397f

Browse files
authored
Switch public inference default model to DeepSeek in Agentic Env Gen (#1181)
## Summary Switch public inference model to DeepSeek ## Detailed description - gpt-oss-120b has been deprecated today without prior notice, caught by SQA regression tests - Switched to deepseek-v4-pro as it has been tested on all 5 example workflows - The inference call time increases to 100+sec from 10+sec, but given the quality outperforms than nemotran, the cost is acceptable. - Updated doc to track perf --------- Signed-off-by: Xinjie Yao <xyao@nvidia.com>
1 parent 0b63649 commit 3bc397f

2 files changed

Lines changed: 8 additions & 6 deletions

File tree

docs/pages/concepts/agentic_environment_generation/model_selection.rst

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,10 @@ resolution.
6060
- Mean runtime
6161
* - ``public`` (default)
6262
- Public (free)
63-
- ``openai/gpt-oss-120b``
63+
- ``deepseek-ai/deepseek-v4-pro-0813``
6464
- ``NVIDIA_API_KEY``
65-
- 13/15 (86.7%)
66-
- 22.28 s
65+
- 15/15 (100%)
66+
- 150.66 s
6767
* - ``internal``
6868
- NVIDIA internal
6969
- ``openai/openai/gpt-5.6-terra``
@@ -81,7 +81,10 @@ resolution.
8181
The benchmark ran each of five documented prompts three times. Pass rate is the fraction of
8282
generated specs that matched the expected structure; runtime is the mean end-to-end
8383
``generate_spec`` runtime. These results are snapshots rather than guarantees: model output is
84-
non-deterministic, and service load affects runtime.
84+
non-deterministic, and service load affects runtime. On the public endpoint, overriding the
85+
default with ``--model nvidia/nemotron-3-ultra-550b-a55b`` passed 9/15 runs (60%) with a
86+
149.86-second mean runtime; ``--model openai/gpt-oss-20b`` passed 7/15 runs (46.7%) with a
87+
333.17-second mean runtime.
8588

8689
Reviewing the Generated Spec
8790
----------------------------

isaaclab_arena/agentic_environment_generation/inference_backend.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,7 @@ class InferenceEndpoint:
5050
PUBLIC_ENDPOINT = InferenceEndpoint(
5151
name="public",
5252
base_url="https://integrate.api.nvidia.com/v1",
53-
# If you cannot access the model in your region, try "nvidia/nemotron-3-super-120b-a12b".
54-
model="openai/gpt-oss-120b",
53+
model="deepseek-ai/deepseek-v4-pro-0813",
5554
api_key_env_var="NVIDIA_API_KEY",
5655
)
5756
"""Publicly reachable build.nvidia.com endpoint, reached with an NGC API key."""

0 commit comments

Comments
 (0)