Skip to content

Commit e11f8b4

Browse files
committed
Address review: clarify model comparisons
Signed-off-by: Xinjie Yao <xyao@nvidia.com>
1 parent 4498e8b commit e11f8b4

2 files changed

Lines changed: 6 additions & 11 deletions

File tree

docs/pages/concepts/agentic_environment_generation/model_selection.rst

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,7 @@ Selecting the Model
4343
Each endpoint preset has its own default model, so switching endpoints switches models (see
4444
:ref:`agentic-env-gen-prerequisites`). The CLI runner overrides it per run with ``--model`` and
4545
``--temperature``; the GUI runner selects the endpoint in the generation panel and uses that
46-
endpoint's default model. The public endpoint defaults to ``deepseek-ai/deepseek-v4-pro-0813``.
47-
For the public endpoint, any model in the
46+
endpoint's default model. For the public endpoint, any model in the
4847
`NVIDIA NIM LLM API reference <https://docs.api.nvidia.com/nim/reference/llm-apis>`_ works, as long
4948
as it supports strict structured outputs — a larger context window buys more reliable prim path
5049
resolution.
@@ -55,7 +54,7 @@ resolution.
5554

5655
* - ``ARENA_INFERENCE_ENDPOINT``
5756
- Accessibility
58-
- Model
57+
- Default model
5958
- API key variable
6059
- Pass rate
6160
- Mean runtime
@@ -65,12 +64,6 @@ resolution.
6564
- ``NVIDIA_API_KEY``
6665
- 15/15 (100%)
6766
- 150.66 s
68-
* - ``public`` (comparison)
69-
- Public (free)
70-
- ``nvidia/nemotron-3-ultra-550b-a55b``
71-
- ``NVIDIA_API_KEY``
72-
- 9/15 (60%)
73-
- 149.86 s
7467
* - ``internal``
7568
- NVIDIA internal
7669
- ``openai/openai/gpt-5.6-terra``
@@ -88,7 +81,10 @@ resolution.
8881
The benchmark ran each of five documented prompts three times. Pass rate is the fraction of
8982
generated specs that matched the expected structure; runtime is the mean end-to-end
9083
``generate_spec`` runtime. These results are snapshots rather than guarantees: model output is
91-
non-deterministic, and service load affects runtime.
84+
non-deterministic, and service load affects runtime. On the public endpoint, overriding the
85+
default with ``--model nvidia/nemotron-3-ultra-550b-a55b`` passed 9/15 runs (60%) with a
86+
149.86-second mean runtime; ``--model openai/gpt-oss-20b`` passed 7/15 runs (46.7%) with a
87+
333.17-second mean runtime.
9288

9389
Reviewing the Generated Spec
9490
----------------------------

isaaclab_arena/agentic_environment_generation/inference_backend.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,6 @@ class InferenceEndpoint:
5050
PUBLIC_ENDPOINT = InferenceEndpoint(
5151
name="public",
5252
base_url="https://integrate.api.nvidia.com/v1",
53-
# If you cannot access the model in your region, try "nvidia/nemotron-3-super-120b-a12b".
5453
model="deepseek-ai/deepseek-v4-pro-0813",
5554
api_key_env_var="NVIDIA_API_KEY",
5655
)

0 commit comments

Comments
 (0)