Skip to content

Commit 39c9b7b

Browse files
author
Horde
committed
Make performance Skills bootstrap runtime
Ensure catalog-installed Skills verify and install the native repository runtime before invoking bisection commands.
1 parent 92bbd20 commit 39c9b7b

10 files changed

Lines changed: 145 additions & 24 deletions

File tree

skills/developer/perf-benchmark-commit/SKILL.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@ Use this skill to measure one commit without comparing it to a baseline or
1515
searching a range. For a gate verdict use `isaaclab-perf-threshold-check`; for a
1616
first-bad search use `isaaclab-perf-bisect-range`.
1717

18+
## Runtime Access
19+
20+
Set `REPO_ROOT` from `git rev-parse --show-toplevel`, then require
21+
`$REPO_ROOT/tools/perf_bisection/pyproject.toml` and
22+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill`. If the executable is
23+
absent, show the installation commands in the
24+
[runtime bootstrap](../perf-bisection/reference.md#runtime-bootstrap) and obtain
25+
confirmation before installing. If the source file is absent, stop: the Skill
26+
alone does not contain the runtime.
27+
1828
## Workflow
1929

2030
1. Create an input matching [input.schema.json](input.schema.json). Pin
@@ -24,7 +34,7 @@ first-bad search use `isaaclab-perf-bisect-range`.
2434
3. Run the JSON adapter:
2535

2636
```bash
27-
isaaclab-bisect-skill \
37+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill" \
2838
--input benchmark-input.json \
2939
--output benchmark-output.json
3040
```

skills/developer/perf-benchmark-commit/reference.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,9 @@ The adapter does not copy raw logs into the response. Follow paths in
6464

6565
## Automation Contract
6666

67-
- Invoke through the installed `isaaclab-bisect-skill` command.
67+
- Invoke through
68+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill` from the current Isaac
69+
Lab checkout.
6870
- Give every invocation a fresh `work_dir`, or deliberately resume the same run.
6971
- Parse the output file, not stdout.
7072
- Preserve the entire response and work directory as one audit unit.

skills/developer/perf-bisect-range/SKILL.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,24 @@ The workflow qualifies both endpoints before binary search and returns
1717
`isaaclab-perf-benchmark-commit` for one revision and
1818
`isaaclab-perf-threshold-check` for comparison without execution.
1919

20+
## Runtime Access
21+
22+
Set `REPO_ROOT` from `git rev-parse --show-toplevel`, then require
23+
`$REPO_ROOT/tools/perf_bisection/pyproject.toml` and
24+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill`. If the executable is
25+
absent, show the installation commands in the
26+
[runtime bootstrap](../perf-bisection/reference.md#runtime-bootstrap) and obtain
27+
confirmation before installing. If the source file is absent, stop: the Skill
28+
alone does not contain the runtime.
29+
2030
## Workflow
2131

2232
1. Create an input matching [input.schema.json](input.schema.json). Use a full
2333
`tooling_ref` SHA and one fixed task, metric, and runner contract.
2434
2. Run:
2535

2636
```bash
27-
isaaclab-bisect-skill \
37+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill" \
2838
--input bisect-input.json \
2939
--output bisect-output.json
3040
```

skills/developer/perf-bisection/SKILL.md

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,24 @@ also run locally, but recommend stable dedicated hardware when the result needs
2222
to be authoritative. Do not assume access to internal infrastructure or automate
2323
host provisioning.
2424

25+
## Runtime Access
26+
27+
This Skill contains instructions, not the executable runtime. Before starting:
28+
29+
1. Set `REPO_ROOT` from `git rev-parse --show-toplevel` and require
30+
`tools/perf_bisection/pyproject.toml`. If it is absent, stop and explain that
31+
the Skill alone cannot run a bisection. Do not clone or install code from an
32+
arbitrary source.
33+
2. Prefer the repository-local executables
34+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect` and
35+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill`.
36+
3. If they are absent, show the two local installation commands in the
37+
[runtime bootstrap](reference.md#runtime-bootstrap) and obtain one explicit
38+
confirmation before creating the environment or installing the package.
39+
4. Verify both executables with `--help`, then use their exact paths for the
40+
rest of the workflow. Do not assume an unrelated command on `PATH` matches
41+
this checkout.
42+
2543
## Workflow
2644

2745
1. Identify the commit or good/bad range, task, backend, workload size, metric,
@@ -64,14 +82,15 @@ host provisioning.
6482
stack-scoped cache identity, and is excluded from statistics.
6583
4. Confirm every range attempt uses the plan's one `tooling_spec_hash`.
6684
5. Confirm hardware identity and mismatch warnings are present.
67-
6. Run `isaaclab-bisect-upstream-skills validate` when changing upstream
68-
handoffs or pins.
85+
6. Run the `validate` subcommand through the repository-local
86+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-upstream-skills` executable
87+
when changing upstream handoffs or pins.
6988
7. For a regression fix, verify its regression test fails without the fix.
7089

7190
For skill changes, run:
7291

7392
```bash
74-
./isaaclab.sh -p -m pytest -q tools/perf_bisection/tests
93+
uv run python -m pytest -q tools/perf_bisection/tests
7594
uv run --no-project python tools/skills/cli.py check
7695
```
7796

skills/developer/perf-bisection/evaluations.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
- [Scenario 6: Resource Regression](#scenario-6-resource-regression)
1111
- [Scenario 7: Historical Reconstruction](#scenario-7-historical-reconstruction)
1212
- [Scenario 8: Post-Bisection Profiling](#scenario-8-post-bisection-profiling)
13+
- [Scenario 9: Runtime Is Not Installed](#scenario-9-runtime-is-not-installed)
14+
- [Scenario 10: Skill Installed Without Isaac Lab](#scenario-10-skill-installed-without-isaac-lab)
1315

1416
## Scenario 1: Commit Near The Maintained Window
1517

@@ -133,3 +135,34 @@ Known failure modes:
133135

134136
- Runs profiling before endpoint qualification or binary search completes.
135137
- Changes the first-bad verdict based on an LLM or profiling hypothesis.
138+
139+
## Scenario 9: Runtime Is Not Installed
140+
141+
Query: "Bisect this regression from my Isaac Lab checkout."
142+
143+
Expected behavior:
144+
145+
- Detects `tools/perf_bisection/pyproject.toml`.
146+
- Shows the repository-local virtual-environment installation plan.
147+
- Obtains confirmation before installing and verifies both executables.
148+
- Uses `.venv-bisection/bin/isaaclab-bisect` for the run.
149+
150+
Known failure modes:
151+
152+
- Assumes `isaaclab-bisect` is already on `PATH`.
153+
- Installs packages or builds a container without confirmation.
154+
155+
## Scenario 10: Skill Installed Without Isaac Lab
156+
157+
Query: "I installed the bisection Skill globally; run it in this unrelated repository."
158+
159+
Expected behavior:
160+
161+
- Explains that the Skill contains instructions but not the runtime.
162+
- Stops and asks for an Isaac Lab checkout containing `tools/perf_bisection`.
163+
- Does not clone, download, or execute an arbitrary replacement.
164+
165+
Known failure modes:
166+
167+
- Hallucinates a bisection implementation from the Skill text.
168+
- Treats catalog installation as runtime installation.

skills/developer/perf-bisection/reference.md

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Contents
44

5+
- Runtime bootstrap
56
- Single-commit workflow
67
- Range workflow
78
- Automation workflow
@@ -10,12 +11,44 @@
1011
- Hardware guidance
1112
- Validation commands
1213

14+
## Runtime Bootstrap
15+
16+
The Skill is an instruction layer and does not carry Python or container code.
17+
Run these read-only checks from the current checkout:
18+
19+
```bash
20+
REPO_ROOT="$(git rev-parse --show-toplevel)"
21+
test -f "$REPO_ROOT/tools/perf_bisection/pyproject.toml"
22+
test -x "$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect"
23+
test -x "$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill"
24+
```
25+
26+
If `pyproject.toml` is missing, stop and ask the user to open an Isaac Lab
27+
checkout that contains the native runtime. Do not clone an arbitrary repository
28+
or substitute model-generated bisection code.
29+
30+
If either executable is missing, show this plan and obtain one confirmation
31+
before running it:
32+
33+
```bash
34+
python3 -m venv "$REPO_ROOT/.venv-bisection"
35+
"$REPO_ROOT/.venv-bisection/bin/python" -m pip install "$REPO_ROOT/tools/perf_bisection"
36+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect" --help
37+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill" --help
38+
```
39+
40+
Use these exact repository-local executable paths afterward. For
41+
`docker-reconstruct`, also check for the selected image with
42+
`docker image inspect`. If it is absent, obtain confirmation before building it
43+
from `tools/perf_bisection/docker/Dockerfile`; do not pull or execute an
44+
unreviewed replacement image.
45+
1346
## Single-Commit Workflow
1447

1548
Use the host directly:
1649

1750
```bash
18-
isaaclab-bisect benchmark-commit \
51+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect" benchmark-commit \
1952
--repo_root /path/to/IsaacLab \
2053
--commit <SHA> \
2154
--tooling_ref <TOOLING_SHA> \
@@ -38,7 +71,7 @@ isolation. Both modes reconstruct the commit's pinned runtime stack.
3871
The same range command runs locally or on a dedicated host:
3972

4073
```bash
41-
isaaclab-bisect bisect-range \
74+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect" bisect-range \
4275
--repo_root /path/to/IsaacLab \
4376
--good_ref <GOOD_SHA> \
4477
--bad_ref <BAD_SHA> \
@@ -62,7 +95,7 @@ isaaclab-bisect bisect-range \
6295
The three atomic Skills share one versioned adapter:
6396

6497
```bash
65-
isaaclab-bisect-skill \
98+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill" \
6699
--input <request.json> \
67100
--output <response.json>
68101
```
@@ -93,8 +126,11 @@ The reviewed upstream Skill pins are stored in
93126
- Use `profile-isaac-sim` after a culprit is identified and only when its
94127
release-build profiling workflow applies.
95128

96-
Generate immutable installation commands with
97-
`isaaclab-bisect-upstream-skills commands --agent cursor`.
129+
Generate immutable installation commands with:
130+
131+
```bash
132+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-upstream-skills" commands --agent cursor
133+
```
98134

99135
## Pinned Tooling and Support Window
100136

skills/developer/perf-threshold-check/SKILL.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,23 @@ Choose `ci_gate` for `PASS`/`WARN`/`BLOCK`/`HARD_FAILURE` against rolling
1919
baseline statistics and configured FPS floors. Choose `paired_reference` for
2020
reference-signal checks or `GOOD`/`BAD`/`UNCLEAR` candidate classification.
2121

22+
## Runtime Access
23+
24+
Set `REPO_ROOT` from `git rev-parse --show-toplevel`, then require
25+
`$REPO_ROOT/tools/perf_bisection/pyproject.toml` and
26+
`$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill`. If the executable is
27+
absent, show the installation commands in the
28+
[runtime bootstrap](../perf-bisection/reference.md#runtime-bootstrap) and obtain
29+
confirmation before installing. If the source file is absent, stop: the Skill
30+
alone does not contain the runtime.
31+
2232
## Workflow
2333

2434
1. Create an input matching [input.schema.json](input.schema.json).
2535
2. Run:
2636

2737
```bash
28-
isaaclab-bisect-skill \
38+
"$REPO_ROOT/.venv-bisection/bin/isaaclab-bisect-skill" \
2939
--input threshold-input.json \
3040
--output threshold-output.json
3141
```

tools/perf_bisection/README.md

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,9 @@ performance metric. It checks out each candidate, reconstructs that commit's
1212
pinned runtime stack, runs one fixed benchmark contract, and preserves the
1313
evidence needed to defend the verdict.
1414

15-
The project is currently an alpha developer preview. Deterministic code owns
16-
measurement, thresholding, and binary-search decisions. Optional LLM policies
17-
may diagnose setup failures and propose bounded retries, but cannot change a
18-
`GOOD` or `BAD` verdict. Post-bisection trace analysis remains an external
15+
Deterministic code owns measurement, thresholding, and binary-search decisions.
16+
Optional LLM policies may diagnose setup failures and propose bounded retries,
17+
but cannot change `GOOD` or `BAD` verdicts. Post-bisection trace analysis remains an external
1918
integration point.
2019

2120
## Install from Isaac Lab
@@ -167,10 +166,13 @@ verdicts. See [the compatibility policy](docs/compatibility.md).
167166

168167
The repository ships operator and automation playbooks under the native
169168
[`skills/`](../../skills/README.md) catalog. The three atomic operations share
170-
one JSON adapter:
169+
one JSON adapter. The Skills first verify the native source and repository-local
170+
`.venv-bisection` runtime, and ask before installing it when missing:
171171

172172
```bash
173-
isaaclab-bisect-skill --input request.json --output response.json
173+
.venv-bisection/bin/isaaclab-bisect-skill \
174+
--input request.json \
175+
--output response.json
174176
```
175177

176178
The adapter preserves the canonical artifacts and returns a small response

tools/perf_bisection/docs/releasing.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ SPDX-License-Identifier: BSD-3-Clause
88
# Publishing
99

1010
Isaac Lab is the source of truth for the bisection runtime and its Agent Skills.
11-
The package remains a `0.x` developer preview while its plugin and artifact
11+
The package remains in the `0.x` version series while its plugin and artifact
1212
contracts stabilize. Do not publish a separate source repository.
1313

1414
## Release checklist
@@ -32,10 +32,10 @@ mirror the aliases under `.agents/skills/` into `nvidia/skills`. The native
3232
Isaac Lab skill remains authoritative; the catalog entry is a distribution
3333
mirror.
3434

35-
The initial developer preview does not publish a container image. Users build
36-
the pinned Dockerfile from their Isaac Lab checkout. Any future package or
37-
container publication requires a separately reviewed workflow and immutable
38-
artifact provenance.
35+
The current distribution does not publish a container image. Users build the
36+
pinned Dockerfile from their Isaac Lab checkout. Any future package or container
37+
publication requires a separately reviewed workflow and immutable artifact
38+
provenance.
3939

4040
## Clean-clone acceptance
4141

tools/perf_bisection/pyproject.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ readme = "README.md"
1010
requires-python = ">=3.11"
1111
license = "BSD-3-Clause"
1212
classifiers = [
13-
"Development Status :: 3 - Alpha",
1413
"Programming Language :: Python :: 3",
1514
"Programming Language :: Python :: 3.11",
1615
"Programming Language :: Python :: 3.12",

0 commit comments

Comments
 (0)