Skip to content

Commit 92bbd20

Browse files
author
Neil Mehta
committed
Add native performance bisection agent
Make Isaac Lab the source of truth for the validated bisection runtime, pinned benchmark tooling, security controls, and project-discoverable Skills.
1 parent 796d1ad commit 92bbd20

115 files changed

Lines changed: 21220 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/isaaclab-perf-benchmark-commit‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../skills/developer/perf-benchmark-commit

‎.agents/skills/isaaclab-perf-bisect-range‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../skills/developer/perf-bisect-range

‎.agents/skills/isaaclab-perf-bisection‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../skills/developer/perf-bisection

‎.agents/skills/isaaclab-perf-threshold-check‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../skills/developer/perf-threshold-check

‎.github/workflows/bisection-agent-ci.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Copyright (c) 2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2+
# All rights reserved.
3+
#
4+
# SPDX-License-Identifier: BSD-3-Clause
5+
6+
name: Bisection Agent CI
7+
8+
on:
9+
pull_request:
10+
paths:
11+
- "tools/perf_bisection/**"
12+
- "tools/perf_smoke_test/**"
13+
- "skills/developer/perf-*/**"
14+
- ".agents/skills/isaaclab-perf-*"
15+
- ".github/workflows/bisection-agent-ci.yml"
16+
push:
17+
branches:
18+
- develop
19+
paths:
20+
- "tools/perf_bisection/**"
21+
- "tools/perf_smoke_test/**"
22+
- "skills/developer/perf-*/**"
23+
- ".agents/skills/isaaclab-perf-*"
24+
- ".github/workflows/bisection-agent-ci.yml"
25+
workflow_dispatch:
26+
27+
permissions:
28+
contents: read
29+
30+
jobs:
31+
test:
32+
name: Python ${{ matrix.python-version }}
33+
runs-on: ubuntu-latest
34+
strategy:
35+
fail-fast: false
36+
matrix:
37+
python-version:
38+
- "3.11"
39+
- "3.12"
40+
41+
steps:
42+
- uses: actions/checkout@v6
43+
- uses: actions/setup-python@v5
44+
with:
45+
python-version: ${{ matrix.python-version }}
46+
cache: pip
47+
cache-dependency-path: tools/perf_bisection/pyproject.toml
48+
- name: Install
49+
run: python -m pip install --editable "./tools/perf_bisection[test]"
50+
- name: Lint
51+
run: |
52+
ruff check tools/perf_bisection tools/perf_smoke_test
53+
ruff format --check tools/perf_bisection tools/perf_smoke_test
54+
- name: Test
55+
run: pytest -q tools/perf_bisection/tests
56+
- name: Verify CLI installation
57+
run: isaaclab-bisect --help
58+
- name: Build wheel
59+
if: matrix.python-version == '3.12'
60+
run: python -m pip wheel ./tools/perf_bisection --no-deps --wheel-dir dist
61+
- name: Verify clean wheel installation
62+
if: matrix.python-version == '3.12'
63+
run: |
64+
python -m venv /tmp/isaaclab-bisection-agent
65+
/tmp/isaaclab-bisection-agent/bin/python -m pip install --no-deps dist/*.whl
66+
/tmp/isaaclab-bisection-agent/bin/isaaclab-bisect --help
67+
/tmp/isaaclab-bisection-agent/bin/isaaclab-bisect-skill --help
68+
69+
container:
70+
name: Container build
71+
runs-on: ubuntu-latest
72+
73+
steps:
74+
- uses: actions/checkout@v6
75+
- name: Build reconstruction image
76+
run: |
77+
docker build \
78+
--file tools/perf_bisection/docker/Dockerfile \
79+
--tag isaaclab-bisection-agent:test \
80+
tools/perf_bisection

‎.github/workflows/bisection-agent-security.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Copyright (c) 2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2+
# All rights reserved.
3+
#
4+
# SPDX-License-Identifier: BSD-3-Clause
5+
6+
name: Bisection Agent Security
7+
8+
on:
9+
pull_request:
10+
paths:
11+
- "tools/perf_bisection/**"
12+
- ".github/workflows/bisection-agent-security.yml"
13+
push:
14+
branches:
15+
- develop
16+
paths:
17+
- "tools/perf_bisection/**"
18+
- ".github/workflows/bisection-agent-security.yml"
19+
workflow_dispatch:
20+
21+
permissions:
22+
contents: read
23+
24+
jobs:
25+
codeql:
26+
name: CodeQL
27+
runs-on: ubuntu-latest
28+
timeout-minutes: 20
29+
permissions:
30+
contents: read
31+
security-events: write
32+
33+
steps:
34+
- uses: actions/checkout@v6
35+
- uses: github/codeql-action/init@v4
36+
with:
37+
languages: python
38+
- uses: github/codeql-action/analyze@v4
39+
40+
dependencies:
41+
name: Python dependency audit
42+
runs-on: ubuntu-latest
43+
timeout-minutes: 15
44+
45+
steps:
46+
- uses: actions/checkout@v6
47+
- uses: actions/setup-python@v5
48+
with:
49+
python-version: "3.12"
50+
- uses: pypa/gh-action-pip-audit@v1
51+
with:
52+
inputs: tools/perf_bisection
53+
54+
container:
55+
name: Container vulnerability scan
56+
runs-on: ubuntu-latest
57+
timeout-minutes: 45
58+
59+
steps:
60+
- uses: actions/checkout@v6
61+
- name: Build reconstruction image
62+
run: |
63+
docker build \
64+
--file tools/perf_bisection/docker/Dockerfile \
65+
--tag isaaclab-bisection-agent:security-scan \
66+
tools/perf_bisection
67+
- uses: anchore/scan-action@v7
68+
with:
69+
image: isaaclab-bisection-agent:security-scan
70+
fail-build: true
71+
severity-cutoff: high

‎skills/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ Developer skills:
99
- `developer/pr-workflow/`: prepare changes for review using Isaac Lab's PR, commit, changelog, and validation conventions.
1010
- `developer/changelog-fragments/`: add and validate package changelog fragments.
1111
- `developer/coding-style/`: apply Isaac Lab coding style, API design, docstring, type-hint, lazy export, and contribution conventions.
12+
- `developer/perf-bisection/`: route performance investigations through reproducible single-commit benchmarking, deterministic threshold checks, or paired-reference bisection.
13+
- `developer/perf-benchmark-commit/`: benchmark one Isaac Lab revision with pinned harness-owned tooling.
14+
- `developer/perf-threshold-check/`: classify existing performance evidence without running simulation.
15+
- `developer/perf-bisect-range/`: qualify good and bad references and locate the first regressing commit.
1216

1317
User skills:
1418

‎skills/developer/perf-benchmark-commit/SKILL.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
name: isaaclab-perf-benchmark-commit
3+
description: Reconstructs and benchmarks one Isaac Lab commit with pinned perf-smoke tooling and a versioned JSON contract. Use when measuring a historical revision, collecting a canonical metric, or invoking single-commit benchmarking from Fanes Agent or other automation.
4+
audience: developer
5+
status: experimental
6+
owners:
7+
- isaaclab-maintainers
8+
---
9+
10+
# Benchmark One Commit
11+
12+
## When To Use
13+
14+
Use this skill to measure one commit without comparing it to a baseline or
15+
searching a range. For a gate verdict use `isaaclab-perf-threshold-check`; for a
16+
first-bad search use `isaaclab-perf-bisect-range`.
17+
18+
## Workflow
19+
20+
1. Create an input matching [input.schema.json](input.schema.json). Pin
21+
`tooling_ref` to a full commit SHA for authoritative runs.
22+
2. Use `local-reconstruct` for host isolation or `docker-reconstruct` with an
23+
explicit image for container isolation.
24+
3. Run the JSON adapter:
25+
26+
```bash
27+
isaaclab-bisect-skill \
28+
--input benchmark-input.json \
29+
--output benchmark-output.json
30+
```
31+
32+
4. Read the output using [output.schema.json](output.schema.json). Treat
33+
`result.succeeded: false` and categorized skips as evidence, not measured
34+
regressions.
35+
36+
The adapter streams benchmark logs to the caller and writes the output envelope
37+
even when the harness returns a nonzero status.
38+
39+
## Validation
40+
41+
- Require `tooling_manifest.json` and matching tooling hashes for authoritative results.
42+
- Require a numeric selected metric in every measured attempt.
43+
- Confirm hardware identity and mismatch warnings are retained.
44+
- Validate the request and response against the linked schemas.
45+
46+
## Maintenance
47+
48+
Keep this skill synchronized with
49+
`tools/perf_bisection/src/isaaclab_bisection/skill_api.py`,
50+
`tools/perf_bisection/src/isaaclab_bisection/cli.py`, and
51+
`tools/perf_bisection/src/isaaclab_bisection/bisection/measurement.py`. Add fields additively; bump
52+
the JSON `schema_version` before making an incompatible contract change.
53+
54+
## References
55+
56+
- [Evaluations](evaluations.md)
57+
- [Invocation reference](reference.md)
58+
- [Input schema](input.schema.json)
59+
- [Output schema](output.schema.json)
60+
- [Bisection README](../../../tools/perf_bisection/README.md)

‎skills/developer/perf-benchmark-commit/evaluations.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Benchmark One Commit Evaluations
2+
3+
## Scenario 1: Authoritative Historical Measurement
4+
5+
Query: "Benchmark commit `<SHA>` on the L40S using the current perf-smoke ruler."
6+
7+
Expected behavior:
8+
9+
- Selects `isaaclab-perf-benchmark-commit`.
10+
- Creates a schema-valid `benchmark_commit` request with full candidate and
11+
tooling SHAs, a fresh work directory, and one fixed task/backend/metric
12+
contract.
13+
- Uses `docker-reconstruct` for the authoritative run and accepts success only
14+
when `status` is `completed` and `result.succeeded` is true.
15+
- Preserves the response and complete work directory as one audit unit.
16+
17+
Known failure modes:
18+
19+
- Uses `WORKTREE` tooling while describing the result as authoritative.
20+
- Invokes benchmark code from the candidate checkout instead of the pinned
21+
tooling snapshot.
22+
- Treats process exit code zero as sufficient without checking the result.
23+
24+
## Scenario 2: Local Tooling Development
25+
26+
Query: "Test my uncommitted bisection tooling against one commit before I commit it."
27+
28+
Expected behavior:
29+
30+
- Uses `tooling_ref: WORKTREE` and clearly labels the result
31+
non-authoritative.
32+
- Keeps the candidate commit, task, metric, and measurement policy explicit.
33+
- Does not present the output as a portable cross-host measurement.
34+
35+
Known failure modes:
36+
37+
- Requires a committed tooling SHA for this explicitly developmental run.
38+
- Omits the non-authoritative limitation.
39+
- Reuses the output as an authoritative reference in a later bisection.
40+
41+
## Scenario 3: Candidate Cannot Be Measured
42+
43+
Query: "The single-commit benchmark failed during reconstruction. Is that commit a regression?"
44+
45+
Expected behavior:
46+
47+
- Reads the response status, `result.succeeded`, categorized skip or blocker,
48+
and linked attempt artifacts.
49+
- Reports an environment, host, or tooling failure as evidence that no metric
50+
was obtained.
51+
- Does not classify the commit as a performance regression.
52+
53+
Known failure modes:
54+
55+
- Converts any installation or runtime failure into a BAD verdict.
56+
- Falls back to a different benchmark ruler to obtain a number.
57+
- Discards the install log, tooling manifest, or attempt summary.

0 commit comments

Comments
 (0)