Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ Every PR description must include an **AI model disclosure** section inside a co
3. Ping a core maintainer on Slack for final approval, after obtaining the checklist sign-off when required.
4. An authorized maintainer posts `/use <run_id>` (see below) and the PR is merged via the reuse path.

H3 video deployments follow the same flow. The [H3 submission guide](video-generation/SUBMITTING.md) lists their video-specific steps in order.

**Performance changelog:** A PR that changes what any benchmark runs **MUST** append one new entry covering the affected config keys to the physical end of `inferencex-e2e/perf-changelog.yaml`; `python -m infx.matrix.changed --base origin/main` (run from `inferencex-e2e/`) lists them. A PR that changes no benchmark point adds none. Historical entries **MUST NOT** be edited.

For full config retirement, list the deleted config keys with `retired: true`. The planner verifies that every selected key existed in the base revision and is absent from the current master configs, retains the entry in changelog metadata, and schedules no benchmarks or evals for it. Retirement entries cannot use scenario filters, `append-only`, or eval-selection fields or modifiers; partial scenario retirement uses a regular entry for the surviving config.
Expand Down
79 changes: 19 additions & 60 deletions inferencex-e2e/docs/video-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ explicitly selected model view, records the final command, then starts it.
It adds no engine flags or performance environment defaults. The SGLang child
checks the assigned UUID order before importing its engine. The previous B200
`frozen-c124` four-clip JSON and YAML artifacts remain readable as historical evidence.
The App shows their four-sample cells in History, below its ten-valid-sample chart
floor.
Each recipe's `readiness.timeout_seconds` sets how long the server may load and
warm up.

Expand Down Expand Up @@ -95,67 +97,24 @@ the first native video reuse request; a PR-local checker cannot bypass that gate

## Contribute a video deployment

For an existing video runtime/API, change its native recipe and master config rather
than adding a launcher or modifying the Python client. The configured deployments
are the `*-video` keys in [`configs/nvidia-master.yaml`](../configs/nvidia-master.yaml)
and [`configs/amd-master.yaml`](../configs/amd-master.yaml);
each names its recipe directory under
[`benchmarks/single_node/srt-slurm-recipes/h3/`](../benchmarks/single_node/srt-slurm-recipes/h3/).
Their presence does not establish final-head GPU qualification.
To add or change a deployment, follow the
[H3 submission guide](../../video-generation/SUBMITTING.md).

1. Choose an existing combination and keep its frozen workload, model revision,
concurrency population and measurement boundary. Put the full engine command
after `services[].args`' `--`; put nonsecret process settings in `services[].env`
or recipe-wide `environment`. Keep the native serve/client entrypoints and
telemetry binding. A new model, engine API, topology or model-view format
needs maintainer adapter validation before it becomes a recipe-only contribution.
2. Change the existing row in its master config, or copy it with a
distinct key and recipe for a deployment that must remain separately selectable.
Pin the image by digest and use its real interpreter and architecture. The
H100/H200/B200 SGLang rows use the public CUDA amd64 image and the MI355X row its
ROCm MI35x build. A shared image does
not qualify a different topology or SKU: validate
the selected deployments through their sweep. Maintainers stage model snapshots in the runner's
existing HF cache; contributors do not patch the cache or SSH extra launch flags.
3. Render locally from `inferencex-e2e/`, using the repository's pinned srt-slurm
submodule and locked dependencies. This writes files only and submits no job:
### Render a deployment locally

```bash
recipe_output=$(mktemp -d)
uv run --locked --extra recipes infx generate \
--config-key h3-bf16-h100-sglang-video --output-dir "$recipe_output"
uv run --locked --all-extras --group test python -m infx.matrix.changed \
--base origin/main --json
```
Render locally from `inferencex-e2e/`, using the repository's pinned srt-slurm
submodule and locked dependencies. This writes files only and submits no job:

Inspect `manifest.json` and its generated YAML for the selected config, image,
model, GPU count, full command, environment and client concurrency. The recipe
fingerprint binds the matrix row and rendered recipe; a successful render
proves schema/binding, not model execution. Run the focused binder, entrypoint,
matrix and result tests for changed behavior.
4. Append the PR's single `perf-changelog.yaml` entry with every affected config
key and a `scenario-type` that lists `video-generation`. The planner rejects an
entry whose named video config would otherwise plan no jobs. Follow [the contribution workflow](../../CONTRIBUTING.md) for review,
CODEOWNER sign-off, the primary sweep label and trusted reuse. A maintainer
controls fork scheduling. Use the final PR-selected matrix and preserve failed
attempts; an ad-hoc diagnostic cannot replace its qualifying sweep.
5. Inspect the original run artifacts. `video-sweep-manifest` identifies the
selected attempt and deployments. Each `h3-results-*` bundle retains
`manifest.json`, `ci.json`, `SHA256SUMS` and `gpu/c*/` request, media and power
evidence. A valid `release-v1` deployment needs 20 valid measured C1 clips plus
its separate warmup. Four-sample B200 cells remain History evidence below the
App's ten-valid-sample chart floor.
```bash
recipe_output=$(mktemp -d)
uv run --locked --extra recipes infx generate \
--config-key h3-bf16-h100-sglang-video --output-dir "$recipe_output"
uv run --locked --all-extras --group test python -m infx.matrix.changed \
--base origin/main --json
```

During execution, native `recipe.lock.yaml` and the scheduler job record identify
the submitted configuration. Inspect `logs/video/server/launch.json` for the actual
argv/environment and assigned GPU UUIDs; `spec.json` retains the selected model
and workload. Service startup failures are in `logs/service_video-server.out` and
`logs/sweep_<job>.log`. Client failures remain in each cell's `baseline/run.json`;
native power failures retain `power-validation.json`. Keep run ID, attempt, source
head, executed checkout, recipe fingerprint and plan hash distinct when reporting
a problem. Never hand-edit a bundle to make its result eligible.

After authorized merge and publication, use the App's published History and
read-only video API to retrieve the original source. An unpublished source is not
an official result, and opening a comparison must not publish it. Follow the
publication handoff above rather than copying media by hand.
Inspect `manifest.json` and its generated YAML for the selected config, image,
model, GPU count, full command, environment and client concurrency. The recipe
fingerprint binds the matrix row and rendered recipe; a successful render
proves schema/binding, not model execution. Run the focused binder, entrypoint,
matrix and result tests for changed behavior.
172 changes: 172 additions & 0 deletions video-generation/SUBMITTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
# Submit an H3 video deployment

Follow these steps to add or change an H3 (VideoGenX) deployment, from choosing a
config to a published result. Each step links the section that holds its detail.
[Native video delivery](../inferencex-e2e/docs/video-generation.md) explains how the
sweep runs, what a result bundle contains, and how reuse is validated.

| Step | Owner |
| --- | --- |
| [1. Choose the deployment](#1-choose-the-deployment) | You |
| [2. Change the recipe and config row](#2-change-the-recipe-and-config-row) | You |
| [3. Check locally](#3-check-locally) | You |
| [4. Append the changelog entry](#4-append-the-changelog-entry) | You |
| [5. Open the PR and start the sweep](#5-open-the-pr-and-start-the-sweep) | You open the PR. A maintainer labels a fork PR. |
| [6. Confirm the run qualifies](#6-confirm-the-run-qualifies) | You |
| [7. Get review and sign-off](#7-get-review-and-sign-off) | One eligible CODEOWNER, then a core maintainer |
| [8. Reuse and merge](#8-reuse-and-merge) | An authorized maintainer |
| [9. Verify publication](#9-verify-publication) | Merge Ingest publishes. You verify. |

## 1. Choose the deployment

The configured deployments are the `*-video` keys in
[`nvidia-master.yaml`](../inferencex-e2e/configs/nvidia-master.yaml) and
[`amd-master.yaml`](../inferencex-e2e/configs/amd-master.yaml). Each key names its
recipe directory under
[`srt-slurm-recipes/h3/`](../inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/h3/).

Start from an existing combination. Keep its frozen workload, model revision,
concurrency population, and measurement boundary. To change a deployment, edit its
row. To add a deployment that must stay separately selectable, copy the row under a
distinct key with its own recipe. A new model, engine API, topology, or model-view
format needs maintainer adapter validation before it can be a recipe-only
contribution.

## 2. Change the recipe and config row

Change the native recipe and the master config row. Do not add a launcher or modify
the Python client. Keep the native serve and client entrypoints and the telemetry
binding.

- Put the complete engine command after the `--` in `services[].args`.
- Put nonsecret process settings in `services[].env` or the recipe-wide
`environment`. [Native video delivery](../inferencex-e2e/docs/video-generation.md)
lists the placeholders the binder substitutes and the names it rejects.
- Set `services[].env.VIDEO_RUNTIME_REVISION` to the engine's 40-character commit.
The binder rejects any other value, and the server records it as the engine
revision.
- Pin the image by SHA256 digest, and use its real interpreter and architecture.
The H100, H200, and B200 SGLang rows use the public CUDA amd64 image. The MI355X
row uses its ROCm MI35x build.

A shared image does not qualify a different topology or SKU. Each selected
deployment qualifies through its own sweep. Maintainers stage model snapshots in the
runner's existing Hugging Face cache. Do not patch the cache or pass launch flags
outside the recipe.

## 3. Check locally

Render the changed deployments and list the changed config keys as
[Render a deployment locally](../inferencex-e2e/docs/video-generation.md#render-a-deployment-locally)
describes. If you changed code under `video-generation/`, run its
[local checks](README.md#local-checks). For code under `inferencex-e2e/infx/`, run
the [repository local checks](../AGENTS.md#local-checks).

## 4. Append the changelog entry

Append one entry to the physical end of
[`perf-changelog.yaml`](../inferencex-e2e/perf-changelog.yaml). List every affected
config key, and include `video-generation` in `scenario-type`. The planner rejects
an entry whose named video configs would otherwise plan no jobs.

```yaml
- config-keys:
- h3-bf16-h100-sglang-video
scenario-type: [video-generation]
description:
- "<what changes in the deployment>"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
```

Leave `append-only` unset, because video runs only as a regular sweep. The
[performance changelog rule](../CONTRIBUTING.md#pr-review-flow) covers the one-entry
limit and byte preservation.

## 5. Open the PR and start the sweep

Write the title and body as [PR descriptions](../CONTRIBUTING.md#pr-descriptions)
requires. Then start the sweep.

- **Same-repository PR.** Apply exactly one primary label, normally
`full-sweep-fail-fast`. `run-sweep.yml` then sweeps the changelog matrix.
- **Fork PR.** Mark the PR ready for review and resolve merge conflicts. A
maintainer applies the label, which approves only the current head and dispatches
the [trusted sweep](../.github/workflows/README.md#trusted-external-fork-sweep-dispatch-poc).
After each push, the maintainer removes and re-adds the label.

Do not add `evals-only` or `agentx-fast`, because a run with either label cannot be
reused. [PR sweep labels](../.github/workflows/README.md#pr-sweep-labels) describes
every label and modifier.

## 6. Confirm the run qualifies

The qualifying run is a full sweep of the final changelog matrix on a commit in the
PR. Its `validate-video-results` job must pass. That job checks the
[reuse conditions](../inferencex-e2e/docs/video-generation.md) against the retained
evidence of every selected deployment. A valid `release-v1` deployment needs 20
valid measured C1 clips plus its separate warmup. Keep failed attempts. An ad hoc
diagnostic run cannot replace the qualifying sweep.

Inspect the run's artifacts:

- `video-sweep-manifest` identifies the selected attempt and deployments.
- Each `h3-results-*` bundle holds `manifest.json`, `ci.json`, `SHA256SUMS`, and the
request, media, and power evidence under `gpu/c*/`.

Never hand-edit a bundle to make its result eligible. If a deployment fails, see
[Troubleshoot a run](#troubleshoot-a-run).

## 7. Get review and sign-off

The master configs and the recipes have vendor owners in
[CODEOWNERS](../.github/CODEOWNERS). One eligible CODEOWNER posts the
[PR Review Checklist](../CONTRIBUTING.md#the-pr-review-checklist-codeowner-sign-off).
A core maintainer gives final approval.

## 8. Reuse and merge

After the final sweep passes, an authorized maintainer comments `/use <run_id>` to
select it. [Reuse at merge](../CONTRIBUTING.md#reusing-your-prs-green-sweep-at-merge-with-use)
describes the rules.

- **Same-repository PR.** The maintainer merges with
`infx.workflows.merge_with_reuse`.
- **Fork PR.** The maintainer pins the trusted `e2e-tests.yml` run by its ID, then
merges. Automatic selection and `merge_with_reuse` find only `run-sweep.yml` runs.

## 9. Verify publication

On the merge to `main`, Merge Ingest publishes the reused run as
[native video delivery](../inferencex-e2e/docs/video-generation.md) describes.
Confirm that the post-merge jobs pass, as [After merging](../CONTRIBUTING.md#after-merging)
requires. Then use the App's published History and read-only video API to retrieve
the original source. An unpublished source is not an official result, and opening a
comparison must not publish it. Do not copy media by hand.

## Reference quality (PSNR)

After a same-repository Run Sweep finishes, the
[Video Quality](../.github/workflows/video-quality.yml) workflow scores each measured
clip against the public Diffusers reference videos. It uploads the scores as the
`h3-quality-<run>-<attempt>` artifact. It does not score a trusted fork sweep
automatically. A maintainer dispatches it with that run's `run_id` and
`run_attempt`.

PSNR measures how closely each clip matches the reference implementation's output
for the same prompt and seed. Other hardware and engines legitimately diverge from
that output, so the score is not a quality grade. `quality.json` keeps
`quality_status: unjudged` and `release_qualified: false`, and no workflow gates
reuse or merge on the score.
[Reference PSNR](README.md#reference-psnr) describes the method and the reasons a
clip stays unscored.

## Troubleshoot a run

During execution, the native `recipe.lock.yaml` and the scheduler job record identify
the submitted configuration. Inspect `logs/video/server/launch.json` for the actual
argv, environment, and assigned GPU UUIDs. `spec.json` retains the selected model
and workload. Service startup failures are in `logs/service_video-server.out` and
`logs/sweep_<job>.log`. Client failures remain in each cell's `baseline/run.json`.
Native power failures retain `power-validation.json`. When you report a problem,
keep the run ID, attempt, source head, executed checkout, recipe fingerprint, and
plan hash distinct.
Loading