How to Test Workflows

Run end-to-end commands from inferencex-e2e/, which owns pyproject.toml, uv.lock, and .python-version. Workflow dispatch generator arguments also resolve paths relative to that directory. The project-local .python-version selects its Python version.

In order to test configurations described in inferencex-e2e/configs, the primary workflow file used is .github/workflows/e2e-tests.yml. As input, this workflow takes in the CLI arguments for the python -m infx.matrix.generate command. The command usage is shown below:

usage: python -m infx.matrix.generate [-h] {full-sweep,test-config} ...

Generate benchmark configurations from YAML config files

positional arguments:
  {full-sweep,test-config}
                        Available commands
    full-sweep          Generate full sweep configurations with optional
                        filtering by model, precision, framework, runner type,
                        and sequence lengths
    test-config         Generate full sweep for specific config keys.
                        Validates that all specified keys exist before
                        generating.

options:
  -h, --help            show this help message and exit

`full-sweep` Command

The full-sweep command generates benchmark configurations with optional filtering. You can specify --single-node, --multi-node, or both. If neither is specified, both types are generated.

usage: python -m infx.matrix.generate full-sweep
    --config-files CONFIG_FILES [CONFIG_FILES ...]
    [--runner-config RUNNER_CONFIG]
    [--no-evals | --evals-only] [--all-evals]
    [--smoke] [--trim-conc]
    [--runner-node-filter RUNNER_NODE_FILTER]
    [--scenario-type {fixed-seq-len,agentic-coding} [{fixed-seq-len,agentic-coding} ...]]
    [--model-prefix MODEL_PREFIX [MODEL_PREFIX ...]]
    [--precision PRECISION [PRECISION ...]]
    [--framework FRAMEWORK [FRAMEWORK ...]]
    [--runner-type RUNNER_TYPE [RUNNER_TYPE ...]]
    [--seq-lens {1k1k,8k1k} [{1k1k,8k1k} ...]]
    [--step-size STEP_SIZE]
    [--min-conc MIN_CONC]
    [--max-conc MAX_CONC]
    [--max-tp MAX_TP]
    [--max-ep MAX_EP]
    [--single-node] [--multi-node]

If neither --single-node nor --multi-node is specified, both types are generated.

By default, throughput runs for every generated config and eval-only jobs run for the selected 8k1k subset and the AgentX GSM8K subset. --no-evals disables eval jobs, --evals-only emits only that selected subset, and adding --all-evals expands it to every eligible 8k1k fixed-sequence and agentic config. --all-evals always emits only those eval-only jobs, with or without --evals-only, but it cannot be combined with --no-evals.

--step-size must be greater than 1 and applies to concurrency ranges. --min-conc / --max-conc drop concurrencies outside the bounds, including explicit conc-list values. A single-node fixed-sequence conc-start/conc-end range that crosses a bound is clamped to it, and a fixed-sequence config with no concurrency left under --max-conc runs at the cap. When both bounds are set, --min-conc must not exceed --max-conc.

--trim-conc (the trim-conc input of e2e-tests.yml) keeps only the minimum concurrency of every generated single- and multi-node deployment shape, after eval selection, for a lowest-concurrency smoke run; in changelog-ref mode only throughput rows are trimmed, and evals keep their selected concurrency. No PR label enables trimming.

Examples

Generate all single-node and multi-node configurations (default):

full-sweep --config-files configs/nvidia-master.yaml

Test all single-node dsr1 configurations on B200 with 8k1k sequence lengths:

full-sweep --single-node --model-prefix dsr1 --runner-type cluster:b200-nscale --seq-lens 8k1k --config-files configs/nvidia-master.yaml

Test all single-node fp8 precision configs for 8k1k workloads:

full-sweep --single-node --precision fp8 --seq-lens 8k1k --config-files configs/nvidia-master.yaml configs/amd-master.yaml

Test all single-node TRT configs on H200 runners:

full-sweep --single-node --framework trt --runner-type h200 cluster:h200-dgxc --config-files configs/nvidia-master.yaml

Test specific single-node model on specific hardware with specific sequence lengths:

full-sweep --single-node --model-prefix dsr1 --runner-type cluster:b200-nscale --precision fp4 --framework sglang --seq-lens 8k1k --config-files configs/nvidia-master.yaml

Limit concurrency and parallelism for faster testing:

full-sweep --single-node --max-conc 64 --max-tp 4 --config-files configs/nvidia-master.yaml

Test all multi-node configurations:

full-sweep --multi-node --config-files configs/nvidia-master.yaml

Test agentic configurations:

full-sweep --scenario-type agentic-coding --config-files configs/nvidia-master.yaml configs/amd-master.yaml

`test-config` Command

The test-config command generates the full sweep for one or more specific config keys. This is useful for testing individual configurations without filtering by model prefix, framework, etc.

usage: python -m infx.matrix.generate test-config
    --config-files CONFIG_FILES [CONFIG_FILES ...]
    [--runner-config RUNNER_CONFIG]
    [--no-evals | --evals-only] [--all-evals]
    [--smoke] [--trim-conc]
    [--runner-node-filter RUNNER_NODE_FILTER]
    [--scenario-type {fixed-seq-len,agentic-coding} [{fixed-seq-len,agentic-coding} ...]]
    --config-keys CONFIG_KEYS [CONFIG_KEYS ...]
    [--conc CONC [CONC ...]]
    [--exp-names EXP_NAMES [EXP_NAMES ...]]
    [--seq-lens {1k1k,8k1k} [{1k1k,8k1k} ...]]

Config keys support wildcard patterns using * (matches any characters) and ? (matches a single character). Patterns that match no keys will raise an error.

Examples

Test a single config by exact name:

test-config --config-keys dsr1-fp4-b200-sglang --config-files configs/nvidia-master.yaml

Test multiple exact configs:

test-config --config-keys dsr1-fp4-b200-sglang dsr1-fp8-h200-trt --config-files configs/nvidia-master.yaml

Use wildcard to test all B200 configs:

test-config --config-keys *-b200-* --config-files configs/nvidia-master.yaml

Use wildcard to test all sglang configs:

test-config --config-keys *-sglang --config-files configs/nvidia-master.yaml configs/amd-master.yaml

Use wildcard to test all dsr1 model configs:

test-config --config-keys dsr1* --config-files configs/nvidia-master.yaml

Mix exact keys and patterns:

test-config --config-keys dsr1-fp4-b200-sglang qwen3.5* --config-files configs/nvidia-master.yaml

Override concurrency for targeted testing:

test-config --config-keys *-b200-* --conc 4 8 --config-files configs/nvidia-master.yaml

Run eval-only jobs for every generated fixed-sequence config:

test-config --config-keys dsr1-fp8-h200-sglang --evals-only --all-evals --config-files configs/nvidia-master.yaml

PR Sweep Labels

run-sweep.yml sweeps only same-repository PRs that change inferencex-e2e/perf-changelog.yaml, using the appended entries as the matrix. Apply exactly one primary label, normally full-sweep-fail-fast. Fork PRs use the trusted dispatch.

Primary label	Canary	Per-matrix fail-fast
`full-sweep-fail-fast`	Yes	Yes
`full-sweep-enabled`	Yes	No
`non-canary-full-sweep-enabled`	No	No

All primary labels run the full selected concurrency matrix. Modifiers require a primary label: all-evals expands eligible evals while retaining throughput; evals-only suppresses throughput; agentx-fast uses one extra AgentX warmup request after mandatory primers and a 20-minute profile, leaving fixed-sequence and eval settings canonical. all-evals/evals-only reject changelog additions with append-only: true or no-evals: true. Do not reuse an evals-only or agentx-fast run. For lowest-concurrency smoke, dispatch e2e-tests.yml with trim-conc: true; no PR label trims concurrency.

Labels authorize same-repository sweeps on draft or ready PRs. Marking ready does not dispatch a sweep. Changing a recognized primary or modifier label cancels the active sweep; another starts only if a primary label remains. To retrigger an already-labeled PR, remove and reapply its primary label. Merge conflicts block sweeps. Queue priority comes from the PR diff and the PR description's checklist, not priority labels. A PR gets a small bonus when every item under its Checklist heading is checked, except conditional items ("If ...", "Before merging ...") that do not apply yet. The checklist is read when the sweep starts, so later edits count from the next sweep. Trusted external sweeps classify the approved PR diff the same way. Manual e2e-tests.yml and profile.yml dispatches of a revision not yet on main classify its diff from the merge base only for patchwork, so engine-patching branches still run last. Klaud runs and revisions already on main keep unclassified scoring.

The canary is the lowest-concurrency non-eval candidate among single-node fixed-sequence 1k/1k, 8k/1k, or AgentX entries, falling back to multi-node AgentX. It is removed from the remaining matrix. If none qualify, fan-out proceeds without a canary; otherwise all matrices require canary success. Fail-fast cancels siblings within each matrix family, not every independent family. Find the first actual failure before treating a cancelled run as a flake. Manual e2e-tests.yml has no canary and defaults fail-fast to false.

Changelog Conflict Recovery

Run from inferencex-e2e/. After fetching and merging origin/main, if the changelog conflicts, use the helper while index stages 1/2/3 still exist:

uv run --extra workflows python -m infx.workflows.prepare_perf_changelog_merge resolve-conflict \
  --changelog-file perf-changelog.yaml --pr-number "$PR" --repo SemiAnalysisAI/InferenceX
git add perf-changelog.yaml
git commit --no-edit
uv run --extra workflows python -m infx.workflows.validate_perf_changelog \
  --changelog-file perf-changelog.yaml --base-ref origin/main --head-ref HEAD

The helper preserves main's bytes and appends only unique PR contributions. Stop on any refusal or unexpected historical edit. Never accept only one side, hand-merge, or reformat history. Resolve non-changelog conflicts separately. For authorized reuse, prefer infx.workflows.merge_with_reuse, which performs the synchronization and validation together.

Trusted External-Fork Sweep Dispatch (PoC)

Public-fork pull_request workflows receive no repository secrets, and every run-sweep.yml job requires a same-repository head. For an external PR, run-sweep.yml therefore runs no jobs: it neither validates the changelog nor fans out onto GPU runners. A maintainer with write, maintain, or admin permission can add any modifier labels first, then apply one primary sweep label to approve the PR's exact current head SHA. The PR must be open and non-draft. Dispatch waits briefly for GitHub to settle mergeability, then is refused unless the PR is mergeable and GitHub's test merge commit has the approved head as its second parent; resolve conflicts or re-add the label if GitHub is still recomputing it. trusted-external-sweep.yml then dispatches e2e-tests.yml from main and pins both the approved head and that verified merge SHA. e2e-tests.yml plans the changelog matrix itself and runs it with the trusted workflow's secrets.

The approval is revision-specific. A later push is not trusted automatically. Remove and re-add the primary sweep label to approve the new SHA. The trusted dispatcher never checks out or executes PR code itself.

This proof of concept produces benchmark and evaluation artifacts through the End-to-End Tests workflow. The dispatch also uploads a source-identity artifact (PR number, full approved head SHA, merge SHA, and the evals-only, agentx-fast, all-evals, and fail-fast modifiers) and changelog-metadata. A pinned /use <run_id> accepts these runs; reuse and staging report the recorded PR head and judge modifiers from the recorded dispatch rather than the PR's current labels. Older runs without the artifact fall back to the run title and the PR's current labels. On the fork PR, /stage-results <run_id> stages such a run when its recorded head is one of the PR's commits. The PoC also fans out the selected matrix immediately. It does not reproduce run-sweep.yml's canary-first sequencing: no label runs a canary here, and only full-sweep-fail-fast sets fail-fast, so the other two primary labels behave identically.

Reusing an Approved PR Full Sweep

[skip-sweep] skips PR benchmark setup only. Changelog and reuse checks still run. The push-to-main merge-ingest.yml run ignores it.

An authorized maintainer can reuse an eligible completed sweep without keeping a primary sweep label on the PR:

/use <run_id>

Keep the command and required run ID on one line. This pins an eligible completed run-sweep.yml PR run, including failed or cancelled runs with usable results. The run may come from any PR; its commit need not be part of this one.

The legacy /reuse-sweep-run <run_id> remains equivalent. Bare /reuse-sweep-run selects the latest successful eligible run automatically; bare /use is rejected. Both names share authorization, validation, and reactions.

Source validation checks identity and artifacts, not full-matrix coverage. Acceptance does not certify a green full sweep. Verify coverage and pin the run ID when a full sweep is required by the review process.

The latest matching comment across both names by an OWNER, MEMBER, or COLLABORATOR wins. The bot reacts with 👍 after validating the request, or 👎 on rejection. When the pinned run's commit is not part of the PR, it also posts one warning comment for that request and run; reuse still proceeds. Details are in the Actions run summary. Edits replace the bot's old reaction. /use <run_id> also triggers stage-results.yml. Staging requires a primary sweep label on the PR and a source run created while one was applied. A PR without a primary label can instead stage only the run pinned by its latest authorized /use <run_id>, after the same validation that acknowledges the request. It posts a staging comment, or a rejection comment when neither applies. Comments do not trigger or cancel GPU sweeps. Later commits skip a new sweep after changelog/matrix and source-run validation. Merge-time validation remains authoritative; an acknowledgment cannot override expired or invalid artifacts. Reuse is rejected while the PR currently carries evals-only or agentx-fast, checked when /use is acknowledged and at merge (and by merge_with_reuse). On a push these labels skip the reuse gate, so a primary label starts a fresh sweep. Source-run label history is not inspected, so pin only runs produced without them. To force a fresh sweep after a reuse command, remove and re-add the primary sweep label.

uv run --extra workflows python -m infx.workflows.merge_with_reuse <pr-number> is the supported merge path for reuse. It merges main, preserves changelog bytes, fixes an appended XXX PR link, pushes a synchronization commit, waits for checks, then merges.

At merge, merge-ingest.yml ("Merge Ingest") publishes the reused sweep. It runs on pushes to main that change inferencex-e2e/perf-changelog.yaml. Its single ingest job first computes the changelog delta with infx.matrix.plan. For entries with results, it resolves the merge commit's PR, reuse command, and source run with infx.workflows.reuse, and fails before uploading or dispatching anything unless reuse is validly authorized. It then uploads merge-time changelog-metadata and sends one repository_dispatch to InferenceX-app: ingest-agentic-results (with database-target: production) when the delta has agentic entries, otherwise ingest-results. The payload's source-run-id is the reused PR run-sweep.yml run and its merge-run-id is the Merge Ingest run.

For a nonempty delta consisting entirely of retired: true entries, the planner verifies that the selected configs existed in the base and are absent from the head. There are no result artifacts to reuse: merge these PRs directly after review and required checks, without /use or merge_with_reuse. Merge Ingest uploads the validated changelog metadata but skips reuse lookup and database dispatch. Empty or mixed deltas do not receive this exception. Ingest recovery excludes retired entries from mixed changes and rejects retirement-only changes.

The app downloads source artifacts, keeps the newest upload for each exact artifact name, and ingests them with changelog metadata from the merge run. The normal ingestion code skips failed benchmark rows. Benchmark rows and public links retain source-run provenance. Source coverage is authoritative, so later matrix/eval policy changes do not invalidate reuse.

When the app's ingest finishes, on success or failure, it sends ingest-results-completed back with per-run counts (benchmark, eval, trace and power rows: new, duplicate, skipped, unmapped, failed). ingest-results-callback.yml checks that merge-run-id is a Merge Ingest push to main, resolves the PR merged by that commit, and posts one comment per merge run, updated in place on reruns (marker ).

Reuse fails closed when authorized but ineligible or invalid. Pushes to main never run a sweep: run-sweep.yml is PR-only, and without reuse authorization the Merge Ingest run fails and nothing is benchmarked or ingested.

Validation Architecture

The benchmarking system uses a strict validation methodology to ensure correctness at every stage. This is implemented in inferencex-e2e/infx/matrix/validation.py using Pydantic models.

Validation Methodology

The system validates both ends of the configuration pipeline:

Input Validation (Master Configs): Validates the structure of inferencex-e2e/configs/*.yaml files before any processing occurs
Output Validation (Matrix Entries): Validates the generated matrix entries that are passed to workflow templates

This dual-validation approach ensures:

No malformed configurations enter the pipeline
No invalid parameters reach the benchmark workflows
Workflow templates (benchmark-tmpl.yml, benchmark-multinode-tmpl.yml) can assume all inputs are valid, with no runtime validation needed

Input Validation: Master Config Files

Master config files (e.g., nvidia-master.yaml, amd-master.yaml) are validated against strict Pydantic schemas:

SingleNodeMasterConfigEntry: Validates single-node configurations
MultiNodeMasterConfigEntry: Validates multi-node configurations

Each config must specify:

Required fields: image, model, model-prefix, precision, framework, runner, multinode
Sequence length configs with search spaces defining TP, EP, concurrency ranges, etc.
Optional fields like disagg, spec-decoding, dp-attn

Invalid or missing fields raise immediate validation errors before any matrix generation.

Output Validation: Matrix Entries

Generated matrix entries (the actual workflow inputs) are validated against:

SingleNodeMatrixEntry: Matches the inputs expected by benchmark-tmpl.yml
MultiNodeMatrixEntry: Matches the inputs expected by benchmark-multinode-tmpl.yml

These Pydantic models mirror the workflow template input definitions exactly. For example, benchmark-tmpl.yml expects:

inputs:
  runner: required
  image: required
  model: required
  model-prefix: required
  precision: required
  framework: required
  ...

The corresponding SingleNodeMatrixEntry enforces these same fields with appropriate types.

Key Design Principles

No defaults in output validation: Matrix entry models don't set defaults. Missing values must fail validation rather than silently using fallbacks.
extra='forbid': Unknown fields are rejected, preventing typos or deprecated fields from slipping through.
Strict typing: Fields like spec-decoding use Literal["mtp", "draft_model", "none"] to restrict values to known options.
Concurrency validation: The system ensures either conc-list OR conc-start/conc-end is provided, but not both.

Validation Flow

inferencex-e2e/configs/*.yaml
        │
        ▼
┌─────────────────────────┐
│  validate_master_config │  ← Input validation (Pydantic)
└─────────────────────────┘
        │
        ▼
┌─────────────────────────┐
│  infx.matrix.generate   │  ← Matrix generation
└─────────────────────────┘
        │
        ▼
┌─────────────────────────┐
│  validate_matrix_entry  │  ← Output validation (Pydantic)
└─────────────────────────┘
        │
        ▼
  benchmark-tmpl.yml or
  benchmark-multinode-tmpl.yml

Navigation Menu

Repository navigation

README.md

README.md

How to Test Workflows

`full-sweep` Command

Examples

`test-config` Command

Examples

PR Sweep Labels

Changelog Conflict Recovery

Trusted External-Fork Sweep Dispatch (PoC)

Reusing an Approved PR Full Sweep

Validation Architecture

Validation Methodology

Input Validation: Master Config Files

Output Validation: Matrix Entries

Key Design Principles

Validation Flow

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

How to Test Workflows

full-sweep Command

Examples

test-config Command

Examples

PR Sweep Labels

Changelog Conflict Recovery

Trusted External-Fork Sweep Dispatch (PoC)

Reusing an Approved PR Full Sweep

Validation Architecture

Validation Methodology

Input Validation: Master Config Files

Output Validation: Matrix Entries

Key Design Principles

Validation Flow

`full-sweep` Command

`test-config` Command