Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
6e691cd
feat: integrate H3 workflows with the current infx planner
edwingao28 Sep 26, 2026
eba49c7
test: preserve the parent H3 report assertion cleanup
edwingao28 Sep 26, 2026
8247212
fix: use canonical infx entrypoints for H3
edwingao28 Sep 27, 2026
8492e47
merge: reconcile H3 cross-hardware with current main
edwingao28 Sep 28, 2026
1e648b1
fix: point H3 paths at inferencex-e2e after repo reorg
edwingao28 Sep 28, 2026
5cbc665
docs: add Cam/Alec H3 interface handoff after main reconcile
edwingao28 Sep 28, 2026
77f2080
feat: add Wan serving and deployment evidence
edwingao28 Oct 5, 2026
7f1fc18
feat: add the H3 video-generation harness and its native srt-slurm ev…
edwingao28 Oct 17, 2026
a2e37f2
feat: run video-generation points through the native srt-slurm launcher
edwingao28 Oct 17, 2026
0cb4c84
ci: add the video-generation sweep bucket and H3 workflows
edwingao28 Oct 17, 2026
a1a8d4e
test: add the PR #2916 H3 serving spec as a fixture
edwingao28 Oct 17, 2026
d2a5e52
feat: add a shared MiniMax-H3 workload spec with plan hashing
edwingao28 Oct 17, 2026
f9c8154
ci: test and lint the H3 workload spec
edwingao28 Oct 17, 2026
64a0472
docs: add vLLM-Omni serving notes for MiniMax-H3
edwingao28 Oct 17, 2026
460fffa
feat: add Wan vLLM-Omni transport and shared spec
edwingao28 Oct 17, 2026
90f9c57
chore: synchronize Wan with H3 parent
edwingao28 Oct 17, 2026
c0eaf33
docs: include Wan spec dependencies in CPU checks
edwingao28 Oct 17, 2026
7cbb82e
test: remove redundant Wan test narration
edwingao28 Oct 17, 2026
d06ac06
chore: synchronize native video delivery with main
edwingao28 Oct 18, 2026
7eb6747
test: add the PR #2916 H3 serving spec as a fixture
edwingao28 Oct 17, 2026
cda654a
feat: add a shared MiniMax-H3 workload spec with plan hashing
edwingao28 Oct 17, 2026
12d064b
ci: test and lint the H3 workload spec
edwingao28 Oct 17, 2026
6589814
docs: add vLLM-Omni serving notes for MiniMax-H3
edwingao28 Oct 17, 2026
221cd0f
feat: bind H3 engine comparison to native video sweep
edwingao28 Oct 18, 2026
19991ae
chore: reconcile H3 native engine foundation for Wan
edwingao28 Oct 18, 2026
1bd5e98
fix: connect native video sweeps and validate complete artifacts
edwingao28 Oct 18, 2026
85bfbbb
test: add the PR #2916 H3 serving spec as a fixture
edwingao28 Oct 17, 2026
5b02863
feat: add a shared MiniMax-H3 workload spec with plan hashing
edwingao28 Oct 17, 2026
ae4ee4b
ci: test and lint the H3 workload spec
edwingao28 Oct 17, 2026
41579ea
docs: add vLLM-Omni serving notes for MiniMax-H3
edwingao28 Oct 17, 2026
fcc5f08
feat: bind H3 engine comparison to native video sweep
edwingao28 Oct 18, 2026
2936089
feat: add Wan native comparison recipes for both engines
edwingao28 Oct 18, 2026
a205147
chore: align Wan comparison with native video sealing
edwingao28 Oct 18, 2026
95688b2
fix: connect native video sweeps and validate complete artifacts
edwingao28 Oct 18, 2026
964df42
fix: pin vLLM video rows to the tested image digest
edwingao28 Oct 18, 2026
9143bea
fix: exclude video-only sweeps from LLM aggregation
edwingao28 Oct 18, 2026
752a76b
ci: document native video validation job permissions
edwingao28 Oct 18, 2026
b594803
test: add the PR #2916 H3 serving spec as a fixture
edwingao28 Oct 17, 2026
ac9eef0
feat: add a shared MiniMax-H3 workload spec with plan hashing
edwingao28 Oct 17, 2026
dfcf6db
ci: test and lint the H3 workload spec
edwingao28 Oct 17, 2026
4227658
docs: add vLLM-Omni serving notes for MiniMax-H3
edwingao28 Oct 17, 2026
c77faf1
feat: bind H3 engine comparison to native video sweep
edwingao28 Oct 18, 2026
179696b
chore: preserve H3 public branch ancestry for native comparison
edwingao28 Oct 18, 2026
2e14b4b
chore: align H3 comparison with final native sweep gate
edwingao28 Oct 18, 2026
f115373
chore: reconcile Wan comparison with final native sweep gate
edwingao28 Oct 18, 2026
0d7ce1d
chore: align Wan comparison with H3 public branch history
edwingao28 Oct 18, 2026
2c5c085
chore: declare H3 engine rows for native full sweep
edwingao28 Oct 18, 2026
7b60a2b
docs: keep video serving notes factual and links valid
edwingao28 Oct 18, 2026
b441779
chore: inherit H3 full sweep selection for Wan comparison
edwingao28 Oct 18, 2026
c07b403
chore: declare Wan engine rows for native full sweep
edwingao28 Oct 18, 2026
dbe495e
test: align empty collector coverage
edwingao28 Oct 18, 2026
76dfbbf
docs: remove obsolete video translation [skip-sweep]
edwingao28 Oct 18, 2026
48c3783
fix: seal selected native video execution spec
edwingao28 Oct 18, 2026
3f9839c
chore: inherit parent native sweep follow-up
edwingao28 Oct 18, 2026
f0428b7
fix: align Wan service with native power validation
edwingao28 Oct 18, 2026
4bfb66c
chore: inherit selected native execution spec
edwingao28 Oct 18, 2026
334004b
docs: remove private video notes and dead links
edwingao28 Oct 18, 2026
0763804
chore: format selected video binder for CI
edwingao28 Oct 18, 2026
4e8f161
chore: inherit H3 binder formatting for CI
edwingao28 Oct 18, 2026
4475151
fix: export only retained video result guides
edwingao28 Oct 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ usage: python -m infx.matrix.generate full-sweep
[--no-evals | --evals-only] [--all-evals]
[--smoke] [--trim-conc]
[--runner-node-filter RUNNER_NODE_FILTER]
[--scenario-type {fixed-seq-len,agentic-coding} [{fixed-seq-len,agentic-coding} ...]]
[--scenario-type {fixed-seq-len,agentic-coding,video-generation} [{fixed-seq-len,agentic-coding,video-generation} ...]]
[--model-prefix MODEL_PREFIX [MODEL_PREFIX ...]]
[--precision PRECISION [PRECISION ...]]
[--framework FRAMEWORK [FRAMEWORK ...]]
Expand Down Expand Up @@ -109,7 +109,7 @@ usage: python -m infx.matrix.generate test-config
[--no-evals | --evals-only] [--all-evals]
[--smoke] [--trim-conc]
[--runner-node-filter RUNNER_NODE_FILTER]
[--scenario-type {fixed-seq-len,agentic-coding} [{fixed-seq-len,agentic-coding} ...]]
[--scenario-type {fixed-seq-len,agentic-coding,video-generation} [{fixed-seq-len,agentic-coding,video-generation} ...]]
--config-keys CONFIG_KEYS [CONFIG_KEYS ...]
[--conc CONC [CONC ...]]
[--exp-names EXP_NAMES [EXP_NAMES ...]]
Expand Down
47 changes: 37 additions & 10 deletions .github/workflows/benchmark-tmpl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ env:
DSV41_MIN_CUDAGRAPH_CAPTURE_SIZE: '1'
MODEL_DOWNLOAD_LOCK_TIMEOUT: '21600'
ENROOT_IMPORT_TIME_LIMIT: '120'
SALLOC_TIME_LIMIT: '480'
SALLOC_TIME_LIMIT: ${{ inputs.scenario-type == 'video-generation' && '75' || '480' }}
KEEP_LOGS: '0'
IS_MULTINODE: 'false'
REQUIRE_POWER: ${{ inputs.require-power && '1' || '0' }}
Expand All @@ -111,9 +111,9 @@ env:
MODEL: ${{ fromJSON(inputs.config).model }}
THINKING_MODE: thinking_on
MODEL_PREFIX: ${{ fromJSON(inputs.config).model-prefix }}
ISL: ${{ inputs.scenario-type == 'agentic-coding' && '0' || fromJSON(inputs.config).isl }}
OSL: ${{ inputs.scenario-type == 'agentic-coding' && '0' || fromJSON(inputs.config).osl }}
MAX_MODEL_LEN: ${{ inputs.scenario-type == 'agentic-coding' && '0' || fromJSON(inputs.config).max-model-len }}
ISL: ${{ (inputs.scenario-type == 'agentic-coding' || inputs.scenario-type == 'video-generation') && '0' || fromJSON(inputs.config).isl }}
OSL: ${{ (inputs.scenario-type == 'agentic-coding' || inputs.scenario-type == 'video-generation') && '0' || fromJSON(inputs.config).osl }}
MAX_MODEL_LEN: ${{ (inputs.scenario-type == 'agentic-coding' || inputs.scenario-type == 'video-generation') && '0' || fromJSON(inputs.config).max-model-len }}
IMAGE: ${{ fromJSON(inputs.config).image }}
FRAMEWORK: ${{ fromJSON(inputs.config).framework }}
PRECISION: ${{ fromJSON(inputs.config).precision }}
Expand Down Expand Up @@ -143,6 +143,13 @@ env:
DURATION: ${{ inputs.duration }}
AIPERF_EXPERIMENTAL_FAST: ${{ inputs.agentx-fast && '1' || '0' }}
EVAL_LIMIT: ${{ inputs.eval-limit }}
# Video-generation env (one H3 job per row, every concurrency in CONC_LIST). Other rows leave these empty.
GITHUB_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
VIDEO_ARTIFACT_NAME: ${{ inputs.scenario-type == 'video-generation' && format('h3-results-{0}-{1}-{2}', github.run_id, github.run_attempt, fromJSON(inputs.config).recipe-fingerprint) || '' }}
H3_SPEC: ${{ inputs.scenario-type == 'video-generation' && fromJSON(inputs.config).spec || '' }}
H3_SET: ${{ inputs.scenario-type == 'video-generation' && fromJSON(inputs.config).set || '' }}
VIDEO_GPUS: ${{ inputs.scenario-type == 'video-generation' && fromJSON(inputs.config).gpus || '' }}
VIDEO_CONC_LIST: ${{ inputs.scenario-type == 'video-generation' && join(fromJSON(inputs.config).conc-list, ' ') || '' }}
AIPERF_FAILED_REQUEST_THRESHOLD: '0.10'
RESULT_DIR: /workspace/results
PYTHONDONTWRITEBYTECODE: '1'
Expand Down Expand Up @@ -177,14 +184,14 @@ jobs:
) ||
format('[{0}]', toJSON(inputs.runner))
) }}
timeout-minutes: ${{ inputs.runner == 'cluster:h200-dgxc' && fromJSON(inputs.config).model-prefix == 'dsv41flash' && fromJSON(inputs.config).framework == 'sglang' && inputs.scenario-type == 'agentic-coding' && !inputs.eval-only && fromJSON(inputs.config).conc >= 64 && 1470 || 500 }}
timeout-minutes: ${{ inputs.scenario-type == 'video-generation' && 90 || inputs.runner == 'cluster:h200-dgxc' && fromJSON(inputs.config).model-prefix == 'dsv41flash' && fromJSON(inputs.config).framework == 'sglang' && inputs.scenario-type == 'agentic-coding' && !inputs.eval-only && fromJSON(inputs.config).conc >= 64 && 1470 || 500 }}
name: >-
${{ inputs.klaud-run && 'klaud | ' || '' }}p${{ inputs.priority }} | ${{ fromJSON(inputs.config).model-prefix }} ${{ fromJSON(inputs.config).precision }} ${{ inputs.runner }} ${{ fromJSON(inputs.config).framework == 'sglang' && 'sgl' || fromJSON(inputs.config).framework == 'dynamo-sglang' && 'dyn-sgl' || fromJSON(inputs.config).framework == 'sglang-disagg' && 'sgl-disagg' || fromJSON(inputs.config).framework }}
TP${{ fromJSON(inputs.config).tp }}${{ format('{0}', fromJSON(inputs.config).pp) != '' && format('{0}', fromJSON(inputs.config).pp) != '1' && format('/PP{0}', fromJSON(inputs.config).pp) || '' }}${{ format('{0}', fromJSON(inputs.config).dcp-size) != '' && format('{0}', fromJSON(inputs.config).dcp-size) != '1' && format('/DCP{0}', fromJSON(inputs.config).dcp-size) || '' }}${{ format('{0}', fromJSON(inputs.config).pcp-size) != '' && format('{0}', fromJSON(inputs.config).pcp-size) != '1' && format('/PCP{0}', fromJSON(inputs.config).pcp-size) || '' }}${{ format('{0}', fromJSON(inputs.config).ep) != '' && format('{0}', fromJSON(inputs.config).ep) != '1' && format('/EP{0}', fromJSON(inputs.config).ep) || '' }}${{ inputs.dp-attn && '/DPA' || '' }}
${{ inputs.scenario-type == 'video-generation' && format('{0} GPU', fromJSON(inputs.config).gpus) || format('TP{0}', fromJSON(inputs.config).tp) }}${{ format('{0}', fromJSON(inputs.config).pp) != '' && format('{0}', fromJSON(inputs.config).pp) != '1' && format('/PP{0}', fromJSON(inputs.config).pp) || '' }}${{ format('{0}', fromJSON(inputs.config).dcp-size) != '' && format('{0}', fromJSON(inputs.config).dcp-size) != '1' && format('/DCP{0}', fromJSON(inputs.config).dcp-size) || '' }}${{ format('{0}', fromJSON(inputs.config).pcp-size) != '' && format('{0}', fromJSON(inputs.config).pcp-size) != '1' && format('/PCP{0}', fromJSON(inputs.config).pcp-size) || '' }}${{ format('{0}', fromJSON(inputs.config).ep) != '' && format('{0}', fromJSON(inputs.config).ep) != '1' && format('/EP{0}', fromJSON(inputs.config).ep) || '' }}${{ inputs.dp-attn && '/DPA' || '' }}
${{ fromJSON(inputs.config).spec-decoding != 'none' && fromJSON(inputs.config).spec-decoding || '' }}
${{ inputs.scenario-type == 'agentic-coding' && format('{0}', fromJSON(inputs.config).kv-offloading) != '' && fromJSON(inputs.config).kv-offloading != 'none' && format('{0} KV offload', fromJSON(inputs.config).kv-offloading) || '' }}
${{ inputs.scenario-type == 'agentic-coding' && fromJSON(inputs.config).kv-offload-backend.name != 'none' && fromJSON(inputs.config).kv-offload-backend.name != 'default' && format('{0}', fromJSON(inputs.config).kv-offload-backend.name) || '' }}
c${{ fromJSON(inputs.config).conc }}${{ inputs.eval-only && ' | eval-only' || (inputs.run-eval && ' | eval' || '') }}
c${{ inputs.scenario-type == 'video-generation' && join(fromJSON(inputs.config).conc-list, 'x') || fromJSON(inputs.config).conc }}${{ inputs.eval-only && ' | eval-only' || (inputs.run-eval && ' | eval' || '') }}
steps:
- name: Repair root-owned artifacts (pre-run)
if: ${{ inputs.runner == 'cluster:mi355x-amds' }}
Expand Down Expand Up @@ -281,7 +288,13 @@ jobs:
echo "INFERENCEX_E2E_ROOT=$PWD" >> "$GITHUB_ENV"
RESULT_FILENAME=$(python3 -m infx.results.result_filename)
export RESULT_FILENAME
export GPU_COUNT=$((TP * PP_SIZE * PCP_SIZE))
if [[ "$SCENARIO_TYPE" == video-generation ]]; then
export GPU_COUNT="$VIDEO_GPUS"
export CONC_LIST="$VIDEO_CONC_LIST"
echo "CONC_LIST=${CONC_LIST}" >> "$GITHUB_ENV"
else
export GPU_COUNT=$((TP * PP_SIZE * PCP_SIZE))
fi
echo "GPU_COUNT=${GPU_COUNT}" >> "$GITHUB_ENV"

# Export RESULT_FILENAME early so it's available for artifact uploads even if cancelled
Expand Down Expand Up @@ -326,8 +339,17 @@ jobs:
results/aiperf_artifacts \
--failed-request-threshold "$AIPERF_FAILED_REQUEST_THRESHOLD"

- name: Upload video results
id: upload-video-results
if: ${{ always() && inputs.scenario-type == 'video-generation' && env.RESULT_FILENAME != '' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ env.VIDEO_ARTIFACT_NAME }}
path: ${{ env.INFERENCEX_E2E_ROOT }}/h3-results
if-no-files-found: error

- name: Process result
if: ${{ always() && env.RESULT_FILENAME != '' && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' }}
if: ${{ always() && env.RESULT_FILENAME != '' && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' && inputs.scenario-type != 'video-generation' }}
env:
RUNNER_TYPE: ${{ inputs.runner }}
PYTHONPATH: ${{ github.workspace }}/.result-tooling/inferencex-e2e
Expand All @@ -347,7 +369,7 @@ jobs:

- name: Upload result
id: upload-result
if: ${{ success() && env.RESULT_FILENAME != '' && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' }}
if: ${{ success() && env.RESULT_FILENAME != '' && !inputs.eval-only && inputs.scenario-type != 'agentic-coding' && inputs.scenario-type != 'video-generation' }}
continue-on-error: true
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down Expand Up @@ -473,6 +495,7 @@ jobs:
${{ env.INFERENCEX_E2E_ROOT }}/results/*.log
${{ env.INFERENCEX_E2E_ROOT }}/results/*_config.json
${{ env.INFERENCEX_E2E_ROOT }}/srt-single-node-logs.tar.gz
${{ env.INFERENCEX_E2E_ROOT }}/srt-video-logs.tar.gz
${{ env.INFERENCEX_E2E_ROOT }}/srt-single-node-submission.json
${{ env.INFERENCEX_E2E_ROOT }}/srt-slurm-sha.txt
${{ env.INFERENCEX_E2E_ROOT }}/srt-setup.log
Expand All @@ -488,9 +511,13 @@ jobs:
${{ env.INFERENCEX_E2E_ROOT }}/${{ env.RESULT_FILENAME }}.json
${{ env.INFERENCEX_E2E_ROOT }}/agg_${{ env.RESULT_FILENAME }}.json
${{ env.INFERENCEX_E2E_ROOT }}/power_validation_${{ env.RESULT_FILENAME }}.json
${{ env.INFERENCEX_E2E_ROOT }}/${{ env.RESULT_FILENAME }}_conc*.json
${{ env.INFERENCEX_E2E_ROOT }}/power_validation_${{ env.RESULT_FILENAME }}_conc*.json
${{ env.INFERENCEX_E2E_ROOT }}/results/agentic_power_timezone_offset.txt
${{ env.INFERENCEX_E2E_ROOT }}/LOGS/power/**
${{ env.INFERENCEX_E2E_ROOT }}/LOGS/agentic/**/agentic_power_concurrency_*.json
${{ env.INFERENCEX_E2E_ROOT }}/LOGS/h3/**/benchmark_concurrency_*.json
${{ env.INFERENCEX_E2E_ROOT }}/LOGS/h3/server/*.json
${{ env.INFERENCEX_E2E_ROOT }}/LOGS/${{ env.RESULT_FILENAME }}.json
${{ env.INFERENCEX_E2E_ROOT }}/power-producer-sha.txt
${{ env.INFERENCEX_E2E_ROOT }}/exporter-image.sha256
Expand Down
33 changes: 30 additions & 3 deletions .github/workflows/e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ jobs:
agentic-eval-config: ${{ steps.get-jobs.outputs.agentic-eval-config }}
multi-node-agentic-config: ${{ steps.get-jobs.outputs.multi-node-agentic-config }}
multi-node-agentic-eval-config: ${{ steps.get-jobs.outputs.multi-node-agentic-eval-config }}
video-config: ${{ steps.get-jobs.outputs.video-config }}
steps:
- name: Checkout code (ref)
id: checkout-ref
Expand Down Expand Up @@ -317,9 +318,10 @@ jobs:
AGENTIC_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' not in x and x.get('run-eval', False)]))" | score_matrix agentic-eval)
MULTI_AGENTIC=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' in x and not x.get('eval-only', False)]))" | score_matrix multi-agentic)
MULTI_AGENTIC_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'agentic-coding' and 'prefill' in x and x.get('run-eval', False)]))" | score_matrix multi-agentic-eval)
SINGLE=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))" | score_matrix single)
VIDEO=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if x.get('scenario-type') == 'video-generation']))" | score_matrix video)
SINGLE=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') not in ('agentic-coding', 'video-generation') and not x.get('eval-only', False)]))" | score_matrix single)
MULTI=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('scenario-type') != 'agentic-coding' and not x.get('eval-only', False)]))" | score_matrix multi)
EVALS=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') != 'agentic-coding' and x.get('run-eval', False)]))" | score_matrix eval)
EVALS=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' not in x and x.get('scenario-type') not in ('agentic-coding', 'video-generation') and x.get('run-eval', False)]))" | score_matrix eval)
MULTI_EVAL=$(echo "$CONFIG_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps([x for x in d if 'prefill' in x and x.get('scenario-type') != 'agentic-coding' and x.get('run-eval', False)]))" | score_matrix multi-eval)
{
echo "agentic-config=$AGENTIC"
Expand All @@ -330,6 +332,7 @@ jobs:
echo "multi-node-config=$MULTI"
echo "eval-config=$EVALS"
echo "multi-node-eval-config=$MULTI_EVAL"
echo "video-config=$VIDEO"
} >> "$GITHUB_OUTPUT"

test-sweep-multi-node:
Expand Down Expand Up @@ -500,6 +503,30 @@ jobs:
ref: ${{ needs.get-jobs.outputs.benchmark-ref }}
test-name: ${{ inputs.test-name }}

test-sweep-video:
needs: get-jobs
if: ${{ needs.get-jobs.outputs.video-config != '' && needs.get-jobs.outputs.video-config != '[]' }}
uses: $/.github/workflows/benchmark-tmpl.yml
name: video /
strategy:
fail-fast: ${{ inputs.fail-fast }}
matrix:
config: ${{ fromJson(needs.get-jobs.outputs.video-config) }}
secrets:
INFERENCEX_OFFICIAL_RO_HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
with:
require-power: ${{ inputs.require-power }}
config: ${{ toJSON(matrix.config) }}
klaud-run: ${{ inputs.klaud-run }}
runner: ${{ matrix.config.runner }}
test-name: ${{ inputs.test-name }}
priority: ${{ matrix.config.priority }}
queue-token: ${{ matrix.config['queue-token'] }}
dp-attn: ${{ matrix.config.dp-attn }}
run-eval: false
scenario-type: video-generation
ref: ${{ needs.get-jobs.outputs.benchmark-ref }}

test-sweep-single-node:
needs: get-jobs
if: ${{ needs.get-jobs.outputs.single-node-config != '' && needs.get-jobs.outputs.single-node-config != '[]' }}
Expand Down Expand Up @@ -568,7 +595,7 @@ jobs:

collect-job-events:
name: collect-job-events
needs: [get-jobs, test-sweep-multi-node, test-sweep-multi-node-evals, test-sweep-agentic, test-sweep-agentic-evals, test-sweep-multi-node-agentic, test-sweep-multi-node-agentic-evals, test-sweep-single-node, test-sweep-evals]
needs: [get-jobs, test-sweep-multi-node, test-sweep-multi-node-evals, test-sweep-agentic, test-sweep-agentic-evals, test-sweep-multi-node-agentic, test-sweep-multi-node-agentic-evals, test-sweep-single-node, test-sweep-evals, test-sweep-video]
if: ${{ !cancelled() && needs.get-jobs.result == 'success' }}
permissions:
actions: read # List the run's jobs to record the jobs that wrote no event.
Expand Down
66 changes: 66 additions & 0 deletions .github/workflows/h3-fidelity.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: H3 Retained Media Fidelity

on:
workflow_dispatch:
inputs:
source-run-ids:
description: "Two original H3 serving CI runs, baseline then candidate"
required: true
type: string

permissions:
contents: read
actions: read # Verify original serving runs and download their retained media artifacts.

concurrency:
group: h3-fidelity-${{ inputs.source-run-ids }}
cancel-in-progress: false

jobs:
compare:
name: Compare original C1 video and audio on CPU
runs-on: ubuntu-latest
timeout-minutes: 30
env:
GH_TOKEN: ${{ github.token }}
H3_SOURCE_RUN_IDS: ${{ inputs.source-run-ids }}
PYTHONPATH: ${{ github.workspace }}/inferencex-e2e
steps:
- name: Authorize manual repository execution
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
if (context.eventName !== 'workflow_dispatch' ||
context.repo.owner !== 'SemiAnalysisAI' || context.repo.repo !== 'InferenceX' ||
!context.ref.startsWith('refs/heads/')) {
throw new Error('H3 fidelity requires manual dispatch of an InferenceX repository branch.');
}
for (const username of new Set([context.actor, process.env.GITHUB_TRIGGERING_ACTOR])) {
const {data} = await github.rest.repos.getCollaboratorPermissionLevel({
...context.repo, username,
});
if (!['write', 'maintain', 'admin'].includes(data.permission)) {
throw new Error(username + ' must have write, maintain, or admin permission.');
}
}
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.sha }}
persist-credentials: false
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Verify sealed sources and compare every C1 media pair
run: |
source inferencex-e2e/benchmarks/check_env.sh
check_env_vars GH_TOKEN H3_SOURCE_RUN_IDS PYTHONPATH RUNNER_TEMP
uv run --no-project --python 3.12 --with 'av==16.1.0' --with 'numpy==2.3.5' \
python experimental/video-generation/compare_serving_ci.py \
--source-run-ids "$H3_SOURCE_RUN_IDS" --output "$RUNNER_TEMP/h3-fidelity"
- name: Preserve comparisons, original media and any failure
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: h3-fidelity-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/h3-fidelity/
if-no-files-found: error
compression-level: 0
retention-days: 14
Loading
Loading