From 74bd3e2abdcd17f21074324e5145c0178f26149c Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Tue, 15 Sep 2026 16:59:38 -0700 Subject: [PATCH 001/104] feat: add read-only H3 performance history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增只读 H3 性能历史。复用已发布产物索引,保留原始执行与导出身份、失败和资格状态,提供英中筛选与原结果直链,并以保留数据验证桌面和移动端交互。未创建新的性能测量或发布结果。 --- docs/index.md | 1 + docs/video-history.md | 31 ++ .../cypress/component/video-ci-runs.cy.tsx | 70 +++ packages/app/cypress/e2e/navigation.cy.ts | 35 +- .../app/cypress/fixtures/api/_manifest.json | 7 + .../cypress/fixtures/api/video-history.json | 361 ++++++++++++++++ .../app/src/app/api/video-runs/route.test.ts | 14 + packages/app/src/app/api/video-runs/route.ts | 34 +- .../video-benchmark/VideoCIRuns.tsx | 39 +- .../video-benchmark/VideoHistory.tsx | 399 ++++++++++++++++++ .../video-benchmark/history.test.ts | 86 ++++ .../src/components/video-benchmark/history.ts | 146 +++++++ packages/app/src/lib/api-route-catalog.ts | 6 +- packages/app/src/lib/video-storage.test.ts | 55 ++- packages/app/src/lib/video-storage.ts | 57 +++ 15 files changed, 1321 insertions(+), 20 deletions(-) create mode 100644 docs/video-history.md create mode 100644 packages/app/cypress/fixtures/api/video-history.json create mode 100644 packages/app/src/components/video-benchmark/VideoHistory.tsx create mode 100644 packages/app/src/components/video-benchmark/history.test.ts create mode 100644 packages/app/src/components/video-benchmark/history.ts diff --git a/docs/index.md b/docs/index.md index 73e2c2b43..9e05ce715 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,6 +13,7 @@ Design rationale and non-obvious conventions. See [CLAUDE.md](../CLAUDE.md) for - [Architecture](./architecture.md) — Why client-first, route navigation, URL state, provider nesting, server-side caching (unstable_cache + blob), in-memory client cache, color system, analytics enforcement - [D3 Charts](./d3-charts.md) — Why 4 effects, in-place mutation, refs for zoom, rAF throttling, HTML tooltips, Pareto directions, gradient labels +- [Video History](./video-history.md) — Read-only published H3 catalog, metric boundaries, sharing, and local retained-data replay - [Data Pipeline](./data-pipeline.md) — DB schema reasoning, ETL design, transform pipeline, spline method choice, normalizer resolution order (model/GPU/framework) - [Pitfalls](./pitfalls.md) — Failure modes: token type consistency, schema evolution, empty objects, zoom loss, stale closures, disaggregated metrics, negative splines, date stamping, ref stability, cost inheritance - [GPU Specs](./gpu-specs.md) — Unit conventions, topology invariants, SVG layout rationale, hardware gotchas diff --git a/docs/video-history.md b/docs/video-history.md new file mode 100644 index 000000000..0240ea047 --- /dev/null +++ b/docs/video-history.md @@ -0,0 +1,31 @@ +# Video performance history + +`/video` and `/zh/video` default to **Performance history**. Explicit run/artifact/source/cell URLs retain the existing video viewer. `?view=results` opens the CI browser; `?view=history` restores history. History filters use `history-hardware`, `history-concurrency`, and `history-query` in the URL. + +## Read path + +`GET /api/video-runs?format=history&page=1` returns a versioned UI projection of existing published H3 indexes. It lists `h3-video-media/v1/runs/` metadata, validates pathname identities, sorts by index publication time (artifact ID breaks ties), then reads at most ten indexes for one page. It neither downloads media nor calls the publication path. The repository visibility check still applies. This UI transport is not a stable public benchmark API. + +The projection groups all cells under the original source inside one artifact entry. It reuses `storedBundle`, `servingCells`, `storedFidelityBundle`, and the existing tradeoff calculations. Invalid or unavailable indexes remain explicit error entries. Original execution time comes from `ci.started_at`; index publication time is separate and must never be substituted for execution time. Original run IDs and export run IDs remain distinct. + +## Evidence boundaries + +- Only descriptive observations are shown: this version has no selected matched baseline and computes no before/after delta. +- Valid/completed/scheduled/failed counts stay separate. Missing or invalid metrics stay null. P90 retains the existing per-cell ten-sample display floor. +- Latency and throughput use the submission-to-downloaded-media window. Energy uses the recorded generation window and participating GPU boards. Board energy is not facility energy. +- Fidelity, calibration and release qualification are displayed independently. Stored/published does not mean qualified. +- Filters apply to loaded pages. The user can load older pages without losing already loaded entries. Global execution-time sorting and an editorial change/PR catalog require a later materialized catalog; the current implementation scans index metadata only, never all full indexes. + +## Local replay + +`E2E_FIXTURES=1` serves `cypress/fixtures/api/video-history.json` through the existing fixture loader and checksum manifest. The fixture is a metadata-only projection of retained H100/H200/B200 observations and two fidelity artifacts; it contains no infrastructure paths or video bytes. Unknown publication timestamps remain null. The response header `X-VideoGenX-Replay: 1` causes a visible replay notice. These are historical retained observations, not newly measured results or a new publication. + +## 中文说明 + +`/video` 与 `/zh/video` 默认打开“性能历史”。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 进入 CI 浏览器。历史筛选条件保存在 URL,刷新后恢复。 + +历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。 + +当前展示观测值,未选择匹配基线,也不计算 before/after。延迟、吞吐量使用提交到下载完成的窗口;板卡能耗使用记录的生成窗口。保真度、阈值校准、发布验收单独展示。筛选仅作用于已加载的页面,可继续加载较早结果。 + +`E2E_FIXTURES=1` 使用保留结果的脱敏元数据回放,遵循现有 fixture 校验和清单,并显示回放提示。未知发布时间保持为空;这不是新测量或新发布。 diff --git a/packages/app/cypress/component/video-ci-runs.cy.tsx b/packages/app/cypress/component/video-ci-runs.cy.tsx index aaf4429ac..f6d9d9f72 100644 --- a/packages/app/cypress/component/video-ci-runs.cy.tsx +++ b/packages/app/cypress/component/video-ci-runs.cy.tsx @@ -2,12 +2,82 @@ import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.share import VideoCIRuns from '@/components/video-benchmark/VideoCIRuns'; import ResultPower from '@/components/video-benchmark/ResultPower'; import VideoSelect from '@/components/video-benchmark/VideoSelect'; +import { videoHistoryEntry } from '@/components/video-benchmark/history'; import { servingArtifact, videoRun as run } from '../support/video-artifacts'; describe('H3 automatic CI viewer (synthetic API fixtures)', () => { beforeEach(() => { cy.intercept('GET', '**/api/video-runs*format=media', { statusCode: 204 }); + cy.window().then((win) => + win.history.replaceState(null, '', `${win.location.pathname}?view=results`), + ); + }); + it('defaults to published history and keeps filters and source selection in the shared URL', () => { + const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); cy.window().then((win) => win.history.replaceState(null, '', win.location.pathname)); + cy.intercept('GET', '/api/video-runs?format=history&page=1', { + schemaVersion: 1, + entries: [entry], + nextPage: null, + }).as('history'); + cy.intercept('GET', '/api/video-runs?page=1', () => { + throw new Error('Published discovery must not scan recent Actions'); + }); + cy.intercept('GET', '/api/video-runs?run=123', { + run: run(123, 'success'), + artifacts: [entry.artifact], + }); + cy.intercept('GET', '**/api/video-runs*format=media', servingArtifact()); + cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); + cy.mount( + + + , + ); + cy.wait('@history'); + cy.get('[data-testid="video-history-observation"]').should('have.length', 3); + cy.get('[data-testid="video-history"]').should('contain', 'no matched version baseline'); + cy.get('[aria-label="Client concurrency"]').click(); + cy.contains('[role="option"]', 'C4').click(); + cy.get('[data-testid="video-history-observation"]') + .should('have.length', 1) + .and('contain', '300 / —'); + cy.location('search').should('contain', 'history-concurrency=4'); + cy.get('[data-testid="video-history-observation"] button').click(); + cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C4'); + cy.location('search').should('contain', 'cell=c4').and('contain', 'history-concurrency=4'); + cy.contains('button', 'Performance history').click(); + cy.get('[data-testid="video-history-observation"]').should('have.length', 1); + }); + it('renders retained metrics and missing data in Chinese within a mobile viewport', () => { + cy.viewport(390, 844); + const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); + cy.window().then((win) => + win.history.replaceState( + null, + '', + `${win.location.pathname}?view=history&history-concurrency=1`, + ), + ); + cy.intercept('GET', '/api/video-runs?format=history&page=1', { + schemaVersion: 1, + entries: [entry], + nextPage: null, + }); + cy.mount( + + + , + ); + cy.get('[data-testid="video-history-observation"]') + .should('have.length', 1) + .and('contain', '120 / —') + .and('contain', '168') + .and('contain', '已完成'); + cy.contains('尚未选择匹配的版本基线').should('be.visible'); + cy.get('[data-testid="video-history"]').then(($history) => + expect($history[0].getBoundingClientRect().right).to.be.at.most(390), + ); }); it('keeps an open long-label menu within mobile bounds and supports keyboard selection', () => { cy.viewport(390, 844); diff --git a/packages/app/cypress/e2e/navigation.cy.ts b/packages/app/cypress/e2e/navigation.cy.ts index aa088182b..4d7232900 100644 --- a/packages/app/cypress/e2e/navigation.cy.ts +++ b/packages/app/cypress/e2e/navigation.cy.ts @@ -1,5 +1,6 @@ import { SUPPLEMENTAL_BENCHMARK_ROWS } from '../../src/lib/supplemental-benchmarks'; import { OVERLAY_RUN_ID, OVERLAY_RUN_URL } from '../support/overlay-fixtures'; +import { videoHistoryEntry } from '../../src/components/video-benchmark/history'; import { servingArtifact, videoRun } from '../support/video-artifacts'; const datum = (el: Element) => (el as Element & { __data__: { x: number; y: number } }).__data__; @@ -375,6 +376,34 @@ describe('TPUv7 results', { testIsolation: true }, () => { describe('H3 video artifact viewer', () => { beforeEach(() => { cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + cy.intercept('GET', '/api/video-runs?format=history&page=*', { + schemaVersion: 1, + entries: [], + nextPage: null, + }); + }); + it('opens old published entries by default and restores history filters after reload', () => { + const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); + cy.intercept('GET', '/api/video-runs?format=history&page=1', { + schemaVersion: 1, + entries: [entry], + nextPage: null, + }).as('history'); + cy.visit('/video'); + cy.wait('@history'); + cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); + cy.get('[data-testid="video-history-observation"]').should('have.length', 3); + cy.get('[aria-label="Client concurrency"]').click(); + cy.contains('[role="option"]', 'C2').click(); + cy.get('[data-testid="video-history-observation"]') + .should('have.length', 1) + .and('contain', 'C2'); + cy.reload(); + cy.get('[data-testid="video-history-observation"]') + .should('have.length', 1) + .and('contain', 'C2') + .and('contain', '240 / —'); + cy.location('search').should('contain', 'history-concurrency=2'); }); it('shares a comparison built by opening two CI runs through the page', () => { for (const saved of [ @@ -451,7 +480,7 @@ describe('H3 video artifact viewer', () => { }); it('uses the shared unlock for navigation and keeps an empty viewer free of sample results', () => { cy.viewport(1440, 1000); - cy.visit('/video', { + cy.visit('/video?view=results', { onBeforeLoad(win) { win.localStorage.removeItem('inferencex-feature-gate'); }, @@ -465,14 +494,14 @@ describe('H3 video artifact viewer', () => { cy.get('head meta[name="robots"]').should('have.attr', 'content', 'noindex, nofollow'); }); it('shows a recoverable load error and the Chinese empty state', () => { - cy.visit('/video'); + cy.visit('/video?view=results'); cy.contains('summary', 'Local artifact tools').click(); cy.get('[data-testid="video-benchmark"]').contains('summary', 'Manifest URL').click(); cy.get('input[aria-label="Manifest URL"]').type('https://example.com/wrong.json'); cy.contains('button', 'Load manifest').click(); cy.get('[role="alert"]').should('contain', 'Could not load this bundle'); cy.get('video[data-role]').should('not.exist'); - cy.visit('/zh/video'); + cy.visit('/zh/video?view=results'); cy.get('[data-testid="video-ci-runs"]').should('contain', '本页 GitHub 历史中没有 H3 运行'); cy.get('head meta[name="robots"]').should('have.attr', 'content', 'noindex, nofollow'); }); diff --git a/packages/app/cypress/fixtures/api/_manifest.json b/packages/app/cypress/fixtures/api/_manifest.json index bf77ec673..88a0b26ad 100644 --- a/packages/app/cypress/fixtures/api/_manifest.json +++ b/packages/app/cypress/fixtures/api/_manifest.json @@ -92,6 +92,13 @@ "sha256": "f28d2220e5246ad70113e38e49394bc7a0bfe9afba8cedf08b90e39042e53357", "source": null, "topLevel": "object" + }, + "video-history": { + "bytes": 14506, + "capturedAt": "2026-09-15T23:50:12.033Z", + "sha256": "278dbafd690465ea01d0a05b6c972751a1a2a277a6d1b43362407abd7d42ed2c", + "source": "Retained public H3 metadata replay from original runs; publication time unavailable. No new benchmark or publication.", + "topLevel": "object" } } } diff --git a/packages/app/cypress/fixtures/api/video-history.json b/packages/app/cypress/fixtures/api/video-history.json new file mode 100644 index 000000000..45be3d3fe --- /dev/null +++ b/packages/app/cypress/fixtures/api/video-history.json @@ -0,0 +1,361 @@ +{ + "schemaVersion": 1, + "entries": [ + { + "id": "34336635063.10103071687", + "runId": "34336635063", + "artifact": { + "id": 10103071687, + "name": "h3-video-34336635063-1", + "expired": false, + "size_in_bytes": 118562850, + "digest": "sha256:575c0016fa96f9fa080706285611509db6f2d72be5f8ffd0248cda3b91f2de42", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34336635063", + "sha256": "7df99e50c015669f4dd9ef62549317870f197cfe0fc5d163cf8bf0fb3b88dd5e", + "sourceSha": "25de33c57f5c2ac0189c3f3a8de396057266dfb9", + "hardware": "H100", + "execution": "complete", + "observedAt": "2026-09-09T09:47:33.704519+00:00", + "kind": "observation", + "observations": [ + { + "id": "7df99e50c015669f4dd9ef62549317870f197cfe0fc5d163cf8bf0fb3b88dd5e:c1", + "cell": "c1", + "hardware": "NVIDIA H100 80GB HBM3", + "concurrency": 1, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 167.33339739358053, + "p90": 168.11354550393298, + "clipsGpuHour": 5.3741390883412175, + "energyKj": 431.14963489170344 + }, + { + "id": "7df99e50c015669f4dd9ef62549317870f197cfe0fc5d163cf8bf0fb3b88dd5e:c2", + "cell": "c2", + "hardware": "NVIDIA H100 80GB HBM3", + "concurrency": 2, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 334.78318604291417, + "p90": 336.9140758761205, + "clipsGpuHour": 5.368742995784034, + "energyKj": 431.32628015924786 + } + ], + "fidelity": null, + "calibration": null, + "releaseQualified": false, + "error": null + } + ], + "error": null + }, + { + "id": "34344378350.10106004321", + "runId": "34344378350", + "artifact": { + "id": 10106004321, + "name": "h3-video-34344378350-1", + "expired": false, + "size_in_bytes": 61499259, + "digest": "sha256:a0bd28cc430c443c1be086bb33f16636a04e51acb0d92f554228e1f1b7409329", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34344378350", + "sha256": "98206c73d05226c162cad14f456d7671640cbd1d090945f456f5427713fdde7f", + "sourceSha": "b04b09c505cfec39f71a7971b28c8af6c93882c9", + "hardware": "H100", + "execution": "complete", + "observedAt": "2026-09-09T12:11:43.125736+00:00", + "kind": "observation", + "observations": [ + { + "id": "98206c73d05226c162cad14f456d7671640cbd1d090945f456f5427713fdde7f:c4", + "cell": "c4", + "hardware": "NVIDIA H100 80GB HBM3", + "concurrency": 4, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 669.2438137787394, + "p90": 673.7058754772879, + "clipsGpuHour": 5.368345559881099, + "energyKj": 430.80324425396446 + } + ], + "fidelity": null, + "calibration": null, + "releaseQualified": false, + "error": null + } + ], + "error": null + }, + { + "id": "34342452354.10107476604", + "runId": "34342452354", + "artifact": { + "id": 10107476604, + "name": "h3-video-34342452354-1", + "expired": false, + "size_in_bytes": 175903798, + "digest": "sha256:ac85510c000f69098db37149ae56b48964563c7a1dc4b26a31f71317b8b3599a", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34342452354", + "sha256": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35", + "sourceSha": "6ea0bf8750d97b2f89a913536dd0dc4aad5bf74e", + "hardware": "H200", + "execution": "complete", + "observedAt": "2026-09-09T10:52:07.062086+00:00", + "kind": "observation", + "observations": [ + { + "id": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35:c1", + "cell": "c1", + "hardware": "NVIDIA H200", + "concurrency": 1, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 150.61379817902343, + "p90": 151.10486977093387, + "clipsGpuHour": 5.973235869122123, + "energyKj": 410.954300515549 + }, + { + "id": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35:c2", + "cell": "c2", + "hardware": "NVIDIA H200", + "concurrency": 2, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 300.09254947002046, + "p90": 300.1774324030848, + "clipsGpuHour": 5.997056099146397, + "energyKj": 409.91272507134966 + }, + { + "id": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35:c4", + "cell": "c4", + "hardware": "NVIDIA H200", + "concurrency": 4, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 601.2122122829896, + "p90": 601.8977232379839, + "clipsGpuHour": 5.986775861646954, + "energyKj": 413.58283766782114 + } + ], + "fidelity": null, + "calibration": null, + "releaseQualified": false, + "error": null + } + ], + "error": null + }, + { + "id": "34341996789.10104690935", + "runId": "34341996789", + "artifact": { + "id": 10104690935, + "name": "h3-video-34341996789-1", + "expired": false, + "size_in_bytes": 163563283, + "digest": "sha256:6b05cafd4c014e02c904f65c0e0d2f3d9a5b541ddc4f030372ed08d4a588c002", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34341996789", + "sha256": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82", + "sourceSha": "6ea0bf8750d97b2f89a913536dd0dc4aad5bf74e", + "hardware": "B200", + "execution": "complete", + "observedAt": "2026-09-09T10:47:03.922884+00:00", + "kind": "observation", + "observations": [ + { + "id": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82:c1", + "cell": "c1", + "hardware": "NVIDIA B200", + "concurrency": 1, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 77.94495720259147, + "p90": 78.31597462110221, + "clipsGpuHour": 11.52605424609145, + "energyKj": 301.08754501100987 + }, + { + "id": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82:c2", + "cell": "c2", + "hardware": "NVIDIA B200", + "concurrency": 2, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 153.6016560649732, + "p90": 154.1683327689534, + "clipsGpuHour": 11.691563929313219, + "energyKj": 300.10551714629605 + }, + { + "id": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82:c4", + "cell": "c4", + "hardware": "NVIDIA B200", + "concurrency": 4, + "runtime": "71de97b264b04dcd514cf904003028aefe9775c8", + "workload": "1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · seed 11, 11, 29, 47, 29, 29, 29, 47, 83, 11, 47, 83, 47, 11, 11, 83, 83, 29, 83, 47 · A single continuous medium close-up of an adult presenter facing the camera in a", + "model": "MiniMaxAI/MiniMax-H3", + "status": "complete", + "valid": 20, + "completed": 20, + "scheduled": 20, + "failed": 0, + "samples": 20, + "p50": 307.28381997847464, + "p90": 308.79631440097, + "clipsGpuHour": 11.69172635792307, + "energyKj": 299.96858693604275 + } + ], + "fidelity": null, + "calibration": null, + "releaseQualified": false, + "error": null + } + ], + "error": null + }, + { + "id": "34410724145.10127189553", + "runId": "34410724145", + "artifact": { + "id": 10127189553, + "name": "h3-fidelity-34410724145-1", + "expired": false, + "size_in_bytes": 47838865, + "digest": "sha256:e8f364dae253274e954003fd06b177b17215381520bbc5924a10607839873479", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34410724145", + "sha256": "c40cc39acd26f046f84b307d7f771ed38f28d96ba21860d6679bba7ba3e7cf29", + "sourceSha": "df5e5cd434ef262cac029925d8782e4290ea6d16", + "hardware": "", + "execution": "", + "observedAt": null, + "kind": "fidelity", + "observations": [], + "fidelity": "fail", + "calibration": "uncalibrated", + "releaseQualified": false, + "error": null + } + ], + "error": null + }, + { + "id": "34410729828.10127190894", + "runId": "34410729828", + "artifact": { + "id": 10127190894, + "name": "h3-fidelity-34410729828-1", + "expired": false, + "size_in_bytes": 47585397, + "digest": "sha256:bb701c77431bb999181d74cb99e4d957da13a6ff45ba927c3f89cf5add49ef69", + "stored": true + }, + "publishedAt": null, + "sources": [ + { + "id": "34410729828", + "sha256": "c2f6a036328e4b90fa1b364b16ca10353baa7eb2fef2d71571a1ab1465fc5382", + "sourceSha": "df5e5cd434ef262cac029925d8782e4290ea6d16", + "hardware": "", + "execution": "", + "observedAt": null, + "kind": "fidelity", + "observations": [], + "fidelity": "fail", + "calibration": "uncalibrated", + "releaseQualified": false, + "error": null + } + ], + "error": null + } + ], + "nextPage": null +} diff --git a/packages/app/src/app/api/video-runs/route.test.ts b/packages/app/src/app/api/video-runs/route.test.ts index 24c95f95a..a24935e91 100644 --- a/packages/app/src/app/api/video-runs/route.test.ts +++ b/packages/app/src/app/api/video-runs/route.test.ts @@ -2,6 +2,7 @@ import { NextRequest } from 'next/server'; import { afterEach, describe, expect, it, vi } from 'vitest'; import { GET } from './route'; import { + publishedVideoHistory, readStoredArtifact, storedArtifacts, storeVideoArtifact, @@ -10,6 +11,9 @@ import { vi.mock('@/lib/video-storage', () => ({ videoStorageEnabled: vi.fn(() => false), + publishedVideoHistory: vi.fn(() => + Promise.resolve({ schemaVersion: 1, entries: [], nextPage: null }), + ), storedArtifacts: vi.fn(() => Promise.resolve([])), readStoredArtifact: vi.fn(), storeVideoArtifact: vi.fn(), @@ -22,6 +26,7 @@ afterEach(() => { fetchMock.mockReset(); vi.mocked(videoStorageEnabled).mockReturnValue(false); vi.mocked(storedArtifacts).mockResolvedValue([]); + vi.mocked(publishedVideoHistory).mockClear(); vi.mocked(readStoredArtifact).mockReset(); vi.mocked(storeVideoArtifact).mockClear(); vi.unstubAllEnvs(); @@ -29,6 +34,15 @@ afterEach(() => { }); describe('H3 CI artifact access', () => { + it('discovers old published results without scanning unrelated recent Actions or publishing', async () => { + fetchMock.mockResolvedValueOnce(response({ private: false })); + const result = await GET(request('?format=history&page=2')); + expect(await result.json()).toEqual({ schemaVersion: 1, entries: [], nextPage: null }); + expect(publishedVideoHistory).toHaveBeenCalledWith(2); + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(storeVideoArtifact).not.toHaveBeenCalled(); + }); + it('discovers benchmark runs without including branch-named unit jobs', async () => { fetchMock.mockResolvedValueOnce(response({ private: false })).mockResolvedValueOnce( response({ diff --git a/packages/app/src/app/api/video-runs/route.ts b/packages/app/src/app/api/video-runs/route.ts index c722cebd8..1b0d6189c 100644 --- a/packages/app/src/app/api/video-runs/route.ts +++ b/packages/app/src/app/api/video-runs/route.ts @@ -6,7 +6,10 @@ import { SITE_URL, } from '@semianalysisai/inferencex-constants'; import { getGithubToken } from '@/lib/github-artifacts'; +import { loadFixture } from '@/lib/test-fixtures'; +import type { VideoHistoryPage } from '@/components/video-benchmark/history'; import { + publishedVideoHistory, readStoredArtifact, storedArtifacts, storeVideoArtifact, @@ -38,13 +41,26 @@ export async function GET(request: NextRequest) { const runId = query.get('run'); const artifactId = query.get('artifact'); const page = query.get('page') ?? '1'; + const history = query.get('format') === 'history'; const publishedOnly = query.get('format') === 'published'; const media = query.get('format') === 'media' || publishedOnly; - if ((runId && !id(runId)) || (artifactId && (!runId || !id(artifactId))) || !id(page)) + if ( + (runId && !id(runId)) || + (artifactId && (!runId || !id(artifactId))) || + !id(page) || + (history && (runId || artifactId || !Number.isSafeInteger(Number(page)))) + ) return NextResponse.json( { error: 'Invalid CI run, artifact or page' }, { status: 400, headers }, ); + if (history && process.env.E2E_FIXTURES === '1') { + const fixture = loadFixture('video-history'); + return NextResponse.json( + Number(page) === 1 ? fixture : { schemaVersion: 1, entries: [], nextPage: null }, + { headers: { ...headers, 'X-VideoGenX-Replay': '1' } }, + ); + } const deadline = AbortSignal.timeout(270000); try { if ( @@ -54,6 +70,7 @@ export async function GET(request: NextRequest) { ) { // Production checks repository visibility; local reads must never publish on a cache miss. const url = new URL('/api/video-runs', SITE_URL); + if (history) url.searchParams.set('format', 'history'); if (runId) url.searchParams.set('run', runId); if (artifactId) { url.searchParams.set('artifact', artifactId); @@ -82,9 +99,11 @@ export async function GET(request: NextRequest) { ) throw new Error('Published artifact identity mismatch'); } else if ( - runId - ? String(data.run?.id) !== runId || !Array.isArray(data.artifacts) - : !Array.isArray(data.runs) + history + ? data.schemaVersion !== 1 || !Array.isArray(data.entries) + : runId + ? String(data.run?.id) !== runId || !Array.isArray(data.artifacts) + : !Array.isArray(data.runs) ) { throw new Error('Published run identity mismatch'); } @@ -99,6 +118,7 @@ export async function GET(request: NextRequest) { { error: 'Public H3 repository unavailable' }, { status: 503, headers }, ); + if (history) return NextResponse.json(await publishedVideoHistory(Number(page)), { headers }); if (artifactId) { if (media && !videoStorageEnabled()) return new Response(null, { status: 204, headers }); if (media) { @@ -207,7 +227,11 @@ export async function GET(request: NextRequest) { ); } catch { return NextResponse.json( - { error: 'GitHub is unavailable; retry shortly' }, + { + error: history + ? 'Published video history is unavailable; retry shortly' + : 'GitHub is unavailable; retry shortly', + }, { status: 502, headers }, ); } diff --git a/packages/app/src/components/video-benchmark/VideoCIRuns.tsx b/packages/app/src/components/video-benchmark/VideoCIRuns.tsx index 532da66c7..8d677ead1 100644 --- a/packages/app/src/components/video-benchmark/VideoCIRuns.tsx +++ b/packages/app/src/components/video-benchmark/VideoCIRuns.tsx @@ -8,6 +8,7 @@ import { Skeleton } from '@/components/ui/skeleton'; import { Input } from '@/components/ui/input'; import { useLocale } from '@/lib/use-locale'; import VideoBenchmark from './VideoBenchmark'; +import VideoHistory from './VideoHistory'; import FidelityResults from './FidelityResults'; import VideoSelect from './VideoSelect'; import { servingCells } from './serving'; @@ -23,6 +24,7 @@ const STRINGS = { title: 'H3 video benchmark', results: 'Videos & result', tradeoffs: 'Hardware tradeoffs', + history: 'Performance history', view: 'Benchmark view', preparing: 'Loading results and preparing media. The first publication of a run takes longer.', downloading: 'Downloading CI archive', @@ -53,6 +55,7 @@ const STRINGS = { title: 'H3 视频基准测试', results: '视频与结果', tradeoffs: '硬件延迟与效率权衡', + history: '性能历史', view: '基准测试视图', preparing: '正在加载结果并准备媒体。首次发布该运行的产物需要更多时间。', downloading: '正在下载 CI 产物', @@ -92,9 +95,11 @@ function share(runId: number, artifactId?: number, source?: string, cell?: strin const url = new URL(location.href); const view = url.searchParams.get('view'); const comparison = url.searchParams.get('compare'); + const historyFilters = [...url.searchParams].filter(([key]) => key.startsWith('history-')); url.search = ''; - if (view === 'tradeoff') url.searchParams.set('view', view); + if (view === 'tradeoff' || view === 'history') url.searchParams.set('view', view); if (comparison) url.searchParams.set('compare', comparison); + for (const [key, value] of historyFilters) url.searchParams.set(key, value); url.searchParams.set('run', String(runId)); if (artifactId) url.searchParams.set('artifact', String(artifactId)); if (source) url.searchParams.set('source', source); @@ -124,7 +129,7 @@ export default function VideoCIRuns() { const [direct, setDirect] = useState(''); const [manual, setManual] = useState(false); const [cellId, setCellId] = useState(''); - const [view, setView] = useState('results'); + const [view, setView] = useState(''); const [compared, setCompared] = useState([]); const [comparisonLoading, setComparisonLoading] = useState(false); const [comparisonError, setComparisonError] = useState(false); @@ -132,7 +137,7 @@ export default function VideoCIRuns() { const changeView = (value: string) => { setView(value); const url = new URL(location.href); - if (value === 'tradeoff') url.searchParams.set('view', 'tradeoff'); + if (value === 'tradeoff' || value === 'history') url.searchParams.set('view', value); else url.searchParams.delete('view'); history.replaceState(null, '', url); }; @@ -399,12 +404,14 @@ export default function VideoCIRuns() { } useEffect(() => { const params = new URLSearchParams(location.search); - if (params.get('view') === 'tradeoff') setView('tradeoff'); + const initialView = params.get('view') ?? (params.has('run') ? 'results' : 'history'); + setView(['history', 'tradeoff'].includes(initialView) ? initialView : 'results'); if (params.has('compare')) void restoreComparison(); const directRun = params.get('run'); if (directRun) void selectRun(directRun, params.get('artifact'), params.get('source'), params.get('cell')); - else void list(1, true); + else if (initialView === 'results') void list(1, true); + return () => { request.current++; download.current?.abort(); @@ -415,7 +422,7 @@ export default function VideoCIRuns() { const selectedSource = sources.find((item) => item.id === sourceId); return (
- +
From 2ada5998aae70b1ce74626f7a03eae62fe081599 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:53:33 -0700 Subject: [PATCH 003/104] feat: map H3 artifact hardware to HW_REGISTRY keys and TCO tiers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:将 H3 产物中的 GPU 名称映射到 HW_REGISTRY 键、标签、排序和 TCO 分档单价,并加入含 MI355X 不可用原因的固定硬件名单。 --- .../video-benchmark/hardware.test.ts | 45 +++++++++++++++ .../components/video-benchmark/hardware.ts | 57 +++++++++++++++++++ 2 files changed, 102 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/hardware.test.ts create mode 100644 packages/app/src/components/video-benchmark/hardware.ts diff --git a/packages/app/src/components/video-benchmark/hardware.test.ts b/packages/app/src/components/video-benchmark/hardware.test.ts new file mode 100644 index 000000000..fb3e1dc42 --- /dev/null +++ b/packages/app/src/components/video-benchmark/hardware.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest'; +import { + costPerGpuHour, + hardwareKey, + hardwareLabel, + hardwareSort, + VIDEO_HARDWARE_ROSTER, +} from './hardware'; + +describe('video hardware mapping', () => { + it('maps artifact device names to HW_REGISTRY keys', () => { + expect(hardwareKey('NVIDIA H100 80GB HBM3')).toBe('h100'); + expect(hardwareKey('NVIDIA H200')).toBe('h200'); + expect(hardwareKey('NVIDIA B200')).toBe('b200'); + expect(hardwareKey('MI355X')).toBe('mi355x'); + expect(hardwareKey('AMD Instinct MI355X OAM')).toBe('mi355x'); + expect(hardwareKey('NVIDIA GB200')).toBe('gb200'); + expect(hardwareKey('')).toBeNull(); + expect(hardwareKey('Synthetic GPU')).toBeNull(); + }); + it('does not confuse H100 with H1000-style names or MI300X with MI355X', () => { + expect(hardwareKey('NVIDIA H1000')).toBeNull(); + expect(hardwareKey('AMD Instinct MI300X')).toBe('mi300x'); + }); + it('exposes label, sort and TCO tiers from HW_REGISTRY', () => { + expect(hardwareLabel('h200')).toBe('H200'); + expect(hardwareLabel('unknown')).toBe('unknown'); + expect(hardwareSort('b200')).toBeLessThan(hardwareSort('h100')); + expect(costPerGpuHour('h200', 'h')).toBe(1.22); + expect(costPerGpuHour('h200', 'r')).toBe(2.9); + expect(costPerGpuHour('unknown', 'h')).toBeNull(); + }); + it('keeps MI355X in the campaign roster as unavailable with its failed run', () => { + const mi355x = VIDEO_HARDWARE_ROSTER.find((item) => item.key === 'mi355x'); + expect(mi355x?.unavailable?.runUrl).toBe( + 'https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34411615416', + ); + expect(VIDEO_HARDWARE_ROSTER.map((item) => item.key)).toEqual([ + 'b200', + 'mi355x', + 'h200', + 'h100', + ]); + }); +}); diff --git a/packages/app/src/components/video-benchmark/hardware.ts b/packages/app/src/components/video-benchmark/hardware.ts new file mode 100644 index 000000000..36a4b14b5 --- /dev/null +++ b/packages/app/src/components/video-benchmark/hardware.ts @@ -0,0 +1,57 @@ +import { HW_REGISTRY } from '@semianalysisai/inferencex-constants'; + +/** `h` = Owning at Large Hyperscaler Volume, `r` = Rent - 3 Year Commit (HW_REGISTRY costh/costr). */ +export type CostTier = 'h' | 'r'; + +/** Registry keys that can appear as a whole token in an artifact GPU name, longest first. */ +const TOKENS = Object.keys(HW_REGISTRY) + .filter((key) => /^(?:h|b|gb|mi)\d/u.test(key)) + .toSorted((a, b) => b.length - a.length); + +/** Map a raw device name (`NVIDIA H100 80GB HBM3`, `MI355X`) to a HW_REGISTRY key. */ +export function hardwareKey(name: string): string | null { + const lower = name.toLowerCase(); + for (const key of TOKENS) { + if (new RegExp(`(?:^|[^a-z0-9])${key}(?![a-z0-9])`, 'u').test(lower)) return key; + } + return null; +} + +export const hardwareLabel = (key: string): string => HW_REGISTRY[key]?.label ?? key; + +export const hardwareSort = (key: string): number => + HW_REGISTRY[key]?.sort ?? Number.MAX_SAFE_INTEGER; + +export function costPerGpuHour(key: string, tier: CostTier): number | null { + const entry = HW_REGISTRY[key]; + if (!entry) return null; + const value = tier === 'h' ? entry.costh : entry.costr; + return Number.isFinite(value) && value > 0 ? value : null; +} + +export interface VideoHardwareRosterEntry { + key: string; + /** Present when the campaign targeted this hardware but no valid observation exists. */ + unavailable?: { en: string; zh: string; runUrl: string }; +} + +/** + * Hardware the frozen H3 campaign targets. Hardware without a published + * observation still renders as an unavailable legend/table row so a missing + * result is never mistaken for an omission. Sorted by HW_REGISTRY order. + */ +export const VIDEO_HARDWARE_ROSTER: readonly VideoHardwareRosterEntry[] = ( + [ + { key: 'h100' }, + { key: 'h200' }, + { key: 'b200' }, + { + key: 'mi355x', + unavailable: { + en: 'No valid published run yet: the 2026-09-09 attempt failed before generation (AMD SMI parser).', + zh: '尚无有效的已发布运行:2026-09-09 的尝试在生成前失败(AMD SMI 解析器)。', + runUrl: 'https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34411615416', + }, + }, + ] satisfies VideoHardwareRosterEntry[] +).toSorted((a, b) => hardwareSort(a.key) - hardwareSort(b.key)); From 72669b3e593bd25cd5386ba1803d95f5276c99c6 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:53:44 -0700 Subject: [PATCH 004/104] feat: add video benchmark metric registry with TCO tiers and GPU basis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增视频基准指标注册表:出片时间、生成速度、每 GPU 小时视频数、每美元 TCO 视频数、每条视频成本与能耗、功率上限占比;支持成本分档与参与/已分配 GPU 口径,缺失值一律为 null。 --- .../video-benchmark/metrics.test.ts | 107 +++++++ .../src/components/video-benchmark/metrics.ts | 271 ++++++++++++++++++ 2 files changed, 378 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/metrics.test.ts create mode 100644 packages/app/src/components/video-benchmark/metrics.ts diff --git a/packages/app/src/components/video-benchmark/metrics.test.ts b/packages/app/src/components/video-benchmark/metrics.test.ts new file mode 100644 index 000000000..2d427d0d5 --- /dev/null +++ b/packages/app/src/components/video-benchmark/metrics.test.ts @@ -0,0 +1,107 @@ +import { describe, expect, it } from 'vitest'; +import { formatMetric, metricLabel, metricValue, type VideoPoint } from './metrics'; + +// H200 and H100 C1 cells from the retained 2026-09-09 serving-smoke artifacts. +const h200: VideoPoint = { + id: '34342452354.10107476604:c1', + runId: '34342452354', + artifactId: 10107476604, + cell: 'c1', + hardwareKey: 'h200', + hardwareName: 'NVIDIA H200', + runtime: '71de97b264b04dcd514cf904003028aefe9775c8', + concurrency: 1, + participating: 4, + allocated: 4, + valid: 20, + completed: 20, + scheduled: 20, + failed: 0, + samples: 20, + p50: 150.61379817902343, + p90: 151.10486977093387, + wallSeconds: 3013.44202613004, + durationSeconds: 8, + frameCount: 192, + energyKj: 410.954300515549, + avgPowerW: 2727.484797810607, + enforcedLimitW: 2800, + server: { tp: 2, ulysses: 2, attention: 'auto' }, + status: 'complete', + observedAt: '2026-09-09T10:00:00Z', +}; +const h100: VideoPoint = { + ...h200, + id: 'x', + hardwareKey: 'h100', + hardwareName: 'NVIDIA H100 80GB HBM3', + p50: 167.33339739358053, + p90: 168.11354550393298, + wallSeconds: 3349.37367718108, + allocated: 8, + energyKj: 431.14963489170344, + avgPowerW: 2574.550733165641, +}; +const opts = { tier: 'h', basis: 'participating' } as const; + +describe('video metrics', () => { + it('derives throughput per GPU-hour from valid clips, wall time and participating GPUs', () => { + expect(metricValue(h200, 'videosPerGpuHour', opts)).toBeCloseTo(5.9732, 3); + expect(metricValue(h200, 'videoSecondsPerGpuHour', opts)).toBeCloseTo(47.786, 2); + }); + it('switches the denominator to allocated GPUs on request', () => { + expect(metricValue(h100, 'videosPerGpuHour', opts)).toBeCloseTo(5.3741, 3); + expect(metricValue(h100, 'videosPerGpuHour', { ...opts, basis: 'allocated' })).toBeCloseTo( + 2.687, + 3, + ); + }); + it('prices videos with HW_REGISTRY tiers', () => { + expect(metricValue(h200, 'videosPerDollar', opts)).toBeCloseTo(4.896, 2); + expect(metricValue(h200, 'dollarsPerVideo', opts)).toBeCloseTo(0.2042, 3); + expect(metricValue(h200, 'dollarsPerVideo', { ...opts, tier: 'r' })).toBeCloseTo(0.4855, 3); + expect(metricValue(h200, 'dollarsPerVideoSecond', opts)).toBeCloseTo(0.02553, 4); + expect(metricValue(h100, 'dollarsPerVideo', { ...opts, basis: 'allocated' })).toBeCloseTo( + 0.4354, + 3, + ); + }); + it('exposes latency, generation speed, energy and power ratios', () => { + expect(metricValue(h200, 'p90Latency', opts)).toBeCloseTo(151.105, 3); + expect(metricValue(h200, 'p50Latency', opts)).toBeCloseTo(150.614, 3); + expect(metricValue(h200, 'genSpeed', opts)).toBeCloseTo(192 / 150.61379817902343, 6); + expect(metricValue(h200, 'kjPerVideo', opts)).toBeCloseTo(410.954, 2); + expect(metricValue(h200, 'videosPerKwh', opts)).toBeCloseTo(8.76, 2); + expect(metricValue(h200, 'powerPctCap', opts)).toBeCloseTo(97.41, 1); + }); + it('returns null instead of zero for missing inputs', () => { + const broken: VideoPoint = { + ...h200, + hardwareKey: null, + energyKj: null, + valid: 0, + wallSeconds: null, + enforcedLimitW: null, + }; + expect(metricValue(broken, 'videosPerDollar', opts)).toBeNull(); + expect(metricValue(broken, 'videosPerGpuHour', opts)).toBeNull(); + expect(metricValue(broken, 'kjPerVideo', opts)).toBeNull(); + expect(metricValue(broken, 'videosPerKwh', opts)).toBeNull(); + expect(metricValue(broken, 'powerPctCap', opts)).toBeNull(); + expect(metricValue({ ...h200, samples: 9 }, 'p90Latency', opts)).toBeNull(); + expect(metricValue({ ...h200, participating: null }, 'videosPerGpuHour', opts)).toBeNull(); + }); + it('labels tiered metrics with the cost tier and formats by metric', () => { + expect(metricLabel('videosPerDollar', 'en', opts)).toBe( + 'Videos per $1 TCO (Owning at Large Hyperscaler Volume)', + ); + expect(metricLabel('videosPerDollar', 'zh', { ...opts, tier: 'r' })).toBe( + '每 1 美元 TCO 生成视频数(租赁 - 3 年承诺)', + ); + expect(metricLabel('p90Latency', 'en', opts)).toBe('P90 time to video (s)'); + expect(formatMetric(0.20419, 'dollarsPerVideo')).toBe('$0.204'); + expect(formatMetric(null, 'kjPerVideo')).toBe('—'); + expect(formatMetric(151.10486, 'p90Latency')).toBe('151.1'); + expect(formatMetric(1234.5, 'videoSecondsPerGpuHour')).toBe('1,234.5'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts new file mode 100644 index 000000000..f0ab62b04 --- /dev/null +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -0,0 +1,271 @@ +import { costPerGpuHour, type CostTier } from './hardware'; + +/** Denominator for per-GPU metrics: boards that generated the clip vs. boards the job reserved. */ +export type GpuBasis = 'participating' | 'allocated'; + +/** One published serving cell (hardware × client concurrency) flattened for the dashboard. */ +export interface VideoPoint { + id: string; + runId: string; + artifactId: number; + cell: string | null; + hardwareKey: string | null; + hardwareName: string; + runtime: string; + concurrency: number | null; + participating: number | null; + allocated: number | null; + valid: number | null; + completed: number | null; + scheduled: number | null; + failed: number | null; + samples: number; + p50: number | null; + p90: number | null; + wallSeconds: number | null; + durationSeconds: number | null; + frameCount: number | null; + /** GPU-board kJ per valid clip over the recorded generation window; null when power is invalid. */ + energyKj: number | null; + /** Sum of participating boards' time-weighted mean W. */ + avgPowerW: number | null; + /** Sum of the same boards' recorded enforced limits; null when a limit was not recorded. */ + enforcedLimitW: number | null; + server: { tp: number | null; ulysses: number | null; attention: string | null } | null; + status: string; + observedAt: string | null; +} + +export type XMetricId = 'p90Latency' | 'p50Latency' | 'genSpeed'; +export type YMetricId = + | 'videosPerDollar' + | 'videosPerGpuHour' + | 'videoSecondsPerGpuHour' + | 'dollarsPerVideo' + | 'kjPerVideo' + | 'videosPerKwh'; +export type MetricId = XMetricId | YMetricId | 'powerPctCap' | 'dollarsPerVideoSecond'; + +export interface MetricOptions { + tier: CostTier; + basis: GpuBasis; +} + +interface MetricDefinition { + label: string; + labelZh: string; + unit: string; + polarity: 'higher' | 'lower'; + digits: number; + /** Label gets the cost tier appended. */ + tiered?: boolean; + /** Prefix a dollar sign when formatting. */ + currency?: boolean; +} + +export const X_METRICS: readonly XMetricId[] = ['p90Latency', 'p50Latency', 'genSpeed']; +export const Y_METRICS: readonly YMetricId[] = [ + 'videosPerDollar', + 'videosPerGpuHour', + 'videoSecondsPerGpuHour', + 'dollarsPerVideo', + 'kjPerVideo', + 'videosPerKwh', +]; +export const COST_TIERS: readonly CostTier[] = ['h', 'r']; + +/** Same tier names the inference calculator uses for HW_REGISTRY costh/costr. */ +export const TIER_LABELS: Record = { + h: { en: 'Owning at Large Hyperscaler Volume', zh: 'Hyperscaler 自有设备' }, + r: { en: 'Rent - 3 Year Commit', zh: '租赁 - 3 年承诺' }, +}; + +export const VIDEO_METRICS: Record = { + p90Latency: { + label: 'P90 time to video (s)', + labelZh: 'P90 出片时间(s)', + unit: 's', + polarity: 'lower', + digits: 1, + }, + p50Latency: { + label: 'P50 time to video (s)', + labelZh: 'P50 出片时间(s)', + unit: 's', + polarity: 'lower', + digits: 1, + }, + genSpeed: { + label: 'Generation speed (frames/s per request)', + labelZh: '生成速度(每请求 frames/s)', + unit: 'frames/s', + polarity: 'higher', + digits: 2, + }, + videosPerDollar: { + label: 'Videos per $1 TCO', + labelZh: '每 1 美元 TCO 生成视频数', + unit: 'videos/$', + polarity: 'higher', + digits: 2, + tiered: true, + }, + videosPerGpuHour: { + label: 'Videos per GPU-hour', + labelZh: '每 GPU 小时视频数', + unit: 'videos/GPU-hr', + polarity: 'higher', + digits: 2, + }, + videoSecondsPerGpuHour: { + label: 'Video seconds per GPU-hour', + labelZh: '每 GPU 小时生成视频秒数', + unit: 'video-s/GPU-hr', + polarity: 'higher', + digits: 1, + }, + dollarsPerVideo: { + label: 'TCO cost per video', + labelZh: '每条视频 TCO 成本', + unit: '$/video', + polarity: 'lower', + digits: 3, + tiered: true, + currency: true, + }, + dollarsPerVideoSecond: { + label: 'TCO cost per video-second', + labelZh: '每视频秒 TCO 成本', + unit: '$/video-s', + polarity: 'lower', + digits: 4, + tiered: true, + currency: true, + }, + kjPerVideo: { + label: 'GPU-board energy per video (kJ)', + labelZh: '每条视频 GPU 板卡能耗(kJ)', + unit: 'kJ/video', + polarity: 'lower', + digits: 1, + }, + videosPerKwh: { + label: 'Videos per GPU-board kWh', + labelZh: '每 GPU 板卡 kWh 视频数', + unit: 'videos/kWh', + polarity: 'higher', + digits: 2, + }, + powerPctCap: { + label: 'Mean board power / enforced limit (%)', + labelZh: '板卡平均功率 / 生效功率上限(%)', + unit: '%', + polarity: 'higher', + digits: 1, + }, +}; + +const finite = (n: number | null | undefined): n is number => + typeof n === 'number' && Number.isFinite(n); +const positive = (n: number | null | undefined): n is number => finite(n) && n > 0; + +function gpus(point: VideoPoint, basis: GpuBasis): number | null { + const n = basis === 'allocated' ? point.allocated : point.participating; + return positive(n) && Number.isSafeInteger(n) ? n : null; +} + +function videosPerGpuHour(point: VideoPoint, basis: GpuBasis): number | null { + const n = gpus(point, basis); + if (!positive(point.valid) || !positive(point.wallSeconds) || n === null) return null; + return (point.valid * 3600) / (point.wallSeconds * n); +} + +function dollarsPerVideo(point: VideoPoint, options: MetricOptions): number | null { + const rate = videosPerGpuHour(point, options.basis); + const cost = point.hardwareKey ? costPerGpuHour(point.hardwareKey, options.tier) : null; + return rate !== null && cost !== null ? cost / rate : null; +} + +/** Every formula returns null (never 0) when an input is missing or invalid. */ +export function metricValue( + point: VideoPoint, + id: MetricId, + options: MetricOptions, +): number | null { + let value: number | null = null; + switch (id) { + case 'p90Latency': { + // Same display floor as the run views: P90 needs at least ten valid samples. + value = point.samples >= 10 ? point.p90 : null; + break; + } + case 'p50Latency': { + value = point.p50; + break; + } + case 'genSpeed': { + value = + positive(point.frameCount) && positive(point.p50) ? point.frameCount / point.p50 : null; + break; + } + case 'videosPerGpuHour': { + value = videosPerGpuHour(point, options.basis); + break; + } + case 'videoSecondsPerGpuHour': { + const rate = videosPerGpuHour(point, options.basis); + value = + rate !== null && positive(point.durationSeconds) ? rate * point.durationSeconds : null; + break; + } + case 'videosPerDollar': { + const cost = dollarsPerVideo(point, options); + value = cost === null ? null : 1 / cost; + break; + } + case 'dollarsPerVideo': { + value = dollarsPerVideo(point, options); + break; + } + case 'dollarsPerVideoSecond': { + const cost = dollarsPerVideo(point, options); + value = + cost !== null && positive(point.durationSeconds) ? cost / point.durationSeconds : null; + break; + } + case 'kjPerVideo': { + value = positive(point.energyKj) ? point.energyKj : null; + break; + } + case 'videosPerKwh': { + value = positive(point.energyKj) ? 3600 / point.energyKj : null; + break; + } + case 'powerPctCap': { + value = + positive(point.avgPowerW) && positive(point.enforcedLimitW) + ? (point.avgPowerW / point.enforcedLimitW) * 100 + : null; + break; + } + } + return finite(value) ? value : null; +} + +export function metricLabel(id: MetricId, locale: 'en' | 'zh', options: MetricOptions): string { + const def = VIDEO_METRICS[id]; + const base = locale === 'zh' ? def.labelZh : def.label; + if (!def.tiered) return base; + const tier = TIER_LABELS[options.tier][locale]; + return locale === 'zh' ? `${base}(${tier})` : `${base} (${tier})`; +} + +export function formatMetric(value: number | null, id: MetricId): string { + if (value === null) return '—'; + const def = VIDEO_METRICS[id]; + const text = value.toLocaleString('en-US', { + minimumFractionDigits: def.currency ? def.digits : 0, + maximumFractionDigits: def.digits, + }); + return def.currency ? `$${text}` : text; +} From 61a658903b9d6e780945307dc223e422bb010781 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:53:54 -0700 Subject: [PATCH 005/104] feat: add direction-aware Pareto frontier for video hardware points MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增支持坐标轴方向声明的 Pareto 前沿计算,复用 inference 图的前沿算法。 --- .../video-benchmark/frontier.test.ts | 40 +++++++++++++++++++ .../components/video-benchmark/frontier.ts | 24 +++++++++++ 2 files changed, 64 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/frontier.test.ts create mode 100644 packages/app/src/components/video-benchmark/frontier.ts diff --git a/packages/app/src/components/video-benchmark/frontier.test.ts b/packages/app/src/components/video-benchmark/frontier.test.ts new file mode 100644 index 000000000..6d3215a02 --- /dev/null +++ b/packages/app/src/components/video-benchmark/frontier.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from 'vitest'; +import { paretoFrontier } from './frontier'; + +const b200 = { id: 'b200', x: 78.3, y: 6.66 }; +const h200 = { id: 'h200', x: 151.1, y: 4.9 }; +const h100 = { id: 'h100', x: 168.1, y: 4.59 }; + +describe('paretoFrontier', () => { + it('keeps only the dominating point when one hardware is faster and more efficient', () => { + expect(paretoFrontier([h100, h200, b200], 'lower', 'higher').map((p) => p.id)).toEqual([ + 'b200', + ]); + }); + it('keeps a slower but more efficient point on a lower/higher frontier, sorted by x', () => { + const cheapSlow = { id: 'slow', x: 400, y: 9 }; + expect( + paretoFrontier([cheapSlow, h100, h200, b200], 'lower', 'higher').map((p) => p.id), + ).toEqual(['b200', 'slow']); + }); + it('handles higher/higher (generation speed vs efficiency)', () => { + const fast = { id: 'fast', x: 2.46, y: 6.66 }; + const mid = { id: 'mid', x: 1.27, y: 4.9 }; + const midEfficient = { id: 'midEff', x: 1.2, y: 7 }; + expect(paretoFrontier([fast, mid, midEfficient], 'higher', 'higher').map((p) => p.id)).toEqual([ + 'midEff', + 'fast', + ]); + }); + it('handles lower/lower (time to video vs $/video)', () => { + const cost = [ + { id: 'b200', x: 78.3, y: 0.15 }, + { id: 'h200', x: 151.1, y: 0.204 }, + { id: 'cheap', x: 300, y: 0.1 }, + ]; + expect(paretoFrontier(cost, 'lower', 'lower').map((p) => p.id)).toEqual(['b200', 'cheap']); + }); + it('returns an empty frontier for no points', () => { + expect(paretoFrontier([], 'lower', 'higher')).toEqual([]); + }); +}); diff --git a/packages/app/src/components/video-benchmark/frontier.ts b/packages/app/src/components/video-benchmark/frontier.ts new file mode 100644 index 000000000..102adf9cd --- /dev/null +++ b/packages/app/src/components/video-benchmark/frontier.ts @@ -0,0 +1,24 @@ +import { paretoFrontUpperLeft } from '@/components/calculator/interpolation'; + +type Better = 'lower' | 'higher'; + +/** + * Non-dominated set for two axes with declared directions. `paretoFrontUpperLeft` + * keeps the points no other point beats on both a larger x and a larger y, so a + * lower-is-better axis is fed negated. The same hull code therefore serves + * time-to-video vs efficiency, generation speed vs efficiency and time-to-video + * vs cost. Result is sorted by ascending x. + */ +export function paretoFrontier( + points: T[], + xBetter: Better, + yBetter: Better, +): T[] { + const sx = xBetter === 'higher' ? 1 : -1; + const sy = yBetter === 'higher' ? 1 : -1; + return paretoFrontUpperLeft( + points, + (p) => sx * p.x, + (p) => sy * p.y, + ).toSorted((a, b) => a.x - b.x); +} From 49059f745b21c64df82db2c3c111af4cd191d69e Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:54:04 -0700 Subject: [PATCH 006/104] feat: add v_ URL state for the video hardware dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:为视频硬件仪表板新增 v_ 前缀的 URL 状态读写:坐标轴、成本分档、GPU 口径、排队尾迹、仅最优、图表/表格视图;默认值不写入 URL。 --- .../video-benchmark/video-url-state.test.ts | 47 +++++++++++++ .../video-benchmark/video-url-state.ts | 68 +++++++++++++++++++ 2 files changed, 115 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/video-url-state.test.ts create mode 100644 packages/app/src/components/video-benchmark/video-url-state.ts diff --git a/packages/app/src/components/video-benchmark/video-url-state.test.ts b/packages/app/src/components/video-benchmark/video-url-state.test.ts new file mode 100644 index 000000000..31fca027e --- /dev/null +++ b/packages/app/src/components/video-benchmark/video-url-state.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'vitest'; +import { + DEFAULT_VIDEO_DASHBOARD_STATE, + readVideoDashboardState, + writeVideoDashboardState, +} from './video-url-state'; + +describe('video dashboard URL state', () => { + it('falls back to defaults for missing or invalid params', () => { + expect(readVideoDashboardState('')).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); + expect( + readVideoDashboardState('?v_x=bogus&v_y=nope&v_tier=z&v_basis=q&v_view=pie&v_queue=yes'), + ).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); + }); + it('reads every supported param', () => { + expect( + readVideoDashboardState( + '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_view=table', + ), + ).toEqual({ + x: 'genSpeed', + y: 'kjPerVideo', + tier: 'r', + basis: 'allocated', + queue: true, + optimal: true, + view: 'table', + }); + }); + it('writes only non-default params and keeps unrelated params', () => { + const url = new URL('https://x.test/video?run=1&view=results'); + const out = writeVideoDashboardState(url, { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + tier: 'r', + queue: true, + }); + expect(out.searchParams.get('run')).toBe('1'); + expect(out.searchParams.get('view')).toBe('results'); + expect(out.searchParams.get('v_tier')).toBe('r'); + expect(out.searchParams.get('v_queue')).toBe('1'); + expect(out.searchParams.has('v_x')).toBe(false); + expect(url.searchParams.has('v_tier')).toBe(false); + const reset = writeVideoDashboardState(out, DEFAULT_VIDEO_DASHBOARD_STATE); + expect([...reset.searchParams.keys()].filter((k) => k.startsWith('v_'))).toEqual([]); + expect(reset.searchParams.get('run')).toBe('1'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/video-url-state.ts b/packages/app/src/components/video-benchmark/video-url-state.ts new file mode 100644 index 000000000..aad6c229e --- /dev/null +++ b/packages/app/src/components/video-benchmark/video-url-state.ts @@ -0,0 +1,68 @@ +import type { CostTier } from './hardware'; +import { + COST_TIERS, + X_METRICS, + Y_METRICS, + type GpuBasis, + type XMetricId, + type YMetricId, +} from './metrics'; + +export interface VideoDashboardState { + x: XMetricId; + y: YMetricId; + tier: CostTier; + basis: GpuBasis; + /** Show C2/C4 client-concurrency cells as dotted queueing tails. */ + queue: boolean; + /** Show only Pareto-optimal hardware. */ + optimal: boolean; + view: 'chart' | 'table'; +} + +export const DEFAULT_VIDEO_DASHBOARD_STATE: VideoDashboardState = { + x: 'p90Latency', + y: 'videosPerDollar', + tier: 'h', + basis: 'participating', + queue: false, + optimal: false, + view: 'chart', +}; + +const BASES: readonly GpuBasis[] = ['participating', 'allocated']; +const VIEWS: readonly VideoDashboardState['view'][] = ['chart', 'table']; + +const pick = (allowed: readonly T[], value: string | null, fallback: T): T => + allowed.includes(value as T) ? (value as T) : fallback; + +/** Dashboard controls live under the `v_` prefix so the run/artifact/view params stay untouched. */ +export function readVideoDashboardState(search: string): VideoDashboardState { + const p = new URLSearchParams(search); + const d = DEFAULT_VIDEO_DASHBOARD_STATE; + return { + x: pick(X_METRICS, p.get('v_x'), d.x), + y: pick(Y_METRICS, p.get('v_y'), d.y), + tier: pick(COST_TIERS, p.get('v_tier'), d.tier), + basis: pick(BASES, p.get('v_basis'), d.basis), + queue: p.get('v_queue') === '1', + optimal: p.get('v_opt') === '1', + view: pick(VIEWS, p.get('v_view'), d.view), + }; +} + +/** Returns a new URL with defaults omitted; the input URL is not mutated. */ +export function writeVideoDashboardState(url: URL, state: VideoDashboardState): URL { + const out = new URL(url); + const d = DEFAULT_VIDEO_DASHBOARD_STATE; + const set = (key: string, value: string | null) => + value === null ? out.searchParams.delete(key) : out.searchParams.set(key, value); + set('v_x', state.x === d.x ? null : state.x); + set('v_y', state.y === d.y ? null : state.y); + set('v_tier', state.tier === d.tier ? null : state.tier); + set('v_basis', state.basis === d.basis ? null : state.basis); + set('v_queue', state.queue ? '1' : null); + set('v_opt', state.optimal ? '1' : null); + set('v_view', state.view === d.view ? null : state.view); + return out; +} From b2406829f1c763aa4bee63cdaec78ef45456d57e Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:55:19 -0700 Subject: [PATCH 007/104] feat: project GPU counts, power and server layout per published video cell MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:已发布历史投影为每个 cell 增加参与/已分配 GPU 数、测量窗口、视频时长与帧数、板卡功率与生效上限、配置的 TP/Ulysses/attention;均为可空的附加字段,schemaVersion 不变。 --- docs/video-history.md | 4 ++- .../video-benchmark/history.test.ts | 18 +++++++++++++ .../src/components/video-benchmark/history.ts | 27 +++++++++++++++++++ 3 files changed, 48 insertions(+), 1 deletion(-) diff --git a/docs/video-history.md b/docs/video-history.md index c3a86e9a5..67b12a032 100644 --- a/docs/video-history.md +++ b/docs/video-history.md @@ -8,6 +8,8 @@ The projection groups all cells under the original source inside one artifact entry. It reuses `storedBundle`, `servingCells`, `storedFidelityBundle`, and the existing tradeoff calculations. Invalid or unavailable indexes remain explicit error entries. Original execution time comes from `ci.started_at`; index publication time is separate and must never be substituted for execution time. Original run IDs and export run IDs remain distinct. +Each observation also carries the cell's participating and allocated GPU counts, measurement wall seconds, clip duration and frame count, summed mean board watts, summed recorded enforced limits (null when no limit snapshot exists) and the configured `tp_size` / `ulysses_degree` / `attention_backend`. These are additive, nullable fields under the same `schemaVersion: 1`; the hardware dashboard derives every displayed metric from them, so the browser never parses an artifact bundle for the chart. + ## Evidence boundaries - Only descriptive observations are shown: this version has no selected matched baseline and computes no before/after delta. @@ -24,7 +26,7 @@ The projection groups all cells under the original source inside one artifact en `/video` 与 `/zh/video` 默认打开“性能历史”。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 打开最新发布页面中首个可读取的产物与原始运行;未选择运行时,“视频与结果”也使用该入口。空列表或读取失败会显示可恢复的提示;只有点击“浏览 CI 运行”才扫描 GitHub Actions。返回性能历史会取消正在进行的结果选择;刷新和重新加载保留产物、原始运行及并发配置。历史筛选条件保存在 URL,刷新后恢复。 -历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。 +历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。每个 cell 另附参与/已分配 GPU 数、测量窗口秒数、视频时长与帧数、板卡平均功率之和、记录的生效功率上限之和(无快照时为 null)以及配置的 `tp_size` / `ulysses_degree` / `attention_backend`;这些都是同一 `schemaVersion: 1` 下可为空的附加字段,硬件仪表板据此推导全部指标,浏览器不再解析产物包。 当前展示观测值,未选择匹配基线,也不计算 before/after。延迟、吞吐量使用提交到下载完成的窗口;板卡能耗使用记录的生成窗口。保真度、阈值校准、发布验收单独展示。筛选仅作用于已加载的页面,可继续加载较早结果。 diff --git a/packages/app/src/components/video-benchmark/history.test.ts b/packages/app/src/components/video-benchmark/history.test.ts index 58b770230..72673fb03 100644 --- a/packages/app/src/components/video-benchmark/history.test.ts +++ b/packages/app/src/components/video-benchmark/history.test.ts @@ -50,6 +50,21 @@ describe('video history projection', () => { [4, 300, null, 15, 168], ]); }); + it('carries GPU counts, wall time, clip shape, board power and server layout per cell', () => { + const entry = videoHistoryEntry(saved(), null); + const c1 = entry.sources[0].observations[0]; + expect(c1).toMatchObject({ + participating: 2, + allocated: 2, + wallSeconds: 480, + durationSeconds: 4, + frameCount: 107, + avgPowerW: 1400, + // The synthetic bundle records no power-limit snapshots, so no limit is invented. + enforcedLimitW: null, + server: { tp: 1, ulysses: 2, attention: null }, + }); + }); it('preserves failed cells and null power without inventing zero-valued performance', () => { const artifact = saved(); const source = artifact.sources[0]; @@ -71,6 +86,9 @@ describe('video history projection', () => { p90: null, clipsGpuHour: null, energyKj: null, + wallSeconds: null, + avgPowerW: null, + enforcedLimitW: null, }); }); it('retains source provenance and failure when a new artifact contract is unsupported', () => { diff --git a/packages/app/src/components/video-benchmark/history.ts b/packages/app/src/components/video-benchmark/history.ts index fac1eef34..aa9f9d541 100644 --- a/packages/app/src/components/video-benchmark/history.ts +++ b/packages/app/src/components/video-benchmark/history.ts @@ -22,6 +22,18 @@ export interface VideoHistoryObservation { p90: number | null; clipsGpuHour: number | null; energyKj: number | null; + /** GPU boards that generated the clips (distinct bound UUIDs); null when the binding is ambiguous. */ + participating: number | null; + /** GPU boards the job reserved (Slurm AllocTRES or the retained AMD binding); null when unknown. */ + allocated: number | null; + wallSeconds: number | null; + durationSeconds: number | null; + frameCount: number | null; + /** Summed mean board W over the recorded generation window; null when the power phase is invalid. */ + avgPowerW: number | null; + /** Summed recorded enforced limits of the same boards; null when a limit was not recorded. */ + enforcedLimitW: number | null; + server: { tp: number | null; ulysses: number | null; attention: string | null } | null; } export interface VideoHistorySource { id: string; @@ -120,6 +132,21 @@ export function videoHistoryEntry( p90: latencyValue(point, 'p90'), clipsGpuHour: efficiencyValue(point, 'clipsGpu'), energyKj: point.energy === null ? null : point.energy / 1000, + participating: point.participating, + allocated: point.allocated, + wallSeconds: point.wall, + durationSeconds: number(at(point.workload, 'generation', 'duration_seconds')), + frameCount: number(at(point.workload, 'generation', 'frame_count')), + avgPowerW: point.power, + enforcedLimitW: point.powerLimit?.watts ?? null, + server: + point.server !== null && typeof point.server === 'object' + ? { + tp: number(at(point.server, 'tp_size')), + ulysses: number(at(point.server, 'ulysses_degree')), + attention: text(at(point.server, 'attention_backend')) || null, + } + : null, }; }); } From 22187d5c7bef1402cba8b7a7b840ff79b8660caf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:58:47 -0700 Subject: [PATCH 008/104] feat: flatten published video history into chart points MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:把已发布历史页面展开为可绘图的 VideoPoint 列表:映射硬件键、按来源去重保留最新发布、按 HW_REGISTRY 顺序排序、兼容缺少附加字段的旧投影。 --- .../components/video-benchmark/points.test.ts | 156 ++++++++++++++++++ .../src/components/video-benchmark/points.ts | 76 +++++++++ 2 files changed, 232 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/points.test.ts create mode 100644 packages/app/src/components/video-benchmark/points.ts diff --git a/packages/app/src/components/video-benchmark/points.test.ts b/packages/app/src/components/video-benchmark/points.test.ts new file mode 100644 index 000000000..a893e2a47 --- /dev/null +++ b/packages/app/src/components/video-benchmark/points.test.ts @@ -0,0 +1,156 @@ +import { describe, expect, it } from 'vitest'; +import type { VideoHistoryObservation, VideoHistoryPage } from './history'; +import { videoPoints } from './points'; + +const observation: VideoHistoryObservation = { + id: 'sha:c1', + cell: 'c1', + hardware: 'NVIDIA H200', + concurrency: 1, + runtime: '71de97b2', + workload: 'w', + model: 'MiniMaxAI/MiniMax-H3', + status: 'complete', + valid: 20, + completed: 20, + scheduled: 20, + failed: 0, + samples: 20, + p50: 150.6, + p90: 151.1, + clipsGpuHour: 5.97, + energyKj: 410.95, + participating: 4, + allocated: 4, + wallSeconds: 3013.4, + durationSeconds: 8, + frameCount: 192, + avgPowerW: 2727.5, + enforcedLimitW: 2800, + server: { tp: 2, ulysses: 2, attention: 'auto' }, +}; +const artifact = { + id: 10107476604, + name: 'h3-video-34342452354-1', + expired: false, + size_in_bytes: 1, + stored: true, +}; +const source = { + id: '34342452354', + sha256: 'a', + sourceSha: 'b', + hardware: 'NVIDIA H200', + execution: 'complete', + observedAt: '2026-09-09T10:00:00Z', + kind: 'observation' as const, + observations: [observation, { ...observation, id: 'sha:c2', cell: 'c2', concurrency: 2 }], + fidelity: null, + calibration: null, + releaseQualified: false, + error: null, +}; +const page: VideoHistoryPage = { + schemaVersion: 1, + nextPage: null, + entries: [ + { + id: '34342452354.10107476604', + runId: '34342452354', + artifact, + publishedAt: '2026-09-10T00:00:00Z', + error: null, + sources: [ + source, + { + ...source, + id: '1', + kind: 'fidelity', + observations: [], + fidelity: 'fail', + calibration: 'uncalibrated', + }, + ], + }, + { + // An older re-export of the same source: same observation ids, stale numbers. + id: '99.1', + runId: '34342452354', + artifact: { ...artifact, id: 1 }, + publishedAt: '2026-09-01T00:00:00Z', + error: null, + sources: [{ ...source, observations: [{ ...observation, p50: 999 }] }], + }, + ], +}; + +describe('videoPoints', () => { + it('flattens observation cells, maps hardware keys and keeps the newest publication of a source', () => { + const points = videoPoints([page]); + expect(points.map((p) => [p.id, p.concurrency])).toEqual([ + ['sha:c1', 1], + ['sha:c2', 2], + ]); + expect(points[0]).toMatchObject({ + runId: '34342452354', + artifactId: 10107476604, + hardwareKey: 'h200', + hardwareName: 'NVIDIA H200', + p50: 150.6, + enforcedLimitW: 2800, + server: { tp: 2, ulysses: 2, attention: 'auto' }, + observedAt: '2026-09-09T10:00:00Z', + }); + }); + it('sorts by HW_REGISTRY order, then concurrency', () => { + const b200 = { + ...page.entries[0], + id: '2.2', + runId: '2', + sources: [ + { + ...source, + id: '2', + observations: [{ ...observation, id: 'b:c1', hardware: 'NVIDIA B200' }], + }, + ], + }; + const h100 = { + ...page.entries[0], + id: '3.3', + runId: '3', + sources: [ + { + ...source, + id: '3', + observations: [ + { ...observation, id: 'h:c4', hardware: 'NVIDIA H100 80GB HBM3', concurrency: 4 }, + { ...observation, id: 'h:c1', hardware: 'NVIDIA H100 80GB HBM3' }, + ], + }, + ], + }; + const points = videoPoints([{ ...page, entries: [h100, page.entries[0], b200] }]); + expect(points.map((p) => p.id)).toEqual(['b:c1', 'sha:c1', 'sha:c2', 'h:c1', 'h:c4']); + }); + it('tolerates pre-upgrade observations that lack the additive fields', () => { + const legacy = structuredClone(page); + const obs = legacy.entries[0].sources[0].observations[0] as unknown as Record; + for (const key of [ + 'participating', + 'allocated', + 'wallSeconds', + 'durationSeconds', + 'frameCount', + 'avgPowerW', + 'enforcedLimitW', + 'server', + ]) + delete obs[key]; + const [point] = videoPoints([legacy]); + expect(point.participating).toBeNull(); + expect(point.wallSeconds).toBeNull(); + expect(point.server).toBeNull(); + expect(point.hardwareKey).toBe('h200'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/points.ts b/packages/app/src/components/video-benchmark/points.ts new file mode 100644 index 000000000..519696f9b --- /dev/null +++ b/packages/app/src/components/video-benchmark/points.ts @@ -0,0 +1,76 @@ +import { hardwareKey, hardwareSort } from './hardware'; +import type { VideoHistoryObservation, VideoHistoryPage } from './history'; +import type { VideoPoint } from './metrics'; + +const num = (value: unknown): number | null => + typeof value === 'number' && Number.isFinite(value) ? value : null; + +/** + * Flatten published history pages (newest publication first) into chartable + * points. A source re-exported under a later artifact repeats its observation + * ids; the first occurrence wins, which is the newest publication. + */ +export function videoPoints(pages: VideoHistoryPage[]): VideoPoint[] { + const seen = new Set(); + const points: VideoPoint[] = []; + for (const page of pages) + for (const entry of page.entries) + for (const source of entry.sources) { + if (source.kind !== 'observation') continue; + for (const observation of source.observations) { + if (seen.has(observation.id)) continue; + seen.add(observation.id); + points.push(toPoint(observation, entry.runId, entry.artifact.id, source.observedAt)); + } + } + return points.toSorted( + (a, b) => + hardwareSort(a.hardwareKey ?? '') - hardwareSort(b.hardwareKey ?? '') || + (a.concurrency ?? 0) - (b.concurrency ?? 0), + ); +} + +/** Observations published before the additive fields existed read as null, never 0. */ +function toPoint( + o: Partial & Pick, + runId: string, + artifactId: number, + observedAt: string | null, +): VideoPoint { + const server = o.server; + return { + id: o.id, + runId, + artifactId, + cell: o.cell ?? null, + hardwareKey: hardwareKey(o.hardware), + hardwareName: o.hardware, + runtime: o.runtime ?? '', + concurrency: num(o.concurrency), + participating: num(o.participating), + allocated: num(o.allocated), + valid: num(o.valid), + completed: num(o.completed), + scheduled: num(o.scheduled), + failed: num(o.failed), + samples: num(o.samples) ?? 0, + p50: num(o.p50), + p90: num(o.p90), + wallSeconds: num(o.wallSeconds), + durationSeconds: num(o.durationSeconds), + frameCount: num(o.frameCount), + energyKj: num(o.energyKj), + avgPowerW: num(o.avgPowerW), + enforcedLimitW: num(o.enforcedLimitW), + server: + server !== null && typeof server === 'object' + ? { + tp: num(server.tp), + ulysses: num(server.ulysses), + attention: typeof server.attention === 'string' ? server.attention : null, + } + : null, + status: o.status ?? '', + observedAt, + }; +} From 89b247f325f17919c71541cfe76246f9bbf1f3b4 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 19:58:57 -0700 Subject: [PATCH 009/104] test: rebuild video history fixture with per-cell GPU, power and layout fields MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增从保留的 StoredArtifact 导出重建 video-history fixture 的脚本,并更新 fixture 与校验清单;无新测量、无新发布。 --- .../app/cypress/fixtures/api/_manifest.json | 10 +- .../cypress/fixtures/api/video-history.json | 126 ++++++++++++++++-- packages/app/package.json | 1 + .../scripts/build-video-history-fixture.ts | 71 ++++++++++ 4 files changed, 194 insertions(+), 14 deletions(-) create mode 100644 packages/app/scripts/build-video-history-fixture.ts diff --git a/packages/app/cypress/fixtures/api/_manifest.json b/packages/app/cypress/fixtures/api/_manifest.json index 88a0b26ad..183661168 100644 --- a/packages/app/cypress/fixtures/api/_manifest.json +++ b/packages/app/cypress/fixtures/api/_manifest.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, - "generatedAt": "2026-08-20T08:25:30.857Z", + "generatedAt": "2026-09-20T02:56:12.184Z", "fixtures": { "agentic-catalog": { "bytes": 10027, @@ -94,10 +94,10 @@ "topLevel": "object" }, "video-history": { - "bytes": 14506, - "capturedAt": "2026-09-15T23:50:12.033Z", - "sha256": "278dbafd690465ea01d0a05b6c972751a1a2a277a6d1b43362407abd7d42ed2c", - "source": "Retained public H3 metadata replay from original runs; publication time unavailable. No new benchmark or publication.", + "bytes": 18083, + "capturedAt": "2026-09-20T02:56:12.184Z", + "sha256": "8a748582af68a5d259ad06f7295a3a5b92989ed0b5c0798eefb69600e22cdb77", + "source": "Retained public H3 metadata replay from original runs via videoHistoryEntry (per-cell GPU counts, board power and server layout); publication time unavailable. No new benchmark or publication.", "topLevel": "object" } } diff --git a/packages/app/cypress/fixtures/api/video-history.json b/packages/app/cypress/fixtures/api/video-history.json index 45be3d3fe..65a72f04c 100644 --- a/packages/app/cypress/fixtures/api/video-history.json +++ b/packages/app/cypress/fixtures/api/video-history.json @@ -40,7 +40,19 @@ "p50": 167.33339739358053, "p90": 168.11354550393298, "clipsGpuHour": 5.3741390883412175, - "energyKj": 431.14963489170344 + "energyKj": 431.14963489170344, + "participating": 4, + "allocated": 8, + "wallSeconds": 3349.37367718108, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2574.550733165641, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } }, { "id": "7df99e50c015669f4dd9ef62549317870f197cfe0fc5d163cf8bf0fb3b88dd5e:c2", @@ -59,7 +71,19 @@ "p50": 334.78318604291417, "p90": 336.9140758761205, "clipsGpuHour": 5.368742995784034, - "energyKj": 431.32628015924786 + "energyKj": 431.32628015924786, + "participating": 4, + "allocated": 8, + "wallSeconds": 3352.7401133067906, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2573.016825749808, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } } ], "fidelity": null, @@ -109,7 +133,19 @@ "p50": 669.2438137787394, "p90": 673.7058754772879, "clipsGpuHour": 5.368345559881099, - "energyKj": 430.80324425396446 + "energyKj": 430.80324425396446, + "participating": 4, + "allocated": 8, + "wallSeconds": 3352.988327450119, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2569.709833308179, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } } ], "fidelity": null, @@ -159,7 +195,19 @@ "p50": 150.61379817902343, "p90": 151.10486977093387, "clipsGpuHour": 5.973235869122123, - "energyKj": 410.954300515549 + "energyKj": 410.954300515549, + "participating": 4, + "allocated": 4, + "wallSeconds": 3013.44202613004, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2727.484797810607, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } }, { "id": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35:c2", @@ -178,7 +226,19 @@ "p50": 300.09254947002046, "p90": 300.1774324030848, "clipsGpuHour": 5.997056099146397, - "energyKj": 409.91272507134966 + "energyKj": 409.91272507134966, + "participating": 4, + "allocated": 4, + "wallSeconds": 3001.4726729940157, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2731.4196931869433, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } }, { "id": "ad62f1b44954b9733cb513b7dd1d54c224759a20956d9a1f257b190ce8d4da35:c4", @@ -197,7 +257,19 @@ "p50": 601.2122122829896, "p90": 601.8977232379839, "clipsGpuHour": 5.986775861646954, - "energyKj": 413.58283766782114 + "energyKj": 413.58283766782114, + "participating": 4, + "allocated": 4, + "wallSeconds": 3006.626674519968, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 2751.151140021047, + "enforcedLimitW": 2800, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } } ], "fidelity": null, @@ -247,7 +319,19 @@ "p50": 77.94495720259147, "p90": 78.31597462110221, "clipsGpuHour": 11.52605424609145, - "energyKj": 301.08754501100987 + "energyKj": 301.08754501100987, + "participating": 4, + "allocated": 8, + "wallSeconds": 1561.6792716470081, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 3856.016973742528, + "enforcedLimitW": 4000, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } }, { "id": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82:c2", @@ -266,7 +350,19 @@ "p50": 153.6016560649732, "p90": 154.1683327689534, "clipsGpuHour": 11.691563929313219, - "energyKj": 300.10551714629605 + "energyKj": 300.10551714629605, + "participating": 4, + "allocated": 8, + "wallSeconds": 1539.571618376067, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 3898.6355959505895, + "enforcedLimitW": 4000, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } }, { "id": "ab90da9236114cd13de833a364159b8f84e69060de888f3e813635d5b7b26b82:c4", @@ -285,7 +381,19 @@ "p50": 307.28381997847464, "p90": 308.79631440097, "clipsGpuHour": 11.69172635792307, - "energyKj": 299.96858693604275 + "energyKj": 299.96858693604275, + "participating": 4, + "allocated": 8, + "wallSeconds": 1539.5502297059866, + "durationSeconds": 8, + "frameCount": 192, + "avgPowerW": 3896.903348609339, + "enforcedLimitW": 4000, + "server": { + "tp": 2, + "ulysses": 2, + "attention": null + } } ], "fidelity": null, diff --git a/packages/app/package.json b/packages/app/package.json index 3ee80be32..f327bfc92 100644 --- a/packages/app/package.json +++ b/packages/app/package.json @@ -31,6 +31,7 @@ "cache:invalidate": "bun --env-file=../../.env scripts/invalidate-cache.ts", "cache:warmup": "bun --env-file=../../.env scripts/warmup-cache.ts", "capture:fixtures": "bun scripts/capture-cypress-fixtures.ts", + "fixtures:video-history": "bun scripts/build-video-history-fixture.ts", "guard:zh-copy:chinese-only": "bun scripts/check-zh-chinese-only.ts --chinese-only" }, "dependencies": { diff --git a/packages/app/scripts/build-video-history-fixture.ts b/packages/app/scripts/build-video-history-fixture.ts new file mode 100644 index 000000000..b8752583d --- /dev/null +++ b/packages/app/scripts/build-video-history-fixture.ts @@ -0,0 +1,71 @@ +/** + * Rebuild cypress/fixtures/api/video-history.json from retained StoredArtifact + * exports (frontend exports of the original CI artifacts; no new benchmark and + * no new publication). Entries whose run has no local export are kept verbatim, + * so fidelity-only entries survive a rebuild untouched. + * + * Usage: + * bun scripts/build-video-history-fixture.ts .json> + */ + +import { readdir, readFile, writeFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; + +import { + videoHistoryEntry, + type VideoHistoryEntry, + type VideoHistoryPage, +} from '../src/components/video-benchmark/history'; +import type { StoredArtifact } from '../src/components/video-benchmark/stored'; +import { + FIXTURE_MANIFEST_FILENAME, + type FixtureManifest, + assertFixtureContent, + fixtureSha256, + fixtureTopLevel, +} from '../src/lib/test-fixture-manifest'; + +const FIXTURE_NAME = 'video-history'; +const fixturesDir = resolve(import.meta.dirname, '..', 'cypress', 'fixtures', 'api'); +const source = process.argv[2]; +if (!source) throw new Error('Pass the directory holding retained .json exports'); + +const current = JSON.parse( + await readFile(resolve(fixturesDir, `${FIXTURE_NAME}.json`), 'utf8'), +) as VideoHistoryPage; +const exports = new Map(); +const files = await readdir(source); +for (const file of files.filter((name) => /^\d+\.json$/u.test(name))) { + const saved = JSON.parse(await readFile(resolve(source, file), 'utf8')) as StoredArtifact; + if (saved.storageVersion !== 1) throw new Error(`${file} is not a storageVersion 1 export`); + exports.set(`${saved.runId}.${saved.artifact.id}`, saved); +} + +let rebuilt = 0; +const entries: VideoHistoryEntry[] = current.entries.map((entry) => { + const saved = exports.get(entry.id); + if (!saved) return entry; + rebuilt += 1; + const next = videoHistoryEntry(saved, entry.publishedAt); + // The existing fixture already carries the artifact identity (size, digest); keep it. + return { ...next, artifact: { ...next.artifact, ...entry.artifact } }; +}); +const page: VideoHistoryPage = { schemaVersion: 1, entries, nextPage: current.nextPage }; +assertFixtureContent(FIXTURE_NAME, page); +const body = `${JSON.stringify(page, null, 2)}\n`; +await writeFile(resolve(fixturesDir, `${FIXTURE_NAME}.json`), body); + +const manifestPath = resolve(fixturesDir, FIXTURE_MANIFEST_FILENAME); +const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as FixtureManifest; +const capturedAt = new Date().toISOString(); +manifest.fixtures[FIXTURE_NAME] = { + bytes: Buffer.byteLength(body), + capturedAt, + sha256: fixtureSha256(body), + source: + 'Retained public H3 metadata replay from original runs via videoHistoryEntry (per-cell GPU counts, board power and server layout); publication time unavailable. No new benchmark or publication.', + topLevel: fixtureTopLevel(page), +}; +manifest.generatedAt = capturedAt; +await writeFile(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); +console.log(`Wrote ${entries.length} entries; ${rebuilt} rebuilt from ${exports.size} exports`); From 2124786b77355d871ba47c1d0093e4431ec63506 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:08:56 -0700 Subject: [PATCH 010/104] feat: add video dashboard config bar, hardware chart, KPI cards and table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增视频仪表板的构件:读取已发布历史的 hook、基准测试/图表配置面板、带 Pareto 包络与排队尾迹的 D3 硬件图、每硬件 KPI 卡(含 MI355X 未测得卡)、可排序数据表;VideoPoint 增加 model/workload 字段并按 cell 保留最新观测。 --- .../video-benchmark/VideoConfigBar.tsx | 132 +++++++++++ .../video-benchmark/VideoHardwareChart.tsx | 211 ++++++++++++++++++ .../video-benchmark/VideoKpiCards.tsx | 117 ++++++++++ .../video-benchmark/VideoPointsTable.tsx | 121 ++++++++++ .../video-benchmark/metrics.test.ts | 2 + .../src/components/video-benchmark/metrics.ts | 3 + .../components/video-benchmark/points.test.ts | 23 +- .../src/components/video-benchmark/points.ts | 18 ++ .../video-benchmark/use-video-points.ts | 53 +++++ 9 files changed, 679 insertions(+), 1 deletion(-) create mode 100644 packages/app/src/components/video-benchmark/VideoConfigBar.tsx create mode 100644 packages/app/src/components/video-benchmark/VideoHardwareChart.tsx create mode 100644 packages/app/src/components/video-benchmark/VideoKpiCards.tsx create mode 100644 packages/app/src/components/video-benchmark/VideoPointsTable.tsx create mode 100644 packages/app/src/components/video-benchmark/use-video-points.ts diff --git a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx new file mode 100644 index 000000000..d7b1d6d98 --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx @@ -0,0 +1,132 @@ +'use client'; + +import { ControlPanel } from '@/components/ui/control-panel'; +import { track } from '@/lib/analytics'; +import { useLocale } from '@/lib/use-locale'; +import type { CostTier } from './hardware'; +import { + COST_TIERS, + metricLabel, + TIER_LABELS, + X_METRICS, + Y_METRICS, + type GpuBasis, + type XMetricId, + type YMetricId, +} from './metrics'; +import VideoSelect from './VideoSelect'; +import type { VideoDashboardState } from './video-url-state'; + +const STRINGS = { + en: { + benchmark: 'Benchmark config', + chart: 'Chart config', + model: 'Model', + workload: 'Workload', + load: 'Load', + c1: 'C1 · one request per deployment', + all: 'C1 + C2/C4 queueing tails', + x: 'X-axis metric', + y: 'Y-axis metric', + tier: 'Cost tier', + basis: 'GPU basis', + participating: 'Participating GPUs (boards generating the clip)', + allocated: 'Allocated GPUs (boards the job reserved)', + }, + zh: { + benchmark: '基准测试配置', + chart: '图表配置', + model: '模型', + workload: '工作负载', + load: '负载', + c1: 'C1 · 每个部署一个请求', + all: 'C1 + C2/C4 排队尾迹', + x: 'X 轴指标', + y: 'Y 轴指标', + tier: '成本分档', + basis: 'GPU 口径', + participating: '参与计算的 GPU(实际生成视频的板卡)', + allocated: '已分配的 GPU(作业预留的板卡)', + }, +}; + +/** + * Two control panels mirroring the inference tab: what is being compared + * (frozen for this campaign, so model and workload are single-option selects + * that state the comparison rather than change it) and how it is plotted. + */ +export default function VideoConfigBar({ + state, + onChange, + modelLabel, + workloadLabel, +}: { + state: VideoDashboardState; + onChange: (patch: Partial) => void; + modelLabel: string; + workloadLabel: string; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const options = { tier: state.tier, basis: state.basis }; + const change = (key: K, value: VideoDashboardState[K]) => { + onChange({ [key]: value } as Partial); + track(`video_${key}_changed`, { value: String(value) }); + }; + return ( +
+ + {}} + options={[{ value: 'model', label: modelLabel }]} + /> + {}} + options={[{ value: 'workload', label: workloadLabel }]} + /> + change('queue', value === 'all')} + options={[ + { value: 'c1', label: s.c1 }, + { value: 'all', label: s.all }, + ]} + /> + + + change('x', value as XMetricId)} + options={X_METRICS.map((id) => ({ value: id, label: metricLabel(id, locale, options) }))} + /> + change('y', value as YMetricId)} + options={Y_METRICS.map((id) => ({ value: id, label: metricLabel(id, locale, options) }))} + /> + change('tier', value as CostTier)} + options={COST_TIERS.map((tier) => ({ value: tier, label: TIER_LABELS[tier][locale] }))} + /> + change('basis', value as GpuBasis)} + options={[ + { value: 'participating', label: s.participating }, + { value: 'allocated', label: s.allocated }, + ]} + /> + +
+ ); +} diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx new file mode 100644 index 000000000..b9b91d560 --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -0,0 +1,211 @@ +'use client'; + +import { curveLinear, type Selection } from 'd3'; +import { D3Chart } from '@/lib/d3-chart/D3Chart'; +import type { ContinuousScale } from '@/lib/d3-chart/types'; +import { CHART_TYPE, px } from '@/lib/d3-chart/typography'; +import { useLocale } from '@/lib/use-locale'; +import { escapeHtml } from '@/lib/utils'; +import { paretoFrontier } from './frontier'; +import { hardwareLabel } from './hardware'; +import { formatMetric, metricLabel, metricValue, VIDEO_METRICS, type VideoPoint } from './metrics'; +import { latestVideoCells } from './points'; +import type { VideoDashboardState } from './video-url-state'; + +export const VIDEO_CHART_ID = 'video-hardware'; +/** Matches ChartSection's default `${analyticsPrefix}_zoom_reset_${chartId}`. */ +export const VIDEO_ZOOM_RESET_EVENT = `video_zoom_reset_${VIDEO_CHART_ID}`; + +export interface PlottedVideoPoint extends VideoPoint { + x: number; + y: number; + color: string; + label: string; +} + +/** + * Points with both axes available (newest observation per hardware cell), the + * cross-hardware Pareto hull over the C1 cells, and per-hardware queueing tails + * (C1→C2→C4) when the load control shows them. + */ +export function plotVideoPoints( + points: VideoPoint[], + state: VideoDashboardState, + colorFor: (hardwareKey: string) => string, + hidden: ReadonlySet, +) { + const options = { tier: state.tier, basis: state.basis }; + const plotted: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { + if (!p.hardwareKey || hidden.has(p.hardwareKey)) return []; + if (!state.queue && p.concurrency !== 1) return []; + const x = metricValue(p, state.x, options); + const y = metricValue(p, state.y, options); + if (x === null || y === null) return []; + return [{ ...p, x, y, color: colorFor(p.hardwareKey), label: hardwareLabel(p.hardwareKey) }]; + }); + const frontier = paretoFrontier( + plotted.filter((p) => p.concurrency === 1), + VIDEO_METRICS[state.x].polarity, + VIDEO_METRICS[state.y].polarity, + ); + const optimal = new Set(frontier.map((p) => p.hardwareKey)); + const visible = state.optimal ? plotted.filter((p) => optimal.has(p.hardwareKey)) : plotted; + const tails: Record = {}; + if (state.queue) + for (const p of visible) { + if (!p.hardwareKey) continue; + (tails[p.hardwareKey] ??= []).push({ x: p.x, y: p.y }); + } + for (const key of Object.keys(tails)) { + tails[key].sort((a, b) => a.x - b.x); + if (tails[key].length < 2) delete tails[key]; + } + return { plotted: visible, frontier, tails }; +} + +const STRINGS = { + en: { + hull: 'Dashed hull joins the Pareto-optimal hardware at C1. Dotted tails show C2/C4 client concurrency on the batch-one server: requests queue, throughput stays flat, so they are evidence, not a frontier.', + controls: + 'Shift+scroll to zoom; drag to pan; double-click to reset. Click a point to pin its details.', + n: 'valid samples', + run: 'run', + noData: 'No hardware has both selected metrics for this configuration.', + }, + zh: { + hull: '虚线连接 C1 下 Pareto 最优的硬件;点线尾迹为 batch-one 服务器上 C2/C4 客户端并发的表现:请求排队、吞吐量不变,它们是证据而不是前沿。', + controls: 'Shift+滚轮缩放,拖动平移,双击重置。点击数据点可固定详情。', + n: '有效样本', + run: '运行', + noData: '当前配置下没有硬件同时具备所选的两个指标。', + }, +}; + +export default function VideoHardwareChart({ + points, + state, + colorFor, + hidden, + onSelect, +}: { + points: VideoPoint[]; + state: VideoDashboardState; + colorFor: (hardwareKey: string) => string; + hidden: ReadonlySet; + onSelect?: (point: VideoPoint) => void; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const options = { tier: state.tier, basis: state.basis }; + const { plotted, frontier, tails } = plotVideoPoints(points, state, colorFor, hidden); + const xLabel = metricLabel(state.x, locale, options); + const yLabel = metricLabel(state.y, locale, options); + const drawLabels = ( + group: Selection, + x: ContinuousScale, + y: ContinuousScale, + ) => { + group + .selectAll('text.video-point-label') + .data(plotted, (p) => p.id) + .join('text') + .attr('class', 'video-point-label') + .attr('x', (p) => x(p.x) + 10) + .attr('y', (p) => y(p.y) - 10) + .attr('font-size', px(CHART_TYPE.axisLabel)) + .attr('font-weight', (p) => (p.concurrency === 1 ? 600 : 400)) + .attr('fill', 'var(--foreground)') + .attr('pointer-events', 'none') + .text((p) => (p.concurrency === 1 ? p.label : `C${p.concurrency}`)); + }; + const maxX = Math.max(0, ...plotted.map((p) => p.x)); + const maxY = Math.max(0, ...plotted.map((p) => p.y)); + return ( + 1 ? { hull: frontier.map((p) => ({ x: p.x, y: p.y })) } : {}, + config: { + getColor: () => 'var(--foreground)', + strokeWidth: 1.5, + strokeDasharray: '6 4', + curve: curveLinear, + }, + }, + { + type: 'line', + key: 'queueing-tails', + lines: tails, + config: { + getColor: (key) => colorFor(key), + getStrokeDasharray: () => '2 4', + strokeWidth: 1.5, + curve: curveLinear, + }, + }, + { + type: 'point', + data: plotted, + config: { + getCx: () => 0, + getCy: () => 0, + getX: (p) => p.x, + getY: (p) => p.y, + getColor: (p) => p.color, + getRadius: (p) => (p.concurrency === 1 ? 7 : 4), + stroke: 'var(--foreground)', + strokeWidth: 1, + keyFn: (p) => p.id, + }, + }, + { + type: 'custom', + key: 'video-point-labels', + render: (group, ctx) => { + drawLabels( + group, + (ctx.renderedXScale ?? ctx.xScale) as ContinuousScale, + (ctx.renderedYScale ?? ctx.yScale) as ContinuousScale, + ); + }, + onZoom: (group, ctx) => + drawLabels(group, ctx.newXScale as ContinuousScale, ctx.newYScale as ContinuousScale), + }, + ]} + tooltip={{ + rulerType: 'none', + content: (p) => + `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}
${escapeHtml(xLabel)}: ${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}: ${formatMetric(p.y, state.y)}
${s.n}: ${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, + onPointClick: onSelect, + }} + zoom={{ + enabled: true, + axes: 'both', + scaleExtent: [1, 20], + resetEventName: VIDEO_ZOOM_RESET_EVENT, + }} + instructions={s.controls} + noDataOverlay={ + plotted.length === 0 ? ( +

+ {s.noData} +

+ ) : undefined + } + caption={

{s.hull}

} + /> + ); +} diff --git a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx new file mode 100644 index 000000000..825c277cb --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx @@ -0,0 +1,117 @@ +'use client'; + +import { Card } from '@/components/ui/card'; +import { useLocale } from '@/lib/use-locale'; +import { hardwareLabel, VIDEO_HARDWARE_ROSTER } from './hardware'; +import { formatMetric, metricValue, type MetricId, type VideoPoint } from './metrics'; +import { latestVideoCells } from './points'; +import type { VideoDashboardState } from './video-url-state'; + +const CARD_METRICS = [ + 'p50Latency', + 'videosPerGpuHour', + 'dollarsPerVideo', + 'kjPerVideo', + 'powerPctCap', +] as const satisfies readonly MetricId[]; +const STRINGS = { + en: { + p50Latency: 'P50 time to video (s)', + videosPerGpuHour: 'Videos / GPU-hr', + dollarsPerVideo: 'TCO / video', + kjPerVideo: 'kJ / video', + powerPctCap: 'Board power / limit', + p90: 'P90', + gpus: 'GPUs', + of: 'of', + unavailable: 'Not measured', + why: 'Failed run', + }, + zh: { + p50Latency: 'P50 出片时间(s)', + videosPerGpuHour: '视频数 / GPU 小时', + dollarsPerVideo: 'TCO / 条', + kjPerVideo: 'kJ / 条', + powerPctCap: '板卡功率 / 上限', + p90: 'P90', + gpus: '张 GPU', + of: '/', + unavailable: '未测得', + why: '失败的运行', + }, +}; + +/** One card per campaign hardware at C1; hardware without a valid run says so instead of vanishing. */ +export default function VideoKpiCards({ + points, + state, + colorFor, +}: { + points: VideoPoint[]; + state: VideoDashboardState; + colorFor: (hardwareKey: string) => string; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const options = { tier: state.tier, basis: state.basis }; + const cells = latestVideoCells(points); + return ( +
+ {VIDEO_HARDWARE_ROSTER.map(({ key, unavailable }) => { + const point = cells.find((p) => p.hardwareKey === key && p.concurrency === 1); + return ( + +
+ + {hardwareLabel(key)} + {point && ( + + {point.participating ?? '—'} {s.of} {point.allocated ?? '—'} {s.gpus} + + )} +
+ {point ? ( +
+ {CARD_METRICS.map((id) => ( +
+
{s[id]}
+
+ {formatMetric(metricValue(point, id, options), id)} + {id === 'powerPctCap' && metricValue(point, id, options) !== null && '%'} + {id === 'p50Latency' && ( + + {s.p90}{' '} + {formatMetric(metricValue(point, 'p90Latency', options), 'p90Latency')} + + )} +
+
+ ))} +
+ ) : ( +

+ {s.unavailable}.{' '} + {unavailable && ( + <> + {unavailable[locale]}{' '} + + {s.why} + + + )} +

+ )} +
+ ); + })} +
+ ); +} diff --git a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx new file mode 100644 index 000000000..0051a3914 --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx @@ -0,0 +1,121 @@ +'use client'; + +import { DataTable, type DataTableColumn } from '@/components/ui/data-table'; +import { useLocale } from '@/lib/use-locale'; +import { hardwareLabel } from './hardware'; +import { formatMetric, metricLabel, metricValue, type MetricId, type VideoPoint } from './metrics'; +import { latestVideoCells } from './points'; +import type { VideoDashboardState } from './video-url-state'; + +const STRINGS = { + en: { + hardware: 'Hardware', + c: 'C', + counts: 'Valid / scheduled', + power: 'Board power (W)', + run: 'CI run', + }, + zh: { + hardware: '硬件', + c: 'C', + counts: '有效 / 计划', + power: '板卡功率(W)', + run: 'CI 运行', + }, +}; +const METRIC_COLUMNS: readonly MetricId[] = [ + 'p50Latency', + 'p90Latency', + 'videosPerGpuHour', + 'videosPerDollar', + 'dollarsPerVideo', + 'kjPerVideo', + 'powerPctCap', +]; + +/** Rows for the plotted cells: every metric the chart can show, plus provenance. */ +export function videoTableRows( + points: VideoPoint[], + state: VideoDashboardState, + hidden: ReadonlySet, +): VideoPoint[] { + return latestVideoCells(points).filter( + (p) => p.hardwareKey && !hidden.has(p.hardwareKey) && (state.queue || p.concurrency === 1), + ); +} + +export default function VideoPointsTable({ + points, + state, + hidden, +}: { + points: VideoPoint[]; + state: VideoDashboardState; + hidden: ReadonlySet; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const options = { tier: state.tier, basis: state.basis }; + const rows = videoTableRows(points, state, hidden); + const columns: DataTableColumn[] = [ + { + header: s.hardware, + cell: (p) => hardwareLabel(p.hardwareKey ?? ''), + sortValue: (p) => hardwareLabel(p.hardwareKey ?? ''), + importance: 'key', + pinned: true, + }, + { + header: s.c, + align: 'right', + cell: (p) => p.concurrency ?? '—', + sortValue: (p) => p.concurrency ?? 0, + importance: 'key', + }, + { + header: s.counts, + align: 'right', + cell: (p) => `${p.valid ?? '—'} / ${p.scheduled ?? '—'}`, + }, + ...METRIC_COLUMNS.map((id): DataTableColumn => ({ + header: metricLabel(id, locale, options), + align: 'right', + cell: (p) => formatMetric(metricValue(p, id, options), id), + sortValue: (p) => metricValue(p, id, options) ?? Number.NEGATIVE_INFINITY, + importance: id === 'p50Latency' || id === 'videosPerDollar' ? 'key' : 'secondary', + })), + { + header: s.power, + align: 'right', + cell: (p) => + p.avgPowerW === null + ? '—' + : p.avgPowerW.toLocaleString('en-US', { maximumFractionDigits: 0 }), + sortValue: (p) => p.avgPowerW ?? 0, + importance: 'secondary', + }, + { + header: s.run, + cell: (p) => ( + + #{p.runId} + + ), + importance: 'secondary', + }, + ]; + return ( + + ); +} diff --git a/packages/app/src/components/video-benchmark/metrics.test.ts b/packages/app/src/components/video-benchmark/metrics.test.ts index 2d427d0d5..e26bc1752 100644 --- a/packages/app/src/components/video-benchmark/metrics.test.ts +++ b/packages/app/src/components/video-benchmark/metrics.test.ts @@ -10,6 +10,8 @@ const h200: VideoPoint = { hardwareKey: 'h200', hardwareName: 'NVIDIA H200', runtime: '71de97b264b04dcd514cf904003028aefe9775c8', + model: 'MiniMaxAI/MiniMax-H3', + workload: '1344 × 768 · 8 s · 24 fps · 50 steps · MiniMaxAI/MiniMax-H3 @ 42ed227ee7df', concurrency: 1, participating: 4, allocated: 4, diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts index f0ab62b04..5f09eebe8 100644 --- a/packages/app/src/components/video-benchmark/metrics.ts +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -12,6 +12,9 @@ export interface VideoPoint { hardwareKey: string | null; hardwareName: string; runtime: string; + model: string; + /** Human-readable workload summary from the projection (resolution · duration · fps · steps · model @ rev · seeds · prompt). */ + workload: string; concurrency: number | null; participating: number | null; allocated: number | null; diff --git a/packages/app/src/components/video-benchmark/points.test.ts b/packages/app/src/components/video-benchmark/points.test.ts index a893e2a47..64a25c31b 100644 --- a/packages/app/src/components/video-benchmark/points.test.ts +++ b/packages/app/src/components/video-benchmark/points.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; import type { VideoHistoryObservation, VideoHistoryPage } from './history'; -import { videoPoints } from './points'; +import { latestVideoCells, videoPoints } from './points'; const observation: VideoHistoryObservation = { id: 'sha:c1', @@ -153,4 +153,25 @@ describe('videoPoints', () => { expect(point.server).toBeNull(); expect(point.hardwareKey).toBe('h200'); }); + it('keeps only the newest observation per hardware cell', () => { + const rerun = { + ...page.entries[0], + id: '5.5', + runId: '5', + sources: [ + { + ...source, + id: '5', + observations: [{ ...observation, id: 'again:c1', p50: 140 }], + }, + ], + }; + // Entry order is publication order (newest first): the rerun supersedes the older cell. + const points = videoPoints([{ ...page, entries: [rerun, page.entries[0]] }]); + expect(points.map((p) => p.id)).toEqual(['again:c1', 'sha:c1', 'sha:c2']); + expect(latestVideoCells(points).map((p) => [p.id, p.p50])).toEqual([ + ['again:c1', 140], + ['sha:c2', 150.6], + ]); + }); }); diff --git a/packages/app/src/components/video-benchmark/points.ts b/packages/app/src/components/video-benchmark/points.ts index 519696f9b..bac2848fa 100644 --- a/packages/app/src/components/video-benchmark/points.ts +++ b/packages/app/src/components/video-benchmark/points.ts @@ -46,6 +46,8 @@ function toPoint( hardwareKey: hardwareKey(o.hardware), hardwareName: o.hardware, runtime: o.runtime ?? '', + model: o.model ?? '', + workload: o.workload ?? '', concurrency: num(o.concurrency), participating: num(o.participating), allocated: num(o.allocated), @@ -74,3 +76,19 @@ function toPoint( observedAt, }; } + +/** + * Newest published observation per (hardware, concurrency) cell. Points arrive + * newest publication first, so the first occurrence wins; hardware without a + * registry key is kept as-is because nothing can be compared against it. + */ +export function latestVideoCells(points: VideoPoint[]): VideoPoint[] { + const seen = new Set(); + return points.filter((point) => { + if (point.hardwareKey === null) return true; + const key = `${point.hardwareKey}:${point.concurrency ?? 'na'}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); +} diff --git a/packages/app/src/components/video-benchmark/use-video-points.ts b/packages/app/src/components/video-benchmark/use-video-points.ts new file mode 100644 index 000000000..1892c7d79 --- /dev/null +++ b/packages/app/src/components/video-benchmark/use-video-points.ts @@ -0,0 +1,53 @@ +'use client'; + +import { useCallback, useEffect, useState } from 'react'; +import type { VideoHistoryPage } from './history'; +import type { VideoPoint } from './metrics'; +import { videoPoints } from './points'; + +/** Published pages are read newest first; the frozen campaign fits in far fewer. */ +const MAX_PAGES = 5; + +/** Load every published history page (bounded) and flatten it into chart points. */ +export function useVideoPoints() { + const [points, setPoints] = useState([]); + const [loading, setLoading] = useState(true); + const [loadError, setLoadError] = useState(null); + const [replay, setReplay] = useState(false); + const [attempt, setAttempt] = useState(0); + useEffect(() => { + const controller = new AbortController(); + setLoading(true); + setLoadError(null); + (async () => { + const pages: VideoHistoryPage[] = []; + let replayed = false; + for (let page = 1; page <= MAX_PAGES; page++) { + const response = await fetch(`/api/video-runs?format=history&page=${page}`, { + signal: controller.signal, + cache: 'no-store', + }); + if (!response.ok) throw new Error(`Published history HTTP ${response.status}`); + replayed ||= response.headers.get('x-videogenx-replay') === '1'; + const data = (await response.json()) as VideoHistoryPage; + if (data.schemaVersion !== 1 || !Array.isArray(data.entries)) + throw new Error('Unsupported published history projection'); + pages.push(data); + if (data.nextPage === null) break; + } + if (controller.signal.aborted) return; + setPoints(videoPoints(pages)); + setReplay(replayed); + })() + .catch((error: unknown) => { + if (controller.signal.aborted) return; + setLoadError(error instanceof Error ? error.message : String(error)); + }) + .finally(() => { + if (!controller.signal.aborted) setLoading(false); + }); + return () => controller.abort(); + }, [attempt]); + const retry = useCallback(() => setAttempt((n) => n + 1), []); + return { points, loading, error: loadError, replay, retry }; +} From 490e4be55455c5ba5e5e791b963d24cdadf4f9f3 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:09:20 -0700 Subject: [PATCH 011/104] feat: make the cross-hardware dashboard the /video landing view MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:/video 与 /zh/video 改为以跨硬件仪表板为首屏:配置栏、带 Pareto 包络的硬件图与图例侧栏、TCO 单价与来源、KPI 卡、图表/表格切换与 CSV 导出;原运行/视频/历史视图收进可展开的“运行、视频与证据”区块,含 run/view/compare/history-* 参数的深链接自动展开。 --- .../cypress/component/video-dashboard.cy.tsx | 100 +++++ packages/app/cypress/e2e/navigation.cy.ts | 2 + .../app/src/app/(dashboard)/video/page.tsx | 4 +- .../app/src/app/zh/(dashboard)/video/page.tsx | 4 +- .../video-benchmark/VideoDashboard.tsx | 400 ++++++++++++++++++ 5 files changed, 506 insertions(+), 4 deletions(-) create mode 100644 packages/app/cypress/component/video-dashboard.cy.tsx create mode 100644 packages/app/src/components/video-benchmark/VideoDashboard.tsx diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx new file mode 100644 index 000000000..4464ce9cf --- /dev/null +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -0,0 +1,100 @@ +import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; +import VideoDashboard from '@/components/video-benchmark/VideoDashboard'; + +// Retained H100/H200/B200 C1/C2/C4 observations (cypress/fixtures/api/video-history.json). +function mount(pathname = '/video', search = '') { + cy.intercept('GET', '/api/video-runs?format=history&page=1', { + fixture: 'api/video-history.json', + }).as('history'); + cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + cy.window().then((win) => + win.history.replaceState(null, '', `${win.location.pathname}${search}`), + ); + cy.mount( + + + , + ); + cy.wait('@history'); +} +const kpi = (hardware: string) => + cy.get(`[data-testid="video-kpi-card"][data-hardware="${hardware}"]`); +const points = () => cy.get('[data-testid="video-hardware-chart"] circle.point'); +const tails = () => cy.get('[data-testid="video-hardware-chart"] path.line-path'); + +describe('Video hardware dashboard (retained fixture)', () => { + it('plots one C1 point per measured hardware and lists MI355X as unavailable', () => { + mount(); + points().should('have.length', 3); + tails().should('have.length', 0); + cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); + kpi('mi355x').find('[data-testid="video-kpi-unavailable"]').should('contain', 'Not measured'); + kpi('h200').should('contain', '$0.204').and('contain', '150.6').and('contain', '97.4'); + kpi('b200').should('contain', '$0.150'); + kpi('h100').should('contain', '$0.218').and('contain', '4 of 8 GPUs'); + cy.get('[data-testid="video-config-bar"]').should( + 'contain', + '1344 × 768 · 8 s · 24 fps · 50 steps', + ); + cy.get('[data-testid="video-chart-card"]').should( + 'contain', + 'MiniMaxAI/MiniMax-H3 @ 42ed227ee7df', + ); + }); + it('switches cost tier and GPU basis, reflecting both in the URL and the cards', () => { + mount(); + cy.get('[role="combobox"][aria-label="Cost tier"]').click(); + cy.contains('[role="option"]', 'Rent - 3 Year Commit').click(); + // 2.9 $/hr ÷ 5.9732 videos/GPU-hr = 0.48550 → rounds down at three decimals. + kpi('h200').should('contain', '$0.485'); + cy.location('search').should('contain', 'v_tier=r'); + cy.get('[data-testid="video-tco-badge"]') + .first() + .should('contain', 'B200') + .and('contain', '3.70'); + cy.get('[role="combobox"][aria-label="GPU basis"]').click(); + cy.contains('[role="option"]', 'Allocated GPUs').click(); + kpi('h100').should('contain', '$0.744'); + cy.location('search').should('contain', 'v_basis=allocated'); + }); + it('adds dotted C2/C4 tails behind the queueing switch and isolates the Pareto-optimal hardware', () => { + mount(); + cy.get('[data-testid="video-queue"]').click(); + points().should('have.length', 9); + tails().should('have.length', 3); + cy.location('search').should('contain', 'v_queue=1'); + cy.get('[data-testid="video-optimal"]').click(); + points().should('have.length', 3); + tails().should('have.length', 1); + cy.location('search').should('contain', 'v_opt=1'); + }); + it('hides a hardware from the legend and shows the table view', () => { + mount(); + cy.contains('[data-testid="video-legend"] li', 'H100').click(); + points().should('have.length', 2); + cy.contains('button', 'Table').click(); + cy.get('[data-testid="video-points-table"] tbody tr').should('have.length', 2); + cy.get('[data-testid="video-points-table"]') + .should('contain', 'H200') + .and('contain', 'B200') + .and('not.contain', 'H100'); + cy.location('search').should('contain', 'v_view=table'); + cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); + }); + it('restores v_ params from the URL and opens the runs section for run deep links', () => { + mount('/video', '?v_y=kjPerVideo&v_x=genSpeed&v_queue=1&view=history'); + points().should('have.length', 9); + cy.get('[data-testid="video-chart-card"]').should('contain', 'GPU-board energy per video (kJ)'); + cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); + cy.get('[data-testid="video-ci-runs"]').should('exist'); + }); + it('renders Chinese copy on /zh/video', () => { + mount('/zh/video'); + cy.get('[data-testid="video-chart-card"]').should( + 'contain', + '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', + ); + kpi('mi355x').should('contain', '未测得'); + cy.get('[data-testid="video-config-bar"]').should('contain', '成本分档'); + }); +}); diff --git a/packages/app/cypress/e2e/navigation.cy.ts b/packages/app/cypress/e2e/navigation.cy.ts index 2a1391ddb..fbe8cf70e 100644 --- a/packages/app/cypress/e2e/navigation.cy.ts +++ b/packages/app/cypress/e2e/navigation.cy.ts @@ -391,6 +391,8 @@ describe('H3 video artifact viewer', () => { }).as('history'); cy.visit('/video'); cy.wait('@history'); + // The hardware dashboard leads; the run/history views live in a collapsible section. + cy.get('[data-testid="video-runs-section"] summary').click(); cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); cy.get('[data-testid="video-history-observation"]').should('have.length', 3); cy.get('[aria-label="Client concurrency"]').click(); diff --git a/packages/app/src/app/(dashboard)/video/page.tsx b/packages/app/src/app/(dashboard)/video/page.tsx index b2a7a8458..df20cb200 100644 --- a/packages/app/src/app/(dashboard)/video/page.tsx +++ b/packages/app/src/app/(dashboard)/video/page.tsx @@ -1,5 +1,5 @@ import type { Metadata } from 'next'; -import VideoCIRuns from '@/components/video-benchmark/VideoCIRuns'; +import VideoDashboard from '@/components/video-benchmark/VideoDashboard'; import { tabMetadata } from '@/lib/tab-meta'; export const metadata: Metadata = { @@ -7,5 +7,5 @@ export const metadata: Metadata = { robots: { index: false, follow: false }, }; export default function VideoPage() { - return ; + return ; } diff --git a/packages/app/src/app/zh/(dashboard)/video/page.tsx b/packages/app/src/app/zh/(dashboard)/video/page.tsx index 7716d19ca..f85e612d5 100644 --- a/packages/app/src/app/zh/(dashboard)/video/page.tsx +++ b/packages/app/src/app/zh/(dashboard)/video/page.tsx @@ -1,5 +1,5 @@ import type { Metadata } from 'next'; -import VideoCIRuns from '@/components/video-benchmark/VideoCIRuns'; +import VideoDashboard from '@/components/video-benchmark/VideoDashboard'; import { tabMetadataZh } from '@/lib/tab-meta-zh'; export const metadata: Metadata = { @@ -7,5 +7,5 @@ export const metadata: Metadata = { robots: { index: false, follow: false }, }; export default function VideoPage() { - return ; + return ; } diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx new file mode 100644 index 000000000..2feb7dd5c --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -0,0 +1,400 @@ +'use client'; + +import { useCallback, useEffect, useMemo, useState } from 'react'; +import { + HW_REGISTRY, + TCO_SOURCE_TITLE, + TCO_SOURCE_URL, +} from '@semianalysisai/inferencex-constants'; +import { Button } from '@/components/ui/button'; +import ChartLegend from '@/components/ui/chart-legend'; +import { ChartSection } from '@/components/ui/chart-section'; +import { Heading } from '@/components/ui/heading'; +import { useThemeColors } from '@/hooks/useThemeColors'; +import { track } from '@/lib/analytics'; +import { useLocale } from '@/lib/use-locale'; +import { costPerGpuHour, hardwareLabel, VIDEO_HARDWARE_ROSTER } from './hardware'; +import { + metricLabel, + metricValue, + TIER_LABELS, + VIDEO_METRICS, + type MetricId, + type VideoPoint, +} from './metrics'; +import { latestVideoCells } from './points'; +import { useVideoPoints } from './use-video-points'; +import VideoCIRuns from './VideoCIRuns'; +import VideoConfigBar from './VideoConfigBar'; +import VideoHardwareChart, { VIDEO_CHART_ID } from './VideoHardwareChart'; +import VideoKpiCards from './VideoKpiCards'; +import VideoPointsTable, { videoTableRows } from './VideoPointsTable'; +import { + DEFAULT_VIDEO_DASHBOARD_STATE, + readVideoDashboardState, + writeVideoDashboardState, + type VideoDashboardState, +} from './video-url-state'; + +/** URL params owned by the run/results/history views; any of them opens that section on load. */ +const RUNS_SECTION_PARAMS = [ + 'run', + 'artifact', + 'source', + 'cell', + 'view', + 'compare', + 'history-hardware', + 'history-concurrency', + 'history-query', +]; +const CSV_METRICS: readonly MetricId[] = [ + 'p50Latency', + 'p90Latency', + 'genSpeed', + 'videosPerGpuHour', + 'videoSecondsPerGpuHour', + 'videosPerDollar', + 'dollarsPerVideo', + 'dollarsPerVideoSecond', + 'kjPerVideo', + 'videosPerKwh', + 'powerPctCap', +]; + +const STRINGS = { + en: { + title: 'VideoGenX · MiniMax-H3 across hardware', + subtitle: + 'One frozen workload, one model and runtime, measured on each GPU: time to video, useful output per GPU-hour and per TCO dollar, and metered GPU-board energy.', + chart: 'Chart', + table: 'Table', + viewToggle: 'Chart or table view', + tier: 'Cost tier', + badges: 'TCO $/chip/hr', + source: 'Source', + queue: 'Show queueing (C2/C4)', + optimal: 'Optimal only', + runs: 'Runs, videos & evidence', + runsHint: 'Per-run results, generated clips, fidelity checks and the performance history list.', + loading: 'Loading published results…', + error: 'Could not load published results', + retry: 'Retry', + replay: 'Replaying retained results (local fixture), not a new measurement.', + vs: 'vs.', + notMeasured: 'not measured', + runtime: 'runtime', + workloads: (n: number) => ` (+${n} more)`, + }, + zh: { + title: 'VideoGenX · MiniMax-H3 跨硬件对比', + subtitle: + '同一冻结的工作负载、同一模型与运行时,在每种 GPU 上实测:出片时间、每 GPU 小时与每美元 TCO 的有效产出、GPU 板卡实测能耗。', + chart: '图表', + table: '表格', + viewToggle: '图表或表格视图', + tier: '成本分档', + badges: 'TCO $/chip/hr', + source: '来源', + queue: '显示排队尾迹(C2/C4)', + optimal: '仅最优', + runs: '运行、视频与证据', + runsHint: '按运行查看结果、生成的视频、保真度检查以及性能历史列表。', + loading: '正在加载已发布结果…', + error: '无法加载已发布结果', + retry: '重试', + replay: '正在回放保留结果(本地 fixture),不是新的测量。', + vs: 'vs.', + notMeasured: '未测得', + runtime: 'runtime', + workloads: (n: number) => `(另有 ${n} 个)`, + }, +}; + +function csvCell(value: string | number | null): string { + if (value === null) return ''; + const text = String(value); + return /[",\n]/u.test(text) ? `"${text.replaceAll('"', '""')}"` : text; +} + +export default function VideoDashboard() { + const locale = useLocale(); + const s = STRINGS[locale]; + const { points, loading, error, replay, retry } = useVideoPoints(); + const [state, setState] = useState(DEFAULT_VIDEO_DASHBOARD_STATE); + const [hidden, setHidden] = useState>(() => new Set()); + const [legendExpanded, setLegendExpanded] = useState(true); + const [runsOpen, setRunsOpen] = useState(false); + useEffect(() => { + setState(readVideoDashboardState(location.search)); + const params = new URLSearchParams(location.search); + setRunsOpen(RUNS_SECTION_PARAMS.some((key) => params.has(key))); + }, []); + const update = useCallback((patch: Partial) => { + setState((old) => { + const next = { ...old, ...patch }; + history.replaceState(null, '', writeVideoDashboardState(new URL(location.href), next)); + return next; + }); + }, []); + + const hardwareKeys = useMemo(() => VIDEO_HARDWARE_ROSTER.map((item) => item.key), []); + const { resolveColor, getCssColor } = useThemeColors({ + highContrast: false, + activeKeys: hardwareKeys, + }); + const colorFor = useCallback( + (key: string) => getCssColor(resolveColor(key)), + [resolveColor, getCssColor], + ); + + const options = { tier: state.tier, basis: state.basis }; + const cells = useMemo(() => latestVideoCells(points), [points]); + const measured = useMemo( + () => new Map(cells.filter((p) => p.concurrency === 1).map((p) => [p.hardwareKey, p])), + [cells], + ); + const lead = cells.find((p) => p.concurrency === 1) ?? cells[0]; + // "1344 × 768 · 8 s · 24 fps · 50 steps · model @ rev · seeds · prompt" → shape, then model @ rev. + const workloadParts = lead?.workload.split(' · ') ?? []; + const workloadLabel = workloadParts.slice(0, 4).join(' · ') || '—'; + const modelLabel = workloadParts[4] ?? lead?.model ?? '—'; + const otherWorkloads = new Set(cells.map((p) => p.workload.split(' · ').slice(0, 5).join(' · '))) + .size; + const workloadSuffix = otherWorkloads > 1 ? s.workloads(otherWorkloads - 1) : ''; + + const legendItems = VIDEO_HARDWARE_ROSTER.map(({ key, unavailable }) => { + const point = measured.get(key); + return { + name: key, + hw: key, + label: point + ? `${hardwareLabel(key)} · ${s.runtime} ${point.runtime.slice(0, 8)}` + : `${hardwareLabel(key)} · ${s.notMeasured}`, + color: colorFor(key), + isActive: point !== undefined && !hidden.has(key), + isRemovable: point !== undefined, + title: unavailable?.[locale], + onClick: (name: string) => { + if (!measured.has(name)) return; + setHidden((old) => { + const next = new Set(old); + if (next.has(name)) next.delete(name); + else next.add(name); + return next; + }); + track('video_legend_toggled', { hardware: name }); + }, + }; + }); + + const exportCsv = () => { + const rows = videoTableRows(points, state, hidden); + const header = [ + 'hardware', + 'concurrency', + 'valid', + 'scheduled', + 'participating_gpus', + 'allocated_gpus', + ...CSV_METRICS.map((id) => `${id} (${VIDEO_METRICS[id].unit})`), + 'board_power_w', + 'enforced_limit_w', + 'runtime', + 'ci_run', + ]; + const lines = rows.map((p) => + [ + hardwareLabel(p.hardwareKey ?? ''), + p.concurrency, + p.valid, + p.scheduled, + p.participating, + p.allocated, + ...CSV_METRICS.map((id) => metricValue(p, id, options)), + p.avgPowerW, + p.enforcedLimitW, + p.runtime, + `https://github.com/SemiAnalysisAI/InferenceX/actions/runs/${p.runId}`, + ] + .map(csvCell) + .join(','), + ); + const blob = new Blob([`${[header.join(','), ...lines].join('\n')}\n`], { + type: 'text/csv;charset=utf-8', + }); + const url = URL.createObjectURL(blob); + const anchor = document.createElement('a'); + anchor.href = url; + anchor.download = `videogenx-${state.y}-vs-${state.x}-${state.tier}-${state.basis}.csv`; + anchor.click(); + URL.revokeObjectURL(url); + }; + + return ( +
+
+ + {s.title} + +

{s.subtitle}

+
+ + + {(['chart', 'table'] as const).map((view) => ( + + ))} +
+ } + > +
+
+ + {modelLabel} · {metricLabel(state.y, locale, options)} {s.vs}{' '} + {metricLabel(state.x, locale, options)} + +

+ + {s.tier}: {TIER_LABELS[state.tier][locale]} + + + {s.badges}: + {hardwareKeys.map((key) => ( + + {HW_REGISTRY[key]?.badgeLabel ?? hardwareLabel(key)}{' '} + {costPerGpuHour(key, state.tier)?.toFixed(2) ?? '—'} + + ))} + + + {s.source}:{' '} + + {TCO_SOURCE_TITLE} + + +

+
+ {replay && ( +

+ {s.replay} +

+ )} + {loading && ( +

+ {s.loading} +

+ )} + {error && ( +
+

+ {s.error}: {error} +

+ +
+ )} + {!loading && !error && ( +
+
+ {state.view === 'chart' ? ( +
+
+ { + update({ queue: checked }); + track('video_queue_changed', { value: String(checked) }); + }, + }, + { + id: 'video-optimal', + label: s.optimal, + checked: state.optimal, + onCheckedChange: (checked) => { + update({ optimal: checked }); + track('video_optimal_changed', { value: String(checked) }); + }, + }, + ]} + /> +
+
+ )} +
+ + +
{ + const open = event.currentTarget.open; + if (open === runsOpen) return; + setRunsOpen(open); + track('video_runs_section_toggled', { open }); + }} + > + + {s.runs} + {s.runsHint} + +
{runsOpen && }
+
+ + ); +} From cebb64565f84e022dea037e7ded6fa824c304ed9 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:09:30 -0700 Subject: [PATCH 012/104] test: cover the video dashboard end to end with retained fixtures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增视频仪表板 E2E:v_ 参数恢复、排队尾迹、回放提示、history 深链接展开运行区块、中文页面;登记 timings 基线。 --- .../app/cypress/e2e/video-dashboard.cy.ts | 29 +++++++++++++++++++ packages/app/timings.json | 4 +++ 2 files changed, 33 insertions(+) create mode 100644 packages/app/cypress/e2e/video-dashboard.cy.ts diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts new file mode 100644 index 000000000..0d55d052a --- /dev/null +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -0,0 +1,29 @@ +// Runs against the E2E_FIXTURES=1 server, which serves the retained +// H100/H200/B200 observations from cypress/fixtures/api/video-history.json. +describe('Video hardware dashboard (E2E fixtures)', () => { + it('leads with the cross-hardware chart and restores v_ params from the URL', () => { + cy.visit('/video?v_y=kjPerVideo&v_tier=r&v_queue=1'); + cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 9); + cy.get('[data-testid="video-hardware-chart"] path.line-path').should('have.length', 3); + cy.get('[data-testid="video-chart-card"]') + .should('contain', 'GPU-board energy per video (kJ)') + .and('contain', 'Rent - 3 Year Commit'); + cy.get('[data-testid="video-tco-badge"]').first().should('contain', 'B200'); + cy.get('[data-testid="video-replay"]').should('be.visible'); + cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); + cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); + cy.get('[data-testid="video-ci-runs"]').should('not.exist'); + }); + it('opens the runs section for history deep links and renders the Chinese dashboard', () => { + cy.visit('/video?view=history'); + cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); + cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); + cy.visit('/zh/video'); + cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 3); + cy.get('[data-testid="video-chart-card"]').should( + 'contain', + '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', + ); + cy.get('[data-testid="video-kpi-card"][data-hardware="mi355x"]').should('contain', '未测得'); + }); +}); diff --git a/packages/app/timings.json b/packages/app/timings.json index 5ad312efa..bffd90811 100644 --- a/packages/app/timings.json +++ b/packages/app/timings.json @@ -276,6 +276,10 @@ "spec": "cypress/e2e/url-params.cy.ts", "duration": 27250 }, + { + "spec": "cypress/e2e/video-dashboard.cy.ts", + "duration": 8000 + }, { "spec": "cypress/e2e/yaxis-metrics-render.cy.ts", "duration": 7006 From 2cab668588f21d0f55cd2c34326d0ac2cee7347d Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:16:49 -0700 Subject: [PATCH 013/104] docs: explain the VideoGenX dashboard data path, axes and C1 default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增 VideoGenX 仪表板文档(数据路径、坐标轴与默认值、C1 默认原因、Pareto 包络、GPU 口径、未测得硬件与缺失值规则),在文档索引登记,并更新性能历史文档的入口说明。 --- docs/index.md | 1 + docs/video-history.md | 4 ++-- docs/videogenx-dashboard.md | 29 +++++++++++++++++++++++++++++ 3 files changed, 32 insertions(+), 2 deletions(-) create mode 100644 docs/videogenx-dashboard.md diff --git a/docs/index.md b/docs/index.md index 9e05ce715..0727cbcf6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -25,6 +25,7 @@ Design rationale and non-obvious conventions. See [CLAUDE.md](../CLAUDE.md) for - [State Ownership](./state-ownership.md) — Which context owns which state, availability filtering cascade, comparison date mechanics, URL param sync - [Blog](./blog.md) — MDX content system, SEO features (OG images, RSS, llms.txt, JSON-LD), TOC sidebar, reading progress, heading links, analytics events - [CollectiveX](./collectivex.md) — lazy ingest-on-read pipeline (separate Neon DB as a durable GitHub-artifact cache), tombstoned deletes, raw-docs storage + shared reader +- [VideoGenX dashboard](./videogenx-dashboard.md) — `/video` cross-hardware chart: published-history projection → points → metrics, TCO tiers and GPU basis, C1 default with queueing tails, Pareto hull, unavailable-hardware roster, null rules - [Chinese Pages (/zh)](./i18n.md) — Why hand-authored /zh pages instead of an i18n framework, hreflang pairing, blog translation pairing, html lang workaround, CJK reading time/slugs - [Chinese Copy Editorial Guide](./chinese-copy.md) — Audience, surface-specific register, context-aware terminology, rewrite workflow, manual-review process, and PR checklist diff --git a/docs/video-history.md b/docs/video-history.md index 67b12a032..9bdb2f128 100644 --- a/docs/video-history.md +++ b/docs/video-history.md @@ -1,6 +1,6 @@ # Video performance history -`/video` and `/zh/video` default to **Performance history**. Explicit run/artifact/source/cell URLs retain the existing video viewer. `?view=results` opens the first readable artifact/source from the newest published page; `?view=history` restores history. History filters use `history-hardware`, `history-concurrency`, and `history-query` in the URL. Videos & result follows the same published-first path when no run is selected; empty or failed reads show a recoverable message. Browse CI runs explicitly opens legacy Actions discovery. Returning to History cancels pending result selection; Refresh and reload preserve the selected artifact/source/cell. +`/video` and `/zh/video` lead with the cross-hardware dashboard (see [VideoGenX dashboard](./videogenx-dashboard.md)); the run, results and **Performance history** views described here live in the collapsible "Runs, videos & evidence" section below it, which opens automatically when the URL carries any of `run`, `artifact`, `source`, `cell`, `view`, `compare` or `history-*`. Explicit run/artifact/source/cell URLs retain the existing video viewer. `?view=results` opens the first readable artifact/source from the newest published page; `?view=history` restores history. History filters use `history-hardware`, `history-concurrency`, and `history-query` in the URL. Videos & result follows the same published-first path when no run is selected; empty or failed reads show a recoverable message. Browse CI runs explicitly opens legacy Actions discovery. Returning to History cancels pending result selection; Refresh and reload preserve the selected artifact/source/cell. ## Read path @@ -24,7 +24,7 @@ Each observation also carries the cell's participating and allocated GPU counts, ## 中文说明 -`/video` 与 `/zh/video` 默认打开“性能历史”。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 打开最新发布页面中首个可读取的产物与原始运行;未选择运行时,“视频与结果”也使用该入口。空列表或读取失败会显示可恢复的提示;只有点击“浏览 CI 运行”才扫描 GitHub Actions。返回性能历史会取消正在进行的结果选择;刷新和重新加载保留产物、原始运行及并发配置。历史筛选条件保存在 URL,刷新后恢复。 +`/video` 与 `/zh/video` 首屏为跨硬件仪表板(见 [VideoGenX 仪表板](./videogenx-dashboard.md));本文描述的运行、结果与“性能历史”视图位于其下方可展开的“运行、视频与证据”区块,URL 带有 `run`、`artifact`、`source`、`cell`、`view`、`compare` 或 `history-*` 参数时自动展开。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 打开最新发布页面中首个可读取的产物与原始运行;未选择运行时,“视频与结果”也使用该入口。空列表或读取失败会显示可恢复的提示;只有点击“浏览 CI 运行”才扫描 GitHub Actions。返回性能历史会取消正在进行的结果选择;刷新和重新加载保留产物、原始运行及并发配置。历史筛选条件保存在 URL,刷新后恢复。 历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。每个 cell 另附参与/已分配 GPU 数、测量窗口秒数、视频时长与帧数、板卡平均功率之和、记录的生效功率上限之和(无快照时为 null)以及配置的 `tp_size` / `ulysses_degree` / `attention_backend`;这些都是同一 `schemaVersion: 1` 下可为空的附加字段,硬件仪表板据此推导全部指标,浏览器不再解析产物包。 diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md new file mode 100644 index 000000000..1ba76f8d9 --- /dev/null +++ b/docs/videogenx-dashboard.md @@ -0,0 +1,29 @@ +# VideoGenX dashboard (/video) + +Why `/video` leads with a cross-hardware chart instead of a CI-run browser, and the rules its metrics obey. Component code lives in `packages/app/src/components/video-benchmark/` (`VideoDashboard.tsx` is the root). + +## Data path + +Published blob indexes → `GET /api/video-runs?format=history` (the per-cell projection described in [Video performance history](./video-history.md), with additive nullable fields for GPU counts, wall time, clip shape, board power, enforced limit and server layout) → `points.ts` (`videoPoints`, `latestVideoCells`) → `metrics.ts` (`metricValue`). The browser never parses an artifact bundle for the dashboard; media loads only inside the collapsed "Runs, videos & evidence" section, which mounts the previous `VideoCIRuns` views unchanged so `?run=`, `?view=`, `?compare=` and `history-*` deep links keep working and open that section automatically. + +## Axes and defaults + +`video-url-state.ts` owns the `v_*` URL params (`v_x`, `v_y`, `v_tier`, `v_basis`, `v_queue`, `v_opt`, `v_view`); defaults are omitted from the URL and every other param is left alone. + +- **X = P90 time to video (s)**, lower is better. `p50Latency` and `genSpeed` (frames/s per request, the closest analogue to interactivity) are the alternatives. +- **Y = videos per $1 TCO** at the _Owning at Large Hyperscaler Volume_ tier. Cost per GPU-hour comes from `HW_REGISTRY` (`costh`/`costr`), the same SemiAnalysis TCO source `/inference` uses; the badge row and source link sit above the chart. +- **GPU basis**: _participating_ GPUs (the boards that generated the clip — 4 per video with TP2 × Ulysses 2) is chip efficiency; _allocated_ GPUs (H100/B200 reserved 8) is what a deployment bills. One toggle apart, never mixed in one number. +- **Why C1 by default**: the pinned H3 server is batch-one and single-replica, so client concurrency only queues requests — throughput stays flat while latency scales with C. C2/C4 render as dotted "queueing tails" behind the _Show queueing_ switch; they are evidence that batching does not help, not a frontier. +- **Pareto hull** (`frontier.ts`) flips lower-is-better axes and reuses `paretoFrontUpperLeft` from the calculator. With one C1 point per hardware the hull is a cross-hardware comparison; a per-hardware curve needs a GPUs-per-video sweep (design plan P6, GPU work not authorised yet). When one hardware dominates outright (today: B200), no hull path is drawn. + +## Rules + +- **Unavailable hardware** stays visible. `VIDEO_HARDWARE_ROSTER` in `hardware.ts` lists the campaign targets; MI355X has no valid run and renders as a muted legend row, a "Not measured" KPI card with the failed-run link, and no table row. A missing point is never a zero. +- **Nulls**: every formula returns `null` when an input is missing or invalid — power phase invalid, fewer than ten samples for P90, unknown hardware key, missing GPU count. `formatMetric` prints `—`. +- **Newest observation wins**: `latestVideoCells` keeps the first (newest publication) observation per hardware × concurrency, so a re-run replaces, never duplicates, a point. +- **Strings**: component-local `STRINGS = { en, zh }` via `useLocale()`; SKUs, units and identifiers stay English. +- **Fixture**: `cypress/fixtures/api/video-history.json` is rebuilt from retained `StoredArtifact` exports with `bun run fixtures:video-history ` (`scripts/build-video-history-fixture.ts`); it is metadata only and registered in `_manifest.json`. + +## 中文说明 + +`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。默认 X 为 P90 出片时间、Y 为每 1 美元 TCO 视频数(Hyperscaler 自有设备分档,单价与 `/inference` 同源)。默认只画 C1:pinned H3 服务是 batch-one、单副本,客户端并发只会排队,C2/C4 作为可选的点线尾迹展示。Pareto 包络复用 calculator 的前沿算法;参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 From 5c501466a5cd1ea4b2353a42be954155113b3c4a Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:20:35 -0700 Subject: [PATCH 014/104] fix: keep video KPI swatches hydration-safe and show placeholders while loading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:KPI 卡色块在首次渲染使用中性色,水合完成后再取主题色,消除 SSR/CSR 不一致;历史数据加载中显示占位骨架而不是误报“未测得”;E2E 增加 390 px 视口无横向滚动检查并输出截图。 --- .../cypress/component/video-dashboard.cy.tsx | 17 +++++++++++++++++ packages/app/cypress/e2e/video-dashboard.cy.ts | 17 +++++++++++++++++ .../video-benchmark/VideoDashboard.tsx | 18 ++++++++++++++---- .../video-benchmark/VideoKpiCards.tsx | 12 +++++++++++- 4 files changed, 59 insertions(+), 5 deletions(-) diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index 4464ce9cf..a48853455 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -88,6 +88,23 @@ describe('Video hardware dashboard (retained fixture)', () => { cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); cy.get('[data-testid="video-ci-runs"]').should('exist'); }); + it('shows placeholders while loading instead of a false "Not measured"', () => { + cy.intercept('GET', '/api/video-runs?format=history&page=1', (req) => { + req.reply({ fixture: 'api/video-history.json', delay: 800 }); + }).as('slowHistory'); + cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + cy.mount( + + + , + ); + cy.get('[data-testid="video-kpi-skeleton"]').should('have.length', 4); + cy.get('[data-testid="video-kpi-unavailable"]').should('not.exist'); + cy.wait('@slowHistory'); + cy.get('[data-testid="video-kpi-skeleton"]').should('not.exist'); + kpi('h200').should('contain', '$0.204'); + kpi('mi355x').find('[data-testid="video-kpi-unavailable"]').should('exist'); + }); it('renders Chinese copy on /zh/video', () => { mount('/zh/video'); cy.get('[data-testid="video-chart-card"]').should( diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts index 0d55d052a..17860917b 100644 --- a/packages/app/cypress/e2e/video-dashboard.cy.ts +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -26,4 +26,21 @@ describe('Video hardware dashboard (E2E fixtures)', () => { ); cy.get('[data-testid="video-kpi-card"][data-hardware="mi355x"]').should('contain', '未测得'); }); + it('fits a phone viewport without horizontal page scroll in both locales', () => { + cy.viewport(390, 844); + for (const path of ['/video?v_queue=1', '/zh/video']) { + cy.visit(path); + cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length.at.least', 3); + cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); + cy.document().then((doc) => { + expect(doc.documentElement.scrollWidth, `${path} page width`).to.be.at.most( + doc.documentElement.clientWidth, + ); + }); + cy.screenshot(`video-dashboard-390${path.startsWith('/zh') ? '-zh' : '-en'}`, { + capture: 'fullPage', + overwrite: true, + }); + } + }); }); diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx index 2feb7dd5c..7147cbdf4 100644 --- a/packages/app/src/components/video-benchmark/VideoDashboard.tsx +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -1,6 +1,6 @@ 'use client'; -import { useCallback, useEffect, useMemo, useState } from 'react'; +import { useCallback, useEffect, useMemo, useState, useSyncExternalStore } from 'react'; import { HW_REGISTRY, TCO_SOURCE_TITLE, @@ -111,6 +111,8 @@ const STRINGS = { }, }; +const subscribeNoop = () => () => {}; + function csvCell(value: string | number | null): string { if (value === null) return ''; const text = String(value); @@ -143,9 +145,17 @@ export default function VideoDashboard() { highContrast: false, activeKeys: hardwareKeys, }); + // Vendor hues depend on the resolved theme and computed styles, which the + // server cannot know; render the neutral token until hydration completes so + // the first client render matches the server HTML. + const mounted = useSyncExternalStore( + subscribeNoop, + () => true, + () => false, + ); const colorFor = useCallback( - (key: string) => getCssColor(resolveColor(key)), - [resolveColor, getCssColor], + (key: string) => (mounted ? getCssColor(resolveColor(key)) : 'var(--muted-foreground)'), + [mounted, resolveColor, getCssColor], ); const options = { tier: state.tier, basis: state.basis }; @@ -377,7 +387,7 @@ export default function VideoDashboard() { )} - +
string; + /** While the published history loads, show placeholders rather than a false "Not measured". */ + loading?: boolean; }) { const locale = useLocale(); const s = STRINGS[locale]; @@ -73,7 +77,13 @@ export default function VideoKpiCards({ )} - {point ? ( + {!point && loading ? ( +
+ {CARD_METRICS.map((id) => ( + + ))} +
+ ) : point ? (
{CARD_METRICS.map((id) => (
From d189dce8b571ab1079806860974f06d5972d0ad6 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:23:37 -0700 Subject: [PATCH 015/104] fix: align video dashboard Chinese labels with existing tier and metric wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:统一“成本档位”“仅最优”等既有用词;指标中文改为“每 GPU 小时生成视频数”“每秒视频时长 TCO 成本”“每 kWh GPU 板卡电能生成视频数”;重写包络说明与副标题,使其更贴近中文技术表达且不夸大“吞吐量不变”的结论。 --- packages/app/cypress/component/video-dashboard.cy.tsx | 2 +- .../app/src/components/video-benchmark/VideoConfigBar.tsx | 2 +- .../app/src/components/video-benchmark/VideoDashboard.tsx | 4 ++-- .../src/components/video-benchmark/VideoHardwareChart.tsx | 2 +- .../app/src/components/video-benchmark/VideoKpiCards.tsx | 4 ++-- packages/app/src/components/video-benchmark/metrics.ts | 6 +++--- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index a48853455..ab71bc582 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -112,6 +112,6 @@ describe('Video hardware dashboard (retained fixture)', () => { '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', ); kpi('mi355x').should('contain', '未测得'); - cy.get('[data-testid="video-config-bar"]').should('contain', '成本分档'); + cy.get('[data-testid="video-config-bar"]').should('contain', '成本档位'); }); }); diff --git a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx index d7b1d6d98..d3927ce09 100644 --- a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx +++ b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx @@ -43,7 +43,7 @@ const STRINGS = { all: 'C1 + C2/C4 排队尾迹', x: 'X 轴指标', y: 'Y 轴指标', - tier: '成本分档', + tier: '成本档位', basis: 'GPU 口径', participating: '参与计算的 GPU(实际生成视频的板卡)', allocated: '已分配的 GPU(作业预留的板卡)', diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx index 7147cbdf4..ef9d81c7c 100644 --- a/packages/app/src/components/video-benchmark/VideoDashboard.tsx +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -89,11 +89,11 @@ const STRINGS = { zh: { title: 'VideoGenX · MiniMax-H3 跨硬件对比', subtitle: - '同一冻结的工作负载、同一模型与运行时,在每种 GPU 上实测:出片时间、每 GPU 小时与每美元 TCO 的有效产出、GPU 板卡实测能耗。', + '固定的工作负载、模型与运行时,在每种 GPU 上实测:出片时间、每 GPU 小时和每美元 TCO 的有效产出,以及 GPU 板卡的实测能耗。', chart: '图表', table: '表格', viewToggle: '图表或表格视图', - tier: '成本分档', + tier: '成本档位', badges: 'TCO $/chip/hr', source: '来源', queue: '显示排队尾迹(C2/C4)', diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx index b9b91d560..26880b2e6 100644 --- a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -73,7 +73,7 @@ const STRINGS = { noData: 'No hardware has both selected metrics for this configuration.', }, zh: { - hull: '虚线连接 C1 下 Pareto 最优的硬件;点线尾迹为 batch-one 服务器上 C2/C4 客户端并发的表现:请求排队、吞吐量不变,它们是证据而不是前沿。', + hull: '虚线包络连接 C1 下 Pareto 最优的硬件;点线尾迹是 batch-one 服务器上 C2/C4 客户端并发的结果:请求排队、吞吐量基本持平,因此只是证据,不构成前沿。', controls: 'Shift+滚轮缩放,拖动平移,双击重置。点击数据点可固定详情。', n: '有效样本', run: '运行', diff --git a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx index 8912f173f..0b5a985f2 100644 --- a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx +++ b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx @@ -31,8 +31,8 @@ const STRINGS = { zh: { p50Latency: 'P50 出片时间(s)', videosPerGpuHour: '视频数 / GPU 小时', - dollarsPerVideo: 'TCO / 条', - kjPerVideo: 'kJ / 条', + dollarsPerVideo: 'TCO / 条视频', + kjPerVideo: 'kJ / 条视频', powerPctCap: '板卡功率 / 上限', p90: 'P90', gpus: '张 GPU', diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts index 5f09eebe8..68e80eab4 100644 --- a/packages/app/src/components/video-benchmark/metrics.ts +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -115,7 +115,7 @@ export const VIDEO_METRICS: Record = { }, videosPerGpuHour: { label: 'Videos per GPU-hour', - labelZh: '每 GPU 小时视频数', + labelZh: '每 GPU 小时生成视频数', unit: 'videos/GPU-hr', polarity: 'higher', digits: 2, @@ -138,7 +138,7 @@ export const VIDEO_METRICS: Record = { }, dollarsPerVideoSecond: { label: 'TCO cost per video-second', - labelZh: '每视频秒 TCO 成本', + labelZh: '每秒视频时长 TCO 成本', unit: '$/video-s', polarity: 'lower', digits: 4, @@ -154,7 +154,7 @@ export const VIDEO_METRICS: Record = { }, videosPerKwh: { label: 'Videos per GPU-board kWh', - labelZh: '每 GPU 板卡 kWh 视频数', + labelZh: '每 kWh GPU 板卡电能生成视频数', unit: 'videos/kWh', polarity: 'higher', digits: 2, From 5d451914ad369c91db4c4d9a67982362adc01aff Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:51:38 -0700 Subject: [PATCH 016/104] feat: key video cells by deployment and carry the replica count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A deployment is the server layout that produced a cell (participating GPUs, tp_size × ulysses_degree); cells that differ only in client concurrency share it. latestVideoCells now dedupes per hardware × deployment × concurrency so a GPUs-per-video sweep adds points instead of replacing them. Project the endpoint replica count (execution.deployment.replica_count, null before that record existed) and classify concurrency above one request per replica as queueing; leadCell picks each hardware's most efficient deployment. Fixture rebuilt from the retained exports (replicas: null). 中文:视频 cell 改为按“部署”(参与 GPU 数、tp_size × ulysses_degree)作键,仅并发不同的 cell 属于同一部署,GPUs-per-video 扫描会新增点而不是覆盖。投影新增端点副本数 (早于该记录的产物包为 null),并发超过每副本一个请求判定为排队;leadCell 选出每种硬件 最高效的部署。fixture 已按保留导出重建。 --- docs/video-history.md | 4 +- .../app/cypress/fixtures/api/_manifest.json | 8 +- .../cypress/fixtures/api/video-history.json | 9 +++ .../video-benchmark/deployment.test.ts | 81 +++++++++++++++++++ .../components/video-benchmark/deployment.ts | 62 ++++++++++++++ .../video-benchmark/history.test.ts | 1 + .../src/components/video-benchmark/history.ts | 3 + .../video-benchmark/metrics.test.ts | 1 + .../src/components/video-benchmark/metrics.ts | 2 + .../components/video-benchmark/points.test.ts | 1 + .../src/components/video-benchmark/points.ts | 12 ++- .../components/video-benchmark/tradeoff.ts | 2 + 12 files changed, 176 insertions(+), 10 deletions(-) create mode 100644 packages/app/src/components/video-benchmark/deployment.test.ts create mode 100644 packages/app/src/components/video-benchmark/deployment.ts diff --git a/docs/video-history.md b/docs/video-history.md index 9bdb2f128..6063551bf 100644 --- a/docs/video-history.md +++ b/docs/video-history.md @@ -8,7 +8,7 @@ The projection groups all cells under the original source inside one artifact entry. It reuses `storedBundle`, `servingCells`, `storedFidelityBundle`, and the existing tradeoff calculations. Invalid or unavailable indexes remain explicit error entries. Original execution time comes from `ci.started_at`; index publication time is separate and must never be substituted for execution time. Original run IDs and export run IDs remain distinct. -Each observation also carries the cell's participating and allocated GPU counts, measurement wall seconds, clip duration and frame count, summed mean board watts, summed recorded enforced limits (null when no limit snapshot exists) and the configured `tp_size` / `ulysses_degree` / `attention_backend`. These are additive, nullable fields under the same `schemaVersion: 1`; the hardware dashboard derives every displayed metric from them, so the browser never parses an artifact bundle for the chart. +Each observation also carries the cell's participating and allocated GPU counts, measurement wall seconds, clip duration and frame count, summed mean board watts, summed recorded enforced limits (null when no limit snapshot exists), the configured `tp_size` / `ulysses_degree` / `attention_backend` and the endpoint's replica count (`execution.deployment.replica_count`, null for bundles that predate it). These are additive, nullable fields under the same `schemaVersion: 1`; the hardware dashboard derives every displayed metric from them, so the browser never parses an artifact bundle for the chart. ## Evidence boundaries @@ -26,7 +26,7 @@ Each observation also carries the cell's participating and allocated GPU counts, `/video` 与 `/zh/video` 首屏为跨硬件仪表板(见 [VideoGenX 仪表板](./videogenx-dashboard.md));本文描述的运行、结果与“性能历史”视图位于其下方可展开的“运行、视频与证据”区块,URL 带有 `run`、`artifact`、`source`、`cell`、`view`、`compare` 或 `history-*` 参数时自动展开。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 打开最新发布页面中首个可读取的产物与原始运行;未选择运行时,“视频与结果”也使用该入口。空列表或读取失败会显示可恢复的提示;只有点击“浏览 CI 运行”才扫描 GitHub Actions。返回性能历史会取消正在进行的结果选择;刷新和重新加载保留产物、原始运行及并发配置。历史筛选条件保存在 URL,刷新后恢复。 -历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。每个 cell 另附参与/已分配 GPU 数、测量窗口秒数、视频时长与帧数、板卡平均功率之和、记录的生效功率上限之和(无快照时为 null)以及配置的 `tp_size` / `ulysses_degree` / `attention_backend`;这些都是同一 `schemaVersion: 1` 下可为空的附加字段,硬件仪表板据此推导全部指标,浏览器不再解析产物包。 +历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。每个 cell 另附参与/已分配 GPU 数、测量窗口秒数、视频时长与帧数、板卡平均功率之和、记录的生效功率上限之和(无快照时为 null)、配置的 `tp_size` / `ulysses_degree` / `attention_backend`,以及端点的副本数(`execution.deployment.replica_count`,早于该记录的产物包为 null);这些都是同一 `schemaVersion: 1` 下可为空的附加字段,硬件仪表板据此推导全部指标,浏览器不再解析产物包。 当前展示观测值,未选择匹配基线,也不计算 before/after。延迟、吞吐量使用提交到下载完成的窗口;板卡能耗使用记录的生成窗口。保真度、阈值校准、发布验收单独展示。筛选仅作用于已加载的页面,可继续加载较早结果。 diff --git a/packages/app/cypress/fixtures/api/_manifest.json b/packages/app/cypress/fixtures/api/_manifest.json index 183661168..3b01d2b7f 100644 --- a/packages/app/cypress/fixtures/api/_manifest.json +++ b/packages/app/cypress/fixtures/api/_manifest.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, - "generatedAt": "2026-09-20T02:56:12.184Z", + "generatedAt": "2026-09-20T03:49:19.738Z", "fixtures": { "agentic-catalog": { "bytes": 10027, @@ -94,9 +94,9 @@ "topLevel": "object" }, "video-history": { - "bytes": 18083, - "capturedAt": "2026-09-20T02:56:12.184Z", - "sha256": "8a748582af68a5d259ad06f7295a3a5b92989ed0b5c0798eefb69600e22cdb77", + "bytes": 18371, + "capturedAt": "2026-09-20T03:49:19.738Z", + "sha256": "16c1b9c97824fc68160454c14aeb79f2c3404595a5bd97b9bb8f286de42539f9", "source": "Retained public H3 metadata replay from original runs via videoHistoryEntry (per-cell GPU counts, board power and server layout); publication time unavailable. No new benchmark or publication.", "topLevel": "object" } diff --git a/packages/app/cypress/fixtures/api/video-history.json b/packages/app/cypress/fixtures/api/video-history.json index 65a72f04c..5d8443a7b 100644 --- a/packages/app/cypress/fixtures/api/video-history.json +++ b/packages/app/cypress/fixtures/api/video-history.json @@ -43,6 +43,7 @@ "energyKj": 431.14963489170344, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 3349.37367718108, "durationSeconds": 8, "frameCount": 192, @@ -74,6 +75,7 @@ "energyKj": 431.32628015924786, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 3352.7401133067906, "durationSeconds": 8, "frameCount": 192, @@ -136,6 +138,7 @@ "energyKj": 430.80324425396446, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 3352.988327450119, "durationSeconds": 8, "frameCount": 192, @@ -198,6 +201,7 @@ "energyKj": 410.954300515549, "participating": 4, "allocated": 4, + "replicas": null, "wallSeconds": 3013.44202613004, "durationSeconds": 8, "frameCount": 192, @@ -229,6 +233,7 @@ "energyKj": 409.91272507134966, "participating": 4, "allocated": 4, + "replicas": null, "wallSeconds": 3001.4726729940157, "durationSeconds": 8, "frameCount": 192, @@ -260,6 +265,7 @@ "energyKj": 413.58283766782114, "participating": 4, "allocated": 4, + "replicas": null, "wallSeconds": 3006.626674519968, "durationSeconds": 8, "frameCount": 192, @@ -322,6 +328,7 @@ "energyKj": 301.08754501100987, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 1561.6792716470081, "durationSeconds": 8, "frameCount": 192, @@ -353,6 +360,7 @@ "energyKj": 300.10551714629605, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 1539.571618376067, "durationSeconds": 8, "frameCount": 192, @@ -384,6 +392,7 @@ "energyKj": 299.96858693604275, "participating": 4, "allocated": 8, + "replicas": null, "wallSeconds": 1539.5502297059866, "durationSeconds": 8, "frameCount": 192, diff --git a/packages/app/src/components/video-benchmark/deployment.test.ts b/packages/app/src/components/video-benchmark/deployment.test.ts new file mode 100644 index 000000000..37f3586d8 --- /dev/null +++ b/packages/app/src/components/video-benchmark/deployment.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, it } from 'vitest'; +import { deploymentKey, isQueueing, layoutLabel, leadCell } from './deployment'; +import type { VideoPoint } from './metrics'; + +const base: VideoPoint = { + id: 'a', + runId: '1', + artifactId: 1, + cell: 'c1', + hardwareKey: 'h200', + hardwareName: 'NVIDIA H200', + runtime: 'r', + model: 'm', + workload: 'w', + concurrency: 1, + participating: 4, + allocated: 8, + replicas: null, + valid: 20, + completed: 20, + scheduled: 20, + failed: 0, + samples: 20, + p50: 150, + p90: 151, + wallSeconds: 3000, + durationSeconds: 8, + frameCount: 192, + energyKj: 400, + avgPowerW: 2700, + enforcedLimitW: 2800, + server: { tp: 2, ulysses: 2, attention: null }, + status: 'complete', + observedAt: null, +}; +const options = { tier: 'h', basis: 'participating' } as const; + +describe('deployment', () => { + it('keys a deployment by boards and model split, not by concurrency', () => { + const c4: VideoPoint = { ...base, concurrency: 4 }; + expect(deploymentKey(base)).toBe('4g:tp2:u2'); + expect(deploymentKey(c4)).toBe('4g:tp2:u2'); + expect(deploymentKey({ ...base, server: { tp: 1, ulysses: 4, attention: null } })).toBe( + '4g:tp1:u4', + ); + expect(deploymentKey({ ...base, participating: null, server: null })).toBe('nag:tpna:una'); + }); + it('treats concurrency above the replica count as queueing, with one replica when unknown', () => { + expect(isQueueing(base)).toBe(false); + expect(isQueueing({ ...base, concurrency: 2 })).toBe(true); + expect(isQueueing({ ...base, concurrency: 4, replicas: 4 })).toBe(false); + expect(isQueueing({ ...base, concurrency: 8, replicas: 4 })).toBe(true); + expect(isQueueing({ ...base, concurrency: null })).toBe(false); + }); + it('labels the layout in both locales and omits unknown parts', () => { + expect(layoutLabel(base, 'en')).toBe('4 GPU · TP2 × Ulysses 2'); + expect(layoutLabel({ ...base, replicas: 2 }, 'zh')).toBe( + '4 张 GPU · TP2 × Ulysses 2 · 2 个副本', + ); + expect(layoutLabel({ ...base, server: null }, 'en')).toBe('4 GPU'); + expect(layoutLabel({ ...base, participating: null, server: null }, 'en')).toBe('—'); + }); + it('leads with the most efficient non-queued deployment of the hardware', () => { + const eightBoards = { + ...base, + id: 'b', + participating: 8, + server: { tp: 4, ulysses: 2, attention: null }, + p50: 90, + wallSeconds: 1800, // 20 × 3600 / (1800 × 8) = 5.0 videos per GPU-hour + }; + const queued = { ...base, id: 'c', concurrency: 2, wallSeconds: 2900 }; // higher rate, but queued + // base: 20 × 3600 / (3000 × 4) = 6.0 videos per GPU-hour + expect(leadCell([eightBoards, queued, base], 'h200', options)?.id).toBe('a'); + expect(leadCell([eightBoards, queued, base], 'b200', options)).toBeUndefined(); + const tie = { ...eightBoards, id: 'd', participating: 4, wallSeconds: 3000, p50: 120 }; + expect(leadCell([base, tie], 'h200', options)?.id).toBe('d'); + const noRate = { ...base, id: 'e', wallSeconds: null }; + expect(leadCell([noRate], 'h200', options)?.id).toBe('e'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/deployment.ts b/packages/app/src/components/video-benchmark/deployment.ts new file mode 100644 index 000000000..9aa17b6e9 --- /dev/null +++ b/packages/app/src/components/video-benchmark/deployment.ts @@ -0,0 +1,62 @@ +import { metricValue, type MetricOptions, type VideoPoint } from './metrics'; + +type Layout = Pick; + +/** + * A deployment is the server layout that produced a cell: how many GPU boards + * served one request and how they split the model (tensor parallel × Ulysses + * sequence parallel). Cells that differ only in client concurrency belong to + * the same deployment, so they share this key. + */ +export function deploymentKey(p: Pick): string { + return `${p.participating ?? 'na'}g:tp${p.server?.tp ?? 'na'}:u${p.server?.ulysses ?? 'na'}`; +} + +/** + * On a batch-one server every request beyond one per replica waits in a queue: + * latency grows with concurrency while throughput stays flat. Such a cell is + * evidence about the same deployment, never a new point on its frontier. + * Bundles that predate the deployment record ran one supervised endpoint, so + * an unknown replica count reads as one. + */ +export function isQueueing(p: Pick): boolean { + return (p.concurrency ?? 1) > (p.replicas ?? 1); +} + +/** "4 GPU · TP2 × Ulysses 2", plus the replica count when more than one. */ +export function layoutLabel(p: Layout, locale: 'en' | 'zh'): string { + const parts: string[] = []; + if (p.participating !== null) + parts.push(locale === 'zh' ? `${p.participating} 张 GPU` : `${p.participating} GPU`); + const tp = p.server?.tp ?? null; + const ulysses = p.server?.ulysses ?? null; + if (tp !== null && ulysses !== null) parts.push(`TP${tp} × Ulysses ${ulysses}`); + if (p.replicas !== null && p.replicas > 1) + parts.push(locale === 'zh' ? `${p.replicas} 个副本` : `${p.replicas} replicas`); + return parts.length > 0 ? parts.join(' · ') : '—'; +} + +/** + * The cell that stands for a hardware in cards and comparisons: its most + * efficient deployment under the current GPU basis (highest videos per + * GPU-hour), lowest P50 on ties. Queued cells never lead; a hardware whose + * cells all lack the rate falls back to its first deployment cell. + */ +export function leadCell( + cells: VideoPoint[], + hardwareKey: string, + options: MetricOptions, +): VideoPoint | undefined { + let best: VideoPoint | undefined; + let bestRate = Number.NEGATIVE_INFINITY; + for (const p of cells) { + if (p.hardwareKey !== hardwareKey || isQueueing(p)) continue; + const rate = metricValue(p, 'videosPerGpuHour', options) ?? Number.NEGATIVE_INFINITY; + const faster = rate === bestRate && (p.p50 ?? Infinity) < (best?.p50 ?? Infinity); + if (best === undefined || rate > bestRate || faster) { + best = p; + bestRate = rate; + } + } + return best; +} diff --git a/packages/app/src/components/video-benchmark/history.test.ts b/packages/app/src/components/video-benchmark/history.test.ts index 72673fb03..337089f9a 100644 --- a/packages/app/src/components/video-benchmark/history.test.ts +++ b/packages/app/src/components/video-benchmark/history.test.ts @@ -56,6 +56,7 @@ describe('video history projection', () => { expect(c1).toMatchObject({ participating: 2, allocated: 2, + replicas: null, wallSeconds: 480, durationSeconds: 4, frameCount: 107, diff --git a/packages/app/src/components/video-benchmark/history.ts b/packages/app/src/components/video-benchmark/history.ts index aa9f9d541..337b60158 100644 --- a/packages/app/src/components/video-benchmark/history.ts +++ b/packages/app/src/components/video-benchmark/history.ts @@ -26,6 +26,8 @@ export interface VideoHistoryObservation { participating: number | null; /** GPU boards the job reserved (Slurm AllocTRES or the retained AMD binding); null when unknown. */ allocated: number | null; + /** Replicas behind the endpoint (`execution.deployment.replica_count`); null before that record existed. */ + replicas: number | null; wallSeconds: number | null; durationSeconds: number | null; frameCount: number | null; @@ -134,6 +136,7 @@ export function videoHistoryEntry( energyKj: point.energy === null ? null : point.energy / 1000, participating: point.participating, allocated: point.allocated, + replicas: point.replicas, wallSeconds: point.wall, durationSeconds: number(at(point.workload, 'generation', 'duration_seconds')), frameCount: number(at(point.workload, 'generation', 'frame_count')), diff --git a/packages/app/src/components/video-benchmark/metrics.test.ts b/packages/app/src/components/video-benchmark/metrics.test.ts index e26bc1752..19f963dfd 100644 --- a/packages/app/src/components/video-benchmark/metrics.test.ts +++ b/packages/app/src/components/video-benchmark/metrics.test.ts @@ -15,6 +15,7 @@ const h200: VideoPoint = { concurrency: 1, participating: 4, allocated: 4, + replicas: null, valid: 20, completed: 20, scheduled: 20, diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts index 68e80eab4..da55c8675 100644 --- a/packages/app/src/components/video-benchmark/metrics.ts +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -18,6 +18,8 @@ export interface VideoPoint { concurrency: number | null; participating: number | null; allocated: number | null; + /** Model replicas behind the endpoint; null for bundles that predate the deployment record. */ + replicas: number | null; valid: number | null; completed: number | null; scheduled: number | null; diff --git a/packages/app/src/components/video-benchmark/points.test.ts b/packages/app/src/components/video-benchmark/points.test.ts index 64a25c31b..1d54e8947 100644 --- a/packages/app/src/components/video-benchmark/points.test.ts +++ b/packages/app/src/components/video-benchmark/points.test.ts @@ -22,6 +22,7 @@ const observation: VideoHistoryObservation = { energyKj: 410.95, participating: 4, allocated: 4, + replicas: null, wallSeconds: 3013.4, durationSeconds: 8, frameCount: 192, diff --git a/packages/app/src/components/video-benchmark/points.ts b/packages/app/src/components/video-benchmark/points.ts index bac2848fa..ec3d73208 100644 --- a/packages/app/src/components/video-benchmark/points.ts +++ b/packages/app/src/components/video-benchmark/points.ts @@ -1,3 +1,4 @@ +import { deploymentKey } from './deployment'; import { hardwareKey, hardwareSort } from './hardware'; import type { VideoHistoryObservation, VideoHistoryPage } from './history'; import type { VideoPoint } from './metrics'; @@ -51,6 +52,7 @@ function toPoint( concurrency: num(o.concurrency), participating: num(o.participating), allocated: num(o.allocated), + replicas: num(o.replicas), valid: num(o.valid), completed: num(o.completed), scheduled: num(o.scheduled), @@ -78,15 +80,17 @@ function toPoint( } /** - * Newest published observation per (hardware, concurrency) cell. Points arrive - * newest publication first, so the first occurrence wins; hardware without a - * registry key is kept as-is because nothing can be compared against it. + * Newest published observation per (hardware, deployment, concurrency) cell. + * Points arrive newest publication first, so the first occurrence wins; hardware + * without a registry key is kept as-is because nothing can be compared against + * it. Two server layouts of one hardware are different cells, so a GPUs-per-video + * sweep adds points instead of replacing the existing one. */ export function latestVideoCells(points: VideoPoint[]): VideoPoint[] { const seen = new Set(); return points.filter((point) => { if (point.hardwareKey === null) return true; - const key = `${point.hardwareKey}:${point.concurrency ?? 'na'}`; + const key = `${point.hardwareKey}:${deploymentKey(point)}:${point.concurrency ?? 'na'}`; if (seen.has(key)) return false; seen.add(key); return true; diff --git a/packages/app/src/components/video-benchmark/tradeoff.ts b/packages/app/src/components/video-benchmark/tradeoff.ts index aef07f516..ffdaec7d3 100644 --- a/packages/app/src/components/video-benchmark/tradeoff.ts +++ b/packages/app/src/components/video-benchmark/tradeoff.ts @@ -154,6 +154,7 @@ function pairedTradeoffPoints(run: TradeoffRun) { boundary: text(at(metrics, 'measurement', 'boundary')), allocated: positive(at(result, 'hardware', 'reserved_gpu_count')), participating: positive(at(result, 'hardware', 'selected_gpu_count')), + replicas: positive(at(result, 'execution', 'deployment', 'replica_count')), energy, powerLimit, timing: serverTimingSummary(null, false), @@ -233,6 +234,7 @@ function servingTradeoffPoints(run: TradeoffRun) { boundary: text(boundary), allocated, participating: uuids.length > 0 && new Set(uuids).size === uuids.length ? uuids.length : null, + replicas: positive(at(b.result, 'execution', 'deployment', 'replica_count')), energy: powerValid ? positive(at(phase, 'aggregate', 'joules_per_valid_clip')) : null, power: powerValid ? number(at(phase, 'aggregate', 'avg_power_w')) : null, powerWindow: powerValid ? number(at(phase, 'duration_seconds')) : null, From 63a3d7733f767554ec07ccc756a060f3e1a9a9ef Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:51:50 -0700 Subject: [PATCH 017/104] feat: draw per-hardware Pareto frontiers over deployments, never through queued cells MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Client concurrency on the batch-one H3 server only queues requests (retained C1/C2/C4 throughput is flat within ±1.5 %), so the dotted C1→C2→C4 tails were not a frontier and are gone. plotVideoPoints builds each hardware's Pareto frontier over its deployment cells and draws a solid line once two or more frontier deployments exist; dominated deployments render faded and "Optimal only" hides them. Queued cells are opt-in faded markers that never join a line. A "Pareto frontier" switch (v_frontier) adds the dashed cross-hardware frontier as on /inference; when one hardware dominates the caption says so. With one deployment per hardware today the chart is a scatter whose caption names the measured layout and the sweep a curve needs. The config bar states the measured deployments, the table and CSV carry deployment and queued columns, and KPI cards show each hardware's lead deployment. 中文:batch-one H3 服务上客户端并发只会排队(保留数据 C1/C2/C4 吞吐量相差不到 1.5%), 原来的 C1→C2→C4 点线尾迹不是前沿,已移除。plotVideoPoints 按硬件对部署点求 Pareto 前沿, 两个及以上前沿点才画实线;被支配部署淡色显示,“仅最优”可隐藏;排队点需手动开启, 只画淡色小标记、不连线。新增“Pareto 前沿”开关(v_frontier)叠加跨硬件虚线,某硬件全面 占优时由说明文字指出。目前每种硬件一种部署,图表为散点,说明文字给出实测布局及曲线所需的 扫描。配置栏展示实测部署,表格与 CSV 增加部署/排队列,KPI 卡展示各硬件的代表部署。 --- docs/videogenx-dashboard.md | 18 ++- .../cypress/component/video-dashboard.cy.tsx | 31 +++- .../app/cypress/e2e/video-dashboard.cy.ts | 7 +- .../video-benchmark/VideoConfigBar.tsx | 28 ++-- .../video-benchmark/VideoDashboard.tsx | 45 +++++- .../video-benchmark/VideoHardwareChart.tsx | 139 +++++++++--------- .../video-benchmark/VideoKpiCards.tsx | 9 +- .../video-benchmark/VideoPointsTable.tsx | 17 ++- .../components/video-benchmark/plot.test.ts | 135 +++++++++++++++++ .../src/components/video-benchmark/plot.ts | 84 +++++++++++ .../video-benchmark/video-url-state.test.ts | 3 +- .../video-benchmark/video-url-state.ts | 9 +- 12 files changed, 418 insertions(+), 107 deletions(-) create mode 100644 packages/app/src/components/video-benchmark/plot.test.ts create mode 100644 packages/app/src/components/video-benchmark/plot.ts diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md index 1ba76f8d9..0bce20b01 100644 --- a/docs/videogenx-dashboard.md +++ b/docs/videogenx-dashboard.md @@ -8,13 +8,21 @@ Published blob indexes → `GET /api/video-runs?format=history` (the per-cell pr ## Axes and defaults -`video-url-state.ts` owns the `v_*` URL params (`v_x`, `v_y`, `v_tier`, `v_basis`, `v_queue`, `v_opt`, `v_view`); defaults are omitted from the URL and every other param is left alone. +`video-url-state.ts` owns the `v_*` URL params (`v_x`, `v_y`, `v_tier`, `v_basis`, `v_queue`, `v_opt`, `v_frontier`, `v_view`); defaults are omitted from the URL and every other param is left alone. - **X = P90 time to video (s)**, lower is better. `p50Latency` and `genSpeed` (frames/s per request, the closest analogue to interactivity) are the alternatives. - **Y = videos per $1 TCO** at the _Owning at Large Hyperscaler Volume_ tier. Cost per GPU-hour comes from `HW_REGISTRY` (`costh`/`costr`), the same SemiAnalysis TCO source `/inference` uses; the badge row and source link sit above the chart. - **GPU basis**: _participating_ GPUs (the boards that generated the clip — 4 per video with TP2 × Ulysses 2) is chip efficiency; _allocated_ GPUs (H100/B200 reserved 8) is what a deployment bills. One toggle apart, never mixed in one number. -- **Why C1 by default**: the pinned H3 server is batch-one and single-replica, so client concurrency only queues requests — throughput stays flat while latency scales with C. C2/C4 render as dotted "queueing tails" behind the _Show queueing_ switch; they are evidence that batching does not help, not a frontier. -- **Pareto hull** (`frontier.ts`) flips lower-is-better axes and reuses `paretoFrontUpperLeft` from the calculator. With one C1 point per hardware the hull is a cross-hardware comparison; a per-hardware curve needs a GPUs-per-video sweep (design plan P6, GPU work not authorised yet). When one hardware dominates outright (today: B200), no hull path is drawn. + +## Deployments, not concurrency, make the curve + +`/inference` sweeps batch size to trace each hardware's latency-vs-throughput frontier. A diffusion video server has no batching lever: the pinned H3 SGLang server is batch-one and single-replica, so client concurrency only queues requests — the retained C1/C2/C4 cells show throughput flat within ±1.5 % while latency scales with C. Joining those cells would draw a flat "curve" that measures the queue, not the hardware, so the chart never does. + +The knob that does trade latency for efficiency is the **deployment**: how many GPU boards serve one request and how they split the model (`tp_size` × `ulysses_degree`). `deployment.ts` keys cells by it (`deploymentKey`), classifies `concurrency > replicas` as queueing (`isQueueing`; bundles that predate `execution.deployment.replica_count` ran one endpoint, so null reads as one), and picks each hardware's most efficient deployment as its representative (`leadCell`, used by the KPI cards and the legend). + +`plot.ts` (`plotVideoPoints`) then builds the chart: per hardware, the Pareto frontier over its deployment cells (`frontier.ts` flips lower-is-better axes and reuses `paretoFrontUpperLeft` from the calculator); a solid line appears once a hardware has two or more frontier deployments, dominated deployments render faded, and _Optimal only_ hides them. Queued cells are opt-in (_Show queued requests_), drawn as small faded markers and never joined. _Pareto frontier_ adds the dashed cross-hardware frontier over all deployments, as on `/inference`; when one hardware dominates outright (today: B200) the caption says so instead of drawing a line. + +Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 × Ulysses 2), so the chart is an honest scatter and the caption says a per-hardware curve needs the GPUs-per-video sweep. That sweep (design plan P6) is backend work in InferenceX `experimental/video-generation` — the serving matrix currently varies only concurrency — and needs GPU authorisation; the frontend needs no change when its cells arrive, because they differ in `deploymentKey` and therefore add points rather than replace them. ## Rules @@ -26,4 +34,6 @@ Published blob indexes → `GET /api/video-runs?format=history` (the per-cell pr ## 中文说明 -`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。默认 X 为 P90 出片时间、Y 为每 1 美元 TCO 视频数(Hyperscaler 自有设备分档,单价与 `/inference` 同源)。默认只画 C1:pinned H3 服务是 batch-one、单副本,客户端并发只会排队,C2/C4 作为可选的点线尾迹展示。Pareto 包络复用 calculator 的前沿算法;参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 +`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。默认 X 为 P90 出片时间、Y 为每 1 美元 TCO 视频数(Hyperscaler 自有设备档位,单价与 `/inference` 同源)。 + +曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表;`plot.ts` 按硬件对部署点求 Pareto 前沿,达到两个及以上前沿点时才画实线,被支配的部署以淡色显示,“仅最优”可将其隐藏;排队点需手动开启,只画小号淡色标记、不连线;“Pareto 前沿”开关叠加跨硬件虚线。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index ab71bc582..eb64482fd 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -20,18 +20,25 @@ function mount(pathname = '/video', search = '') { const kpi = (hardware: string) => cy.get(`[data-testid="video-kpi-card"][data-hardware="${hardware}"]`); const points = () => cy.get('[data-testid="video-hardware-chart"] circle.point'); -const tails = () => cy.get('[data-testid="video-hardware-chart"] path.line-path'); +const lines = () => cy.get('[data-testid="video-hardware-chart"] path.line-path'); +const hull = () => cy.get('[data-testid="video-hardware-chart"] path.roofline-path'); describe('Video hardware dashboard (retained fixture)', () => { it('plots one C1 point per measured hardware and lists MI355X as unavailable', () => { mount(); points().should('have.length', 3); - tails().should('have.length', 0); + lines().should('have.length', 0); + hull().should('have.length', 0); + cy.get('[data-testid="video-chart-caption"]').should( + 'contain', + 'One deployment measured per hardware so far (4 GPU · TP2 × Ulysses 2)', + ); + cy.get('[data-testid="video-config-bar"]').should('contain', '4 GPU · TP2 × Ulysses 2'); cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); kpi('mi355x').find('[data-testid="video-kpi-unavailable"]').should('contain', 'Not measured'); kpi('h200').should('contain', '$0.204').and('contain', '150.6').and('contain', '97.4'); kpi('b200').should('contain', '$0.150'); - kpi('h100').should('contain', '$0.218').and('contain', '4 of 8 GPUs'); + kpi('h100').should('contain', '$0.218').and('contain', '4 of 8 GPUs · TP2 × Ulysses 2'); cy.get('[data-testid="video-config-bar"]').should( 'contain', '1344 × 768 · 8 s · 24 fps · 50 steps', @@ -57,17 +64,29 @@ describe('Video hardware dashboard (retained fixture)', () => { kpi('h100').should('contain', '$0.744'); cy.location('search').should('contain', 'v_basis=allocated'); }); - it('adds dotted C2/C4 tails behind the queueing switch and isolates the Pareto-optimal hardware', () => { + it('plots queued cells as unjoined markers and never draws a curve through them', () => { mount(); cy.get('[data-testid="video-queue"]').click(); points().should('have.length', 9); - tails().should('have.length', 3); + lines().should('have.length', 0); + cy.get('[data-testid="video-chart-caption"]').should('contain', 'never joined into a curve'); cy.location('search').should('contain', 'v_queue=1'); + cy.contains('button', 'Table').click(); + cy.get('[data-testid="video-points-table"] tbody tr').should('have.length', 9); + cy.get('[data-testid="video-points-table"]').should('contain', '2 · queued'); + cy.contains('button', 'Chart').click(); + // Optimal-only keeps every hardware's single deployment and hides the queued cells. cy.get('[data-testid="video-optimal"]').click(); points().should('have.length', 3); - tails().should('have.length', 1); cy.location('search').should('contain', 'v_opt=1'); }); + it('explains the cross-hardware frontier switch when one hardware dominates', () => { + mount(); + cy.get('[data-testid="video-frontier"]').click(); + cy.location('search').should('contain', 'v_frontier=1'); + hull().should('have.length', 0); + cy.get('[data-testid="video-chart-caption"]').should('contain', 'one hardware dominates'); + }); it('hides a hardware from the legend and shows the table view', () => { mount(); cy.contains('[data-testid="video-legend"] li', 'H100').click(); diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts index 17860917b..c0c4e0a1b 100644 --- a/packages/app/cypress/e2e/video-dashboard.cy.ts +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -4,7 +4,12 @@ describe('Video hardware dashboard (E2E fixtures)', () => { it('leads with the cross-hardware chart and restores v_ params from the URL', () => { cy.visit('/video?v_y=kjPerVideo&v_tier=r&v_queue=1'); cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 9); - cy.get('[data-testid="video-hardware-chart"] path.line-path').should('have.length', 3); + // Queued C2/C4 cells are markers only: no curve is drawn through them. + cy.get('[data-testid="video-hardware-chart"] path.line-path').should('have.length', 0); + cy.get('[data-testid="video-chart-caption"]').should( + 'contain', + 'One deployment measured per hardware', + ); cy.get('[data-testid="video-chart-card"]') .should('contain', 'GPU-board energy per video (kJ)') .and('contain', 'Rent - 3 Year Commit'); diff --git a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx index d3927ce09..7f04a34aa 100644 --- a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx +++ b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx @@ -23,9 +23,7 @@ const STRINGS = { chart: 'Chart config', model: 'Model', workload: 'Workload', - load: 'Load', - c1: 'C1 · one request per deployment', - all: 'C1 + C2/C4 queueing tails', + deployment: 'Deployment', x: 'X-axis metric', y: 'Y-axis metric', tier: 'Cost tier', @@ -38,9 +36,7 @@ const STRINGS = { chart: '图表配置', model: '模型', workload: '工作负载', - load: '负载', - c1: 'C1 · 每个部署一个请求', - all: 'C1 + C2/C4 排队尾迹', + deployment: '部署', x: 'X 轴指标', y: 'Y 轴指标', tier: '成本档位', @@ -52,19 +48,24 @@ const STRINGS = { /** * Two control panels mirroring the inference tab: what is being compared - * (frozen for this campaign, so model and workload are single-option selects - * that state the comparison rather than change it) and how it is plotted. + * (frozen for this campaign, so model, workload and the measured deployments + * are single-option selects that state the comparison rather than change it) + * and how it is plotted. Queued cells, optimal-only and the cross-hardware + * frontier are legend switches, as on the inference chart. */ export default function VideoConfigBar({ state, onChange, modelLabel, workloadLabel, + deploymentLabel, }: { state: VideoDashboardState; onChange: (patch: Partial) => void; modelLabel: string; workloadLabel: string; + /** Measured server layouts (GPUs per video and model split), stated rather than selectable. */ + deploymentLabel: string; }) { const locale = useLocale(); const s = STRINGS[locale]; @@ -89,13 +90,10 @@ export default function VideoConfigBar({ options={[{ value: 'workload', label: workloadLabel }]} /> change('queue', value === 'all')} - options={[ - { value: 'c1', label: s.c1 }, - { value: 'all', label: s.all }, - ]} + label={s.deployment} + value="deployment" + onValueChange={() => {}} + options={[{ value: 'deployment', label: deploymentLabel }]} /> diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx index ef9d81c7c..e3bacacb1 100644 --- a/packages/app/src/components/video-benchmark/VideoDashboard.tsx +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -13,6 +13,7 @@ import { Heading } from '@/components/ui/heading'; import { useThemeColors } from '@/hooks/useThemeColors'; import { track } from '@/lib/analytics'; import { useLocale } from '@/lib/use-locale'; +import { isQueueing, layoutLabel, leadCell } from './deployment'; import { costPerGpuHour, hardwareLabel, VIDEO_HARDWARE_ROSTER } from './hardware'; import { metricLabel, @@ -73,8 +74,10 @@ const STRINGS = { tier: 'Cost tier', badges: 'TCO $/chip/hr', source: 'Source', - queue: 'Show queueing (C2/C4)', + queue: 'Show queued requests (C2/C4)', optimal: 'Optimal only', + frontier: 'Pareto frontier', + deployments: (n: number) => `${n} deployments`, runs: 'Runs, videos & evidence', runsHint: 'Per-run results, generated clips, fidelity checks and the performance history list.', loading: 'Loading published results…', @@ -96,8 +99,10 @@ const STRINGS = { tier: '成本档位', badges: 'TCO $/chip/hr', source: '来源', - queue: '显示排队尾迹(C2/C4)', + queue: '显示排队请求(C2/C4)', optimal: '仅最优', + frontier: 'Pareto 前沿', + deployments: (n: number) => `${n} 种部署`, runs: '运行、视频与证据', runsHint: '按运行查看结果、生成的视频、保真度检查以及性能历史列表。', loading: '正在加载已发布结果…', @@ -160,11 +165,23 @@ export default function VideoDashboard() { const options = { tier: state.tier, basis: state.basis }; const cells = useMemo(() => latestVideoCells(points), [points]); + // Each hardware is represented by its most efficient measured deployment. const measured = useMemo( - () => new Map(cells.filter((p) => p.concurrency === 1).map((p) => [p.hardwareKey, p])), - [cells], + () => + new Map( + hardwareKeys.flatMap((key) => { + const point = leadCell(cells, key, { tier: state.tier, basis: state.basis }); + return point ? [[key, point] as const] : []; + }), + ), + [cells, hardwareKeys, state.tier, state.basis], ); - const lead = cells.find((p) => p.concurrency === 1) ?? cells[0]; + const lead = measured.values().next().value ?? cells[0]; + const layouts = [ + ...new Set(cells.filter((p) => !isQueueing(p)).map((p) => layoutLabel(p, locale))), + ]; + const deploymentLabel = + layouts.length > 2 ? s.deployments(layouts.length) : layouts.join(' | ') || '—'; // "1344 × 768 · 8 s · 24 fps · 50 steps · model @ rev · seeds · prompt" → shape, then model @ rev. const workloadParts = lead?.workload.split(' · ') ?? []; const workloadLabel = workloadParts.slice(0, 4).join(' · ') || '—'; @@ -207,6 +224,10 @@ export default function VideoDashboard() { 'scheduled', 'participating_gpus', 'allocated_gpus', + 'tp_size', + 'ulysses_degree', + 'replicas', + 'queued', ...CSV_METRICS.map((id) => `${id} (${VIDEO_METRICS[id].unit})`), 'board_power_w', 'enforced_limit_w', @@ -221,6 +242,10 @@ export default function VideoDashboard() { p.scheduled, p.participating, p.allocated, + p.server?.tp ?? null, + p.server?.ulysses ?? null, + p.replicas, + isQueueing(p) ? 1 : 0, ...CSV_METRICS.map((id) => metricValue(p, id, options)), p.avgPowerW, p.enforcedLimitW, @@ -254,6 +279,7 @@ export default function VideoDashboard() { onChange={update} modelLabel={modelLabel} workloadLabel={`${workloadLabel}${workloadSuffix}`} + deploymentLabel={deploymentLabel} /> { + update({ frontier: checked }); + track('video_frontier_changed', { value: String(checked) }); + }, + }, ]} />
diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx index 26880b2e6..5b59e721c 100644 --- a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -6,81 +6,54 @@ import type { ContinuousScale } from '@/lib/d3-chart/types'; import { CHART_TYPE, px } from '@/lib/d3-chart/typography'; import { useLocale } from '@/lib/use-locale'; import { escapeHtml } from '@/lib/utils'; -import { paretoFrontier } from './frontier'; -import { hardwareLabel } from './hardware'; -import { formatMetric, metricLabel, metricValue, VIDEO_METRICS, type VideoPoint } from './metrics'; -import { latestVideoCells } from './points'; +import { layoutLabel } from './deployment'; +import { formatMetric, metricLabel, type VideoPoint } from './metrics'; +import { plotVideoPoints, type PlottedVideoPoint } from './plot'; import type { VideoDashboardState } from './video-url-state'; export const VIDEO_CHART_ID = 'video-hardware'; /** Matches ChartSection's default `${analyticsPrefix}_zoom_reset_${chartId}`. */ export const VIDEO_ZOOM_RESET_EVENT = `video_zoom_reset_${VIDEO_CHART_ID}`; -export interface PlottedVideoPoint extends VideoPoint { - x: number; - y: number; - color: string; - label: string; -} - -/** - * Points with both axes available (newest observation per hardware cell), the - * cross-hardware Pareto hull over the C1 cells, and per-hardware queueing tails - * (C1→C2→C4) when the load control shows them. - */ -export function plotVideoPoints( - points: VideoPoint[], - state: VideoDashboardState, - colorFor: (hardwareKey: string) => string, - hidden: ReadonlySet, -) { - const options = { tier: state.tier, basis: state.basis }; - const plotted: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { - if (!p.hardwareKey || hidden.has(p.hardwareKey)) return []; - if (!state.queue && p.concurrency !== 1) return []; - const x = metricValue(p, state.x, options); - const y = metricValue(p, state.y, options); - if (x === null || y === null) return []; - return [{ ...p, x, y, color: colorFor(p.hardwareKey), label: hardwareLabel(p.hardwareKey) }]; - }); - const frontier = paretoFrontier( - plotted.filter((p) => p.concurrency === 1), - VIDEO_METRICS[state.x].polarity, - VIDEO_METRICS[state.y].polarity, - ); - const optimal = new Set(frontier.map((p) => p.hardwareKey)); - const visible = state.optimal ? plotted.filter((p) => optimal.has(p.hardwareKey)) : plotted; - const tails: Record = {}; - if (state.queue) - for (const p of visible) { - if (!p.hardwareKey) continue; - (tails[p.hardwareKey] ??= []).push({ x: p.x, y: p.y }); - } - for (const key of Object.keys(tails)) { - tails[key].sort((a, b) => a.x - b.x); - if (tails[key].length < 2) delete tails[key]; - } - return { plotted: visible, frontier, tails }; -} - const STRINGS = { en: { - hull: 'Dashed hull joins the Pareto-optimal hardware at C1. Dotted tails show C2/C4 client concurrency on the batch-one server: requests queue, throughput stays flat, so they are evidence, not a frontier.', + single: (layouts: string) => + `One deployment measured per hardware so far (${layouts}), so each hardware is a single point. A per-hardware Pareto curve needs the GPUs-per-video sweep.`, + multi: + "Solid lines join each hardware's Pareto-optimal deployments (GPUs per video and model split); faded points are dominated deployments.", + global: 'Dashed line: cross-hardware Pareto frontier over all deployments.', + globalOne: + 'Cross-hardware Pareto frontier: one hardware dominates, so there is no line to draw.', + queued: + 'Small faded markers: C2/C4 client concurrency on the same batch-one deployment. Requests queue and throughput stays flat, so they are evidence, never joined into a curve.', controls: 'Shift+scroll to zoom; drag to pan; double-click to reset. Click a point to pin its details.', + deployment: 'Deployment', + queuedTag: 'queued', n: 'valid samples', run: 'run', noData: 'No hardware has both selected metrics for this configuration.', }, zh: { - hull: '虚线包络连接 C1 下 Pareto 最优的硬件;点线尾迹是 batch-one 服务器上 C2/C4 客户端并发的结果:请求排队、吞吐量基本持平,因此只是证据,不构成前沿。', + single: (layouts: string) => + `目前每种硬件只测得一种部署(${layouts}),因此每种硬件只有一个点;要得到每硬件的 Pareto 曲线,还需扫描每条视频占用的 GPU 数。`, + multi: + '实线连接同一硬件的 Pareto 最优部署(每条视频占用的 GPU 数及模型切分方式);淡色点为被支配的部署。', + global: '虚线为所有部署的跨硬件 Pareto 前沿。', + globalOne: '跨硬件 Pareto 前沿:有一种硬件全面占优,因此没有可画的连线。', + queued: + '小号淡色标记为同一 batch-one 部署上的 C2/C4 客户端并发:请求排队、吞吐量持平,仅作为证据展示,不连成曲线。', controls: 'Shift+滚轮缩放,拖动平移,双击重置。点击数据点可固定详情。', + deployment: '部署', + queuedTag: '排队', n: '有效样本', run: '运行', noData: '当前配置下没有硬件同时具备所选的两个指标。', }, }; +const faded = (color: string) => `color-mix(in oklab, ${color} 45%, transparent)`; + export default function VideoHardwareChart({ points, state, @@ -97,9 +70,28 @@ export default function VideoHardwareChart({ const locale = useLocale(); const s = STRINGS[locale]; const options = { tier: state.tier, basis: state.basis }; - const { plotted, frontier, tails } = plotVideoPoints(points, state, colorFor, hidden); + const { plotted, frontiers, global, multiLayout } = plotVideoPoints( + points, + state, + colorFor, + hidden, + ); const xLabel = metricLabel(state.x, locale, options); const yLabel = metricLabel(state.y, locale, options); + const lines: Record = {}; + const lineColors: Record = {}; + for (const [key, frontier] of Object.entries(frontiers)) { + if (frontier.length < 2) continue; + lines[key] = frontier.map((p) => ({ x: p.x, y: p.y })); + lineColors[key] = frontier[0].color; + } + const layouts = [ + ...new Set(plotted.filter((p) => !p.queued).map((p) => layoutLabel(p, locale))), + ].join(locale === 'zh' ? ';' : '; '); + const pointLabel = (p: PlottedVideoPoint) => { + if (p.queued) return `C${p.concurrency}`; + return multiLayout ? `${p.label} · ${layoutLabel(p, locale)}` : p.label; + }; const drawLabels = ( group: Selection, x: ContinuousScale, @@ -113,13 +105,18 @@ export default function VideoHardwareChart({ .attr('x', (p) => x(p.x) + 10) .attr('y', (p) => y(p.y) - 10) .attr('font-size', px(CHART_TYPE.axisLabel)) - .attr('font-weight', (p) => (p.concurrency === 1 ? 600 : 400)) - .attr('fill', 'var(--foreground)') + .attr('font-weight', (p) => (p.optimal ? 600 : 400)) + .attr('fill', (p) => (p.optimal ? 'var(--foreground)' : 'var(--muted-foreground)')) .attr('pointer-events', 'none') - .text((p) => (p.concurrency === 1 ? p.label : `C${p.concurrency}`)); + .text(pointLabel); }; const maxX = Math.max(0, ...plotted.map((p) => p.x)); const maxY = Math.max(0, ...plotted.map((p) => p.y)); + const caption = [ + multiLayout ? s.multi : s.single(layouts || '—'), + state.frontier ? (global.length > 1 ? s.global : s.globalOne) : null, + state.queue && !state.optimal ? s.queued : null, + ].filter((text): text is string => text !== null); return ( 1 ? { hull: frontier.map((p) => ({ x: p.x, y: p.y })) } : {}, + key: 'pareto-global', + rooflines: + state.frontier && global.length > 1 + ? { global: global.map((p) => ({ x: p.x, y: p.y })) } + : {}, config: { getColor: () => 'var(--foreground)', strokeWidth: 1.5, @@ -147,12 +147,11 @@ export default function VideoHardwareChart({ }, { type: 'line', - key: 'queueing-tails', - lines: tails, + key: 'hardware-frontiers', + lines, config: { - getColor: (key) => colorFor(key), - getStrokeDasharray: () => '2 4', - strokeWidth: 1.5, + getColor: (key) => lineColors[key] ?? 'var(--foreground)', + strokeWidth: 2, curve: curveLinear, }, }, @@ -164,8 +163,8 @@ export default function VideoHardwareChart({ getCy: () => 0, getX: (p) => p.x, getY: (p) => p.y, - getColor: (p) => p.color, - getRadius: (p) => (p.concurrency === 1 ? 7 : 4), + getColor: (p) => (p.queued || !p.optimal ? faded(p.color) : p.color), + getRadius: (p) => (p.queued ? 3.5 : p.optimal ? 7 : 5), stroke: 'var(--foreground)', strokeWidth: 1, keyFn: (p) => p.id, @@ -188,7 +187,7 @@ export default function VideoHardwareChart({ tooltip={{ rulerType: 'none', content: (p) => - `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}
${escapeHtml(xLabel)}: ${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}: ${formatMetric(p.y, state.y)}
${s.n}: ${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, + `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}${p.queued ? ` (${s.queuedTag})` : ''}
${s.deployment}: ${escapeHtml(layoutLabel(p, locale))}
${escapeHtml(xLabel)}: ${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}: ${formatMetric(p.y, state.y)}
${s.n}: ${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, onPointClick: onSelect, }} zoom={{ @@ -205,7 +204,11 @@ export default function VideoHardwareChart({

) : undefined } - caption={

{s.hull}

} + caption={ +

+ {caption.join(' ')} +

+ } /> ); } diff --git a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx index 0b5a985f2..62712c418 100644 --- a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx +++ b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx @@ -3,6 +3,7 @@ import { Card } from '@/components/ui/card'; import { Skeleton } from '@/components/ui/skeleton'; import { useLocale } from '@/lib/use-locale'; +import { leadCell } from './deployment'; import { hardwareLabel, VIDEO_HARDWARE_ROSTER } from './hardware'; import { formatMetric, metricValue, type MetricId, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; @@ -42,7 +43,7 @@ const STRINGS = { }, }; -/** One card per campaign hardware at C1; hardware without a valid run says so instead of vanishing. */ +/** One card per campaign hardware, showing its most efficient deployment; hardware without a valid run says so instead of vanishing. */ export default function VideoKpiCards({ points, state, @@ -62,7 +63,7 @@ export default function VideoKpiCards({ return (
{VIDEO_HARDWARE_ROSTER.map(({ key, unavailable }) => { - const point = cells.find((p) => p.hardwareKey === key && p.concurrency === 1); + const point = leadCell(cells, key, options); return (
@@ -74,6 +75,10 @@ export default function VideoKpiCards({ {point && ( {point.participating ?? '—'} {s.of} {point.allocated ?? '—'} {s.gpus} + {point.server?.tp !== null && + point.server?.tp !== undefined && + point.server.ulysses !== null && + ` · TP${point.server.tp} × Ulysses ${point.server.ulysses}`} )}
diff --git a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx index 0051a3914..3e488b6e9 100644 --- a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx +++ b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx @@ -2,6 +2,7 @@ import { DataTable, type DataTableColumn } from '@/components/ui/data-table'; import { useLocale } from '@/lib/use-locale'; +import { isQueueing, layoutLabel } from './deployment'; import { hardwareLabel } from './hardware'; import { formatMetric, metricLabel, metricValue, type MetricId, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; @@ -10,14 +11,18 @@ import type { VideoDashboardState } from './video-url-state'; const STRINGS = { en: { hardware: 'Hardware', + deployment: 'Deployment', c: 'C', + queued: 'queued', counts: 'Valid / scheduled', power: 'Board power (W)', run: 'CI run', }, zh: { hardware: '硬件', + deployment: '部署', c: 'C', + queued: '排队', counts: '有效 / 计划', power: '板卡功率(W)', run: 'CI 运行', @@ -33,14 +38,14 @@ const METRIC_COLUMNS: readonly MetricId[] = [ 'powerPctCap', ]; -/** Rows for the plotted cells: every metric the chart can show, plus provenance. */ +/** Rows for the plotted cells: every deployment, queued cells on request, plus provenance. */ export function videoTableRows( points: VideoPoint[], state: VideoDashboardState, hidden: ReadonlySet, ): VideoPoint[] { return latestVideoCells(points).filter( - (p) => p.hardwareKey && !hidden.has(p.hardwareKey) && (state.queue || p.concurrency === 1), + (p) => p.hardwareKey && !hidden.has(p.hardwareKey) && (state.queue || !isQueueing(p)), ); } @@ -65,10 +70,16 @@ export default function VideoPointsTable({ importance: 'key', pinned: true, }, + { + header: s.deployment, + cell: (p) => layoutLabel(p, locale), + sortValue: (p) => p.participating ?? 0, + importance: 'key', + }, { header: s.c, align: 'right', - cell: (p) => p.concurrency ?? '—', + cell: (p) => (isQueueing(p) ? `${p.concurrency} · ${s.queued}` : (p.concurrency ?? '—')), sortValue: (p) => p.concurrency ?? 0, importance: 'key', }, diff --git a/packages/app/src/components/video-benchmark/plot.test.ts b/packages/app/src/components/video-benchmark/plot.test.ts new file mode 100644 index 000000000..bbbda936c --- /dev/null +++ b/packages/app/src/components/video-benchmark/plot.test.ts @@ -0,0 +1,135 @@ +import { describe, expect, it } from 'vitest'; +import type { VideoPoint } from './metrics'; +import { plotVideoPoints } from './plot'; +import { DEFAULT_VIDEO_DASHBOARD_STATE } from './video-url-state'; + +const base: VideoPoint = { + id: 'h200-4g-c1', + runId: '1', + artifactId: 1, + cell: 'c1', + hardwareKey: 'h200', + hardwareName: 'NVIDIA H200', + runtime: 'r', + model: 'm', + workload: 'w', + concurrency: 1, + participating: 4, + allocated: 8, + replicas: null, + valid: 20, + completed: 20, + scheduled: 20, + failed: 0, + samples: 20, + p50: 150, + p90: 151, + wallSeconds: 3000, // 6.0 videos per GPU-hour + durationSeconds: 8, + frameCount: 192, + energyKj: 400, + avgPowerW: 2700, + enforcedLimitW: 2800, + server: { tp: 2, ulysses: 2, attention: null }, + status: 'complete', + observedAt: null, +}; +const cell = (patch: Partial): VideoPoint => ({ ...base, ...patch }); +// H200 deployments: 8 boards are faster but less efficient, 2 boards slower but more efficient, +// and a second 4-board split that is dominated by the first (slower and less efficient). +const h200 = [ + base, + cell({ + id: 'h200-8g', + participating: 8, + server: { tp: 4, ulysses: 2, attention: null }, + p90: 85, + p50: 84, + wallSeconds: 1700, + }), // 5.29 + cell({ + id: 'h200-2g', + participating: 2, + server: { tp: 2, ulysses: 1, attention: null }, + p90: 290, + p50: 289, + wallSeconds: 5400, + }), // 6.67 + cell({ + id: 'h200-4g-tp4', + server: { tp: 4, ulysses: 1, attention: null }, + p90: 160, + p50: 159, + wallSeconds: 3100, + }), // 5.81 + cell({ id: 'h200-4g-c2', concurrency: 2, p90: 300, p50: 299, wallSeconds: 2990 }), // queued, marginally higher rate +]; +const b200 = cell({ + id: 'b200-4g', + hardwareKey: 'b200', + hardwareName: 'NVIDIA B200', + p90: 78, + p50: 77, + wallSeconds: 1560, +}); +const colorFor = (key: string) => `color-${key}`; +const state = { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + x: 'p90Latency' as const, + y: 'videosPerGpuHour' as const, +}; + +describe('plotVideoPoints', () => { + it('builds a per-hardware frontier over deployments and leaves queued cells out of it', () => { + const plot = plotVideoPoints([...h200, b200], state, colorFor, new Set()); + expect(plot.frontiers.h200.map((p) => p.id)).toEqual(['h200-8g', 'h200-4g-c1', 'h200-2g']); + expect(plot.frontiers.b200.map((p) => p.id)).toEqual(['b200-4g']); + expect(plot.plotted.map((p) => p.id)).toEqual([ + 'h200-4g-c1', + 'h200-8g', + 'h200-2g', + 'h200-4g-tp4', + 'b200-4g', + ]); + expect(plot.plotted.find((p) => p.id === 'h200-4g-tp4')?.optimal).toBe(false); + expect(plot.multiLayout).toBe(true); + // B200 is faster and more efficient than every H200 deployment, so it is the whole global frontier. + expect(plot.global.map((p) => p.id)).toEqual(['b200-4g']); + }); + it('shows queued cells only on request, faded from every frontier', () => { + const plot = plotVideoPoints(h200, { ...state, queue: true }, colorFor, new Set()); + const queued = plot.plotted.filter((p) => p.queued); + expect(queued.map((p) => p.id)).toEqual(['h200-4g-c2']); + expect(plot.frontiers.h200.map((p) => p.id)).not.toContain('h200-4g-c2'); + expect(plot.global.map((p) => p.id)).not.toContain('h200-4g-c2'); + }); + it('optimal-only hides dominated deployments and queued cells', () => { + const plot = plotVideoPoints( + h200, + { ...state, queue: true, optimal: true }, + colorFor, + new Set(), + ); + expect(plot.plotted.map((p) => p.id)).toEqual(['h200-4g-c1', 'h200-8g', 'h200-2g']); + }); + it('is a scatter of single deployments when each hardware has one layout', () => { + const plot = plotVideoPoints([base, b200], state, colorFor, new Set()); + expect(plot.multiLayout).toBe(false); + expect(Object.values(plot.frontiers).every((f) => f.length === 1)).toBe(true); + expect(plot.global.map((p) => p.id)).toEqual(['b200-4g']); + }); + it('drops hidden hardware and cells missing either metric', () => { + const plot = plotVideoPoints( + [ + base, + b200, + cell({ id: 'no-p90', hardwareKey: 'h100', hardwareName: 'H100', samples: 4, p90: null }), + ], + state, + colorFor, + new Set(['b200']), + ); + expect(plot.plotted.map((p) => p.id)).toEqual(['h200-4g-c1']); + expect(plot.frontiers).not.toHaveProperty('b200'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/plot.ts b/packages/app/src/components/video-benchmark/plot.ts new file mode 100644 index 000000000..fa1c2037d --- /dev/null +++ b/packages/app/src/components/video-benchmark/plot.ts @@ -0,0 +1,84 @@ +import { deploymentKey, isQueueing } from './deployment'; +import { paretoFrontier } from './frontier'; +import { hardwareLabel } from './hardware'; +import { metricValue, VIDEO_METRICS, type VideoPoint } from './metrics'; +import { latestVideoCells } from './points'; +import type { VideoDashboardState } from './video-url-state'; + +export interface PlottedVideoPoint extends VideoPoint { + x: number; + y: number; + color: string; + label: string; + /** Client concurrency above one request per replica: the same deployment under load. */ + queued: boolean; + /** On its hardware's Pareto frontier (deployment cells only). */ + optimal: boolean; +} + +export interface VideoPlot { + /** Cells to draw: deployments (dominated ones unless `optimal`), plus queued cells when `queue`. */ + plotted: PlottedVideoPoint[]; + /** Per-hardware Pareto frontier over its deployment cells, ascending x; lines need two or more. */ + frontiers: Record; + /** Cross-hardware Pareto frontier over every deployment cell. */ + global: PlottedVideoPoint[]; + /** True once any hardware has more than one measured deployment. */ + multiLayout: boolean; +} + +/** + * The chart's data model. A hardware's curve is the Pareto frontier over its + * measured deployments (GPUs per video and how they split the model); client + * concurrency on a batch-one server only queues requests, so queued cells are + * plotted as evidence but never join a frontier. + */ +export function plotVideoPoints( + points: VideoPoint[], + state: VideoDashboardState, + colorFor: (hardwareKey: string) => string, + hidden: ReadonlySet, +): VideoPlot { + const options = { tier: state.tier, basis: state.basis }; + const cells: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { + if (!p.hardwareKey || hidden.has(p.hardwareKey)) return []; + const x = metricValue(p, state.x, options); + const y = metricValue(p, state.y, options); + if (x === null || y === null) return []; + return [ + { + ...p, + x, + y, + color: colorFor(p.hardwareKey), + label: hardwareLabel(p.hardwareKey), + queued: isQueueing(p), + optimal: false, + }, + ]; + }); + const deployments = cells.filter((p) => !p.queued); + const byHardware = new Map(); + for (const p of deployments) { + const list = byHardware.get(p.hardwareKey!) ?? []; + list.push(p); + byHardware.set(p.hardwareKey!, list); + } + const xBetter = VIDEO_METRICS[state.x].polarity; + const yBetter = VIDEO_METRICS[state.y].polarity; + const frontiers: Record = {}; + let multiLayout = false; + for (const [key, list] of byHardware) { + frontiers[key] = paretoFrontier(list, xBetter, yBetter); + for (const p of frontiers[key]) p.optimal = true; + if (new Set(list.map(deploymentKey)).size > 1) multiLayout = true; + } + const shown = state.optimal ? deployments.filter((p) => p.optimal) : deployments; + const queued = state.queue && !state.optimal ? cells.filter((p) => p.queued) : []; + return { + plotted: [...shown, ...queued], + frontiers, + global: paretoFrontier(deployments, xBetter, yBetter), + multiLayout, + }; +} diff --git a/packages/app/src/components/video-benchmark/video-url-state.test.ts b/packages/app/src/components/video-benchmark/video-url-state.test.ts index 31fca027e..f7cd3ed44 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.test.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.test.ts @@ -15,7 +15,7 @@ describe('video dashboard URL state', () => { it('reads every supported param', () => { expect( readVideoDashboardState( - '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_view=table', + '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_frontier=1&v_view=table', ), ).toEqual({ x: 'genSpeed', @@ -24,6 +24,7 @@ describe('video dashboard URL state', () => { basis: 'allocated', queue: true, optimal: true, + frontier: true, view: 'table', }); }); diff --git a/packages/app/src/components/video-benchmark/video-url-state.ts b/packages/app/src/components/video-benchmark/video-url-state.ts index aad6c229e..435fe1b97 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.ts @@ -13,10 +13,12 @@ export interface VideoDashboardState { y: YMetricId; tier: CostTier; basis: GpuBasis; - /** Show C2/C4 client-concurrency cells as dotted queueing tails. */ + /** Also plot queued client-concurrency cells (C > replicas) as faded markers. */ queue: boolean; - /** Show only Pareto-optimal hardware. */ + /** Hide deployments dominated by another deployment of the same hardware. */ optimal: boolean; + /** Draw the cross-hardware Pareto frontier over all deployments. */ + frontier: boolean; view: 'chart' | 'table'; } @@ -27,6 +29,7 @@ export const DEFAULT_VIDEO_DASHBOARD_STATE: VideoDashboardState = { basis: 'participating', queue: false, optimal: false, + frontier: false, view: 'chart', }; @@ -47,6 +50,7 @@ export function readVideoDashboardState(search: string): VideoDashboardState { basis: pick(BASES, p.get('v_basis'), d.basis), queue: p.get('v_queue') === '1', optimal: p.get('v_opt') === '1', + frontier: p.get('v_frontier') === '1', view: pick(VIEWS, p.get('v_view'), d.view), }; } @@ -63,6 +67,7 @@ export function writeVideoDashboardState(url: URL, state: VideoDashboardState): set('v_basis', state.basis === d.basis ? null : state.basis); set('v_queue', state.queue ? '1' : null); set('v_opt', state.optimal ? '1' : null); + set('v_frontier', state.frontier ? '1' : null); set('v_view', state.view === d.view ? null : state.view); return out; } From 85329460342f11b7aab41f5477b567934977c001 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:30:01 -0700 Subject: [PATCH 018/104] feat: add H3 API price reference and API-priced video metrics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增带日期与来源的 H3 API 标价参考常量(0.034 $/视频秒,区间 0.034–0.047,采集于 2026-09-19),并在视频指标注册表中加入每条视频 API 标价、按标价计的每 GPU 小时收入、每 GPU 小时利润与标价倍数;缺少价格时全部返回 null,货币格式的负号置于 $ 之前。 --- .../video-benchmark/api-reference.test.ts | 24 +++++ .../video-benchmark/api-reference.ts | 31 ++++++ .../video-benchmark/metrics.test.ts | 65 +++++++++++++ .../src/components/video-benchmark/metrics.ts | 97 ++++++++++++++++++- 4 files changed, 213 insertions(+), 4 deletions(-) create mode 100644 packages/app/src/components/video-benchmark/api-reference.test.ts create mode 100644 packages/app/src/components/video-benchmark/api-reference.ts diff --git a/packages/app/src/components/video-benchmark/api-reference.test.ts b/packages/app/src/components/video-benchmark/api-reference.test.ts new file mode 100644 index 000000000..f032eaf20 --- /dev/null +++ b/packages/app/src/components/video-benchmark/api-reference.test.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from 'vitest'; +import { formatApiPrice, H3_API_REFERENCE } from './api-reference'; + +describe('H3 API price reference', () => { + it('is a dated, sourced list price inside its own listed range', () => { + const { pricePerVideoSecondUsd, rangeUsd, capturedOn, label } = H3_API_REFERENCE; + expect(pricePerVideoSecondUsd).toBe(0.034); + expect(rangeUsd).toEqual([0.034, 0.047]); + expect(pricePerVideoSecondUsd).toBeGreaterThanOrEqual(rangeUsd[0]); + expect(pricePerVideoSecondUsd).toBeLessThanOrEqual(rangeUsd[1]); + expect(capturedOn).toBe('2026-09-19'); + expect(Number.isNaN(Date.parse(capturedOn))).toBe(false); + // The source text must keep saying what was not verified, in both locales. + expect(label.en).toContain('MiniMax Design'); + expect(label.en).toContain('not verified'); + expect(label.zh).toContain('MiniMax Design'); + expect(label.zh).toContain('未在按量付费页面核实'); + }); + it('formats USD per video-second to the input step, keeping a fourth decimal when typed', () => { + expect(formatApiPrice(0.034)).toBe('$0.034'); + expect(formatApiPrice(0.05)).toBe('$0.050'); + expect(formatApiPrice(0.0345)).toBe('$0.0345'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/api-reference.ts b/packages/app/src/components/video-benchmark/api-reference.ts new file mode 100644 index 000000000..17d668a89 --- /dev/null +++ b/packages/app/src/components/video-benchmark/api-reference.ts @@ -0,0 +1,31 @@ +export interface ApiPriceReference { + /** USD an API bills per generated video-second at the captured tier. */ + pricePerVideoSecondUsd: number; + /** Listed range across the captured tiers, USD per video-second. */ + rangeUsd: readonly [number, number]; + /** ISO date the price was captured. */ + capturedOn: string; + /** Where the price was seen, including what was not verified. */ + label: { en: string; zh: string }; +} + +/** + * Dated, sourced API list price the dashboard compares self-hosted TCO against. + * It is a reference input, not a measurement: the reader can override it + * (`v_api`) and every derived number is labelled as list price. Change the + * numbers only together with a new capture date and source text. + */ +export const H3_API_REFERENCE: ApiPriceReference = { + pricePerVideoSecondUsd: 0.034, + rangeUsd: [0.034, 0.047], + capturedOn: '2026-09-19', + label: { + en: 'MiniMax Design · H3 768p subscription tier (web search, not verified on the pay-as-you-go page; H3 was listed as not yet supported on the API packages page)', + zh: 'MiniMax Design · H3 768p 订阅档(来源为网页搜索,未在按量付费页面核实;API 套餐页当时标注 H3 尚未支持)', + }, +}; + +/** `$0.034` style USD per video-second: at least the three decimals the input steps by. */ +export function formatApiPrice(value: number): string { + return `$${value.toLocaleString('en-US', { minimumFractionDigits: 3, maximumFractionDigits: 4 })}`; +} diff --git a/packages/app/src/components/video-benchmark/metrics.test.ts b/packages/app/src/components/video-benchmark/metrics.test.ts index 19f963dfd..d7e7d5f0d 100644 --- a/packages/app/src/components/video-benchmark/metrics.test.ts +++ b/packages/app/src/components/video-benchmark/metrics.test.ts @@ -46,6 +46,8 @@ const h100: VideoPoint = { avgPowerW: 2574.550733165641, }; const opts = { tier: 'h', basis: 'participating' } as const; +// H3 768p list price captured 2026-09-19 (api-reference.ts), USD per video-second. +const priced = { ...opts, apiPricePerVideoSecond: 0.034 }; describe('video metrics', () => { it('derives throughput per GPU-hour from valid clips, wall time and participating GPUs', () => { @@ -77,6 +79,52 @@ describe('video metrics', () => { expect(metricValue(h200, 'videosPerKwh', opts)).toBeCloseTo(8.76, 2); expect(metricValue(h200, 'powerPctCap', opts)).toBeCloseTo(97.41, 1); }); + it('prices the clip at the API reference and derives revenue, profit and headroom', () => { + // 0.034 $/s × 8 s clip; 5.9732 videos/GPU-hr × $0.272; − $1.22 H200 hyperscaler GPU-hr. + expect(metricValue(h200, 'apiPricePerVideo', priced)).toBeCloseTo(0.272, 6); + expect(metricValue(h200, 'revenuePerGpuHour', priced)).toBeCloseTo(1.6247, 3); + expect(metricValue(h200, 'profitPerGpuHour', priced)).toBeCloseTo(0.4047, 3); + // $0.272 API ÷ $0.2042 TCO per video. + expect(metricValue(h200, 'apiPriceMultiple', priced)).toBeCloseTo(1.332, 3); + // Rental tier: same revenue, $2.90 GPU-hr → loss; headroom drops below 1. + expect(metricValue(h200, 'profitPerGpuHour', { ...priced, tier: 'r' })).toBeCloseTo(-1.2753, 3); + expect(metricValue(h200, 'apiPriceMultiple', { ...priced, tier: 'r' })).toBeCloseTo(0.5603, 3); + }); + it('halves revenue on the allocated basis when only half the reserved GPUs generate', () => { + // H100: 4 of 8 allocated boards participate → 2.687 videos/GPU-hr × $0.272 − $1.17. + expect(metricValue(h100, 'revenuePerGpuHour', priced)).toBeCloseTo(1.4618, 3); + expect(metricValue(h100, 'profitPerGpuHour', priced)).toBeCloseTo(0.2918, 3); + const allocated = { ...priced, basis: 'allocated' } as const; + expect(metricValue(h100, 'revenuePerGpuHour', allocated)).toBeCloseTo(0.7309, 3); + expect(metricValue(h100, 'profitPerGpuHour', allocated)).toBeCloseTo(-0.4391, 3); + // The clip price itself does not depend on the GPU basis. + expect(metricValue(h100, 'apiPricePerVideo', allocated)).toBeCloseTo(0.272, 6); + expect(metricValue(h100, 'apiPriceMultiple', allocated)).toBeCloseTo(0.6247, 3); + }); + it('returns null for every API-priced metric without a positive price or clip length', () => { + const ids = [ + 'apiPricePerVideo', + 'revenuePerGpuHour', + 'profitPerGpuHour', + 'apiPriceMultiple', + ] as const; + for (const id of ids) { + expect(metricValue(h200, id, opts)).toBeNull(); + expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: null })).toBeNull(); + expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: 0 })).toBeNull(); + expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: -0.01 })).toBeNull(); + expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: Number.NaN })).toBeNull(); + expect(metricValue({ ...h200, durationSeconds: null }, id, priced)).toBeNull(); + } + // A price alone is not enough: revenue, profit and headroom still need throughput and a tier cost. + expect(metricValue({ ...h200, wallSeconds: null }, 'revenuePerGpuHour', priced)).toBeNull(); + expect(metricValue({ ...h200, hardwareKey: null }, 'profitPerGpuHour', priced)).toBeNull(); + expect(metricValue({ ...h200, hardwareKey: null }, 'apiPriceMultiple', priced)).toBeNull(); + expect(metricValue({ ...h200, wallSeconds: null }, 'apiPricePerVideo', priced)).toBeCloseTo( + 0.272, + 6, + ); + }); it('returns null instead of zero for missing inputs', () => { const broken: VideoPoint = { ...h200, @@ -107,4 +155,21 @@ describe('video metrics', () => { expect(formatMetric(151.10486, 'p90Latency')).toBe('151.1'); expect(formatMetric(1234.5, 'videoSecondsPerGpuHour')).toBe('1,234.5'); }); + it('labels and formats the API-priced metrics, with the sign before the currency symbol', () => { + expect(metricLabel('profitPerGpuHour', 'en', opts)).toBe( + 'Profit per GPU-hour, API list price − TCO (Owning at Large Hyperscaler Volume)', + ); + expect(metricLabel('apiPriceMultiple', 'zh', { ...opts, tier: 'r' })).toBe( + 'API 标价 ÷ 每条视频 TCO 成本(租赁 - 3 年承诺)', + ); + expect(metricLabel('revenuePerGpuHour', 'en', opts)).toBe( + 'Revenue per GPU-hour at API list price', + ); + expect(metricLabel('apiPricePerVideo', 'zh', opts)).toBe('每条视频 API 标价'); + expect(formatMetric(0.272, 'apiPricePerVideo')).toBe('$0.272'); + expect(formatMetric(1.62471, 'revenuePerGpuHour')).toBe('$1.62'); + expect(formatMetric(-0.43913, 'profitPerGpuHour')).toBe('-$0.44'); + expect(formatMetric(1.3318, 'apiPriceMultiple')).toBe('1.33'); + expect(formatMetric(-1.5, 'p50Latency')).toBe('-1.5'); + }); }); diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts index da55c8675..aea6ee6e3 100644 --- a/packages/app/src/components/video-benchmark/metrics.ts +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -48,12 +48,26 @@ export type YMetricId = | 'videoSecondsPerGpuHour' | 'dollarsPerVideo' | 'kjPerVideo' - | 'videosPerKwh'; -export type MetricId = XMetricId | YMetricId | 'powerPctCap' | 'dollarsPerVideoSecond'; + | 'videosPerKwh' + | 'revenuePerGpuHour' + | 'profitPerGpuHour' + | 'apiPriceMultiple'; +export type MetricId = + | XMetricId + | YMetricId + | 'powerPctCap' + | 'dollarsPerVideoSecond' + | 'apiPricePerVideo'; export interface MetricOptions { tier: CostTier; basis: GpuBasis; + /** + * USD an API bills per video-second of the same clip (`H3_API_REFERENCE` or + * the reader's override). Missing or non-positive → every API-priced metric + * is null. + */ + apiPricePerVideoSecond?: number | null; } interface MetricDefinition { @@ -76,6 +90,9 @@ export const Y_METRICS: readonly YMetricId[] = [ 'dollarsPerVideo', 'kjPerVideo', 'videosPerKwh', + 'revenuePerGpuHour', + 'profitPerGpuHour', + 'apiPriceMultiple', ]; export const COST_TIERS: readonly CostTier[] = ['h', 'r']; @@ -168,6 +185,42 @@ export const VIDEO_METRICS: Record = { polarity: 'higher', digits: 1, }, + // API-priced metrics: list price × clip length, seen from the self-hosting + // operator's side, so more revenue, profit or price headroom is "higher". + apiPricePerVideo: { + label: 'API list price per video', + labelZh: '每条视频 API 标价', + unit: '$/video', + polarity: 'higher', + digits: 3, + currency: true, + }, + revenuePerGpuHour: { + label: 'Revenue per GPU-hour at API list price', + labelZh: '按 API 标价计的每 GPU 小时收入', + unit: '$/GPU-hr', + polarity: 'higher', + digits: 2, + currency: true, + }, + // Tiered labels take the cost tier in parentheses, so they carry no unit parentheses of their own. + profitPerGpuHour: { + label: 'Profit per GPU-hour, API list price − TCO', + labelZh: '每 GPU 小时利润,API 标价 − TCO', + unit: '$/GPU-hr', + polarity: 'higher', + digits: 2, + tiered: true, + currency: true, + }, + apiPriceMultiple: { + label: 'API list price ÷ TCO cost per video', + labelZh: 'API 标价 ÷ 每条视频 TCO 成本', + unit: '×', + polarity: 'higher', + digits: 2, + tiered: true, + }, }; const finite = (n: number | null | undefined): n is number => @@ -191,6 +244,18 @@ function dollarsPerVideo(point: VideoPoint, options: MetricOptions): number | nu return rate !== null && cost !== null ? cost / rate : null; } +/** What an API would bill for one clip of this cell's length at the reference price. */ +function apiPricePerVideo(point: VideoPoint, options: MetricOptions): number | null { + const price = options.apiPricePerVideoSecond; + return positive(price) && positive(point.durationSeconds) ? price * point.durationSeconds : null; +} + +function revenuePerGpuHour(point: VideoPoint, options: MetricOptions): number | null { + const rate = videosPerGpuHour(point, options.basis); + const price = apiPricePerVideo(point, options); + return rate !== null && price !== null ? rate * price : null; +} + /** Every formula returns null (never 0) when an input is missing or invalid. */ export function metricValue( point: VideoPoint, @@ -253,6 +318,28 @@ export function metricValue( : null; break; } + case 'apiPricePerVideo': { + value = apiPricePerVideo(point, options); + break; + } + case 'revenuePerGpuHour': { + value = revenuePerGpuHour(point, options); + break; + } + case 'profitPerGpuHour': { + // Negative when the API list price does not cover the tier's GPU-hour. + const revenue = revenuePerGpuHour(point, options); + const cost = point.hardwareKey ? costPerGpuHour(point.hardwareKey, options.tier) : null; + value = revenue !== null && cost !== null ? revenue - cost : null; + break; + } + case 'apiPriceMultiple': { + // > 1 means the API list price exceeds the self-hosted TCO cost of the clip. + const price = apiPricePerVideo(point, options); + const cost = dollarsPerVideo(point, options); + value = price !== null && cost !== null ? price / cost : null; + break; + } } return finite(value) ? value : null; } @@ -265,12 +352,14 @@ export function metricLabel(id: MetricId, locale: 'en' | 'zh', options: MetricOp return locale === 'zh' ? `${base}(${tier})` : `${base} (${tier})`; } +/** Sign precedes the currency symbol (`-$0.44`), as profit per GPU-hour can be negative. */ export function formatMetric(value: number | null, id: MetricId): string { if (value === null) return '—'; const def = VIDEO_METRICS[id]; - const text = value.toLocaleString('en-US', { + const text = Math.abs(value).toLocaleString('en-US', { minimumFractionDigits: def.currency ? def.digits : 0, maximumFractionDigits: def.digits, }); - return def.currency ? `$${text}` : text; + const sign = value < 0 ? '-' : ''; + return def.currency ? `${sign}$${text}` : `${sign}${text}`; } From 8751d9e00f629927ad8cb8ccc764b31688c7f1a9 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:30:12 -0700 Subject: [PATCH 019/104] feat: carry the API price in the video dashboard URL state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:仪表板状态新增 apiPrice(默认取参考值);URL 参数 v_api 仅接受正的有限数并保留四位小数,无效值回退默认、等于默认时省略;新增 metricOptions(state) 把状态映射为指标选项。 --- .../video-benchmark/video-url-state.test.ts | 69 ++++++++++++++++++- .../video-benchmark/video-url-state.ts | 28 ++++++++ 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/packages/app/src/components/video-benchmark/video-url-state.test.ts b/packages/app/src/components/video-benchmark/video-url-state.test.ts index f7cd3ed44..e7dcb3b19 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.test.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from 'vitest'; +import { H3_API_REFERENCE } from './api-reference'; import { DEFAULT_VIDEO_DASHBOARD_STATE, + metricOptions, + parseApiPrice, readVideoDashboardState, writeVideoDashboardState, } from './video-url-state'; @@ -9,13 +12,15 @@ describe('video dashboard URL state', () => { it('falls back to defaults for missing or invalid params', () => { expect(readVideoDashboardState('')).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); expect( - readVideoDashboardState('?v_x=bogus&v_y=nope&v_tier=z&v_basis=q&v_view=pie&v_queue=yes'), + readVideoDashboardState( + '?v_x=bogus&v_y=nope&v_tier=z&v_basis=q&v_view=pie&v_queue=yes&v_api=free', + ), ).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); }); it('reads every supported param', () => { expect( readVideoDashboardState( - '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_frontier=1&v_view=table', + '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_frontier=1&v_view=table&v_api=0.047', ), ).toEqual({ x: 'genSpeed', @@ -26,7 +31,24 @@ describe('video dashboard URL state', () => { optimal: true, frontier: true, view: 'table', + apiPrice: 0.047, }); + expect(readVideoDashboardState('?v_y=profitPerGpuHour').y).toBe('profitPerGpuHour'); + }); + it('defaults the API price to the dated reference and rejects non-positive or malformed prices', () => { + expect(DEFAULT_VIDEO_DASHBOARD_STATE.apiPrice).toBe(H3_API_REFERENCE.pricePerVideoSecondUsd); + for (const bad of ['0', '-0.034', 'NaN', 'Infinity', '', ' ', 'abc', '0.00004']) { + expect(readVideoDashboardState(`?v_api=${bad}`).apiPrice).toBe(0.034); + expect(parseApiPrice(bad)).toBeNull(); + } + expect(parseApiPrice(null)).toBeNull(); + expect(parseApiPrice(undefined)).toBeNull(); + expect(parseApiPrice(Number.NaN)).toBeNull(); + // Positive prices are kept to four decimals, so the URL and the input agree. + expect(parseApiPrice('0.05')).toBe(0.05); + expect(parseApiPrice('0.03456')).toBe(0.0346); + expect(parseApiPrice(0.0001)).toBe(0.0001); + expect(readVideoDashboardState('?v_api=1e-2').apiPrice).toBe(0.01); }); it('writes only non-default params and keeps unrelated params', () => { const url = new URL('https://x.test/video?run=1&view=results'); @@ -40,9 +62,52 @@ describe('video dashboard URL state', () => { expect(out.searchParams.get('v_tier')).toBe('r'); expect(out.searchParams.get('v_queue')).toBe('1'); expect(out.searchParams.has('v_x')).toBe(false); + expect(out.searchParams.has('v_api')).toBe(false); expect(url.searchParams.has('v_tier')).toBe(false); const reset = writeVideoDashboardState(out, DEFAULT_VIDEO_DASHBOARD_STATE); expect([...reset.searchParams.keys()].filter((k) => k.startsWith('v_'))).toEqual([]); expect(reset.searchParams.get('run')).toBe('1'); }); + it('round-trips an overridden API price and drops an invalid one', () => { + const url = new URL('https://x.test/video'); + const out = writeVideoDashboardState(url, { ...DEFAULT_VIDEO_DASHBOARD_STATE, apiPrice: 0.05 }); + expect(out.searchParams.get('v_api')).toBe('0.05'); + expect(readVideoDashboardState(out.search).apiPrice).toBe(0.05); + expect( + writeVideoDashboardState(url, { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + apiPrice: 0.03456, + }).searchParams.get('v_api'), + ).toBe('0.0346'); + expect( + writeVideoDashboardState(out, { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + apiPrice: 0, + }).searchParams.has('v_api'), + ).toBe(false); + expect( + writeVideoDashboardState(out, { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + apiPrice: Number.NaN, + }).searchParams.has('v_api'), + ).toBe(false); + }); + it('maps state to metric options, including the API price', () => { + expect(metricOptions(DEFAULT_VIDEO_DASHBOARD_STATE)).toEqual({ + tier: 'h', + basis: 'participating', + apiPricePerVideoSecond: 0.034, + }); + expect( + metricOptions({ + ...DEFAULT_VIDEO_DASHBOARD_STATE, + tier: 'r', + basis: 'allocated', + apiPrice: 0.05, + }), + ).toEqual({ tier: 'r', basis: 'allocated', apiPricePerVideoSecond: 0.05 }); + expect( + metricOptions({ ...DEFAULT_VIDEO_DASHBOARD_STATE, apiPrice: -1 }).apiPricePerVideoSecond, + ).toBeNull(); + }); }); diff --git a/packages/app/src/components/video-benchmark/video-url-state.ts b/packages/app/src/components/video-benchmark/video-url-state.ts index 435fe1b97..29626b1e1 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.ts @@ -1,9 +1,11 @@ +import { H3_API_REFERENCE } from './api-reference'; import type { CostTier } from './hardware'; import { COST_TIERS, X_METRICS, Y_METRICS, type GpuBasis, + type MetricOptions, type XMetricId, type YMetricId, } from './metrics'; @@ -20,6 +22,8 @@ export interface VideoDashboardState { /** Draw the cross-hardware Pareto frontier over all deployments. */ frontier: boolean; view: 'chart' | 'table'; + /** USD per video-second the API-priced metrics assume; the dated reference unless the reader overrides it. */ + apiPrice: number; } export const DEFAULT_VIDEO_DASHBOARD_STATE: VideoDashboardState = { @@ -31,6 +35,7 @@ export const DEFAULT_VIDEO_DASHBOARD_STATE: VideoDashboardState = { optimal: false, frontier: false, view: 'chart', + apiPrice: H3_API_REFERENCE.pricePerVideoSecondUsd, }; const BASES: readonly GpuBasis[] = ['participating', 'allocated']; @@ -39,6 +44,26 @@ const VIEWS: readonly VideoDashboardState['view'][] = ['chart', 'table']; const pick = (allowed: readonly T[], value: string | null, fallback: T): T => allowed.includes(value as T) ? (value as T) : fallback; +/** + * A USD/video-second price as typed or read from `v_api`: positive, finite, + * kept to four decimals so the URL and the input agree. Anything else is null, + * so callers fall back to the reference instead of pricing at 0. + */ +export function parseApiPrice(raw: string | number | null | undefined): number | null { + if (raw === null || raw === undefined) return null; + const n = typeof raw === 'number' ? raw : Number(raw.trim() || Number.NaN); + if (!Number.isFinite(n) || n <= 0) return null; + const rounded = Math.round(n * 1e4) / 1e4; + return rounded > 0 ? rounded : null; +} + +/** Metric inputs the dashboard controls select, so no component rebuilds `{ tier, basis }` by hand. */ +export const metricOptions = (state: VideoDashboardState): MetricOptions => ({ + tier: state.tier, + basis: state.basis, + apiPricePerVideoSecond: parseApiPrice(state.apiPrice), +}); + /** Dashboard controls live under the `v_` prefix so the run/artifact/view params stay untouched. */ export function readVideoDashboardState(search: string): VideoDashboardState { const p = new URLSearchParams(search); @@ -52,6 +77,7 @@ export function readVideoDashboardState(search: string): VideoDashboardState { optimal: p.get('v_opt') === '1', frontier: p.get('v_frontier') === '1', view: pick(VIEWS, p.get('v_view'), d.view), + apiPrice: parseApiPrice(p.get('v_api')) ?? d.apiPrice, }; } @@ -69,5 +95,7 @@ export function writeVideoDashboardState(url: URL, state: VideoDashboardState): set('v_opt', state.optimal ? '1' : null); set('v_frontier', state.frontier ? '1' : null); set('v_view', state.view === d.view ? null : state.view); + const apiPrice = parseApiPrice(state.apiPrice); + set('v_api', apiPrice === null || apiPrice === d.apiPrice ? null : String(apiPrice)); return out; } From 92e9130e075e9bd412b6f8a54e7b2678ddd1095e Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:30:25 -0700 Subject: [PATCH 020/104] feat: add API price reference control and revenue/profit views to the video dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:基准测试配置面板新增 API 价格参考控件(数值输入、重置按钮,以及含参考值、区间、采集日期与来源的说明);KPI 卡新增“API 标价 / 条”与“利润 / GPU 小时”,数据表新增四个 API 定价次级列;图表与各组件统一改用 metricOptions(state),Y 轴允许负的每 GPU 小时利润。 --- .../video-benchmark/VideoApiReference.tsx | 128 ++++++++++++++++++ .../video-benchmark/VideoConfigBar.tsx | 8 +- .../video-benchmark/VideoHardwareChart.tsx | 4 +- .../video-benchmark/VideoKpiCards.tsx | 10 +- .../video-benchmark/VideoPointsTable.tsx | 8 +- .../src/components/video-benchmark/plot.ts | 4 +- 6 files changed, 151 insertions(+), 11 deletions(-) create mode 100644 packages/app/src/components/video-benchmark/VideoApiReference.tsx diff --git a/packages/app/src/components/video-benchmark/VideoApiReference.tsx b/packages/app/src/components/video-benchmark/VideoApiReference.tsx new file mode 100644 index 000000000..b128f84ba --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoApiReference.tsx @@ -0,0 +1,128 @@ +'use client'; + +import { useId, useRef, useState } from 'react'; +import { Button } from '@/components/ui/button'; +import { Input } from '@/components/ui/input'; +import { track } from '@/lib/analytics'; +import { useLocale } from '@/lib/use-locale'; +import { formatApiPrice, H3_API_REFERENCE } from './api-reference'; +import { parseApiPrice } from './video-url-state'; + +const STRINGS = { + en: { + label: 'API price reference ($/video-second)', + reset: 'Reset', + resetTitle: 'Reset to the dated reference price', + reference: 'Reference', + range: 'listed range', + captured: 'captured', + source: 'Source: ', + disclaimer: + 'List price, not realized revenue; self-hosted TCO excludes utilization, CPU, storage and network.', + }, + zh: { + label: 'API 价格参考($/视频秒)', + reset: '重置', + resetTitle: '恢复为带日期的参考价格', + reference: '参考值', + range: '标价区间', + captured: '采集于', + source: '来源:', + disclaimer: '为标价而非实际收入;自托管 TCO 不含利用率、CPU、存储与网络成本。', + }, +}; + +/** + * The one editable economics input: USD an API bills per video-second. It + * sits in the Benchmark config panel next to the frozen model and workload, + * and its caption keeps the dated reference and source in view so a changed + * price is always read against what was captured. Valid prices apply as they + * are typed; an empty or non-positive box leaves the last valid price in + * place and snaps back on blur. + */ +export default function VideoApiReference({ + value, + onChange, +}: { + value: number; + onChange: (value: number) => void; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const id = useId(); + const { pricePerVideoSecondUsd: reference, rangeUsd, capturedOn, label } = H3_API_REFERENCE; + const [raw, setRaw] = useState(() => String(value)); + const [synced, setSynced] = useState(value); + // Adopt a value set elsewhere (URL restore, reset) without clobbering text mid-edit. + if (value !== synced) { + setSynced(value); + if (parseApiPrice(raw) !== value) setRaw(String(value)); + } + const valueAtFocus = useRef(value); + const commit = (next: number, reset = false) => { + if (next === value) return; + onChange(next); + if (reset) track('video_api_price_changed', { value: String(next), reset: true }); + }; + return ( + <> +
+ +
+ { + valueAtFocus.current = value; + }} + onChange={(event) => { + setRaw(event.target.value); + const next = parseApiPrice(event.target.value); + if (next !== null) commit(next); + }} + onBlur={() => { + if (parseApiPrice(raw) === null) setRaw(String(value)); + if (value !== valueAtFocus.current) { + track('video_api_price_changed', { value: String(value) }); + } + }} + onWheel={(event) => event.currentTarget.blur()} + /> + +
+
+
+

+ {s.reference} {formatApiPrice(reference)}/video-s · {s.range}{' '} + {formatApiPrice(rangeUsd[0])}–{formatApiPrice(rangeUsd[1])} · {s.captured} {capturedOn} ·{' '} + {s.source} + {label[locale]} +

+

{s.disclaimer}

+
+ + ); +} diff --git a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx index 7f04a34aa..71beba436 100644 --- a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx +++ b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx @@ -14,8 +14,9 @@ import { type XMetricId, type YMetricId, } from './metrics'; +import VideoApiReference from './VideoApiReference'; import VideoSelect from './VideoSelect'; -import type { VideoDashboardState } from './video-url-state'; +import { metricOptions, type VideoDashboardState } from './video-url-state'; const STRINGS = { en: { @@ -69,14 +70,14 @@ export default function VideoConfigBar({ }) { const locale = useLocale(); const s = STRINGS[locale]; - const options = { tier: state.tier, basis: state.basis }; + const options = metricOptions(state); const change = (key: K, value: VideoDashboardState[K]) => { onChange({ [key]: value } as Partial); track(`video_${key}_changed`, { value: String(value) }); }; return (
- + {}} options={[{ value: 'deployment', label: deploymentLabel }]} /> + onChange({ apiPrice })} /> diff --git a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx index 3e488b6e9..d85893240 100644 --- a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx +++ b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx @@ -6,7 +6,7 @@ import { isQueueing, layoutLabel } from './deployment'; import { hardwareLabel } from './hardware'; import { formatMetric, metricLabel, metricValue, type MetricId, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; -import type { VideoDashboardState } from './video-url-state'; +import { metricOptions, type VideoDashboardState } from './video-url-state'; const STRINGS = { en: { @@ -34,6 +34,10 @@ const METRIC_COLUMNS: readonly MetricId[] = [ 'videosPerGpuHour', 'videosPerDollar', 'dollarsPerVideo', + 'apiPricePerVideo', + 'revenuePerGpuHour', + 'profitPerGpuHour', + 'apiPriceMultiple', 'kjPerVideo', 'powerPctCap', ]; @@ -60,7 +64,7 @@ export default function VideoPointsTable({ }) { const locale = useLocale(); const s = STRINGS[locale]; - const options = { tier: state.tier, basis: state.basis }; + const options = metricOptions(state); const rows = videoTableRows(points, state, hidden); const columns: DataTableColumn[] = [ { diff --git a/packages/app/src/components/video-benchmark/plot.ts b/packages/app/src/components/video-benchmark/plot.ts index fa1c2037d..dd705ac32 100644 --- a/packages/app/src/components/video-benchmark/plot.ts +++ b/packages/app/src/components/video-benchmark/plot.ts @@ -3,7 +3,7 @@ import { paretoFrontier } from './frontier'; import { hardwareLabel } from './hardware'; import { metricValue, VIDEO_METRICS, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; -import type { VideoDashboardState } from './video-url-state'; +import { metricOptions, type VideoDashboardState } from './video-url-state'; export interface PlottedVideoPoint extends VideoPoint { x: number; @@ -39,7 +39,7 @@ export function plotVideoPoints( colorFor: (hardwareKey: string) => string, hidden: ReadonlySet, ): VideoPlot { - const options = { tier: state.tier, basis: state.basis }; + const options = metricOptions(state); const cells: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { if (!p.hardwareKey || hidden.has(p.hardwareKey)) return []; const x = metricValue(p, state.x, options); From c7273ac59b462e6b8e018f7149a11d0f236701fd Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:30:36 -0700 Subject: [PATCH 021/104] test: cover the API price reference in video component specs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增 VideoApiReference 组件测试(参考说明、输入即生效、空值/零值保留上次有效价、失焦回退、重置与外部同步、埋点、中文文案),并扩展仪表板测试:改价后卡片、表格与 v_api 同步更新,allocated 口径下负利润仍在画布内。 --- .../component/video-api-reference.cy.tsx | 78 +++++++++++++++++++ .../cypress/component/video-dashboard.cy.tsx | 49 +++++++++++- 2 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 packages/app/cypress/component/video-api-reference.cy.tsx diff --git a/packages/app/cypress/component/video-api-reference.cy.tsx b/packages/app/cypress/component/video-api-reference.cy.tsx new file mode 100644 index 000000000..958303597 --- /dev/null +++ b/packages/app/cypress/component/video-api-reference.cy.tsx @@ -0,0 +1,78 @@ +import { useState } from 'react'; +import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; +import VideoApiReference from '@/components/video-benchmark/VideoApiReference'; +import { registerAnalyticsClient } from '@/lib/analytics'; + +/** Owns the price like VideoDashboard does and exposes it, plus an external setter. */ +function Harness({ initial, pathname = '/video' }: { initial: number; pathname?: string }) { + const [value, setValue] = useState(initial); + return ( + +
+ +
+ {value} + +
+ ); +} +const input = () => cy.get('[data-testid="video-api-price"]'); +const value = () => cy.get('[data-testid="api-price-value"]'); +const caption = () => cy.get('[data-testid="video-api-reference-caption"]'); + +describe('VideoApiReference', () => { + it('shows the dated reference with its source and applies valid prices as typed', () => { + cy.mount(); + input().should('have.value', '0.034').and('have.attr', 'step', '0.001'); + caption() + .should('contain', 'Reference $0.034/video-s') + .and('contain', 'listed range $0.034–$0.047') + .and('contain', 'captured 2026-09-19') + .and('contain', 'Source: MiniMax Design · H3 768p subscription tier') + .and('contain', 'not verified on the pay-as-you-go page') + .and('contain', 'List price, not realized revenue'); + input().clear().type('0.05'); + value().should('have.text', '0.05'); + }); + it('keeps the last valid price while the box is empty or zero and snaps back on blur', () => { + cy.mount(); + input().clear(); + value().should('have.text', '0.034'); + input().type('0'); + value().should('have.text', '0.034'); + input().blur().should('have.value', '0.034'); + }); + it('resets to the reference, adopts external values and tracks committed changes', () => { + const capture = cy.stub().as('capture'); + registerAnalyticsClient({ capture }); + cy.mount(); + input().should('have.value', '0.05'); + cy.get('[data-testid="video-api-price-reset"]').click(); + value().should('have.text', '0.034'); + input().should('have.value', '0.034'); + cy.get('@capture').should('have.been.calledWith', 'video_api_price_changed', { + value: '0.034', + reset: true, + }); + cy.get('[data-testid="api-price-external"]').click(); + input().should('have.value', '0.047'); + input().clear().type('0.04').blur(); + value().should('have.text', '0.04'); + cy.get('@capture').should('have.been.calledWith', 'video_api_price_changed', { + value: '0.04', + }); + }); + it('renders Chinese copy under /zh', () => { + cy.mount(); + cy.contains('label', 'API 价格参考($/视频秒)').should('exist'); + cy.contains('button', '重置').should('exist'); + caption() + .should('contain', '参考值 $0.034/video-s') + .and('contain', '标价区间 $0.034–$0.047') + .and('contain', '采集于 2026-09-19') + .and('contain', '来源:MiniMax Design · H3 768p 订阅档') + .and('contain', '为标价而非实际收入'); + }); +}); diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index eb64482fd..0fc71ba50 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -107,6 +107,49 @@ describe('Video hardware dashboard (retained fixture)', () => { cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); cy.get('[data-testid="video-ci-runs"]').should('exist'); }); + it('reprices the API reference, updating the cards, the table and the URL', () => { + mount(); + // 0.034 $/video-s × 8 s clip = $0.272; 5.9732 videos/GPU-hr × $0.272 − $1.22 = $0.40 profit. + kpi('h200').should('contain', 'API $/video').and('contain', '$0.272').and('contain', '$0.40'); + // H100 on the participating basis: 5.3741 × $0.272 − $1.17 = $0.29. + kpi('h100').should('contain', '$0.29'); + cy.get('[data-testid="video-api-reference-caption"]') + .should('contain', 'Reference $0.034/video-s') + .and('contain', '$0.034–$0.047') + .and('contain', 'captured 2026-09-19') + .and('contain', 'MiniMax Design'); + cy.get('[data-testid="video-api-price"]').clear().type('0.05'); + // $0.400 per clip; 5.9732 × $0.400 − $1.22 = $1.17 profit. + kpi('h200').should('contain', '$0.400').and('contain', '$1.17'); + cy.location('search').should('contain', 'v_api=0.05'); + cy.contains('button', 'Table').click(); + cy.get('[data-testid="data-table-preset-all"]').click(); + cy.get('[data-testid="video-points-table"] thead') + .should('contain', 'Revenue per GPU-hour at API list price') + .and('contain', 'API list price ÷ TCO cost per video (Owning at Large Hyperscaler Volume)'); + // H200 row: revenue 5.9732 × $0.400 = $2.39; multiple $0.400 ÷ $0.2042 = 1.96. + cy.contains('[data-testid="video-points-table"] tbody tr', 'H200') + .should('contain', '$2.39') + .and('contain', '1.96'); + cy.get('[data-testid="video-api-price-reset"]').click(); + cy.contains('[data-testid="video-points-table"] tbody tr', 'H200').should('contain', '$1.62'); + cy.location('search').should('not.contain', 'v_api'); + }); + it('keeps negative profit per GPU-hour on the canvas on the allocated basis', () => { + mount('/video', '?v_y=profitPerGpuHour&v_basis=allocated'); + // H100 (−$0.44) and B200 (−$0.16) bill idle boards; only H200 (+$0.40) stays positive. + points().should('have.length', 3); + cy.get('[data-testid="video-hardware-chart"] .y-axis .tick text').should(($ticks) => { + const labels = [...$ticks].map((el) => el.textContent ?? ''); + expect(labels.some((label) => /^[-−]/u.test(label))).to.equal(true); + }); + cy.get('[data-testid="video-chart-card"]').should( + 'contain', + 'Profit per GPU-hour, API list price − TCO (Owning at Large Hyperscaler Volume)', + ); + kpi('h100').should('contain', '-$0.44'); + kpi('b200').should('contain', '-$0.16'); + }); it('shows placeholders while loading instead of a false "Not measured"', () => { cy.intercept('GET', '/api/video-runs?format=history&page=1', (req) => { req.reply({ fixture: 'api/video-history.json', delay: 800 }); @@ -131,6 +174,10 @@ describe('Video hardware dashboard (retained fixture)', () => { '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', ); kpi('mi355x').should('contain', '未测得'); - cy.get('[data-testid="video-config-bar"]').should('contain', '成本档位'); + kpi('h200').should('contain', 'API 标价 / 条视频').and('contain', '利润 / GPU 小时'); + cy.get('[data-testid="video-config-bar"]') + .should('contain', '成本档位') + .and('contain', 'API 价格参考($/视频秒)') + .and('contain', '采集于 2026-09-19'); }); }); From 77e62577820261c9be1fbd926919b10392744ab9 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:55:41 -0700 Subject: [PATCH 022/104] fix: cap the video chart y axis at break-even when every hardware loses money MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The y domain fell back to a ceiling of 1 whenever the true ceiling was 0, so any all-negative profit view (default price and basis, Rent - 3 Year Commit tier) drew the three losses in the bottom third of the canvas under an empty band. Move the padding into zeroAnchoredDomain(), which pads past the data anchored at zero and reserves the unit fallback for no data, and cover the all-negative case in Vitest and in the dashboard component spec (the spec fails on the old chart code with "expected 1 to equal 0"). 中文:当所有硬件都亏损时,将视频图表的 y 轴上限固定在盈亏平衡点。此前只要真实上限为 0, y 轴就会退回到 1 的合成上限,把三个亏损点挤在画布底部;现改为 zeroAnchoredDomain() 以零为锚点外扩,仅在没有数据时使用单位区间,并在 Vitest 与仪表板组件测试中覆盖全负值场景。 --- .../cypress/component/video-dashboard.cy.tsx | 14 ++++++++ .../video-benchmark/VideoHardwareChart.tsx | 17 ++++++--- .../video-benchmark/chart-domain.test.ts | 35 +++++++++++++++++++ .../video-benchmark/chart-domain.ts | 16 +++++++++ 4 files changed, 78 insertions(+), 4 deletions(-) create mode 100644 packages/app/src/components/video-benchmark/chart-domain.test.ts create mode 100644 packages/app/src/components/video-benchmark/chart-domain.ts diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index 0fc71ba50..9a2a1d6c1 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -150,6 +150,20 @@ describe('Video hardware dashboard (retained fixture)', () => { kpi('h100').should('contain', '-$0.44'); kpi('b200').should('contain', '-$0.16'); }); + it('caps the y axis at break-even when every hardware loses money at the rental tier', () => { + mount('/video', '?v_y=profitPerGpuHour&v_tier=r'); + // $0.034/video-s on the participating basis against $2.00/$2.90/$3.70 GPU-hr: H100 −$0.54, H200 −$1.28, B200 −$0.56. + points().should('have.length', 3); + kpi('h100').should('contain', '-$0.54'); + kpi('h200').should('contain', '-$1.28'); + kpi('b200').should('contain', '-$0.56'); + cy.get('[data-testid="video-hardware-chart"] .y-axis .tick text').should(($ticks) => { + const values = [...$ticks].map((el) => Number((el.textContent ?? '').replace('−', '-'))); + // Regression: a zero ceiling used to fall back to 1, leaving the top half of the canvas empty. + expect(Math.max(...values)).to.equal(0); + expect(Math.min(...values)).to.be.below(-1.2); + }); + }); it('shows placeholders while loading instead of a false "Not measured"', () => { cy.intercept('GET', '/api/video-runs?format=history&page=1', (req) => { req.reply({ fixture: 'api/video-history.json', delay: 800 }); diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx index 74f4ea24d..de17f945b 100644 --- a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -6,6 +6,7 @@ import type { ContinuousScale } from '@/lib/d3-chart/types'; import { CHART_TYPE, px } from '@/lib/d3-chart/typography'; import { useLocale } from '@/lib/use-locale'; import { escapeHtml } from '@/lib/utils'; +import { zeroAnchoredDomain } from './chart-domain'; import { layoutLabel } from './deployment'; import { formatMetric, metricLabel, type VideoPoint } from './metrics'; import { plotVideoPoints, type PlottedVideoPoint } from './plot'; @@ -110,8 +111,16 @@ export default function VideoHardwareChart({ .attr('pointer-events', 'none') .text(pointLabel); }; - const maxX = Math.max(0, ...plotted.map((p) => p.x)); - const maxY = Math.max(0, ...plotted.map((p) => p.y)); + const xDomain = zeroAnchoredDomain( + plotted.map((p) => p.x), + 1.12, + ); + // Profit per GPU-hour goes negative when the API price misses the tier cost: those + // points stay on the canvas, and when every hardware loses money the top is break-even. + const yDomain = zeroAnchoredDomain( + plotted.map((p) => p.y), + 1.15, + ); const caption = [ multiLayout ? s.multi : s.single(layouts || '—'), state.frontier ? (global.length > 1 ? s.global : s.globalOne) : null, @@ -126,8 +135,8 @@ export default function VideoHardwareChart({ margin={{ top: 20, right: 28, bottom: 80, left: 85 }} watermark="logo" transitionDuration={0} - xScale={{ type: 'linear', domain: [0, maxX * 1.12 || 1], nice: true }} - yScale={{ type: 'linear', domain: [0, maxY * 1.15 || 1], nice: true }} + xScale={{ type: 'linear', domain: xDomain, nice: true }} + yScale={{ type: 'linear', domain: yDomain, nice: true }} xAxis={{ label: xLabel, tickCount: 6 }} yAxis={{ label: yLabel, tickCount: 6 }} layers={[ diff --git a/packages/app/src/components/video-benchmark/chart-domain.test.ts b/packages/app/src/components/video-benchmark/chart-domain.test.ts new file mode 100644 index 000000000..578de9fb7 --- /dev/null +++ b/packages/app/src/components/video-benchmark/chart-domain.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { zeroAnchoredDomain } from './chart-domain'; + +// Profit per GPU-hour for the retained H100/H200/B200 C1 cells at $0.034/video-s +// (cypress/fixtures/api/video-history.json, asserted per point in metrics.test.ts): +// every hardware loses money at the 3-year rental tier ($2.00/$2.90/$3.70 GPU-hr) +// on the participating basis, while on the allocated basis at the hyperscaler +// tier only H200 stays positive. +const RENTAL_LOSSES = [-0.5382, -1.2753, -0.5649]; +const ALLOCATED_HYPERSCALER = [-0.4391, 0.4047, -0.16]; +// Videos per $1 TCO at the hyperscaler tier: H100, H200, B200. +const VIDEOS_PER_DOLLAR = [4.593, 4.896, 6.663]; + +describe('zeroAnchoredDomain', () => { + it('pads positive-only data above zero', () => { + const [lo, hi] = zeroAnchoredDomain(VIDEOS_PER_DOLLAR, 1.15); + expect(lo).toBe(0); + expect(hi).toBeCloseTo(6.663 * 1.15, 6); + }); + it('caps all-negative data at break-even instead of a synthetic ceiling', () => { + // Regression: `maxY * 1.15 || 1` read a zero ceiling as missing data and pinned the top at 1. + const [lo, hi] = zeroAnchoredDomain(RENTAL_LOSSES, 1.15); + expect(hi).toBe(0); + expect(lo).toBeCloseTo(-1.2753 * 1.15, 6); + }); + it('pads both ends of mixed data', () => { + const [lo, hi] = zeroAnchoredDomain(ALLOCATED_HYPERSCALER, 1.15); + expect(lo).toBeCloseTo(-0.4391 * 1.15, 6); + expect(hi).toBeCloseTo(0.4047 * 1.15, 6); + }); + it('falls back to a unit domain only without data, or when every value is zero', () => { + expect(zeroAnchoredDomain([], 1.15)).toEqual([0, 1]); + expect(zeroAnchoredDomain([0, 0], 1.15)).toEqual([0, 1]); + }); +}); diff --git a/packages/app/src/components/video-benchmark/chart-domain.ts b/packages/app/src/components/video-benchmark/chart-domain.ts new file mode 100644 index 000000000..cc75aa311 --- /dev/null +++ b/packages/app/src/components/video-benchmark/chart-domain.ts @@ -0,0 +1,16 @@ +/** + * Linear axis domain for the hardware chart, anchored at zero and padded past + * the data by the `pad` multiplier so point labels clear the plot edges. + * Positive-only data spans [0, pad·max]; all-negative data — profit per + * GPU-hour when the API price misses every tier cost — spans [pad·min, 0], so + * break-even is the top edge rather than a synthetic ceiling above the data; + * mixed data pads both ends. Only empty data gets the unit domain that keeps + * the axes drawn, and all-zero data keeps that unit height instead of + * collapsing the scale to a single tick. + */ +export function zeroAnchoredDomain(values: readonly number[], pad: number): [number, number] { + if (values.length === 0) return [0, 1]; + const lo = Math.min(0, ...values) * pad; + const hi = Math.max(0, ...values) * pad; + return lo === hi ? [0, 1] : [lo, hi]; +} From 5bd497bd14e80b2cd7019eab8e2cdf9db28eeffe Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:35:14 -0700 Subject: [PATCH 023/104] feat: add video compare metric deltas and case pairing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增 VideoGenX 对比纯模块:按 VIDEO_METRICS 极性计算基线/候选的指标比值、变化百分比与优劣判断(缺失输入保持 null);从 serving cell 的测量记录提取 case 身份(prompt + seed,缺失时用 case_id)并按重复次数配对;从已存储产物解析 C1 记录与媒体 URL 映射。单测断言保留 fixture 的实测数值(B200 vs H100 P50 比值 ≈ 0.466,每条视频 TCO ≈ −31%)。 --- .../video-benchmark/compare.test.ts | 277 ++++++++++++++++++ .../src/components/video-benchmark/compare.ts | 223 ++++++++++++++ 2 files changed, 500 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/compare.test.ts create mode 100644 packages/app/src/components/video-benchmark/compare.ts diff --git a/packages/app/src/components/video-benchmark/compare.test.ts b/packages/app/src/components/video-benchmark/compare.test.ts new file mode 100644 index 000000000..3dbcbcb12 --- /dev/null +++ b/packages/app/src/components/video-benchmark/compare.test.ts @@ -0,0 +1,277 @@ +import { describe, expect, it } from 'vitest'; +import history from '../../../cypress/fixtures/api/video-history.json'; +import { at, type Json } from './bundle'; +import { + caseKey, + caseRecords, + comparablePoints, + compareMetrics, + compareSide, + pairCases, + type CaseRecord, +} from './compare'; +import type { VideoHistoryPage } from './history'; +import { videoPoints } from './points'; +import { servingCells } from './serving'; +import { servingFixture } from './serving.fixture'; +import type { StoredArtifact } from './stored'; + +// Retained H100/H200/B200 C1/C2/C4 observations of the 2026-09-09 campaign. +const points = videoPoints([history as unknown as VideoHistoryPage]); +const c1 = (key: string) => comparablePoints(points).find((p) => p.hardwareKey === key)!; +const opts = { tier: 'h', basis: 'participating' } as const; +const row = (rows: ReturnType, id: string) => + rows.find((item) => item.id === id)!; + +function set(value: Json, key: string, replacement: Json) { + if (value === null || typeof value !== 'object' || Array.isArray(value)) + throw new Error('Expected object'); + value[key] = replacement; +} + +// Same shape cypress/support/video-artifacts.ts publishes: synthetic contract data, never evidence. +function stored( + runId: string, + artifactId: number, + hardware: string, + requests = 4, + manifestSha = runId.padEnd(64, '0'), +): StoredArtifact { + const fixture = servingFixture(runId, hardware, requests); + fixture.documents.set('manifest.json', fixture.manifest); + fixture.documents.set('ci.json', fixture.ci); + fixture.checksums.set('manifest.json', manifestSha); + return { + storageVersion: 1, + runId, + artifact: { + id: artifactId, + name: `h3-video-${runId}-1`, + expired: false, + size_in_bytes: 1, + stored: true, + }, + sources: [ + { + id: runId, + documents: [...fixture.documents], + checksums: [...fixture.checksums], + texts: [], + assets: [...fixture.checksums.keys()] + .filter((path) => path.endsWith('.mp4')) + .map((path) => [ + path, + { url: `https://media.test/${path}`, downloadUrl: `https://media.test/${path}?d=1` }, + ]), + }, + ], + }; +} + +describe('comparablePoints', () => { + it('keeps one measured C1 cell per registry hardware, in registry order', () => { + expect(comparablePoints(points).map((p) => [p.hardwareKey, p.concurrency])).toEqual([ + ['b200', 1], + ['h200', 1], + ['h100', 1], + ]); + expect(comparablePoints(points.map((p) => ({ ...p, p50: null })))).toEqual([]); + }); +}); + +describe('compareMetrics', () => { + it('reads B200 against H100 at C1: faster, cheaper, less energy, closer to its cap', () => { + const rows = compareMetrics(c1('h100'), c1('b200'), opts); + expect(rows.map((r) => r.id)).toEqual([ + 'p50Latency', + 'p90Latency', + 'videosPerGpuHour', + 'videosPerDollar', + 'dollarsPerVideo', + 'kjPerVideo', + 'powerPctCap', + ]); + const p50 = row(rows, 'p50Latency'); + expect(p50.baseline).toBeCloseTo(167.33, 2); + expect(p50.candidate).toBeCloseTo(77.94, 2); + expect(p50.ratio).toBeCloseTo(0.4658, 3); + expect(p50.deltaPercent).toBeCloseTo(-53.42, 1); + expect(p50.candidateBetter).toBe(true); + expect(row(rows, 'p90Latency').ratio).toBeCloseTo(0.4659, 3); + const rate = row(rows, 'videosPerGpuHour'); + expect(rate.baseline).toBeCloseTo(5.374, 3); + expect(rate.candidate).toBeCloseTo(11.526, 3); + expect(rate.ratio).toBeCloseTo(2.1447, 3); + expect(rate.candidateBetter).toBe(true); + const cost = row(rows, 'dollarsPerVideo'); + expect(cost.baseline).toBeCloseTo(0.2177, 4); + expect(cost.candidate).toBeCloseTo(0.1501, 4); + expect(cost.deltaPercent).toBeCloseTo(-31.06, 1); + expect(cost.candidateBetter).toBe(true); + expect(row(rows, 'videosPerDollar').deltaPercent).toBeCloseTo(45.05, 1); + const energy = row(rows, 'kjPerVideo'); + expect(energy.baseline).toBeCloseTo(431.1, 1); + expect(energy.candidate).toBeCloseTo(301.1, 1); + expect(energy.candidateBetter).toBe(true); + const power = row(rows, 'powerPctCap'); + expect(power.baseline).toBeCloseTo(91.95, 2); + expect(power.candidate).toBeCloseTo(96.4, 2); + expect(power.candidateBetter).toBe(true); + }); + it('flips the verdict when the slower hardware is the candidate', () => { + const rows = compareMetrics(c1('b200'), c1('h200'), opts); + expect(row(rows, 'p50Latency').ratio).toBeCloseTo(1.9323, 3); + expect(row(rows, 'p50Latency').candidateBetter).toBe(false); + expect(row(rows, 'dollarsPerVideo').candidateBetter).toBe(false); + }); + it('propagates missing inputs as null and treats equal values as neither better', () => { + const noPower = { ...c1('b200'), energyKj: null, enforcedLimitW: null, samples: 5 }; + const rows = compareMetrics(c1('h100'), noPower, opts); + expect(row(rows, 'kjPerVideo')).toEqual({ + id: 'kjPerVideo', + baseline: c1('h100').energyKj, + candidate: null, + ratio: null, + deltaPercent: null, + candidateBetter: null, + }); + expect(row(rows, 'powerPctCap').candidate).toBeNull(); + expect(row(rows, 'p90Latency').candidate).toBeNull(); + expect(row(rows, 'p50Latency').ratio).not.toBeNull(); + for (const same of compareMetrics(c1('h100'), c1('h100'), opts)) { + expect(same.ratio).toBe(1); + expect(same.deltaPercent).toBe(0); + expect(same.candidateBetter).toBeNull(); + } + }); + it('moves only the cost rows with the tier and GPU basis', () => { + const rows = compareMetrics(c1('h100'), c1('b200'), { tier: 'r', basis: 'allocated' }); + // H100 8 allocated: 2.00 / 2.687 videos per GPU-hr; B200 8 allocated: 3.70 / 5.763. + expect(row(rows, 'dollarsPerVideo').baseline).toBeCloseTo(0.7443, 3); + expect(row(rows, 'dollarsPerVideo').candidate).toBeCloseTo(0.642, 3); + expect(row(rows, 'videosPerGpuHour').baseline).toBeCloseTo(2.687, 3); + expect(row(rows, 'p50Latency').ratio).toBeCloseTo(0.4658, 3); + expect(row(rows, 'kjPerVideo').ratio).toBeCloseTo(0.6984, 3); + }); +}); + +describe('caseRecords', () => { + it('lists measurement requests with plan-derived case identity and safe media paths', () => { + const [cell] = servingCells(servingFixture('123', 'NVIDIA H200', 4)); + const records = caseRecords(cell); + expect(records).toHaveLength(4); + expect(records[0]).toEqual({ + slotId: 'measurement-r001-c001', + caseId: 'drum-taps', + prompt: 'A drummer taps a snare drum.', + seed: 11, + status: 'succeeded', + valid: true, + seconds: 120, + mediaPath: 'gpu/c1/baseline/artifacts/measurement-r001-c001.mp4', + }); + expect(records.map((r) => r.slotId)).not.toContain('warmup-001'); + }); + it('prefers the record’s own case fields and drops unsafe or missing paths', () => { + const fixture = servingFixture('123', 'NVIDIA H200', 4); + const run = fixture.documents.get('gpu/c1/baseline/run.json')!; + const first = at(run, 'records', 1); + set(first, 'case_id', 'own-case'); + set(first, 'prompt', 'Own prompt'); + set(first, 'seed', 99); + set(first, 'artifact_path', '../candidate/movie.mp4'); + const second = at(run, 'records', 2); + set(second, 'artifact_path', null); + set(second, 'submit_to_media_seconds', 0); + set(at(second, 'media'), 'valid', null); + const records = caseRecords({ run, runPath: 'gpu/c1/baseline/run.json' }); + expect(records[0]).toMatchObject({ + caseId: 'own-case', + prompt: 'Own prompt', + seed: 99, + mediaPath: null, + }); + expect(records[1]).toMatchObject({ mediaPath: null, seconds: null, valid: null }); + // A slot that names no plan case and carries no identity has nothing to pair on. + set(at(run, 'records', 3), 'slot_id', 'measurement-r003'); + expect(caseKey(caseRecords({ run, runPath: 'gpu/c1/baseline/run.json' })[2])).toBeNull(); + }); +}); + +const records = (requests: number) => + caseRecords(servingCells(servingFixture('1', 'NVIDIA H100 80GB HBM3', requests))[0]); + +describe('pairCases', () => { + it('pairs the k-th repetition of a prompt + seed on each side and counts the rest', () => { + const { pairs, unmatched } = pairCases(records(4), records(6)); + expect(pairs).toHaveLength(4); + expect(pairs.map((p) => p.repetition)).toEqual([0, 1, 2, 3]); + expect(pairs[1]).toMatchObject({ + caseId: 'drum-taps', + prompt: 'A drummer taps a snare drum.', + seed: 11, + baseline: { slotId: 'measurement-r002-c001' }, + candidate: { slotId: 'measurement-r002-c001' }, + }); + expect(unmatched).toEqual({ baseline: 0, candidate: 2 }); + expect(pairCases(records(6), records(4)).unmatched).toEqual({ baseline: 2, candidate: 0 }); + }); + it('never pairs different seeds or records without identity', () => { + const other = records(4).map((r): CaseRecord => ({ ...r, seed: 12 })); + expect(pairCases(records(4), other)).toEqual({ + pairs: [], + unmatched: { baseline: 4, candidate: 4 }, + }); + const blank = records(2).map((r): CaseRecord => ({ + ...r, + caseId: null, + prompt: null, + seed: null, + })); + expect(pairCases(blank, records(2)).unmatched).toEqual({ baseline: 2, candidate: 2 }); + }); + it('falls back to the case id only when prompt or seed is unknown', () => { + const [record] = records(1); + expect(caseKey(record)).toBe('ps\u000011\u0000A drummer taps a snare drum.'); + expect(caseKey({ ...record, seed: null })).toBe('id\u0000drum-taps'); + expect(caseKey({ ...record, prompt: null, caseId: null })).toBeNull(); + const byId = pairCases([{ ...record, seed: null }], [{ ...record, prompt: null }]); + expect(byId.pairs).toHaveLength(1); + expect(byId.pairs[0]).toMatchObject({ prompt: 'A drummer taps a snare drum.', seed: 11 }); + }); +}); + +describe('compareSide', () => { + const h100 = c1('h100'); + it('resolves the point’s C1 records and published media URLs from a stored artifact', () => { + const side = compareSide(stored(h100.runId, h100.artifactId, 'NVIDIA H100 80GB HBM3'), h100); + expect(side.records).toHaveLength(4); + expect(side.urls.get(side.records[0].mediaPath!)).toBe( + 'https://media.test/gpu/c1/baseline/artifacts/measurement-r001-c001.mp4', + ); + }); + it('prefers the source whose manifest SHA the point id names', () => { + const sha = h100.id.split(':')[0]; + const other = stored(h100.runId, h100.artifactId, 'NVIDIA H100 80GB HBM3', 3); + const exact = stored(h100.runId, h100.artifactId, 'NVIDIA H100 80GB HBM3', 6, sha); + const saved: StoredArtifact = { ...exact, sources: [...other.sources, ...exact.sources] }; + expect(compareSide(saved, h100).records).toHaveLength(6); + }); + it('rejects another run’s artifact and artifacts without the cell', () => { + expect(() => + compareSide(stored('999', h100.artifactId, 'NVIDIA H100 80GB HBM3'), h100), + ).toThrow('Stored artifact identity mismatch'); + expect(() => compareSide(stored(h100.runId, 1, 'NVIDIA H100 80GB HBM3'), h100)).toThrow( + 'Stored artifact identity mismatch', + ); + expect(() => + compareSide(stored(h100.runId, h100.artifactId, 'NVIDIA H100 80GB HBM3'), { + ...h100, + cell: 'c3', + }), + ).toThrow('Published artifact has no c3 cell'); + const fidelityOnly = stored(h100.runId, h100.artifactId, 'NVIDIA H100 80GB HBM3'); + fidelityOnly.sources[0].kind = 'fidelity'; + expect(() => compareSide(fidelityOnly, h100)).toThrow('Published artifact has no c1 cell'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/compare.ts b/packages/app/src/components/video-benchmark/compare.ts new file mode 100644 index 000000000..5d80b8614 --- /dev/null +++ b/packages/app/src/components/video-benchmark/compare.ts @@ -0,0 +1,223 @@ +import { at, number, rows, safePath, text } from './bundle'; +import { + metricValue, + VIDEO_METRICS, + type MetricId, + type MetricOptions, + type VideoPoint, +} from './metrics'; +import { latestVideoCells } from './points'; +import { servingCells, type ServingCell } from './serving'; +import { storedBundle, type StoredArtifact } from './stored'; + +/** Metrics the compare table shows, in row order. */ +export const COMPARE_METRICS = [ + 'p50Latency', + 'p90Latency', + 'videosPerGpuHour', + 'videosPerDollar', + 'dollarsPerVideo', + 'kjPerVideo', + 'powerPctCap', +] as const satisfies readonly MetricId[]; +export type CompareMetricId = (typeof COMPARE_METRICS)[number]; + +export interface CompareRow { + id: CompareMetricId; + baseline: number | null; + candidate: number | null; + /** candidate / baseline; null when either side is missing. */ + ratio: number | null; + /** (ratio − 1) × 100. */ + deltaPercent: number | null; + /** By registry polarity; null when not comparable or when both sides are equal. */ + candidateBetter: boolean | null; +} + +/** Measured C1 cells a reader can pick for comparison: registry hardware with a P50 (newest per hardware). */ +export function comparablePoints(points: VideoPoint[]): VideoPoint[] { + return latestVideoCells(points).filter( + (p) => p.concurrency === 1 && p.hardwareKey !== null && p.p50 !== null, + ); +} + +/** Side-by-side deltas; every null input stays null (never 0). */ +export function compareMetrics( + baseline: VideoPoint, + candidate: VideoPoint, + options: MetricOptions, +): CompareRow[] { + return COMPARE_METRICS.map((id) => { + const b = metricValue(baseline, id, options); + const c = metricValue(candidate, id, options); + if (b === null || c === null || !(b > 0)) + return { + id, + baseline: b, + candidate: c, + ratio: null, + deltaPercent: null, + candidateBetter: null, + }; + const ratio = c / b; + const lower = VIDEO_METRICS[id].polarity === 'lower'; + return { + id, + baseline: b, + candidate: c, + ratio, + deltaPercent: (ratio - 1) * 100, + candidateBetter: c === b ? null : lower ? c < b : c > b, + }; + }); +} + +/** One measurement request of a serving cell, with its case identity and retained media path. */ +export interface CaseRecord { + slotId: string; + caseId: string | null; + prompt: string | null; + seed: number | null; + status: string; + /** Technical validity of the decoded media; null when not recorded. */ + valid: boolean | null; + /** Submit-to-downloaded-media seconds for this request. */ + seconds: number | null; + /** Bundle-relative media path (`gpu/c1/baseline/artifacts/….mp4`); null without a safe path. */ + mediaPath: string | null; +} + +/** + * Measurement-phase records of a cell. A record's own `case_id`/`prompt`/`seed` + * win; older records without them fall back to the plan case their slot id + * names (`measurement-r001-c003` → third plan case). + */ +export function caseRecords(cell: Pick): CaseRecord[] { + const cases = rows(at(cell.run, 'plan', 'cases')); + const directory = cell.runPath.slice(0, cell.runPath.lastIndexOf('/') + 1); + return rows(at(cell.run, 'records')) + .filter((record) => at(record, 'phase') === 'measurement') + .map((record): CaseRecord => { + const slotId = text(at(record, 'slot_id')); + const index = /-c(?\d+)$/u.exec(slotId)?.groups?.n; + const planCase = index === undefined ? null : (cases[Number(index) - 1] ?? null); + const seconds = number(at(record, 'submit_to_media_seconds')); + const valid = at(record, 'media', 'valid'); + let mediaPath: string | null = null; + try { + const relative = text(at(record, 'artifact_path')); + if (relative) mediaPath = safePath(`${directory}${safePath(relative)}`); + } catch { + // Unsafe manifest paths never become media URLs. + } + return { + slotId, + caseId: text(at(record, 'case_id')) || text(at(planCase, 'case_id')) || null, + prompt: text(at(record, 'prompt')) || text(at(planCase, 'prompt')) || null, + seed: number(at(record, 'seed')) ?? number(at(planCase, 'seed')), + status: text(at(record, 'status')), + valid: typeof valid === 'boolean' ? valid : null, + seconds: seconds !== null && seconds > 0 ? seconds : null, + mediaPath, + }; + }); +} + +export interface CasePair { + /** Case identity shared by both records. */ + key: string; + caseId: string | null; + prompt: string | null; + seed: number | null; + /** 0-based repetition among pairs with the same identity. */ + repetition: number; + baseline: CaseRecord; + candidate: CaseRecord; +} + +/** Identity used for pairing: prompt + seed when known, else the plan case id. */ +export function caseKey(record: Pick): string | null { + if (record.prompt !== null && record.seed !== null) + return `ps\u0000${record.seed}\u0000${record.prompt}`; + return record.caseId === null ? null : `id\u0000${record.caseId}`; +} + +/** + * Match the k-th repetition of a case on one side with the k-th on the other, + * in baseline order. Records without any case identity, and repetitions the + * other side lacks, are counted rather than invented. + */ +export function pairCases( + baseline: CaseRecord[], + candidate: CaseRecord[], +): { pairs: CasePair[]; unmatched: { baseline: number; candidate: number } } { + const queues = new Map(); + let unmatchedCandidate = 0; + for (const record of candidate) { + const key = caseKey(record); + if (key === null) unmatchedCandidate++; + else queues.set(key, [...(queues.get(key) ?? []), record]); + } + const pairs: CasePair[] = []; + const repetitions = new Map(); + let unmatchedBaseline = 0; + for (const record of baseline) { + const key = caseKey(record); + const match = key === null ? undefined : queues.get(key)?.shift(); + if (key === null || match === undefined) { + unmatchedBaseline++; + continue; + } + const repetition = repetitions.get(key) ?? 0; + repetitions.set(key, repetition + 1); + pairs.push({ + key, + caseId: record.caseId ?? match.caseId, + prompt: record.prompt ?? match.prompt, + seed: record.seed ?? match.seed, + repetition, + baseline: record, + candidate: match, + }); + } + for (const queue of queues.values()) unmatchedCandidate += queue.length; + return { pairs, unmatched: { baseline: unmatchedBaseline, candidate: unmatchedCandidate } }; +} + +/** Case records of one dashboard point plus the published URL of each media path. */ +export interface CompareSide { + records: CaseRecord[]; + urls: Map; +} + +/** + * Resolve a dashboard point inside its stored artifact (`format=media`): the + * observation source whose manifest SHA the point id names, else the first + * source that publishes the point's cell. Contract violations throw, as in the + * run views; nothing is synthesised for a missing cell. + */ +export function compareSide(saved: StoredArtifact, point: VideoPoint): CompareSide { + if ( + saved.storageVersion !== 1 || + saved.runId !== point.runId || + saved.artifact.id !== point.artifactId + ) + throw new Error('Stored artifact identity mismatch'); + const wanted = point.id.split(':')[0]; + const cellId = point.cell ?? (point.concurrency === null ? 'c1' : `c${point.concurrency}`); + let fallback: CompareSide | null = null; + for (const source of saved.sources) { + if (source.kind) continue; + const bundle = storedBundle(source); + const cell = servingCells(bundle).find((item) => item.id === cellId); + if (!cell) continue; + const side: CompareSide = { + records: caseRecords(cell), + urls: new Map(source.assets.map(([path, asset]) => [path, asset.url])), + }; + if (bundle.manifestSha256 === wanted) return side; + fallback ??= side; + } + if (fallback === null) throw new Error(`Published artifact has no ${cellId} cell`); + return fallback; +} From dccdf037f0c0b0a9d0151efe5ce92dce78b1a2f3 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:35:24 -0700 Subject: [PATCH 024/104] feat: add compare URL selection for baseline, candidate and case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增 v_base / v_cand / v_case 参数的读取、写回与默认值解析:默认基线为 P50 最高(最慢)的已实测 C1 硬件,候选为最低(最快);未知硬件回退到默认值,默认值不写入 URL;不改动 video-url-state.ts。含单测。 --- .../video-benchmark/compare-url-state.test.ts | 85 +++++++++++++++++++ .../video-benchmark/compare-url-state.ts | 80 +++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/compare-url-state.test.ts create mode 100644 packages/app/src/components/video-benchmark/compare-url-state.ts diff --git a/packages/app/src/components/video-benchmark/compare-url-state.test.ts b/packages/app/src/components/video-benchmark/compare-url-state.test.ts new file mode 100644 index 000000000..4898d0216 --- /dev/null +++ b/packages/app/src/components/video-benchmark/compare-url-state.test.ts @@ -0,0 +1,85 @@ +import { describe, expect, it } from 'vitest'; +import history from '../../../cypress/fixtures/api/video-history.json'; +import { + DEFAULT_VIDEO_COMPARE_SELECTION, + defaultComparePair, + readVideoCompareSelection, + resolveCompareSelection, + writeVideoCompareSelection, +} from './compare-url-state'; +import type { VideoHistoryPage } from './history'; +import { videoPoints } from './points'; + +const points = videoPoints([history as unknown as VideoHistoryPage]); + +describe('video compare URL selection', () => { + it('defaults to the slowest measured C1 hardware as baseline and the fastest as candidate', () => { + expect(defaultComparePair(points)).toEqual({ baseline: 'h100', candidate: 'b200' }); + expect(defaultComparePair(points.filter((p) => p.hardwareKey === 'h200'))).toEqual({ + baseline: 'h200', + candidate: null, + }); + expect(defaultComparePair([])).toEqual({ baseline: null, candidate: null }); + }); + it('reads v_base, v_cand and v_case, ignoring malformed values', () => { + expect(readVideoCompareSelection('')).toEqual(DEFAULT_VIDEO_COMPARE_SELECTION); + expect(readVideoCompareSelection('?v_base=h200&v_cand=h100&v_case=3&v_x=genSpeed')).toEqual({ + baseline: 'h200', + candidate: 'h100', + caseIndex: 3, + }); + expect(readVideoCompareSelection('?v_base=H200&v_cand=h1%20x&v_case=-1')).toEqual( + DEFAULT_VIDEO_COMPARE_SELECTION, + ); + expect(readVideoCompareSelection('?v_case=12345').caseIndex).toBe(0); + }); + it('resolves choices against measured points and falls back per side', () => { + const resolved = resolveCompareSelection( + { baseline: 'h200', candidate: null, caseIndex: 2 }, + points, + ); + expect(resolved.baseline?.hardwareKey).toBe('h200'); + expect(resolved.candidate?.hardwareKey).toBe('b200'); + expect(resolved.caseIndex).toBe(2); + const unknown = resolveCompareSelection( + { baseline: 'mi355x', candidate: 'gb200', caseIndex: 0 }, + points, + ); + expect(unknown.baseline?.hardwareKey).toBe('h100'); + expect(unknown.candidate?.hardwareKey).toBe('b200'); + expect(resolveCompareSelection(DEFAULT_VIDEO_COMPARE_SELECTION, [])).toEqual({ + baseline: null, + candidate: null, + caseIndex: 0, + }); + }); + it('writes only non-default params, keeps unrelated params and leaves the input alone', () => { + const defaults = defaultComparePair(points); + const url = new URL('https://x.test/video?v_tier=r&run=1'); + const out = writeVideoCompareSelection( + url, + { baseline: 'h100', candidate: 'h200', caseIndex: 2 }, + defaults, + ); + expect(out.searchParams.get('v_tier')).toBe('r'); + expect(out.searchParams.get('run')).toBe('1'); + expect(out.searchParams.has('v_base')).toBe(false); + expect(out.searchParams.get('v_cand')).toBe('h200'); + expect(out.searchParams.get('v_case')).toBe('2'); + expect(url.searchParams.has('v_cand')).toBe(false); + const swapped = writeVideoCompareSelection( + out, + { baseline: 'b200', candidate: 'h100', caseIndex: 0 }, + defaults, + ); + expect(swapped.searchParams.get('v_base')).toBe('b200'); + expect(swapped.searchParams.get('v_cand')).toBe('h100'); + expect(swapped.searchParams.has('v_case')).toBe(false); + const reset = writeVideoCompareSelection( + swapped, + { baseline: 'h100', candidate: 'b200', caseIndex: 0 }, + defaults, + ); + expect([...reset.searchParams.keys()]).toEqual(['v_tier', 'run']); + }); +}); diff --git a/packages/app/src/components/video-benchmark/compare-url-state.ts b/packages/app/src/components/video-benchmark/compare-url-state.ts new file mode 100644 index 000000000..f1f0e3888 --- /dev/null +++ b/packages/app/src/components/video-benchmark/compare-url-state.ts @@ -0,0 +1,80 @@ +import { comparablePoints } from './compare'; +import type { VideoPoint } from './metrics'; + +/** Reader choices for the compare panel; a null hardware means "the default for the loaded points". */ +export interface VideoCompareSelection { + baseline: string | null; + candidate: string | null; + /** 0-based index into the matched case list. */ + caseIndex: number; +} + +export interface ComparePair { + baseline: string | null; + candidate: string | null; +} + +export const DEFAULT_VIDEO_COMPARE_SELECTION: VideoCompareSelection = { + baseline: null, + candidate: null, + caseIndex: 0, +}; + +const HARDWARE_KEY = /^[a-z0-9][a-z0-9-]{0,31}$/u; +const CASE_INDEX = /^\d{1,4}$/u; + +/** Slowest measured C1 hardware (highest P50) as baseline, fastest as candidate. */ +export function defaultComparePair(points: VideoPoint[]): ComparePair { + const measured = comparablePoints(points); + if (measured.length < 2) return { baseline: measured[0]?.hardwareKey ?? null, candidate: null }; + let slowest = measured[0]; + let fastest = measured[0]; + for (const point of measured) { + if ((point.p50 ?? 0) > (slowest.p50 ?? 0)) slowest = point; + if ((point.p50 ?? 0) < (fastest.p50 ?? 0)) fastest = point; + } + return { baseline: slowest.hardwareKey, candidate: fastest.hardwareKey }; +} + +/** `v_base`, `v_cand` and `v_case` live beside the dashboard's other `v_` params; unknown values read as defaults. */ +export function readVideoCompareSelection(search: string): VideoCompareSelection { + const p = new URLSearchParams(search); + const key = (value: string | null) => (value !== null && HARDWARE_KEY.test(value) ? value : null); + const index = p.get('v_case'); + return { + baseline: key(p.get('v_base')), + candidate: key(p.get('v_cand')), + caseIndex: index !== null && CASE_INDEX.test(index) ? Number(index) : 0, + }; +} + +/** Resolve choices against the loaded points; a hardware without a measured C1 cell falls back to the default. */ +export function resolveCompareSelection( + selection: VideoCompareSelection, + points: VideoPoint[], +): { baseline: VideoPoint | null; candidate: VideoPoint | null; caseIndex: number } { + const measured = comparablePoints(points); + const defaults = defaultComparePair(points); + const find = (key: string | null) => + key === null ? null : (measured.find((point) => point.hardwareKey === key) ?? null); + return { + baseline: find(selection.baseline) ?? find(defaults.baseline), + candidate: find(selection.candidate) ?? find(defaults.candidate), + caseIndex: selection.caseIndex, + }; +} + +/** Returns a new URL with defaults omitted; the input URL is not mutated. */ +export function writeVideoCompareSelection( + url: URL, + resolved: VideoCompareSelection, + defaults: ComparePair, +): URL { + const out = new URL(url); + const set = (key: string, value: string | null) => + value === null ? out.searchParams.delete(key) : out.searchParams.set(key, value); + set('v_base', resolved.baseline === defaults.baseline ? null : resolved.baseline); + set('v_cand', resolved.candidate === defaults.candidate ? null : resolved.candidate); + set('v_case', resolved.caseIndex > 0 ? String(resolved.caseIndex) : null); + return out; +} From 1a5116c705102c5b835c0da35266056c62cdf52f Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:35:35 -0700 Subject: [PATCH 025/104] feat: add arena-style VideoCompare panel with on-demand clips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:新增 VideoCompare 组件:基线/候选硬件选择(不足两种时禁用、选到对侧硬件时自动互换)、按 colorFor 着色的指标差异表与绿/红变化标签、按需读取两次运行的 StoredArtifact 并并排播放同一 prompt 与 seed 的视频、用例选择、加载/空/错误状态与脚注;中英双语文案;video_compare_* 埋点。Cypress 组件测试基于保留 fixture 与合成媒体产物。 --- .../cypress/component/video-compare.cy.tsx | 157 ++++++ .../video-benchmark/VideoCompare.tsx | 479 ++++++++++++++++++ 2 files changed, 636 insertions(+) create mode 100644 packages/app/cypress/component/video-compare.cy.tsx create mode 100644 packages/app/src/components/video-benchmark/VideoCompare.tsx diff --git a/packages/app/cypress/component/video-compare.cy.tsx b/packages/app/cypress/component/video-compare.cy.tsx new file mode 100644 index 000000000..884e3e33b --- /dev/null +++ b/packages/app/cypress/component/video-compare.cy.tsx @@ -0,0 +1,157 @@ +import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; +import type { VideoHistoryPage } from '@/components/video-benchmark/history'; +import type { MetricOptions, VideoPoint } from '@/components/video-benchmark/metrics'; +import { videoPoints } from '@/components/video-benchmark/points'; +import VideoCompare from '@/components/video-benchmark/VideoCompare'; +import history from '../fixtures/api/video-history.json'; +import { servingArtifact } from '../support/video-artifacts'; + +// Retained H100/H200/B200 C1/C2/C4 observations; clips come from synthetic stored artifacts. +const points = videoPoints([history as unknown as VideoHistoryPage]); +const c1 = (key: string) => points.find((p) => p.hardwareKey === key && p.concurrency === 1)!; +const COLORS: Record = { + h100: 'rgb(10, 20, 30)', + h200: 'rgb(40, 50, 60)', + b200: 'rgb(70, 80, 90)', +}; +const colorFor = (key: string) => COLORS[key] ?? 'rgb(0, 0, 0)'; +const OWNING_PARTICIPATING: MetricOptions = { tier: 'h', basis: 'participating' }; + +function media(point: VideoPoint, hardware: string, alias: string) { + cy.intercept( + 'GET', + `/api/video-runs?run=${point.runId}&artifact=${point.artifactId}&format=media`, + servingArtifact(Number(point.runId), point.artifactId, hardware), + ).as(alias); +} + +function mount( + pathname = '/video', + search = '', + data: VideoPoint[] = points, + options: MetricOptions = OWNING_PARTICIPATING, +) { + media(c1('h100'), 'NVIDIA H100 80GB HBM3', 'h100'); + media(c1('b200'), 'NVIDIA B200', 'b200'); + media(c1('h200'), 'NVIDIA H200', 'h200'); + cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); + cy.window().then((win) => + win.history.replaceState(null, '', `${win.location.pathname}${search}`), + ); + cy.mount( + + + , + ); +} +const table = () => cy.get('[data-testid="video-compare-table"]'); +const metricRow = (id: string) => table().find(`tbody tr[data-metric="${id}"]`); +const pick = (label: string, option: string) => { + cy.get(`[role="combobox"][aria-label="${label}"]`).click(); + cy.contains('[role="option"]', option).click(); +}; + +describe('Video compare panel (retained fixture)', () => { + it('defaults to the slowest hardware as baseline and the fastest as candidate', () => { + mount(); + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'B200'); + metricRow('p50Latency') + .should('contain', '167.3') + .and('contain', '77.9') + .and('contain', '-53.4%'); + metricRow('p50Latency').find('[data-tone="better"]').should('exist'); + metricRow('dollarsPerVideo') + .should('contain', '$0.218') + .and('contain', '$0.150') + .and('contain', '-31.1%'); + metricRow('powerPctCap').find('[data-tone="better"]').should('exist'); + cy.get('[data-testid="video-compare"]').should('contain', 'Owning at Large Hyperscaler Volume'); + cy.get('[data-testid="video-compare-clip"]').should('not.exist'); + cy.get('@h100.all').should('have.length', 0); + }); + it('loads both runs’ clips on demand and pairs the same prompt and seed', () => { + mount(); + cy.get('[data-testid="video-compare-load"]').click(); + cy.wait(['@h100', '@b200']); + cy.get('[data-testid="video-compare-clip"]').should('have.length', 2); + cy.get('[data-testid="video-compare-clip"] video').should('have.length', 2); + cy.get('[data-testid="video-compare-clip"][data-role="baseline"]') + .should('contain', 'H100') + .and('contain', 'time to video 120 s') + .find('video') + .should('have.attr', 'src') + .and('include', 'measurement-r001-c001.mp4'); + cy.get('[data-testid="video-compare-clip"][data-role="candidate"]').should('contain', 'B200'); + cy.get('[role="combobox"][aria-label="Case (prompt · seed)"]').should( + 'contain', + 'A drummer taps a snare drum. · seed 11 · #1', + ); + pick('Case (prompt · seed)', '#3'); + cy.get('[data-testid="video-compare-clip"] video').each(($video) => + expect($video.attr('src')).to.include('measurement-r003-c001.mp4'), + ); + cy.location('search').should('contain', 'v_case=2'); + cy.get('[data-testid="video-compare"]').should('contain', 'retained CI outputs'); + }); + it('updates the delta table and the URL when the candidate changes, without refetching', () => { + mount(); + cy.get('[data-testid="video-compare-load"]').click(); + cy.wait(['@h100', '@b200']); + pick('Candidate', 'H200'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H200'); + metricRow('p50Latency').should('contain', '150.6').and('contain', '-10%'); + metricRow('dollarsPerVideo').should('contain', '$0.204'); + cy.location('search').should('contain', 'v_cand=h200'); + // H200 clips were never fetched: the pane returns to the on-demand button. + cy.get('[data-testid="video-compare-load"]').should('exist'); + cy.get('@h200.all').should('have.length', 0); + // Choosing the baseline's hardware as candidate swaps the two sides. + pick('Candidate', 'H100'); + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H200'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); + metricRow('p50Latency').find('[data-tone="worse"]').should('exist'); + }); + it('restores v_base and v_cand from the URL and reports a failed media read', () => { + mount('/video', '?v_base=h200&v_cand=h100'); + cy.intercept( + 'GET', + `/api/video-runs?run=${c1('h200').runId}&artifact=${c1('h200').artifactId}&format=media`, + { statusCode: 404, body: { error: 'gone' } }, + ).as('missing'); + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H200'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); + cy.get('[data-testid="video-compare-load"]').click(); + cy.wait('@missing'); + cy.get('[data-testid="video-compare-clips"] [role="alert"]') + .should('contain', 'Could not load clips') + .and('contain', 'HTTP 404'); + cy.contains('button', 'Retry').should('exist'); + }); + it('disables the selects when fewer than two hardware are measured', () => { + mount( + '/video', + '', + points.filter((p) => p.hardwareKey === 'h200'), + ); + cy.get('[data-testid="video-compare-controls"]').should('have.attr', 'disabled'); + cy.get('[data-testid="video-compare"]').should( + 'contain', + 'Comparison needs at least two measured hardware.', + ); + cy.get('[data-testid="video-compare-table"]').should('not.exist'); + }); + it('renders Chinese copy on /zh/video', () => { + mount('/zh/video', '', points, { tier: 'r', basis: 'allocated' }); + cy.get('[data-testid="video-compare"]') + .should('contain', '基线') + .and('contain', '候选') + .and('contain', '加载视频') + .and('contain', '租赁 - 3 年承诺') + .and('contain', '已分配的 GPU'); + metricRow('p50Latency').should('contain', 'P50 出片时间'); + cy.get('[data-testid="video-compare-load"]').click(); + cy.wait(['@h100', '@b200']); + cy.get('[data-testid="video-compare-clip"]').first().should('contain', '出片时间 120 s'); + }); +}); diff --git a/packages/app/src/components/video-benchmark/VideoCompare.tsx b/packages/app/src/components/video-benchmark/VideoCompare.tsx new file mode 100644 index 000000000..fd77596e2 --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoCompare.tsx @@ -0,0 +1,479 @@ +'use client'; + +import { useEffect, useId, useMemo, useRef, useState } from 'react'; +import { Button } from '@/components/ui/button'; +import { Heading } from '@/components/ui/heading'; +import { track } from '@/lib/analytics'; +import { useLocale } from '@/lib/use-locale'; +import { + comparablePoints, + compareMetrics, + compareSide, + pairCases, + type CasePair, + type CompareRow, + type CompareSide, +} from './compare'; +import { + DEFAULT_VIDEO_COMPARE_SELECTION, + defaultComparePair, + readVideoCompareSelection, + resolveCompareSelection, + writeVideoCompareSelection, + type VideoCompareSelection, +} from './compare-url-state'; +import { hardwareLabel } from './hardware'; +import { + formatMetric, + TIER_LABELS, + VIDEO_METRICS, + type MetricOptions, + type VideoPoint, +} from './metrics'; +import type { StoredArtifact } from './stored'; +import VideoSelect from './VideoSelect'; + +const STRINGS = { + en: { + title: 'Compare', + subtitle: + 'Pick a baseline and a candidate hardware (measured C1 cells): metric deltas side by side, then the same generated case from both runs.', + baseline: 'Baseline', + candidate: 'Candidate', + needTwo: 'Comparison needs at least two measured hardware.', + metric: 'Metric', + change: 'Change', + assumptions: (tier: string, basis: string) => `Cost tier: ${tier} · GPU basis: ${basis}.`, + participating: 'participating GPUs', + allocated: 'allocated GPUs', + better: 'Candidate better', + worse: 'Candidate worse', + neutral: 'No difference', + load: 'Load clips', + loadHint: 'Fetches the retained clips of both runs on demand (two published artifacts).', + loading: 'Loading clips…', + error: 'Could not load clips', + retry: 'Retry', + case: 'Case (prompt · seed)', + noPairs: 'No case with the same prompt and seed exists in both runs.', + unmatched: (baseline: number, candidate: number) => + `${baseline} baseline and ${candidate} candidate records have no counterpart.`, + noClip: 'Clip not published for this record.', + mediaError: 'The clip could not be played.', + timeToVideo: 'time to video', + footnote: + 'Clips are the retained CI outputs of the same prompt and seed on each hardware; nothing is re-generated and nothing is voted on. Time to video is that record’s submit-to-downloaded-media seconds.', + }, + zh: { + title: '对比', + subtitle: + '选择基线与候选硬件(已实测的 C1 单元):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', + baseline: '基线', + candidate: '候选', + needTwo: '至少需要两种已实测的硬件才能对比。', + metric: '指标', + change: '变化', + assumptions: (tier: string, basis: string) => `成本分档:${tier} · GPU 口径:${basis}。`, + participating: '参与计算的 GPU', + allocated: '已分配的 GPU', + better: '候选更优', + worse: '候选更差', + neutral: '无差异', + load: '加载视频', + loadHint: '按需读取两次运行保留的视频(两个已发布产物)。', + loading: '正在加载视频…', + error: '无法加载视频', + retry: '重试', + case: '用例(prompt · seed)', + noPairs: '两次运行中没有 prompt 与 seed 均相同的用例。', + unmatched: (baseline: number, candidate: number) => + `${baseline} 条基线记录与 ${candidate} 条候选记录没有对应项。`, + noClip: '此记录的视频未发布。', + mediaError: '视频无法播放。', + timeToVideo: '出片时间', + footnote: + '视频为各硬件 CI 运行保留的原始输出,prompt 与 seed 相同;不重新生成,也不投票。出片时间为该记录从提交到下载完成的秒数。', + }, +}; +type Strings = (typeof STRINGS)['en']; + +const CHIP = { + better: 'border-emerald-600/40 bg-emerald-500/10 text-emerald-700 dark:text-emerald-300', + worse: 'border-red-600/40 bg-red-500/10 text-red-700 dark:text-red-300', + neutral: 'border-border text-muted-foreground', +} as const; + +const excerpt = (prompt: string, max = 60) => + prompt.length > max ? `${prompt.slice(0, max).trimEnd()}…` : prompt; + +function caseLabel(pair: CasePair, repeated: boolean): string { + const parts = [pair.prompt === null ? (pair.caseId ?? '—') : excerpt(pair.prompt)]; + if (pair.seed !== null) parts.push(`seed ${pair.seed}`); + if (repeated) parts.push(`#${pair.repetition + 1}`); + return parts.join(' · '); +} + +async function fetchSide(point: VideoPoint, signal: AbortSignal): Promise { + const response = await fetch( + `/api/video-runs?run=${encodeURIComponent(point.runId)}&artifact=${encodeURIComponent(String(point.artifactId))}&format=media`, + { signal }, + ); + if (!response.ok || response.status === 204) + throw new Error(`Published media HTTP ${response.status}`); + const saved = (await response.json()) as StoredArtifact; + return compareSide(saved, point); +} + +function DeltaChip({ row, s }: { row: CompareRow; s: Strings }) { + if (row.deltaPercent === null || row.ratio === null) + return —; + const tone = row.candidateBetter === null ? 'neutral' : row.candidateBetter ? 'better' : 'worse'; + return ( + + + {row.deltaPercent.toLocaleString('en-US', { + maximumFractionDigits: 1, + signDisplay: 'exceptZero', + })} + % + + + ×{row.ratio.toLocaleString('en-US', { maximumFractionDigits: 2 })} + + + ); +} + +/** + * Arena-style baseline vs candidate: measured C1 metric deltas, then the + * retained clips of the same prompt and seed from both CI runs. Media is fetched + * only on request, one stored artifact per side. + */ +export default function VideoCompare({ + points, + options, + colorFor, +}: { + points: VideoPoint[]; + options: MetricOptions; + colorFor: (hardwareKey: string) => string; +}) { + const locale = useLocale(); + const s = STRINGS[locale]; + const headingId = useId(); + const [selection, setSelection] = useState( + DEFAULT_VIDEO_COMPARE_SELECTION, + ); + const [sides, setSides] = useState>(() => new Map()); + const [pending, setPending] = useState(null); + const [failure, setFailure] = useState<{ key: string; message: string } | null>(null); + const [failedMedia, setFailedMedia] = useState>(() => new Set()); + const request = useRef(0); + const download = useRef(null); + useEffect(() => { + setSelection(readVideoCompareSelection(location.search)); + return () => download.current?.abort(); + }, []); + + const measured = useMemo(() => comparablePoints(points), [points]); + const defaults = useMemo(() => defaultComparePair(points), [points]); + const { baseline, candidate, caseIndex } = resolveCompareSelection(selection, points); + const pairKey = baseline && candidate ? `${baseline.id}|${candidate.id}` : null; + const rows = baseline && candidate ? compareMetrics(baseline, candidate, options) : []; + const loaded = + baseline && candidate && sides.has(baseline.id) && sides.has(candidate.id) + ? { baseline: sides.get(baseline.id)!, candidate: sides.get(candidate.id)! } + : null; + const matched = loaded ? pairCases(loaded.baseline.records, loaded.candidate.records) : null; + const pairs = matched?.pairs ?? []; + const index = Math.min(caseIndex, Math.max(pairs.length - 1, 0)); + const current = pairs[index]; + const repeated = new Set( + pairs + .filter((pair) => pairs.some((other) => other !== pair && other.key === pair.key)) + .map((pair) => pair.key), + ); + const hardwareOptions = measured.map((point) => ({ + value: point.hardwareKey ?? '', + label: hardwareLabel(point.hardwareKey ?? ''), + })); + + const update = (patch: Partial) => { + const next = { ...selection, ...patch }; + setSelection(next); + const resolved = resolveCompareSelection(next, points); + history.replaceState( + null, + '', + writeVideoCompareSelection( + new URL(location.href), + { + baseline: resolved.baseline?.hardwareKey ?? null, + candidate: resolved.candidate?.hardwareKey ?? null, + caseIndex: next.caseIndex, + }, + defaults, + ), + ); + }; + const choose = (side: 'baseline' | 'candidate', hardware: string) => { + const own = side === 'baseline' ? baseline : candidate; + const other = side === 'baseline' ? candidate : baseline; + const patch: Partial = { [side]: hardware }; + // Picking the other side's hardware swaps the two instead of comparing a hardware with itself. + if (other?.hardwareKey === hardware && own) + patch[side === 'baseline' ? 'candidate' : 'baseline'] = own.hardwareKey; + update(patch); + track(`video_compare_${side}_changed`, { hardware }); + }; + const load = async () => { + if (!baseline || !candidate || pairKey === null) return; + download.current?.abort(); + const controller = new AbortController(); + download.current = controller; + const attempt = ++request.current; + setPending(pairKey); + setFailure(null); + track('video_compare_clips_load', { + baseline: baseline.hardwareKey, + candidate: candidate.hardwareKey, + }); + try { + const missing = [baseline, candidate].filter((point) => !sides.has(point.id)); + const fetched = await Promise.all( + missing.map( + async (point) => [point.id, await fetchSide(point, controller.signal)] as const, + ), + ); + if (attempt !== request.current) return; + setSides((old) => new Map([...old, ...fetched])); + } catch (error) { + if (controller.signal.aborted) return; + setFailure({ key: pairKey, message: error instanceof Error ? error.message : String(error) }); + } finally { + if (attempt === request.current) setPending(null); + } + }; + + return ( +
+
+ + {s.title} + +

{s.subtitle}

+
+
+ choose('baseline', value)} + options={hardwareOptions} + /> + choose('candidate', value)} + options={hardwareOptions} + /> +
+ {measured.length < 2 && ( +

+ {s.needTwo} +

+ )} + {baseline && candidate && ( + <> +
+ + + + + {( + [ + ['baseline', baseline], + ['candidate', candidate], + ] as const + ).map(([role, point]) => ( + + ))} + + + + + {rows.map((row) => ( + + + + + + + ))} + +
+ {s.metric} + + + + {hardwareLabel(point.hardwareKey ?? '')} + · {s[role]} + + + {s.change} +
+ {locale === 'zh' + ? VIDEO_METRICS[row.id].labelZh + : VIDEO_METRICS[row.id].label} + {formatMetric(row.baseline, row.id)}{formatMetric(row.candidate, row.id)} + +
+
+

+ {s.assumptions( + TIER_LABELS[options.tier][locale], + options.basis === 'allocated' ? s.allocated : s.participating, + )} +

+
+ {loaded && matched ? ( + pairs.length === 0 || !current ? ( +

+ {s.noPairs} +

+ ) : ( + <> +
+ { + const next = Number(value); + update({ caseIndex: next }); + track('video_compare_case_changed', { + index: next, + seed: pairs[next]?.seed ?? null, + }); + }} + options={pairs.map((pair, i) => ({ + value: String(i), + label: caseLabel(pair, repeated.has(pair.key)), + }))} + /> +
+
+ {( + [ + ['baseline', baseline, current.baseline, loaded.baseline], + ['candidate', candidate, current.candidate, loaded.candidate], + ] as const + ).map(([role, point, record, side]) => { + const url = + record.mediaPath === null ? undefined : side.urls.get(record.mediaPath); + const label = hardwareLabel(point.hardwareKey ?? ''); + return ( +
+ {url ? ( +
+ ); + })} +
+ {(matched.unmatched.baseline > 0 || matched.unmatched.candidate > 0) && ( +

+ {s.unmatched(matched.unmatched.baseline, matched.unmatched.candidate)} +

+ )} + + ) + ) : pending === pairKey ? ( +

+ {s.loading} +

+ ) : failure?.key === pairKey ? ( +
+

+ {s.error}: {failure.message} +

+ +
+ ) : ( +
+ + {s.loadHint} +
+ )} +

{s.footnote}

+
+ + )} +
+ ); +} From 8a0a092e7fe25a6225dc973b9516a37ea4c10fdd Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:56:07 -0700 Subject: [PATCH 026/104] fix: skip malformed sources when resolving compare clips MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit compareSide read every observation source of the stored artifact inside one loop with no per-source isolation, so a single source failing storedBundle's identity check or servingCells' contract asserts aborted the lookup even when a later source matched the point's manifest SHA or carried its cell. Isolate each source as videoHistoryEntry does: skip a source that throws, keep the first error, and rethrow it only when no source carries the cell, so an unreadable artifact is never reported as one that lacks the cell. Cover both throw sites on the SHA-preferred and first-with-cell paths. 中文:修复 compareSide 在解析对比视频时被单个损坏的 source 中断的问题:逐个隔离 source 的契约错误并跳过,仅在没有任何 source 包含该 cell 时才重新抛出首个错误;补充两条单元测试覆盖两处抛错点。 --- .../video-benchmark/compare.test.ts | 58 ++++++++++++++++++- .../src/components/video-benchmark/compare.ts | 55 +++++++++++++----- 2 files changed, 98 insertions(+), 15 deletions(-) diff --git a/packages/app/src/components/video-benchmark/compare.test.ts b/packages/app/src/components/video-benchmark/compare.test.ts index 3dbcbcb12..99a5ac52a 100644 --- a/packages/app/src/components/video-benchmark/compare.test.ts +++ b/packages/app/src/components/video-benchmark/compare.test.ts @@ -14,7 +14,7 @@ import type { VideoHistoryPage } from './history'; import { videoPoints } from './points'; import { servingCells } from './serving'; import { servingFixture } from './serving.fixture'; -import type { StoredArtifact } from './stored'; +import type { StoredArtifact, StoredSource } from './stored'; // Retained H100/H200/B200 C1/C2/C4 observations of the 2026-09-09 campaign. const points = videoPoints([history as unknown as VideoHistoryPage]); @@ -29,6 +29,10 @@ function set(value: Json, key: string, replacement: Json) { value[key] = replacement; } +/** The stored copy of one bundle document; mutating it corrupts only that source. */ +const document = (source: StoredSource, path: string): Json => + source.documents.find(([name]) => name === path)?.[1] ?? null; + // Same shape cypress/support/video-artifacts.ts publishes: synthetic contract data, never evidence. function stored( runId: string, @@ -274,4 +278,56 @@ describe('compareSide', () => { fidelityOnly.sources[0].kind = 'fidelity'; expect(() => compareSide(fidelityOnly, h100)).toThrow('Published artifact has no c1 cell'); }); + it('skips a malformed source so a readable sibling still resolves', () => { + const sha = h100.id.split(':')[0]; + const hardware = 'NVIDIA H100 80GB HBM3'; + // Manifest names another run, so storedBundle rejects the source. + const misbundled = stored(h100.runId, h100.artifactId, hardware, 3); + set(document(misbundled.sources[0], 'manifest.json'), 'run_id', '999'); + // Manifest mode no longer matches CI, so servingCells rejects the source. + const inconsistent = stored(h100.runId, h100.artifactId, hardware, 5); + set(document(inconsistent.sources[0], 'manifest.json'), 'mode', 'fidelity'); + const corrupt = [...misbundled.sources, ...inconsistent.sources]; + // The readable sibling wins whether the point names its manifest SHA… + const exact = stored(h100.runId, h100.artifactId, hardware, 6, sha); + expect( + compareSide({ ...exact, sources: [...corrupt, ...exact.sources] }, h100).records, + ).toHaveLength(6); + // …or it is merely the first source with the cell, before or after the corrupt ones. + const other = stored(h100.runId, h100.artifactId, hardware, 4); + expect( + compareSide({ ...other, sources: [...corrupt, ...other.sources] }, h100).records, + ).toHaveLength(4); + expect( + compareSide({ ...other, sources: [...other.sources, ...corrupt] }, h100).records, + ).toHaveLength(4); + }); + it('rethrows the first source error only when no source publishes the cell', () => { + const hardware = 'NVIDIA H100 80GB HBM3'; + const misbundled = stored(h100.runId, h100.artifactId, hardware); + set(document(misbundled.sources[0], 'manifest.json'), 'run_id', '999'); + const inconsistent = stored(h100.runId, h100.artifactId, hardware); + set(document(inconsistent.sources[0], 'manifest.json'), 'mode', 'fidelity'); + expect(() => compareSide(misbundled, h100)).toThrow('Stored result identity mismatch'); + expect(() => compareSide(inconsistent, h100)).toThrow( + 'Invalid H3 serving matrix: CI identity does not match the manifest', + ); + expect(() => + compareSide( + { ...inconsistent, sources: [...inconsistent.sources, ...misbundled.sources] }, + h100, + ), + ).toThrow('Invalid H3 serving matrix: CI identity does not match the manifest'); + // A readable source without the cell does not prove the unreadable one lacked it. + const readable = stored(h100.runId, h100.artifactId, hardware); + expect(() => + compareSide( + { ...readable, sources: [...misbundled.sources, ...readable.sources] }, + { + ...h100, + cell: 'c3', + }, + ), + ).toThrow('Stored result identity mismatch'); + }); }); diff --git a/packages/app/src/components/video-benchmark/compare.ts b/packages/app/src/components/video-benchmark/compare.ts index 5d80b8614..cd6d83d54 100644 --- a/packages/app/src/components/video-benchmark/compare.ts +++ b/packages/app/src/components/video-benchmark/compare.ts @@ -8,7 +8,7 @@ import { } from './metrics'; import { latestVideoCells } from './points'; import { servingCells, type ServingCell } from './serving'; -import { storedBundle, type StoredArtifact } from './stored'; +import { storedBundle, type StoredArtifact, type StoredSource } from './stored'; /** Metrics the compare table shows, in row order. */ export const COMPARE_METRICS = [ @@ -190,11 +190,35 @@ export interface CompareSide { urls: Map; } +/** + * The cell's records inside one observation source, with that source's + * manifest SHA; null when the source lacks the cell. Contract violations throw, + * as in the run views. + */ +function sourceSide( + source: StoredSource, + cellId: string, +): { side: CompareSide; manifestSha256: string } | null { + const bundle = storedBundle(source); + const cell = servingCells(bundle).find((item) => item.id === cellId); + if (!cell) return null; + return { + manifestSha256: bundle.manifestSha256, + side: { + records: caseRecords(cell), + urls: new Map(source.assets.map(([path, asset]) => [path, asset.url])), + }, + }; +} + /** * Resolve a dashboard point inside its stored artifact (`format=media`): the * observation source whose manifest SHA the point id names, else the first - * source that publishes the point's cell. Contract violations throw, as in the - * run views; nothing is synthesised for a missing cell. + * source that publishes the point's cell. A malformed source is skipped so a + * readable sibling still resolves (as `videoHistoryEntry` isolates sources); + * its error is rethrown only when no source publishes the cell, so an + * unreadable artifact is never reported as one without the cell. Nothing is + * synthesised for a missing cell. */ export function compareSide(saved: StoredArtifact, point: VideoPoint): CompareSide { if ( @@ -206,18 +230,21 @@ export function compareSide(saved: StoredArtifact, point: VideoPoint): CompareSi const wanted = point.id.split(':')[0]; const cellId = point.cell ?? (point.concurrency === null ? 'c1' : `c${point.concurrency}`); let fallback: CompareSide | null = null; + let sourceError: Error | null = null; for (const source of saved.sources) { if (source.kind) continue; - const bundle = storedBundle(source); - const cell = servingCells(bundle).find((item) => item.id === cellId); - if (!cell) continue; - const side: CompareSide = { - records: caseRecords(cell), - urls: new Map(source.assets.map(([path, asset]) => [path, asset.url])), - }; - if (bundle.manifestSha256 === wanted) return side; - fallback ??= side; + let found: ReturnType; + try { + found = sourceSide(source, cellId); + } catch (error) { + sourceError ??= error instanceof Error ? error : new Error(String(error)); + continue; + } + if (found === null) continue; + if (found.manifestSha256 === wanted) return found.side; + fallback ??= found.side; } - if (fallback === null) throw new Error(`Published artifact has no ${cellId} cell`); - return fallback; + if (fallback !== null) return fallback; + if (sourceError !== null) throw sourceError; + throw new Error(`Published artifact has no ${cellId} cell`); } From 66099c8602681b49182fefc3b1333bd29de22798 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:31:35 -0700 Subject: [PATCH 027/104] feat: derive compute-bound evidence from measured video cells MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pure evidence.ts: board power vs. recorded enforced limit per hardware at C1, concurrency plateau rows with throughput/latency ratios against C1, consecutive-hardware speedups against GPU_SPECS bandwidth and BF16/FP8 ratios (log-space reading with an explicit inconclusive verdict), and caveat facts read from the cells. Missing input reads as null, never 0. Unit tests assert the retained fixture: 91.9/97.4/96.4 % of cap, 1.111x and 1.932x observed, C2/C4 throughput within 1.44 % of C1. 中文:新增纯模块 evidence.ts,从实测 cell 推导算力受限证据:C1 板卡功率对照记录到的生效上限、 各并发相对 C1 的吞吐/延迟比值、相邻硬件实测加速比对照 GPU_SPECS 带宽与 BF16/FP8 比值 (对数空间判读,无法区分时明确给出"不确定"),以及来自 cell 的注意事项事实。缺失输入返回 null 而非 0。 单元测试断言保留 fixture 的数值:功率占上限 91.9/97.4/96.4%、实测加速比 1.111x 与 1.932x、 C2/C4 吞吐量与 C1 偏差不超过 1.44%。 --- .../video-benchmark/evidence.test.ts | 274 +++++++++++++++++ .../components/video-benchmark/evidence.ts | 290 ++++++++++++++++++ 2 files changed, 564 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/evidence.test.ts create mode 100644 packages/app/src/components/video-benchmark/evidence.ts diff --git a/packages/app/src/components/video-benchmark/evidence.test.ts b/packages/app/src/components/video-benchmark/evidence.test.ts new file mode 100644 index 000000000..6002cc694 --- /dev/null +++ b/packages/app/src/components/video-benchmark/evidence.test.ts @@ -0,0 +1,274 @@ +import { describe, expect, it } from 'vitest'; +import fixture from '../../../cypress/fixtures/api/video-history.json'; +import { + concurrencyPlateau, + evidenceFacts, + parseBandwidthTbps, + plateauSummary, + powerRange, + powerUtilization, + readSpeedup, + scalingVsSpec, + specNumbers, +} from './evidence'; +import { hardwareSort } from './hardware'; +import type { VideoHistoryPage } from './history'; +import type { VideoPoint } from './metrics'; +import { videoPoints } from './points'; + +// Retained 2026-09-09 H100/H200/B200 C1/C2/C4 serving cells, n = 20 each; MI355X has no valid run. +const points = videoPoints([fixture as unknown as VideoHistoryPage]); +const cell = (hardwareKey: string, concurrency: number): VideoPoint => { + const point = points.find((p) => p.hardwareKey === hardwareKey && p.concurrency === concurrency); + if (!point) throw new Error(`fixture lacks ${hardwareKey} C${concurrency}`); + return point; +}; +const c1 = points.filter((p) => p.concurrency === 1); +const registryOrder = (keys: string[]) => + keys.toSorted((a, b) => hardwareSort(a) - hardwareSort(b)); + +describe('powerUtilization', () => { + it('reads mean board power against the recorded enforced limit per measured hardware at C1', () => { + const rows = powerUtilization(points); + const keys = rows.map((row) => row.hardwareKey); + expect(keys).toEqual(registryOrder(['h100', 'h200', 'b200'])); + expect(keys).toEqual(registryOrder(keys)); + const byKey = new Map(rows.map((row) => [row.hardwareKey, row])); + expect(byKey.get('h100')?.avgPowerW).toBeCloseTo(2574.55, 1); + expect(byKey.get('h100')?.enforcedLimitW).toBe(2800); + expect(byKey.get('h100')?.percentOfCap).toBeCloseTo(91.95, 1); + expect(byKey.get('h200')?.percentOfCap).toBeCloseTo(97.41, 1); + expect(byKey.get('b200')?.avgPowerW).toBeCloseTo(3856.02, 1); + expect(byKey.get('b200')?.enforcedLimitW).toBe(4000); + expect(byKey.get('b200')?.percentOfCap).toBeCloseTo(96.4, 1); + }); + it('summarises the measured spread and keeps missing limits null', () => { + expect(powerRange(powerUtilization(points))).toEqual({ + measured: 3, + min: expect.closeTo(91.95, 1), + max: expect.closeTo(97.41, 1), + }); + const unlimited = powerUtilization([{ ...cell('h100', 1), enforcedLimitW: null }]); + expect(unlimited).toEqual([ + { + hardwareKey: 'h100', + avgPowerW: expect.closeTo(2574.55, 1), + enforcedLimitW: null, + percentOfCap: null, + }, + ]); + expect(powerRange(unlimited)).toBeNull(); + expect(powerUtilization([{ ...cell('h100', 1), avgPowerW: 0 }])[0].percentOfCap).toBeNull(); + expect(powerUtilization(points.filter((p) => p.concurrency !== 1))).toEqual([]); + }); +}); + +describe('concurrencyPlateau', () => { + it('holds throughput within 1.5 % of C1 while P50 scales with client concurrency', () => { + const rows = concurrencyPlateau(points); + expect(rows).toHaveLength(9); + const row = (key: string, c: number) => + rows.find((r) => r.hardwareKey === key && r.concurrency === c); + expect(row('h100', 1)).toEqual({ + hardwareKey: 'h100', + concurrency: 1, + videosPerGpuHour: expect.closeTo(5.374, 3), + p50: expect.closeTo(167.333, 3), + throughputRatioVsC1: 1, + latencyRatioVsC1: 1, + }); + expect(row('h100', 2)?.throughputRatioVsC1).toBeCloseTo(0.999, 3); + expect(row('h100', 2)?.latencyRatioVsC1).toBeCloseTo(2.0007, 3); + expect(row('h100', 4)?.throughputRatioVsC1).toBeCloseTo(0.9989, 3); + expect(row('h100', 4)?.latencyRatioVsC1).toBeCloseTo(3.9995, 3); + expect(row('h200', 2)?.throughputRatioVsC1).toBeCloseTo(1.004, 3); + expect(row('h200', 2)?.latencyRatioVsC1).toBeCloseTo(1.9925, 3); + expect(row('h200', 4)?.throughputRatioVsC1).toBeCloseTo(1.0023, 3); + expect(row('h200', 4)?.latencyRatioVsC1).toBeCloseTo(3.9917, 3); + expect(row('b200', 2)?.throughputRatioVsC1).toBeCloseTo(1.0144, 3); + expect(row('b200', 2)?.latencyRatioVsC1).toBeCloseTo(1.9706, 3); + expect(row('b200', 4)?.videosPerGpuHour).toBeCloseTo(11.692, 3); + expect(row('b200', 4)?.p50).toBeCloseTo(307.284, 3); + expect(row('b200', 4)?.throughputRatioVsC1).toBeCloseTo(1.0144, 3); + expect(row('b200', 4)?.latencyRatioVsC1).toBeCloseTo(3.9423, 3); + }); + it('switches the throughput denominator with the basis without moving the ratios', () => { + const rows = concurrencyPlateau(points, 'allocated'); + const h100 = rows.filter((r) => r.hardwareKey === 'h100'); + expect(h100[0].videosPerGpuHour).toBeCloseTo(2.687, 3); + expect(h100[2].throughputRatioVsC1).toBeCloseTo(0.9989, 3); + // H200 allocated 4 of 4, so its denominator is unchanged. + expect(rows.find((r) => r.hardwareKey === 'h200')?.videosPerGpuHour).toBeCloseTo(5.973, 3); + }); + it('leaves ratios null without a C1 baseline and never reports 0', () => { + const rows = concurrencyPlateau( + points.filter((p) => !(p.hardwareKey === 'h200' && p.concurrency === 1)), + ); + const h200 = rows.filter((r) => r.hardwareKey === 'h200'); + expect(h200).toHaveLength(2); + expect(h200.every((r) => r.throughputRatioVsC1 === null && r.latencyRatioVsC1 === null)).toBe( + true, + ); + expect(h200[0].videosPerGpuHour).toBeCloseTo(5.997, 3); + const broken = concurrencyPlateau([{ ...cell('b200', 1), wallSeconds: null, p50: 0 }]); + expect(broken[0]).toMatchObject({ + videosPerGpuHour: null, + p50: null, + throughputRatioVsC1: null, + latencyRatioVsC1: null, + }); + }); + it('summarises queued cells: deviation from C1 and latency spread per concurrency', () => { + const summary = plateauSummary(concurrencyPlateau(points)); + expect(summary).toEqual({ + queued: 6, + throughputDeviationPct: expect.closeTo(1.437, 2), + latency: [ + { concurrency: 2, min: expect.closeTo(1.9706, 3), max: expect.closeTo(2.0007, 3) }, + { concurrency: 4, min: expect.closeTo(3.9423, 3), max: expect.closeTo(3.9995, 3) }, + ], + }); + expect(plateauSummary(concurrencyPlateau(c1))).toBeNull(); + expect(plateauSummary([])).toBeNull(); + }); +}); + +describe('scalingVsSpec', () => { + it('parses spec-sheet bandwidth strings in TB/s or GB/s', () => { + expect(parseBandwidthTbps('4.8 TB/s')).toBe(4.8); + expect(parseBandwidthTbps('3.35 TB/s')).toBe(3.35); + expect(parseBandwidthTbps('8 TB/s')).toBe(8); + expect(parseBandwidthTbps('3350 GB/s')).toBeCloseTo(3.35, 6); + expect(parseBandwidthTbps('0 TB/s')).toBeNull(); + expect(parseBandwidthTbps('n/a')).toBeNull(); + expect(parseBandwidthTbps('')).toBeNull(); + expect(parseBandwidthTbps(null)).toBeNull(); + }); + it('looks up the campaign hardware on the GPU specs page by SKU name', () => { + expect(specNumbers('h100')).toEqual({ + memoryBandwidthTbps: 3.35, + bf16Tflops: 989, + fp8Tflops: 1979, + }); + expect(specNumbers('h200')).toEqual({ + memoryBandwidthTbps: 4.8, + bf16Tflops: 989, + fp8Tflops: 1979, + }); + expect(specNumbers('b200')).toEqual({ + memoryBandwidthTbps: 8, + bf16Tflops: 2250, + fp8Tflops: 4500, + }); + expect(specNumbers('unknown')).toEqual({ + memoryBandwidthTbps: null, + bf16Tflops: null, + fp8Tflops: null, + }); + }); + it('pairs consecutive hardware from slowest to fastest C1 P50 against spec ratios', () => { + const rows = scalingVsSpec(points); + expect(rows.map((r) => [r.fromKey, r.toKey])).toEqual([ + ['h100', 'h200'], + ['h200', 'b200'], + ]); + expect(rows[0].observedSpeedup).toBeCloseTo(1.111, 3); + expect(rows[0].memoryBandwidthRatio).toBeCloseTo(1.4328, 3); + expect(rows[0].bf16Ratio).toBe(1); + expect(rows[0].fp8Ratio).toBe(1); + expect(rows[1].observedSpeedup).toBeCloseTo(1.932, 3); + expect(rows[1].memoryBandwidthRatio).toBeCloseTo(1.6667, 3); + expect(rows[1].bf16Ratio).toBeCloseTo(2.275, 3); + expect(rows[1].fp8Ratio).toBeCloseTo(2.2739, 3); + }); + it('reads which spec ratio the observed speedup sits closer to, and says when it cannot tell', () => { + const [h200, b200] = scalingVsSpec(points); + expect(readSpeedup(h200)).toEqual({ closest: 'compute', position: 'between', decisive: true }); + expect(readSpeedup(b200)).toEqual({ + closest: 'bandwidth', + position: 'between', + decisive: false, + }); + expect(readSpeedup({ ...b200, memoryBandwidthRatio: null })).toBeNull(); + expect(readSpeedup({ ...b200, bf16Ratio: null, fp8Ratio: null })).toBeNull(); + expect(readSpeedup({ ...b200, observedSpeedup: 2.5 })).toEqual({ + closest: 'compute', + position: 'above', + decisive: true, + }); + expect(readSpeedup({ ...h200, observedSpeedup: 0.9 })).toEqual({ + closest: 'compute', + position: 'below', + decisive: true, + }); + }); + it('nulls every ratio for hardware missing from the specs page and needs two hardware', () => { + const rows = scalingVsSpec([ + { ...cell('h100', 1), hardwareKey: 'gb200', id: 'gb200:c1' }, + cell('b200', 1), + ]); + expect(rows).toEqual([ + { + fromKey: 'gb200', + toKey: 'b200', + observedSpeedup: expect.closeTo(2.1468, 3), + memoryBandwidthRatio: null, + bf16Ratio: null, + fp8Ratio: null, + }, + ]); + expect(readSpeedup(rows[0])).toBeNull(); + expect(scalingVsSpec([cell('h100', 1)])).toEqual([]); + expect(scalingVsSpec([cell('h100', 1), { ...cell('h200', 1), p50: null }])).toEqual([]); + }); +}); + +describe('evidenceFacts', () => { + it('reads the shared recipe, sample count and workload from the cells', () => { + expect(evidenceFacts(points)).toEqual({ + participating: 4, + tp: 2, + ulysses: 2, + samples: { min: 20, max: 20 }, + workload: '1344 × 768 · 8 s · 24 fps · 50 steps', + attention: [], + }); + }); + it('drops a fact the cells disagree on and lists attention backends only when they differ', () => { + const b200 = cell('b200', 1); + const h200 = cell('h200', 1); + const facts = evidenceFacts([ + { + ...b200, + participating: 8, + samples: 18, + server: { tp: 4, ulysses: 2, attention: 'dynamic_cudnn_sdpa' }, + }, + { ...h200, server: { tp: 2, ulysses: 2, attention: 'fa' } }, + { ...cell('h100', 1), server: null, workload: '' }, + ]); + expect(facts).toEqual({ + participating: null, + tp: null, + ulysses: 2, + samples: { min: 18, max: 20 }, + workload: '1344 × 768 · 8 s · 24 fps · 50 steps', + attention: expect.arrayContaining([ + { hardwareKey: 'b200', attention: 'dynamic_cudnn_sdpa' }, + { hardwareKey: 'h200', attention: 'fa' }, + ]), + }); + expect(facts.attention).toHaveLength(2); + expect( + evidenceFacts([{ ...b200, server: { tp: 2, ulysses: 2, attention: 'fa' } }]).attention, + ).toEqual([]); + expect(evidenceFacts([])).toEqual({ + participating: null, + tp: null, + ulysses: null, + samples: null, + workload: null, + attention: [], + }); + }); +}); diff --git a/packages/app/src/components/video-benchmark/evidence.ts b/packages/app/src/components/video-benchmark/evidence.ts new file mode 100644 index 000000000..8af5283be --- /dev/null +++ b/packages/app/src/components/video-benchmark/evidence.ts @@ -0,0 +1,290 @@ +import { GPU_SPECS } from '@/lib/gpu-specs'; +import { hardwareSort } from './hardware'; +import { metricValue, type GpuBasis, type MetricOptions, type VideoPoint } from './metrics'; +import { latestVideoCells } from './points'; + +/** + * Compute-bound evidence read from the measured cells only: board power against + * the recorded enforced limit, the client-concurrency plateau, and observed + * cross-hardware speedups against spec-sheet ratios. Every number is derived + * from published observations or the GPU specs page; missing input reads as + * null, never 0, so a panel can hide what the data does not support. + */ + +const finite = (n: number | null | undefined): n is number => + typeof n === 'number' && Number.isFinite(n); +const positive = (n: number | null | undefined): n is number => finite(n) && n > 0; +const ratio = (numerator: number | null, denominator: number | null): number | null => + finite(numerator) && positive(denominator) ? numerator / denominator : null; + +type MeasuredPoint = VideoPoint & { hardwareKey: string; concurrency: number }; +const isMeasured = (p: VideoPoint): p is MeasuredPoint => + p.hardwareKey !== null && positive(p.concurrency); + +/** Newest observation per (hardware, concurrency) in HW_REGISTRY order, then ascending concurrency. */ +function measuredCells(points: VideoPoint[]): MeasuredPoint[] { + return latestVideoCells(points) + .filter(isMeasured) + .toSorted( + (a, b) => + hardwareSort(a.hardwareKey) - hardwareSort(b.hardwareKey) || a.concurrency - b.concurrency, + ); +} + +/** The cost tier only enters priced metrics; nothing read here is priced. */ +const metricOptions = (basis: GpuBasis): MetricOptions => ({ tier: 'h', basis }); + +export interface PowerUtilizationRow { + hardwareKey: string; + /** Sum of the participating boards' time-weighted mean power during generation. */ + avgPowerW: number | null; + /** Sum of the same boards' enforced limits recorded around the run, not marketing TDP. */ + enforcedLimitW: number | null; + percentOfCap: number | null; +} + +/** One row per measured hardware at C1. */ +export function powerUtilization(points: VideoPoint[]): PowerUtilizationRow[] { + return measuredCells(points) + .filter((p) => p.concurrency === 1) + .map((p) => ({ + hardwareKey: p.hardwareKey, + avgPowerW: positive(p.avgPowerW) ? p.avgPowerW : null, + enforcedLimitW: positive(p.enforcedLimitW) ? p.enforcedLimitW : null, + percentOfCap: metricValue(p, 'powerPctCap', metricOptions('participating')), + })); +} + +export interface PowerRange { + /** Hardware with both a mean power and a recorded limit. */ + measured: number; + min: number; + max: number; +} + +/** Spread of the measured percentages; null when nothing recorded both power and a limit. */ +export function powerRange(rows: PowerUtilizationRow[]): PowerRange | null { + const values = rows.map((row) => row.percentOfCap).filter(finite); + if (values.length === 0) return null; + return { measured: values.length, min: Math.min(...values), max: Math.max(...values) }; +} + +export interface ConcurrencyPlateauRow { + hardwareKey: string; + concurrency: number; + videosPerGpuHour: number | null; + p50: number | null; + /** Throughput relative to the same hardware's C1 cell; null when C1 is missing. */ + throughputRatioVsC1: number | null; + /** P50 time to video relative to the same hardware's C1 cell; null when C1 is missing. */ + latencyRatioVsC1: number | null; +} + +/** + * Every measured cell against its hardware's C1 cell. On a batch-one server a + * flat throughput ratio with latency scaling by C says client concurrency only + * queues requests. `basis` picks the GPU denominator the rest of the dashboard + * shows; ratios within one hardware are unaffected unless its allocation changed. + */ +export function concurrencyPlateau( + points: VideoPoint[], + basis: GpuBasis = 'participating', +): ConcurrencyPlateauRow[] { + const cells = measuredCells(points); + const options = metricOptions(basis); + const baseline = new Map( + cells.filter((p) => p.concurrency === 1).map((p) => [p.hardwareKey, p] as const), + ); + return cells.map((p) => { + const base = baseline.get(p.hardwareKey); + const videosPerGpuHour = metricValue(p, 'videosPerGpuHour', options); + const p50 = positive(p.p50) ? p.p50 : null; + return { + hardwareKey: p.hardwareKey, + concurrency: p.concurrency, + videosPerGpuHour, + p50, + throughputRatioVsC1: ratio( + videosPerGpuHour, + base ? metricValue(base, 'videosPerGpuHour', options) : null, + ), + latencyRatioVsC1: ratio(p50, base && positive(base.p50) ? base.p50 : null), + }; + }); +} + +type QueuedRow = ConcurrencyPlateauRow & { throughputRatioVsC1: number; latencyRatioVsC1: number }; +const isQueued = (row: ConcurrencyPlateauRow): row is QueuedRow => + row.concurrency > 1 && finite(row.throughputRatioVsC1) && finite(row.latencyRatioVsC1); + +export interface PlateauSummary { + /** Cells above C1 that have a C1 baseline. */ + queued: number; + /** Largest |throughput ratio − 1| across those cells, in percent. */ + throughputDeviationPct: number; + /** Latency ratio spread per concurrency, ascending. */ + latency: { concurrency: number; min: number; max: number }[]; +} + +/** Null when no hardware has a cell above C1 with a baseline: nothing to plateau against. */ +export function plateauSummary(rows: ConcurrencyPlateauRow[]): PlateauSummary | null { + const queued = rows.filter(isQueued); + if (queued.length === 0) return null; + const byConcurrency = new Map(); + for (const row of queued) { + const list = byConcurrency.get(row.concurrency) ?? []; + list.push(row.latencyRatioVsC1); + byConcurrency.set(row.concurrency, list); + } + return { + queued: queued.length, + throughputDeviationPct: + Math.max(...queued.map((row) => Math.abs(row.throughputRatioVsC1 - 1))) * 100, + latency: [...byConcurrency] + .toSorted(([a], [b]) => a - b) + .map(([concurrency, values]) => ({ + concurrency, + min: Math.min(...values), + max: Math.max(...values), + })), + }; +} + +/** GPU_SPECS names for the campaign hardware; the specs page keys rows by SKU name, not registry key. */ +const SPEC_NAME: Readonly> = { + h100: 'H100 SXM', + h200: 'H200 SXM', + b200: 'B200 SXM', + mi355x: 'MI355X', +}; + +export interface SpecNumbers { + memoryBandwidthTbps: number | null; + bf16Tflops: number | null; + fp8Tflops: number | null; +} + +/** `"4.8 TB/s"` → 4.8, `"3350 GB/s"` → 3.35; anything else → null. */ +export function parseBandwidthTbps(text: string | null | undefined): number | null { + const match = /^\s*(?\d+(?:\.\d+)?)\s*(?[GT])B\/s\s*$/iu.exec(text ?? ''); + if (!match?.groups) return null; + const value = Number(match.groups.value); + if (!positive(value)) return null; + return match.groups.unit.toUpperCase() === 'G' ? value / 1000 : value; +} + +/** Per-GPU HBM bandwidth and dense tensor-core peaks from the GPU specs page; null when unlisted. */ +export function specNumbers(hardwareKey: string): SpecNumbers { + const spec = GPU_SPECS.find((entry) => entry.name === SPEC_NAME[hardwareKey]); + return { + memoryBandwidthTbps: spec ? parseBandwidthTbps(spec.memoryBandwidth) : null, + bf16Tflops: spec && positive(spec.bf16) ? spec.bf16 : null, + fp8Tflops: spec && positive(spec.fp8) ? spec.fp8 : null, + }; +} + +export interface ScalingVsSpecRow { + fromKey: string; + toKey: string; + /** C1 P50 of `from` over `to`: how many times faster `to` finished the same clip. */ + observedSpeedup: number | null; + memoryBandwidthRatio: number | null; + bf16Ratio: number | null; + fp8Ratio: number | null; +} + +type TimedPoint = MeasuredPoint & { p50: number }; +const isTimedBaseline = (p: MeasuredPoint): p is TimedPoint => + p.concurrency === 1 && positive(p.p50); + +/** Consecutive measured hardware from slowest to fastest C1 P50, each step against its spec ratios. */ +export function scalingVsSpec(points: VideoPoint[]): ScalingVsSpecRow[] { + const ordered = measuredCells(points) + .filter(isTimedBaseline) + .toSorted((a, b) => b.p50 - a.p50); + return ordered.slice(1).map((to, index) => { + const from = ordered[index]; + const fromSpec = specNumbers(from.hardwareKey); + const toSpec = specNumbers(to.hardwareKey); + return { + fromKey: from.hardwareKey, + toKey: to.hardwareKey, + observedSpeedup: ratio(from.p50, to.p50), + memoryBandwidthRatio: ratio(toSpec.memoryBandwidthTbps, fromSpec.memoryBandwidthTbps), + bf16Ratio: ratio(toSpec.bf16Tflops, fromSpec.bf16Tflops), + fp8Ratio: ratio(toSpec.fp8Tflops, fromSpec.fp8Tflops), + }; + }); +} + +export interface SpeedupReading { + /** Spec ratio the observed speedup sits nearer to, measured in log space because ratios multiply. */ + closest: 'bandwidth' | 'compute'; + /** Where the observed speedup falls relative to the smallest and largest spec ratio. */ + position: 'below' | 'between' | 'above'; + /** False when the two log-distances are within 25 % of each other: the data does not separate them. */ + decisive: boolean; +} + +/** Ratio of near to far log-distance below which one spec ratio is called closer. */ +const DECISIVE_BELOW = 0.75; + +/** Null when the observed speedup or every spec ratio is missing. */ +export function readSpeedup(row: ScalingVsSpecRow): SpeedupReading | null { + const compute = [row.bf16Ratio, row.fp8Ratio].filter(positive); + if (!positive(row.observedSpeedup) || !positive(row.memoryBandwidthRatio) || compute.length === 0) + return null; + const observed = Math.log(row.observedSpeedup); + const toBandwidth = Math.abs(observed - Math.log(row.memoryBandwidthRatio)); + const toCompute = Math.min(...compute.map((value) => Math.abs(observed - Math.log(value)))); + const near = Math.min(toBandwidth, toCompute); + const far = Math.max(toBandwidth, toCompute); + const specs = [row.memoryBandwidthRatio, ...compute]; + const lowest = Math.min(...specs); + const highest = Math.max(...specs); + return { + closest: toBandwidth <= toCompute ? 'bandwidth' : 'compute', + position: + row.observedSpeedup < lowest ? 'below' : row.observedSpeedup > highest ? 'above' : 'between', + decisive: far > 0 && near / far < DECISIVE_BELOW, + }; +} + +export interface EvidenceFacts { + /** Participating GPUs per request when every measured cell agrees. */ + participating: number | null; + tp: number | null; + ulysses: number | null; + /** Valid samples per cell across measured cells. */ + samples: { min: number; max: number } | null; + /** Shape · duration · fps · steps from the projection's workload label, when every cell agrees. */ + workload: string | null; + /** Attention backend per hardware, listed only when the recorded backends differ. */ + attention: { hardwareKey: string; attention: string }[]; +} + +/** The one value every cell shares, or null when they disagree or none recorded it. */ +function uniform(values: (T | null)[]): T | null { + const present = values.filter((value): value is T => value !== null); + if (present.length === 0) return null; + return present.every((value) => value === present[0]) ? present[0] : null; +} + +/** Caveat inputs read from the cells so the panel never states a number the data does not carry. */ +export function evidenceFacts(points: VideoPoint[]): EvidenceFacts { + const cells = measuredCells(points); + const samples = cells.map((p) => p.samples).filter(positive); + const attention = cells + .filter((p) => p.concurrency === 1) + .flatMap((p) => + p.server?.attention ? [{ hardwareKey: p.hardwareKey, attention: p.server.attention }] : [], + ); + return { + participating: uniform(cells.map((p) => (positive(p.participating) ? p.participating : null))), + tp: uniform(cells.map((p) => (positive(p.server?.tp) ? p.server.tp : null))), + ulysses: uniform(cells.map((p) => (positive(p.server?.ulysses) ? p.server.ulysses : null))), + samples: samples.length > 0 ? { min: Math.min(...samples), max: Math.max(...samples) } : null, + workload: uniform(cells.map((p) => p.workload.split(' · ').slice(0, 4).join(' · ') || null)), + attention: uniform(attention.map((entry) => entry.attention)) === null ? attention : [], + }; +} From e49c63be4bd520f0c03983c830b8c8901b632cfd Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:31:55 -0700 Subject: [PATCH 028/104] feat: add VideoEvidence compute-bound panel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three exhibits in Cards, each with a one-sentence reading built from the computed numbers: power vs. enforced limit as per-hardware meters, concurrency plateau table grouped by hardware, and observed speedup vs. spec-sheet ratios with a hedged reading per pair. Bilingual caveats are derived from the cells (participating GPUs and TP x Ulysses recipe, n per cell, frozen workload, differing attention backends). Exhibits without supporting data are hidden; the section disappears when nothing is measured. No interactive controls, so no analytics events. 中文:新增 VideoEvidence 面板,三组展示均放在 Card 中,每组附一句由计算结果生成的判读: 板卡功率对照生效上限(按硬件的 meter)、按硬件分组的并发平台期表格、实测加速比对照规格表比值 (每对硬件给出留有余地的判读)。中英双语注意事项由 cell 数据推导(参与计算的 GPU 数与 TP x Ulysses 配方、每 cell 样本数、冻结工作负载、不同的 attention 后端)。数据不足的展示会隐藏;没有实测时整段不渲染。 无交互控件,因此无 analytics 事件。 --- .../video-benchmark/VideoEvidence.tsx | 493 ++++++++++++++++++ 1 file changed, 493 insertions(+) create mode 100644 packages/app/src/components/video-benchmark/VideoEvidence.tsx diff --git a/packages/app/src/components/video-benchmark/VideoEvidence.tsx b/packages/app/src/components/video-benchmark/VideoEvidence.tsx new file mode 100644 index 000000000..cf98e1410 --- /dev/null +++ b/packages/app/src/components/video-benchmark/VideoEvidence.tsx @@ -0,0 +1,493 @@ +'use client'; + +import { Card } from '@/components/ui/card'; +import { Heading } from '@/components/ui/heading'; +import type { Locale } from '@/lib/i18n'; +import { useLocale } from '@/lib/use-locale'; +import { + concurrencyPlateau, + evidenceFacts, + plateauSummary, + powerRange, + powerUtilization, + readSpeedup, + scalingVsSpec, + type ConcurrencyPlateauRow, + type ScalingVsSpecRow, +} from './evidence'; +import { hardwareLabel } from './hardware'; +import { formatMetric, type MetricOptions, type VideoPoint } from './metrics'; + +const STRINGS = { + en: { + title: 'Compute-bound evidence', + subtitle: + 'Three readings from the measured cells on why this workload behaves like a compute-bound diffusion model on the pinned runtime. Every number is computed from the published runs; the wording follows the numbers.', + power: 'Board power vs. enforced limit', + powerHint: + 'Mean GPU-board power of the participating boards while generating, against the power limit recorded around the run, not marketing TDP.', + powerOne: (hardware: string, share: string) => + `${hardware} ran at ${share} of its enforced power limit while generating: near the cap, as a compute-saturated workload would.`, + powerAll: (n: number, share: string) => + `All ${n} measured GPU types ran at ${share} of their enforced power limit while generating: near the cap, as a compute-saturated workload would.`, + notRecorded: 'power or limit not recorded', + meter: 'share of enforced power limit', + plateau: 'Client concurrency plateau', + plateauHint: + "Newest cell per hardware and client concurrency; ratios are against the same hardware's C1 cell.", + plateauSentence: (queued: number, deviation: string, latency: string) => + `Across ${queued} queued cells, throughput per GPU-hour stays within ${deviation} of C1 while P50 time to video rises to ${latency}: on the batch-one server, extra client concurrency queues requests instead of batching them.`, + at: (concurrency: number, range: string) => `${range} at C${concurrency}`, + hardware: 'Hardware', + c: 'C', + rate: 'Videos / GPU-hr', + p50: 'P50 (s)', + throughput: '× C1 throughput', + latency: '× C1 latency', + scaling: 'Observed speedup vs. spec-sheet ratios', + scalingHint: + 'Consecutive hardware from slowest to fastest C1 P50. Observed = P50 ratio; spec ratios use per-GPU HBM bandwidth and dense tensor-core TFLOPS from the GPU specs page.', + pair: 'Pair', + observed: 'Observed (P50 ratio)', + bandwidth: 'Memory bandwidth', + bf16: 'BF16 dense', + fp8: 'FP8 dense', + reading: 'Reading', + readings: { + missing: 'spec missing', + compute: 'closer to the FLOPS ratio', + bandwidth: 'closer to the bandwidth ratio', + between: 'between the two, not clearly closer to either', + above: 'above both spec ratios, not clearly closer to either', + below: 'below both spec ratios, not clearly closer to either', + }, + scalingStep: ( + from: string, + to: string, + observed: string, + bandwidth: string, + flops: string, + reading: string, + ) => + `${from} → ${to}: ${observed} observed vs. ${bandwidth} bandwidth and ${flops} FLOPS, ${reading}.`, + scalingTail: + 'A bandwidth-bound workload would track the bandwidth ratio; a compute-bound one the FLOPS ratio.', + caveats: 'Read with these caveats', + caveatServer: + 'Batch-one, single-replica server: client concurrency queues requests rather than batching them.', + caveatGpus: (n: number, recipe: string) => + `${n} participating GPUs per video${recipe}; allocated boards beyond those sit idle.`, + caveatGpusUnknown: 'Participating GPUs per video differ between cells or were not recorded.', + recipe: (tp: number, ulysses: number) => ` (TP${tp} × Ulysses ${ulysses})`, + caveatPower: + 'GPU-board power against the recorded enforced limit; not node, rack or facility power.', + caveatSamples: (n: string) => + `n = ${n} clips per cell, one cohort per hardware: no error bars, and P90 is a display floor rather than a tail SLO.`, + caveatSamplesUnknown: 'Sample counts were not recorded.', + caveatWorkload: (workload: string) => + `Frozen workload ${workload}; every ratio compares the same clip across hardware.`, + caveatWorkloadUnknown: + 'Cells do not share one workload label; cross-hardware ratios only hold for the same clip.', + caveatAttention: (backends: string) => + `Attention backend differs by hardware (${backends}); each point is hardware plus its measured recipe.`, + }, + zh: { + title: '算力受限(compute-bound)的证据', + subtitle: + '来自实测 cell 的三组读数,说明该工作负载在固定 runtime 上为何表现为算力受限的扩散模型。所有数字均由已发布运行计算得出,措辞以数字为准。', + power: '板卡功率 vs. 生效功率上限', + powerHint: + '生成期间参与计算板卡的 GPU 板卡平均功率,对照运行前后记录到的功率上限,而非标称 TDP。', + powerOne: (hardware: string, share: string) => + `${hardware} 生成期间运行在生效功率上限的 ${share}:接近上限,符合算力饱和型负载的表现。`, + powerAll: (n: number, share: string) => + `全部 ${n} 种实测硬件生成期间均运行在生效功率上限的 ${share}:接近上限,符合算力饱和型负载的表现。`, + notRecorded: '未记录功率或上限', + meter: '占生效功率上限的比例', + plateau: '客户端并发平台期', + plateauHint: '每种硬件、每个客户端并发下最新的 cell;比值均相对同一硬件的 C1 cell。', + plateauSentence: (queued: number, deviation: string, latency: string) => + `在 ${queued} 个排队 cell 中,每 GPU 小时吞吐量与 C1 的偏差不超过 ${deviation},而 P50 出片时间升至 ${latency}:在 batch-one 服务器上,额外的客户端并发只是让请求排队,并没有形成 batching。`, + at: (concurrency: number, range: string) => `C${concurrency} 时 ${range}`, + hardware: '硬件', + c: 'C', + rate: '视频数 / GPU 小时', + p50: 'P50(s)', + throughput: '吞吐量 × C1', + latency: '延迟 × C1', + scaling: '实测加速比 vs. 规格表比值', + scalingHint: + '按 C1 P50 从慢到快排列的相邻硬件。实测值为 P50 比值;规格比值取自 GPU 规格页的单卡 HBM 带宽与 dense tensor-core TFLOPS。', + pair: '硬件对', + observed: '实测(P50 比值)', + bandwidth: '显存带宽', + bf16: 'BF16 dense', + fp8: 'FP8 dense', + reading: '判读', + readings: { + missing: '缺少规格', + compute: '更接近 FLOPS 比值', + bandwidth: '更接近带宽比值', + between: '介于两者之间,不明显偏向任一方', + above: '高于两个规格比值,不明显偏向任一方', + below: '低于两个规格比值,不明显偏向任一方', + }, + scalingStep: ( + from: string, + to: string, + observed: string, + bandwidth: string, + flops: string, + reading: string, + ) => + `${from} → ${to}:实测 ${observed},对比带宽比值 ${bandwidth}、FLOPS 比值 ${flops},${reading}。`, + scalingTail: '带宽受限的负载会跟随带宽比值,算力受限的负载会跟随 FLOPS 比值。', + caveats: '阅读时请注意', + caveatServer: 'batch-one 单副本服务器:客户端并发只会让请求排队,不会形成 batching。', + caveatGpus: (n: number, recipe: string) => + `每条视频由 ${n} 张 GPU 参与计算${recipe};超出部分的已分配板卡处于空闲。`, + caveatGpusUnknown: '各 cell 参与计算的 GPU 数不一致或未记录。', + recipe: (tp: number, ulysses: number) => `(TP${tp} × Ulysses ${ulysses})`, + caveatPower: 'GPU 板卡级功率,对照记录到的生效上限;不是节点、机柜或设施功率。', + caveatSamples: (n: string) => + `每个 cell n = ${n} 条视频、每种硬件一个批次:没有误差棒,P90 只是显示下限而非尾延迟 SLO。`, + caveatSamplesUnknown: '未记录样本数。', + caveatWorkload: (workload: string) => + `冻结的工作负载 ${workload};所有比值都是同一条视频在不同硬件上的对比。`, + caveatWorkloadUnknown: '各 cell 的工作负载标签不一致;跨硬件比值只在同一条视频下成立。', + caveatAttention: (backends: string) => + `各硬件的 attention 后端不同(${backends});每个数据点都是“硬件 + 实测配方”。`, + }, +}; + +const number = (value: number, digits: number) => + value.toLocaleString('en-US', { minimumFractionDigits: digits, maximumFractionDigits: digits }); +const times = (value: number | null) => (value === null ? '—' : `${number(value, 2)}×`); +/** "92–97" or "97" when both ends print alike. */ +function span(min: number, max: number, digits: number): string { + const low = number(min, digits); + const high = number(max, digits); + return low === high ? low : `${low}–${high}`; +} +/** Percent range: "92–97%" in English, "92%–97%" in Chinese. */ +function percentSpan(min: number, max: number, digits: number, locale: Locale): string { + const low = number(min, digits); + const high = number(max, digits); + if (low === high) return `${low}%`; + return locale === 'zh' ? `${low}%–${high}%` : `${low}–${high}%`; +} +function list(items: string[], locale: Locale): string { + if (locale === 'zh') return items.join('、'); + if (items.length <= 2) return items.join(' and '); + return `${items.slice(0, -1).join(', ')} and ${items.at(-1)}`; +} + +function HardwareName({ hardwareKey, color }: { hardwareKey: string; color: string }) { + return ( + + + {hardwareLabel(hardwareKey)} + + ); +} + +const HEAD = 'pb-2 pr-3 font-normal text-muted-foreground'; +const CELL = 'border-t border-border/40 py-1.5 pr-3'; + +export interface VideoEvidenceProps { + points: VideoPoint[]; + options: MetricOptions; + colorFor: (hardwareKey: string) => string; +} + +/** + * Compute-bound evidence panel: board power at the enforced limit, the + * client-concurrency plateau and observed speedups against spec-sheet ratios. + * Each exhibit renders only when the measured cells support it, and every + * sentence is built from the computed numbers; the section disappears when + * nothing is measured. + */ +export default function VideoEvidence({ points, options, colorFor }: VideoEvidenceProps) { + const locale = useLocale(); + const s = STRINGS[locale]; + const power = powerUtilization(points); + const range = powerRange(power); + const plateau = concurrencyPlateau(points, options.basis); + const summary = plateauSummary(plateau); + const scaling = scalingVsSpec(points); + if (range === null && summary === null && scaling.length === 0) return null; + const facts = evidenceFacts(points); + + const powerSentence = + range === null + ? null + : range.measured === 1 + ? s.powerOne( + hardwareLabel(power.find((row) => row.percentOfCap !== null)?.hardwareKey ?? ''), + percentSpan(range.min, range.max, 0, locale), + ) + : s.powerAll(range.measured, percentSpan(range.min, range.max, 0, locale)); + + const plateauSentence = + summary === null + ? null + : s.plateauSentence( + summary.queued, + `${number(summary.throughputDeviationPct, 1)}%`, + list( + summary.latency.map((entry) => + s.at(entry.concurrency, `${span(entry.min, entry.max, 2)}×`), + ), + locale, + ), + ); + const plateauGroups = new Map(); + for (const row of plateau) { + const group = plateauGroups.get(row.hardwareKey) ?? []; + group.push(row); + plateauGroups.set(row.hardwareKey, group); + } + + const flops = (row: ScalingVsSpecRow) => { + const bf16 = times(row.bf16Ratio); + const fp8 = times(row.fp8Ratio); + return bf16 === fp8 ? bf16 : `${bf16} (BF16) / ${fp8} (FP8)`; + }; + const readingText = (row: ScalingVsSpecRow) => { + const reading = readSpeedup(row); + if (reading === null) return s.readings.missing; + return reading.decisive ? s.readings[reading.closest] : s.readings[reading.position]; + }; + const scalingSentence = + scaling.length === 0 + ? null + : `${scaling + .map((row) => + s.scalingStep( + hardwareLabel(row.fromKey), + hardwareLabel(row.toKey), + times(row.observedSpeedup), + times(row.memoryBandwidthRatio), + flops(row), + readingText(row), + ), + ) + .join(' ')} ${s.scalingTail}`; + + const caveats = [ + s.caveatServer, + facts.participating === null + ? s.caveatGpusUnknown + : s.caveatGpus( + facts.participating, + facts.tp !== null && facts.ulysses !== null ? s.recipe(facts.tp, facts.ulysses) : '', + ), + s.caveatPower, + facts.samples === null + ? s.caveatSamplesUnknown + : s.caveatSamples(span(facts.samples.min, facts.samples.max, 0)), + facts.workload === null ? s.caveatWorkloadUnknown : s.caveatWorkload(facts.workload), + ...(facts.attention.length > 0 + ? [ + s.caveatAttention( + facts.attention + .map((entry) => `${hardwareLabel(entry.hardwareKey)} ${entry.attention}`) + .join(' · '), + ), + ] + : []), + ]; + + return ( +
+
+ + {s.title} + +

{s.subtitle}

+
+
+ {range !== null && ( + +
+ + {s.power} + +

{s.powerHint}

+
+
    + {power.map((row) => ( +
  • +
    + + + {row.avgPowerW !== null && row.enforcedLimitW !== null + ? `${number(row.avgPowerW, 0)} W / ${number(row.enforcedLimitW, 0)} W` + : s.notRecorded} + {row.percentOfCap !== null && ( + + {number(row.percentOfCap, 1)}% + + )} + +
    + {row.percentOfCap !== null && ( +
    +
    +
    + )} +
  • + ))} +
+

+ {powerSentence} +

+
+ )} + {summary !== null && ( + +
+ + {s.plateau} + +

{s.plateauHint}

+
+
+ + + + + + + + + + + + + {[...plateauGroups].map(([hardwareKey, rows]) => + rows.map((row, index) => ( + + {index === 0 && ( + + )} + + + + + + + )), + )} + +
{s.hardware}{s.c}{s.rate}{s.p50}{s.throughput}{s.latency}
+ + C{row.concurrency} + {formatMetric(row.videosPerGpuHour, 'videosPerGpuHour')} + + {formatMetric(row.p50, 'p50Latency')} + {times(row.throughputRatioVsC1)}{times(row.latencyRatioVsC1)}
+
+

+ {plateauSentence} +

+
+ )} + {scaling.length > 0 && ( + +
+ + {s.scaling} + +

{s.scalingHint}

+
+
+ + + + + + + + + + + + + {scaling.map((row) => ( + + + + + + + + + ))} + +
{s.pair}{s.observed}{s.bandwidth}{s.bf16}{s.fp8}{s.reading}
+ + + + + + + {times(row.observedSpeedup)} + {times(row.memoryBandwidthRatio)}{times(row.bf16Ratio)}{times(row.fp8Ratio)}{readingText(row)}
+
+

+ {scalingSentence} +

+
+ )} +
+
+ + {s.caveats} + +
    + {caveats.map((caveat) => ( +
  • {caveat}
  • + ))} +
+
+
+ ); +} From d7cfae91668a107afb012aa61e6bd32c0e4f5dbb Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:32:06 -0700 Subject: [PATCH 029/104] test: cover the video evidence panel with the retained fixture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cypress component spec mounts VideoEvidence with points from cypress/fixtures/api/video-history.json through videoPoints and asserts the rendered percentages and ratios (91.9/97.4/96.4 %, 1.11x vs 1.43x, 1.93x between 1.67x and 2.27-2.28x), the allocated-basis column, the Chinese copy under /zh/video, hidden exhibits with one C1 cell, and an empty render with no measured cells. 中文:新增 Cypress 组件测试,通过 videoPoints 用 cypress/fixtures/api/video-history.json 挂载 VideoEvidence,断言渲染出的百分比与比值(91.9/97.4/96.4%、1.11x 对 1.43x、 1.93x 介于 1.67x 与 2.27-2.28x 之间)、allocated 口径列、/zh/video 下的中文文案、 仅有一个 C1 cell 时隐藏的展示,以及无实测 cell 时不渲染。 --- .../cypress/component/video-evidence.cy.tsx | 170 ++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 packages/app/cypress/component/video-evidence.cy.tsx diff --git a/packages/app/cypress/component/video-evidence.cy.tsx b/packages/app/cypress/component/video-evidence.cy.tsx new file mode 100644 index 000000000..00792f0f8 --- /dev/null +++ b/packages/app/cypress/component/video-evidence.cy.tsx @@ -0,0 +1,170 @@ +import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; +import type { VideoHistoryPage } from '@/components/video-benchmark/history'; +import type { GpuBasis } from '@/components/video-benchmark/metrics'; +import { videoPoints } from '@/components/video-benchmark/points'; +import VideoEvidence from '@/components/video-benchmark/VideoEvidence'; + +// Retained H100/H200/B200 C1/C2/C4 observations (cypress/fixtures/api/video-history.json). +function mount(pathname = '/video', basis: GpuBasis = 'participating') { + cy.fixture('api/video-history.json').then((page: VideoHistoryPage) => { + cy.mount( + + (key === 'b200' ? 'rgb(1, 2, 3)' : 'rgb(4, 5, 6)')} + /> + , + ); + }); +} +const exhibit = (name: string) => cy.get(`[data-testid="video-evidence-${name}"]`); + +describe('Video compute-bound evidence (retained fixture)', () => { + it('reads board power against the recorded enforced limit per measured hardware', () => { + mount(); + exhibit('power').find('[data-testid="video-evidence-power-row"]').should('have.length', 3); + exhibit('power') + .find('[data-hardware="h100"]') + .should('contain', '2,575 W / 2,800 W') + .and('contain', '91.9%'); + exhibit('power').find('[data-hardware="h200"]').should('contain', '97.4%'); + exhibit('power') + .find('[data-hardware="b200"]') + .should('contain', '3,856 W / 4,000 W') + .and('contain', '96.4%'); + exhibit('power').find('[role="meter"]').should('have.length', 3); + exhibit('power') + .find('[data-hardware="b200"] [role="meter"]') + .should('have.attr', 'aria-valuenow', '96.4') + .find('div') + .should('have.css', 'background-color', 'rgb(1, 2, 3)'); + exhibit('power-reading').should( + 'contain', + 'All 3 measured GPU types ran at 92–97% of their enforced power limit', + ); + }); + it('tabulates the concurrency plateau with ratios against each hardware C1 cell', () => { + mount(); + exhibit('plateau').find('tbody tr').should('have.length', 9); + exhibit('plateau').find('tbody th[scope="rowgroup"]').should('have.length', 3); + exhibit('plateau') + .find('tr[data-hardware="b200"][data-concurrency="4"]') + .should('contain', '11.69') + .and('contain', '307.3') + .and('contain', '1.01×') + .and('contain', '3.94×'); + exhibit('plateau') + .find('tr[data-hardware="h100"][data-concurrency="4"]') + .should('contain', '5.37') + .and('contain', '669.2') + .and('contain', '1.00×') + .and('contain', '4.00×'); + exhibit('plateau') + .find('tr[data-hardware="h200"][data-concurrency="1"]') + .should('contain', '5.97') + .and('contain', '150.6'); + exhibit('plateau-reading') + .should('contain', 'Across 6 queued cells') + .and('contain', 'within 1.4% of C1') + .and('contain', '1.97–2.00× at C2 and 3.94–4.00× at C4') + .and('contain', 'queues requests instead of batching them'); + }); + it('compares observed speedups with spec-sheet ratios and hedges when they do not separate', () => { + mount(); + exhibit('scaling').find('[data-testid="video-evidence-scaling-row"]').should('have.length', 2); + exhibit('scaling') + .find('[data-pair="h100-h200"]') + .should('contain', 'H100') + .and('contain', 'H200') + .and('contain', '1.11×') + .and('contain', '1.43×') + .and('contain', '1.00×') + .and('contain', 'closer to the FLOPS ratio'); + exhibit('scaling') + .find('[data-pair="h200-b200"]') + .should('contain', '1.93×') + .and('contain', '1.67×') + .and('contain', '2.28×') + .and('contain', '2.27×') + .and('contain', 'between the two, not clearly closer to either'); + exhibit('scaling-reading') + .should('contain', 'H100 → H200: 1.11× observed vs. 1.43× bandwidth and 1.00× FLOPS') + .and( + 'contain', + 'H200 → B200: 1.93× observed vs. 1.67× bandwidth and 2.28× (BF16) / 2.27× (FP8) FLOPS', + ) + .and('contain', 'A bandwidth-bound workload would track the bandwidth ratio'); + exhibit('caveats') + .should('contain', 'Batch-one, single-replica server') + .and('contain', '4 participating GPUs per video (TP2 × Ulysses 2)') + .and('contain', 'not node, rack or facility power') + .and('contain', 'n = 20 clips per cell') + .and('contain', 'Frozen workload 1344 × 768 · 8 s · 24 fps · 50 steps'); + }); + it('switches the throughput column to the allocated basis without moving the ratios', () => { + mount('/video', 'allocated'); + exhibit('plateau') + .find('tr[data-hardware="h100"][data-concurrency="1"]') + .should('contain', '2.69') + .and('contain', '1.00×'); + exhibit('plateau') + .find('tr[data-hardware="h100"][data-concurrency="4"]') + .should('contain', '2.68') + .and('contain', '4.00×'); + exhibit('plateau-reading').should('contain', 'within 1.4% of C1'); + }); + it('renders Chinese copy on /zh/video', () => { + mount('/zh/video'); + cy.get('[data-testid="video-evidence"]') + .should('contain', '算力受限(compute-bound)的证据') + .and('contain', '板卡功率 vs. 生效功率上限') + .and('contain', '客户端并发平台期') + .and('contain', '实测加速比 vs. 规格表比值'); + exhibit('power-reading').should( + 'contain', + '全部 3 种实测硬件生成期间均运行在生效功率上限的 92%–97%', + ); + exhibit('plateau-reading') + .should('contain', '偏差不超过 1.4%') + .and('contain', 'C2 时 1.97–2.00×、C4 时 3.94–4.00×'); + exhibit('scaling') + .should('contain', '更接近 FLOPS 比值') + .and('contain', '介于两者之间,不明显偏向任一方'); + exhibit('caveats') + .should('contain', '每条视频由 4 张 GPU 参与计算(TP2 × Ulysses 2)') + .and('contain', '每个 cell n = 20 条视频'); + }); + it('hides the plateau and scaling exhibits when only one hardware at C1 is measured', () => { + cy.fixture('api/video-history.json').then((page: VideoHistoryPage) => { + cy.mount( + + p.hardwareKey === 'h200' && p.concurrency === 1, + )} + options={{ tier: 'h', basis: 'participating' }} + colorFor={() => 'rgb(0, 0, 0)'} + /> + , + ); + }); + exhibit('power').find('[data-testid="video-evidence-power-row"]').should('have.length', 1); + exhibit('power-reading').should('contain', 'H200 ran at 97% of its enforced power limit'); + exhibit('plateau').should('not.exist'); + exhibit('scaling').should('not.exist'); + exhibit('caveats').should('contain', '4 participating GPUs per video (TP2 × Ulysses 2)'); + }); + it('renders nothing when no cell is measured', () => { + cy.mount( + + 'rgb(0, 0, 0)'} + /> + , + ); + cy.get('[data-testid="video-evidence"]').should('not.exist'); + }); +}); From 3172afa041373a0b3801737b3b3a844b3ebc3555 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:51:18 -0700 Subject: [PATCH 030/104] =?UTF-8?q?fix:=20bound=20the=20power=20meter=20AR?= =?UTF-8?q?IA=20value=20to=20its=20declared=200=E2=80=93100=20range?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fill width was clamped to 100% but aria-valuenow was not, so a mean board power above the recorded enforced limit emitted a range-widget value outside aria-valuemax. One meterPercent helper in evidence.ts now drives both the fill and the ARIA value at 0.1% resolution; the label beside the meter keeps the measured share. Regression: an over-cap H200 C1 cell in the component spec fails on the old markup ('102' vs '100') and passes now. 中文:将功率仪表的 ARIA 值限制在其声明的 0–100 范围内。此前填充宽度已截断到 100%,但 aria-valuenow 未截断,板卡平均功率超过记录的生效上限时会输出超出 aria-valuemax 的值。现由 evidence.ts 中的 meterPercent 统一驱动填充与 ARIA 值, 标签仍显示实测比例;新增超上限 H200 C1 cell 的单元测试与组件回归用例。 --- .../cypress/component/video-evidence.cy.tsx | 31 +++++++++++++++++-- .../video-benchmark/VideoEvidence.tsx | 5 +-- .../video-benchmark/evidence.test.ts | 11 +++++++ .../components/video-benchmark/evidence.ts | 10 ++++++ 4 files changed, 52 insertions(+), 5 deletions(-) diff --git a/packages/app/cypress/component/video-evidence.cy.tsx b/packages/app/cypress/component/video-evidence.cy.tsx index 00792f0f8..c21d7d287 100644 --- a/packages/app/cypress/component/video-evidence.cy.tsx +++ b/packages/app/cypress/component/video-evidence.cy.tsx @@ -1,16 +1,20 @@ import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; import type { VideoHistoryPage } from '@/components/video-benchmark/history'; -import type { GpuBasis } from '@/components/video-benchmark/metrics'; +import type { GpuBasis, VideoPoint } from '@/components/video-benchmark/metrics'; import { videoPoints } from '@/components/video-benchmark/points'; import VideoEvidence from '@/components/video-benchmark/VideoEvidence'; // Retained H100/H200/B200 C1/C2/C4 observations (cypress/fixtures/api/video-history.json). -function mount(pathname = '/video', basis: GpuBasis = 'participating') { +function mount( + pathname = '/video', + basis: GpuBasis = 'participating', + select = (points: VideoPoint[]) => points, +) { cy.fixture('api/video-history.json').then((page: VideoHistoryPage) => { cy.mount( (key === 'b200' ? 'rgb(1, 2, 3)' : 'rgb(4, 5, 6)')} /> @@ -44,6 +48,27 @@ describe('Video compute-bound evidence (retained fixture)', () => { 'All 3 measured GPU types ran at 92–97% of their enforced power limit', ); }); + it('bounds the power meter to its declared range when mean power exceeds the recorded limit', () => { + // 2,856 W against the recorded 2,800 W limit: the label keeps the measured 102.0%, + // the meter reports and fills its 0–100 range. + mount('/video', 'participating', (points) => + points + .filter((p) => p.hardwareKey === 'h200' && p.concurrency === 1) + .map((p) => ({ ...p, avgPowerW: 2856 })), + ); + exhibit('power') + .find('[data-hardware="h200"]') + .should('contain', '2,856 W / 2,800 W') + .and('contain', '102.0%'); + exhibit('power') + .find('[data-hardware="h200"] [role="meter"]') + .should('have.attr', 'aria-valuemax', '100') + .and('have.attr', 'aria-valuenow', '100') + .find('div') + .invoke('attr', 'style') + .should('match', /width:\s*100%/u); + exhibit('power-reading').should('contain', 'H200 ran at 102% of its enforced power limit'); + }); it('tabulates the concurrency plateau with ratios against each hardware C1 cell', () => { mount(); exhibit('plateau').find('tbody tr').should('have.length', 9); diff --git a/packages/app/src/components/video-benchmark/VideoEvidence.tsx b/packages/app/src/components/video-benchmark/VideoEvidence.tsx index cf98e1410..63dbc4036 100644 --- a/packages/app/src/components/video-benchmark/VideoEvidence.tsx +++ b/packages/app/src/components/video-benchmark/VideoEvidence.tsx @@ -7,6 +7,7 @@ import { useLocale } from '@/lib/use-locale'; import { concurrencyPlateau, evidenceFacts, + meterPercent, plateauSummary, powerRange, powerUtilization, @@ -346,13 +347,13 @@ export default function VideoEvidence({ points, options, colorFor }: VideoEviden aria-label={`${hardwareLabel(row.hardwareKey)} ${s.meter}`} aria-valuemin={0} aria-valuemax={100} - aria-valuenow={Math.round(row.percentOfCap * 10) / 10} + aria-valuenow={meterPercent(row.percentOfCap)} className="h-2 w-full overflow-hidden rounded-full bg-muted" >
diff --git a/packages/app/src/components/video-benchmark/evidence.test.ts b/packages/app/src/components/video-benchmark/evidence.test.ts index 6002cc694..d28e425b3 100644 --- a/packages/app/src/components/video-benchmark/evidence.test.ts +++ b/packages/app/src/components/video-benchmark/evidence.test.ts @@ -3,6 +3,7 @@ import fixture from '../../../cypress/fixtures/api/video-history.json'; import { concurrencyPlateau, evidenceFacts, + meterPercent, parseBandwidthTbps, plateauSummary, powerRange, @@ -61,6 +62,16 @@ describe('powerUtilization', () => { expect(powerUtilization([{ ...cell('h100', 1), avgPowerW: 0 }])[0].percentOfCap).toBeNull(); expect(powerUtilization(points.filter((p) => p.concurrency !== 1))).toEqual([]); }); + it('keeps an over-cap share as measured while the meter position stays within 0–100', () => { + const [over] = powerUtilization([{ ...cell('h200', 1), avgPowerW: 2856 }]); + expect(over.enforcedLimitW).toBe(2800); + expect(over.percentOfCap).toBeCloseTo(102, 6); + expect(meterPercent(102)).toBe(100); + expect(meterPercent(100.04)).toBe(100); + expect(meterPercent(97.41)).toBe(97.4); + expect(meterPercent(91.948)).toBe(91.9); + expect(meterPercent(0)).toBe(0); + }); }); describe('concurrencyPlateau', () => { diff --git a/packages/app/src/components/video-benchmark/evidence.ts b/packages/app/src/components/video-benchmark/evidence.ts index 8af5283be..c9784eb90 100644 --- a/packages/app/src/components/video-benchmark/evidence.ts +++ b/packages/app/src/components/video-benchmark/evidence.ts @@ -69,6 +69,16 @@ export function powerRange(rows: PowerUtilizationRow[]): PowerRange | null { return { measured: values.length, min: Math.min(...values), max: Math.max(...values) }; } +/** + * Meter position for a share of the enforced limit. The label beside the meter + * keeps the measured share, which can exceed 100 % when the mean-power window + * and the limit recorded around the run skew; the widget declares 0–100, so its + * fill and ARIA value stay inside that range at 0.1 % resolution. + */ +export function meterPercent(percentOfCap: number): number { + return Math.min(100, Math.max(0, Math.round(percentOfCap * 10) / 10)); +} + export interface ConcurrencyPlateauRow { hardwareKey: string; concurrency: number; From 14963654cb8199381643520be7cd069ebb015e4d Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 20:51:51 -0700 Subject: [PATCH 031/104] docs: note VideoEvidence reads only the basis from its metric options MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The panel takes the same MetricOptions as the sibling panels so the dashboard can pass one object, but nothing in it is priced, so the cost tier is never read. Say so on the prop instead of leaving it implicit. 中文:说明 VideoEvidence 只读取 metric options 中的 basis。该面板与同级面板共用同一 MetricOptions 类型以便仪表板统一传参,但其中没有任何计价指标,因此不会读取 成本 tier;在 prop 上明确写出,而不是留作隐含约定。 --- packages/app/src/components/video-benchmark/VideoEvidence.tsx | 1 + 1 file changed, 1 insertion(+) diff --git a/packages/app/src/components/video-benchmark/VideoEvidence.tsx b/packages/app/src/components/video-benchmark/VideoEvidence.tsx index 63dbc4036..828806b72 100644 --- a/packages/app/src/components/video-benchmark/VideoEvidence.tsx +++ b/packages/app/src/components/video-benchmark/VideoEvidence.tsx @@ -197,6 +197,7 @@ const CELL = 'border-t border-border/40 py-1.5 pr-3'; export interface VideoEvidenceProps { points: VideoPoint[]; + /** Shared with the sibling panels; only `basis` is read, since nothing here is priced. */ options: MetricOptions; colorFor: (hardwareKey: string) => string; } From 67e91ce173032678a423c63fd55126c48e4e2ef3 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 21:07:09 -0700 Subject: [PATCH 032/104] refactor: pick compare and evidence cells by deployment layout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Compare offers one cell per hardware, its most efficient non-queued deployment, and the evidence exhibits read every hardware on the layout most of them share (sharedLayoutCells) so power, plateau and spec-scaling compare like with like once the GPUs-per-video sweep adds layouts. Queued cells never qualify; a hardware with no non-queued cell keeps its plateau rows with null ratios. 中文:Compare 每种硬件只提供一个 cell(最高效的非排队部署);证据面板改为读取各硬件 共有的部署布局(sharedLayoutCells),这样 GPUs-per-video 扫描加入新布局后,功耗、 排队平台与规格缩放仍是同类比较。排队 cell 不参与;没有非排队 cell 的硬件保留其 平台行,比值为 null。 --- .../src/components/video-benchmark/compare.ts | 15 ++++-- .../video-benchmark/deployment.test.ts | 26 +++++++++- .../components/video-benchmark/deployment.ts | 26 ++++++++++ .../components/video-benchmark/evidence.ts | 50 ++++++++++--------- 4 files changed, 88 insertions(+), 29 deletions(-) diff --git a/packages/app/src/components/video-benchmark/compare.ts b/packages/app/src/components/video-benchmark/compare.ts index cd6d83d54..753e5491d 100644 --- a/packages/app/src/components/video-benchmark/compare.ts +++ b/packages/app/src/components/video-benchmark/compare.ts @@ -6,6 +6,7 @@ import { type MetricOptions, type VideoPoint, } from './metrics'; +import { leadCell } from './deployment'; import { latestVideoCells } from './points'; import { servingCells, type ServingCell } from './serving'; import { storedBundle, type StoredArtifact, type StoredSource } from './stored'; @@ -34,11 +35,17 @@ export interface CompareRow { candidateBetter: boolean | null; } -/** Measured C1 cells a reader can pick for comparison: registry hardware with a P50 (newest per hardware). */ +/** + * One cell a reader can pick per registry hardware: its most efficient + * non-queued deployment (chip basis) with a P50. Queued cells never qualify. + */ export function comparablePoints(points: VideoPoint[]): VideoPoint[] { - return latestVideoCells(points).filter( - (p) => p.concurrency === 1 && p.hardwareKey !== null && p.p50 !== null, - ); + const cells = latestVideoCells(points); + const keys = [...new Set(cells.flatMap((p) => (p.hardwareKey === null ? [] : [p.hardwareKey])))]; + return keys.flatMap((key) => { + const lead = leadCell(cells, key, { tier: 'h', basis: 'participating' }); + return lead && lead.p50 !== null ? [lead] : []; + }); } /** Side-by-side deltas; every null input stays null (never 0). */ diff --git a/packages/app/src/components/video-benchmark/deployment.test.ts b/packages/app/src/components/video-benchmark/deployment.test.ts index 37f3586d8..726f41fb7 100644 --- a/packages/app/src/components/video-benchmark/deployment.test.ts +++ b/packages/app/src/components/video-benchmark/deployment.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { deploymentKey, isQueueing, layoutLabel, leadCell } from './deployment'; +import { deploymentKey, isQueueing, layoutLabel, leadCell, sharedLayoutCells } from './deployment'; import type { VideoPoint } from './metrics'; const base: VideoPoint = { @@ -60,6 +60,30 @@ describe('deployment', () => { expect(layoutLabel({ ...base, server: null }, 'en')).toBe('4 GPU'); expect(layoutLabel({ ...base, participating: null, server: null }, 'en')).toBe('—'); }); + it('picks the layout most hardware share, falling back to the lead cell', () => { + const b200Eight = { + ...base, + id: 'b8', + hardwareKey: 'b200', + participating: 8, + server: { tp: 4, ulysses: 2, attention: null }, + wallSeconds: 900, + }; + const b200Four = { ...base, id: 'b4', hardwareKey: 'b200', wallSeconds: 1500 }; + const h100Two = { + ...base, + id: 'h2', + hardwareKey: 'h100', + participating: 2, + server: { tp: 2, ulysses: 1, attention: null }, + }; + const queued = { ...base, id: 'q', concurrency: 2 }; + // 4g:tp2:u2 is shared by h200 and b200; h100 only has a 2-GPU cell and keeps it. + expect( + sharedLayoutCells([base, queued, b200Eight, b200Four, h100Two]).map((p) => p.id), + ).toEqual(['a', 'b4', 'h2']); + expect(sharedLayoutCells([])).toEqual([]); + }); it('leads with the most efficient non-queued deployment of the hardware', () => { const eightBoards = { ...base, diff --git a/packages/app/src/components/video-benchmark/deployment.ts b/packages/app/src/components/video-benchmark/deployment.ts index 9aa17b6e9..02ffbca0e 100644 --- a/packages/app/src/components/video-benchmark/deployment.ts +++ b/packages/app/src/components/video-benchmark/deployment.ts @@ -36,6 +36,32 @@ export function layoutLabel(p: Layout, locale: 'en' | 'zh'): string { return parts.length > 0 ? parts.join(' · ') : '—'; } +/** + * One non-queued cell per hardware on the deployment layout that the most + * hardware share (ties keep the first layout in cell order), so cross-hardware + * evidence compares like with like. A hardware without that layout falls back + * to its lead cell. Cells arrive in hardware order and stay in it. + */ +export function sharedLayoutCells(cells: T[]): T[] { + const deployments = cells.filter((p) => p.hardwareKey !== null && !isQueueing(p)); + const hardwareByLayout = new Map>(); + for (const p of deployments) { + const set = hardwareByLayout.get(deploymentKey(p)) ?? new Set(); + set.add(p.hardwareKey!); + hardwareByLayout.set(deploymentKey(p), set); + } + let shared: string | null = null; + for (const [key, set] of hardwareByLayout) + if (shared === null || set.size > hardwareByLayout.get(shared)!.size) shared = key; + const keys = [...new Set(deployments.map((p) => p.hardwareKey!))]; + return keys.flatMap((hardwareKey) => { + const cell = + deployments.find((p) => p.hardwareKey === hardwareKey && deploymentKey(p) === shared) ?? + leadCell(deployments, hardwareKey, { tier: 'h', basis: 'participating' }); + return cell ? [cell as T] : []; + }); +} + /** * The cell that stands for a hardware in cards and comparisons: its most * efficient deployment under the current GPU basis (highest videos per diff --git a/packages/app/src/components/video-benchmark/evidence.ts b/packages/app/src/components/video-benchmark/evidence.ts index c9784eb90..57bcaa545 100644 --- a/packages/app/src/components/video-benchmark/evidence.ts +++ b/packages/app/src/components/video-benchmark/evidence.ts @@ -1,4 +1,5 @@ import { GPU_SPECS } from '@/lib/gpu-specs'; +import { deploymentKey, sharedLayoutCells } from './deployment'; import { hardwareSort } from './hardware'; import { metricValue, type GpuBasis, type MetricOptions, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; @@ -43,16 +44,14 @@ export interface PowerUtilizationRow { percentOfCap: number | null; } -/** One row per measured hardware at C1. */ +/** One row per measured hardware, on the deployment layout the hardware share. */ export function powerUtilization(points: VideoPoint[]): PowerUtilizationRow[] { - return measuredCells(points) - .filter((p) => p.concurrency === 1) - .map((p) => ({ - hardwareKey: p.hardwareKey, - avgPowerW: positive(p.avgPowerW) ? p.avgPowerW : null, - enforcedLimitW: positive(p.enforcedLimitW) ? p.enforcedLimitW : null, - percentOfCap: metricValue(p, 'powerPctCap', metricOptions('participating')), - })); + return sharedLayoutCells(measuredCells(points)).map((p) => ({ + hardwareKey: p.hardwareKey, + avgPowerW: positive(p.avgPowerW) ? p.avgPowerW : null, + enforcedLimitW: positive(p.enforcedLimitW) ? p.enforcedLimitW : null, + percentOfCap: metricValue(p, 'powerPctCap', metricOptions('participating')), + })); } export interface PowerRange { @@ -100,11 +99,16 @@ export function concurrencyPlateau( points: VideoPoint[], basis: GpuBasis = 'participating', ): ConcurrencyPlateauRow[] { - const cells = measuredCells(points); const options = metricOptions(basis); const baseline = new Map( - cells.filter((p) => p.concurrency === 1).map((p) => [p.hardwareKey, p] as const), + sharedLayoutCells(measuredCells(points)).map((p) => [p.hardwareKey, p] as const), ); + // Only cells of the baseline's own layout are concurrency steps of that deployment; + // a hardware with no non-queued cell keeps its rows with null ratios. + const cells = measuredCells(points).filter((p) => { + const base = baseline.get(p.hardwareKey); + return base === undefined || deploymentKey(p) === deploymentKey(base); + }); return cells.map((p) => { const base = baseline.get(p.hardwareKey); const videosPerGpuHour = metricValue(p, 'videosPerGpuHour', options); @@ -204,13 +208,12 @@ export interface ScalingVsSpecRow { } type TimedPoint = MeasuredPoint & { p50: number }; -const isTimedBaseline = (p: MeasuredPoint): p is TimedPoint => - p.concurrency === 1 && positive(p.p50); +const isTimed = (p: MeasuredPoint): p is TimedPoint => positive(p.p50); -/** Consecutive measured hardware from slowest to fastest C1 P50, each step against its spec ratios. */ +/** Consecutive measured hardware from slowest to fastest P50 on the shared layout, each step against its spec ratios. */ export function scalingVsSpec(points: VideoPoint[]): ScalingVsSpecRow[] { - const ordered = measuredCells(points) - .filter(isTimedBaseline) + const ordered = sharedLayoutCells(measuredCells(points)) + .filter(isTimed) .toSorted((a, b) => b.p50 - a.p50); return ordered.slice(1).map((to, index) => { const from = ordered[index]; @@ -283,16 +286,15 @@ function uniform(values: (T | null)[]): T | null { /** Caveat inputs read from the cells so the panel never states a number the data does not carry. */ export function evidenceFacts(points: VideoPoint[]): EvidenceFacts { const cells = measuredCells(points); + const shared = sharedLayoutCells(cells); const samples = cells.map((p) => p.samples).filter(positive); - const attention = cells - .filter((p) => p.concurrency === 1) - .flatMap((p) => - p.server?.attention ? [{ hardwareKey: p.hardwareKey, attention: p.server.attention }] : [], - ); + const attention = shared.flatMap((p) => + p.server?.attention ? [{ hardwareKey: p.hardwareKey, attention: p.server.attention }] : [], + ); return { - participating: uniform(cells.map((p) => (positive(p.participating) ? p.participating : null))), - tp: uniform(cells.map((p) => (positive(p.server?.tp) ? p.server.tp : null))), - ulysses: uniform(cells.map((p) => (positive(p.server?.ulysses) ? p.server.ulysses : null))), + participating: uniform(shared.map((p) => (positive(p.participating) ? p.participating : null))), + tp: uniform(shared.map((p) => (positive(p.server?.tp) ? p.server.tp : null))), + ulysses: uniform(shared.map((p) => (positive(p.server?.ulysses) ? p.server.ulysses : null))), samples: samples.length > 0 ? { min: Math.min(...samples), max: Math.max(...samples) } : null, workload: uniform(cells.map((p) => p.workload.split(' · ').slice(0, 4).join(' · ') || null)), attention: uniform(attention.map((entry) => entry.attention)) === null ? attention : [], From e7b4be93c20d737798ba3938b24a0e7b961c4946 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 21:07:18 -0700 Subject: [PATCH 033/104] feat: wire the API reference, Compare and Evidence panels into the video dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Metric options now come from the URL state (tier, GPU basis and the editable API price), the subtitle states the dated API reference, the CSV export carries the four API-priced metrics and the price assumption, and the Compare and compute-bound Evidence panels mount under the KPI cards once results have loaded. The loading-placeholder component test resets the URL it inherits from the previous test. 中文:指标选项统一来自 URL 状态(成本档位、GPU 口径、可编辑的 API 参考价),副标题 给出带日期的 API 参考价,CSV 导出加入四个按 API 标价计算的指标及价格假设;结果加载 完成后在 KPI 卡下方挂载 Compare 与 compute-bound 证据面板。加载态组件测试会重置 上一个测试留下的 URL。 --- .../cypress/component/video-dashboard.cy.tsx | 2 ++ .../video-benchmark/VideoDashboard.tsx | 24 ++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index 9a2a1d6c1..d5e940ffe 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -169,6 +169,8 @@ describe('Video hardware dashboard (retained fixture)', () => { req.reply({ fixture: 'api/video-history.json', delay: 800 }); }).as('slowHistory'); cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + // The AUT keeps the previous test's v_ params; start from the defaults. + cy.window().then((win) => win.history.replaceState(null, '', win.location.pathname)); cy.mount( diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx index e3bacacb1..b83d40374 100644 --- a/packages/app/src/components/video-benchmark/VideoDashboard.tsx +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -24,14 +24,18 @@ import { type VideoPoint, } from './metrics'; import { latestVideoCells } from './points'; +import { formatApiPrice, H3_API_REFERENCE } from './api-reference'; import { useVideoPoints } from './use-video-points'; import VideoCIRuns from './VideoCIRuns'; +import VideoCompare from './VideoCompare'; import VideoConfigBar from './VideoConfigBar'; +import VideoEvidence from './VideoEvidence'; import VideoHardwareChart, { VIDEO_CHART_ID } from './VideoHardwareChart'; import VideoKpiCards from './VideoKpiCards'; import VideoPointsTable, { videoTableRows } from './VideoPointsTable'; import { DEFAULT_VIDEO_DASHBOARD_STATE, + metricOptions, readVideoDashboardState, writeVideoDashboardState, type VideoDashboardState, @@ -61,6 +65,10 @@ const CSV_METRICS: readonly MetricId[] = [ 'kjPerVideo', 'videosPerKwh', 'powerPctCap', + 'apiPricePerVideo', + 'revenuePerGpuHour', + 'profitPerGpuHour', + 'apiPriceMultiple', ]; const STRINGS = { @@ -74,6 +82,7 @@ const STRINGS = { tier: 'Cost tier', badges: 'TCO $/chip/hr', source: 'Source', + apiReference: 'API reference', queue: 'Show queued requests (C2/C4)', optimal: 'Optimal only', frontier: 'Pareto frontier', @@ -99,6 +108,7 @@ const STRINGS = { tier: '成本档位', badges: 'TCO $/chip/hr', source: '来源', + apiReference: 'API 参考价', queue: '显示排队请求(C2/C4)', optimal: '仅最优', frontier: 'Pareto 前沿', @@ -163,7 +173,7 @@ export default function VideoDashboard() { [mounted, resolveColor, getCssColor], ); - const options = { tier: state.tier, basis: state.basis }; + const options = metricOptions(state); const cells = useMemo(() => latestVideoCells(points), [points]); // Each hardware is represented by its most efficient measured deployment. const measured = useMemo( @@ -224,6 +234,7 @@ export default function VideoDashboard() { 'scheduled', 'participating_gpus', 'allocated_gpus', + 'api_price_usd_per_video_second', 'tp_size', 'ulysses_degree', 'replicas', @@ -242,6 +253,7 @@ export default function VideoDashboard() { p.scheduled, p.participating, p.allocated, + state.apiPrice, p.server?.tp ?? null, p.server?.ulysses ?? null, p.replicas, @@ -330,6 +342,10 @@ export default function VideoDashboard() { ))} + + {s.apiReference}: {formatApiPrice(state.apiPrice)}/video-s ( + {H3_API_REFERENCE.capturedOn}) + {s.source}:{' '} + {!loading && !error && ( + <> + + + + )}
Date: Sat, 19 Sep 2026 21:08:26 -0700 Subject: [PATCH 034/104] docs: describe the API reference, Compare and Evidence sections of the video dashboard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文:在 /video 设计文档中补充 API 参考价、Compare 与 compute-bound 证据三个区块的说明。 --- docs/videogenx-dashboard.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md index 0bce20b01..7a7813e6a 100644 --- a/docs/videogenx-dashboard.md +++ b/docs/videogenx-dashboard.md @@ -24,6 +24,12 @@ The knob that does trade latency for efficiency is the **deployment**: how many Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 × Ulysses 2), so the chart is an honest scatter and the caption says a per-hardware curve needs the GPUs-per-video sweep. That sweep (design plan P6) is backend work in InferenceX `experimental/video-generation` — the serving matrix currently varies only concurrency — and needs GPU authorisation; the frontend needs no change when its cells arrive, because they differ in `deploymentKey` and therefore add points rather than replace them. +## Sections below the chart + +- **API price reference** (`api-reference.ts`, `VideoApiReference.tsx`): a dated, editable USD per video-second list price (`v_api`, default `H3_API_REFERENCE`) feeds the API-priced metrics in `metrics.ts` — API $/video, revenue and profit per GPU-hour, API price ÷ TCO multiple. It is a web-search reference, labelled with its capture date and range, never presented as an official quote; a null or non-positive price nulls every derived metric. The chart's y axis is zero-anchored (`chart-domain.ts`) so negative profit stays visible and an all-negative view tops out at break-even. +- **Compare** (`compare.ts`, `compare-url-state.ts`, `VideoCompare.tsx`): arena-style baseline vs. candidate — one selectable cell per hardware (its lead deployment), metric deltas by registry polarity, and the same prompt + seed clips loaded on demand from the published artifacts (`v_base`, `v_cand`, `v_case`). +- **Compute-bound evidence** (`evidence.ts`, `VideoEvidence.tsx`): board power against the recorded enforced limit, the concurrency plateau (throughput flat, latency × C), and observed speed-ups against memory-bandwidth and FLOPS spec ratios. Every hardware is read on the deployment layout most of them share (`sharedLayoutCells`) so the exhibits compare like with like; sentences stay conditional when ratios do not separate. + ## Rules - **Unavailable hardware** stays visible. `VIDEO_HARDWARE_ROSTER` in `hardware.ts` lists the campaign targets; MI355X has no valid run and renders as a muted legend row, a "Not measured" KPI card with the failed-run link, and no table row. A missing point is never a zero. @@ -37,3 +43,5 @@ Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 `/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。默认 X 为 P90 出片时间、Y 为每 1 美元 TCO 视频数(Hyperscaler 自有设备档位,单价与 `/inference` 同源)。 曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表;`plot.ts` 按硬件对部署点求 Pareto 前沿,达到两个及以上前沿点时才画实线,被支配的部署以淡色显示,“仅最优”可将其隐藏;排队点需手动开启,只画小号淡色标记、不连线;“Pareto 前沿”开关叠加跨硬件虚线。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 + +图表下方还有三个区块:带日期、可编辑的 API 参考价(`v_api`,来自网页搜索,标注采集日期与区间,不是官方报价)驱动 API 标价 / 条视频、每 GPU 小时收入与利润、API 标价 ÷ TCO 倍数等指标,价格无效时全部为 null,y 轴锚定 0 以便显示负利润;arena 式 Compare 按硬件选基线与候选(各取代表部署),给出按指标极性判定的差值,并按需加载同一 prompt 与 seed 的视频;compute-bound 证据面板展示板卡功率相对生效上限、并发排队平台以及相对带宽/FLOPS 规格比的实测加速比,所有硬件都读取它们共有的部署布局,比值无法区分时措辞保持保守。 From d0a5982e15d2fc863a08f9baf1d3abf5cefcb153 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 21:11:00 -0700 Subject: [PATCH 035/104] fix: keep the API price legible and describe Compare cells as lead deployments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The number input shrank to one digit beside its Reset button inside the four-column benchmark panel; it now keeps a minimum width and the pair wraps instead of clipping. The Compare intro no longer calls its cells "measured C1 cells": each hardware is compared on its most efficient measured deployment. 中文:API 参考价的数字输入框在四列基准配置面板中被 Reset 按钮挤到只剩一位数字,现在保留 最小宽度并在必要时换行。Compare 说明不再写“已实测的 C1 单元”,改为各硬件最高效的实测部署。 --- .../src/components/video-benchmark/VideoApiReference.tsx | 5 +++-- .../app/src/components/video-benchmark/VideoCompare.tsx | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/packages/app/src/components/video-benchmark/VideoApiReference.tsx b/packages/app/src/components/video-benchmark/VideoApiReference.tsx index b128f84ba..72ec1f76b 100644 --- a/packages/app/src/components/video-benchmark/VideoApiReference.tsx +++ b/packages/app/src/components/video-benchmark/VideoApiReference.tsx @@ -70,7 +70,7 @@ export default function VideoApiReference({ -
+
{ valueAtFocus.current = value; }} @@ -102,6 +102,7 @@ export default function VideoApiReference({ variant="outline" title={s.resetTitle} data-testid="video-api-price-reset" + className="shrink-0" onClick={() => { setRaw(String(reference)); commit(reference, true); diff --git a/packages/app/src/components/video-benchmark/VideoCompare.tsx b/packages/app/src/components/video-benchmark/VideoCompare.tsx index fd77596e2..72a0fa575 100644 --- a/packages/app/src/components/video-benchmark/VideoCompare.tsx +++ b/packages/app/src/components/video-benchmark/VideoCompare.tsx @@ -37,7 +37,7 @@ const STRINGS = { en: { title: 'Compare', subtitle: - 'Pick a baseline and a candidate hardware (measured C1 cells): metric deltas side by side, then the same generated case from both runs.', + 'Pick a baseline and a candidate hardware (each on its most efficient measured deployment): metric deltas side by side, then the same generated case from both runs.', baseline: 'Baseline', candidate: 'Candidate', needTwo: 'Comparison needs at least two measured hardware.', @@ -67,7 +67,7 @@ const STRINGS = { zh: { title: '对比', subtitle: - '选择基线与候选硬件(已实测的 C1 单元):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', + '选择基线与候选硬件(各取其实测中最高效的部署):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', baseline: '基线', candidate: '候选', needTwo: '至少需要两种已实测的硬件才能对比。', @@ -149,7 +149,7 @@ function DeltaChip({ row, s }: { row: CompareRow; s: Strings }) { } /** - * Arena-style baseline vs candidate: measured C1 metric deltas, then the + * Arena-style baseline vs candidate: metric deltas between each hardware's lead deployment, then the * retained clips of the same prompt and seed from both CI runs. Media is fetched * only on request, one stored artifact per side. */ From 19cd5f26fb2d2573e5785cf9b51c10e5baa9dc4a Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 21:14:24 -0700 Subject: [PATCH 036/104] fix: polish the Chinese copy of the video dashboard panels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apply the advisory review-zh-copy findings on the API reference, Compare and Evidence panels: one name for the cost tier, the API reference label uses the same /video-s unit as its caption, ratio headers read as multiples of C1, the P90 caveat says what the display threshold means, "recipe" becomes 实测配置, and the chart tooltip uses full-width punctuation in Chinese. 中文:按 review-zh-copy 的建议修订 API 参考价、Compare 与证据面板的中文:成本档位统一命名, API 参考价标签与下方说明同用 /video-s 单位,比值表头改为“相对 C1”,P90 注释说明显示门槛的 含义,“配方”改为“实测配置”,图表 tooltip 在中文下使用全角标点。 --- .../app/cypress/component/video-api-reference.cy.tsx | 4 ++-- packages/app/cypress/component/video-dashboard.cy.tsx | 2 +- .../components/video-benchmark/VideoApiReference.tsx | 7 ++++--- .../src/components/video-benchmark/VideoCompare.tsx | 4 ++-- .../src/components/video-benchmark/VideoEvidence.tsx | 10 +++++----- .../components/video-benchmark/VideoHardwareChart.tsx | 6 ++++-- .../src/components/video-benchmark/api-reference.ts | 2 +- 7 files changed, 19 insertions(+), 16 deletions(-) diff --git a/packages/app/cypress/component/video-api-reference.cy.tsx b/packages/app/cypress/component/video-api-reference.cy.tsx index 958303597..dac99d7a3 100644 --- a/packages/app/cypress/component/video-api-reference.cy.tsx +++ b/packages/app/cypress/component/video-api-reference.cy.tsx @@ -66,13 +66,13 @@ describe('VideoApiReference', () => { }); it('renders Chinese copy under /zh', () => { cy.mount(); - cy.contains('label', 'API 价格参考($/视频秒)').should('exist'); + cy.contains('label', 'API 参考价($/video-s)').should('exist'); cy.contains('button', '重置').should('exist'); caption() .should('contain', '参考值 $0.034/video-s') .and('contain', '标价区间 $0.034–$0.047') .and('contain', '采集于 2026-09-19') .and('contain', '来源:MiniMax Design · H3 768p 订阅档') - .and('contain', '为标价而非实际收入'); + .and('contain', '此处为 API 标价,而非实际收入'); }); }); diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index d5e940ffe..28fa4e182 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -193,7 +193,7 @@ describe('Video hardware dashboard (retained fixture)', () => { kpi('h200').should('contain', 'API 标价 / 条视频').and('contain', '利润 / GPU 小时'); cy.get('[data-testid="video-config-bar"]') .should('contain', '成本档位') - .and('contain', 'API 价格参考($/视频秒)') + .and('contain', 'API 参考价($/video-s)') .and('contain', '采集于 2026-09-19'); }); }); diff --git a/packages/app/src/components/video-benchmark/VideoApiReference.tsx b/packages/app/src/components/video-benchmark/VideoApiReference.tsx index 72ec1f76b..b4441a12f 100644 --- a/packages/app/src/components/video-benchmark/VideoApiReference.tsx +++ b/packages/app/src/components/video-benchmark/VideoApiReference.tsx @@ -21,14 +21,15 @@ const STRINGS = { 'List price, not realized revenue; self-hosted TCO excludes utilization, CPU, storage and network.', }, zh: { - label: 'API 价格参考($/视频秒)', + label: 'API 参考价($/video-s)', reset: '重置', - resetTitle: '恢复为带日期的参考价格', + resetTitle: '恢复为标注采集日期的参考价', reference: '参考值', range: '标价区间', captured: '采集于', source: '来源:', - disclaimer: '为标价而非实际收入;自托管 TCO 不含利用率、CPU、存储与网络成本。', + disclaimer: + '此处为 API 标价,而非实际收入;自托管 TCO 未计入利用率,也不含 CPU、存储与网络成本。', }, }; diff --git a/packages/app/src/components/video-benchmark/VideoCompare.tsx b/packages/app/src/components/video-benchmark/VideoCompare.tsx index 72a0fa575..bac93a70f 100644 --- a/packages/app/src/components/video-benchmark/VideoCompare.tsx +++ b/packages/app/src/components/video-benchmark/VideoCompare.tsx @@ -67,13 +67,13 @@ const STRINGS = { zh: { title: '对比', subtitle: - '选择基线与候选硬件(各取其实测中最高效的部署):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', + '选择基线与候选硬件(各自取实测中最高效的部署):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', baseline: '基线', candidate: '候选', needTwo: '至少需要两种已实测的硬件才能对比。', metric: '指标', change: '变化', - assumptions: (tier: string, basis: string) => `成本分档:${tier} · GPU 口径:${basis}。`, + assumptions: (tier: string, basis: string) => `成本档位:${tier} · GPU 口径:${basis}。`, participating: '参与计算的 GPU', allocated: '已分配的 GPU', better: '候选更优', diff --git a/packages/app/src/components/video-benchmark/VideoEvidence.tsx b/packages/app/src/components/video-benchmark/VideoEvidence.tsx index 828806b72..81c147061 100644 --- a/packages/app/src/components/video-benchmark/VideoEvidence.tsx +++ b/packages/app/src/components/video-benchmark/VideoEvidence.tsx @@ -98,7 +98,7 @@ const STRINGS = { '来自实测 cell 的三组读数,说明该工作负载在固定 runtime 上为何表现为算力受限的扩散模型。所有数字均由已发布运行计算得出,措辞以数字为准。', power: '板卡功率 vs. 生效功率上限', powerHint: - '生成期间参与计算板卡的 GPU 板卡平均功率,对照运行前后记录到的功率上限,而非标称 TDP。', + '生成期间各参与计算 GPU 板卡的平均功率,对照运行前后记录到的功率上限,而非标称 TDP。', powerOne: (hardware: string, share: string) => `${hardware} 生成期间运行在生效功率上限的 ${share}:接近上限,符合算力饱和型负载的表现。`, powerAll: (n: number, share: string) => @@ -114,8 +114,8 @@ const STRINGS = { c: 'C', rate: '视频数 / GPU 小时', p50: 'P50(s)', - throughput: '吞吐量 × C1', - latency: '延迟 × C1', + throughput: '吞吐量(相对 C1)', + latency: '延迟(相对 C1)', scaling: '实测加速比 vs. 规格表比值', scalingHint: '按 C1 P50 从慢到快排列的相邻硬件。实测值为 P50 比值;规格比值取自 GPU 规格页的单卡 HBM 带宽与 dense tensor-core TFLOPS。', @@ -151,13 +151,13 @@ const STRINGS = { recipe: (tp: number, ulysses: number) => `(TP${tp} × Ulysses ${ulysses})`, caveatPower: 'GPU 板卡级功率,对照记录到的生效上限;不是节点、机柜或设施功率。', caveatSamples: (n: string) => - `每个 cell n = ${n} 条视频、每种硬件一个批次:没有误差棒,P90 只是显示下限而非尾延迟 SLO。`, + `每个 cell n = ${n} 条视频、每种硬件一个批次:没有误差棒;P90 仅在样本数达到显示门槛时展示,不能当作尾延迟 SLO。`, caveatSamplesUnknown: '未记录样本数。', caveatWorkload: (workload: string) => `冻结的工作负载 ${workload};所有比值都是同一条视频在不同硬件上的对比。`, caveatWorkloadUnknown: '各 cell 的工作负载标签不一致;跨硬件比值只在同一条视频下成立。', caveatAttention: (backends: string) => - `各硬件的 attention 后端不同(${backends});每个数据点都是“硬件 + 实测配方”。`, + `各硬件的 attention 后端不同(${backends});每个数据点都是“硬件 + 实测配置”。`, }, }; diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx index de17f945b..07180120d 100644 --- a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -37,7 +37,7 @@ const STRINGS = { }, zh: { single: (layouts: string) => - `目前每种硬件只测得一种部署(${layouts}),因此每种硬件只有一个点;要得到每硬件的 Pareto 曲线,还需扫描每条视频占用的 GPU 数。`, + `目前每种硬件只测得一种部署(${layouts}),因此每种硬件只有一个点;要得到各硬件自己的 Pareto 曲线,还需扫描每条视频占用的 GPU 数。`, multi: '实线连接同一硬件的 Pareto 最优部署(每条视频占用的 GPU 数及模型切分方式);淡色点为被支配的部署。', global: '虚线为所有部署的跨硬件 Pareto 前沿。', @@ -89,6 +89,8 @@ export default function VideoHardwareChart({ const layouts = [ ...new Set(plotted.filter((p) => !p.queued).map((p) => layoutLabel(p, locale))), ].join(locale === 'zh' ? ';' : '; '); + const colon = locale === 'zh' ? ':' : ': '; + const queuedTag = locale === 'zh' ? `(${s.queuedTag})` : ` (${s.queuedTag})`; const pointLabel = (p: PlottedVideoPoint) => { if (p.queued) return `C${p.concurrency}`; return multiLayout ? `${p.label} · ${layoutLabel(p, locale)}` : p.label; @@ -196,7 +198,7 @@ export default function VideoHardwareChart({ tooltip={{ rulerType: 'none', content: (p) => - `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}${p.queued ? ` (${s.queuedTag})` : ''}
${s.deployment}: ${escapeHtml(layoutLabel(p, locale))}
${escapeHtml(xLabel)}: ${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}: ${formatMetric(p.y, state.y)}
${s.n}: ${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, + `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}${p.queued ? queuedTag : ''}
${s.deployment}${colon}${escapeHtml(layoutLabel(p, locale))}
${escapeHtml(xLabel)}${colon}${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}${colon}${formatMetric(p.y, state.y)}
${s.n}${colon}${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, onPointClick: onSelect, }} zoom={{ diff --git a/packages/app/src/components/video-benchmark/api-reference.ts b/packages/app/src/components/video-benchmark/api-reference.ts index 17d668a89..87636495f 100644 --- a/packages/app/src/components/video-benchmark/api-reference.ts +++ b/packages/app/src/components/video-benchmark/api-reference.ts @@ -21,7 +21,7 @@ export const H3_API_REFERENCE: ApiPriceReference = { capturedOn: '2026-09-19', label: { en: 'MiniMax Design · H3 768p subscription tier (web search, not verified on the pay-as-you-go page; H3 was listed as not yet supported on the API packages page)', - zh: 'MiniMax Design · H3 768p 订阅档(来源为网页搜索,未在按量付费页面核实;API 套餐页当时标注 H3 尚未支持)', + zh: 'MiniMax Design · H3 768p 订阅档(价格来自网页搜索,未在按量付费页面核实;API 套餐页当时标注 H3 尚未支持)', }, }; From 9e507dd2e5f7839687b8afa0cbc70e9885bf3335 Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sat, 19 Sep 2026 23:46:20 -0700 Subject: [PATCH 037/104] feat: make the video Compare panel an arena-style side-by-side player MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The clips are the point of the comparison, so they lead: the same prompt and seed from both CI runs play side by side with shared play, pause and restart, case arrows with a full prompt card, and a blind mode that labels the sides A and B (and hides the metric deltas) until revealed. Media loads on its own once the panel scrolls into view instead of behind a button; a 204 side reads as "clips not published" rather than an error. The delta table moves below the players. Component and e2e specs stub the clip requests so they stay offline. 中文:视频才是对比的核心,因此改为视频优先:同一 prompt 与 seed 在两种硬件上的视频并排 播放,支持同步播放/暂停/重播、用例翻页与完整 prompt 卡片,“盲测模式”把两侧标为 A/B 并隐藏指标差异直到揭晓。面板滚入视口时自动加载视频,不再需要点按钮;204 视为“未发布 视频”而非错误。指标差值表移到播放器下方。组件与 e2e 测试对视频请求打桩以保持离线。 --- docs/videogenx-dashboard.md | 4 +- .../cypress/component/video-compare.cy.tsx | 137 +++-- .../cypress/component/video-dashboard.cy.tsx | 3 + .../app/cypress/e2e/video-dashboard.cy.ts | 4 + .../video-benchmark/VideoCompare.tsx | 516 +++++++++++++----- 5 files changed, 488 insertions(+), 176 deletions(-) diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md index 7a7813e6a..1446fa891 100644 --- a/docs/videogenx-dashboard.md +++ b/docs/videogenx-dashboard.md @@ -27,7 +27,7 @@ Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 ## Sections below the chart - **API price reference** (`api-reference.ts`, `VideoApiReference.tsx`): a dated, editable USD per video-second list price (`v_api`, default `H3_API_REFERENCE`) feeds the API-priced metrics in `metrics.ts` — API $/video, revenue and profit per GPU-hour, API price ÷ TCO multiple. It is a web-search reference, labelled with its capture date and range, never presented as an official quote; a null or non-positive price nulls every derived metric. The chart's y axis is zero-anchored (`chart-domain.ts`) so negative profit stays visible and an all-negative view tops out at break-even. -- **Compare** (`compare.ts`, `compare-url-state.ts`, `VideoCompare.tsx`): arena-style baseline vs. candidate — one selectable cell per hardware (its lead deployment), metric deltas by registry polarity, and the same prompt + seed clips loaded on demand from the published artifacts (`v_base`, `v_cand`, `v_case`). +- **Compare** (`compare.ts`, `compare-url-state.ts`, `VideoCompare.tsx`): arena-style baseline vs. candidate, video first — the same prompt + seed clips from both CI runs play side by side with shared play/pause/restart, case arrows, and a blind mode that hides which hardware is which until revealed; the metric deltas (registry polarity) sit below. One selectable cell per hardware (its lead deployment); `v_base`, `v_cand`, `v_case` in the URL. The two stored artifacts (several MB each) load once the panel scrolls into view; an HTTP 204 side reads as "clips not published", never as an error. - **Compute-bound evidence** (`evidence.ts`, `VideoEvidence.tsx`): board power against the recorded enforced limit, the concurrency plateau (throughput flat, latency × C), and observed speed-ups against memory-bandwidth and FLOPS spec ratios. Every hardware is read on the deployment layout most of them share (`sharedLayoutCells`) so the exhibits compare like with like; sentences stay conditional when ratios do not separate. ## Rules @@ -44,4 +44,4 @@ Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表;`plot.ts` 按硬件对部署点求 Pareto 前沿,达到两个及以上前沿点时才画实线,被支配的部署以淡色显示,“仅最优”可将其隐藏;排队点需手动开启,只画小号淡色标记、不连线;“Pareto 前沿”开关叠加跨硬件虚线。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 -图表下方还有三个区块:带日期、可编辑的 API 参考价(`v_api`,来自网页搜索,标注采集日期与区间,不是官方报价)驱动 API 标价 / 条视频、每 GPU 小时收入与利润、API 标价 ÷ TCO 倍数等指标,价格无效时全部为 null,y 轴锚定 0 以便显示负利润;arena 式 Compare 按硬件选基线与候选(各取代表部署),给出按指标极性判定的差值,并按需加载同一 prompt 与 seed 的视频;compute-bound 证据面板展示板卡功率相对生效上限、并发排队平台以及相对带宽/FLOPS 规格比的实测加速比,所有硬件都读取它们共有的部署布局,比值无法区分时措辞保持保守。 +图表下方还有三个区块:带日期、可编辑的 API 参考价(`v_api`,来自网页搜索,标注采集日期与区间,不是官方报价)驱动 API 标价 / 条视频、每 GPU 小时收入与利润、API 标价 ÷ TCO 倍数等指标,价格无效时全部为 null,y 轴锚定 0 以便显示负利润;arena 式 Compare 以视频为主:同一 prompt 与 seed 在两种硬件上的视频并排播放,支持同步播放/暂停/重播、用例翻页和“盲测模式”(揭晓前隐藏硬件),指标差值列在视频下方,面板滚入视口时才加载两个已发布产物;compute-bound 证据面板展示板卡功率相对生效上限、并发排队平台以及相对带宽/FLOPS 规格比的实测加速比,所有硬件都读取它们共有的部署布局,比值无法区分时措辞保持保守。 diff --git a/packages/app/cypress/component/video-compare.cy.tsx b/packages/app/cypress/component/video-compare.cy.tsx index 884e3e33b..aad54730f 100644 --- a/packages/app/cypress/component/video-compare.cy.tsx +++ b/packages/app/cypress/component/video-compare.cy.tsx @@ -17,24 +17,29 @@ const COLORS: Record = { const colorFor = (key: string) => COLORS[key] ?? 'rgb(0, 0, 0)'; const OWNING_PARTICIPATING: MetricOptions = { tier: 'h', basis: 'participating' }; +const mediaUrl = (point: VideoPoint) => + `/api/video-runs?run=${point.runId}&artifact=${point.artifactId}&format=media`; function media(point: VideoPoint, hardware: string, alias: string) { cy.intercept( 'GET', - `/api/video-runs?run=${point.runId}&artifact=${point.artifactId}&format=media`, + mediaUrl(point), servingArtifact(Number(point.runId), point.artifactId, hardware), ).as(alias); } +/** Clips load on their own once the panel is in view, so overrides must be registered before mounting. */ function mount( pathname = '/video', search = '', data: VideoPoint[] = points, options: MetricOptions = OWNING_PARTICIPATING, + beforeMount?: () => void, ) { media(c1('h100'), 'NVIDIA H100 80GB HBM3', 'h100'); media(c1('b200'), 'NVIDIA B200', 'b200'); media(c1('h200'), 'NVIDIA H200', 'h200'); cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); + beforeMount?.(); cy.window().then((win) => win.history.replaceState(null, '', `${win.location.pathname}${search}`), ); @@ -46,14 +51,29 @@ function mount( } const table = () => cy.get('[data-testid="video-compare-table"]'); const metricRow = (id: string) => table().find(`tbody tr[data-metric="${id}"]`); +const clip = (role: 'baseline' | 'candidate') => + cy.get(`[data-testid="video-compare-clip"][data-role="${role}"]`); const pick = (label: string, option: string) => { cy.get(`[role="combobox"][aria-label="${label}"]`).click(); cy.contains('[role="option"]', option).click(); }; describe('Video compare panel (retained fixture)', () => { - it('defaults to the slowest hardware as baseline and the fastest as candidate', () => { + it('plays the same case from the slowest and fastest hardware side by side by default', () => { mount(); + cy.wait(['@h100', '@b200']); + cy.get('[data-testid="video-compare-prompt"]') + .should('contain', 'A drummer taps a snare drum.') + .and('contain', 'seed 11'); + cy.get('[data-testid="video-compare-case-of"]').should('contain', 'Case 1 of'); + clip('baseline') + .should('contain', 'H100') + .and('contain', '4 GPU · TP2 × Ulysses 2') + .and('contain', 'time to video 120 s') + .find('video') + .should('have.attr', 'src') + .and('include', 'measurement-r001-c001.mp4'); + clip('candidate').should('contain', 'B200').find('video').should('exist'); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'B200'); metricRow('p50Latency') @@ -66,68 +86,108 @@ describe('Video compare panel (retained fixture)', () => { .and('contain', '$0.150') .and('contain', '-31.1%'); metricRow('powerPctCap').find('[data-tone="better"]').should('exist'); - cy.get('[data-testid="video-compare"]').should('contain', 'Owning at Large Hyperscaler Volume'); - cy.get('[data-testid="video-compare-clip"]').should('not.exist'); - cy.get('@h100.all').should('have.length', 0); + cy.get('[data-testid="video-compare"]') + .should('contain', 'Owning at Large Hyperscaler Volume') + .and('contain', 'retained CI outputs'); + cy.get('@h200.all').should('have.length', 0); }); - it('loads both runs’ clips on demand and pairs the same prompt and seed', () => { + it('steps through cases with the arrows and the select, keeping v_case in the URL', () => { mount(); - cy.get('[data-testid="video-compare-load"]').click(); cy.wait(['@h100', '@b200']); - cy.get('[data-testid="video-compare-clip"]').should('have.length', 2); - cy.get('[data-testid="video-compare-clip"] video').should('have.length', 2); - cy.get('[data-testid="video-compare-clip"][data-role="baseline"]') - .should('contain', 'H100') - .and('contain', 'time to video 120 s') - .find('video') - .should('have.attr', 'src') - .and('include', 'measurement-r001-c001.mp4'); - cy.get('[data-testid="video-compare-clip"][data-role="candidate"]').should('contain', 'B200'); - cy.get('[role="combobox"][aria-label="Case (prompt · seed)"]').should( - 'contain', - 'A drummer taps a snare drum. · seed 11 · #1', + cy.get('[data-testid="video-compare-prev"]').should('be.disabled'); + cy.get('[data-testid="video-compare-next"]').click(); + cy.get('[data-testid="video-compare-clip"] video').each(($video) => + expect($video.attr('src')).to.include('measurement-r002-c001.mp4'), ); + cy.location('search').should('contain', 'v_case=1'); pick('Case (prompt · seed)', '#3'); cy.get('[data-testid="video-compare-clip"] video').each(($video) => expect($video.attr('src')).to.include('measurement-r003-c001.mp4'), ); + cy.get('[data-testid="video-compare-case-of"]').should('contain', 'Case 3 of'); cy.location('search').should('contain', 'v_case=2'); - cy.get('[data-testid="video-compare"]').should('contain', 'retained CI outputs'); + cy.get('[data-testid="video-compare-prev"]').click(); + cy.location('search').should('contain', 'v_case=1'); }); - it('updates the delta table and the URL when the candidate changes, without refetching', () => { + it('drives both players together', () => { + mount(); + cy.wait(['@h100', '@b200']); + cy.window().then((win) => { + cy.stub(win.HTMLMediaElement.prototype, 'play').as('play').resolves(); + cy.stub(win.HTMLMediaElement.prototype, 'pause').as('pause'); + }); + cy.get('[data-testid="video-compare-play"]').click(); + cy.get('@play').should('have.callCount', 2); + cy.get('[data-testid="video-compare-pause"]').click(); + cy.get('@pause').should('have.callCount', 2); + cy.get('[data-testid="video-compare-restart"]').click(); + cy.get('@play').should('have.callCount', 4); + }); + it('hides which hardware is which in blind mode until revealed', () => { + mount(); + cy.wait(['@h100', '@b200']); + cy.get('[data-testid="video-compare-blind"]').click(); + cy.get('[data-testid="video-compare-pane-label"]').should(($labels) => { + expect([...$labels].map((el) => el.textContent)).to.deep.equal(['A', 'B']); + }); + clip('baseline').should('not.contain', 'H100').and('contain', 'time to video 120 s'); + cy.get('[data-testid="video-compare-table"]').should('not.exist'); + cy.get('[data-testid="video-compare-metrics-hidden"]').should('exist'); + cy.get('[data-testid="video-compare-reveal"]').click(); + cy.get('[data-testid="video-compare-pane-label"]').first().should('contain', 'H100'); + cy.get('[data-testid="video-compare-table"]').should('exist'); + // The next case is blind again. + cy.get('[data-testid="video-compare-next"]').click(); + cy.get('[data-testid="video-compare-pane-label"]').first().should('have.text', 'A'); + }); + it('switches the candidate, fetching only the new side, and swaps on collision or request', () => { mount(); - cy.get('[data-testid="video-compare-load"]').click(); cy.wait(['@h100', '@b200']); pick('Candidate', 'H200'); + cy.wait('@h200'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H200'); + clip('candidate').should('contain', 'H200'); metricRow('p50Latency').should('contain', '150.6').and('contain', '-10%'); metricRow('dollarsPerVideo').should('contain', '$0.204'); cy.location('search').should('contain', 'v_cand=h200'); - // H200 clips were never fetched: the pane returns to the on-demand button. - cy.get('[data-testid="video-compare-load"]').should('exist'); - cy.get('@h200.all').should('have.length', 0); - // Choosing the baseline's hardware as candidate swaps the two sides. + cy.get('@h100.all').should('have.length', 1); + cy.get('@h200.all').should('have.length', 1); + // Choosing the baseline's hardware as candidate swaps the two sides; the swap button does too. pick('Candidate', 'H100'); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H200'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); metricRow('p50Latency').find('[data-tone="worse"]').should('exist'); + cy.get('[data-testid="video-compare-swap"]').click(); + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H200'); + cy.get('@h200.all').should('have.length', 1); }); - it('restores v_base and v_cand from the URL and reports a failed media read', () => { - mount('/video', '?v_base=h200&v_cand=h100'); - cy.intercept( - 'GET', - `/api/video-runs?run=${c1('h200').runId}&artifact=${c1('h200').artifactId}&format=media`, - { statusCode: 404, body: { error: 'gone' } }, - ).as('missing'); + it('restores v_base and v_cand from the URL and reports a failed media read with a retry', () => { + mount('/video', '?v_base=h200&v_cand=h100', points, OWNING_PARTICIPATING, () => { + cy.intercept('GET', mediaUrl(c1('h200')), { statusCode: 404, body: { error: 'gone' } }).as( + 'missing', + ); + }); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H200'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); - cy.get('[data-testid="video-compare-load"]').click(); cy.wait('@missing'); cy.get('[data-testid="video-compare-clips"] [role="alert"]') .should('contain', 'Could not load clips') .and('contain', 'HTTP 404'); cy.contains('button', 'Retry').should('exist'); }); + it('says when a run publishes no clips and keeps the metric deltas', () => { + mount('/video', '', points, OWNING_PARTICIPATING, () => { + cy.intercept('GET', mediaUrl(c1('b200')), { statusCode: 204 }).as('unpublished'); + }); + cy.wait(['@h100', '@unpublished']); + cy.get('[data-testid="video-compare-clips"]').should( + 'contain', + 'Clips are not published for one of these runs', + ); + cy.get('[data-testid="video-compare-clip"]').should('not.exist'); + metricRow('p50Latency').should('contain', '-53.4%'); + }); it('disables the selects when fewer than two hardware are measured', () => { mount( '/video', @@ -143,15 +203,16 @@ describe('Video compare panel (retained fixture)', () => { }); it('renders Chinese copy on /zh/video', () => { mount('/zh/video', '', points, { tier: 'r', basis: 'allocated' }); + cy.wait(['@h100', '@b200']); cy.get('[data-testid="video-compare"]') .should('contain', '基线') .and('contain', '候选') - .and('contain', '加载视频') + .and('contain', '同时播放') + .and('contain', '盲测模式') .and('contain', '租赁 - 3 年承诺') .and('contain', '已分配的 GPU'); metricRow('p50Latency').should('contain', 'P50 出片时间'); - cy.get('[data-testid="video-compare-load"]').click(); - cy.wait(['@h100', '@b200']); - cy.get('[data-testid="video-compare-clip"]').first().should('contain', '出片时间 120 s'); + cy.get('[data-testid="video-compare-case-of"]').should('contain', '第 1 /'); + clip('baseline').should('contain', '出片时间 120 s'); }); }); diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index 28fa4e182..f460ae9a6 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -7,6 +7,8 @@ function mount(pathname = '/video', search = '') { fixture: 'api/video-history.json', }).as('history'); cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + // The Compare panel fetches clips once in view; the component runner publishes none. + cy.intercept('GET', '/api/video-runs?run=*', { statusCode: 204 }); cy.window().then((win) => win.history.replaceState(null, '', `${win.location.pathname}${search}`), ); @@ -169,6 +171,7 @@ describe('Video hardware dashboard (retained fixture)', () => { req.reply({ fixture: 'api/video-history.json', delay: 800 }); }).as('slowHistory'); cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); + cy.intercept('GET', '/api/video-runs?run=*', { statusCode: 204 }); // The AUT keeps the previous test's v_ params; start from the defaults. cy.window().then((win) => win.history.replaceState(null, '', win.location.pathname)); cy.mount( diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts index c0c4e0a1b..438c771fc 100644 --- a/packages/app/cypress/e2e/video-dashboard.cy.ts +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -1,6 +1,10 @@ // Runs against the E2E_FIXTURES=1 server, which serves the retained // H100/H200/B200 observations from cypress/fixtures/api/video-history.json. describe('Video hardware dashboard (E2E fixtures)', () => { + beforeEach(() => { + // The Compare panel fetches two published artifacts once in view; keep the spec offline. + cy.intercept('GET', '/api/video-runs?run=*', { statusCode: 204 }); + }); it('leads with the cross-hardware chart and restores v_ params from the URL', () => { cy.visit('/video?v_y=kjPerVideo&v_tier=r&v_queue=1'); cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 9); diff --git a/packages/app/src/components/video-benchmark/VideoCompare.tsx b/packages/app/src/components/video-benchmark/VideoCompare.tsx index bac93a70f..ab56117e6 100644 --- a/packages/app/src/components/video-benchmark/VideoCompare.tsx +++ b/packages/app/src/components/video-benchmark/VideoCompare.tsx @@ -1,8 +1,10 @@ 'use client'; -import { useEffect, useId, useMemo, useRef, useState } from 'react'; +import { useCallback, useEffect, useId, useMemo, useRef, useState } from 'react'; import { Button } from '@/components/ui/button'; import { Heading } from '@/components/ui/heading'; +import { Skeleton } from '@/components/ui/skeleton'; +import { Switch } from '@/components/ui/switch'; import { track } from '@/lib/analytics'; import { useLocale } from '@/lib/use-locale'; import { @@ -22,6 +24,7 @@ import { writeVideoCompareSelection, type VideoCompareSelection, } from './compare-url-state'; +import { layoutLabel } from './deployment'; import { hardwareLabel } from './hardware'; import { formatMetric, @@ -37,10 +40,17 @@ const STRINGS = { en: { title: 'Compare', subtitle: - 'Pick a baseline and a candidate hardware (each on its most efficient measured deployment): metric deltas side by side, then the same generated case from both runs.', + 'Arena-style: the same prompt and seed generated on two hardware, side by side. Pick a baseline and a candidate (each on its most efficient measured deployment); the metric deltas follow below the clips.', baseline: 'Baseline', candidate: 'Candidate', + swap: 'Swap sides', + blind: 'Blind mode', + blindHint: 'Hide which hardware is which until you reveal it.', + reveal: 'Reveal hardware', + hidden: { baseline: 'A', candidate: 'B' }, needTwo: 'Comparison needs at least two measured hardware.', + metrics: 'Metric deltas', + metricsHidden: 'Metric deltas are hidden in blind mode; reveal the hardware to see them.', metric: 'Metric', change: 'Change', assumptions: (tier: string, basis: string) => `Cost tier: ${tier} · GPU basis: ${basis}.`, @@ -49,12 +59,21 @@ const STRINGS = { better: 'Candidate better', worse: 'Candidate worse', neutral: 'No difference', - load: 'Load clips', - loadHint: 'Fetches the retained clips of both runs on demand (two published artifacts).', loading: 'Loading clips…', + loadingHint: 'Fetching the retained clips of both runs (two published artifacts).', error: 'Could not load clips', retry: 'Retry', + noClips: 'Clips are not published for one of these runs; the metric deltas below still apply.', + prompt: 'Prompt', + seed: 'seed', + take: (n: number) => `take ${n}`, + caseOf: (i: number, n: number) => `Case ${i} of ${n}`, + prev: 'Previous case', + next: 'Next case', case: 'Case (prompt · seed)', + playBoth: 'Play both', + pauseBoth: 'Pause both', + restart: 'Restart both', noPairs: 'No case with the same prompt and seed exists in both runs.', unmatched: (baseline: number, candidate: number) => `${baseline} baseline and ${candidate} candidate records have no counterpart.`, @@ -67,10 +86,17 @@ const STRINGS = { zh: { title: '对比', subtitle: - '选择基线与候选硬件(各自取实测中最高效的部署):并排查看指标差异,再对照两次运行中同一 prompt 与 seed 生成的视频。', + 'Arena 式对比:同一 prompt 与 seed 在两种硬件上生成的视频并排播放。选择基线与候选硬件(各自取实测中最高效的部署),指标差异列在视频下方。', baseline: '基线', candidate: '候选', + swap: '交换两侧', + blind: '盲测模式', + blindHint: '揭晓前隐藏两侧各是哪种硬件。', + reveal: '揭晓硬件', + hidden: { baseline: 'A', candidate: 'B' }, needTwo: '至少需要两种已实测的硬件才能对比。', + metrics: '指标差异', + metricsHidden: '盲测模式下隐藏指标差异,揭晓硬件后显示。', metric: '指标', change: '变化', assumptions: (tier: string, basis: string) => `成本档位:${tier} · GPU 口径:${basis}。`, @@ -79,12 +105,21 @@ const STRINGS = { better: '候选更优', worse: '候选更差', neutral: '无差异', - load: '加载视频', - loadHint: '按需读取两次运行保留的视频(两个已发布产物)。', loading: '正在加载视频…', + loadingHint: '正在读取两次运行保留的视频(两个已发布产物)。', error: '无法加载视频', retry: '重试', + noClips: '其中一次运行没有发布视频;下方的指标差异仍然有效。', + prompt: 'Prompt', + seed: 'seed', + take: (n: number) => `第 ${n} 次`, + caseOf: (i: number, n: number) => `第 ${i} / ${n} 个用例`, + prev: '上一个用例', + next: '下一个用例', case: '用例(prompt · seed)', + playBoth: '同时播放', + pauseBoth: '同时暂停', + restart: '从头同时播放', noPairs: '两次运行中没有 prompt 与 seed 均相同的用例。', unmatched: (baseline: number, candidate: number) => `${baseline} 条基线记录与 ${candidate} 条候选记录没有对应项。`, @@ -96,6 +131,8 @@ const STRINGS = { }, }; type Strings = (typeof STRINGS)['en']; +type Role = 'baseline' | 'candidate'; +const ROLES: readonly Role[] = ['baseline', 'candidate']; const CHIP = { better: 'border-emerald-600/40 bg-emerald-500/10 text-emerald-700 dark:text-emerald-300', @@ -113,13 +150,14 @@ function caseLabel(pair: CasePair, repeated: boolean): string { return parts.join(' · '); } -async function fetchSide(point: VideoPoint, signal: AbortSignal): Promise { +/** The stored artifact of one side, or null when the deployment publishes no media (HTTP 204). */ +async function fetchSide(point: VideoPoint, signal: AbortSignal): Promise { const response = await fetch( `/api/video-runs?run=${encodeURIComponent(point.runId)}&artifact=${encodeURIComponent(String(point.artifactId))}&format=media`, { signal }, ); - if (!response.ok || response.status === 204) - throw new Error(`Published media HTTP ${response.status}`); + if (response.status === 204) return null; + if (!response.ok) throw new Error(`Published media HTTP ${response.status}`); const saved = (await response.json()) as StoredArtifact; return compareSide(saved, point); } @@ -148,10 +186,21 @@ function DeltaChip({ row, s }: { row: CompareRow; s: Strings }) { ); } +function ClipPlaceholders() { + return ( + + ); +} + /** - * Arena-style baseline vs candidate: metric deltas between each hardware's lead deployment, then the - * retained clips of the same prompt and seed from both CI runs. Media is fetched - * only on request, one stored artifact per side. + * Arena-style baseline vs candidate: the same prompt and seed from both CI runs + * play side by side (blind mode hides which hardware is which), with the metric + * deltas between each hardware's lead deployment below. Media is fetched once + * the panel scrolls into view, one stored artifact per side. */ export default function VideoCompare({ points, @@ -165,27 +214,53 @@ export default function VideoCompare({ const locale = useLocale(); const s = STRINGS[locale]; const headingId = useId(); + const blindId = useId(); + const sectionRef = useRef(null); + const videos = useRef>({ baseline: null, candidate: null }); const [selection, setSelection] = useState( DEFAULT_VIDEO_COMPARE_SELECTION, ); - const [sides, setSides] = useState>(() => new Map()); + const [sides, setSides] = useState>(() => new Map()); const [pending, setPending] = useState(null); const [failure, setFailure] = useState<{ key: string; message: string } | null>(null); const [failedMedia, setFailedMedia] = useState>(() => new Set()); + const [visible, setVisible] = useState(false); + const [blind, setBlind] = useState(false); + const [revealedFor, setRevealedFor] = useState(null); const request = useRef(0); const download = useRef(null); useEffect(() => { setSelection(readVideoCompareSelection(location.search)); return () => download.current?.abort(); }, []); + // Two stored artifacts are several MB each: fetch them once the panel is near the viewport. + useEffect(() => { + const node = sectionRef.current; + if (!node || typeof IntersectionObserver === 'undefined') { + setVisible(true); + return; + } + const observer = new IntersectionObserver( + (entries) => { + if (entries.some((entry) => entry.isIntersecting)) { + setVisible(true); + observer.disconnect(); + } + }, + { rootMargin: '240px' }, + ); + observer.observe(node); + return () => observer.disconnect(); + }, []); const measured = useMemo(() => comparablePoints(points), [points]); const defaults = useMemo(() => defaultComparePair(points), [points]); const { baseline, candidate, caseIndex } = resolveCompareSelection(selection, points); const pairKey = baseline && candidate ? `${baseline.id}|${candidate.id}` : null; const rows = baseline && candidate ? compareMetrics(baseline, candidate, options) : []; + const fetched = baseline && candidate && sides.has(baseline.id) && sides.has(candidate.id); const loaded = - baseline && candidate && sides.has(baseline.id) && sides.has(candidate.id) + fetched && sides.get(baseline.id) && sides.get(candidate.id) ? { baseline: sides.get(baseline.id)!, candidate: sides.get(candidate.id)! } : null; const matched = loaded ? pairCases(loaded.baseline.records, loaded.candidate.records) : null; @@ -201,6 +276,12 @@ export default function VideoCompare({ value: point.hardwareKey ?? '', label: hardwareLabel(point.hardwareKey ?? ''), })); + const masked = blind && revealedFor !== `${pairKey}:${index}`; + const sideOf = { baseline, candidate } as Record; + const paneLabel = (role: Role) => + masked ? s.hidden[role] : hardwareLabel(sideOf[role]?.hardwareKey ?? ''); + const paneColor = (role: Role) => + masked ? 'var(--muted-foreground)' : colorFor(sideOf[role]?.hardwareKey ?? ''); const update = (patch: Partial) => { const next = { ...selection, ...patch }; @@ -220,9 +301,9 @@ export default function VideoCompare({ ), ); }; - const choose = (side: 'baseline' | 'candidate', hardware: string) => { - const own = side === 'baseline' ? baseline : candidate; - const other = side === 'baseline' ? candidate : baseline; + const choose = (side: Role, hardware: string) => { + const own = sideOf[side]; + const other = sideOf[side === 'baseline' ? 'candidate' : 'baseline']; const patch: Partial = { [side]: hardware }; // Picking the other side's hardware swaps the two instead of comparing a hardware with itself. if (other?.hardwareKey === hardware && own) @@ -230,7 +311,19 @@ export default function VideoCompare({ update(patch); track(`video_compare_${side}_changed`, { hardware }); }; - const load = async () => { + const swap = () => { + if (!baseline || !candidate) return; + update({ baseline: candidate.hardwareKey, candidate: baseline.hardwareKey }); + track('video_compare_swapped', { + baseline: candidate.hardwareKey, + candidate: baseline.hardwareKey, + }); + }; + const goTo = (next: number) => { + update({ caseIndex: next }); + track('video_compare_case_changed', { index: next, seed: pairs[next]?.seed ?? null }); + }; + const load = useCallback(async () => { if (!baseline || !candidate || pairKey === null) return; download.current?.abort(); const controller = new AbortController(); @@ -244,36 +337,83 @@ export default function VideoCompare({ }); try { const missing = [baseline, candidate].filter((point) => !sides.has(point.id)); - const fetched = await Promise.all( + const results = await Promise.all( missing.map( async (point) => [point.id, await fetchSide(point, controller.signal)] as const, ), ); if (attempt !== request.current) return; - setSides((old) => new Map([...old, ...fetched])); + setSides((old) => new Map([...old, ...results])); } catch (error) { if (controller.signal.aborted) return; setFailure({ key: pairKey, message: error instanceof Error ? error.message : String(error) }); } finally { if (attempt === request.current) setPending(null); } + }, [baseline, candidate, pairKey, sides]); + useEffect(() => { + if (!visible || pairKey === null || fetched) return; + if (pending === pairKey || failure?.key === pairKey) return; + void load(); + }, [visible, pairKey, fetched, pending, failure, load]); + + const eachVideo = (fn: (video: HTMLVideoElement) => void) => { + for (const video of Object.values(videos.current)) if (video) fn(video); + }; + const playback = (action: 'play' | 'pause' | 'restart') => { + eachVideo((video) => { + if (action === 'pause') { + video.pause(); + return; + } + if (action === 'restart') video.currentTime = 0; + void video.play().catch(() => {}); + }); + track('video_compare_playback', { action }); + }; + const reveal = () => { + setRevealedFor(`${pairKey}:${index}`); + track('video_compare_revealed', { + baseline: baseline?.hardwareKey ?? null, + candidate: candidate?.hardwareKey ?? null, + }); }; return (
-
- - {s.title} - -

{s.subtitle}

+
+
+ + {s.title} + +

{s.subtitle}

+
+
choose('baseline', value)} options={hardwareOptions} /> + -
- - - - - {( - [ - ['baseline', baseline], - ['candidate', candidate], - ] as const - ).map(([role, point]) => ( - - ))} - - - - - {rows.map((row) => ( - - - - - - - ))} - -
- {s.metric} - - - - {hardwareLabel(point.hardwareKey ?? '')} - · {s[role]} - - - {s.change} -
- {locale === 'zh' - ? VIDEO_METRICS[row.id].labelZh - : VIDEO_METRICS[row.id].label} - {formatMetric(row.baseline, row.id)}{formatMetric(row.candidate, row.id)} - -
-
-

- {s.assumptions( - TIER_LABELS[options.tier][locale], - options.basis === 'allocated' ? s.allocated : s.participating, - )} -

{loaded && matched ? ( pairs.length === 0 || !current ? ( @@ -362,34 +455,56 @@ export default function VideoCompare({

) : ( <> -
- { - const next = Number(value); - update({ caseIndex: next }); - track('video_compare_case_changed', { - index: next, - seed: pairs[next]?.seed ?? null, - }); - }} - options={pairs.map((pair, i) => ({ - value: String(i), - label: caseLabel(pair, repeated.has(pair.key)), - }))} - /> +
+
+ {s.prompt} + + + + {s.caseOf(index + 1, pairs.length)} + + + +
+

{current.prompt ?? current.caseId ?? '—'}

+

+ {current.seed !== null && `${s.seed} ${current.seed}`} + {repeated.has(current.key) && ` · ${s.take(current.repetition + 1)}`} +

- {( - [ - ['baseline', baseline, current.baseline, loaded.baseline], - ['candidate', candidate, current.candidate, loaded.candidate], - ] as const - ).map(([role, point, record, side]) => { + {ROLES.map((role) => { + const point = sideOf[role]!; + const record = current[role]; + const side = loaded[role]; const url = record.mediaPath === null ? undefined : side.urls.get(record.mediaPath); - const label = hardwareLabel(point.hardwareKey ?? ''); + const label = paneLabel(role); return (
+
+ + + {label} + + · {s[role]} + {!masked && ( + + · {layoutLabel(point, locale)} + + )} + {record.seconds !== null && ( + + · {s.timeToVideo}{' '} + {record.seconds.toLocaleString('en-US', { + maximumFractionDigits: 1, + })}{' '} + s + + )} +
{url ? (
); })}
+
+
+ + + + {masked && ( + + )} +
+
+ goTo(Number(value))} + options={pairs.map((pair, i) => ({ + value: String(i), + label: caseLabel(pair, repeated.has(pair.key)), + }))} + /> +
+
{(matched.unmatched.baseline > 0 || matched.unmatched.candidate > 0) && (

) - ) : pending === pairKey ? ( + ) : fetched ? (

- {s.loading} + {s.noClips}

+ ) : pending === pairKey ? ( +
+

+ {s.loading} {s.loadingHint} +

+ +
) : failure?.key === pairKey ? (

{s.error}: {failure.message}

-
) : ( -
- - {s.loadHint} -
+ )} -

{s.footnote}

+ {masked ? ( +

+ {s.metricsHidden} +

+ ) : ( +
+

{s.metrics}

+
+ + + + + {ROLES.map((role) => ( + + ))} + + + + + {rows.map((row) => ( + + + + + + + ))} + +
+ {s.metric} + + + + {paneLabel(role)} + · {s[role]} + + + {s.change} +
+ {locale === 'zh' + ? VIDEO_METRICS[row.id].labelZh + : VIDEO_METRICS[row.id].label} + {formatMetric(row.baseline, row.id)}{formatMetric(row.candidate, row.id)} + +
+
+

+ {s.assumptions( + TIER_LABELS[options.tier][locale], + options.basis === 'allocated' ? s.allocated : s.participating, + )} +

+
+ )} +

{s.footnote}

)}
From a0ee76970e6e14b079c082aa610742bb821f05cb Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sun, 20 Sep 2026 00:34:35 -0700 Subject: [PATCH 038/104] refactor: trim the video dashboard to the cost and compute-bound claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keep only what the two claims need: X = P90/P50 time to video, Y = videos per $1 TCO, TCO cost per video, videos per GPU-hour or kJ per video, one cost tier select. Every per-GPU metric divides by participating GPUs; an idle-boards note under the chart replaces the GPU-basis toggle. Drop the queued/optimal/frontier switches (queued cells never reach the chart, and per-hardware frontier lines appear on their own once a hardware has two deployments), the single-option Model/Workload/Deployment selects (now plain facts), the revenue/profit/multiple/genSpeed/video-seconds/kWh/ $ per video-second metrics, the KPI profit and board-power rows, blind mode and the case dropdown in Compare, and four of its seven delta rows. The API reference now yields one number, the API list price per video, shown beside the TCO cost. Tests, docs and the Chinese copy follow. 中文:围绕两个 claim 收敛 /video 仪表板:X 轴只留 P90/P50 出片时间,Y 轴只留 每 1 美元 TCO 视频数、每条视频 TCO 成本、每 GPU 小时视频数和每条视频能耗, 保留成本档位。所有按 GPU 计的指标以参与计算的 GPU 为分母,图表下方的空闲 板卡说明取代 GPU 口径切换。去掉排队/仅最优/前沿三个开关(排队 cell 不进入 图表,某硬件有两种部署时自动画出前沿线)、三个单选项假下拉(改为文字事实)、 收入/利润/倍数/生成速度/视频秒数/kWh/每视频秒成本等指标、KPI 卡的利润与 板卡功率行、Compare 的盲测模式与用例下拉,以及七行差值中的四行。API 参考价 只推导每条视频 API 标价,显示在 TCO 成本旁。测试、文档与中文文案同步更新。 --- docs/index.md | 2 +- docs/videogenx-dashboard.md | 29 +- .../cypress/component/video-compare.cy.tsx | 82 +++-- .../cypress/component/video-dashboard.cy.tsx | 272 +++++++++++------ .../cypress/component/video-evidence.cy.tsx | 54 +--- .../app/cypress/e2e/video-dashboard.cy.ts | 222 +++++++++++++- .../video-benchmark/VideoCompare.tsx | 279 ++++++------------ .../video-benchmark/VideoConfigBar.tsx | 58 ++-- .../video-benchmark/VideoDashboard.tsx | 73 ++--- .../video-benchmark/VideoEvidence.tsx | 8 +- .../video-benchmark/VideoHardwareChart.tsx | 61 +--- .../video-benchmark/VideoKpiCards.tsx | 57 ++-- .../video-benchmark/VideoPointsTable.tsx | 26 +- .../video-benchmark/chart-domain.test.ts | 11 +- .../video-benchmark/compare.test.ts | 71 ++--- .../src/components/video-benchmark/compare.ts | 8 +- .../components/video-benchmark/deployment.ts | 2 +- .../video-benchmark/evidence.test.ts | 8 - .../components/video-benchmark/evidence.ts | 15 +- .../video-benchmark/metrics.test.ts | 204 ++++++++----- .../src/components/video-benchmark/metrics.ts | 189 +++--------- .../components/video-benchmark/plot.test.ts | 110 ++++--- .../src/components/video-benchmark/plot.ts | 31 +- .../video-benchmark/video-url-state.test.ts | 71 +++-- .../video-benchmark/video-url-state.ts | 26 +- 25 files changed, 999 insertions(+), 970 deletions(-) diff --git a/docs/index.md b/docs/index.md index 0727cbcf6..86e02ec5e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -25,7 +25,7 @@ Design rationale and non-obvious conventions. See [CLAUDE.md](../CLAUDE.md) for - [State Ownership](./state-ownership.md) — Which context owns which state, availability filtering cascade, comparison date mechanics, URL param sync - [Blog](./blog.md) — MDX content system, SEO features (OG images, RSS, llms.txt, JSON-LD), TOC sidebar, reading progress, heading links, analytics events - [CollectiveX](./collectivex.md) — lazy ingest-on-read pipeline (separate Neon DB as a durable GitHub-artifact cache), tombstoned deletes, raw-docs storage + shared reader -- [VideoGenX dashboard](./videogenx-dashboard.md) — `/video` cross-hardware chart: published-history projection → points → metrics, TCO tiers and GPU basis, C1 default with queueing tails, Pareto hull, unavailable-hardware roster, null rules +- [VideoGenX dashboard](./videogenx-dashboard.md) — `/video` cross-hardware chart: published-history projection → points → metrics, TCO tiers per participating GPU vs. the API list price, deployment cells (queued cells never plotted), arena Compare, compute-bound evidence, unavailable-hardware roster, null rules - [Chinese Pages (/zh)](./i18n.md) — Why hand-authored /zh pages instead of an i18n framework, hreflang pairing, blog translation pairing, html lang workaround, CJK reading time/slugs - [Chinese Copy Editorial Guide](./chinese-copy.md) — Audience, surface-specific register, context-aware terminology, rewrite workflow, manual-review process, and PR checklist diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md index 1446fa891..b747f75d5 100644 --- a/docs/videogenx-dashboard.md +++ b/docs/videogenx-dashboard.md @@ -8,40 +8,41 @@ Published blob indexes → `GET /api/video-runs?format=history` (the per-cell pr ## Axes and defaults -`video-url-state.ts` owns the `v_*` URL params (`v_x`, `v_y`, `v_tier`, `v_basis`, `v_queue`, `v_opt`, `v_frontier`, `v_view`); defaults are omitted from the URL and every other param is left alone. +`video-url-state.ts` owns the `v_*` URL params (`v_x`, `v_y`, `v_tier`, `v_view`, `v_api`); defaults are omitted from the URL and every other param is left alone. -- **X = P90 time to video (s)**, lower is better. `p50Latency` and `genSpeed` (frames/s per request, the closest analogue to interactivity) are the alternatives. -- **Y = videos per $1 TCO** at the _Owning at Large Hyperscaler Volume_ tier. Cost per GPU-hour comes from `HW_REGISTRY` (`costh`/`costr`), the same SemiAnalysis TCO source `/inference` uses; the badge row and source link sit above the chart. -- **GPU basis**: _participating_ GPUs (the boards that generated the clip — 4 per video with TP2 × Ulysses 2) is chip efficiency; _allocated_ GPUs (H100/B200 reserved 8) is what a deployment bills. One toggle apart, never mixed in one number. +- **X = P90 time to video (s)**, lower is better; `p50Latency` is the alternative. +- **Y = videos per $1 TCO** at the _Owning at Large Hyperscaler Volume_ tier; TCO cost per video, videos per GPU-hour and GPU-board energy per video (kJ) are the alternatives. Cost per GPU-hour comes from `HW_REGISTRY` (`costh`/`costr`), the same SemiAnalysis TCO source `/inference` uses; the badge row and source link sit above the chart. +- **Per participating GPU**: every per-GPU metric divides by the boards that generated the clip (4 per video with TP2 × Ulysses 2), never by the boards a job reserved. The retained H100 and B200 jobs reserved eight and used four, so a note under the chart header names them ("B200 (4 of 8), H100 (4 of 8) … the idle boards are not counted") and their KPI card headers read "4 of 8 GPUs". Idle boards are stated, not billed; there is no GPU-basis toggle. +- **Both axes start at zero** (`chart-domain.ts`), so per-dollar and per-video readings compare by length, not by offset. ## Deployments, not concurrency, make the curve `/inference` sweeps batch size to trace each hardware's latency-vs-throughput frontier. A diffusion video server has no batching lever: the pinned H3 SGLang server is batch-one and single-replica, so client concurrency only queues requests — the retained C1/C2/C4 cells show throughput flat within ±1.5 % while latency scales with C. Joining those cells would draw a flat "curve" that measures the queue, not the hardware, so the chart never does. -The knob that does trade latency for efficiency is the **deployment**: how many GPU boards serve one request and how they split the model (`tp_size` × `ulysses_degree`). `deployment.ts` keys cells by it (`deploymentKey`), classifies `concurrency > replicas` as queueing (`isQueueing`; bundles that predate `execution.deployment.replica_count` ran one endpoint, so null reads as one), and picks each hardware's most efficient deployment as its representative (`leadCell`, used by the KPI cards and the legend). +The knob that does trade latency for efficiency is the **deployment**: how many GPU boards serve one request and how they split the model (`tp_size` × `ulysses_degree`). `deployment.ts` keys cells by it (`deploymentKey`), classifies `concurrency > replicas` as queueing (`isQueueing`; bundles that predate `execution.deployment.replica_count` ran one endpoint, so null reads as one), and picks each hardware's most efficient deployment as its representative (`leadCell`, used by the KPI cards, the Compare panel and the legend). -`plot.ts` (`plotVideoPoints`) then builds the chart: per hardware, the Pareto frontier over its deployment cells (`frontier.ts` flips lower-is-better axes and reuses `paretoFrontUpperLeft` from the calculator); a solid line appears once a hardware has two or more frontier deployments, dominated deployments render faded, and _Optimal only_ hides them. Queued cells are opt-in (_Show queued requests_), drawn as small faded markers and never joined. _Pareto frontier_ adds the dashed cross-hardware frontier over all deployments, as on `/inference`; when one hardware dominates outright (today: B200) the caption says so instead of drawing a line. +`plot.ts` (`plotVideoPoints`) then builds the chart from the non-queued cells only: a queued cell never reaches the chart (or the table and CSV) and lives in the evidence panel instead. Per hardware it takes the Pareto frontier over the remaining deployment cells (`frontier.ts` flips lower-is-better axes and reuses `paretoFrontUpperLeft` from the calculator); a solid line appears automatically once a hardware has two or more frontier deployments, and dominated deployments render faded. None of this has a switch — no "show queued requests", no "optimal only", no cross-hardware frontier — so every reader sees the same chart. -Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 × Ulysses 2), so the chart is an honest scatter and the caption says a per-hardware curve needs the GPUs-per-video sweep. That sweep (design plan P6) is backend work in InferenceX `experimental/video-generation` — the serving matrix currently varies only concurrency — and needs GPU authorisation; the frontend needs no change when its cells arrive, because they differ in `deploymentKey` and therefore add points rather than replace them. +Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 × Ulysses 2), so the chart is an honest three-point scatter and the caption says a per-hardware curve needs the GPUs-per-video sweep. That sweep (design plan P6) is backend work in InferenceX `experimental/video-generation` — the serving matrix currently varies only concurrency — and needs GPU authorisation; the frontend needs no change when its cells arrive, because they differ in `deploymentKey` and therefore add points rather than replace them. ## Sections below the chart -- **API price reference** (`api-reference.ts`, `VideoApiReference.tsx`): a dated, editable USD per video-second list price (`v_api`, default `H3_API_REFERENCE`) feeds the API-priced metrics in `metrics.ts` — API $/video, revenue and profit per GPU-hour, API price ÷ TCO multiple. It is a web-search reference, labelled with its capture date and range, never presented as an official quote; a null or non-positive price nulls every derived metric. The chart's y axis is zero-anchored (`chart-domain.ts`) so negative profit stays visible and an all-negative view tops out at break-even. -- **Compare** (`compare.ts`, `compare-url-state.ts`, `VideoCompare.tsx`): arena-style baseline vs. candidate, video first — the same prompt + seed clips from both CI runs play side by side with shared play/pause/restart, case arrows, and a blind mode that hides which hardware is which until revealed; the metric deltas (registry polarity) sit below. One selectable cell per hardware (its lead deployment); `v_base`, `v_cand`, `v_case` in the URL. The two stored artifacts (several MB each) load once the panel scrolls into view; an HTTP 204 side reads as "clips not published", never as an error. -- **Compute-bound evidence** (`evidence.ts`, `VideoEvidence.tsx`): board power against the recorded enforced limit, the concurrency plateau (throughput flat, latency × C), and observed speed-ups against memory-bandwidth and FLOPS spec ratios. Every hardware is read on the deployment layout most of them share (`sharedLayoutCells`) so the exhibits compare like with like; sentences stay conditional when ratios do not separate. +- **API price reference** (`api-reference.ts`, `VideoApiReference.tsx`): a dated, editable USD per video-second list price (`v_api`, default `H3_API_REFERENCE`) feeds exactly one derived number, the API list price per video (`apiPricePerVideo` = price × clip seconds). It is shown beside the TCO cost per video on the KPI cards and as a table and CSV column; the chart never plots it, and no revenue, profit or price multiple is derived from it. The price is a web-search reference, labelled with its capture date and range, never presented as an official quote; a null or non-positive price nulls the list price, and `parseApiPrice` falls back to the reference rather than pricing at 0. +- **Compare** (`compare.ts`, `compare-url-state.ts`, `VideoCompare.tsx`): arena-style baseline vs. candidate, video first — the same prompt + seed clips from both CI runs play side by side with shared play/pause/restart and previous/next case arrows (no blind mode, no case dropdown); the deltas below are fixed to time to video (P50), TCO cost per video and energy per video (`COMPARE_METRICS`, registry polarity), with the cost tier stated in a footnote. One selectable cell per hardware (its lead deployment), defaulting to the slowest hardware as baseline and the fastest as candidate; `v_base`, `v_cand`, `v_case` in the URL. The two stored artifacts (several MB each) load once the panel scrolls into view; an HTTP 204 side reads as "clips not published", never as an error. +- **Compute-bound evidence** (`evidence.ts`, `VideoEvidence.tsx`): board power against the recorded enforced limit, the concurrency plateau (throughput flat, latency × C — the only place the queued cells appear), and observed speed-ups against memory-bandwidth and FLOPS spec ratios. Every hardware is read on the deployment layout most of them share (`sharedLayoutCells`) so the exhibits compare like with like; sentences stay conditional when ratios do not separate. ## Rules - **Unavailable hardware** stays visible. `VIDEO_HARDWARE_ROSTER` in `hardware.ts` lists the campaign targets; MI355X has no valid run and renders as a muted legend row, a "Not measured" KPI card with the failed-run link, and no table row. A missing point is never a zero. - **Nulls**: every formula returns `null` when an input is missing or invalid — power phase invalid, fewer than ten samples for P90, unknown hardware key, missing GPU count. `formatMetric` prints `—`. -- **Newest observation wins**: `latestVideoCells` keeps the first (newest publication) observation per hardware × concurrency, so a re-run replaces, never duplicates, a point. +- **Newest observation wins**: `latestVideoCells` keeps the first (newest publication) observation per hardware × deployment × concurrency, so a re-run replaces, never duplicates, a point, while a new server layout adds one. - **Strings**: component-local `STRINGS = { en, zh }` via `useLocale()`; SKUs, units and identifiers stay English. - **Fixture**: `cypress/fixtures/api/video-history.json` is rebuilt from retained `StoredArtifact` exports with `bun run fixtures:video-history
` (`scripts/build-video-history-fixture.ts`); it is metadata only and registered in `_manifest.json`. ## 中文说明 -`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。默认 X 为 P90 出片时间、Y 为每 1 美元 TCO 视频数(Hyperscaler 自有设备档位,单价与 `/inference` 同源)。 +`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。`video-url-state.ts` 管理 `v_x`、`v_y`、`v_tier`、`v_view`、`v_api` 五个 URL 参数,默认值不写入 URL。X 轴可选 P90(默认)或 P50 出片时间;Y 轴可选每 1 美元 TCO 视频数(默认,Hyperscaler 自有设备档位,单价与 `/inference` 同源)、每条视频 TCO 成本、每 GPU 小时视频数和每条视频 GPU 板卡能耗(kJ)。所有按 GPU 计的指标都以参与计算的 GPU 为分母(每条视频 4 张,TP2 × Ulysses 2),而不是任务预留的板卡数;保留数据中 H100 与 B200 的运行预留了 8 张却只用 4 张,图表标题下方的说明会点名这些硬件,对应 KPI 卡标题显示“4 / 8 张 GPU”。空闲板卡只作说明、不计成本,没有 GPU 口径切换。两条坐标轴都从 0 起,便于按长度比较。 -曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表;`plot.ts` 按硬件对部署点求 Pareto 前沿,达到两个及以上前沿点时才画实线,被支配的部署以淡色显示,“仅最优”可将其隐藏;排队点需手动开启,只画小号淡色标记、不连线;“Pareto 前沿”开关叠加跨硬件虚线。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。参与/已分配 GPU 口径一键切换;MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 +曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表(KPI 卡、Compare 面板和图例都用它);`plot.ts` 只用未排队的 cell 绘图——排队 cell 不会进入图表、表格和 CSV,只出现在证据面板——再按硬件对剩余部署点求 Pareto 前沿:某硬件有两个及以上前沿部署时自动画出实线,被支配的部署以淡色显示。这些行为都没有开关,没有“显示排队请求”,没有“仅最优”,也没有跨硬件前沿,所有读者看到的是同一张图。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的三点散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 -图表下方还有三个区块:带日期、可编辑的 API 参考价(`v_api`,来自网页搜索,标注采集日期与区间,不是官方报价)驱动 API 标价 / 条视频、每 GPU 小时收入与利润、API 标价 ÷ TCO 倍数等指标,价格无效时全部为 null,y 轴锚定 0 以便显示负利润;arena 式 Compare 以视频为主:同一 prompt 与 seed 在两种硬件上的视频并排播放,支持同步播放/暂停/重播、用例翻页和“盲测模式”(揭晓前隐藏硬件),指标差值列在视频下方,面板滚入视口时才加载两个已发布产物;compute-bound 证据面板展示板卡功率相对生效上限、并发排队平台以及相对带宽/FLOPS 规格比的实测加速比,所有硬件都读取它们共有的部署布局,比值无法区分时措辞保持保守。 +图表下方还有三个区块。带日期、可编辑的 API 参考价(`v_api`,来自网页搜索,标注采集日期与区间,不是官方报价)只推导一个数字——每条视频 API 标价(单价 × 视频秒数),显示在 KPI 卡每条视频 TCO 成本旁边,并作为表格和 CSV 的一列;它不进入图表,也不再推导收入、利润或价格倍数。价格缺失或不为正时该值为 null,`parseApiPrice` 会回退到参考价,而不是按 0 计价。Arena 式 Compare 以视频为主:同一 prompt 与 seed 在两种硬件上的视频并排播放,支持同步播放/暂停/重播和上一个/下一个用例箭头,没有盲测模式,也没有用例下拉框;下方的差值行固定为出片时间(P50)、每条视频 TCO 成本和每条视频能耗,并以脚注注明成本档位。每种硬件只有一个可选 cell(其代表部署),默认以最慢的硬件为基线、最快的为候选,选择记录在 `v_base`、`v_cand`、`v_case` 中;面板滚入视口时才加载两个已发布产物,HTTP 204 读作“未发布视频”而不是错误。compute-bound 证据面板展示板卡功率相对生效上限、并发排队平台(排队 cell 唯一出现的地方)以及相对带宽/FLOPS 规格比的实测加速比,所有硬件都读取它们共有的部署布局,比值无法区分时措辞保持保守。 diff --git a/packages/app/cypress/component/video-compare.cy.tsx b/packages/app/cypress/component/video-compare.cy.tsx index aad54730f..c2c8fa0f2 100644 --- a/packages/app/cypress/component/video-compare.cy.tsx +++ b/packages/app/cypress/component/video-compare.cy.tsx @@ -15,7 +15,7 @@ const COLORS: Record = { b200: 'rgb(70, 80, 90)', }; const colorFor = (key: string) => COLORS[key] ?? 'rgb(0, 0, 0)'; -const OWNING_PARTICIPATING: MetricOptions = { tier: 'h', basis: 'participating' }; +const OWNING: MetricOptions = { tier: 'h' }; const mediaUrl = (point: VideoPoint) => `/api/video-runs?run=${point.runId}&artifact=${point.artifactId}&format=media`; @@ -32,7 +32,7 @@ function mount( pathname = '/video', search = '', data: VideoPoint[] = points, - options: MetricOptions = OWNING_PARTICIPATING, + options: MetricOptions = OWNING, beforeMount?: () => void, ) { media(c1('h100'), 'NVIDIA H100 80GB HBM3', 'h100'); @@ -76,37 +76,67 @@ describe('Video compare panel (retained fixture)', () => { clip('candidate').should('contain', 'B200').find('video').should('exist'); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'B200'); + // Exactly three delta rows: time to video, TCO cost per video and energy per video. + table() + .find('tbody tr') + .should(($rows) => { + expect([...$rows].map((el) => el.dataset.metric)).to.deep.equal([ + 'p50Latency', + 'dollarsPerVideo', + 'kjPerVideo', + ]); + }); metricRow('p50Latency') .should('contain', '167.3') .and('contain', '77.9') .and('contain', '-53.4%'); metricRow('p50Latency').find('[data-tone="better"]').should('exist'); + // $1.17 ÷ 5.3741 videos/GPU-hr vs. $1.73 ÷ 11.526, both per participating GPU. metricRow('dollarsPerVideo') .should('contain', '$0.218') .and('contain', '$0.150') .and('contain', '-31.1%'); - metricRow('powerPctCap').find('[data-tone="better"]').should('exist'); + metricRow('kjPerVideo') + .should('contain', '431.1') + .and('contain', '301.1') + .and('contain', '-30.2%'); + metricRow('kjPerVideo').find('[data-tone="better"]').should('exist'); + cy.contains('[data-testid="video-compare"] p', 'Cost tier:').should( + 'have.text', + 'Cost tier: Owning at Large Hyperscaler Volume.', + ); cy.get('[data-testid="video-compare"]') - .should('contain', 'Owning at Large Hyperscaler Volume') - .and('contain', 'retained CI outputs'); + .should('contain', 'retained CI outputs') + .and('not.contain', 'GPU basis'); + // Arena only: no blind mode and no case dropdown, just the two hardware selects. + cy.get('[data-testid="video-compare-blind"]').should('not.exist'); + cy.get('[data-testid="video-compare-controls"] [role="combobox"]').should(($boxes) => { + expect([...$boxes].map((el) => el.getAttribute('aria-label'))).to.deep.equal([ + 'Baseline', + 'Candidate', + ]); + }); cy.get('@h200.all').should('have.length', 0); }); - it('steps through cases with the arrows and the select, keeping v_case in the URL', () => { + it('steps through cases with the arrows, keeping v_case in the URL', () => { mount(); cy.wait(['@h100', '@b200']); cy.get('[data-testid="video-compare-prev"]').should('be.disabled'); + cy.get('[role="combobox"][aria-label="Case (prompt · seed)"]').should('not.exist'); cy.get('[data-testid="video-compare-next"]').click(); cy.get('[data-testid="video-compare-clip"] video').each(($video) => expect($video.attr('src')).to.include('measurement-r002-c001.mp4'), ); + cy.get('[data-testid="video-compare-case-of"]').should('contain', 'Case 2 of'); cy.location('search').should('contain', 'v_case=1'); - pick('Case (prompt · seed)', '#3'); + cy.get('[data-testid="video-compare-next"]').click(); cy.get('[data-testid="video-compare-clip"] video').each(($video) => expect($video.attr('src')).to.include('measurement-r003-c001.mp4'), ); cy.get('[data-testid="video-compare-case-of"]').should('contain', 'Case 3 of'); cy.location('search').should('contain', 'v_case=2'); cy.get('[data-testid="video-compare-prev"]').click(); + cy.get('[data-testid="video-compare-case-of"]').should('contain', 'Case 2 of'); cy.location('search').should('contain', 'v_case=1'); }); it('drives both players together', () => { @@ -123,23 +153,6 @@ describe('Video compare panel (retained fixture)', () => { cy.get('[data-testid="video-compare-restart"]').click(); cy.get('@play').should('have.callCount', 4); }); - it('hides which hardware is which in blind mode until revealed', () => { - mount(); - cy.wait(['@h100', '@b200']); - cy.get('[data-testid="video-compare-blind"]').click(); - cy.get('[data-testid="video-compare-pane-label"]').should(($labels) => { - expect([...$labels].map((el) => el.textContent)).to.deep.equal(['A', 'B']); - }); - clip('baseline').should('not.contain', 'H100').and('contain', 'time to video 120 s'); - cy.get('[data-testid="video-compare-table"]').should('not.exist'); - cy.get('[data-testid="video-compare-metrics-hidden"]').should('exist'); - cy.get('[data-testid="video-compare-reveal"]').click(); - cy.get('[data-testid="video-compare-pane-label"]').first().should('contain', 'H100'); - cy.get('[data-testid="video-compare-table"]').should('exist'); - // The next case is blind again. - cy.get('[data-testid="video-compare-next"]').click(); - cy.get('[data-testid="video-compare-pane-label"]').first().should('have.text', 'A'); - }); it('switches the candidate, fetching only the new side, and swaps on collision or request', () => { mount(); cy.wait(['@h100', '@b200']); @@ -148,7 +161,8 @@ describe('Video compare panel (retained fixture)', () => { cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H200'); clip('candidate').should('contain', 'H200'); metricRow('p50Latency').should('contain', '150.6').and('contain', '-10%'); - metricRow('dollarsPerVideo').should('contain', '$0.204'); + metricRow('dollarsPerVideo').should('contain', '$0.204').and('contain', '-6.2%'); + metricRow('kjPerVideo').should('contain', '411').and('contain', '-4.7%'); cy.location('search').should('contain', 'v_cand=h200'); cy.get('@h100.all').should('have.length', 1); cy.get('@h200.all').should('have.length', 1); @@ -156,14 +170,14 @@ describe('Video compare panel (retained fixture)', () => { pick('Candidate', 'H100'); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H200'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); - metricRow('p50Latency').find('[data-tone="worse"]').should('exist'); + metricRow('p50Latency').should('contain', '+11.1%').find('[data-tone="worse"]').should('exist'); cy.get('[data-testid="video-compare-swap"]').click(); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H200'); cy.get('@h200.all').should('have.length', 1); }); it('restores v_base and v_cand from the URL and reports a failed media read with a retry', () => { - mount('/video', '?v_base=h200&v_cand=h100', points, OWNING_PARTICIPATING, () => { + mount('/video', '?v_base=h200&v_cand=h100', points, OWNING, () => { cy.intercept('GET', mediaUrl(c1('h200')), { statusCode: 404, body: { error: 'gone' } }).as( 'missing', ); @@ -177,7 +191,7 @@ describe('Video compare panel (retained fixture)', () => { cy.contains('button', 'Retry').should('exist'); }); it('says when a run publishes no clips and keeps the metric deltas', () => { - mount('/video', '', points, OWNING_PARTICIPATING, () => { + mount('/video', '', points, OWNING, () => { cy.intercept('GET', mediaUrl(c1('b200')), { statusCode: 204 }).as('unpublished'); }); cy.wait(['@h100', '@unpublished']); @@ -202,16 +216,20 @@ describe('Video compare panel (retained fixture)', () => { cy.get('[data-testid="video-compare-table"]').should('not.exist'); }); it('renders Chinese copy on /zh/video', () => { - mount('/zh/video', '', points, { tier: 'r', basis: 'allocated' }); + mount('/zh/video', '', points, { tier: 'r' }); cy.wait(['@h100', '@b200']); cy.get('[data-testid="video-compare"]') .should('contain', '基线') .and('contain', '候选') .and('contain', '同时播放') - .and('contain', '盲测模式') - .and('contain', '租赁 - 3 年承诺') - .and('contain', '已分配的 GPU'); + .and('contain', '成本档位:租赁 - 3 年承诺。'); metricRow('p50Latency').should('contain', 'P50 出片时间'); + // Rent - 3 Year Commit: $2.00 ÷ 5.3741 vs. $3.70 ÷ 11.526 per participating GPU-hour. + metricRow('dollarsPerVideo') + .should('contain', '每条视频 TCO 成本') + .and('contain', '$0.372') + .and('contain', '$0.321'); + metricRow('kjPerVideo').should('contain', '每条视频 GPU 板卡能耗'); cy.get('[data-testid="video-compare-case-of"]').should('contain', '第 1 /'); clip('baseline').should('contain', '出片时间 120 s'); }); diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index f460ae9a6..07e035b6f 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -23,78 +23,161 @@ const kpi = (hardware: string) => cy.get(`[data-testid="video-kpi-card"][data-hardware="${hardware}"]`); const points = () => cy.get('[data-testid="video-hardware-chart"] circle.point'); const lines = () => cy.get('[data-testid="video-hardware-chart"] path.line-path'); -const hull = () => cy.get('[data-testid="video-hardware-chart"] path.roofline-path'); +const tableRows = () => cy.get('[data-testid="video-points-table"] tbody tr'); +const headers = () => cy.get('[data-testid="video-points-table"] thead th'); +// Roster order (HW_REGISTRY sort): the B200 and H100 jobs reserved 8 boards and generated on 4. +const IDLE_NOTE = + 'Per participating GPU: B200 (4 of 8), H100 (4 of 8) reserved more boards than one video uses; the idle boards are not counted.'; +const IDLE_NOTE_ZH = + '按参与计算的 GPU 计:B200(4 / 8 张)、H100(4 / 8 张)预留的板卡多于单条视频所需,空闲板卡未计入。'; +const ALL_COLUMNS = [ + 'Hardware', + 'Deployment', + 'Valid / scheduled', + 'P50 time to video (s)', + 'P90 time to video (s)', + 'Videos per GPU-hour', + 'Videos per $1 TCO (Owning at Large Hyperscaler Volume)', + 'TCO cost per video (Owning at Large Hyperscaler Volume)', + 'API list price per video', + 'GPU-board energy per video (kJ)', + 'Mean board power / enforced limit (%)', + 'Board power (W)', + 'CI run', +]; describe('Video hardware dashboard (retained fixture)', () => { it('plots one C1 point per measured hardware and lists MI355X as unavailable', () => { mount(); points().should('have.length', 3); lines().should('have.length', 0); - hull().should('have.length', 0); + cy.get('[data-testid="video-hardware-chart"] path.roofline-path').should('not.exist'); cy.get('[data-testid="video-chart-caption"]').should( 'contain', 'One deployment measured per hardware so far (4 GPU · TP2 × Ulysses 2)', ); - cy.get('[data-testid="video-config-bar"]').should('contain', '4 GPU · TP2 × Ulysses 2'); + cy.get('[data-testid="video-chart-card"]') + .should( + 'contain', + 'MiniMaxAI/MiniMax-H3 @ 42ed227ee7df · Videos per $1 TCO (Owning at Large Hyperscaler Volume) vs. P90 time to video (s)', + ) + .and('contain', 'Cost tier: Owning at Large Hyperscaler Volume'); cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); kpi('mi355x').find('[data-testid="video-kpi-unavailable"]').should('contain', 'Not measured'); - kpi('h200').should('contain', '$0.204').and('contain', '150.6').and('contain', '97.4'); - kpi('b200').should('contain', '$0.150'); - kpi('h100').should('contain', '$0.218').and('contain', '4 of 8 GPUs · TP2 × Ulysses 2'); - cy.get('[data-testid="video-config-bar"]').should( - 'contain', - '1344 × 768 · 8 s · 24 fps · 50 steps', - ); - cy.get('[data-testid="video-chart-card"]').should( - 'contain', - 'MiniMaxAI/MiniMax-H3 @ 42ed227ee7df', - ); + // H200: 20 valid ÷ (3013.4 s × 4 participating GPUs) = 5.97 videos/GPU-hr; $1.22 ÷ 5.9732 = $0.204. + kpi('h200') + .should('contain', '4 of 4 GPUs · TP2 × Ulysses 2') + .and('contain', '150.6') + .and('contain', 'P90 151.1') + .and('contain', '5.97') + .and('contain', '$0.204') + .and('contain', '411'); + kpi('b200').should('contain', '11.53').and('contain', '$0.150').and('contain', '301.1'); + // H100 divides by the 4 boards that generated, not the 8 the job reserved: $1.17 ÷ 5.3741. + kpi('h100') + .should('contain', '4 of 8 GPUs · TP2 × Ulysses 2') + .and('contain', '5.37') + .and('contain', '$0.218') + .and('contain', '431.1'); + kpi('h100').should('not.contain', 'Profit').and('not.contain', 'Board power'); + }); + it('states model, workload and deployment as facts and offers no GPU basis or legend switches', () => { + mount(); + cy.get('[data-testid="video-config-fact"]').should(($facts) => { + expect([...$facts].map((el) => el.textContent)).to.deep.equal([ + 'ModelMiniMaxAI/MiniMax-H3 @ 42ed227ee7df', + 'Workload1344 × 768 · 8 s · 24 fps · 50 steps', + 'Deployment4 GPU · TP2 × Ulysses 2', + ]); + }); + cy.get('[data-testid="video-config-bar"] [role="combobox"]').should(($boxes) => { + expect([...$boxes].map((el) => el.getAttribute('aria-label'))).to.deep.equal([ + 'X-axis metric', + 'Y-axis metric', + 'Cost tier', + ]); + }); + cy.get('[role="combobox"][aria-label="GPU basis"]').should('not.exist'); + cy.get('[data-testid="video-legend"]').should('contain', 'H100').and('contain', 'MI355X'); + cy.get('[data-testid="video-legend"] [role="switch"]').should('not.exist'); + for (const id of ['video-queue', 'video-optimal', 'video-frontier']) + cy.get(`[data-testid="${id}"]`).should('not.exist'); + cy.get('[data-testid="video-idle-note"]').should('have.text', IDLE_NOTE); }); - it('switches cost tier and GPU basis, reflecting both in the URL and the cards', () => { + it('switches the cost tier, repricing the cards, the badges and the URL', () => { mount(); cy.get('[role="combobox"][aria-label="Cost tier"]').click(); cy.contains('[role="option"]', 'Rent - 3 Year Commit').click(); - // 2.9 $/hr ÷ 5.9732 videos/GPU-hr = 0.48550 → rounds down at three decimals. + // $2.90/GPU-hr ÷ 5.9732 videos/GPU-hr = 0.48550 → three decimals. kpi('h200').should('contain', '$0.485'); + kpi('h100').should('contain', '$0.372'); + kpi('b200').should('contain', '$0.321'); cy.location('search').should('contain', 'v_tier=r'); + cy.get('[data-testid="video-chart-card"]') + .should('contain', 'Cost tier: Rent - 3 Year Commit') + .and('contain', 'Videos per $1 TCO (Rent - 3 Year Commit)'); cy.get('[data-testid="video-tco-badge"]') .first() .should('contain', 'B200') .and('contain', '3.70'); - cy.get('[role="combobox"][aria-label="GPU basis"]').click(); - cy.contains('[role="option"]', 'Allocated GPUs').click(); - kpi('h100').should('contain', '$0.744'); - cy.location('search').should('contain', 'v_basis=allocated'); }); - it('plots queued cells as unjoined markers and never draws a curve through them', () => { - mount(); - cy.get('[data-testid="video-queue"]').click(); - points().should('have.length', 9); + it('never plots queued cells, even when a stale URL asks for them', () => { + // C2/C4 exceed the single replica, so they stay in the evidence panel; the retired + // v_queue/v_opt/v_frontier/v_basis params change neither what is drawn nor what is billed. + mount('/video', '?v_queue=1&v_opt=1&v_frontier=1&v_basis=allocated'); + points().should('have.length', 3); lines().should('have.length', 0); - cy.get('[data-testid="video-chart-caption"]').should('contain', 'never joined into a curve'); - cy.location('search').should('contain', 'v_queue=1'); + cy.get('[data-testid="video-chart-caption"]').should('not.contain', 'queued'); + kpi('h100').should('contain', '$0.218'); cy.contains('button', 'Table').click(); - cy.get('[data-testid="video-points-table"] tbody tr').should('have.length', 9); - cy.get('[data-testid="video-points-table"]').should('contain', '2 · queued'); - cy.contains('button', 'Chart').click(); - // Optimal-only keeps every hardware's single deployment and hides the queued cells. - cy.get('[data-testid="video-optimal"]').click(); - points().should('have.length', 3); - cy.location('search').should('contain', 'v_opt=1'); + tableRows().should('have.length', 3); + cy.get('[data-testid="video-points-table"]').should('not.contain', 'queued'); + // Only C1 is scheduled once per replica: every row shows the full 20 / 20 cohort once. + cy.get('[data-testid="data-table-preset-all"]').click(); + tableRows().each(($row) => expect($row.text()).to.include('20 / 20')); + cy.get('[data-testid="video-idle-note"]').should('have.text', IDLE_NOTE); }); - it('explains the cross-hardware frontier switch when one hardware dominates', () => { - mount(); - cy.get('[data-testid="video-frontier"]').click(); - cy.location('search').should('contain', 'v_frontier=1'); - hull().should('have.length', 0); - cy.get('[data-testid="video-chart-caption"]').should('contain', 'one hardware dominates'); + it('lists the plotted cells with the full column set on demand and no concurrency column', () => { + mount('/video', '?v_view=table'); + cy.get('[data-testid="video-hardware-chart"]').should('not.exist'); + tableRows().should('have.length', 3); + headers().should(($ths) => { + expect([...$ths].map((el) => el.textContent?.trim())).to.deep.equal([ + ...ALL_COLUMNS.slice(0, 2), + ALL_COLUMNS[3], + ALL_COLUMNS[6], + ]); + }); + cy.get('[data-testid="data-table-preset-all"]').click(); + headers().should(($ths) => { + expect([...$ths].map((el) => el.textContent?.trim())).to.deep.equal(ALL_COLUMNS); + }); + // B200 C1: 77.9 s / 78.3 s, 11.53 videos/GPU-hr, 6.66 videos per $1, $0.150, $0.272 API, 301.1 kJ, 96.4 %, 3,856 W. + cy.contains('[data-testid="video-points-table"] tbody tr', 'B200').should(($row) => { + const cells = [...$row.find('td')].map((el) => el.textContent?.trim()); + expect(cells.slice(0, 12)).to.deep.equal([ + 'B200', + '4 GPU · TP2 × Ulysses 2', + '20 / 20', + '77.9', + '78.3', + '11.53', + '6.66', + '$0.150', + '$0.272', + '301.1', + '96.4', + '3,856', + ]); + expect(cells[12]).to.equal('#34341996789'); + }); }); it('hides a hardware from the legend and shows the table view', () => { mount(); cy.contains('[data-testid="video-legend"] li', 'H100').click(); points().should('have.length', 2); cy.contains('button', 'Table').click(); - cy.get('[data-testid="video-points-table"] tbody tr').should('have.length', 2); + tableRows().should('have.length', 2); cy.get('[data-testid="video-points-table"]') .should('contain', 'H200') .and('contain', 'B200') @@ -103,69 +186,50 @@ describe('Video hardware dashboard (retained fixture)', () => { cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); }); it('restores v_ params from the URL and opens the runs section for run deep links', () => { - mount('/video', '?v_y=kjPerVideo&v_x=genSpeed&v_queue=1&view=history'); - points().should('have.length', 9); - cy.get('[data-testid="video-chart-card"]').should('contain', 'GPU-board energy per video (kJ)'); + mount('/video', '?v_y=kjPerVideo&v_x=p50Latency&v_tier=r&view=history'); + points().should('have.length', 3); + cy.get('[data-testid="video-chart-card"]') + .should('contain', 'GPU-board energy per video (kJ) vs. P50 time to video (s)') + .and('contain', 'Cost tier: Rent - 3 Year Commit'); + kpi('h200').should('contain', '$0.485'); cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); cy.get('[data-testid="video-ci-runs"]').should('exist'); }); - it('reprices the API reference, updating the cards, the table and the URL', () => { + it('reprices the API list price beside the TCO cost, in the cards, the table and the URL', () => { mount(); - // 0.034 $/video-s × 8 s clip = $0.272; 5.9732 videos/GPU-hr × $0.272 − $1.22 = $0.40 profit. - kpi('h200').should('contain', 'API $/video').and('contain', '$0.272').and('contain', '$0.40'); - // H100 on the participating basis: 5.3741 × $0.272 − $1.17 = $0.29. - kpi('h100').should('contain', '$0.29'); + // 0.034 $/video-s × 8 s clip = $0.272, read beside each hardware's TCO cost per video. + kpi('h200').should('contain', 'API list $0.272').and('contain', '$0.204'); + kpi('h100').should('contain', 'API list $0.272'); + kpi('b200').should('contain', 'API list $0.272'); + cy.get('[data-testid="video-chart-card"]').should( + 'contain', + 'API reference: $0.034/video-s (2026-09-19)', + ); cy.get('[data-testid="video-api-reference-caption"]') .should('contain', 'Reference $0.034/video-s') .and('contain', '$0.034–$0.047') .and('contain', 'captured 2026-09-19') .and('contain', 'MiniMax Design'); cy.get('[data-testid="video-api-price"]').clear().type('0.05'); - // $0.400 per clip; 5.9732 × $0.400 − $1.22 = $1.17 profit. - kpi('h200').should('contain', '$0.400').and('contain', '$1.17'); + // $0.400 per 8 s clip; the TCO cost itself does not move. + kpi('h200').should('contain', 'API list $0.400').and('contain', '$0.204'); + kpi('h200').should('not.contain', '$0.272'); cy.location('search').should('contain', 'v_api=0.05'); cy.contains('button', 'Table').click(); cy.get('[data-testid="data-table-preset-all"]').click(); cy.get('[data-testid="video-points-table"] thead') - .should('contain', 'Revenue per GPU-hour at API list price') - .and('contain', 'API list price ÷ TCO cost per video (Owning at Large Hyperscaler Volume)'); - // H200 row: revenue 5.9732 × $0.400 = $2.39; multiple $0.400 ÷ $0.2042 = 1.96. + .should('contain', 'API list price per video') + .and('contain', 'TCO cost per video (Owning at Large Hyperscaler Volume)') + .and('not.contain', 'Revenue') + .and('not.contain', 'Profit'); cy.contains('[data-testid="video-points-table"] tbody tr', 'H200') - .should('contain', '$2.39') - .and('contain', '1.96'); + .should('contain', '$0.400') + .and('contain', '$0.204'); cy.get('[data-testid="video-api-price-reset"]').click(); - cy.contains('[data-testid="video-points-table"] tbody tr', 'H200').should('contain', '$1.62'); + cy.contains('[data-testid="video-points-table"] tbody tr', 'H200').should('contain', '$0.272'); + kpi('h200').should('contain', 'API list $0.272'); cy.location('search').should('not.contain', 'v_api'); }); - it('keeps negative profit per GPU-hour on the canvas on the allocated basis', () => { - mount('/video', '?v_y=profitPerGpuHour&v_basis=allocated'); - // H100 (−$0.44) and B200 (−$0.16) bill idle boards; only H200 (+$0.40) stays positive. - points().should('have.length', 3); - cy.get('[data-testid="video-hardware-chart"] .y-axis .tick text').should(($ticks) => { - const labels = [...$ticks].map((el) => el.textContent ?? ''); - expect(labels.some((label) => /^[-−]/u.test(label))).to.equal(true); - }); - cy.get('[data-testid="video-chart-card"]').should( - 'contain', - 'Profit per GPU-hour, API list price − TCO (Owning at Large Hyperscaler Volume)', - ); - kpi('h100').should('contain', '-$0.44'); - kpi('b200').should('contain', '-$0.16'); - }); - it('caps the y axis at break-even when every hardware loses money at the rental tier', () => { - mount('/video', '?v_y=profitPerGpuHour&v_tier=r'); - // $0.034/video-s on the participating basis against $2.00/$2.90/$3.70 GPU-hr: H100 −$0.54, H200 −$1.28, B200 −$0.56. - points().should('have.length', 3); - kpi('h100').should('contain', '-$0.54'); - kpi('h200').should('contain', '-$1.28'); - kpi('b200').should('contain', '-$0.56'); - cy.get('[data-testid="video-hardware-chart"] .y-axis .tick text').should(($ticks) => { - const values = [...$ticks].map((el) => Number((el.textContent ?? '').replace('−', '-'))); - // Regression: a zero ceiling used to fall back to 1, leaving the top half of the canvas empty. - expect(Math.max(...values)).to.equal(0); - expect(Math.min(...values)).to.be.below(-1.2); - }); - }); it('shows placeholders while loading instead of a false "Not measured"', () => { cy.intercept('GET', '/api/video-runs?format=history&page=1', (req) => { req.reply({ fixture: 'api/video-history.json', delay: 800 }); @@ -181,22 +245,52 @@ describe('Video hardware dashboard (retained fixture)', () => { ); cy.get('[data-testid="video-kpi-skeleton"]').should('have.length', 4); cy.get('[data-testid="video-kpi-unavailable"]').should('not.exist'); + cy.get('[data-testid="video-idle-note"]').should('not.exist'); cy.wait('@slowHistory'); cy.get('[data-testid="video-kpi-skeleton"]').should('not.exist'); kpi('h200').should('contain', '$0.204'); kpi('mi355x').find('[data-testid="video-kpi-unavailable"]').should('exist'); + cy.get('[data-testid="video-idle-note"]').should('have.text', IDLE_NOTE); }); it('renders Chinese copy on /zh/video', () => { mount('/zh/video'); - cy.get('[data-testid="video-chart-card"]').should( + cy.get('[data-testid="video-chart-card"]') + .should('contain', '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)') + .and('contain', '成本档位: Hyperscaler 自有设备'); + cy.get('[data-testid="video-chart-caption"]').should( 'contain', - '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', + '目前每种硬件只测得一种部署(4 张 GPU · TP2 × Ulysses 2)', ); + cy.get('[data-testid="video-idle-note"]').should('have.text', IDLE_NOTE_ZH); kpi('mi355x').should('contain', '未测得'); - kpi('h200').should('contain', 'API 标价 / 条视频').and('contain', '利润 / GPU 小时'); + kpi('h200') + .should('contain', '4 / 4 张 GPU') + .and('contain', 'P50 出片时间(s)') + .and('contain', '视频数 / GPU 小时') + .and('contain', 'TCO / 条视频') + .and('contain', 'API 标价 $0.272') + .and('contain', 'kJ / 条视频'); + cy.get('[data-testid="video-config-fact"]').should(($facts) => { + expect([...$facts].map((el) => el.textContent)).to.deep.equal([ + '模型MiniMaxAI/MiniMax-H3 @ 42ed227ee7df', + '工作负载1344 × 768 · 8 s · 24 fps · 50 steps', + '部署4 张 GPU · TP2 × Ulysses 2', + ]); + }); cy.get('[data-testid="video-config-bar"]') - .should('contain', '成本档位') + .should('contain', 'X 轴指标') + .and('contain', 'Y 轴指标') + .and('contain', '成本档位') .and('contain', 'API 参考价($/video-s)') .and('contain', '采集于 2026-09-19'); + cy.contains('button', '表格').click(); + headers().should(($ths) => { + expect([...$ths].map((el) => el.textContent?.trim())).to.deep.equal([ + '硬件', + '部署', + 'P50 出片时间(s)', + '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', + ]); + }); }); }); diff --git a/packages/app/cypress/component/video-evidence.cy.tsx b/packages/app/cypress/component/video-evidence.cy.tsx index c21d7d287..129287cfb 100644 --- a/packages/app/cypress/component/video-evidence.cy.tsx +++ b/packages/app/cypress/component/video-evidence.cy.tsx @@ -1,21 +1,18 @@ import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; import type { VideoHistoryPage } from '@/components/video-benchmark/history'; -import type { GpuBasis, VideoPoint } from '@/components/video-benchmark/metrics'; +import type { VideoPoint } from '@/components/video-benchmark/metrics'; import { videoPoints } from '@/components/video-benchmark/points'; import VideoEvidence from '@/components/video-benchmark/VideoEvidence'; // Retained H100/H200/B200 C1/C2/C4 observations (cypress/fixtures/api/video-history.json). -function mount( - pathname = '/video', - basis: GpuBasis = 'participating', - select = (points: VideoPoint[]) => points, -) { +// Nothing in the panel is priced, so it takes no metric options: every per-GPU number +// divides by the boards that generated the clip. +function mount(pathname = '/video', select = (points: VideoPoint[]) => points) { cy.fixture('api/video-history.json').then((page: VideoHistoryPage) => { cy.mount( (key === 'b200' ? 'rgb(1, 2, 3)' : 'rgb(4, 5, 6)')} /> , @@ -51,7 +48,7 @@ describe('Video compute-bound evidence (retained fixture)', () => { it('bounds the power meter to its declared range when mean power exceeds the recorded limit', () => { // 2,856 W against the recorded 2,800 W limit: the label keeps the measured 102.0%, // the meter reports and fills its 0–100 range. - mount('/video', 'participating', (points) => + mount('/video', (points) => points .filter((p) => p.hardwareKey === 'h200' && p.concurrency === 1) .map((p) => ({ ...p, avgPowerW: 2856 })), @@ -73,12 +70,18 @@ describe('Video compute-bound evidence (retained fixture)', () => { mount(); exhibit('plateau').find('tbody tr').should('have.length', 9); exhibit('plateau').find('tbody th[scope="rowgroup"]').should('have.length', 3); + // Throughput divides by the 4 participating boards, never the 8 the B200/H100 jobs reserved. exhibit('plateau') .find('tr[data-hardware="b200"][data-concurrency="4"]') .should('contain', '11.69') .and('contain', '307.3') .and('contain', '1.01×') .and('contain', '3.94×'); + exhibit('plateau') + .find('tr[data-hardware="h100"][data-concurrency="1"]') + .should('contain', '5.37') + .and('contain', '167.3') + .and('contain', '1.00×'); exhibit('plateau') .find('tr[data-hardware="h100"][data-concurrency="4"]') .should('contain', '5.37') @@ -123,22 +126,11 @@ describe('Video compute-bound evidence (retained fixture)', () => { exhibit('caveats') .should('contain', 'Batch-one, single-replica server') .and('contain', '4 participating GPUs per video (TP2 × Ulysses 2)') + .and('contain', 'allocated boards beyond those sit idle') .and('contain', 'not node, rack or facility power') .and('contain', 'n = 20 clips per cell') .and('contain', 'Frozen workload 1344 × 768 · 8 s · 24 fps · 50 steps'); }); - it('switches the throughput column to the allocated basis without moving the ratios', () => { - mount('/video', 'allocated'); - exhibit('plateau') - .find('tr[data-hardware="h100"][data-concurrency="1"]') - .should('contain', '2.69') - .and('contain', '1.00×'); - exhibit('plateau') - .find('tr[data-hardware="h100"][data-concurrency="4"]') - .should('contain', '2.68') - .and('contain', '4.00×'); - exhibit('plateau-reading').should('contain', 'within 1.4% of C1'); - }); it('renders Chinese copy on /zh/video', () => { mount('/zh/video'); cy.get('[data-testid="video-evidence"]') @@ -161,19 +153,9 @@ describe('Video compute-bound evidence (retained fixture)', () => { .and('contain', '每个 cell n = 20 条视频'); }); it('hides the plateau and scaling exhibits when only one hardware at C1 is measured', () => { - cy.fixture('api/video-history.json').then((page: VideoHistoryPage) => { - cy.mount( - - p.hardwareKey === 'h200' && p.concurrency === 1, - )} - options={{ tier: 'h', basis: 'participating' }} - colorFor={() => 'rgb(0, 0, 0)'} - /> - , - ); - }); + mount('/video', (points) => + points.filter((p) => p.hardwareKey === 'h200' && p.concurrency === 1), + ); exhibit('power').find('[data-testid="video-evidence-power-row"]').should('have.length', 1); exhibit('power-reading').should('contain', 'H200 ran at 97% of its enforced power limit'); exhibit('plateau').should('not.exist'); @@ -183,11 +165,7 @@ describe('Video compute-bound evidence (retained fixture)', () => { it('renders nothing when no cell is measured', () => { cy.mount( - 'rgb(0, 0, 0)'} - /> + 'rgb(0, 0, 0)'} /> , ); cy.get('[data-testid="video-evidence"]').should('not.exist'); diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts index 438c771fc..8cc60af1b 100644 --- a/packages/app/cypress/e2e/video-dashboard.cy.ts +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -1,14 +1,18 @@ // Runs against the E2E_FIXTURES=1 server, which serves the retained -// H100/H200/B200 observations from cypress/fixtures/api/video-history.json. +// H100/H200/B200 C1/C2/C4 observations from cypress/fixtures/api/video-history.json. +// Lead (C1) cells, per participating GPU: H100 5.37 videos/GPU-hr at $1.17/GPU-hr, +// H200 5.97 at $1.22, B200 11.53 at $1.73. Every clip is 8 s, so the $0.034/video-s +// API reference lists at $0.272 per video. describe('Video hardware dashboard (E2E fixtures)', () => { beforeEach(() => { // The Compare panel fetches two published artifacts once in view; keep the spec offline. - cy.intercept('GET', '/api/video-runs?run=*', { statusCode: 204 }); + cy.intercept('GET', '/api/video-runs?run=*', { statusCode: 204 }).as('media'); }); it('leads with the cross-hardware chart and restores v_ params from the URL', () => { - cy.visit('/video?v_y=kjPerVideo&v_tier=r&v_queue=1'); - cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 9); - // Queued C2/C4 cells are markers only: no curve is drawn through them. + cy.visit('/video?v_y=kjPerVideo&v_tier=r'); + // One measured deployment per hardware: three points, no frontier line, and the + // queued C2/C4 cells never reach the chart. + cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 3); cy.get('[data-testid="video-hardware-chart"] path.line-path').should('have.length', 0); cy.get('[data-testid="video-chart-caption"]').should( 'contain', @@ -17,12 +21,202 @@ describe('Video hardware dashboard (E2E fixtures)', () => { cy.get('[data-testid="video-chart-card"]') .should('contain', 'GPU-board energy per video (kJ)') .and('contain', 'Rent - 3 Year Commit'); + cy.get('[data-testid="video-idle-note"]').should( + 'have.text', + 'Per participating GPU: B200 (4 of 8), H100 (4 of 8) reserved more boards than one video uses; the idle boards are not counted.', + ); cy.get('[data-testid="video-tco-badge"]').first().should('contain', 'B200'); cy.get('[data-testid="video-replay"]').should('be.visible'); + // The campaign is frozen: model, workload and deployment are facts; only axes and tier select. + cy.get('[data-testid="video-config-fact"]').should('have.length', 3); + cy.get('[data-testid="video-config-fact"]') + .eq(0) + .should('contain', 'Model') + .and('contain', 'MiniMaxAI/MiniMax-H3 @ 42ed227ee7df'); + cy.get('[data-testid="video-config-fact"]') + .eq(1) + .should('contain', 'Workload') + .and('contain', '1344 × 768 · 8 s · 24 fps · 50 steps'); + cy.get('[data-testid="video-config-fact"]') + .eq(2) + .should('contain', 'Deployment') + .and('contain', '4 GPU · TP2 × Ulysses 2'); + cy.get('[data-testid="video-config-bar"] [role="combobox"]').should('have.length', 3); + cy.get('[data-testid="video-config-bar"] [role="combobox"]') + .eq(0) + .should('contain', 'P90 time to video (s)'); + cy.get('[data-testid="video-config-bar"] [role="combobox"]') + .eq(1) + .should('contain', 'GPU-board energy per video (kJ)'); + cy.get('[data-testid="video-config-bar"] [role="combobox"]') + .eq(2) + .should('contain', 'Rent - 3 Year Commit'); + cy.get('[data-testid="video-config-bar"]').should('not.contain', 'GPU basis'); + cy.get('[data-testid="video-legend"]').should('contain', 'MI355X · not measured'); + cy.get('[data-testid="video-legend"] [role="switch"]').should('not.exist'); + // KPI cards read each hardware's lead deployment at the selected tier (Rent - 3 Year Commit: + // $2 / $2.9 / $3.7 per GPU-hour); the API list price sits beside the TCO cost. cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); + cy.get('[data-testid="video-kpi-card"][data-hardware="b200"]') + .should('contain', '4 of 8 GPUs · TP2 × Ulysses 2') + .and('contain', 'P50 time to video (s)') + .and('contain', '77.9') + .and('contain', 'P90 78.3') + .and('contain', 'Videos / GPU-hr') + .and('contain', '11.53') + .and('contain', 'TCO / video') + .and('contain', '$0.321') + .and('contain', 'API list $0.272') + .and('contain', 'kJ / video') + .and('contain', '301.1') + .and('not.contain', 'Profit'); + cy.get('[data-testid="video-kpi-card"][data-hardware="h200"]') + .should('contain', '4 of 4 GPUs') + .and('contain', '$0.485'); + cy.get('[data-testid="video-kpi-card"][data-hardware="h100"]') + .should('contain', '4 of 8 GPUs') + .and('contain', '$0.372'); cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); cy.get('[data-testid="video-ci-runs"]').should('not.exist'); }); + it('lists only the measured deployments in the table view, without a concurrency column', () => { + cy.visit('/video?v_view=table'); + cy.get('[data-testid="video-hardware-chart"]').should('not.exist'); + // One row per hardware: the queued C2/C4 cells stay out, so the C4-only H100 run never appears. + cy.get('[data-testid="video-points-table"] tbody tr').should('have.length', 3); + cy.get('[data-testid="video-points-table"]').should('not.contain', '#34344378350'); + cy.get('[data-testid="data-table-preset-all"]').click(); + cy.get('[data-testid="video-points-table"] thead th').should(($headers) => { + expect([...$headers].map((th) => th.textContent?.trim())).to.deep.equal([ + 'Hardware', + 'Deployment', + 'Valid / scheduled', + 'P50 time to video (s)', + 'P90 time to video (s)', + 'Videos per GPU-hour', + 'Videos per $1 TCO (Owning at Large Hyperscaler Volume)', + 'TCO cost per video (Owning at Large Hyperscaler Volume)', + 'API list price per video', + 'GPU-board energy per video (kJ)', + 'Mean board power / enforced limit (%)', + 'Board power (W)', + 'CI run', + ]); + }); + cy.contains('[data-testid="video-points-table"] tbody tr', 'B200') + .should('contain', '4 GPU · TP2 × Ulysses 2') + .and('contain', '20 / 20') + .and('contain', '77.9') + .and('contain', '78.3') + .and('contain', '11.53') + .and('contain', '6.66') + .and('contain', '$0.150') + .and('contain', '$0.272') + .and('contain', '301.1') + .and('contain', '96.4') + .and('contain', '3,856') + .and('contain', '#34341996789'); + cy.contains('[data-testid="video-points-table"] tbody tr', 'H200') + .should('contain', '150.6') + .and('contain', '5.97') + .and('contain', '4.9') + .and('contain', '$0.204') + .and('contain', '411') + .and('contain', '97.4') + .and('contain', '2,727') + .and('contain', '#34342452354'); + cy.contains('[data-testid="video-points-table"] tbody tr', 'H100') + .should('contain', '167.3') + .and('contain', '5.37') + .and('contain', '4.59') + .and('contain', '$0.218') + .and('contain', '431.1') + .and('contain', '91.9') + .and('contain', '2,575') + .and('contain', '#34336635063'); + }); + it('prices the API list reference from v_api and resets it to the dated default', () => { + cy.visit('/video?v_api=0.05'); + cy.get('[data-testid="video-api-price"]').should('have.value', '0.05'); + cy.get('[data-testid="video-chart-card"]').should('contain', 'API reference: $0.050/video-s'); + // 8 s clips: $0.05/video-s lists at $0.400 per video on every hardware. + cy.get('[data-testid="video-kpi-card"][data-hardware="b200"]').should( + 'contain', + 'API list $0.400', + ); + cy.get('[data-testid="video-kpi-card"][data-hardware="h100"]').should( + 'contain', + 'API list $0.400', + ); + cy.get('[data-testid="video-api-price-reset"]').click(); + cy.get('[data-testid="video-api-price"]').should('have.value', '0.034'); + cy.get('[data-testid="video-kpi-card"][data-hardware="b200"]').should( + 'contain', + 'API list $0.272', + ); + cy.location('search').should('not.contain', 'v_api'); + // A non-positive price never prices at 0: the dated reference stays in force. + cy.visit('/video?v_api=0'); + cy.get('[data-testid="video-api-price"]').should('have.value', '0.034'); + cy.get('[data-testid="video-kpi-card"][data-hardware="h200"]').should( + 'contain', + 'API list $0.272', + ); + cy.get('[data-testid="video-api-reference-caption"]').should('contain', 'captured 2026-09-19'); + }); + it('compares the slowest lead deployment against the fastest with time, cost and energy deltas', () => { + cy.visit('/video'); + cy.get('[data-testid="video-compare"]').scrollIntoView(); + cy.wait('@media'); + // Default pair: slowest C1 P50 (H100) as baseline, fastest (B200) as candidate. Two hardware + // selects and nothing else to pick: no case dropdown, no blind mode. + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'B200'); + cy.get('[data-testid="video-compare"] [role="combobox"]').should('have.length', 2); + cy.get('[data-testid="video-compare-clips"]').should( + 'contain', + 'Clips are not published for one of these runs', + ); + cy.get('[data-testid="video-compare-table"] tbody tr').should(($rows) => { + expect([...$rows].map((row) => row.dataset.metric)).to.deep.equal([ + 'p50Latency', + 'dollarsPerVideo', + 'kjPerVideo', + ]); + }); + cy.get('[data-testid="video-compare-table"] tr[data-metric="p50Latency"]') + .should('contain', '167.3') + .and('contain', '77.9') + .and('contain', '-53.4%') + .and('contain', '×0.47') + .find('[data-tone]') + .should('have.attr', 'data-tone', 'better'); + cy.get('[data-testid="video-compare-table"] tr[data-metric="dollarsPerVideo"]') + .should('contain', '$0.218') + .and('contain', '$0.150') + .and('contain', '-31.1%') + .find('[data-tone]') + .should('have.attr', 'data-tone', 'better'); + cy.get('[data-testid="video-compare-table"] tr[data-metric="kjPerVideo"]') + .should('contain', '431.1') + .and('contain', '301.1') + .and('contain', '-30.2%') + .find('[data-tone]') + .should('have.attr', 'data-tone', 'better'); + cy.get('[data-testid="video-compare"]').should( + 'contain', + 'Cost tier: Owning at Large Hyperscaler Volume.', + ); + // Swapping sides flips the deltas and records the pair in the URL. + cy.get('[data-testid="video-compare-swap"]').click(); + cy.get('[data-testid="video-compare-baseline"]').should('contain', 'B200'); + cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); + cy.get('[data-testid="video-compare-table"] tr[data-metric="p50Latency"]') + .should('contain', '+114.7%') + .find('[data-tone]') + .should('have.attr', 'data-tone', 'worse'); + cy.location('search').should('contain', 'v_base=b200').and('contain', 'v_cand=h100'); + }); it('opens the runs section for history deep links and renders the Chinese dashboard', () => { cy.visit('/video?view=history'); cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); @@ -33,13 +227,27 @@ describe('Video hardware dashboard (E2E fixtures)', () => { 'contain', '每 1 美元 TCO 生成视频数(Hyperscaler 自有设备)', ); + cy.get('[data-testid="video-idle-note"]').should( + 'have.text', + '按参与计算的 GPU 计:B200(4 / 8 张)、H100(4 / 8 张)预留的板卡多于单条视频所需,空闲板卡未计入。', + ); + cy.get('[data-testid="video-config-bar"]') + .should('contain', '模型') + .and('contain', '工作负载') + .and('contain', '4 张 GPU · TP2 × Ulysses 2') + .and('contain', 'X 轴指标') + .and('contain', '成本档位'); + cy.get('[data-testid="video-kpi-card"][data-hardware="b200"]') + .should('contain', 'TCO / 条视频') + .and('contain', '$0.150') + .and('contain', 'API 标价 $0.272'); cy.get('[data-testid="video-kpi-card"][data-hardware="mi355x"]').should('contain', '未测得'); }); it('fits a phone viewport without horizontal page scroll in both locales', () => { cy.viewport(390, 844); - for (const path of ['/video?v_queue=1', '/zh/video']) { + for (const path of ['/video', '/zh/video']) { cy.visit(path); - cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length.at.least', 3); + cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 3); cy.get('[data-testid="video-kpi-card"]').should('have.length', 4); cy.document().then((doc) => { expect(doc.documentElement.scrollWidth, `${path} page width`).to.be.at.most( diff --git a/packages/app/src/components/video-benchmark/VideoCompare.tsx b/packages/app/src/components/video-benchmark/VideoCompare.tsx index ab56117e6..425c7d7c3 100644 --- a/packages/app/src/components/video-benchmark/VideoCompare.tsx +++ b/packages/app/src/components/video-benchmark/VideoCompare.tsx @@ -4,7 +4,6 @@ import { useCallback, useEffect, useId, useMemo, useRef, useState } from 'react' import { Button } from '@/components/ui/button'; import { Heading } from '@/components/ui/heading'; import { Skeleton } from '@/components/ui/skeleton'; -import { Switch } from '@/components/ui/switch'; import { track } from '@/lib/analytics'; import { useLocale } from '@/lib/use-locale'; import { @@ -12,7 +11,6 @@ import { compareMetrics, compareSide, pairCases, - type CasePair, type CompareRow, type CompareSide, } from './compare'; @@ -44,18 +42,11 @@ const STRINGS = { baseline: 'Baseline', candidate: 'Candidate', swap: 'Swap sides', - blind: 'Blind mode', - blindHint: 'Hide which hardware is which until you reveal it.', - reveal: 'Reveal hardware', - hidden: { baseline: 'A', candidate: 'B' }, needTwo: 'Comparison needs at least two measured hardware.', metrics: 'Metric deltas', - metricsHidden: 'Metric deltas are hidden in blind mode; reveal the hardware to see them.', metric: 'Metric', change: 'Change', - assumptions: (tier: string, basis: string) => `Cost tier: ${tier} · GPU basis: ${basis}.`, - participating: 'participating GPUs', - allocated: 'allocated GPUs', + assumptions: (tier: string) => `Cost tier: ${tier}.`, better: 'Candidate better', worse: 'Candidate worse', neutral: 'No difference', @@ -70,7 +61,6 @@ const STRINGS = { caseOf: (i: number, n: number) => `Case ${i} of ${n}`, prev: 'Previous case', next: 'Next case', - case: 'Case (prompt · seed)', playBoth: 'Play both', pauseBoth: 'Pause both', restart: 'Restart both', @@ -90,18 +80,11 @@ const STRINGS = { baseline: '基线', candidate: '候选', swap: '交换两侧', - blind: '盲测模式', - blindHint: '揭晓前隐藏两侧各是哪种硬件。', - reveal: '揭晓硬件', - hidden: { baseline: 'A', candidate: 'B' }, needTwo: '至少需要两种已实测的硬件才能对比。', metrics: '指标差异', - metricsHidden: '盲测模式下隐藏指标差异,揭晓硬件后显示。', metric: '指标', change: '变化', - assumptions: (tier: string, basis: string) => `成本档位:${tier} · GPU 口径:${basis}。`, - participating: '参与计算的 GPU', - allocated: '已分配的 GPU', + assumptions: (tier: string) => `成本档位:${tier}。`, better: '候选更优', worse: '候选更差', neutral: '无差异', @@ -116,7 +99,6 @@ const STRINGS = { caseOf: (i: number, n: number) => `第 ${i} / ${n} 个用例`, prev: '上一个用例', next: '下一个用例', - case: '用例(prompt · seed)', playBoth: '同时播放', pauseBoth: '同时暂停', restart: '从头同时播放', @@ -140,16 +122,6 @@ const CHIP = { neutral: 'border-border text-muted-foreground', } as const; -const excerpt = (prompt: string, max = 60) => - prompt.length > max ? `${prompt.slice(0, max).trimEnd()}…` : prompt; - -function caseLabel(pair: CasePair, repeated: boolean): string { - const parts = [pair.prompt === null ? (pair.caseId ?? '—') : excerpt(pair.prompt)]; - if (pair.seed !== null) parts.push(`seed ${pair.seed}`); - if (repeated) parts.push(`#${pair.repetition + 1}`); - return parts.join(' · '); -} - /** The stored artifact of one side, or null when the deployment publishes no media (HTTP 204). */ async function fetchSide(point: VideoPoint, signal: AbortSignal): Promise { const response = await fetch( @@ -198,7 +170,7 @@ function ClipPlaceholders() { /** * Arena-style baseline vs candidate: the same prompt and seed from both CI runs - * play side by side (blind mode hides which hardware is which), with the metric + * play side by side, with the metric * deltas between each hardware's lead deployment below. Media is fetched once * the panel scrolls into view, one stored artifact per side. */ @@ -214,7 +186,6 @@ export default function VideoCompare({ const locale = useLocale(); const s = STRINGS[locale]; const headingId = useId(); - const blindId = useId(); const sectionRef = useRef(null); const videos = useRef>({ baseline: null, candidate: null }); const [selection, setSelection] = useState( @@ -225,8 +196,6 @@ export default function VideoCompare({ const [failure, setFailure] = useState<{ key: string; message: string } | null>(null); const [failedMedia, setFailedMedia] = useState>(() => new Set()); const [visible, setVisible] = useState(false); - const [blind, setBlind] = useState(false); - const [revealedFor, setRevealedFor] = useState(null); const request = useRef(0); const download = useRef(null); useEffect(() => { @@ -276,12 +245,9 @@ export default function VideoCompare({ value: point.hardwareKey ?? '', label: hardwareLabel(point.hardwareKey ?? ''), })); - const masked = blind && revealedFor !== `${pairKey}:${index}`; const sideOf = { baseline, candidate } as Record; - const paneLabel = (role: Role) => - masked ? s.hidden[role] : hardwareLabel(sideOf[role]?.hardwareKey ?? ''); - const paneColor = (role: Role) => - masked ? 'var(--muted-foreground)' : colorFor(sideOf[role]?.hardwareKey ?? ''); + const paneLabel = (role: Role) => hardwareLabel(sideOf[role]?.hardwareKey ?? ''); + const paneColor = (role: Role) => colorFor(sideOf[role]?.hardwareKey ?? ''); const update = (patch: Partial) => { const next = { ...selection, ...patch }; @@ -371,13 +337,6 @@ export default function VideoCompare({ }); track('video_compare_playback', { action }); }; - const reveal = () => { - setRevealedFor(`${pairKey}:${index}`); - track('video_compare_revealed', { - baseline: baseline?.hardwareKey ?? null, - candidate: candidate?.hardwareKey ?? null, - }); - }; return (
-
-
- - {s.title} - -

{s.subtitle}

-
- +
+ + {s.title} + +

{s.subtitle}

· {s[role]} - {!masked && ( - - · {layoutLabel(point, locale)} - - )} + + · {layoutLabel(point, locale)} + {record.seconds !== null && ( · {s.timeToVideo}{' '} @@ -564,57 +502,33 @@ export default function VideoCompare({ ); })}
-
-
- - - - {masked && ( - - )} -
-
- goTo(Number(value))} - options={pairs.map((pair, i) => ({ - value: String(i), - label: caseLabel(pair, repeated.has(pair.key)), - }))} - /> -
+
+ + +
{(matched.unmatched.baseline > 0 || matched.unmatched.candidate > 0) && (

)}

- {masked ? ( -

- {s.metricsHidden} -

- ) : ( -
-

{s.metrics}

-
- - - -
- {s.metric} +
+

{s.metrics}

+
+ + + + + {ROLES.map((role) => ( + - {ROLES.map((role) => ( - - ))} - + + + + {rows.map((row) => ( + + + + + - - - {rows.map((row) => ( - - - - - - - ))} - -
+ {s.metric} + + + + {paneLabel(role)} + · {s[role]} + - - - {paneLabel(role)} - · {s[role]} - - - {s.change} + ))} + + {s.change} +
+ {locale === 'zh' + ? VIDEO_METRICS[row.id].labelZh + : VIDEO_METRICS[row.id].label} {formatMetric(row.baseline, row.id)}{formatMetric(row.candidate, row.id)} + +
- {locale === 'zh' - ? VIDEO_METRICS[row.id].labelZh - : VIDEO_METRICS[row.id].label} - {formatMetric(row.baseline, row.id)}{formatMetric(row.candidate, row.id)} - -
-
-

- {s.assumptions( - TIER_LABELS[options.tier][locale], - options.basis === 'allocated' ? s.allocated : s.participating, - )} -

+ ))} + +
- )} +

+ {s.assumptions(TIER_LABELS[options.tier][locale])} +

+

{s.footnote}

)} diff --git a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx index 71beba436..2eea0b2fa 100644 --- a/packages/app/src/components/video-benchmark/VideoConfigBar.tsx +++ b/packages/app/src/components/video-benchmark/VideoConfigBar.tsx @@ -10,7 +10,6 @@ import { TIER_LABELS, X_METRICS, Y_METRICS, - type GpuBasis, type XMetricId, type YMetricId, } from './metrics'; @@ -28,9 +27,6 @@ const STRINGS = { x: 'X-axis metric', y: 'Y-axis metric', tier: 'Cost tier', - basis: 'GPU basis', - participating: 'Participating GPUs (boards generating the clip)', - allocated: 'Allocated GPUs (boards the job reserved)', }, zh: { benchmark: '基准测试配置', @@ -41,18 +37,24 @@ const STRINGS = { x: 'X 轴指标', y: 'Y 轴指标', tier: '成本档位', - basis: 'GPU 口径', - participating: '参与计算的 GPU(实际生成视频的板卡)', - allocated: '已分配的 GPU(作业预留的板卡)', }, }; +/** A fixed fact of the campaign, laid out like a control so the panel reads as one row. */ +function Fact({ label, value }: { label: string; value: string }) { + return ( +
+ {label} + {value} +
+ ); +} + /** * Two control panels mirroring the inference tab: what is being compared - * (frozen for this campaign, so model, workload and the measured deployments - * are single-option selects that state the comparison rather than change it) - * and how it is plotted. Queued cells, optimal-only and the cross-hardware - * frontier are legend switches, as on the inference chart. + * (frozen for this campaign, so model, workload and deployment are stated, + * not selectable, beside the one editable input, the API list price) and + * how it is plotted (axes and cost tier). */ export default function VideoConfigBar({ state, @@ -65,7 +67,7 @@ export default function VideoConfigBar({ onChange: (patch: Partial) => void; modelLabel: string; workloadLabel: string; - /** Measured server layouts (GPUs per video and model split), stated rather than selectable. */ + /** Measured server layouts (GPUs per video and model split). */ deploymentLabel: string; }) { const locale = useLocale(); @@ -78,27 +80,12 @@ export default function VideoConfigBar({ return (
- {}} - options={[{ value: 'model', label: modelLabel }]} - /> - {}} - options={[{ value: 'workload', label: workloadLabel }]} - /> - {}} - options={[{ value: 'deployment', label: deploymentLabel }]} - /> + + + onChange({ apiPrice })} /> - + change('tier', value as CostTier)} options={COST_TIERS.map((tier) => ({ value: tier, label: TIER_LABELS[tier][locale] }))} /> - change('basis', value as GpuBasis)} - options={[ - { value: 'participating', label: s.participating }, - { value: 'allocated', label: s.allocated }, - ]} - />
); diff --git a/packages/app/src/components/video-benchmark/VideoDashboard.tsx b/packages/app/src/components/video-benchmark/VideoDashboard.tsx index b83d40374..bdfea7350 100644 --- a/packages/app/src/components/video-benchmark/VideoDashboard.tsx +++ b/packages/app/src/components/video-benchmark/VideoDashboard.tsx @@ -56,19 +56,12 @@ const RUNS_SECTION_PARAMS = [ const CSV_METRICS: readonly MetricId[] = [ 'p50Latency', 'p90Latency', - 'genSpeed', 'videosPerGpuHour', - 'videoSecondsPerGpuHour', 'videosPerDollar', 'dollarsPerVideo', - 'dollarsPerVideoSecond', 'kjPerVideo', - 'videosPerKwh', 'powerPctCap', 'apiPricePerVideo', - 'revenuePerGpuHour', - 'profitPerGpuHour', - 'apiPriceMultiple', ]; const STRINGS = { @@ -83,9 +76,10 @@ const STRINGS = { badges: 'TCO $/chip/hr', source: 'Source', apiReference: 'API reference', - queue: 'Show queued requests (C2/C4)', - optimal: 'Optimal only', - frontier: 'Pareto frontier', + idle: (list: string) => + `Per participating GPU: ${list} reserved more boards than one video uses; the idle boards are not counted.`, + idleItem: (hardware: string, used: number, reserved: number) => + `${hardware} (${used} of ${reserved})`, deployments: (n: number) => `${n} deployments`, runs: 'Runs, videos & evidence', runsHint: 'Per-run results, generated clips, fidelity checks and the performance history list.', @@ -109,9 +103,10 @@ const STRINGS = { badges: 'TCO $/chip/hr', source: '来源', apiReference: 'API 参考价', - queue: '显示排队请求(C2/C4)', - optimal: '仅最优', - frontier: 'Pareto 前沿', + idle: (list: string) => + `按参与计算的 GPU 计:${list}预留的板卡多于单条视频所需,空闲板卡未计入。`, + idleItem: (hardware: string, used: number, reserved: number) => + `${hardware}(${used} / ${reserved} 张)`, deployments: (n: number) => `${n} 种部署`, runs: '运行、视频与证据', runsHint: '按运行查看结果、生成的视频、保真度检查以及性能历史列表。', @@ -180,11 +175,11 @@ export default function VideoDashboard() { () => new Map( hardwareKeys.flatMap((key) => { - const point = leadCell(cells, key, { tier: state.tier, basis: state.basis }); + const point = leadCell(cells, key, { tier: state.tier }); return point ? [[key, point] as const] : []; }), ), - [cells, hardwareKeys, state.tier, state.basis], + [cells, hardwareKeys, state.tier], ); const lead = measured.values().next().value ?? cells[0]; const layouts = [ @@ -199,6 +194,12 @@ export default function VideoDashboard() { const otherWorkloads = new Set(cells.map((p) => p.workload.split(' · ').slice(0, 5).join(' · '))) .size; const workloadSuffix = otherWorkloads > 1 ? s.workloads(otherWorkloads - 1) : ''; + // Jobs that reserved a whole node but generated on part of it: say so beside the per-GPU numbers. + const idle = [...measured.values()] + .filter( + (p) => p.participating !== null && p.allocated !== null && p.allocated > p.participating, + ) + .map((p) => s.idleItem(hardwareLabel(p.hardwareKey ?? ''), p.participating!, p.allocated!)); const legendItems = VIDEO_HARDWARE_ROSTER.map(({ key, unavailable }) => { const point = measured.get(key); @@ -226,7 +227,7 @@ export default function VideoDashboard() { }); const exportCsv = () => { - const rows = videoTableRows(points, state, hidden); + const rows = videoTableRows(points, hidden); const header = [ 'hardware', 'concurrency', @@ -238,7 +239,6 @@ export default function VideoDashboard() { 'tp_size', 'ulysses_degree', 'replicas', - 'queued', ...CSV_METRICS.map((id) => `${id} (${VIDEO_METRICS[id].unit})`), 'board_power_w', 'enforced_limit_w', @@ -257,7 +257,6 @@ export default function VideoDashboard() { p.server?.tp ?? null, p.server?.ulysses ?? null, p.replicas, - isQueueing(p) ? 1 : 0, ...CSV_METRICS.map((id) => metricValue(p, id, options)), p.avgPowerW, p.enforcedLimitW, @@ -273,7 +272,7 @@ export default function VideoDashboard() { const url = URL.createObjectURL(blob); const anchor = document.createElement('a'); anchor.href = url; - anchor.download = `videogenx-${state.y}-vs-${state.x}-${state.tier}-${state.basis}.csv`; + anchor.download = `videogenx-${state.y}-vs-${state.x}-${state.tier}.csv`; anchor.click(); URL.revokeObjectURL(url); }; @@ -358,6 +357,11 @@ export default function VideoDashboard() {

+ {idle.length > 0 && ( +

+ {s.idle(idle.join(locale === 'zh' ? '、' : ', '))} +

+ )}
{replay && (

@@ -403,35 +407,6 @@ export default function VideoDashboard() { isLegendExpanded={legendExpanded} onExpandedChange={setLegendExpanded} disableActiveSort - switches={[ - { - id: 'video-queue', - label: s.queue, - checked: state.queue, - onCheckedChange: (checked) => { - update({ queue: checked }); - track('video_queue_changed', { value: String(checked) }); - }, - }, - { - id: 'video-optimal', - label: s.optimal, - checked: state.optimal, - onCheckedChange: (checked) => { - update({ optimal: checked }); - track('video_optimal_changed', { value: String(checked) }); - }, - }, - { - id: 'video-frontier', - label: s.frontier, - checked: state.frontier, - onCheckedChange: (checked) => { - update({ frontier: checked }); - track('video_frontier_changed', { value: String(checked) }); - }, - }, - ]} />

@@ -442,7 +417,7 @@ export default function VideoDashboard() { {!loading && !error && ( <> - + )}
string; } @@ -209,12 +207,12 @@ export interface VideoEvidenceProps { * sentence is built from the computed numbers; the section disappears when * nothing is measured. */ -export default function VideoEvidence({ points, options, colorFor }: VideoEvidenceProps) { +export default function VideoEvidence({ points, colorFor }: VideoEvidenceProps) { const locale = useLocale(); const s = STRINGS[locale]; const power = powerUtilization(points); const range = powerRange(power); - const plateau = concurrencyPlateau(points, options.basis); + const plateau = concurrencyPlateau(points); const summary = plateauSummary(plateau); const scaling = scalingVsSpec(points); if (range === null && summary === null && scaling.length === 0) return null; diff --git a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx index 07180120d..7d5e2aa1f 100644 --- a/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx +++ b/packages/app/src/components/video-benchmark/VideoHardwareChart.tsx @@ -22,15 +22,9 @@ const STRINGS = { `One deployment measured per hardware so far (${layouts}), so each hardware is a single point. A per-hardware Pareto curve needs the GPUs-per-video sweep.`, multi: "Solid lines join each hardware's Pareto-optimal deployments (GPUs per video and model split); faded points are dominated deployments.", - global: 'Dashed line: cross-hardware Pareto frontier over all deployments.', - globalOne: - 'Cross-hardware Pareto frontier: one hardware dominates, so there is no line to draw.', - queued: - 'Small faded markers: C2/C4 client concurrency on the same batch-one deployment. Requests queue and throughput stays flat, so they are evidence, never joined into a curve.', controls: 'Shift+scroll to zoom; drag to pan; double-click to reset. Click a point to pin its details.', deployment: 'Deployment', - queuedTag: 'queued', n: 'valid samples', run: 'run', noData: 'No hardware has both selected metrics for this configuration.', @@ -40,13 +34,8 @@ const STRINGS = { `目前每种硬件只测得一种部署(${layouts}),因此每种硬件只有一个点;要得到各硬件自己的 Pareto 曲线,还需扫描每条视频占用的 GPU 数。`, multi: '实线连接同一硬件的 Pareto 最优部署(每条视频占用的 GPU 数及模型切分方式);淡色点为被支配的部署。', - global: '虚线为所有部署的跨硬件 Pareto 前沿。', - globalOne: '跨硬件 Pareto 前沿:有一种硬件全面占优,因此没有可画的连线。', - queued: - '小号淡色标记为同一 batch-one 部署上的 C2/C4 客户端并发:请求排队、吞吐量持平,仅作为证据展示,不连成曲线。', controls: 'Shift+滚轮缩放,拖动平移,双击重置。点击数据点可固定详情。', deployment: '部署', - queuedTag: '排队', n: '有效样本', run: '运行', noData: '当前配置下没有硬件同时具备所选的两个指标。', @@ -71,12 +60,7 @@ export default function VideoHardwareChart({ const locale = useLocale(); const s = STRINGS[locale]; const options = metricOptions(state); - const { plotted, frontiers, global, multiLayout } = plotVideoPoints( - points, - state, - colorFor, - hidden, - ); + const { plotted, frontiers, multiLayout } = plotVideoPoints(points, state, colorFor, hidden); const xLabel = metricLabel(state.x, locale, options); const yLabel = metricLabel(state.y, locale, options); const lines: Record = {}; @@ -86,15 +70,12 @@ export default function VideoHardwareChart({ lines[key] = frontier.map((p) => ({ x: p.x, y: p.y })); lineColors[key] = frontier[0].color; } - const layouts = [ - ...new Set(plotted.filter((p) => !p.queued).map((p) => layoutLabel(p, locale))), - ].join(locale === 'zh' ? ';' : '; '); + const layouts = [...new Set(plotted.map((p) => layoutLabel(p, locale)))].join( + locale === 'zh' ? ';' : '; ', + ); const colon = locale === 'zh' ? ':' : ': '; - const queuedTag = locale === 'zh' ? `(${s.queuedTag})` : ` (${s.queuedTag})`; - const pointLabel = (p: PlottedVideoPoint) => { - if (p.queued) return `C${p.concurrency}`; - return multiLayout ? `${p.label} · ${layoutLabel(p, locale)}` : p.label; - }; + const pointLabel = (p: PlottedVideoPoint) => + multiLayout ? `${p.label} · ${layoutLabel(p, locale)}` : p.label; const drawLabels = ( group: Selection, x: ContinuousScale, @@ -113,21 +94,15 @@ export default function VideoHardwareChart({ .attr('pointer-events', 'none') .text(pointLabel); }; + // Both axes start at zero so per-dollar and per-video readings compare by length, not offset. const xDomain = zeroAnchoredDomain( plotted.map((p) => p.x), 1.12, ); - // Profit per GPU-hour goes negative when the API price misses the tier cost: those - // points stay on the canvas, and when every hardware loses money the top is break-even. const yDomain = zeroAnchoredDomain( plotted.map((p) => p.y), 1.15, ); - const caption = [ - multiLayout ? s.multi : s.single(layouts || '—'), - state.frontier ? (global.length > 1 ? s.global : s.globalOne) : null, - state.queue && !state.optimal ? s.queued : null, - ].filter((text): text is string => text !== null); return ( 1 - ? { global: global.map((p) => ({ x: p.x, y: p.y })) } - : {}, - config: { - getColor: () => 'var(--foreground)', - strokeWidth: 1.5, - strokeDasharray: '6 4', - curve: curveLinear, - }, - }, { type: 'line', key: 'hardware-frontiers', @@ -174,8 +135,8 @@ export default function VideoHardwareChart({ getCy: () => 0, getX: (p) => p.x, getY: (p) => p.y, - getColor: (p) => (p.queued || !p.optimal ? faded(p.color) : p.color), - getRadius: (p) => (p.queued ? 3.5 : p.optimal ? 7 : 5), + getColor: (p) => (p.optimal ? p.color : faded(p.color)), + getRadius: (p) => (p.optimal ? 7 : 5), stroke: 'var(--foreground)', strokeWidth: 1, keyFn: (p) => p.id, @@ -198,7 +159,7 @@ export default function VideoHardwareChart({ tooltip={{ rulerType: 'none', content: (p) => - `
${escapeHtml(p.label)} · C${p.concurrency ?? '?'}${p.queued ? queuedTag : ''}
${s.deployment}${colon}${escapeHtml(layoutLabel(p, locale))}
${escapeHtml(xLabel)}${colon}${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}${colon}${formatMetric(p.y, state.y)}
${s.n}${colon}${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, + `
${escapeHtml(p.label)}
${s.deployment}${colon}${escapeHtml(layoutLabel(p, locale))}
${escapeHtml(xLabel)}${colon}${formatMetric(p.x, state.x)}
${escapeHtml(yLabel)}${colon}${formatMetric(p.y, state.y)}
${s.n}${colon}${p.samples} · ${s.run} #${escapeHtml(p.runId)}
`, onPointClick: onSelect, }} zoom={{ @@ -217,7 +178,7 @@ export default function VideoHardwareChart({ } caption={

- {caption.join(' ')} + {multiLayout ? s.multi : s.single(layouts || '—')}

} /> diff --git a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx index 01bfafed8..9520d49fc 100644 --- a/packages/app/src/components/video-benchmark/VideoKpiCards.tsx +++ b/packages/app/src/components/video-benchmark/VideoKpiCards.tsx @@ -13,21 +13,16 @@ const CARD_METRICS = [ 'p50Latency', 'videosPerGpuHour', 'dollarsPerVideo', - 'apiPricePerVideo', - 'profitPerGpuHour', 'kjPerVideo', - 'powerPctCap', ] as const satisfies readonly MetricId[]; const STRINGS = { en: { p50Latency: 'P50 time to video (s)', videosPerGpuHour: 'Videos / GPU-hr', dollarsPerVideo: 'TCO / video', - apiPricePerVideo: 'API $/video', - profitPerGpuHour: 'Profit / GPU-hr', kjPerVideo: 'kJ / video', - powerPctCap: 'Board power / limit', p90: 'P90', + api: 'API list', gpus: 'GPUs', of: 'of', unavailable: 'Not measured', @@ -37,11 +32,9 @@ const STRINGS = { p50Latency: 'P50 出片时间(s)', videosPerGpuHour: '视频数 / GPU 小时', dollarsPerVideo: 'TCO / 条视频', - apiPricePerVideo: 'API 标价 / 条视频', - profitPerGpuHour: '利润 / GPU 小时', kjPerVideo: 'kJ / 条视频', - powerPctCap: '板卡功率 / 上限', p90: 'P90', + api: 'API 标价', gpus: '张 GPU', of: '/', unavailable: '未测得', @@ -49,7 +42,11 @@ const STRINGS = { }, }; -/** One card per campaign hardware, showing its most efficient deployment; hardware without a valid run says so instead of vanishing. */ +/** + * One card per campaign hardware on its most efficient deployment: time to + * video, output per GPU-hour, TCO cost per video beside the API list price, + * and energy per video. Hardware without a valid run says so instead of vanishing. + */ export default function VideoKpiCards({ points, state, @@ -66,6 +63,15 @@ export default function VideoKpiCards({ const s = STRINGS[locale]; const options = metricOptions(state); const cells = latestVideoCells(points); + const secondary = (point: VideoPoint, id: (typeof CARD_METRICS)[number]) => { + if (id === 'p50Latency') + return `${s.p90} ${formatMetric(metricValue(point, 'p90Latency', options), 'p90Latency')}`; + if (id === 'dollarsPerVideo') { + const api = metricValue(point, 'apiPricePerVideo', options); + return api === null ? null : `${s.api} ${formatMetric(api, 'apiPricePerVideo')}`; + } + return null; + }; return (
{VIDEO_HARDWARE_ROSTER.map(({ key, unavailable }) => { @@ -96,21 +102,22 @@ export default function VideoKpiCards({
) : point ? (
- {CARD_METRICS.map((id) => ( -
-
{s[id]}
-
- {formatMetric(metricValue(point, id, options), id)} - {id === 'powerPctCap' && metricValue(point, id, options) !== null && '%'} - {id === 'p50Latency' && ( - - {s.p90}{' '} - {formatMetric(metricValue(point, 'p90Latency', options), 'p90Latency')} - - )} -
-
- ))} + {CARD_METRICS.map((id) => { + const note = secondary(point, id); + return ( +
+
{s[id]}
+
+ {formatMetric(metricValue(point, id, options), id)} + {note && ( + + {note} + + )} +
+
+ ); + })}
) : (

diff --git a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx index d85893240..c9ccc77cf 100644 --- a/packages/app/src/components/video-benchmark/VideoPointsTable.tsx +++ b/packages/app/src/components/video-benchmark/VideoPointsTable.tsx @@ -12,8 +12,6 @@ const STRINGS = { en: { hardware: 'Hardware', deployment: 'Deployment', - c: 'C', - queued: 'queued', counts: 'Valid / scheduled', power: 'Board power (W)', run: 'CI run', @@ -21,8 +19,6 @@ const STRINGS = { zh: { hardware: '硬件', deployment: '部署', - c: 'C', - queued: '排队', counts: '有效 / 计划', power: '板卡功率(W)', run: 'CI 运行', @@ -35,21 +31,14 @@ const METRIC_COLUMNS: readonly MetricId[] = [ 'videosPerDollar', 'dollarsPerVideo', 'apiPricePerVideo', - 'revenuePerGpuHour', - 'profitPerGpuHour', - 'apiPriceMultiple', 'kjPerVideo', 'powerPctCap', ]; -/** Rows for the plotted cells: every deployment, queued cells on request, plus provenance. */ -export function videoTableRows( - points: VideoPoint[], - state: VideoDashboardState, - hidden: ReadonlySet, -): VideoPoint[] { +/** Rows for the plotted cells: every measured deployment, plus provenance. Queued cells stay in the evidence panel. */ +export function videoTableRows(points: VideoPoint[], hidden: ReadonlySet): VideoPoint[] { return latestVideoCells(points).filter( - (p) => p.hardwareKey && !hidden.has(p.hardwareKey) && (state.queue || !isQueueing(p)), + (p) => p.hardwareKey && !hidden.has(p.hardwareKey) && !isQueueing(p), ); } @@ -65,7 +54,7 @@ export default function VideoPointsTable({ const locale = useLocale(); const s = STRINGS[locale]; const options = metricOptions(state); - const rows = videoTableRows(points, state, hidden); + const rows = videoTableRows(points, hidden); const columns: DataTableColumn[] = [ { header: s.hardware, @@ -80,13 +69,6 @@ export default function VideoPointsTable({ sortValue: (p) => p.participating ?? 0, importance: 'key', }, - { - header: s.c, - align: 'right', - cell: (p) => (isQueueing(p) ? `${p.concurrency} · ${s.queued}` : (p.concurrency ?? '—')), - sortValue: (p) => p.concurrency ?? 0, - importance: 'key', - }, { header: s.counts, align: 'right', diff --git a/packages/app/src/components/video-benchmark/chart-domain.test.ts b/packages/app/src/components/video-benchmark/chart-domain.test.ts index 578de9fb7..2ccc93ad6 100644 --- a/packages/app/src/components/video-benchmark/chart-domain.test.ts +++ b/packages/app/src/components/video-benchmark/chart-domain.test.ts @@ -1,11 +1,12 @@ import { describe, expect, it } from 'vitest'; import { zeroAnchoredDomain } from './chart-domain'; -// Profit per GPU-hour for the retained H100/H200/B200 C1 cells at $0.034/video-s -// (cypress/fixtures/api/video-history.json, asserted per point in metrics.test.ts): -// every hardware loses money at the 3-year rental tier ($2.00/$2.90/$3.70 GPU-hr) -// on the participating basis, while on the allocated basis at the hyperscaler -// tier only H200 stays positive. +// Every Y metric the chart plots today is positive-only, but the helper stays +// sign-agnostic (chart-domain.ts), so two signed samples keep the shapes it must +// still handle: an all-negative set and a mixed-sign set, sized like $/GPU-hr +// margins of the retained H100/H200/B200 C1 cells. VIDEOS_PER_DOLLAR is their +// hyperscaler-tier videos per $1 TCO (cypress/fixtures/api/video-history.json, +// asserted per point in metrics.test.ts). const RENTAL_LOSSES = [-0.5382, -1.2753, -0.5649]; const ALLOCATED_HYPERSCALER = [-0.4391, 0.4047, -0.16]; // Videos per $1 TCO at the hyperscaler tier: H100, H200, B200. diff --git a/packages/app/src/components/video-benchmark/compare.test.ts b/packages/app/src/components/video-benchmark/compare.test.ts index 99a5ac52a..8c9ac4a0d 100644 --- a/packages/app/src/components/video-benchmark/compare.test.ts +++ b/packages/app/src/components/video-benchmark/compare.test.ts @@ -19,7 +19,7 @@ import type { StoredArtifact, StoredSource } from './stored'; // Retained H100/H200/B200 C1/C2/C4 observations of the 2026-09-09 campaign. const points = videoPoints([history as unknown as VideoHistoryPage]); const c1 = (key: string) => comparablePoints(points).find((p) => p.hardwareKey === key)!; -const opts = { tier: 'h', basis: 'participating' } as const; +const opts = { tier: 'h' } as const; const row = (rows: ReturnType, id: string) => rows.find((item) => item.id === id)!; @@ -84,43 +84,28 @@ describe('comparablePoints', () => { }); describe('compareMetrics', () => { - it('reads B200 against H100 at C1: faster, cheaper, less energy, closer to its cap', () => { + it('reads B200 against H100 at C1: faster, cheaper and less energy per video', () => { const rows = compareMetrics(c1('h100'), c1('b200'), opts); - expect(rows.map((r) => r.id)).toEqual([ - 'p50Latency', - 'p90Latency', - 'videosPerGpuHour', - 'videosPerDollar', - 'dollarsPerVideo', - 'kjPerVideo', - 'powerPctCap', - ]); + expect(rows.map((r) => r.id)).toEqual(['p50Latency', 'dollarsPerVideo', 'kjPerVideo']); const p50 = row(rows, 'p50Latency'); expect(p50.baseline).toBeCloseTo(167.33, 2); expect(p50.candidate).toBeCloseTo(77.94, 2); expect(p50.ratio).toBeCloseTo(0.4658, 3); expect(p50.deltaPercent).toBeCloseTo(-53.42, 1); expect(p50.candidateBetter).toBe(true); - expect(row(rows, 'p90Latency').ratio).toBeCloseTo(0.4659, 3); - const rate = row(rows, 'videosPerGpuHour'); - expect(rate.baseline).toBeCloseTo(5.374, 3); - expect(rate.candidate).toBeCloseTo(11.526, 3); - expect(rate.ratio).toBeCloseTo(2.1447, 3); - expect(rate.candidateBetter).toBe(true); + // Hyperscaler tier: $1.17 / 5.374 and $1.73 / 11.526 videos per participating GPU-hour. const cost = row(rows, 'dollarsPerVideo'); expect(cost.baseline).toBeCloseTo(0.2177, 4); expect(cost.candidate).toBeCloseTo(0.1501, 4); + expect(cost.ratio).toBeCloseTo(0.6894, 3); expect(cost.deltaPercent).toBeCloseTo(-31.06, 1); expect(cost.candidateBetter).toBe(true); - expect(row(rows, 'videosPerDollar').deltaPercent).toBeCloseTo(45.05, 1); const energy = row(rows, 'kjPerVideo'); expect(energy.baseline).toBeCloseTo(431.1, 1); expect(energy.candidate).toBeCloseTo(301.1, 1); + expect(energy.ratio).toBeCloseTo(0.6983, 3); + expect(energy.deltaPercent).toBeCloseTo(-30.17, 1); expect(energy.candidateBetter).toBe(true); - const power = row(rows, 'powerPctCap'); - expect(power.baseline).toBeCloseTo(91.95, 2); - expect(power.candidate).toBeCloseTo(96.4, 2); - expect(power.candidateBetter).toBe(true); }); it('flips the verdict when the slower hardware is the candidate', () => { const rows = compareMetrics(c1('b200'), c1('h200'), opts); @@ -129,8 +114,8 @@ describe('compareMetrics', () => { expect(row(rows, 'dollarsPerVideo').candidateBetter).toBe(false); }); it('propagates missing inputs as null and treats equal values as neither better', () => { - const noPower = { ...c1('b200'), energyKj: null, enforcedLimitW: null, samples: 5 }; - const rows = compareMetrics(c1('h100'), noPower, opts); + const unmeasured = { ...c1('b200'), energyKj: null, wallSeconds: null, samples: 5 }; + const rows = compareMetrics(c1('h100'), unmeasured, opts); expect(row(rows, 'kjPerVideo')).toEqual({ id: 'kjPerVideo', baseline: c1('h100').energyKj, @@ -139,21 +124,41 @@ describe('compareMetrics', () => { deltaPercent: null, candidateBetter: null, }); - expect(row(rows, 'powerPctCap').candidate).toBeNull(); - expect(row(rows, 'p90Latency').candidate).toBeNull(); - expect(row(rows, 'p50Latency').ratio).not.toBeNull(); + expect(row(rows, 'dollarsPerVideo')).toMatchObject({ + baseline: expect.closeTo(0.2177, 4), + candidate: null, + ratio: null, + candidateBetter: null, + }); + // P50 has no sample floor, unlike P90 in the run views. + expect(row(rows, 'p50Latency').ratio).toBeCloseTo(0.4658, 3); + // A zero baseline has no ratio either. + expect( + row(compareMetrics({ ...c1('h100'), p50: 0 }, c1('b200'), opts), 'p50Latency'), + ).toMatchObject({ + baseline: 0, + candidate: expect.closeTo(77.94, 2), + ratio: null, + deltaPercent: null, + candidateBetter: null, + }); for (const same of compareMetrics(c1('h100'), c1('h100'), opts)) { expect(same.ratio).toBe(1); expect(same.deltaPercent).toBe(0); expect(same.candidateBetter).toBeNull(); } }); - it('moves only the cost rows with the tier and GPU basis', () => { - const rows = compareMetrics(c1('h100'), c1('b200'), { tier: 'r', basis: 'allocated' }); - // H100 8 allocated: 2.00 / 2.687 videos per GPU-hr; B200 8 allocated: 3.70 / 5.763. - expect(row(rows, 'dollarsPerVideo').baseline).toBeCloseTo(0.7443, 3); - expect(row(rows, 'dollarsPerVideo').candidate).toBeCloseTo(0.642, 3); - expect(row(rows, 'videosPerGpuHour').baseline).toBeCloseTo(2.687, 3); + it('moves only the cost row with the tier', () => { + const rows = compareMetrics(c1('h100'), c1('b200'), { tier: 'r' }); + // Rental $2.00 / 5.374 and $3.70 / 11.526 videos per GPU-hour, still per participating GPU. + const cost = row(rows, 'dollarsPerVideo'); + expect(cost.baseline).toBeCloseTo(0.3722, 3); + expect(cost.candidate).toBeCloseTo(0.321, 3); + expect(cost.ratio).toBeCloseTo(0.8626, 3); + expect(cost.candidateBetter).toBe(true); + const atHyperscaler = compareMetrics(c1('h100'), c1('b200'), opts); + expect(row(rows, 'p50Latency')).toEqual(row(atHyperscaler, 'p50Latency')); + expect(row(rows, 'kjPerVideo')).toEqual(row(atHyperscaler, 'kjPerVideo')); expect(row(rows, 'p50Latency').ratio).toBeCloseTo(0.4658, 3); expect(row(rows, 'kjPerVideo').ratio).toBeCloseTo(0.6984, 3); }); diff --git a/packages/app/src/components/video-benchmark/compare.ts b/packages/app/src/components/video-benchmark/compare.ts index 753e5491d..4d7cbe1b1 100644 --- a/packages/app/src/components/video-benchmark/compare.ts +++ b/packages/app/src/components/video-benchmark/compare.ts @@ -14,12 +14,8 @@ import { storedBundle, type StoredArtifact, type StoredSource } from './stored'; /** Metrics the compare table shows, in row order. */ export const COMPARE_METRICS = [ 'p50Latency', - 'p90Latency', - 'videosPerGpuHour', - 'videosPerDollar', 'dollarsPerVideo', 'kjPerVideo', - 'powerPctCap', ] as const satisfies readonly MetricId[]; export type CompareMetricId = (typeof COMPARE_METRICS)[number]; @@ -37,13 +33,13 @@ export interface CompareRow { /** * One cell a reader can pick per registry hardware: its most efficient - * non-queued deployment (chip basis) with a P50. Queued cells never qualify. + * non-queued deployment (per participating GPU) with a P50. Queued cells never qualify. */ export function comparablePoints(points: VideoPoint[]): VideoPoint[] { const cells = latestVideoCells(points); const keys = [...new Set(cells.flatMap((p) => (p.hardwareKey === null ? [] : [p.hardwareKey])))]; return keys.flatMap((key) => { - const lead = leadCell(cells, key, { tier: 'h', basis: 'participating' }); + const lead = leadCell(cells, key, { tier: 'h' }); return lead && lead.p50 !== null ? [lead] : []; }); } diff --git a/packages/app/src/components/video-benchmark/deployment.ts b/packages/app/src/components/video-benchmark/deployment.ts index 02ffbca0e..6183d91d0 100644 --- a/packages/app/src/components/video-benchmark/deployment.ts +++ b/packages/app/src/components/video-benchmark/deployment.ts @@ -57,7 +57,7 @@ export function sharedLayoutCells(cells: T[]): T[] { return keys.flatMap((hardwareKey) => { const cell = deployments.find((p) => p.hardwareKey === hardwareKey && deploymentKey(p) === shared) ?? - leadCell(deployments, hardwareKey, { tier: 'h', basis: 'participating' }); + leadCell(deployments, hardwareKey, { tier: 'h' }); return cell ? [cell as T] : []; }); } diff --git a/packages/app/src/components/video-benchmark/evidence.test.ts b/packages/app/src/components/video-benchmark/evidence.test.ts index d28e425b3..e019763dd 100644 --- a/packages/app/src/components/video-benchmark/evidence.test.ts +++ b/packages/app/src/components/video-benchmark/evidence.test.ts @@ -103,14 +103,6 @@ describe('concurrencyPlateau', () => { expect(row('b200', 4)?.throughputRatioVsC1).toBeCloseTo(1.0144, 3); expect(row('b200', 4)?.latencyRatioVsC1).toBeCloseTo(3.9423, 3); }); - it('switches the throughput denominator with the basis without moving the ratios', () => { - const rows = concurrencyPlateau(points, 'allocated'); - const h100 = rows.filter((r) => r.hardwareKey === 'h100'); - expect(h100[0].videosPerGpuHour).toBeCloseTo(2.687, 3); - expect(h100[2].throughputRatioVsC1).toBeCloseTo(0.9989, 3); - // H200 allocated 4 of 4, so its denominator is unchanged. - expect(rows.find((r) => r.hardwareKey === 'h200')?.videosPerGpuHour).toBeCloseTo(5.973, 3); - }); it('leaves ratios null without a C1 baseline and never reports 0', () => { const rows = concurrencyPlateau( points.filter((p) => !(p.hardwareKey === 'h200' && p.concurrency === 1)), diff --git a/packages/app/src/components/video-benchmark/evidence.ts b/packages/app/src/components/video-benchmark/evidence.ts index 57bcaa545..577efc262 100644 --- a/packages/app/src/components/video-benchmark/evidence.ts +++ b/packages/app/src/components/video-benchmark/evidence.ts @@ -1,7 +1,7 @@ import { GPU_SPECS } from '@/lib/gpu-specs'; import { deploymentKey, sharedLayoutCells } from './deployment'; import { hardwareSort } from './hardware'; -import { metricValue, type GpuBasis, type MetricOptions, type VideoPoint } from './metrics'; +import { metricValue, type MetricOptions, type VideoPoint } from './metrics'; import { latestVideoCells } from './points'; /** @@ -33,7 +33,7 @@ function measuredCells(points: VideoPoint[]): MeasuredPoint[] { } /** The cost tier only enters priced metrics; nothing read here is priced. */ -const metricOptions = (basis: GpuBasis): MetricOptions => ({ tier: 'h', basis }); +const options: MetricOptions = { tier: 'h' }; export interface PowerUtilizationRow { hardwareKey: string; @@ -50,7 +50,7 @@ export function powerUtilization(points: VideoPoint[]): PowerUtilizationRow[] { hardwareKey: p.hardwareKey, avgPowerW: positive(p.avgPowerW) ? p.avgPowerW : null, enforcedLimitW: positive(p.enforcedLimitW) ? p.enforcedLimitW : null, - percentOfCap: metricValue(p, 'powerPctCap', metricOptions('participating')), + percentOfCap: metricValue(p, 'powerPctCap', options), })); } @@ -92,14 +92,9 @@ export interface ConcurrencyPlateauRow { /** * Every measured cell against its hardware's C1 cell. On a batch-one server a * flat throughput ratio with latency scaling by C says client concurrency only - * queues requests. `basis` picks the GPU denominator the rest of the dashboard - * shows; ratios within one hardware are unaffected unless its allocation changed. + * queues requests. Throughput is per participating GPU, as everywhere else. */ -export function concurrencyPlateau( - points: VideoPoint[], - basis: GpuBasis = 'participating', -): ConcurrencyPlateauRow[] { - const options = metricOptions(basis); +export function concurrencyPlateau(points: VideoPoint[]): ConcurrencyPlateauRow[] { const baseline = new Map( sharedLayoutCells(measuredCells(points)).map((p) => [p.hardwareKey, p] as const), ); diff --git a/packages/app/src/components/video-benchmark/metrics.test.ts b/packages/app/src/components/video-benchmark/metrics.test.ts index d7e7d5f0d..03cb96e0a 100644 --- a/packages/app/src/components/video-benchmark/metrics.test.ts +++ b/packages/app/src/components/video-benchmark/metrics.test.ts @@ -1,7 +1,18 @@ import { describe, expect, it } from 'vitest'; -import { formatMetric, metricLabel, metricValue, type VideoPoint } from './metrics'; +import { + formatMetric, + metricLabel, + metricValue, + VIDEO_METRICS, + X_METRICS, + Y_METRICS, + type MetricId, + type MetricOptions, + type VideoPoint, +} from './metrics'; -// H200 and H100 C1 cells from the retained 2026-09-09 serving-smoke artifacts. +// H200, H100 and B200 C1 cells from the retained 2026-09-09 serving-smoke artifacts +// (cypress/fixtures/api/video-history.json). const h200: VideoPoint = { id: '34342452354.10107476604:c1', runId: '34342452354', @@ -33,6 +44,7 @@ const h200: VideoPoint = { status: 'complete', observedAt: '2026-09-09T10:00:00Z', }; +// H100 and B200 reserved eight boards and generated on four. const h100: VideoPoint = { ...h200, id: 'x', @@ -45,84 +57,101 @@ const h100: VideoPoint = { energyKj: 431.14963489170344, avgPowerW: 2574.550733165641, }; -const opts = { tier: 'h', basis: 'participating' } as const; +const b200: VideoPoint = { + ...h200, + id: 'y', + hardwareKey: 'b200', + hardwareName: 'NVIDIA B200', + p50: 77.94495720259147, + p90: 78.31597462110221, + wallSeconds: 1561.6792716470081, + allocated: 8, + energyKj: 301.08754501100987, + avgPowerW: 3856.016973742528, + enforcedLimitW: 4000, +}; +const opts: MetricOptions = { tier: 'h' }; // H3 768p list price captured 2026-09-19 (api-reference.ts), USD per video-second. -const priced = { ...opts, apiPricePerVideoSecond: 0.034 }; +const priced: MetricOptions = { ...opts, apiPricePerVideoSecond: 0.034 }; describe('video metrics', () => { + it('registers the two axis groups plus the two card-only metrics', () => { + expect([...X_METRICS]).toEqual(['p90Latency', 'p50Latency']); + expect([...Y_METRICS]).toEqual([ + 'videosPerDollar', + 'dollarsPerVideo', + 'videosPerGpuHour', + 'kjPerVideo', + ]); + expect(Object.keys(VIDEO_METRICS).toSorted()).toEqual( + [...X_METRICS, ...Y_METRICS, 'powerPctCap', 'apiPricePerVideo'].toSorted(), + ); + }); it('derives throughput per GPU-hour from valid clips, wall time and participating GPUs', () => { + // 20 clips × 3600 s / (wall s × 4 boards). expect(metricValue(h200, 'videosPerGpuHour', opts)).toBeCloseTo(5.9732, 3); - expect(metricValue(h200, 'videoSecondsPerGpuHour', opts)).toBeCloseTo(47.786, 2); - }); - it('switches the denominator to allocated GPUs on request', () => { expect(metricValue(h100, 'videosPerGpuHour', opts)).toBeCloseTo(5.3741, 3); - expect(metricValue(h100, 'videosPerGpuHour', { ...opts, basis: 'allocated' })).toBeCloseTo( + expect(metricValue(b200, 'videosPerGpuHour', opts)).toBeCloseTo(11.526, 3); + }); + it('divides by the participating boards, never the allocation', () => { + // The idle half of an 8-board reservation is stated beside the numbers, not billed. + expect(metricValue({ ...h100, allocated: 4 }, 'videosPerGpuHour', opts)).toBe( + metricValue(h100, 'videosPerGpuHour', opts), + ); + expect(metricValue({ ...h100, allocated: null }, 'dollarsPerVideo', opts)).toBe( + metricValue(h100, 'dollarsPerVideo', opts), + ); + expect(metricValue({ ...h100, participating: 8 }, 'videosPerGpuHour', opts)).toBeCloseTo( 2.687, 3, ); }); it('prices videos with HW_REGISTRY tiers', () => { - expect(metricValue(h200, 'videosPerDollar', opts)).toBeCloseTo(4.896, 2); + // Hyperscaler $1.22/$1.17/$1.73 and rental $2.90/$2.00/$3.70 per GPU-hour for H200/H100/B200. + expect(metricValue(h200, 'videosPerDollar', opts)).toBeCloseTo(4.8961, 3); expect(metricValue(h200, 'dollarsPerVideo', opts)).toBeCloseTo(0.2042, 3); - expect(metricValue(h200, 'dollarsPerVideo', { ...opts, tier: 'r' })).toBeCloseTo(0.4855, 3); - expect(metricValue(h200, 'dollarsPerVideoSecond', opts)).toBeCloseTo(0.02553, 4); - expect(metricValue(h100, 'dollarsPerVideo', { ...opts, basis: 'allocated' })).toBeCloseTo( - 0.4354, - 3, - ); + expect(metricValue(h200, 'dollarsPerVideo', { tier: 'r' })).toBeCloseTo(0.4855, 3); + expect(metricValue(h100, 'videosPerDollar', opts)).toBeCloseTo(4.5933, 3); + expect(metricValue(h100, 'dollarsPerVideo', opts)).toBeCloseTo(0.2177, 3); + expect(metricValue(h100, 'dollarsPerVideo', { tier: 'r' })).toBeCloseTo(0.3722, 3); + expect(metricValue(b200, 'videosPerDollar', opts)).toBeCloseTo(6.6625, 3); + expect(metricValue(b200, 'dollarsPerVideo', opts)).toBeCloseTo(0.1501, 3); + expect(metricValue(b200, 'dollarsPerVideo', { tier: 'r' })).toBeCloseTo(0.321, 3); + // Videos per $1 TCO is exactly the reciprocal of the TCO cost per video. + expect( + metricValue(h200, 'videosPerDollar', opts)! * metricValue(h200, 'dollarsPerVideo', opts)!, + ).toBeCloseTo(1, 12); }); - it('exposes latency, generation speed, energy and power ratios', () => { + it('exposes latency, energy and board power against the enforced limit', () => { expect(metricValue(h200, 'p90Latency', opts)).toBeCloseTo(151.105, 3); expect(metricValue(h200, 'p50Latency', opts)).toBeCloseTo(150.614, 3); - expect(metricValue(h200, 'genSpeed', opts)).toBeCloseTo(192 / 150.61379817902343, 6); expect(metricValue(h200, 'kjPerVideo', opts)).toBeCloseTo(410.954, 2); - expect(metricValue(h200, 'videosPerKwh', opts)).toBeCloseTo(8.76, 2); expect(metricValue(h200, 'powerPctCap', opts)).toBeCloseTo(97.41, 1); + expect(metricValue(b200, 'powerPctCap', opts)).toBeCloseTo(96.4, 1); }); - it('prices the clip at the API reference and derives revenue, profit and headroom', () => { - // 0.034 $/s × 8 s clip; 5.9732 videos/GPU-hr × $0.272; − $1.22 H200 hyperscaler GPU-hr. + it('prices the clip at the API list price, independent of hardware and tier', () => { + // 0.034 $/video-s × 8 s clip. expect(metricValue(h200, 'apiPricePerVideo', priced)).toBeCloseTo(0.272, 6); - expect(metricValue(h200, 'revenuePerGpuHour', priced)).toBeCloseTo(1.6247, 3); - expect(metricValue(h200, 'profitPerGpuHour', priced)).toBeCloseTo(0.4047, 3); - // $0.272 API ÷ $0.2042 TCO per video. - expect(metricValue(h200, 'apiPriceMultiple', priced)).toBeCloseTo(1.332, 3); - // Rental tier: same revenue, $2.90 GPU-hr → loss; headroom drops below 1. - expect(metricValue(h200, 'profitPerGpuHour', { ...priced, tier: 'r' })).toBeCloseTo(-1.2753, 3); - expect(metricValue(h200, 'apiPriceMultiple', { ...priced, tier: 'r' })).toBeCloseTo(0.5603, 3); + expect(metricValue(h100, 'apiPricePerVideo', { ...priced, tier: 'r' })).toBeCloseTo(0.272, 6); + expect( + metricValue({ ...b200, hardwareKey: null, wallSeconds: null }, 'apiPricePerVideo', priced), + ).toBeCloseTo(0.272, 6); + expect( + metricValue(h200, 'apiPricePerVideo', { ...priced, apiPricePerVideoSecond: 0.047 }), + ).toBeCloseTo(0.376, 6); }); - it('halves revenue on the allocated basis when only half the reserved GPUs generate', () => { - // H100: 4 of 8 allocated boards participate → 2.687 videos/GPU-hr × $0.272 − $1.17. - expect(metricValue(h100, 'revenuePerGpuHour', priced)).toBeCloseTo(1.4618, 3); - expect(metricValue(h100, 'profitPerGpuHour', priced)).toBeCloseTo(0.2918, 3); - const allocated = { ...priced, basis: 'allocated' } as const; - expect(metricValue(h100, 'revenuePerGpuHour', allocated)).toBeCloseTo(0.7309, 3); - expect(metricValue(h100, 'profitPerGpuHour', allocated)).toBeCloseTo(-0.4391, 3); - // The clip price itself does not depend on the GPU basis. - expect(metricValue(h100, 'apiPricePerVideo', allocated)).toBeCloseTo(0.272, 6); - expect(metricValue(h100, 'apiPriceMultiple', allocated)).toBeCloseTo(0.6247, 3); - }); - it('returns null for every API-priced metric without a positive price or clip length', () => { - const ids = [ - 'apiPricePerVideo', - 'revenuePerGpuHour', - 'profitPerGpuHour', - 'apiPriceMultiple', - ] as const; - for (const id of ids) { - expect(metricValue(h200, id, opts)).toBeNull(); - expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: null })).toBeNull(); - expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: 0 })).toBeNull(); - expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: -0.01 })).toBeNull(); - expect(metricValue(h200, id, { ...opts, apiPricePerVideoSecond: Number.NaN })).toBeNull(); - expect(metricValue({ ...h200, durationSeconds: null }, id, priced)).toBeNull(); + it('returns null for the API list price without a positive price or clip length', () => { + expect(metricValue(h200, 'apiPricePerVideo', opts)).toBeNull(); + for (const price of [null, 0, -0.01, Number.NaN, Number.POSITIVE_INFINITY]) { + expect( + metricValue(h200, 'apiPricePerVideo', { ...opts, apiPricePerVideoSecond: price }), + ).toBeNull(); } - // A price alone is not enough: revenue, profit and headroom still need throughput and a tier cost. - expect(metricValue({ ...h200, wallSeconds: null }, 'revenuePerGpuHour', priced)).toBeNull(); - expect(metricValue({ ...h200, hardwareKey: null }, 'profitPerGpuHour', priced)).toBeNull(); - expect(metricValue({ ...h200, hardwareKey: null }, 'apiPriceMultiple', priced)).toBeNull(); - expect(metricValue({ ...h200, wallSeconds: null }, 'apiPricePerVideo', priced)).toBeCloseTo( - 0.272, - 6, + expect(metricValue({ ...h200, durationSeconds: null }, 'apiPricePerVideo', priced)).toBeNull(); + expect(metricValue({ ...h200, durationSeconds: 0 }, 'apiPricePerVideo', priced)).toBeNull(); + // The price never leaks into the TCO metrics. + expect(metricValue(h200, 'dollarsPerVideo', priced)).toBe( + metricValue(h200, 'dollarsPerVideo', opts), ); }); it('returns null instead of zero for missing inputs', () => { @@ -134,42 +163,53 @@ describe('video metrics', () => { wallSeconds: null, enforcedLimitW: null, }; - expect(metricValue(broken, 'videosPerDollar', opts)).toBeNull(); - expect(metricValue(broken, 'videosPerGpuHour', opts)).toBeNull(); - expect(metricValue(broken, 'kjPerVideo', opts)).toBeNull(); - expect(metricValue(broken, 'videosPerKwh', opts)).toBeNull(); + for (const id of Y_METRICS) expect(metricValue(broken, id, opts)).toBeNull(); expect(metricValue(broken, 'powerPctCap', opts)).toBeNull(); + // Same display floor as the run views: P90 needs ten valid samples, P50 does not. expect(metricValue({ ...h200, samples: 9 }, 'p90Latency', opts)).toBeNull(); + expect(metricValue({ ...h200, samples: 9 }, 'p50Latency', opts)).toBeCloseTo(150.614, 3); expect(metricValue({ ...h200, participating: null }, 'videosPerGpuHour', opts)).toBeNull(); + expect(metricValue({ ...h200, participating: 0 }, 'videosPerGpuHour', opts)).toBeNull(); + expect(metricValue({ ...h200, participating: 2.5 }, 'videosPerGpuHour', opts)).toBeNull(); + expect(metricValue({ ...h200, energyKj: 0 }, 'kjPerVideo', opts)).toBeNull(); + expect(metricValue({ ...h200, avgPowerW: 0 }, 'powerPctCap', opts)).toBeNull(); + // Hardware outside HW_REGISTRY has a rate but no price. + expect( + metricValue({ ...h200, hardwareKey: 'unknown-gpu' }, 'videosPerGpuHour', opts), + ).toBeCloseTo(5.9732, 3); + expect( + metricValue({ ...h200, hardwareKey: 'unknown-gpu' }, 'dollarsPerVideo', opts), + ).toBeNull(); + expect( + metricValue({ ...h200, hardwareKey: 'unknown-gpu' }, 'videosPerDollar', opts), + ).toBeNull(); }); it('labels tiered metrics with the cost tier and formats by metric', () => { expect(metricLabel('videosPerDollar', 'en', opts)).toBe( 'Videos per $1 TCO (Owning at Large Hyperscaler Volume)', ); - expect(metricLabel('videosPerDollar', 'zh', { ...opts, tier: 'r' })).toBe( + expect(metricLabel('videosPerDollar', 'zh', { tier: 'r' })).toBe( '每 1 美元 TCO 生成视频数(租赁 - 3 年承诺)', ); + expect(metricLabel('dollarsPerVideo', 'zh', opts)).toBe( + '每条视频 TCO 成本(Hyperscaler 自有设备)', + ); expect(metricLabel('p90Latency', 'en', opts)).toBe('P90 time to video (s)'); + expect(metricLabel('kjPerVideo', 'zh', opts)).toBe('每条视频 GPU 板卡能耗(kJ)'); + expect(metricLabel('apiPricePerVideo', 'en', opts)).toBe('API list price per video'); + expect(metricLabel('apiPricePerVideo', 'zh', opts)).toBe('每条视频 API 标价'); + // Only the priced TCO metrics carry the tier. + const tiered = (Object.keys(VIDEO_METRICS) as MetricId[]).filter( + (id) => metricLabel(id, 'en', opts) !== metricLabel(id, 'en', { tier: 'r' }), + ); + expect(tiered).toEqual(['videosPerDollar', 'dollarsPerVideo']); expect(formatMetric(0.20419, 'dollarsPerVideo')).toBe('$0.204'); + expect(formatMetric(0.15, 'dollarsPerVideo')).toBe('$0.150'); + expect(formatMetric(0.272, 'apiPricePerVideo')).toBe('$0.272'); expect(formatMetric(null, 'kjPerVideo')).toBe('—'); expect(formatMetric(151.10486, 'p90Latency')).toBe('151.1'); - expect(formatMetric(1234.5, 'videoSecondsPerGpuHour')).toBe('1,234.5'); - }); - it('labels and formats the API-priced metrics, with the sign before the currency symbol', () => { - expect(metricLabel('profitPerGpuHour', 'en', opts)).toBe( - 'Profit per GPU-hour, API list price − TCO (Owning at Large Hyperscaler Volume)', - ); - expect(metricLabel('apiPriceMultiple', 'zh', { ...opts, tier: 'r' })).toBe( - 'API 标价 ÷ 每条视频 TCO 成本(租赁 - 3 年承诺)', - ); - expect(metricLabel('revenuePerGpuHour', 'en', opts)).toBe( - 'Revenue per GPU-hour at API list price', - ); - expect(metricLabel('apiPricePerVideo', 'zh', opts)).toBe('每条视频 API 标价'); - expect(formatMetric(0.272, 'apiPricePerVideo')).toBe('$0.272'); - expect(formatMetric(1.62471, 'revenuePerGpuHour')).toBe('$1.62'); - expect(formatMetric(-0.43913, 'profitPerGpuHour')).toBe('-$0.44'); - expect(formatMetric(1.3318, 'apiPriceMultiple')).toBe('1.33'); - expect(formatMetric(-1.5, 'p50Latency')).toBe('-1.5'); + expect(formatMetric(1234.5, 'videosPerGpuHour')).toBe('1,234.5'); + expect(formatMetric(4.8961, 'videosPerDollar')).toBe('4.9'); + expect(formatMetric(97.41, 'powerPctCap')).toBe('97.4'); }); }); diff --git a/packages/app/src/components/video-benchmark/metrics.ts b/packages/app/src/components/video-benchmark/metrics.ts index aea6ee6e3..0e86c9319 100644 --- a/packages/app/src/components/video-benchmark/metrics.ts +++ b/packages/app/src/components/video-benchmark/metrics.ts @@ -1,8 +1,5 @@ import { costPerGpuHour, type CostTier } from './hardware'; -/** Denominator for per-GPU metrics: boards that generated the clip vs. boards the job reserved. */ -export type GpuBasis = 'participating' | 'allocated'; - /** One published serving cell (hardware × client concurrency) flattened for the dashboard. */ export interface VideoPoint { id: string; @@ -16,7 +13,9 @@ export interface VideoPoint { /** Human-readable workload summary from the projection (resolution · duration · fps · steps · model @ rev · seeds · prompt). */ workload: string; concurrency: number | null; + /** Boards that generated the clip: the denominator of every per-GPU metric. */ participating: number | null; + /** Boards the job reserved; the surplus sat idle and is shown, not billed. */ allocated: number | null; /** Model replicas behind the endpoint; null for bundles that predate the deployment record. */ replicas: number | null; @@ -41,31 +40,16 @@ export interface VideoPoint { observedAt: string | null; } -export type XMetricId = 'p90Latency' | 'p50Latency' | 'genSpeed'; -export type YMetricId = - | 'videosPerDollar' - | 'videosPerGpuHour' - | 'videoSecondsPerGpuHour' - | 'dollarsPerVideo' - | 'kjPerVideo' - | 'videosPerKwh' - | 'revenuePerGpuHour' - | 'profitPerGpuHour' - | 'apiPriceMultiple'; -export type MetricId = - | XMetricId - | YMetricId - | 'powerPctCap' - | 'dollarsPerVideoSecond' - | 'apiPricePerVideo'; +export type XMetricId = 'p90Latency' | 'p50Latency'; +export type YMetricId = 'videosPerDollar' | 'dollarsPerVideo' | 'videosPerGpuHour' | 'kjPerVideo'; +/** Axis metrics plus the two the cards and evidence read but the chart does not plot. */ +export type MetricId = XMetricId | YMetricId | 'powerPctCap' | 'apiPricePerVideo'; export interface MetricOptions { tier: CostTier; - basis: GpuBasis; /** * USD an API bills per video-second of the same clip (`H3_API_REFERENCE` or - * the reader's override). Missing or non-positive → every API-priced metric - * is null. + * the reader's override). Missing or non-positive → the API list price is null. */ apiPricePerVideoSecond?: number | null; } @@ -82,17 +66,12 @@ interface MetricDefinition { currency?: boolean; } -export const X_METRICS: readonly XMetricId[] = ['p90Latency', 'p50Latency', 'genSpeed']; +export const X_METRICS: readonly XMetricId[] = ['p90Latency', 'p50Latency']; export const Y_METRICS: readonly YMetricId[] = [ 'videosPerDollar', - 'videosPerGpuHour', - 'videoSecondsPerGpuHour', 'dollarsPerVideo', + 'videosPerGpuHour', 'kjPerVideo', - 'videosPerKwh', - 'revenuePerGpuHour', - 'profitPerGpuHour', - 'apiPriceMultiple', ]; export const COST_TIERS: readonly CostTier[] = ['h', 'r']; @@ -117,13 +96,6 @@ export const VIDEO_METRICS: Record = { polarity: 'lower', digits: 1, }, - genSpeed: { - label: 'Generation speed (frames/s per request)', - labelZh: '生成速度(每请求 frames/s)', - unit: 'frames/s', - polarity: 'higher', - digits: 2, - }, videosPerDollar: { label: 'Videos per $1 TCO', labelZh: '每 1 美元 TCO 生成视频数', @@ -132,20 +104,6 @@ export const VIDEO_METRICS: Record = { digits: 2, tiered: true, }, - videosPerGpuHour: { - label: 'Videos per GPU-hour', - labelZh: '每 GPU 小时生成视频数', - unit: 'videos/GPU-hr', - polarity: 'higher', - digits: 2, - }, - videoSecondsPerGpuHour: { - label: 'Video seconds per GPU-hour', - labelZh: '每 GPU 小时生成视频秒数', - unit: 'video-s/GPU-hr', - polarity: 'higher', - digits: 1, - }, dollarsPerVideo: { label: 'TCO cost per video', labelZh: '每条视频 TCO 成本', @@ -155,14 +113,12 @@ export const VIDEO_METRICS: Record = { tiered: true, currency: true, }, - dollarsPerVideoSecond: { - label: 'TCO cost per video-second', - labelZh: '每秒视频时长 TCO 成本', - unit: '$/video-s', - polarity: 'lower', - digits: 4, - tiered: true, - currency: true, + videosPerGpuHour: { + label: 'Videos per GPU-hour', + labelZh: '每 GPU 小时生成视频数', + unit: 'videos/GPU-hr', + polarity: 'higher', + digits: 2, }, kjPerVideo: { label: 'GPU-board energy per video (kJ)', @@ -171,13 +127,6 @@ export const VIDEO_METRICS: Record = { polarity: 'lower', digits: 1, }, - videosPerKwh: { - label: 'Videos per GPU-board kWh', - labelZh: '每 kWh GPU 板卡电能生成视频数', - unit: 'videos/kWh', - polarity: 'higher', - digits: 2, - }, powerPctCap: { label: 'Mean board power / enforced limit (%)', labelZh: '板卡平均功率 / 生效功率上限(%)', @@ -185,8 +134,7 @@ export const VIDEO_METRICS: Record = { polarity: 'higher', digits: 1, }, - // API-priced metrics: list price × clip length, seen from the self-hosting - // operator's side, so more revenue, profit or price headroom is "higher". + // What the API lists for the same clip: a reference beside the TCO cost, never plotted. apiPricePerVideo: { label: 'API list price per video', labelZh: '每条视频 API 标价', @@ -195,67 +143,34 @@ export const VIDEO_METRICS: Record = { digits: 3, currency: true, }, - revenuePerGpuHour: { - label: 'Revenue per GPU-hour at API list price', - labelZh: '按 API 标价计的每 GPU 小时收入', - unit: '$/GPU-hr', - polarity: 'higher', - digits: 2, - currency: true, - }, - // Tiered labels take the cost tier in parentheses, so they carry no unit parentheses of their own. - profitPerGpuHour: { - label: 'Profit per GPU-hour, API list price − TCO', - labelZh: '每 GPU 小时利润,API 标价 − TCO', - unit: '$/GPU-hr', - polarity: 'higher', - digits: 2, - tiered: true, - currency: true, - }, - apiPriceMultiple: { - label: 'API list price ÷ TCO cost per video', - labelZh: 'API 标价 ÷ 每条视频 TCO 成本', - unit: '×', - polarity: 'higher', - digits: 2, - tiered: true, - }, }; const finite = (n: number | null | undefined): n is number => typeof n === 'number' && Number.isFinite(n); const positive = (n: number | null | undefined): n is number => finite(n) && n > 0; -function gpus(point: VideoPoint, basis: GpuBasis): number | null { - const n = basis === 'allocated' ? point.allocated : point.participating; +/** + * Per-GPU metrics divide by the boards that generated the clip. The retained + * H100 and B200 jobs reserved eight boards and used four; the idle ones are + * stated beside the numbers (KPI header, chart note) rather than billed. + */ +function gpus(point: VideoPoint): number | null { + const n = point.participating; return positive(n) && Number.isSafeInteger(n) ? n : null; } -function videosPerGpuHour(point: VideoPoint, basis: GpuBasis): number | null { - const n = gpus(point, basis); +function videosPerGpuHour(point: VideoPoint): number | null { + const n = gpus(point); if (!positive(point.valid) || !positive(point.wallSeconds) || n === null) return null; return (point.valid * 3600) / (point.wallSeconds * n); } function dollarsPerVideo(point: VideoPoint, options: MetricOptions): number | null { - const rate = videosPerGpuHour(point, options.basis); + const rate = videosPerGpuHour(point); const cost = point.hardwareKey ? costPerGpuHour(point.hardwareKey, options.tier) : null; return rate !== null && cost !== null ? cost / rate : null; } -/** What an API would bill for one clip of this cell's length at the reference price. */ -function apiPricePerVideo(point: VideoPoint, options: MetricOptions): number | null { - const price = options.apiPricePerVideoSecond; - return positive(price) && positive(point.durationSeconds) ? price * point.durationSeconds : null; -} - -function revenuePerGpuHour(point: VideoPoint, options: MetricOptions): number | null { - const rate = videosPerGpuHour(point, options.basis); - const price = apiPricePerVideo(point, options); - return rate !== null && price !== null ? rate * price : null; -} - /** Every formula returns null (never 0) when an input is missing or invalid. */ export function metricValue( point: VideoPoint, @@ -273,19 +188,8 @@ export function metricValue( value = point.p50; break; } - case 'genSpeed': { - value = - positive(point.frameCount) && positive(point.p50) ? point.frameCount / point.p50 : null; - break; - } case 'videosPerGpuHour': { - value = videosPerGpuHour(point, options.basis); - break; - } - case 'videoSecondsPerGpuHour': { - const rate = videosPerGpuHour(point, options.basis); - value = - rate !== null && positive(point.durationSeconds) ? rate * point.durationSeconds : null; + value = videosPerGpuHour(point); break; } case 'videosPerDollar': { @@ -297,20 +201,10 @@ export function metricValue( value = dollarsPerVideo(point, options); break; } - case 'dollarsPerVideoSecond': { - const cost = dollarsPerVideo(point, options); - value = - cost !== null && positive(point.durationSeconds) ? cost / point.durationSeconds : null; - break; - } case 'kjPerVideo': { value = positive(point.energyKj) ? point.energyKj : null; break; } - case 'videosPerKwh': { - value = positive(point.energyKj) ? 3600 / point.energyKj : null; - break; - } case 'powerPctCap': { value = positive(point.avgPowerW) && positive(point.enforcedLimitW) @@ -319,25 +213,10 @@ export function metricValue( break; } case 'apiPricePerVideo': { - value = apiPricePerVideo(point, options); - break; - } - case 'revenuePerGpuHour': { - value = revenuePerGpuHour(point, options); - break; - } - case 'profitPerGpuHour': { - // Negative when the API list price does not cover the tier's GPU-hour. - const revenue = revenuePerGpuHour(point, options); - const cost = point.hardwareKey ? costPerGpuHour(point.hardwareKey, options.tier) : null; - value = revenue !== null && cost !== null ? revenue - cost : null; - break; - } - case 'apiPriceMultiple': { - // > 1 means the API list price exceeds the self-hosted TCO cost of the clip. - const price = apiPricePerVideo(point, options); - const cost = dollarsPerVideo(point, options); - value = price !== null && cost !== null ? price / cost : null; + // List price × clip length; the reader's override is validated upstream. + const price = options.apiPricePerVideoSecond; + value = + positive(price) && positive(point.durationSeconds) ? price * point.durationSeconds : null; break; } } @@ -352,14 +231,12 @@ export function metricLabel(id: MetricId, locale: 'en' | 'zh', options: MetricOp return locale === 'zh' ? `${base}(${tier})` : `${base} (${tier})`; } -/** Sign precedes the currency symbol (`-$0.44`), as profit per GPU-hour can be negative. */ export function formatMetric(value: number | null, id: MetricId): string { if (value === null) return '—'; const def = VIDEO_METRICS[id]; - const text = Math.abs(value).toLocaleString('en-US', { + const text = value.toLocaleString('en-US', { minimumFractionDigits: def.currency ? def.digits : 0, maximumFractionDigits: def.digits, }); - const sign = value < 0 ? '-' : ''; - return def.currency ? `${sign}$${text}` : `${sign}${text}`; + return def.currency ? `$${text}` : text; } diff --git a/packages/app/src/components/video-benchmark/plot.test.ts b/packages/app/src/components/video-benchmark/plot.test.ts index bbbda936c..72730b6cd 100644 --- a/packages/app/src/components/video-benchmark/plot.test.ts +++ b/packages/app/src/components/video-benchmark/plot.test.ts @@ -1,7 +1,10 @@ import { describe, expect, it } from 'vitest'; -import type { VideoPoint } from './metrics'; +import history from '../../../cypress/fixtures/api/video-history.json'; +import type { VideoHistoryPage } from './history'; +import { X_METRICS, Y_METRICS, type VideoPoint } from './metrics'; import { plotVideoPoints } from './plot'; -import { DEFAULT_VIDEO_DASHBOARD_STATE } from './video-url-state'; +import { videoPoints } from './points'; +import { DEFAULT_VIDEO_DASHBOARD_STATE, type VideoDashboardState } from './video-url-state'; const base: VideoPoint = { id: 'h200-4g-c1', @@ -62,8 +65,9 @@ const h200 = [ p50: 159, wallSeconds: 3100, }), // 5.81 - cell({ id: 'h200-4g-c2', concurrency: 2, p90: 300, p50: 299, wallSeconds: 2990 }), // queued, marginally higher rate ]; +// Two clients on the one-replica 4-board deployment: queued, with a marginally higher rate than C1. +const queued = cell({ id: 'h200-4g-c2', concurrency: 2, p90: 300, p50: 299, wallSeconds: 2990 }); const b200 = cell({ id: 'b200-4g', hardwareKey: 'b200', @@ -73,63 +77,97 @@ const b200 = cell({ wallSeconds: 1560, }); const colorFor = (key: string) => `color-${key}`; -const state = { +const state: VideoDashboardState = { ...DEFAULT_VIDEO_DASHBOARD_STATE, - x: 'p90Latency' as const, - y: 'videosPerGpuHour' as const, + x: 'p90Latency', + y: 'videosPerGpuHour', }; +const plot = ( + points: VideoPoint[], + s: VideoDashboardState = state, + hidden: ReadonlySet = new Set(), +) => plotVideoPoints(points, s, colorFor, hidden); +const ids = (points: { id: string }[]) => points.map((p) => p.id); describe('plotVideoPoints', () => { - it('builds a per-hardware frontier over deployments and leaves queued cells out of it', () => { - const plot = plotVideoPoints([...h200, b200], state, colorFor, new Set()); - expect(plot.frontiers.h200.map((p) => p.id)).toEqual(['h200-8g', 'h200-4g-c1', 'h200-2g']); - expect(plot.frontiers.b200.map((p) => p.id)).toEqual(['b200-4g']); - expect(plot.plotted.map((p) => p.id)).toEqual([ + it('builds a per-hardware frontier over deployments and marks the points on it', () => { + const out = plot([...h200, b200]); + expect(Object.keys(out).toSorted()).toEqual(['frontiers', 'multiLayout', 'plotted']); + expect(ids(out.frontiers.h200)).toEqual(['h200-8g', 'h200-4g-c1', 'h200-2g']); + expect(ids(out.frontiers.b200)).toEqual(['b200-4g']); + expect(ids(out.plotted)).toEqual([ 'h200-4g-c1', 'h200-8g', 'h200-2g', 'h200-4g-tp4', 'b200-4g', ]); - expect(plot.plotted.find((p) => p.id === 'h200-4g-tp4')?.optimal).toBe(false); - expect(plot.multiLayout).toBe(true); - // B200 is faster and more efficient than every H200 deployment, so it is the whole global frontier. - expect(plot.global.map((p) => p.id)).toEqual(['b200-4g']); + expect(out.plotted.map((p) => p.optimal)).toEqual([true, true, true, false, true]); + expect(out.plotted[1]).toMatchObject({ + x: 85, + y: expect.closeTo(5.2941, 3), + color: 'color-h200', + label: 'H200', + }); + expect(out.multiLayout).toBe(true); }); - it('shows queued cells only on request, faded from every frontier', () => { - const plot = plotVideoPoints(h200, { ...state, queue: true }, colorFor, new Set()); - const queued = plot.plotted.filter((p) => p.queued); - expect(queued.map((p) => p.id)).toEqual(['h200-4g-c2']); - expect(plot.frontiers.h200.map((p) => p.id)).not.toContain('h200-4g-c2'); - expect(plot.global.map((p) => p.id)).not.toContain('h200-4g-c2'); + it('never plots a queued cell: more clients than replicas only queue on the batch-one server', () => { + const out = plot([...h200, queued]); + expect(ids(out.plotted)).not.toContain('h200-4g-c2'); + expect(ids(out.frontiers.h200)).toEqual(['h200-8g', 'h200-4g-c1', 'h200-2g']); + // No axis pair or cost tier re-admits it. + for (const x of X_METRICS) { + for (const y of Y_METRICS) { + expect(ids(plot([base, queued], { ...state, x, y, tier: 'r' }).plotted)).toEqual([ + 'h200-4g-c1', + ]); + } + } + // An unrecorded replica count reads as one; one client per replica is not queueing. + expect(ids(plot([cell({ id: 'c4-r2', concurrency: 4, replicas: 2 })]).plotted)).toEqual([]); + expect(ids(plot([cell({ id: 'c2-r2', concurrency: 2, replicas: 2 })]).plotted)).toEqual([ + 'c2-r2', + ]); + expect(ids(plot([cell({ id: 'c-na', concurrency: null })]).plotted)).toEqual(['c-na']); }); - it('optimal-only hides dominated deployments and queued cells', () => { - const plot = plotVideoPoints( - h200, - { ...state, queue: true, optimal: true }, - colorFor, - new Set(), + it('plots only the C1 cells of the retained campaign, whose C2 and C4 cells queue on one replica', () => { + // H100/H200/B200 C1/C2/C4 of 2026-09-09, n = 20 each, replicas unrecorded → one endpoint. + const out = plot( + videoPoints([history as unknown as VideoHistoryPage]), + DEFAULT_VIDEO_DASHBOARD_STATE, ); - expect(plot.plotted.map((p) => p.id)).toEqual(['h200-4g-c1', 'h200-8g', 'h200-2g']); + expect(out.plotted.map((p) => [p.hardwareKey, p.concurrency])).toEqual([ + ['b200', 1], + ['h200', 1], + ['h100', 1], + ]); + // P90 time to video against videos per $1 TCO at the hyperscaler tier ($1.73/$1.22/$1.17 GPU-hr). + expect(out.plotted.map((p) => [p.x, p.y])).toEqual([ + [expect.closeTo(78.316, 3), expect.closeTo(6.6625, 3)], + [expect.closeTo(151.105, 3), expect.closeTo(4.8961, 3)], + [expect.closeTo(168.1135, 3), expect.closeTo(4.5933, 3)], + ]); + expect(out.multiLayout).toBe(false); + expect(Object.values(out.frontiers).map((f) => f.length)).toEqual([1, 1, 1]); + expect(out.plotted.every((p) => p.optimal)).toBe(true); }); it('is a scatter of single deployments when each hardware has one layout', () => { - const plot = plotVideoPoints([base, b200], state, colorFor, new Set()); - expect(plot.multiLayout).toBe(false); - expect(Object.values(plot.frontiers).every((f) => f.length === 1)).toBe(true); - expect(plot.global.map((p) => p.id)).toEqual(['b200-4g']); + const out = plot([base, b200]); + expect(out.multiLayout).toBe(false); + expect(Object.values(out.frontiers).every((f) => f.length === 1)).toBe(true); + expect(out.plotted.every((p) => p.optimal)).toBe(true); }); it('drops hidden hardware and cells missing either metric', () => { - const plot = plotVideoPoints( + const out = plot( [ base, b200, cell({ id: 'no-p90', hardwareKey: 'h100', hardwareName: 'H100', samples: 4, p90: null }), ], state, - colorFor, new Set(['b200']), ); - expect(plot.plotted.map((p) => p.id)).toEqual(['h200-4g-c1']); - expect(plot.frontiers).not.toHaveProperty('b200'); + expect(ids(out.plotted)).toEqual(['h200-4g-c1']); + expect(out.frontiers).not.toHaveProperty('b200'); }); }); diff --git a/packages/app/src/components/video-benchmark/plot.ts b/packages/app/src/components/video-benchmark/plot.ts index dd705ac32..c0f1b3b8a 100644 --- a/packages/app/src/components/video-benchmark/plot.ts +++ b/packages/app/src/components/video-benchmark/plot.ts @@ -10,28 +10,24 @@ export interface PlottedVideoPoint extends VideoPoint { y: number; color: string; label: string; - /** Client concurrency above one request per replica: the same deployment under load. */ - queued: boolean; - /** On its hardware's Pareto frontier (deployment cells only). */ + /** On its hardware's Pareto frontier over deployments. */ optimal: boolean; } export interface VideoPlot { - /** Cells to draw: deployments (dominated ones unless `optimal`), plus queued cells when `queue`. */ + /** Deployment cells to draw, one per hardware until a GPUs-per-video sweep lands. */ plotted: PlottedVideoPoint[]; /** Per-hardware Pareto frontier over its deployment cells, ascending x; lines need two or more. */ frontiers: Record; - /** Cross-hardware Pareto frontier over every deployment cell. */ - global: PlottedVideoPoint[]; /** True once any hardware has more than one measured deployment. */ multiLayout: boolean; } /** * The chart's data model. A hardware's curve is the Pareto frontier over its - * measured deployments (GPUs per video and how they split the model); client - * concurrency on a batch-one server only queues requests, so queued cells are - * plotted as evidence but never join a frontier. + * measured deployments (GPUs per video and how they split the model). Client + * concurrency above one request per replica only queues on the batch-one + * server, so those cells stay in the evidence panel and never reach the chart. */ export function plotVideoPoints( points: VideoPoint[], @@ -40,8 +36,8 @@ export function plotVideoPoints( hidden: ReadonlySet, ): VideoPlot { const options = metricOptions(state); - const cells: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { - if (!p.hardwareKey || hidden.has(p.hardwareKey)) return []; + const plotted: PlottedVideoPoint[] = latestVideoCells(points).flatMap((p) => { + if (!p.hardwareKey || hidden.has(p.hardwareKey) || isQueueing(p)) return []; const x = metricValue(p, state.x, options); const y = metricValue(p, state.y, options); if (x === null || y === null) return []; @@ -52,14 +48,12 @@ export function plotVideoPoints( y, color: colorFor(p.hardwareKey), label: hardwareLabel(p.hardwareKey), - queued: isQueueing(p), optimal: false, }, ]; }); - const deployments = cells.filter((p) => !p.queued); const byHardware = new Map(); - for (const p of deployments) { + for (const p of plotted) { const list = byHardware.get(p.hardwareKey!) ?? []; list.push(p); byHardware.set(p.hardwareKey!, list); @@ -73,12 +67,5 @@ export function plotVideoPoints( for (const p of frontiers[key]) p.optimal = true; if (new Set(list.map(deploymentKey)).size > 1) multiLayout = true; } - const shown = state.optimal ? deployments.filter((p) => p.optimal) : deployments; - const queued = state.queue && !state.optimal ? cells.filter((p) => p.queued) : []; - return { - plotted: [...shown, ...queued], - frontiers, - global: paretoFrontier(deployments, xBetter, yBetter), - multiLayout, - }; + return { plotted, frontiers, multiLayout }; } diff --git a/packages/app/src/components/video-benchmark/video-url-state.test.ts b/packages/app/src/components/video-benchmark/video-url-state.test.ts index e7dcb3b19..48c12a888 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.test.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.test.ts @@ -1,39 +1,47 @@ import { describe, expect, it } from 'vitest'; import { H3_API_REFERENCE } from './api-reference'; +import { X_METRICS, Y_METRICS } from './metrics'; import { DEFAULT_VIDEO_DASHBOARD_STATE, metricOptions, parseApiPrice, readVideoDashboardState, writeVideoDashboardState, + type VideoDashboardState, } from './video-url-state'; describe('video dashboard URL state', () => { it('falls back to defaults for missing or invalid params', () => { expect(readVideoDashboardState('')).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); + expect(readVideoDashboardState('?v_x=bogus&v_y=nope&v_tier=z&v_view=pie&v_api=free')).toEqual( + DEFAULT_VIDEO_DASHBOARD_STATE, + ); + }); + it('ignores retired params and metric ids, so old share links open on the defaults', () => { + // The GPU-basis switch, the queue/optimal/frontier toggles and the profit axes are gone. expect( readVideoDashboardState( - '?v_x=bogus&v_y=nope&v_tier=z&v_basis=q&v_view=pie&v_queue=yes&v_api=free', + '?v_basis=allocated&v_queue=1&v_opt=1&v_frontier=1&v_x=genSpeed&v_y=profitPerGpuHour', ), ).toEqual(DEFAULT_VIDEO_DASHBOARD_STATE); + // Each axis accepts only its own metrics; the card-only metrics are not axes. + expect(readVideoDashboardState('?v_x=videosPerDollar').x).toBe('p90Latency'); + expect(readVideoDashboardState('?v_y=p50Latency').y).toBe('videosPerDollar'); + expect(readVideoDashboardState('?v_y=apiPricePerVideo').y).toBe('videosPerDollar'); }); it('reads every supported param', () => { expect( - readVideoDashboardState( - '?v_x=genSpeed&v_y=kjPerVideo&v_tier=r&v_basis=allocated&v_queue=1&v_opt=1&v_frontier=1&v_view=table&v_api=0.047', - ), - ).toEqual({ - x: 'genSpeed', - y: 'kjPerVideo', - tier: 'r', - basis: 'allocated', - queue: true, - optimal: true, - frontier: true, - view: 'table', - apiPrice: 0.047, - }); - expect(readVideoDashboardState('?v_y=profitPerGpuHour').y).toBe('profitPerGpuHour'); + readVideoDashboardState('?v_x=p50Latency&v_y=kjPerVideo&v_tier=r&v_view=table&v_api=0.047'), + ).toEqual({ x: 'p50Latency', y: 'kjPerVideo', tier: 'r', view: 'table', apiPrice: 0.047 }); + expect([...X_METRICS]).toEqual(['p90Latency', 'p50Latency']); + expect([...Y_METRICS]).toEqual([ + 'videosPerDollar', + 'dollarsPerVideo', + 'videosPerGpuHour', + 'kjPerVideo', + ]); + for (const x of X_METRICS) expect(readVideoDashboardState(`?v_x=${x}`).x).toBe(x); + for (const y of Y_METRICS) expect(readVideoDashboardState(`?v_y=${y}`).y).toBe(y); }); it('defaults the API price to the dated reference and rejects non-positive or malformed prices', () => { expect(DEFAULT_VIDEO_DASHBOARD_STATE.apiPrice).toBe(H3_API_REFERENCE.pricePerVideoSecondUsd); @@ -55,19 +63,31 @@ describe('video dashboard URL state', () => { const out = writeVideoDashboardState(url, { ...DEFAULT_VIDEO_DASHBOARD_STATE, tier: 'r', - queue: true, + view: 'table', }); expect(out.searchParams.get('run')).toBe('1'); + // The page's own `view` param is not the chart/table switch, which lives under `v_view`. expect(out.searchParams.get('view')).toBe('results'); expect(out.searchParams.get('v_tier')).toBe('r'); - expect(out.searchParams.get('v_queue')).toBe('1'); + expect(out.searchParams.get('v_view')).toBe('table'); expect(out.searchParams.has('v_x')).toBe(false); + expect(out.searchParams.has('v_y')).toBe(false); expect(out.searchParams.has('v_api')).toBe(false); expect(url.searchParams.has('v_tier')).toBe(false); const reset = writeVideoDashboardState(out, DEFAULT_VIDEO_DASHBOARD_STATE); expect([...reset.searchParams.keys()].filter((k) => k.startsWith('v_'))).toEqual([]); expect(reset.searchParams.get('run')).toBe('1'); }); + it('round-trips the axis metrics', () => { + const chosen: VideoDashboardState = { + ...DEFAULT_VIDEO_DASHBOARD_STATE, + x: 'p50Latency', + y: 'dollarsPerVideo', + }; + const out = writeVideoDashboardState(new URL('https://x.test/video'), chosen); + expect(out.search).toBe('?v_x=p50Latency&v_y=dollarsPerVideo'); + expect(readVideoDashboardState(out.search)).toEqual(chosen); + }); it('round-trips an overridden API price and drops an invalid one', () => { const url = new URL('https://x.test/video'); const out = writeVideoDashboardState(url, { ...DEFAULT_VIDEO_DASHBOARD_STATE, apiPrice: 0.05 }); @@ -92,20 +112,15 @@ describe('video dashboard URL state', () => { }).searchParams.has('v_api'), ).toBe(false); }); - it('maps state to metric options, including the API price', () => { + it('maps state to metric options: the cost tier and the API price, nothing else', () => { expect(metricOptions(DEFAULT_VIDEO_DASHBOARD_STATE)).toEqual({ tier: 'h', - basis: 'participating', apiPricePerVideoSecond: 0.034, }); - expect( - metricOptions({ - ...DEFAULT_VIDEO_DASHBOARD_STATE, - tier: 'r', - basis: 'allocated', - apiPrice: 0.05, - }), - ).toEqual({ tier: 'r', basis: 'allocated', apiPricePerVideoSecond: 0.05 }); + expect(metricOptions({ ...DEFAULT_VIDEO_DASHBOARD_STATE, tier: 'r', apiPrice: 0.05 })).toEqual({ + tier: 'r', + apiPricePerVideoSecond: 0.05, + }); expect( metricOptions({ ...DEFAULT_VIDEO_DASHBOARD_STATE, apiPrice: -1 }).apiPricePerVideoSecond, ).toBeNull(); diff --git a/packages/app/src/components/video-benchmark/video-url-state.ts b/packages/app/src/components/video-benchmark/video-url-state.ts index 29626b1e1..9b4e22b99 100644 --- a/packages/app/src/components/video-benchmark/video-url-state.ts +++ b/packages/app/src/components/video-benchmark/video-url-state.ts @@ -4,7 +4,6 @@ import { COST_TIERS, X_METRICS, Y_METRICS, - type GpuBasis, type MetricOptions, type XMetricId, type YMetricId, @@ -14,15 +13,8 @@ export interface VideoDashboardState { x: XMetricId; y: YMetricId; tier: CostTier; - basis: GpuBasis; - /** Also plot queued client-concurrency cells (C > replicas) as faded markers. */ - queue: boolean; - /** Hide deployments dominated by another deployment of the same hardware. */ - optimal: boolean; - /** Draw the cross-hardware Pareto frontier over all deployments. */ - frontier: boolean; view: 'chart' | 'table'; - /** USD per video-second the API-priced metrics assume; the dated reference unless the reader overrides it. */ + /** USD per video-second the API list price assumes; the dated reference unless the reader overrides it. */ apiPrice: number; } @@ -30,15 +22,10 @@ export const DEFAULT_VIDEO_DASHBOARD_STATE: VideoDashboardState = { x: 'p90Latency', y: 'videosPerDollar', tier: 'h', - basis: 'participating', - queue: false, - optimal: false, - frontier: false, view: 'chart', apiPrice: H3_API_REFERENCE.pricePerVideoSecondUsd, }; -const BASES: readonly GpuBasis[] = ['participating', 'allocated']; const VIEWS: readonly VideoDashboardState['view'][] = ['chart', 'table']; const pick = (allowed: readonly T[], value: string | null, fallback: T): T => @@ -57,10 +44,9 @@ export function parseApiPrice(raw: string | number | null | undefined): number | return rounded > 0 ? rounded : null; } -/** Metric inputs the dashboard controls select, so no component rebuilds `{ tier, basis }` by hand. */ +/** Metric inputs the dashboard controls select, so no component rebuilds them by hand. */ export const metricOptions = (state: VideoDashboardState): MetricOptions => ({ tier: state.tier, - basis: state.basis, apiPricePerVideoSecond: parseApiPrice(state.apiPrice), }); @@ -72,10 +58,6 @@ export function readVideoDashboardState(search: string): VideoDashboardState { x: pick(X_METRICS, p.get('v_x'), d.x), y: pick(Y_METRICS, p.get('v_y'), d.y), tier: pick(COST_TIERS, p.get('v_tier'), d.tier), - basis: pick(BASES, p.get('v_basis'), d.basis), - queue: p.get('v_queue') === '1', - optimal: p.get('v_opt') === '1', - frontier: p.get('v_frontier') === '1', view: pick(VIEWS, p.get('v_view'), d.view), apiPrice: parseApiPrice(p.get('v_api')) ?? d.apiPrice, }; @@ -90,10 +72,6 @@ export function writeVideoDashboardState(url: URL, state: VideoDashboardState): set('v_x', state.x === d.x ? null : state.x); set('v_y', state.y === d.y ? null : state.y); set('v_tier', state.tier === d.tier ? null : state.tier); - set('v_basis', state.basis === d.basis ? null : state.basis); - set('v_queue', state.queue ? '1' : null); - set('v_opt', state.optimal ? '1' : null); - set('v_frontier', state.frontier ? '1' : null); set('v_view', state.view === d.view ? null : state.view); const apiPrice = parseApiPrice(state.apiPrice); set('v_api', apiPrice === null || apiPrice === d.apiPrice ? null : String(apiPrice)); From cfa834a5e23e37bbeb7767eb77052ddd6c79dfaf Mon Sep 17 00:00:00 2001 From: Wenyao Gao Date: Sun, 20 Sep 2026 13:08:45 -0700 Subject: [PATCH 039/104] refactor: remove the legacy /video viewer, trade-off chart and CI-run browser MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dashboard, Compare and Evidence panels replace the per-run viewer; the Performance history list stays in the collapsible section and opens for history-* deep links. Keep the data layer (bundle, stored, archive, serving, fidelity, tradeoff points) that the history API and Compare still read, and trim the comments that still described removed metrics. 中文:删除旧的 /video 按运行查看器、trade-off 图表和 CI 运行浏览;仪表板、Compare 与证据面板取而代之,性能历史列表保留在可展开区块并响应 history-* 深链接。保留 history API 与 Compare 仍在读取的数据层(bundle、stored、archive、serving、fidelity、tradeoff points),并修掉仍在描述已删指标的注释。 --- docs/video-history.md | 4 +- docs/videogenx-dashboard.md | 2 +- .../cypress/component/video-ci-runs.cy.tsx | 523 ---------- .../cypress/component/video-compare.cy.tsx | 2 - .../cypress/component/video-dashboard.cy.tsx | 11 +- .../cypress/component/video-fidelity.cy.tsx | 133 --- .../component/video-results-navigation.cy.tsx | 165 --- .../component/video-serving-results.cy.tsx | 399 -------- .../cypress/component/video-tradeoff.cy.tsx | 290 ------ packages/app/cypress/e2e/navigation.cy.ts | 134 +-- .../app/cypress/e2e/video-dashboard.cy.ts | 11 +- .../video-benchmark/FidelityResults.tsx | 493 --------- .../video-benchmark/ResultPower.tsx | 236 ----- .../video-benchmark/ResultSummary.tsx | 246 ----- .../video-benchmark/ServingResults.tsx | 886 ---------------- .../video-benchmark/VideoBenchmark.tsx | 965 ------------------ .../video-benchmark/VideoCIRuns.tsx | 730 ------------- .../video-benchmark/VideoDashboard.tsx | 45 +- .../video-benchmark/VideoHistory.tsx | 29 +- .../video-benchmark/VideoTradeoff.tsx | 605 ----------- .../video-benchmark/chart-domain.ts | 5 +- .../video-benchmark/comparison.test.ts | 36 - .../components/video-benchmark/comparison.ts | 21 - .../components/video-benchmark/deployment.ts | 2 +- .../components/video-benchmark/frontier.ts | 2 +- .../components/video-benchmark/report.test.ts | 42 - .../src/components/video-benchmark/report.ts | 43 - .../video-benchmark/tradeoff.test.ts | 38 - .../components/video-benchmark/tradeoff.ts | 28 - 29 files changed, 48 insertions(+), 6078 deletions(-) delete mode 100644 packages/app/cypress/component/video-ci-runs.cy.tsx delete mode 100644 packages/app/cypress/component/video-fidelity.cy.tsx delete mode 100644 packages/app/cypress/component/video-results-navigation.cy.tsx delete mode 100644 packages/app/cypress/component/video-serving-results.cy.tsx delete mode 100644 packages/app/cypress/component/video-tradeoff.cy.tsx delete mode 100644 packages/app/src/components/video-benchmark/FidelityResults.tsx delete mode 100644 packages/app/src/components/video-benchmark/ResultPower.tsx delete mode 100644 packages/app/src/components/video-benchmark/ResultSummary.tsx delete mode 100644 packages/app/src/components/video-benchmark/ServingResults.tsx delete mode 100644 packages/app/src/components/video-benchmark/VideoBenchmark.tsx delete mode 100644 packages/app/src/components/video-benchmark/VideoCIRuns.tsx delete mode 100644 packages/app/src/components/video-benchmark/VideoTradeoff.tsx delete mode 100644 packages/app/src/components/video-benchmark/comparison.test.ts delete mode 100644 packages/app/src/components/video-benchmark/comparison.ts delete mode 100644 packages/app/src/components/video-benchmark/report.test.ts delete mode 100644 packages/app/src/components/video-benchmark/report.ts diff --git a/docs/video-history.md b/docs/video-history.md index 6063551bf..13de6f18d 100644 --- a/docs/video-history.md +++ b/docs/video-history.md @@ -1,6 +1,6 @@ # Video performance history -`/video` and `/zh/video` lead with the cross-hardware dashboard (see [VideoGenX dashboard](./videogenx-dashboard.md)); the run, results and **Performance history** views described here live in the collapsible "Runs, videos & evidence" section below it, which opens automatically when the URL carries any of `run`, `artifact`, `source`, `cell`, `view`, `compare` or `history-*`. Explicit run/artifact/source/cell URLs retain the existing video viewer. `?view=results` opens the first readable artifact/source from the newest published page; `?view=history` restores history. History filters use `history-hardware`, `history-concurrency`, and `history-query` in the URL. Videos & result follows the same published-first path when no run is selected; empty or failed reads show a recoverable message. Browse CI runs explicitly opens legacy Actions discovery. Returning to History cancels pending result selection; Refresh and reload preserve the selected artifact/source/cell. +`/video` and `/zh/video` lead with the cross-hardware dashboard (see [VideoGenX dashboard](./videogenx-dashboard.md)); the **Performance history** list described here sits in a collapsible section below it, which opens automatically when the URL carries a `history-*` filter (`history-hardware`, `history-concurrency`, `history-query`). Each entry links to its original GitHub Actions run. The earlier per-run viewer, trade-off chart and CI-run browser were removed with the dashboard rebuild; clips are compared in the dashboard's Compare panel. ## Read path @@ -24,7 +24,7 @@ Each observation also carries the cell's participating and allocated GPU counts, ## 中文说明 -`/video` 与 `/zh/video` 首屏为跨硬件仪表板(见 [VideoGenX 仪表板](./videogenx-dashboard.md));本文描述的运行、结果与“性能历史”视图位于其下方可展开的“运行、视频与证据”区块,URL 带有 `run`、`artifact`、`source`、`cell`、`view`、`compare` 或 `history-*` 参数时自动展开。原有 run/artifact/source/cell 链接仍打开对应结果;`?view=results` 打开最新发布页面中首个可读取的产物与原始运行;未选择运行时,“视频与结果”也使用该入口。空列表或读取失败会显示可恢复的提示;只有点击“浏览 CI 运行”才扫描 GitHub Actions。返回性能历史会取消正在进行的结果选择;刷新和重新加载保留产物、原始运行及并发配置。历史筛选条件保存在 URL,刷新后恢复。 +`/video` 与 `/zh/video` 首屏为跨硬件仪表板(见 [VideoGenX 仪表板](./videogenx-dashboard.md));本文描述的“性能历史”列表位于其下方可展开的区块,URL 带有 `history-*` 筛选参数(`history-hardware`、`history-concurrency`、`history-query`)时自动展开。每条记录链接到原始 GitHub Actions 运行。原有的按运行查看结果、trade-off 图表与 CI 运行浏览已随仪表板重建移除;视频对比由仪表板的 Compare 面板提供。 历史读取只列举既有发布索引的元数据,按索引发布时间排序,每页最多读取十份完整索引;不下载或发布媒体。原始执行时间取自 `ci.started_at`,与发布时间、重新导出的时间分开。索引无效时保留错误记录,缺失指标保留 null。每个 cell 另附参与/已分配 GPU 数、测量窗口秒数、视频时长与帧数、板卡平均功率之和、记录的生效功率上限之和(无快照时为 null)、配置的 `tp_size` / `ulysses_degree` / `attention_backend`,以及端点的副本数(`execution.deployment.replica_count`,早于该记录的产物包为 null);这些都是同一 `schemaVersion: 1` 下可为空的附加字段,硬件仪表板据此推导全部指标,浏览器不再解析产物包。 diff --git a/docs/videogenx-dashboard.md b/docs/videogenx-dashboard.md index b747f75d5..90a11a807 100644 --- a/docs/videogenx-dashboard.md +++ b/docs/videogenx-dashboard.md @@ -41,7 +41,7 @@ Today every hardware has exactly one measured deployment (4 GPUs per video, TP2 ## 中文说明 -`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;原有的运行/视频/历史视图收进可展开的“运行、视频与证据”区块,`?run=`、`?view=`、`?compare=` 和 `history-*` 深链接会自动展开该区块。`video-url-state.ts` 管理 `v_x`、`v_y`、`v_tier`、`v_view`、`v_api` 五个 URL 参数,默认值不写入 URL。X 轴可选 P90(默认)或 P50 出片时间;Y 轴可选每 1 美元 TCO 视频数(默认,Hyperscaler 自有设备档位,单价与 `/inference` 同源)、每条视频 TCO 成本、每 GPU 小时视频数和每条视频 GPU 板卡能耗(kJ)。所有按 GPU 计的指标都以参与计算的 GPU 为分母(每条视频 4 张,TP2 × Ulysses 2),而不是任务预留的板卡数;保留数据中 H100 与 B200 的运行预留了 8 张却只用 4 张,图表标题下方的说明会点名这些硬件,对应 KPI 卡标题显示“4 / 8 张 GPU”。空闲板卡只作说明、不计成本,没有 GPU 口径切换。两条坐标轴都从 0 起,便于按长度比较。 +`/video` 以跨硬件图表为首屏:数据来自已发布 blob 索引的 per-cell 投影,浏览器不再为图表解析产物包;已发布结果的“性能历史”列表收进可展开区块,`history-*` 深链接会自动展开;原有的按运行查看、trade-off 图表与 CI 运行浏览已移除。`video-url-state.ts` 管理 `v_x`、`v_y`、`v_tier`、`v_view`、`v_api` 五个 URL 参数,默认值不写入 URL。X 轴可选 P90(默认)或 P50 出片时间;Y 轴可选每 1 美元 TCO 视频数(默认,Hyperscaler 自有设备档位,单价与 `/inference` 同源)、每条视频 TCO 成本、每 GPU 小时视频数和每条视频 GPU 板卡能耗(kJ)。所有按 GPU 计的指标都以参与计算的 GPU 为分母(每条视频 4 张,TP2 × Ulysses 2),而不是任务预留的板卡数;保留数据中 H100 与 B200 的运行预留了 8 张却只用 4 张,图表标题下方的说明会点名这些硬件,对应 KPI 卡标题显示“4 / 8 张 GPU”。空闲板卡只作说明、不计成本,没有 GPU 口径切换。两条坐标轴都从 0 起,便于按长度比较。 曲线来自部署而不是并发:pinned H3 服务是 batch-one、单副本,客户端并发只会排队(保留数据中 C1/C2/C4 的吞吐量相差不到 1.5%,延迟随 C 线性增长),把这些点连起来画出的只是队列,因此图表绝不连接它们。真正在延迟与效率之间取舍的是部署方式:每条视频占用多少张 GPU 以及模型如何切分(`tp_size` × `ulysses_degree`)。`deployment.ts` 以此为 cell 键,把 `concurrency > replicas` 判定为排队,并为每种硬件选出最高效的部署作为代表(KPI 卡、Compare 面板和图例都用它);`plot.ts` 只用未排队的 cell 绘图——排队 cell 不会进入图表、表格和 CSV,只出现在证据面板——再按硬件对剩余部署点求 Pareto 前沿:某硬件有两个及以上前沿部署时自动画出实线,被支配的部署以淡色显示。这些行为都没有开关,没有“显示排队请求”,没有“仅最优”,也没有跨硬件前沿,所有读者看到的是同一张图。目前每种硬件只有一种实测部署(每条视频 4 张 GPU,TP2 × Ulysses 2),所以图表是诚实的三点散点图,说明文字会指出每硬件曲线还需扫描每条视频占用的 GPU 数;该扫描属于 InferenceX 后端工作并需要 GPU 授权,前端无需改动即可接收新 cell。MI355X 以“未测得”形式保留在图例与 KPI 卡中;缺失指标一律为 null,绝不写成 0。 diff --git a/packages/app/cypress/component/video-ci-runs.cy.tsx b/packages/app/cypress/component/video-ci-runs.cy.tsx deleted file mode 100644 index 86d176e78..000000000 --- a/packages/app/cypress/component/video-ci-runs.cy.tsx +++ /dev/null @@ -1,523 +0,0 @@ -import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; -import VideoCIRuns from '@/components/video-benchmark/VideoCIRuns'; -import ResultPower from '@/components/video-benchmark/ResultPower'; -import VideoSelect from '@/components/video-benchmark/VideoSelect'; -import { videoHistoryEntry } from '@/components/video-benchmark/history'; -import { servingArtifact, videoRun as run } from '../support/video-artifacts'; - -describe('H3 automatic CI viewer (synthetic API fixtures)', () => { - beforeEach(() => { - cy.intercept('GET', '**/api/video-runs*format=media', { statusCode: 204 }); - cy.intercept('GET', '/api/video-runs?format=history&page=1', { - schemaVersion: 1, - entries: [], - nextPage: null, - }); - cy.window().then((win) => - win.history.replaceState(null, '', `${win.location.pathname}?view=results`), - ); - }); - it('defaults to published history and keeps filters and source selection in the shared URL', () => { - const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); - cy.window().then((win) => win.history.replaceState(null, '', win.location.pathname)); - cy.intercept('GET', '/api/video-runs?format=history&page=1', { - schemaVersion: 1, - entries: [entry], - nextPage: null, - }).as('history'); - cy.intercept('GET', '/api/video-runs?page=1', () => { - throw new Error('Published discovery must not scan recent Actions'); - }); - cy.intercept('GET', '/api/video-runs?run=123', { - run: run(123, 'success'), - artifacts: [entry.artifact], - }); - cy.intercept('GET', '**/api/video-runs*format=media', servingArtifact()); - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - cy.mount( - - - , - ); - cy.wait('@history'); - cy.get('[data-testid="video-history-observation"]').should('have.length', 3); - cy.get('[data-testid="video-history"]').should('contain', 'no matched version baseline'); - cy.get('[aria-label="Client concurrency"]').click(); - cy.contains('[role="option"]', 'C4').click(); - cy.get('[data-testid="video-history-observation"]') - .should('have.length', 1) - .and('contain', '300 / —'); - cy.location('search').should('contain', 'history-concurrency=4'); - cy.get('[data-testid="video-history-observation"] button').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C4'); - cy.location('search').should('contain', 'cell=c4').and('contain', 'history-concurrency=4'); - cy.contains('button', 'Performance history').click(); - cy.get('[data-testid="video-history-observation"]').should('have.length', 1); - }); - it('renders retained metrics and missing data in Chinese within a mobile viewport', () => { - cy.viewport(390, 844); - const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); - cy.window().then((win) => - win.history.replaceState( - null, - '', - `${win.location.pathname}?view=history&history-concurrency=1`, - ), - ); - cy.intercept('GET', '/api/video-runs?format=history&page=1', { - schemaVersion: 1, - entries: [entry], - nextPage: null, - }); - cy.mount( - - - , - ); - cy.get('[data-testid="video-history-observation"]') - .should('have.length', 1) - .and('contain', '120 / —') - .and('contain', '168') - .and('contain', '已完成'); - cy.contains('尚未选择匹配的版本基线').should('be.visible'); - cy.get('[data-testid="video-history"]').then(($history) => - expect($history[0].getBoundingClientRect().right).to.be.at.most(390), - ); - }); - it('keeps an open long-label menu within mobile bounds and supports keyboard selection', () => { - cy.viewport(390, 844); - const change = cy.stub().as('selection'); - cy.mount( -

- -
, - ); - cy.get('[role="combobox"]').focus().type('{enter}'); - cy.get('[role="listbox"]') - .should('be.visible') - .then(($menu) => { - const rect = $menu[0].getBoundingClientRect(); - expect(rect.left).to.be.at.least(0); - expect(rect.right).to.be.at.most(390); - }); - cy.get('[role="option"]').last().focus().type('{enter}'); - cy.get('@selection').should('have.been.calledWith', 'second'); - cy.get('[role="listbox"]').should('not.exist'); - cy.get('[role="combobox"]').should('have.focus'); - }); - it('browses CI runs explicitly and switches to a failed run without inventing media', () => { - cy.intercept('GET', '/api/video-runs?page=1', { - runs: [run(20, 'success'), run(10, 'failure')], - nextPage: null, - }); - cy.intercept('GET', '/api/video-runs?run=20', { run: run(20, 'success'), artifacts: [] }).as( - 'latest', - ); - cy.intercept('GET', '/api/video-runs?run=10', { run: run(10, 'failure'), artifacts: [] }).as( - 'failed', - ); - cy.mount( - - - , - ); - cy.contains('button', 'Browse CI runs').click(); - cy.wait('@latest'); - cy.contains('No result artifact for this run yet').should('be.visible'); - cy.get('[role="combobox"][aria-label="CI run"]').click(); - cy.contains('[role="option"]', '#10 · failure').click(); - cy.wait('@failed'); - cy.contains('summary', 'Run details and artifact selection').click(); - cy.contains('a', '#10 · completed / failure').should('be.visible'); - cy.get('video').should('not.exist'); - }); - it('starts shared-artifact loading before run metadata finishes', () => { - let releaseRun: (() => void) | undefined; - let mediaStarted = false; - cy.window().then((win) => - win.history.replaceState(null, '', `${win.location.pathname}?run=30&artifact=40`), - ); - cy.intercept( - 'GET', - '/api/video-runs?run=30', - (req) => - new Promise((resolve) => { - releaseRun = () => { - req.reply({ - run: run(30, 'success'), - artifacts: [{ id: 40, name: 'h3-results-30-1', expired: false, size_in_bytes: 100 }], - }); - resolve(); - }; - }), - ); - cy.intercept('GET', '**/api/video-runs*format=media', (req) => { - mediaStarted = true; - req.reply({ statusCode: 204 }); - }); - cy.intercept('GET', '/api/video-runs?run=30&artifact=40', { - statusCode: 502, - body: { error: 'Synthetic ZIP fallback' }, - }); - cy.mount( - - - , - ); - cy.wrap(null).should(() => { - expect(mediaStarted).to.equal(true); - expect(releaseRun).to.be.a('function'); - }); - cy.then(() => releaseRun?.()); - cy.get('[data-testid="video-ci-runs"] [role="alert"]').should( - 'contain', - 'Synthetic ZIP fallback', - ); - }); - it('keeps navigation and bundle validation errors visible on an empty tradeoff link', () => { - cy.window().then((win) => - win.history.replaceState( - null, - '', - `${win.location.pathname}?run=30&artifact=40&view=tradeoff`, - ), - ); - cy.intercept('GET', '/api/video-runs?run=30', { - run: run(30, 'success'), - artifacts: [{ id: 40, name: 'h3-results-30-1', expired: false, size_in_bytes: 100 }], - }); - cy.intercept('GET', '**/api/video-runs*format=media', { - storageVersion: 1, - runId: '30', - artifact: { id: 40, name: 'h3-results-30-1', expired: false, size_in_bytes: 100 }, - sources: [ - { - id: '30', - assets: [], - checksums: [['manifest.json', 'synthetic']], - texts: [['gpu/supervisor/baseline/telemetry.jsonl', 'broken JSON']], - documents: [ - ['manifest.json', { run_id: '30' }], - [ - 'gpu/gpu-job.json', - { - roles: { - baseline: { - telemetry_summary: { - qualified: true, - requested_interval_seconds: 1, - gpu_identity: [{ uuid: 'GPU-synthetic' }], - }, - }, - }, - }, - ], - ], - }, - ], - }); - cy.mount( - - - , - ); - cy.contains('button', 'Videos & result').should('be.visible'); - cy.contains('[role="alert"]', 'JSON').should('be.visible'); - cy.contains('button', 'Videos & result').click(); - cy.get('[data-testid="video-tradeoff"]').should('not.be.visible'); - cy.contains('[role="alert"]', 'JSON').should('be.visible'); - }); - it('keeps the shared run selected when browsing the first catalog page', () => { - cy.window().then((win) => - win.history.replaceState(null, '', `${win.location.pathname}?run=30`), - ); - cy.intercept('GET', '/api/video-runs?run=30', { run: run(30, 'success'), artifacts: [] }).as( - 'shared', - ); - cy.intercept('GET', '/api/video-runs?page=1', { runs: [run(20, 'success')], nextPage: 2 }).as( - 'browse', - ); - cy.mount( - - - , - ); - cy.wait('@shared'); - cy.get('[role="combobox"][aria-label="CI run"]').should('contain.text', '#30'); - cy.contains('button', 'Older runs').should('not.exist'); - cy.contains('button', 'Browse CI runs').click(); - cy.wait('@browse'); - cy.get('[role="combobox"][aria-label="CI run"]').should('contain.text', '#30').click(); - cy.contains('[role="option"]', '#20 · success').should('be.visible'); - cy.get('[role="listbox"]').type('{esc}'); - cy.contains('button', 'Older runs').should('be.visible'); - cy.location('search').should('contain', 'run=30'); - }); - it('keeps a direct selection and its download active when an older catalog arrives', () => { - let releaseList: (() => void) | undefined; - cy.intercept( - 'GET', - '/api/video-runs?page=1', - (req) => - new Promise((resolve) => { - releaseList = () => { - req.reply({ runs: [run(20, 'success')], nextPage: null }); - resolve(); - }; - }), - ).as('catalog'); - cy.intercept('GET', '/api/video-runs?run=30', { - run: run(30, 'success'), - artifacts: [{ id: 40, name: 'h3-results-30-1', expired: false, size_in_bytes: 100 }], - }).as('direct'); - cy.intercept('GET', '/api/video-runs?run=30&artifact=40', { - delay: 3000, - statusCode: 502, - body: { error: 'Synthetic download failure' }, - }).as('zip'); - cy.mount( - - - , - ); - cy.contains('button', 'Browse CI runs').click(); - cy.wrap(null).should(() => expect(releaseList).to.be.a('function')); - cy.contains('summary', 'Run details and artifact selection').click(); - cy.get('input[aria-label="GitHub run ID"]').type('30'); - cy.contains('button', 'Open run').click(); - cy.wait('@direct'); - cy.get('[role="combobox"][aria-label="Result artifact"]').should( - 'contain.text', - 'h3-results-30-1', - ); - cy.then(() => releaseList?.()); - cy.wait('@catalog'); - cy.get('[role="combobox"][aria-label="CI run"]').should('contain.text', '#30'); - cy.get('[role="status"]').should('be.visible'); - cy.wait('@zip'); - cy.get('[role="alert"]').should('contain', 'Synthetic download failure'); - }); - it('shows stored measurements without downloading the archive', () => { - const artifact = { - id: 40, - name: 'h3-results-30-1', - expired: false, - size_in_bytes: 100, - stored: true, - }; - cy.intercept('GET', '/api/video-runs?page=1', { runs: [run(30, 'success')], nextPage: null }); - cy.intercept('GET', '/api/video-runs?run=30', { - run: run(30, 'success'), - artifacts: [artifact], - }); - cy.intercept('GET', '/api/video-runs?run=30&artifact=40', () => { - throw new Error('Stored results must not fetch the ZIP'); - }); - cy.intercept('GET', '**/api/video-runs*format=media', { - storageVersion: 1, - runId: '30', - artifact, - sources: [ - { - id: '30', - assets: [], - texts: [], - checksums: [['manifest.json', 'a'.repeat(64)]], - documents: [ - ['manifest.json', { run_id: '30' }], - [ - 'ci.json', - { phase: 'complete', regression_status: 'inconclusive', release_qualified: false }, - ], - [ - 'report/evidence.json', - { - policy: { calibration_status: 'uncalibrated' }, - roles: { - baseline: { - summary: { latency_median_seconds: 150, valid_clips_per_second: 0.01 }, - }, - candidate: { - summary: { latency_median_seconds: 120, valid_clips_per_second: 0.02 }, - }, - }, - }, - ], - ], - }, - ], - }); - cy.mount( - - - , - ); - cy.contains('button', 'Browse CI runs').click(); - cy.get('[data-testid="result-summary"]') - .should('contain', 'Media stored for direct playback') - .and('contain', 'uncalibrated'); - cy.contains('tr', 'Latency') - .should('contain', '150') - .and('contain', '120') - .and('contain', '-20%'); - cy.contains('tr', 'Valid clips').should('contain', '36').and('contain', '72'); - cy.contains('tr', 'Mean GPU power').find('[aria-label="Unavailable"]').should('have.length', 2); - }); - it('opens the shared serving cell and switches from the chart without reloading the CI artifact', () => { - const saved = servingArtifact(); - cy.window().then((win) => - win.history.replaceState( - null, - '', - `${win.location.pathname}?run=123&artifact=40&source=123&cell=c4`, - ), - ); - cy.intercept('GET', '/api/video-runs?run=123', { - run: run(123, 'success'), - artifacts: [saved.artifact], - }).as('servingRun'); - cy.intercept('GET', '**/api/video-runs*format=media', saved).as('servingMedia'); - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - cy.intercept('GET', '/api/video-runs?run=123&artifact=40', () => { - throw new Error('Stored serving results must not fetch the ZIP'); - }); - cy.mount( - - - , - ); - cy.wait(['@servingRun', '@servingMedia']); - cy.get('[data-testid="serving-selected-metrics"]') - .should('contain', 'C4') - .and('contain', '300 s'); - cy.get('[data-testid="serving-media"] video').should( - 'have.attr', - 'src', - 'https://media.test/gpu/c4/baseline/artifacts/measurement-r001-c001.mp4', - ); - cy.get('[data-testid="serving-media"] [aria-label="Clip / request"]').click(); - cy.contains('[role="option"]', 'Warmup').click(); - cy.get('[data-testid="serving-media"] video').should( - 'have.attr', - 'src', - 'https://media.test/gpu/c4/baseline/artifacts/warmup-001.mp4', - ); - cy.contains('button', 'Hardware tradeoffs').click(); - cy.get('[data-testid="video-tradeoff"] tbody tr') - .should('have.length', 3) - .first() - .find('button') - .click(); - cy.contains('button', 'Open videos and full result').click(); - cy.get('[data-testid="serving-selected-metrics"]') - .should('be.visible') - .and('contain', 'C1') - .and('contain', '120 s'); - cy.get('[data-testid="serving-media"] video').should( - 'have.attr', - 'src', - 'https://media.test/gpu/c1/baseline/artifacts/measurement-r001-c001.mp4', - ); - cy.location('search').should('contain', 'cell=c1'); - cy.get('@servingRun.all').should('have.length', 1); - cy.get('@servingMedia.all').should('have.length', 1); - cy.get('[data-testid="result-summary"]').should('not.exist'); - }); - it('shows a malformed serving matrix error instead of empty paired results', () => { - const saved = servingArtifact(); - const source = saved.sources[0]; - source.documents = source.documents.map(([path, document]) => [ - path, - path === 'serving-smoke.json' - ? { schema_version: '9.0.0', bundle_type: 'h3_serving_smoke_matrix' } - : document, - ]); - cy.window().then((win) => - win.history.replaceState( - null, - '', - `${win.location.pathname}?run=123&artifact=40&view=tradeoff`, - ), - ); - cy.intercept('GET', '/api/video-runs?run=123', { - run: run(123, 'success'), - artifacts: [saved.artifact], - }); - cy.intercept('GET', '**/api/video-runs*format=media', saved); - cy.mount( - - - , - ); - cy.contains('[role="alert"]', 'Invalid H3 serving matrix: unsupported matrix contract').should( - 'be.visible', - ); - cy.get('[data-testid="serving-results"], [data-testid="result-summary"]').should('not.exist'); - cy.get('video').should('not.exist'); - }); - it('reports expiration without downloading or substituting another artifact', () => { - cy.intercept('GET', '/api/video-runs?page=1', { runs: [run(30, 'success')], nextPage: null }); - cy.intercept('GET', '/api/video-runs?run=30', { - run: run(30, 'success'), - artifacts: [{ id: 40, name: 'h3-results-30-1', expired: true, size_in_bytes: 100 }], - }); - cy.mount( - - - , - ); - cy.contains('button', 'Browse CI runs').click(); - cy.get('[role="alert"]').should('contain', 'Expired'); - cy.get('video').should('not.exist'); - cy.get('[role="combobox"][aria-label="Result artifact"]').should( - 'contain.text', - 'h3-results-30-1', - ); - }); - it('withholds invalid generation measurements and separates warmup energy', () => { - const power = { - phases: { - measurement: { - valid: false, - status: 'invalid', - invalid_reasons: ['unbracketed'], - aggregate: { avg_power_w: 999 }, - }, - warmup: { - valid: true, - status: 'valid', - duration_seconds: 2, - aggregate: { - avg_power_w: 100, - energy_j: 200, - joules_per_valid_clip: 200, - observed_peak_power_w: 110, - }, - per_gpu: { 'GPU-fixture': { avg_power_w: 100, observed_peak_power_w: 110 } }, - }, - }, - }; - cy.mount( - - - , - ); - cy.get('[data-testid="result-power"]') - .should('contain', 'unbracketed') - .and('not.contain', '999'); - cy.get('[role="combobox"][aria-label="Power window"]').click(); - cy.contains('[role="option"]', 'Warmup').click(); - cy.get('[data-testid="result-power"]') - .should('contain', '0.2') - .and('contain', 'GPU-fixture') - .and('not.contain', 'unbracketed'); - }); -}); diff --git a/packages/app/cypress/component/video-compare.cy.tsx b/packages/app/cypress/component/video-compare.cy.tsx index c2c8fa0f2..442ab96db 100644 --- a/packages/app/cypress/component/video-compare.cy.tsx +++ b/packages/app/cypress/component/video-compare.cy.tsx @@ -76,7 +76,6 @@ describe('Video compare panel (retained fixture)', () => { clip('candidate').should('contain', 'B200').find('video').should('exist'); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'H100'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'B200'); - // Exactly three delta rows: time to video, TCO cost per video and energy per video. table() .find('tbody tr') .should(($rows) => { @@ -108,7 +107,6 @@ describe('Video compare panel (retained fixture)', () => { cy.get('[data-testid="video-compare"]') .should('contain', 'retained CI outputs') .and('not.contain', 'GPU basis'); - // Arena only: no blind mode and no case dropdown, just the two hardware selects. cy.get('[data-testid="video-compare-blind"]').should('not.exist'); cy.get('[data-testid="video-compare-controls"] [role="combobox"]').should(($boxes) => { expect([...$boxes].map((el) => el.getAttribute('aria-label'))).to.deep.equal([ diff --git a/packages/app/cypress/component/video-dashboard.cy.tsx b/packages/app/cypress/component/video-dashboard.cy.tsx index 07e035b6f..7e7da209e 100644 --- a/packages/app/cypress/component/video-dashboard.cy.tsx +++ b/packages/app/cypress/component/video-dashboard.cy.tsx @@ -152,7 +152,6 @@ describe('Video hardware dashboard (retained fixture)', () => { headers().should(($ths) => { expect([...$ths].map((el) => el.textContent?.trim())).to.deep.equal(ALL_COLUMNS); }); - // B200 C1: 77.9 s / 78.3 s, 11.53 videos/GPU-hr, 6.66 videos per $1, $0.150, $0.272 API, 301.1 kJ, 96.4 %, 3,856 W. cy.contains('[data-testid="video-points-table"] tbody tr', 'B200').should(($row) => { const cells = [...$row.find('td')].map((el) => el.textContent?.trim()); expect(cells.slice(0, 12)).to.deep.equal([ @@ -183,17 +182,17 @@ describe('Video hardware dashboard (retained fixture)', () => { .and('contain', 'B200') .and('not.contain', 'H100'); cy.location('search').should('contain', 'v_view=table'); - cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); + cy.get('[data-testid="video-history-section"]').should('not.have.attr', 'open'); }); - it('restores v_ params from the URL and opens the runs section for run deep links', () => { - mount('/video', '?v_y=kjPerVideo&v_x=p50Latency&v_tier=r&view=history'); + it('restores v_ params from the URL and opens the history section for history deep links', () => { + mount('/video', '?v_y=kjPerVideo&v_x=p50Latency&v_tier=r&history-hardware=H200'); points().should('have.length', 3); cy.get('[data-testid="video-chart-card"]') .should('contain', 'GPU-board energy per video (kJ) vs. P50 time to video (s)') .and('contain', 'Cost tier: Rent - 3 Year Commit'); kpi('h200').should('contain', '$0.485'); - cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); - cy.get('[data-testid="video-ci-runs"]').should('exist'); + cy.get('[data-testid="video-history-section"]').should('have.attr', 'open'); + cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); }); it('reprices the API list price beside the TCO cost, in the cards, the table and the URL', () => { mount(); diff --git a/packages/app/cypress/component/video-fidelity.cy.tsx b/packages/app/cypress/component/video-fidelity.cy.tsx deleted file mode 100644 index e13eab67a..000000000 --- a/packages/app/cypress/component/video-fidelity.cy.tsx +++ /dev/null @@ -1,133 +0,0 @@ -import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; -import FidelityResults from '@/components/video-benchmark/FidelityResults'; -import { fidelityFixture } from '@/components/video-benchmark/fidelity.fixture'; -import { at } from '@/components/video-benchmark/bundle'; - -// Synthetic non-playable bytes validate controls, URL binding, and accounting only. -// Actual decoded video/audio playback is verified separately using retained CI output. -const mount = (props: React.ComponentProps, locale = '/video') => - cy.mount( - - - , - ); - -describe('Native paired fidelity UI (synthetic artifacts)', () => { - it('distinguishes valid media from 18 threshold failures and changes both selected videos', () => { - cy.then(() => fidelityFixture({ pairs: 20, failedPairs: 18 })).then((fixture) => { - mount({ published: fixture.published, runId: fixture.runId }); - cy.contains('p', 'Matched valid pairs').next().should('have.text', '20'); - cy.contains('p', 'Pairs outside fidelity thresholds').next().should('have.text', '18'); - cy.contains('dt', 'Threshold calibration').next().should('have.text', 'Uncalibrated'); - cy.contains('dt', 'Release qualified').next().should('have.text', 'No'); - cy.contains('dt', 'Video/audio technical validity').next().should('have.text', 'Yes'); - for (const role of ['Baseline', 'Candidate']) - cy.get(`video[aria-label="${role}"]`) - .should('have.prop', 'controls', true) - .and('have.prop', 'muted', false) - .and('have.attr', 'preload', 'metadata'); - const initial = ['baseline', 'candidate'].map( - (role) => `report/${at(fixture.portable, 'slots', 0, role, 'artifact_path')}`, - ); - const next = ['baseline', 'candidate'].map( - (role) => `report/${at(fixture.portable, 'slots', 1, role, 'artifact_path')}`, - ); - for (const [index, role] of ['Baseline', 'Candidate'].entries()) - cy.get(`video[aria-label="${role}"]`).should( - 'have.attr', - 'src', - fixture.published.assets.find(([path]) => path === initial[index])![1].url, - ); - cy.get('[aria-label="Paired clip"]').click(); - cy.get('[role="option"]').should('have.length', 20); - cy.contains('[role="option"]', 'synthetic-case-2').click(); - for (const [index, role] of ['Baseline', 'Candidate'].entries()) - cy.get(`video[aria-label="${role}"]`).should( - 'have.attr', - 'src', - fixture.published.assets.find(([path]) => path === next[index])![1].url, - ); - cy.contains('Synthetic test prompt 2').should('be.visible'); - cy.contains('a', 'Raw comparison JSON').should( - 'have.attr', - 'href', - fixture.published.assets.find(([path]) => path === 'comparison.json')![1].downloadUrl, - ); - cy.contains('a', 'CI #101').should( - 'have.attr', - 'href', - 'https://github.com/SemiAnalysisAI/InferenceX/actions/runs/101', - ); - }); - }); - - it('rewrites report media/download links and removes scripts and external resources', () => { - cy.then(() => fidelityFixture()).then((fixture) => { - mount({ published: fixture.published, runId: fixture.runId }); - cy.get('iframe').should('not.exist'); - cy.contains('summary', 'Original report').click(); - cy.get('iframe') - .should('have.attr', 'sandbox', 'allow-same-origin allow-downloads') - .should(($iframe) => { - const doc = $iframe[0].contentDocument!; - const path = `report/${at(fixture.portable, 'slots', 0, 'baseline', 'artifact_path')}`; - expect(doc.querySelector('video')?.getAttribute('src')).to.equal( - fixture.published.assets.find(([key]) => key === path)![1].url, - ); - expect(doc.querySelector('a')?.getAttribute('href')).to.equal( - fixture.published.assets.find(([key]) => key === 'report/index.comparison.json')![1] - .url, - ); - expect(doc.querySelectorAll('script')).to.have.length(0); - expect(doc.querySelector('img')?.hasAttribute('src')).to.equal(false); - expect(doc.body.dataset.scriptRan).to.equal(undefined); - }); - cy.contains('summary', 'Original report').click(); - cy.get('iframe').should('not.exist'); - }); - }); - - it('loads a raw sealed bundle and revokes owned object URLs when unmounted', () => { - cy.window().then((win) => { - cy.spy(win.URL, 'revokeObjectURL').as('revoke'); - }); - cy.then(() => fidelityFixture()).then((fixture) => { - mount({ reader: fixture.read, runId: fixture.runId }); - cy.get('[data-testid="fidelity-results"]').should('be.visible'); - cy.get('video[aria-label="Baseline"]') - .invoke('attr', 'src') - .then((url) => { - expect(url).to.match(/^blob:/u); - cy.mount(

Unmounted synthetic reader

); - cy.get('@revoke').should('have.been.calledWith', url); - }); - }); - }); - - it('clears old evidence and reports an invalid or empty comparison', () => { - cy.then(() => fidelityFixture()).then((fixture) => { - mount({ published: fixture.published, runId: fixture.runId }); - cy.get('[data-testid="fidelity-results"]').should('be.visible'); - const invalid = structuredClone(fixture.published); - for (const [path, value] of invalid.documents) - if (['comparison.json', 'report/index.comparison.json'].includes(path)) - Object.assign(value!, { slots: [] }); - mount({ published: invalid, runId: fixture.runId }); - cy.contains('[role="alert"]', 'Could not load paired evidence').should('be.visible'); - cy.get('[data-testid="fidelity-results"]').should('not.exist'); - cy.get('video').should('not.exist'); - }); - }); - - it('renders Chinese status and original-audio controls without relabeling failures as invalid media', () => { - cy.then(() => fidelityFixture()).then((fixture) => { - mount({ published: fixture.published, runId: fixture.runId }, '/zh/video'); - cy.contains('成对视频保真度').should('be.visible'); - cy.contains('p', '超出保真度阈值的视频对').next().should('have.text', '1'); - cy.contains('dt', '视频与音频技术有效性').next().should('have.text', '是'); - cy.contains('dt', '阈值校准状态').next().should('have.text', '未校准'); - cy.get('video[aria-label="基线"]').should('have.prop', 'muted', false); - cy.get('video[aria-label="候选"]').should('have.prop', 'controls', true); - }); - }); -}); diff --git a/packages/app/cypress/component/video-results-navigation.cy.tsx b/packages/app/cypress/component/video-results-navigation.cy.tsx deleted file mode 100644 index d996c7360..000000000 --- a/packages/app/cypress/component/video-results-navigation.cy.tsx +++ /dev/null @@ -1,165 +0,0 @@ -import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; -import VideoCIRuns from '@/components/video-benchmark/VideoCIRuns'; -import { videoHistoryEntry } from '@/components/video-benchmark/history'; -import { servingArtifact, videoRun } from '../support/video-artifacts'; - -describe('Published video result navigation', () => { - const saved = servingArtifact(); - const page = { schemaVersion: 1, entries: [videoHistoryEntry(saved, null)], nextPage: null }; - beforeEach(() => { - cy.window().then((win) => - win.history.replaceState(null, '', `${win.location.pathname}?view=results`), - ); - cy.intercept('GET', '/api/video-runs?format=history&page=1', page).as('published'); - cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }).as('recent'); - cy.intercept('GET', '/api/video-runs?run=123', { - run: videoRun(123, 'success'), - artifacts: [saved.artifact], - }).as('run'); - cy.intercept('GET', '**/api/video-runs*format=media', saved).as('media'); - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - }); - it('opens published videos from History when recent Actions contain no H3 runs', () => { - cy.window().then((win) => win.history.replaceState(null, '', win.location.pathname)); - cy.mount( - - - , - ); - cy.get('[data-testid="video-history-observation"]').should('have.length', 3); - cy.contains('button', 'Videos & result').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C1'); - cy.location('search').should('contain', 'view=results').and('contain', 'run=123'); - cy.get('@recent.all').should('have.length', 0); - }); - it('shows the Chinese published empty state and scans CI only after an explicit request', () => { - cy.intercept('GET', '/api/video-runs?format=history&page=1', { - schemaVersion: 1, - entries: [], - nextPage: null, - }); - cy.mount( - - - , - ); - cy.contains('最新发布页面中没有可读取的结果').should('be.visible'); - cy.get('@recent.all').should('have.length', 0); - cy.contains('button', '浏览 CI 运行').click(); - cy.wait('@recent'); - cy.contains('本页 GitHub 历史中没有 H3 运行').should('be.visible'); - cy.contains('button', '性能历史').should('be.enabled'); - cy.location('search').should('contain', 'view=results'); - cy.contains('button', '视频与结果').click(); - cy.contains('最新发布页面中没有可读取的结果').should('be.visible'); - cy.contains('button', '浏览 CI 运行').should('be.enabled'); - }); - it('retries a failed published read and skips unreadable sources without scanning CI', () => { - let reads = 0; - cy.intercept('GET', '/api/video-runs?format=history&page=1', (req) => { - if (++reads === 1) req.reply({ statusCode: 503, body: { error: 'Unavailable' } }); - else - req.reply({ - ...page, - entries: [ - { ...page.entries[0], error: 'Unreadable index', sources: [] }, - ...page.entries, - ], - }); - }); - cy.mount( - - - , - ); - cy.get('[role="alert"]').should('contain', 'Could not load published results'); - cy.contains('button', 'Retry').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible'); - cy.get('@recent.all').should('have.length', 0); - }); - it('does not select a stale published response after returning to History', () => { - let release: (() => void) | undefined; - cy.window().then((win) => { - const fetch = win.fetch.bind(win); - let first = true; - cy.stub(win, 'fetch').callsFake((input: RequestInfo | URL, init?: RequestInit) => { - if (String(input).includes('format=history') && first) { - first = false; - return new Promise((resolve) => { - release = () => resolve(Response.json(page)); - }); - } - return fetch(input, init); - }); - }); - cy.mount( - - - , - ); - cy.wrap(null).should(() => expect(release).to.be.a('function')); - cy.contains('button', 'Performance history').click(); - cy.get('[data-testid="video-history-observation"]').should('have.length', 3); - cy.then(() => release?.()); - cy.get('@run.all').should('have.length', 0); - cy.location('search').should('contain', 'view=history').and('not.contain', 'run='); - cy.contains('button', 'Performance history').should('have.attr', 'aria-pressed', 'true'); - }); - it('retries the shared C4 selection after a refresh metadata failure', () => { - cy.window().then((win) => - win.history.replaceState( - null, - '', - `${win.location.pathname}?view=results&run=123&artifact=40&source=123&cell=c4`, - ), - ); - let reads = 0; - cy.intercept('GET', '/api/video-runs?run=123', (req) => { - if (++reads === 2) req.reply({ statusCode: 503, body: { error: 'CI run unavailable' } }); - else req.reply({ run: videoRun(123, 'success'), artifacts: [saved.artifact] }); - }); - cy.mount( - - - , - ); - cy.get('[data-testid="serving-selected-metrics"]').should('contain', 'C4'); - cy.contains('button', 'Refresh').click(); - cy.get('[role="alert"]').should('contain', 'CI run unavailable'); - cy.contains('button', 'Retry').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C4'); - cy.get('@published.all').should('have.length', 0); - cy.location('search').should('contain', 'cell=c4'); - }); - it('resumes the selected artifact when media loading was cancelled by returning to History', () => { - let release: (() => void) | undefined; - cy.window().then((win) => { - const fetch = win.fetch.bind(win); - let first = true; - cy.stub(win, 'fetch').callsFake((input: RequestInfo | URL, init?: RequestInit) => { - if (String(input).includes('format=media') && first) { - first = false; - return new Promise((resolve) => { - release = () => resolve(Response.json(saved)); - }); - } - return fetch(input, init); - }); - }); - cy.mount( - - - , - ); - cy.wait('@run'); - cy.get('[aria-label="CI run"]').should('contain', '#123'); - cy.contains('button', 'Performance history').click(); - cy.then(() => release?.()); - cy.get('[data-testid="video-history-observation"]').should('have.length', 3); - cy.contains('button', 'Performance history').should('have.attr', 'aria-pressed', 'true'); - cy.contains('button', 'Videos & result').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible'); - cy.get('@media.all').should('have.length', 1); - cy.get('@recent.all').should('have.length', 0); - }); -}); diff --git a/packages/app/cypress/component/video-serving-results.cy.tsx b/packages/app/cypress/component/video-serving-results.cy.tsx deleted file mode 100644 index f62b313d8..000000000 --- a/packages/app/cypress/component/video-serving-results.cy.tsx +++ /dev/null @@ -1,399 +0,0 @@ -import { useState } from 'react'; -import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; -import ServingResults from '@/components/video-benchmark/ServingResults'; -import type { ServingCell } from '@/components/video-benchmark/serving'; -import { at, entries, rows, type Bundle } from '@/components/video-benchmark/bundle'; - -// Synthetic fixtures exercise the serving contract; they are never benchmark results. -const cells: ServingCell[] = [1, 2, 4].map((concurrency) => { - const id = `c${concurrency}`; - const request = (phase: string, slot: string) => ({ - phase, - slot_id: slot, - artifact_path: `artifacts/${slot}.mp4`, - prompt: 'Synthetic test prompt only', - seed: 11, - submit_to_media_seconds: 120 * concurrency, - media_validation_seconds: 0.6, - media: { - valid: true, - video: { duration_seconds: 4.4583, width: 1344, height: 768, frame_count: 107, fps: 24 }, - audio: { sample_rate_hz: 32000, channels: 2 }, - checks: [{ name: 'media.decode', status: 'passed' }], - }, - }); - return { - id, - concurrency, - runPath: `gpu/${id}/baseline/run.json`, - jobPath: `gpu/${id}/gpu-job.json`, - powerPath: `gpu/${id}/power.json`, - specPath: `gpu/${id}/spec.json`, - cell: { - status: 'complete', - verified: true, - completion: { - valid: 4, - scheduled: 4, - completed: 4, - failed: 0, - not_started: 0, - unfinished: 0, - }, - metrics: { - client_ready_p50_seconds: 120 * concurrency, - valid_clips_per_second: 1 / 120, - measurement: { wall_seconds: 480 }, - serving: { - client_ready_latency_seconds: { sample_count: 4, p90: null }, - valid_video_seconds_per_second: 4.4583 / 120, - peak_client_in_flight: concurrency, - }, - }, - }, - run: { - records: [request('warmup', 'warmup-001'), request('measurement', 'measurement-r001-c001')], - plan: { generation: { duration_seconds: 4, frame_count: 107, num_inference_steps: 50 } }, - configuration: { - model_id: 'Synthetic model', - model_revision: 'synthetic-model-revision', - runtime_revision: 'synthetic-runtime-revision', - }, - }, - job: { - roles: { - baseline: { - power_configuration_before: { - status: 'recorded', - observed_at: '2026-09-09T01:00:00Z', - gpus: ['GPU-test', 'GPU-test-2'].map((uuid) => ({ - uuid, - configured_limit_w: 700, - enforced_limit_w: 700, - })), - }, - power_configuration_after: { - status: 'recorded', - observed_at: '2026-09-09T02:00:00Z', - gpus: ['GPU-test', 'GPU-test-2'].map((uuid) => ({ - uuid, - configured_limit_w: 700, - enforced_limit_w: 700, - })), - }, - telemetry_summary: { - gpu_identity: [{ name: 'Synthetic GPU', uuid: 'GPU-test' }], - measurement_observed_memory_peak_mib_by_gpu: { 'GPU-test': 100000 }, - }, - }, - }, - }, - spec: { gpu_uuids: ['GPU-test', 'GPU-test-2'], server: { ulysses_degree: 2, tp_size: 1 } }, - power: { - semantics: { power_unit: 'W' }, - phases: { - measurement: { - valid: true, - duration_seconds: 479.9, - aggregate: { - avg_power_w: 1374, - observed_peak_power_w: 1394, - energy_j: 656721, - joules_per_valid_clip: 164180, - }, - per_gpu: { - 'GPU-test': { avg_power_w: 687, observed_peak_power_w: 697 }, - 'GPU-test-2': { avg_power_w: 687, observed_peak_power_w: 697 }, - }, - }, - startup: { - valid: false, - invalid_reasons: ['Synthetic missing telemetry'], - aggregate: { avg_power_w: 999999 }, - }, - }, - }, - }; -}); -const bundle: Bundle = { - manifest: { - run_id: '123', - git_commit: 'synthetic-backend-sha', - resources: { requested: { gpus: 8 } }, - }, - ci: { slurm_job: { AllocTRES: 'cpu=32,mem=512G,gres/gpu=8' } }, - result: null, - job: null, - report: null, - comparison: null, - documents: new Map(), - files: new Map(), - checksums: new Map(), - manifestSha256: 'synthetic-manifest-sha', -}; -const urls = new Map( - cells.flatMap((cell) => - ['measurement-r001-c001', 'warmup-001'].map((slot): [string, string] => [ - `gpu/${cell.id}/baseline/artifacts/${slot}.mp4`, - `https://media.example.test/${cell.id}/${slot}.mp4`, - ]), - ), -); - -function mount( - props: Partial> = {}, - locale = '/video', -) { - cy.mount( - - - , - ); -} - -function LinkedSelection() { - const [cell, setCell] = useState('c1'); - return ( - - - - - ); -} - -describe('H3 serving results (synthetic fixtures)', () => { - it('follows chart-driven cell changes while the result remains mounted', () => { - cy.mount(); - cy.contains('button', 'C1').should('have.attr', 'aria-pressed', 'true'); - cy.get('[aria-label="Clip / request"]').click(); - cy.contains('[role="option"]', 'Warmup').click(); - cy.get('video').should('have.attr', 'src', 'https://media.example.test/c1/warmup-001.mp4'); - cy.contains('button', 'Open C4 from chart').click(); - cy.contains('button', /^C4$/u).should('have.attr', 'aria-pressed', 'true'); - cy.get('video').should( - 'have.attr', - 'src', - 'https://media.example.test/c4/measurement-r001-c001.mp4', - ); - cy.contains('button', 'C2').click(); - cy.contains('button', 'C2').should('have.attr', 'aria-pressed', 'true'); - }); - it('renders a parent-created blob in the report while keeping scripts disabled', () => { - cy.window().then((win) => { - const blobUrl = win.URL.createObjectURL( - new Blob( - [ - '', - ], - { type: 'image/svg+xml' }, - ), - ); - mount({ - html: `Synthetic blob imageDownload synthetic blob`, - }); - cy.get('iframe').should('not.exist'); - cy.contains('summary', 'Original report').click(); - cy.get('iframe') - .should('have.attr', 'sandbox', 'allow-same-origin allow-downloads') - .should(($frame) => { - const doc = $frame[0].contentDocument; - expect(doc?.querySelector('img')?.naturalWidth).to.equal(2); - expect(doc?.querySelector('a')?.getAttribute('href')).to.equal(blobUrl); - expect(doc?.body.dataset.scriptRan).to.equal(undefined); - }) - .then(() => win.URL.revokeObjectURL(blobUrl)); - cy.contains('summary', 'Original report').click(); - cy.get('iframe').should('not.exist'); - }); - }); - - it('shows each concurrency without manufacturing a candidate and separates participating and allocated GPU counts', () => { - const change = cy.stub().as('change'); - mount({ onCellChange: change }); - cy.get('[data-testid="serving-matrix"] tbody tr').should('have.length', 3); - cy.get('[data-testid="serving-matrix"]').within(() => { - cy.contains('th', 'Energy / valid clip (kJ/clip)').should('be.visible'); - cy.contains('td', '164.18').should('be.visible'); - cy.contains('th', 'Mean / enforced limit (%)').should('be.visible'); - cy.contains('td', '98.14').should('be.visible'); - }); - cy.get('[data-testid="serving-selected-metrics"]').within(() => { - cy.contains('120 s').should('be.visible'); - cy.contains('p', '15').should('be.visible'); - cy.contains('dt', 'P90 delivery latency').next().should('have.text', 'Unavailable'); - }); - cy.get('[aria-label="GPU normalization"]').click(); - cy.contains('[role="option"]', 'All allocated GPUs').click(); - cy.get('[data-testid="serving-selected-metrics"]').contains('p', '3.75').should('be.visible'); - cy.get('video') - .should('have.attr', 'src', 'https://media.example.test/c1/measurement-r001-c001.mp4') - .and('have.prop', 'muted', false); - cy.contains('button', /^C4$/u).click(); - cy.get('@change').should('have.been.calledWith', 'c4'); - cy.get('[data-testid="serving-selected-metrics"]').should('contain', '480 s'); - cy.get('video').should( - 'have.attr', - 'src', - 'https://media.example.test/c4/measurement-r001-c001.mp4', - ); - cy.get('[aria-label="Clip / request"]').click(); - cy.contains('[role="option"]', 'Warmup').click(); - cy.get('video').should('have.attr', 'src', 'https://media.example.test/c4/warmup-001.mp4'); - cy.get('[data-testid="serving-selected-metrics"]').should('contain', '4 / 4'); - cy.contains('a', 'Download original MP4').should( - 'have.attr', - 'href', - 'https://media.example.test/c4/warmup-001.mp4', - ); - cy.contains('Baseline').should('not.exist'); - cy.contains('Candidate').should('not.exist'); - }); - - it('shows server stages separately from delivery and retains partial coverage', () => { - const distribution = { - values: [1, 2, 3, 4], - sample_count: 4, - valid_clip_count: 4, - missing_count: 0, - p50: 2.5, - p90: null, - p95: null, - }; - const original = cells[0]; - const record = rows(at(original.run, 'records'))[1]; - const timed = { - ...original, - run: { - ...Object.fromEntries(entries(original.run)), - summary: { valid: 4 }, - serving: { - queue_delay_seconds: distribution, - server_ready_latency_seconds: { - ...distribution, - values: [120, 121], - sample_count: 2, - missing_count: 2, - p50: 999999, - }, - observed_batch_sizes: [1, 1, 1, 1], - observed_replica_ids: [0], - }, - records: [ - { - ...Object.fromEntries(entries(record)), - job_id: 'synthetic-request', - server_timings: { - schema_version: '1.0.0', - status: 'complete', - request_id: 'synthetic-request', - instance_id: 'synthetic-instance', - clock_id: 'synthetic-clock', - clock: 'time.monotonic_ns; same Linux boot and time namespace; nanoseconds', - observed_batch_size: 1, - replica_id: 0, - server_ready_latency_seconds: 119, - prequeue_seconds: 1, - queue_delay_seconds: 0, - execution_seconds: 117, - postprocess_seconds: 1, - }, - }, - ], - }, - }; - mount({ cells: [timed] }); - cy.get('[data-testid="serving-server-timing"]').within(() => { - cy.contains('tr', 'Queue delay').should('contain', '2.5').and('contain', '4 / 4 / 0'); - cy.contains('tr', 'Received → media ready') - .should('contain', 'Unavailable') - .and('contain', '2 / 4 / 2') - .and('not.contain', '999,999'); - cy.contains('dt', 'Observed batch sizes').next().should('have.text', '1'); - cy.contains('dt', 'Observed replica IDs').next().should('have.text', '0'); - }); - cy.get('[data-testid="request-server-timing"]').within(() => { - cy.get('summary').click(); - cy.contains('dt', 'Queue delay').next().should('have.text', '0 s'); - cy.contains('dt', 'Received → media ready').next().should('have.text', '119 s'); - cy.contains('time.monotonic_ns').should('be.visible'); - }); - cy.contains('dt', 'This request: submission → downloaded media') - .next() - .should('have.text', '120 s'); - mount({ cells: [timed] }, '/zh/video'); - cy.get('[data-testid="serving-server-timing"]').within(() => { - cy.contains('tr', '排队时长').should('contain', '2.5'); - cy.contains('tr', '接收请求 → 媒体就绪') - .should('contain', '无数据') - .and('contain', '2 / 4 / 2'); - }); - }); - - it('distinguishes requested and decoded duration and withholds invalid power', () => { - mount(); - cy.contains('dt', 'Requested duration (s)').next().should('have.text', '4'); - cy.contains('dt', 'Decoded video duration (s)').next().should('have.text', '4.4583'); - cy.get('[data-testid="serving-power"]').within(() => { - cy.contains('dt', 'Energy / valid clip (kJ/clip)').next().should('have.text', '164.18'); - cy.get('[aria-label="Power window"]').click(); - }); - cy.contains('[role="option"]', 'Startup').click(); - cy.get('[data-testid="serving-power"]').within(() => { - cy.contains('dt', 'Aggregate mean (W)').next().should('have.text', 'Unavailable'); - cy.contains('dt', 'Mean / enforced limit (%)').next().should('have.text', 'Unavailable'); - cy.contains('Synthetic missing telemetry').should('be.visible'); - cy.contains('999,999').should('not.exist'); - }); - }); - - it('withholds headline metrics and power for an unverified configuration', () => { - mount({ - cells: [ - { - ...cells[0], - cell: { - verified: false, - status: 'failed', - metrics: { client_ready_p50_seconds: 999999, valid_clips_per_second: 999999 }, - }, - }, - ], - }); - cy.get('[data-testid="serving-selected-metrics"]').should('not.contain', '999,999'); - cy.get('[data-testid="serving-matrix"]').should('not.contain', '999,999'); - cy.contains('dt', 'Measurement verified').next().should('have.text', 'No'); - cy.get('[data-testid="serving-power"]').within(() => { - cy.contains('dt', 'Aggregate mean (W)').next().should('have.text', 'Unavailable'); - cy.contains('dt', 'Mean / enforced limit (%)').next().should('have.text', 'Unavailable'); - }); - }); - - it('supports Chinese copy and missing request, media, and allocation data', () => { - mount({ urls: new Map(), bundle: { ...bundle, ci: null } }, '/zh/video'); - cy.contains('并发服务测试结果').should('be.visible'); - cy.contains('此请求暂无可用媒体。').should('be.visible'); - cy.get('video').should('not.exist'); - cy.get('[data-testid="serving-server-timing"]').should('contain', '无数据'); - cy.contains('p', '有效视频 / 参与计算 GPU 小时').next().should('have.text', '15'); - cy.get('[aria-label="GPU 归一化口径"]').click(); - cy.contains('[role="option"]', '所有已分配 GPU').click(); - cy.contains('p', '有效视频 / 已分配 GPU 小时').next().should('have.text', '无数据'); - mount({ cells: [{ ...cells[0], run: null, job: null, spec: null, power: null }] }); - cy.contains('No request records are available for this configuration.').should('be.visible'); - }); -}); diff --git a/packages/app/cypress/component/video-tradeoff.cy.tsx b/packages/app/cypress/component/video-tradeoff.cy.tsx deleted file mode 100644 index 1025b50d9..000000000 --- a/packages/app/cypress/component/video-tradeoff.cy.tsx +++ /dev/null @@ -1,290 +0,0 @@ -import { PathnameContext } from 'next/dist/shared/lib/hooks-client-context.shared-runtime'; -import VideoTradeoff from '@/components/video-benchmark/VideoTradeoff'; -import type { TradeoffRun } from '@/components/video-benchmark/tradeoff'; -import { at, entries, rows, type Json } from '@/components/video-benchmark/bundle'; -import { servingFixture } from '@/components/video-benchmark/serving.fixture'; - -const role = { - metrics: { - status: 'valid', - measurement: { wall_seconds: 3600, concurrency: 2 }, - completion: { valid: 10, completed: 10, scheduled: 11, failed: 1 }, - }, - records: Array.from({ length: 10 }, (_, i) => ({ - phase: 'measurement', - status: 'succeeded', - submit_to_media_seconds: i + 1, - media: { valid: true, video: { duration_seconds: 8 } }, - })), -}; -const run: TradeoffRun = { - exportRun: '20', - artifact: '30', - bundle: { - manifest: { run_id: '10' }, - manifestSha256: 'synthetic-only', - result: { - schema_version: '1.0.0', - bundle_type: 'h3_benchmark_result', - workload: { - plan: { - model_id: 'synthetic', - model_revision: 'synthetic-revision', - generation: { - width: 1280, - height: 720, - fps: 24, - duration_seconds: 8, - num_inference_steps: 50, - }, - cases: [{ prompt: 'Synthetic test only', seed: 1 }], - }, - }, - hardware: { - reserved_gpu_count: 8, - selected_gpu_count: 4, - devices: [{ name: 'Synthetic GPU' }], - }, - roles: { baseline: role, candidate: role }, - }, - }, -}; - -describe('Video tradeoff chart (synthetic fixtures)', () => { - it('plots participating GPU efficiency by default and requires assumptions for the cost axis', () => { - const open = cy.stub().as('open'); - cy.mount( - - - , - ); - cy.get('[role="combobox"][aria-label="Efficiency axis · higher is better"]').should( - 'contain', - 'Valid clips / participating GPU-hour', - ); - cy.get('[data-testid="video-tradeoff-chart"] circle.point').should('have.length', 2); - cy.contains('td', '2.5').should('be.visible'); - cy.get('[role="combobox"][aria-label="Efficiency axis · higher is better"]').click(); - cy.contains('[role="option"]', 'Valid clips / allocated GPU-hour').click(); - cy.contains('td', '1.25').should('be.visible'); - cy.contains('dd', '8 / 4').should('be.visible'); - cy.get('[role="combobox"][aria-label="Efficiency axis · higher is better"]').click(); - cy.contains('[role="option"]', 'Valid clips / USD').click(); - cy.contains('No points qualify for these axes yet.').should('be.visible'); - cy.get('input[aria-label="Whole deployment cost (USD/hour)"]').type('2'); - cy.get('circle.point').should('not.exist'); - cy.get('input[aria-label="Cost source and included items"]').type( - 'Synthetic complete-deployment cost', - ); - cy.get('input[aria-label="Cost as of"]').type('2026-09-09'); - cy.get('circle.point').should('have.length', 1); - cy.contains('td', '5').should('be.visible'); - cy.contains('button', 'Open videos and full result').click(); - cy.get('@open').should('have.been.calledOnce'); - }); - it('still withholds P90 for single-request results until median diagnostics are selected', () => { - const single = structuredClone(run); - const result = single.bundle.result as { roles: Record }; - for (const point of Object.values(result.roles)) { - point.records = point.records.slice(0, 1); - point.metrics.completion = { valid: 1, completed: 1, scheduled: 1, failed: 0 }; - } - cy.mount( - - undefined} /> - , - ); - cy.contains('No points qualify for these axes yet.').should('be.visible'); - cy.get('circle.point').should('not.exist'); - cy.contains('button', 'Inspect available serial results').click(); - cy.get('[role="combobox"][aria-label="Latency axis · lower is better"]').should( - 'contain', - 'Median client-ready latency (s)', - ); - cy.get('circle.point').should('have.length', 2); - }); - it('shows an empty state without inventing fixture results', () => { - cy.mount( - - undefined} /> - , - ); - cy.contains('Open a CI result').should('be.visible'); - cy.get('circle.point').should('not.exist'); - }); - it('distinguishes groups with the same short label and exposes their full workload', () => { - const other = structuredClone(run); - other.bundle.manifestSha256 = 'different-synthetic-workload'; - const result = other.bundle.result as { - workload: { plan: { generation: Record } }; - }; - result.workload.plan.generation.guidance_scale = 4; - cy.mount( - - undefined} /> - , - ); - cy.get('[role="combobox"][aria-label="Matched workload"]').click(); - cy.get('[role="option"]').should('have.length', 2); - cy.contains('[role="option"]', '#1').should('be.visible'); - cy.contains('[role="option"]', '#2').click(); - cy.contains('summary', 'Matched workload').click(); - cy.get('[data-testid="tradeoff-detail"] pre').first().should('contain', '"guidance_scale": 4'); - }); -}); - -const matrix = (): TradeoffRun => ({ - exportRun: '123', - artifact: '456', - bundle: { ...servingFixture(), result: null, manifestSha256: 'synthetic-serving' }, -}); - -describe('Video serving matrix chart (synthetic contract fixture)', () => { - it('shows server coverage in point details without changing the client latency axis', () => { - const timed = matrix(); - const documents = timed.bundle.documents!; - const runPath = 'gpu/c1/baseline/run.json'; - const source = documents.get(runPath)!; - const serving = { - ...Object.fromEntries(entries(at(source, 'serving'))), - observed_batch_sizes: [1, 1, 1, 1], - observed_replica_ids: [0], - queue_delay_seconds: { - values: [0, 1, 2, 3], - sample_count: 4, - valid_clip_count: 4, - missing_count: 0, - p50: 1.5, - p90: null, - p95: null, - }, - }; - documents.set(runPath, { ...Object.fromEntries(entries(source)), serving }); - const matrixDoc = documents.get('serving-smoke.json')!; - documents.set('serving-smoke.json', { - ...Object.fromEntries(entries(matrixDoc)), - cells: rows(at(matrixDoc, 'cells')).map((cell, index) => - index - ? cell - : { - ...Object.fromEntries(entries(cell)), - metrics: { ...Object.fromEntries(entries(at(cell, 'metrics'))), serving }, - }, - ), - }); - cy.mount( - - undefined} /> - , - ); - cy.contains('dt', 'Observed batch sizes').next().should('have.text', '1'); - cy.contains('dt', 'Observed replica IDs').next().should('have.text', '0'); - cy.get('[data-testid="tradeoff-server-timing"]').within(() => { - cy.get('summary').click(); - cy.contains('dt', 'Queue delay') - .next() - .should('contain', '1.5 / — / —') - .and('contain', '4 / 4 / 0'); - }); - cy.get('[aria-label="Latency axis · lower is better"]').should( - 'contain', - 'Median client-ready latency (s)', - ); - cy.get('circle.point').should('have.length', 3); - }); - - it('defaults to P90 when individual serving cells have enough samples', () => { - const formal = matrix(); - formal.bundle = { - ...servingFixture('123', 'NVIDIA H200', 20), - result: null, - manifestSha256: 'synthetic-formal', - }; - cy.mount( - - undefined} /> - , - ); - cy.get('[role="combobox"][aria-label="Latency axis · lower is better"]').should( - 'contain', - 'P90 client-ready latency (s)', - ); - cy.get('circle.point').should('have.length', 3); - cy.get('tbody tr') - .should('have.length', 3) - .each((row) => { - cy.wrap(row).should('contain', '20 / 20 / 20'); - }); - }); - it('shows three labeled cell medians and opens the selected concurrency result', () => { - const open = cy.stub().as('openCell'); - cy.mount( - - - , - ); - cy.get('[role="combobox"][aria-label="Latency axis · lower is better"]').should( - 'contain', - 'Median client-ready latency (s)', - ); - cy.get('circle.point').should('have.length', 3); - cy.get('path.line-path') - .should('have.length', 1) - .invoke('attr', 'd') - .should('match', /^M[^L]+L[^L]+L[^L]+$/u); - cy.contains('Lines connect measured concurrency settings').should('be.visible'); - cy.get('path.line-path') - .invoke('attr', 'd') - .then((before) => { - cy.get('[data-testid="video-tradeoff-chart"] svg').trigger('wheel', { - deltaY: -200, - shiftKey: true, - clientX: 400, - clientY: 300, - force: true, - }); - cy.get('path.line-path').invoke('attr', 'd').should('not.equal', before); - }); - cy.get('text.serving-point-label') - .should('have.length', 3) - .then((labels) => { - expect([...labels].map((label) => label.textContent)).to.deep.equal(['C1', 'C2', 'C4']); - }); - cy.contains('Closed-loop smoke · serving capacity not qualified').should('be.visible'); - cy.get('tbody tr') - .should('have.length', 3) - .each((row) => { - cy.wrap(row).should('contain', '4 / 4 / 4').and('contain', '15'); - }); - cy.get('[role="combobox"][aria-label="Latency axis · lower is better"]').click(); - cy.contains('[role="option"]', 'P90 client-ready latency (s)').click(); - cy.get('circle.point').should('not.exist'); - cy.get('path.line-path').should('not.exist'); - cy.contains('No points qualify for these axes yet.').should('be.visible'); - cy.contains('button', 'Show cell medians').click(); - cy.get('circle.point').should('have.length', 3); - cy.get('path.line-path').should('have.length', 1); - cy.contains('tbody button', 'Client concurrency 4').click(); - cy.contains('button', 'Open videos and full result').click(); - cy.get('@openCell') - .should('have.been.calledOnce') - .its('firstCall.args.0.cellId') - .should('equal', 'c4'); - }); - it('shows the Chinese serving labels with per-cell medians', () => { - cy.mount( - - undefined} /> - , - ); - cy.get('[role="combobox"][aria-label="延迟轴 · 越低越好"]').should( - 'contain', - '客户端就绪延迟中位数(秒)', - ); - cy.get('circle.point').should('have.length', 3); - cy.get('path.line-path').should('have.length', 1); - cy.contains('各并发数的实测结果').should('be.visible'); - cy.contains('客户端并发数 2').should('be.visible'); - cy.contains('闭环冒烟测试 · 服务容量未经验证').should('be.visible'); - }); -}); diff --git a/packages/app/cypress/e2e/navigation.cy.ts b/packages/app/cypress/e2e/navigation.cy.ts index fbe8cf70e..c2b8a42b1 100644 --- a/packages/app/cypress/e2e/navigation.cy.ts +++ b/packages/app/cypress/e2e/navigation.cy.ts @@ -1,7 +1,7 @@ import { SUPPLEMENTAL_BENCHMARK_ROWS } from '../../src/lib/supplemental-benchmarks'; import { OVERLAY_RUN_ID, OVERLAY_RUN_URL } from '../support/overlay-fixtures'; import { videoHistoryEntry } from '../../src/components/video-benchmark/history'; -import { servingArtifact, videoRun } from '../support/video-artifacts'; +import { servingArtifact } from '../support/video-artifacts'; const datum = (el: Element) => (el as Element & { __data__: { x: number; y: number } }).__data__; @@ -373,16 +373,15 @@ describe('TPUv7 results', { testIsolation: true }, () => { }); }); -describe('H3 video artifact viewer', () => { +describe('H3 video performance history', () => { beforeEach(() => { - cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }); cy.intercept('GET', '/api/video-runs?format=history&page=*', { schemaVersion: 1, entries: [], nextPage: null, }); }); - it('opens old published entries by default and restores history filters after reload', () => { + it('opens published history behind the dashboard and restores its filters after reload', () => { const entry = videoHistoryEntry(servingArtifact(), '2026-09-09T00:00:00Z'); cy.intercept('GET', '/api/video-runs?format=history&page=1', { schemaVersion: 1, @@ -391,8 +390,8 @@ describe('H3 video artifact viewer', () => { }).as('history'); cy.visit('/video'); cy.wait('@history'); - // The hardware dashboard leads; the run/history views live in a collapsible section. - cy.get('[data-testid="video-runs-section"] summary').click(); + cy.get('[data-testid="video-history"]').should('not.exist'); + cy.get('[data-testid="video-history-section"] summary').click(); cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); cy.get('[data-testid="video-history-observation"]').should('have.length', 3); cy.get('[aria-label="Client concurrency"]').click(); @@ -407,117 +406,13 @@ describe('H3 video artifact viewer', () => { .and('contain', '240 / —'); cy.location('search').should('contain', 'history-concurrency=2'); }); - it('opens the first published result directly and preserves the selection on refresh and reload', () => { - const saved = servingArtifact(); - cy.intercept('GET', '/api/video-runs?format=history&page=1', { - schemaVersion: 1, - entries: [videoHistoryEntry(saved, null)], - nextPage: null, - }).as('published'); - cy.intercept('GET', '/api/video-runs?page=*', { runs: [], nextPage: null }).as('recent'); - cy.intercept('GET', '/api/video-runs?run=123', { - run: videoRun(123, 'success'), - artifacts: [saved.artifact], - }).as('run'); - cy.intercept('GET', '**/api/video-runs*format=media', saved); - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - cy.visit('/video?view=results'); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C1'); - cy.contains('[data-testid="serving-matrix"] button', 'C4').click(); - cy.get('@published.all').then((requests) => { - const initialReads = requests.length; - cy.contains('button', 'Refresh').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C4'); - cy.get('@published.all').should('have.length', initialReads); - }); - cy.location('search').should('contain', 'view=results').and('contain', 'cell=c4'); - cy.reload(); - cy.get('[data-testid="serving-selected-metrics"]').should('be.visible').and('contain', 'C4'); - cy.contains('button', 'Videos & result').should('have.attr', 'aria-pressed', 'true'); - cy.get('@recent.all').should('have.length', 0); - }); - it('shares a comparison built by opening two CI runs through the page', () => { - for (const saved of [ - servingArtifact(123, 40, 'NVIDIA H200'), - servingArtifact(456, 41, 'NVIDIA B200'), - ]) { - cy.intercept('GET', `/api/video-runs?run=${saved.runId}`, { - run: videoRun(Number(saved.runId), 'success'), - artifacts: [saved.artifact], - }); - cy.intercept( - 'GET', - `/api/video-runs?run=${saved.runId}&artifact=${saved.artifact.id}&format=media`, - saved, - ); - } - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - cy.visit('/video?view=tradeoff&run=123&artifact=40'); - cy.get('[data-testid="video-tradeoff"] tbody tr').should('have.length', 3); - cy.contains('summary', 'Run details and artifact selection').click(); - cy.get('[data-testid="video-ci-runs"] form input').type('456'); - cy.get('[data-testid="video-ci-runs"] form').submit(); - cy.get('[data-testid="video-tradeoff"] tbody tr').should('have.length', 6); - cy.location().should((location) => { - const params = new URLSearchParams(location.search); - expect(params.get('run')).to.equal('456'); - expect(params.get('compare')).to.equal('123.40,456.41'); - }); - cy.reload(); - cy.get('[data-testid="video-tradeoff"] tbody tr').should('have.length', 6); - cy.get('[data-testid="video-tradeoff"]') - .should('contain', 'NVIDIA H200') - .and('contain', 'NVIDIA B200'); - }); - it('restores a cross-hardware comparison on reload and retains good results when one artifact fails', () => { - const h200 = servingArtifact(123, 40, 'NVIDIA H200'); - const b200 = servingArtifact(456, 41, 'NVIDIA B200'); - for (const saved of [h200, b200]) { - cy.intercept('GET', `/api/video-runs?run=${saved.runId}`, { - run: videoRun(Number(saved.runId), 'success'), - artifacts: [saved.artifact], - }); - cy.intercept( - 'GET', - `/api/video-runs?run=${saved.runId}&artifact=${saved.artifact.id}&format=media`, - saved, - ); - } - cy.intercept('GET', '/api/video-runs?run=789&artifact=42&format=media', { - statusCode: 503, - body: { error: 'Synthetic unavailable artifact' }, - }); - cy.intercept('GET', 'https://media.test/**', { statusCode: 204 }); - cy.visit('/video?view=tradeoff&run=123&artifact=40&source=123&compare=123.40,456.41,789.42'); - cy.get('[data-testid="video-tradeoff"] tbody tr').should('have.length', 6); - cy.get('[data-testid="tradeoff-detail"]').should('contain', 'NVIDIA H200'); - cy.get('[data-testid="video-tradeoff"]') - .should('contain', 'NVIDIA H200') - .and('contain', 'NVIDIA B200'); - cy.get('[role="alert"]').should('contain', 'Some comparison results could not be loaded'); - cy.reload(); - cy.get('[data-testid="video-tradeoff"] tbody tr').should('have.length', 6); - cy.get('[data-testid="tradeoff-detail"]').should('contain', 'NVIDIA H200'); - cy.contains( - '[data-testid="video-tradeoff"] tbody button', - /NVIDIA B200.*Client concurrency 4/, - ).click(); - cy.contains('button', 'Open videos and full result').click(); - cy.get('[data-testid="serving-selected-metrics"]').should('contain', 'C4'); - cy.location('search').should('include', 'run=456').and('include', 'compare='); - cy.get('[data-testid="serving-media"] video') - .should('have.attr', 'src') - .and('include', '/gpu/c4/'); - }); - it('uses the shared unlock for navigation and keeps an empty viewer free of sample results', () => { + it('keeps the hidden Video tab behind the shared unlock and noindex', () => { cy.viewport(1440, 1000); - cy.visit('/video?view=results', { + cy.visit('/video', { onBeforeLoad(win) { win.localStorage.removeItem('inferencex-feature-gate'); }, }); - cy.contains('button', 'Browse CI runs').click(); - cy.get('[data-testid="video-ci-runs"]').should('contain', 'No H3 runs in this page'); cy.get('video[data-role]').should('not.exist'); cy.get('[data-testid="tab-trigger-hidden"]').should('not.exist'); cy.get('body').type('{upArrow}{upArrow}{downArrow}{downArrow}'); @@ -525,17 +420,10 @@ describe('H3 video artifact viewer', () => { cy.contains('a', 'Video').should('have.attr', 'href', '/video'); cy.get('head meta[name="robots"]').should('have.attr', 'content', 'noindex, nofollow'); }); - it('shows a recoverable load error and the Chinese empty state', () => { - cy.visit('/video?view=results'); - cy.contains('summary', 'Local artifact tools').click(); - cy.get('[data-testid="video-benchmark"]').contains('summary', 'Manifest URL').click(); - cy.get('input[aria-label="Manifest URL"]').type('https://example.com/wrong.json'); - cy.contains('button', 'Load manifest').click(); - cy.get('[role="alert"]').should('contain', 'Could not load this bundle'); - cy.get('video[data-role]').should('not.exist'); - cy.visit('/zh/video?view=results'); - cy.contains('button', '浏览 CI 运行').click(); - cy.get('[data-testid="video-ci-runs"]').should('contain', '本页 GitHub 历史中没有 H3 运行'); + it('renders the Chinese history section for a history deep link', () => { + cy.visit('/zh/video?history-hardware=H200'); + cy.get('[data-testid="video-history-section"]').should('have.attr', 'open'); + cy.get('[data-testid="video-history"] h1').should('contain', '性能历史'); cy.get('head meta[name="robots"]').should('have.attr', 'content', 'noindex, nofollow'); }); }); diff --git a/packages/app/cypress/e2e/video-dashboard.cy.ts b/packages/app/cypress/e2e/video-dashboard.cy.ts index 8cc60af1b..5fc3cba80 100644 --- a/packages/app/cypress/e2e/video-dashboard.cy.ts +++ b/packages/app/cypress/e2e/video-dashboard.cy.ts @@ -76,8 +76,8 @@ describe('Video hardware dashboard (E2E fixtures)', () => { cy.get('[data-testid="video-kpi-card"][data-hardware="h100"]') .should('contain', '4 of 8 GPUs') .and('contain', '$0.372'); - cy.get('[data-testid="video-runs-section"]').should('not.have.attr', 'open'); - cy.get('[data-testid="video-ci-runs"]').should('not.exist'); + cy.get('[data-testid="video-history-section"]').should('not.have.attr', 'open'); + cy.get('[data-testid="video-history"]').should('not.exist'); }); it('lists only the measured deployments in the table view, without a concurrency column', () => { cy.visit('/video?v_view=table'); @@ -207,7 +207,6 @@ describe('Video hardware dashboard (E2E fixtures)', () => { 'contain', 'Cost tier: Owning at Large Hyperscaler Volume.', ); - // Swapping sides flips the deltas and records the pair in the URL. cy.get('[data-testid="video-compare-swap"]').click(); cy.get('[data-testid="video-compare-baseline"]').should('contain', 'B200'); cy.get('[data-testid="video-compare-candidate"]').should('contain', 'H100'); @@ -217,9 +216,9 @@ describe('Video hardware dashboard (E2E fixtures)', () => { .should('have.attr', 'data-tone', 'worse'); cy.location('search').should('contain', 'v_base=b200').and('contain', 'v_cand=h100'); }); - it('opens the runs section for history deep links and renders the Chinese dashboard', () => { - cy.visit('/video?view=history'); - cy.get('[data-testid="video-runs-section"]').should('have.attr', 'open'); + it('opens the history section for history deep links and renders the Chinese dashboard', () => { + cy.visit('/video?history-hardware=H200'); + cy.get('[data-testid="video-history-section"]').should('have.attr', 'open'); cy.get('[data-testid="video-history"] h1').should('contain', 'Performance history'); cy.visit('/zh/video'); cy.get('[data-testid="video-hardware-chart"] circle.point').should('have.length', 3); diff --git a/packages/app/src/components/video-benchmark/FidelityResults.tsx b/packages/app/src/components/video-benchmark/FidelityResults.tsx deleted file mode 100644 index a2223008c..000000000 --- a/packages/app/src/components/video-benchmark/FidelityResults.tsx +++ /dev/null @@ -1,493 +0,0 @@ -'use client'; - -import { useEffect, useState } from 'react'; -import { Card } from '@/components/ui/card'; -import { Heading } from '@/components/ui/heading'; -import { useLocale } from '@/lib/use-locale'; -import { track } from '@/lib/analytics'; -import VideoSelect from './VideoSelect'; -import { at, ROLES, rows, text, type Json } from './bundle'; -import { fidelitySource, loadFidelityBundle, type FidelityBundle } from './fidelity'; -import { storedFidelityBundle, type StoredSource } from './stored'; -import { renderReportHtml } from './report'; - -const STRINGS = { - en: { - title: 'Paired video fidelity', - intro: 'CPU comparison of retained C1 videos. No new video generation.', - pairs: 'Matched valid pairs', - outside: 'Pairs outside fidelity thresholds', - inconclusive: 'Inconclusive pairs', - calibration: 'Threshold calibration', - qualification: 'Release qualified', - unavailable: 'Unavailable', - uncalibrated: 'Uncalibrated', - yes: 'Yes', - no: 'No', - note: 'Technical validity covers decoding, geometry and continuity. Paired pixel/audio comparisons measure output drift using exploratory thresholds; they do not establish perceptual quality, prompt adherence or a hardware quality winner.', - loading: 'Loading and verifying paired evidence…', - error: 'Could not load paired evidence', - slot: 'Paired clip', - baseline: 'Baseline', - candidate: 'Candidate', - prompt: 'Prompt', - seed: 'Seed', - audio: 'Play one side at a time to compare original audio.', - download: 'Download original MP4', - noMedia: 'No media for this observation', - mediaError: 'The browser could not play this media. Download the original MP4 to inspect it.', - latency: 'Submission to downloaded media (s)', - integrity: 'Video/audio technical validity', - details: 'Generation settings and revisions', - runtime: 'Runtime revision', - model: 'Model revision', - metrics: 'Measured fidelity', - psnr: 'Video PSNR (dB)', - mae: 'Video MAE (normalized 0–1)', - identical: 'Identical frames; finite PSNR undefined', - spectral: 'Audio spectral cosine', - rms: 'Audio RMS ratio (candidate / baseline)', - waveform: 'Audio waveform MAE (PCM amplitude)', - frames: 'Compared / total video frames', - videoCoverage: 'Video sample coverage (fraction)', - audioCoverage: 'Audio sample coverage (fraction)', - audioSamples: 'Compared audio samples / channel', - checks: 'Recorded checks and thresholds', - check: 'Check', - status: 'Recorded status', - observed: 'Observed', - threshold: 'Threshold', - unit: 'Unit', - timing: - 'Delivery latency is descriptive and includes polling and download. These cross-hardware results are not a calibrated performance regression or serving-capacity claim.', - sources: 'Original GPU runs and evidence', - originalResult: 'Original serving result and runtime log', - runtimeNote: - 'Resolved attention backends and runtime settings can differ across sources. Consult the original runtime logs before attributing output differences to hardware.', - producer: 'CPU comparison CI', - artifact: 'Download CI artifact', - digest: 'Comparison SHA256', - raw: 'Raw comparison JSON', - report: 'Original report', - limitations: 'Backend policy and limitations', - }, - zh: { - title: '成对视频保真度', - intro: '使用保留的 C1 视频在 CPU 上进行对比,未重新生成视频。', - pairs: '匹配的有效视频对', - outside: '超出保真度阈值的视频对', - inconclusive: '无法判定的视频对', - calibration: '阈值校准状态', - qualification: '是否通过发布验收', - unavailable: '无数据', - uncalibrated: '未校准', - yes: '是', - no: '否', - note: '技术有效性涵盖解码、画面尺寸与连续性。像素与音频配对比较采用探索性阈值,衡量的是输出偏差,不能证明感知质量或 prompt 遵循程度,也不能据此判定哪种硬件的视频质量更好。', - loading: '正在加载并校验配对证据…', - error: '无法加载配对证据', - slot: '配对视频', - baseline: '基线', - candidate: '候选', - prompt: 'Prompt', - seed: 'Seed', - audio: '每次只播放一侧,以便对比原始音频。', - download: '下载原始 MP4', - noMedia: '此侧暂无可用媒体', - mediaError: '浏览器无法播放此媒体,请下载原始 MP4 检查。', - latency: '提交到视频下载完成(秒)', - integrity: '视频与音频技术有效性', - details: '生成设置与版本', - runtime: '运行时版本', - model: '模型版本', - metrics: '实测保真度', - psnr: '视频 PSNR(dB)', - mae: '视频 MAE(归一化 0–1)', - identical: '帧完全一致,PSNR 无有限值', - spectral: '音频频谱余弦相似度', - rms: '音频 RMS 比值(候选 / 基线)', - waveform: '音频波形 MAE(PCM 幅度)', - frames: '已比较 / 总视频帧数', - videoCoverage: '视频采样覆盖比例', - audioCoverage: '音频采样覆盖比例', - audioSamples: '每声道已比较音频采样数', - checks: '已记录的检查与阈值', - check: '检查项', - status: '已记录状态', - observed: '观测值', - threshold: '阈值', - unit: '单位', - timing: - '交付延迟仅作描述,包含轮询与下载时间。这些跨硬件数据不构成已校准的性能回归或持续服务容量结论。', - sources: '原始 GPU 运行与证据', - originalResult: '原始服务测试结果与运行时日志', - runtimeNote: - '各来源实际使用的 attention backend 与运行时设置可能不同。判断输出差异的原因时,请核对原始运行时日志,不要直接归因于硬件。', - producer: 'CPU 对比 CI', - artifact: '下载 CI 产物', - digest: '对比结果 SHA256', - raw: '原始对比 JSON', - report: '原始报告', - limitations: '后端策略与局限', - }, -}; - -export default function FidelityResults({ - reader, - published, - runId, -}: { - reader?: (path: string) => Promise; - published?: StoredSource; - runId: string; -}) { - const locale = useLocale(); - const s = STRINGS[locale]; - const [loaded, setLoaded] = useState<{ - bundle: FidelityBundle; - urls: Map; - downloads: Map; - html: string; - } | null>(null); - const [loadError, setError] = useState(''); - const [slotId, setSlotId] = useState(''); - const [mediaError, setMediaError] = useState(false); - const [reportOpen, setReportOpen] = useState(false); - useEffect(() => { - let cancelled = false; - const created: string[] = []; - setLoaded(null); - setError(''); - setSlotId(''); - setMediaError(false); - void (async () => { - try { - const bundle = published - ? storedFidelityBundle(published) - : await loadFidelityBundle(reader!, runId); - if (cancelled) return; - const urls = new Map(published?.assets.map(([path, asset]) => [path, asset.url])); - if (!published) - for (const [path, blob] of bundle.files) { - const url = URL.createObjectURL( - new Blob([blob], { - type: path.endsWith('.mp4') ? 'video/mp4' : 'application/octet-stream', - }), - ); - created.push(url); - urls.set(path, url); - } - const raw = await bundle.files.get('report/index.html')?.text(); - if (cancelled) return; - setLoaded({ - bundle, - urls, - downloads: published - ? new Map(published.assets.map(([path, asset]) => [path, asset.downloadUrl])) - : urls, - html: raw ? renderReportHtml(raw, urls) : '', - }); - } catch (error) { - if (!cancelled) setError(error instanceof Error ? error.message : String(error)); - } - })(); - return () => { - cancelled = true; - created.forEach(URL.revokeObjectURL); - }; - }, [reader, published, runId]); - const fmt = (value: Json | undefined) => - typeof value === 'number' - ? value.toLocaleString('en-US', { maximumFractionDigits: 4 }) - : value === true - ? s.yes - : value === false - ? s.no - : value === null || value === undefined || value === '' - ? s.unavailable - : typeof value === 'string' - ? value - : JSON.stringify(value); - if (loadError) - return ( - - {s.error} -

{loadError}

-
- ); - if (!loaded) return {s.loading}; - const { bundle, urls, downloads } = loaded; - const comparison = bundle.comparison; - const slots = rows(at(bundle.portable, 'slots')); - const slot = slots.find((item) => at(item, 'slot_id') === slotId) ?? slots[0]; - const metrics = at(slot, 'metrics'); - const table = (values: [string, Json | undefined][]) => ( -
- {values.map(([label, value]) => ( -
-
{label}
-
{fmt(value)}
-
- ))} -
- ); - return ( -
- -
- {s.title} -

{s.intro}

-
-
- {[ - [s.pairs, at(comparison, 'summary', 'matched_valid_pairs')], - [ - s.outside, - slots.filter((item) => - rows(at(item, 'checks')).some( - (check) => - text(at(check, 'name')).startsWith('fidelity.') && - at(check, 'status') === 'fail', - ), - ).length, - ], - [s.inconclusive, at(comparison, 'summary', 'inconclusive_slots')], - ].map(([label, value]) => ( -
-

{String(label)}

-

{fmt(value)}

-
- ))} -
- {table([ - [ - s.calibration, - at(comparison, 'policy', 'calibration_status') === 'uncalibrated' - ? s.uncalibrated - : at(comparison, 'policy', 'calibration_status'), - ], - [s.qualification, at(comparison, 'release_qualified')], - ])} -

{s.note}

-

{s.runtimeNote}

-
- -
- { - setSlotId(value); - setMediaError(false); - track('video_fidelity_clip_selected', { slot: value }); - }} - options={slots.map((item) => ({ - value: text(at(item, 'slot_id')), - label: `${text(at(item, 'slot_id'))} · ${text(at(item, 'case_id'))}`, - }))} - /> -
-

- {s.prompt}: - {text(at(slot, 'prompt'))} -

-

- {s.seed}: {fmt(at(slot, 'seed'))} · {s.audio} -

-
- {ROLES.map((role) => { - const observation = at(slot, role); - const path = text(at(observation, 'artifact_path')); - const source = fidelitySource(bundle, role); - const sourceId = at(source, 'ci', 'databaseId'); - const ci = bundle.documents.get(`sources/${sourceId}/ci.json`); - return ( -
- - {s[role]} · {fmt(at(ci, 'site', 'gpu_model'))} - - {urls.get(`report/${path}`) ? ( -
- ); - })} -
- {mediaError &&

{s.mediaError}

} -

{s.timing}

-
- {s.details} -
-            {JSON.stringify(
-              {
-                generation: at(comparison, 'plan', 'generation'),
-                baseline: at(comparison, 'baseline', 'configuration'),
-                candidate: at(comparison, 'candidate', 'configuration'),
-              },
-              null,
-              2,
-            )}
-          
-
-
- - {s.metrics} - {table([ - [ - s.psnr, - at(metrics, 'video_identical') === true ? s.identical : at(metrics, 'video_psnr_db'), - ], - [s.mae, at(metrics, 'video_mae')], - [s.spectral, at(metrics, 'audio_spectral_cosine')], - [s.rms, at(metrics, 'audio_rms_ratio')], - [s.waveform, at(metrics, 'audio_waveform_mae')], - [ - s.frames, - `${fmt(at(metrics, 'video_compared_frames'))} / ${fmt(at(metrics, 'video_total_frames'))}`, - ], - [s.videoCoverage, at(metrics, 'video_sample_coverage_fraction')], - [s.audioCoverage, at(metrics, 'audio_sample_coverage_fraction')], - [s.audioSamples, at(metrics, 'audio_compared_samples_per_channel')], - ])} -
- {s.checks} -
- - - - {[s.check, s.status, s.observed, s.threshold, s.unit].map((label) => ( - - ))} - - - - {rows(at(slot, 'checks')).map((check, index) => ( - - {['name', 'status', 'observed', 'threshold', 'unit'].map((key) => ( - - ))} - - ))} - -
- {label} -
- {fmt(at(check, key))} -
-
-
-
- - {s.sources} -
- {ROLES.map((role) => { - const source = fidelitySource(bundle, role); - const id = at(source, 'ci', 'databaseId'); - const artifact = at(source, 'artifact', 'id'); - return ( - - ); - })} -
- - {s.producer} #{runId} - - {table([[s.digest, bundle.comparisonSha256]])} - {downloads.get('comparison.json') && ( - - {s.raw} - - )} - {loaded.html && ( -
setReportOpen(event.currentTarget.open)}> - {s.report} - {reportOpen && ( -