[feat]: Add NVFP4QAT linear layer (Attn-QAT 3/12)

[SolitaryThinker]

Summary

Slice 3 of 12 in the decomposition of #1225 ([feat] Upstream Attn-QAT Video Diffusion Code). Adds the FastVideo-native FP4 linear forward helper module extracted from PR-1225.

Pure deadcode: no existing code path imports fp4_linear_forward yet. The QAT attention backends (Slice 4) and beyond will wire it up; activation lands in Slice 12.

Stacking

• Base: pr1225_s2 (Slice 2, [feat]: Add NVFP4QAT quantization config (Attn-QAT 2/12) #1348, still open).
• When [feat]: Add NVFP4QAT quantization config (Attn-QAT 2/12) #1348 merges, GitHub auto-retargets this PR's base to main.
• Cumulative diff @ Slice 12/12 ≡ full PR [feat] Upstream Attn-QAT Video Diffusion Code #1225 feature.

Naming family

• NVFP4QATConfig (Slice 2, [feat]: Add NVFP4QAT quantization config (Attn-QAT 2/12) #1348)
• FP4 linear forward helper module (this slice)
• Distinct from main's NVFP4Config / NVFP4 quant method (Blackwell hardware FP4 inference — added by PR [infra] Dreamverse 11/14: Add NVFP4 quantization support #1334).

Files

• fastvideo/layers/fp4linear.py (NEW, 115 LOC) — source PR-1225 module. The extracted file does not define a public Fp4Linear* class or import Fp4Config, so no class/file rename or config-import substitution was applied.

Provenance

Extracted from PR #1225 by @RandNMR73 (source SHA: 3f818d0f). Git author is @SolitaryThinker; @jzhang38 and @RandNMR73 are co-authored via commit trailers.

Test plan

• No behavior change (deadcode). pre-commit run --files fastvideo/layers/fp4linear.py passes.
• Unit-test coverage for the FP4 linear helper lands in a later slice once a caller exists.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[gemini-code-assist]

Code Review

This pull request introduces an FP4 linear layer implementation leveraging the flashinfer library. It includes a custom autograd function, _LinearFWD4BWD16Fn, which handles FP4 quantization and matrix multiplication in the forward pass, while maintaining standard precision for the backward pass. The review feedback suggests optimizing the _global_sf helper function by moving it to the module level to avoid re-definition overhead during each forward pass and using the more idiomatic .clamp() method for numerical stability.

[gemini-code-assist]

Move the _global_sf helper to the module level and decorate it with @torch.compile there. Defining and decorating a function inside the forward method of an autograd.Function is highly inefficient as it causes the function to be re-defined (and potentially re-compiled or cache-checked) on every forward pass, leading to significant overhead. Additionally, using .clamp(min=...) is more idiomatic and efficient than torch.maximum with a newly created tensor.

@torch.compile
def _global_sf(t: torch.Tensor) -> torch.Tensor:
    maxabs = t.float().abs().nan_to_num().max()
    maxabs = maxabs.clamp(min=1e-12)
    return (448.0 * 6.0) / maxabs


class _LinearFWD4BWD16Fn(torch.autograd.Function):

[SolitaryThinker]

Done in dbc9e7c. Hoisted _global_sf to module level with @torch.compile and switched to .clamp(min=1e-12). Thanks for the catch.

[gemini-code-assist]

Remove the local definition of _global_sf as it should be moved to the module level for better performance and to avoid re-definition overhead during every forward pass.

[SolitaryThinker]

Done in dbc9e7c. Hoisted _global_sf to module level with @torch.compile and switched to .clamp(min=1e-12). Thanks for the catch.

[SolitaryThinker]

/merge