[bugfix] Gate the fused Ulysses all-to-all kernel on one host, and stop re-voting per call

[shaoxiongduan]

Purpose

Two defects in the fused Ulysses all-to-all from #1740.

On a sequence-parallel group that spans two nodes of a GB200 rack, the kernel engages
and runs about 2.4x slower than the NCCL path it replaces. ncclTeamLsa reports
whether peer memory is addressable, not whether it is fast, and NCCL 2.29 extends the
LSA team across a multi-node NVLink domain. The kernel's fine-grained 16-byte remote
stores lose badly to NCCL's bulk transfers over that fabric.

Separately, the per-call agreement collective costs a flat ~227us, which eats most of
the speedup on a single node: 1.52x drops to 1.19x, and on the smaller gather operand
it goes net negative.

Fixes #

Changes

• Require a single host. The topology gate now compares hostnames alongside the
  LSA check. This is the regime the slab decomposition was tuned for, and matches
  FlashInfer's own gate (ulysses_topology.py:222).

• Make the topology exchange unconditional. _can_attempt computes its local
  verdict without collectives, then runs one all_gather_object carrying
  (hostname, local_ok), then decides. A collective behind a rank-local early return
  hangs the group exactly when ranks disagree, which is what this gate detects.

• Cache the per-call agreement by signature. _agree_call issued a host-side gloo
  all_gather before every collective. A generation has two distinct signatures, the
  scatter and gather shapes, across 50 layers x 4 steps, so this drops ~400 collectives
  to 2. Arming, window growth, and a new resolution each change the signature, so a
  stale verdict cannot be served.

Test Plan

4x GB200 (sm_100a), MiniMax-H3 and Wan2.1-14B at 5s 720p, bf16, p50 of 60 rank-max
samples. sp=8 spans two nodes.

# per-layer collective, fused off vs auto
FASTVIDEO_ULYSSES_A2A={off,auto} torchrun --nproc_per_node=4 bench_a2a.py --iters 60
FASTVIDEO_ULYSSES_A2A={off,auto} torchrun --nnodes=2 --nproc_per_node=4 \
    --node-rank=$R --master_addr=$N1 bench_a2a.py --iters 60

# one rank reports the kernel unavailable; the group must still finish
FAULT_RANK=2 torchrun --nproc_per_node=4 deadlock_probe.py

# end to end
python examples/inference/basic/basic_fasth3_lora_preview.py \
    --num-gpus 4 --repeats 3 --ulysses-a2a {off,auto} [--no-inference-torch-compile] --vsa

Test Results

Test output

per-layer collective (us), p50 of 60 rank-max samples

  sp=4, one node           NCCL     before            after
    Wan2.1-14B  layer      3382     2688 (1.26x)      2243 (1.51x)
    MiniMax-H3  layer      2421     2035 (1.19x)      1590 (1.52x)
      of which gather       734      897 (0.82x)       455 (1.61x)

  sp=8, two nodes          NCCL     before            after
    Wan2.1-14B  layer      1811     3512 (1.9x slow)  declines -> 1818
    MiniMax-H3  layer      1295     3102 (2.4x slow)  declines -> 1300
      logged: "ranks span multiple hosts: ['hpc-rack-3-1', 'hpc-rack-3-2']"

one rank reporting the kernel unavailable (sp=4)
  before : exit=124, timed out at 180s
  after  : all 4 ranks completed, results correct

end to end, FastH3 4-step VSA, 1344x768 124f, 4 GPU, median denoising of 3
  regional compile ON,  a2a off    2.247s
  regional compile ON,  a2a auto   2.253s
  regional compile OFF, a2a off    2.842s
  regional compile OFF, a2a auto   2.657s

The shipped FastH3 profile is unchanged, because the fused path declines inside a
regionally compiled region and that profile enables regional compile. These fixes
apply to uncompiled inference and to training.

Checklist

• I ran pre-commit run --all-files and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU memory impact of my changes

For model/pipeline changes, also check: n/a, this changes a device communicator only.

• I verified SSIM regression tests pass
• I updated the support matrix if adding a new model

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]