Skip to content

Commit 9b9db07

Browse files
committed
[bugfix] matrixgame2 causal denoise: store KV-cache end indices as Python int, not GPU tensor
Storing kv_cache["global_end_index"] / ["local_end_index"] as torch.tensor([0], device=device) forces the consumer sites in causal_model.py and action_module.py through their isinstance(Tensor) branch, which does int(tensor.item()) on every access. Each .item() is a GPU->CPU sync that serializes the autoregressive denoise loop and prevents kernel-queue lookahead. Both call sites already handle the Python int case (their isinstance branch goes to else and uses the int directly). The fix is to initialize as plain int in _initialize_kv_cache / _initialize_action_kv_cache and let the producer ('fill_' vs assignment) and consumer (item vs int cast) branches resolve. Measured on Modal H100x1, num_frames=117 (10 AR blocks * 3 DMD steps = 30 DiT forwards): Denoise stage syncs: 10,244 -> 644 (-94%) Denoise stage sync_ms: 264.1 -> 11.2 (-96%) avg_queue_delay_us: 42,919 -> 20,465 (-52%) Inference wall: 18.3s -> 17.2s (-6%) Note the modest wall improvement: the remaining ~77% GPU-idle is Python-interpreter throughput bound (5,455 kernels/forward at 14.8us avg), not sync-bound. This fix is a prerequisite for any subsequent CUDA Graph / persistent-kernel work but not the full answer.
1 parent d6119c1 commit 9b9db07

1 file changed

Lines changed: 6 additions & 12 deletions

File tree

fastvideo/pipelines/stages/matrixgame2_denoising.py

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -278,10 +278,8 @@ def _initialize_kv_cache(self, batch_size: int, dtype: torch.dtype, device: torc
278278
torch.zeros([batch_size, kv_cache_size, num_attention_heads, attention_head_dim],
279279
dtype=dtype,
280280
device=device),
281-
"global_end_index":
282-
torch.tensor([0], dtype=torch.long, device=device),
283-
"local_end_index":
284-
torch.tensor([0], dtype=torch.long, device=device),
281+
"global_end_index": 0,
282+
"local_end_index": 0,
285283
})
286284

287285
return kv_cache
@@ -302,10 +300,8 @@ def _initialize_action_kv_cache(self, batch_size: int, dtype: torch.dtype, devic
302300
torch.zeros([batch_size, kv_cache_size, action_heads, keyboard_head_dim], dtype=dtype, device=device),
303301
"v":
304302
torch.zeros([batch_size, kv_cache_size, action_heads, keyboard_head_dim], dtype=dtype, device=device),
305-
"global_end_index":
306-
torch.tensor([0], dtype=torch.long, device=device),
307-
"local_end_index":
308-
torch.tensor([0], dtype=torch.long, device=device),
303+
"global_end_index": 0,
304+
"local_end_index": 0,
309305
})
310306
kv_cache_mouse.append({
311307
"k":
@@ -316,10 +312,8 @@ def _initialize_action_kv_cache(self, batch_size: int, dtype: torch.dtype, devic
316312
torch.zeros([batch_size * self.frame_seq_length, kv_cache_size, action_heads, mouse_head_dim],
317313
dtype=dtype,
318314
device=device),
319-
"global_end_index":
320-
torch.tensor([0], dtype=torch.long, device=device),
321-
"local_end_index":
322-
torch.tensor([0], dtype=torch.long, device=device),
315+
"global_end_index": 0,
316+
"local_end_index": 0,
323317
})
324318

325319
return kv_cache_mouse, kv_cache_keyboard

0 commit comments

Comments
 (0)