Skip to content

Fix scaled_mm CuTe compile contract for blockscaled persistent GEMM #18334

Description

@coderabbitai

Summary

scaled_mm in tensorrt_llm/_torch/cute_dsl_kernels/blackwell/dense_blockscaled_gemm_persistent.py compiles Sm100BlockScaledPersistentDenseGemmKernel directly. Its positional arguments do not match Sm100BlockScaledPersistentDenseGemmKernel.__call__.

Required change

Update scaled_mm to compile the wrapper interface. Supply the required runtime dimensions and alpha_tensor. Preserve the existing pointer construction, layout mapping, cluster limit, stream, epilogue operation, and compilation options.

Rationale

The direct cute.compile(gemm_obj, ...) call binds the layout tuple where __call__ expects alpha. It then shifts the remaining positional arguments. This can produce an invalid compiled invocation.

Affected area

  • tensorrt_llm/_torch/cute_dsl_kernels/blackwell/dense_blockscaled_gemm_persistent.py
  • scaled_mm
  • Sm100BlockScaledPersistentDenseGemmKernel.wrapper

Acceptance criteria

  • scaled_mm compiles the wrapper with arguments that match the wrapper signature.
  • The compile-time placeholders include the GEMM dimensions and alpha_tensor.
  • Existing layout, pointer-alignment, stream, epilogue, cluster, and options behavior remains intact.
  • Add or update focused coverage for the compile interface if the test environment supports it.

Backlinks

Requested by: @farazkh80

Metadata

Metadata

Assignees

Labels

Customized kernels<NV>Specialized/modified CUDA kernels in TRTLLM for LLM ops, beyond standard TRT. Dev & perf.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions