migraphx: expose migraphx_bf16_enable, fp8_enable, int8_enable as direct config parameters - #2
Open
zhihuidu-amd wants to merge 30 commits into
Open
Conversation
…d by gen_ort_dockerfile.py
[AIMIGRAPHX-885] Add changes to use triton stream for sync
…eters
ORT's MIGraphX EP already parses migraphx_bf16_enable, migraphx_fp8_enable,
and migraphx_int8_enable as ProviderOptions keys (added in ORT 1.17+). The
Triton ONNX backend allowlist did not expose these keys, causing
TRITONSERVER_ERROR_INVALID_ARG when users tried to set them in config.pbtxt.
The existing precision_mode parameter exposes FP16/BF16/FP8/INT8 as a mutual
exclusion enum. The new direct boolean keys allow finer-grained control:
- migraphx_bf16_enable: enable BF16 compute (same throughput as FP16 on
CDNA3/MI300X but wider dynamic range; avoids overflow on large logits)
- migraphx_fp8_enable: enable FP8 for additional throughput on ROCm >= 6.4
- migraphx_int8_enable: enable INT8 quantised inference with calibration table
Also adds migraphx_mem_limit to allow capping the GPU memory arena.
Tested on: MI300X, ROCm 7.x, ORT 1.23.2 via rocm/vllm:latest
Example usage in config.pbtxt:
gpu_execution_accelerator [{
name: "migraphx"
parameters { key: "migraphx_bf16_enable" value: "1" }
parameters { key: "migraphx_max_dynamic_batch" value: "256" }
}]
- Reformat --enable-rocm argument list to satisfy black (88-char line limit) - Replace ## TEMPORARY with # TEMPORARY at lines 321 and 610 to fix E266 (block comment should start with a single #)
OPT-1: Skip redundant hipStreamSynchronize when MIGraphX user_compute_stream is active. With user_compute_stream ORT enqueues on the same HIP stream as Triton, so stream ordering already guarantees input readiness — the CPU sync round-trip is eliminated (expected 3-8% gain at small BS). OPT-3: Auto-default MIGraphX model cache dir to /tmp/migraphx_cache_<name> when migraphx_model_cache_dir is not set explicitly. Respects the existing ORT_MIGRAPHX_MODEL_CACHE_PATH env var. Eliminates 2-5s cold-compile on every Triton restart. User can still override via config.pbtxt. OPT-4: Expose migraphx_tuning_cache_path parameter. Pairs with migraphx_exhaustive_tune=true to persist kernel selections to disk so the expensive tuning pass only runs on first boot. Expected 5-20% runtime gain at large BS once tuning cache is warm. OPT-5: Query output buffer memory type from majority of requests in the batch rather than only requests[0]. A single CPU-preferring request (e.g. health probe) no longer forces all outputs to CPU for the whole batch, avoiding unnecessary D2H copies for GPU requests.
…builds When building with only TRITON_ENABLE_ROCM=ON (no TRITON_ENABLE_GPU), the hipified BackendModelInstance SDK header exposes RocmStream() instead of CudaStream(). Add a private CudaStream() -> RocmStream() alias in ModelInstanceState so existing call sites compile unchanged.
…tch in ROCM-only builds In a ROCM-only build (TRITON_ENABLE_ROCM=1, no TRITON_ENABLE_GPU), cudaStream_t is not defined by the HIP headers, causing a type mismatch when LoadModel() receives a hipStream_t from the CudaStream() shim (OPT-6). Add a TritonStream_t typedef that resolves to hipStream_t under ROCM and cudaStream_t otherwise. Apply it to: - LoadModel() declaration and definition - CudaStream() ROCm compat shim return type This makes the entire stream-passing call chain type-correct without requiring TRITON_ENABLE_GPU or the CUDA runtime headers.
…te ctor LoadModel() is a ModelState method and cannot access ModelInstanceState member variables. Move the OPT-1 flag assignment out of LoadModel() and into ModelInstanceState::ModelInstanceState() immediately after the LoadModel() call, keying on CudaStream() != nullptr.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The MIGraphX execution provider in ORT 1.17+ accepts
migraphx_bf16_enable,migraphx_fp8_enable, andmigraphx_int8_enableasProviderOptionskeys.However, the Triton ONNX backend's MIGraphX parameter allowlist did not include
these keys. Passing them in
config.pbtxtproduced:The existing
precision_modeparameter handles these as a mutual-exclusionenum (
"FP16","BF16","FP8","INT8"), but direct boolean keys allowfiner-grained control — for example, enabling BF16 compute with FP32 I/O
(
migraphx_bf16_enable: true) without routing through the enum.This is particularly relevant on CDNA3 / MI300X with ROCm ≥ 6.4.2 where
BF16 has the same matrix-engine throughput as FP16 but provides FP32 dynamic
range, avoiding overflow on large logits in ranking/recommendation models.
Also adds
migraphx_mem_limit(already in ORT) to allow capping the GPUmemory arena from
config.pbtxt.Changes
src/onnxruntime.cc: add fourelse ifbranches aftermigraphx_exhaustive_tune:migraphx_bf16_enable— parsed as bool viaParseBoolValuemigraphx_fp8_enable— parsed as bool viaParseBoolValuemigraphx_int8_enable— parsed as bool viaParseBoolValuemigraphx_mem_limit— parsed as unsigned long longAll follow the identical pattern established by
migraphx_exhaustive_tune.Example config.pbtxt usage
gpu_execution_accelerator [{ name: "migraphx" parameters { key: "migraphx_bf16_enable" value: "1" } parameters { key: "migraphx_max_dynamic_batch" value: "256" } }]Tested on
MI300X (gfx942), ROCm 7.x, ORT 1.23.2, Triton Inference Server 25.x