Repository navigation
Conversation
Co-authored-by: mukh1l <mmallaiy@amd.com> Co-authored-by: awelling2801 <anuya.welling@amd.com> Co-authored-by: Tiffany Mintz <Tiffany.Mintz@amd.com> Co-authored-by: Vicky Tsang <vicky.tsang@amd.com> Co-authored-by: Radha Krishna Srimanthula <radha.srimanthula@amd.com>
Co-authored-by: mukh1l <mmallaiy@amd.com> Co-authored-by: awelling2801 <anuya.welling@amd.com> Co-authored-by: Tiffany Mintz <Tiffany.Mintz@amd.com> Co-authored-by: Vicky Tsang <vicky.tsang@amd.com> Co-authored-by: Radha Krishna Srimanthula <radha.srimanthula@amd.com>
This commit also adds an include that was missing during some builds and causing failures.
…rough cmake rather than directly with make
… single file (and the test utils).
…cess. Currently crashing on missing <cuda_fp16.h>
…s crash with missing symbols errors
…ng with import problems
… in warp size macros
Turning on more tests, mostly in graphbolt/impl. Still only one failure (other than the missing torch_geometric dependency) so far. It's a failure in test_hetero_cached_feature where the miss rate is slightly too high. Still have to turn on the dataloader tests.
The hipification of HugeCTR was in progress on the ROCm/dgl default branch, but was missing a few small tweaks. Especially the initialization of the warp_position in nv_gpu_cache.cu.
Pulling the changes directly from the prehipified source in the nod-ai branch and then making minor tweaks.
…LVM host compiler
Removed skip macros for - tests/python/common/cuda/test_gpu_cache.py - tests/python/pytorch/cuda/test_nccl.py
[Feature] Updating with Upstream
- Fixes the test failure in test_hetero_cached_feature.py caused by rocm's constraints on atomics (i.e. it doesn't support 8bit atomics) - Cleaning up comments and unused files from self review
This PR consolidates the graphbolt/CMakeLists.tx as much as possible so that CUDA and ROCm configurations do not have redundant CMake variables. It also switched the configurate of ROCm archs to the more standard CMAKE_HIP_ARCHITECTURES.
[DOC] Update with readme with pypi links
[Feature] Bump hipCollections version
[Feature] Updating dgl ROCm support
… as a separate file
adding changes to mark rocm/dgl deprecated, retaining original readme…
mukh1l
marked this pull request as ready for review
June 22, 2026 15:33
Updates GraphBolt and the tensor adapter to use PyTorch's CUDA-masquerading HIP APIs, fixes hipCUB 5.0 iterator compatibility, and hardens ROCm CI with dynamic test ports and isolated DataLoader test runs.
Pandas can return read-only numpy views from to_numpy(), which breaks in-place node-type offset updates during heterogeneous graph preprocessing. Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce docker/Dockerfile.rocm, sync the legacy CI Dockerfile, apply cuco header patches during GraphBolt deps install, and restore HIP masquerading aliases so GraphBolt builds against recent PyTorch ROCm stacks. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix ROCm/HIP build, GraphBolt, and OnDiskDataset compatibility
- docker/Dockerfile.rocm, docker/Dockerfile.ci_gpu_rocm: BASE_IMAGE -> rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.13.0 (confirmed to exist and pull cleanly on Docker Hub). - Drop gfx90a from GPU_BUILD_TARGETS, add gfx950: gfx942,gfx950. CMakePresets.json "rocm" preset updated to match. - ROCm 10.0's rocm/pytorch image ships ROCm via pip (rocm-sdk-core/ -libraries/-device-*) instead of the classic /opt/rocm tree, and omits the 'devel' package (headers + cmake configs). Added a RUN step to both Dockerfiles that installs the matching rocm[devel] wheel from https://stable.repo.amd.com/rocm/whl-next/, runs `rocm-sdk init`, and symlinks /opt/rocm -> `rocm-sdk path --root`. This is a no-op on older base images that already ship a real /opt/rocm (e.g. rocm7.0.2), so CMakePresets.json and script/install_graphbolt_deps.sh keep working unmodified either way. Validated live on a real MI300X node (ctr-cx66-mi300x-01): - Base image pulls and runs with GPU passthrough; torch reports ROCm 10.0.0 and sees the MI300X. - The /opt/rocm symlink fix works: `cmake --preset rocm` configures successfully against the pip-installed devel tree. KNOWN BLOCKER (upstream, not fixed here): the actual HIP compile fails for both DGL core (~30 .cu.o targets, e.g. src/array/cuda/array_index_select.cu.o) and GraphBolt with the same root cause, purely inside ROCm 10.0.0's rocPRIM 4.6.0 headers: rocprim/device/detail/config/device_batch_copy.hpp:43: error: no template named 'default_batch_memcpy_config_base' `device_config_helper.hpp` (which device_batch_copy.hpp/ device_batch_memcpy.hpp include and depend on) does not define default_batch_memcpy_config_base/default_batch_copy_config_base at all in this release - confirmed by grepping the shipped header. Matches the documented design in rocPRIM's own developer docs, so this looks like a real packaging bug in the ROCm 10.0.0 stable rocPRIM release (a stale/regenerated config header out of sync with device_config_helper.hpp), not a DGL code issue: any Thrust code that touches batch memcpy/copy on the HIP backend (e.g. thrust::merge, thrust::sort) hits it. Confirmed stable.repo.amd.com only serves 10.0.0 right now, no patch release yet. Full build log: ~/dgl_rocm10_build.log on the shared home dir. Do not build/ship from this branch until that's resolved upstream or worked around. Co-authored-by: Cursor <cursoragent@cursor.com>
…13.0 Validated end-to-end on a real MI300X (gfx942), building for gfx942+gfx950, against the RELEASED/stable ROCm 10.0.0 (rocm/pytorch:rocm10.0 image) - not a nightly build. Core DGL, GraphBolt (incl. HugeCTR GPU-cache extension), tensoradapter, and dgl_sparse all now build and link cleanly, and real GPU message-passing + graphbolt import + dgl.sparse import were smoke-tested successfully on-device. 1. script/install_graphbolt_deps.sh, script/device_batch_copy.hpp (deleted), script/device_batch_memcpy.hpp (deleted): remove the unconditional `cp *.hpp .../rocprim/device/detail/config/.` step. This copied two hand-patched rocPRIM config headers (vendored here for an older ROCm release that genuinely lacked gfx942 rocPRIM configs, see upstream fix ROCm/rocm-libraries#1883) directly over ROCm 10.0.0's own shipped headers. ROCm 10.0.0 ships rocPRIM 4.6.0, which already has working gfx942/gfx950 configs but with a different internal API shape - so this step was clobbering good headers with stale, incompatible ones and was the actual root cause of the "no template named 'default_batch_memcpy_config_base'" build failures (previously misdiagnosed as an upstream ROCm/rocPRIM bug; it is not one). 2. graphbolt/src/cuda/common.h: - GRAPHBOLT_CACHING_ALLOCATOR and GRAPHBOLT_C10_KERNEL_LAUNCH_CHECK's HIP branch pointed at c10::hip::HIPCachingAllocator / C10_HIP_KERNEL_LAUNCH_CHECK, neither of which this PyTorch/ROCm version provides. Repointed at c10::cuda::CUDACachingAllocator / C10_CUDA_KERNEL_LAUNCH_CHECK, which this version defines directly and uses for both its CUDA and HIP builds internally (same pattern already used correctly by GRAPHBOLT_C10_CHECK -> C10_HIP_CHECK, which PyTorch aliases to C10_CUDA_CHECK). - CUDAWorkspaceAllocator::operator() took `void*`. Newer libstdc++'s std::unique_ptr destructor requires the deleter to satisfy is_invocable<Deleter&, pointer> as an exact-type call rather than via the (otherwise always-legal) implicit T*->void* conversion, so std::unique_ptr<T, CUDAWorkspaceAllocator> failed a static_assert for several T. Made operator() a template over the pointee type so it matches directly for any T. - CopyScalar::record()'s default argument and CopyScalar(const scalar_t*) both initialize a GPUStream_t (at::hip::HIPStreamMasqueradingAsCUDA) from GetCurrentStream(), which now returns the plain (base-class) c10::cuda::CUDAStream. HIPStreamMasqueradingAsCUDA's constructor-from-Stream is explicit, so this implicit conversion stopped compiling. Wrapped both in an explicit GPUStream_t(...). 3. graphbolt/include/graphbolt/async.h: removed an obsolete `namespace c10::cuda { using CUDAStream = ...; using CUDAStreamGuard = ...; inline auto getCurrentCUDAStream() {...} }` shim. It used to fill a real gap on older PyTorch/ROCm ports where c10::cuda::CUDAStream/ CUDAStreamGuard/getCurrentCUDAStream did not exist for HIP builds. This PyTorch version's own (hipified) <c10/hip/HIPStream.h> and <c10/hip/HIPGuard.h> now declare all three directly, so the shim conflicted with them (typedef redefinition, ambiguous overload on zero-arg calls). Removed; call sites resolve to PyTorch's real declarations, which already provide everything used here (pack3/unpack3, CUDAStreamGuard(Stream)). None of this touches the CUDA (non-HIP) build paths. Co-authored-by: Cursor <cursoragent@cursor.com>
…st env tests/scripts/task_unit_test_rocm.sh already worked around librocm-openblas.so.0 not being on the default loader path for rocm-sdk-wheel installs (torch_shm_manager needs it to share CPU tensors across processes). With ROCm 10.0.0 + PyTorch 2.13.0, torch_shm_manager also needs librocprofiler-sdk.so.1, which lives directly under _rocm_sdk_core/lib (a different subdirectory than the existing host-math/lib fix covers). Without it, torch_shm_manager fails to start and every test that shares CPU tensors across processes (torch.multiprocessing / mp.spawn-based tests, DGL's IPC tests) fails with a generic "no response from torch_shm_manager" RuntimeError - reproduced and fixed on a real MI300X: test_multiprocessing-ipc.py::test_torch_ipc and test_optim.py::test_multiprocess_sparse_adam_state_dict[nccl/gloo-1] all went from failing to passing with this LD_LIBRARY_PATH addition. Also extends .gitignore for pytest_*.xml / lcov_pytest*.info / python_coverage_report.txt / cluster_gcn.pkl / optim_ip_config.txt / _download/, which task_unit_test_rocm.sh writes into the repo root when run from there per its own usage instructions (tests/scripts/../Dockerfile.rocm container welcome message). Co-authored-by: Cursor <cursoragent@cursor.com>
Fixes all 6 test failures that weren't already covered by the librocprofiler-sdk.so.1 LD_LIBRARY_PATH fix. Full suite now passes clean: 10181 passed, 858 skipped, 0 failed (tests/scripts/task_unit_test_rocm.sh pytorch gpu), verified on a real MI300X. 1. python/dgl/utils/data.py (networkx2tensor, networkxbipartite2tensors): `int(attr[edge_id_attr_name])` raised "TypeError: only 0-dimensional arrays can be converted to Python scalars" when edge_id_attr_name's value is a single-element (but not 0-d) array, e.g. np.array([1]) as used by test_heterograph.py::test_create's fixture. NumPy < 2.0 silently allowed converting single-element arrays of any shape via int(); NumPy 2.0 restricts this to true 0-d arrays. Switched to int(np.asarray(x).item()), which numpy's own deprecation notice recommends and which handles plain ints, numpy scalars, 0-d arrays, and single-element N-d arrays uniformly. 2. python/dgl/nn/pytorch/explain/subgraphx.py (SubgraphX.shapley, HeteroSubgraphX.shapley): same root cause, same fix, for int(np.where(permuted_space == split_point)[0]) - fixes test_subgraphx/test_heterosubgraphx. 3. python/dgl/graphbolt/internal/utils.py (numpy_save_aligned): fixes test_numpy_save_aligned, which asserted the written file is 4096-byte aligned but got a plain ~128-byte-header file instead. NumPy's NEP 52 API cleanup (numpy >= 2.1) moved the real .npy-header-writing implementation from numpy.lib.format into a new private numpy.lib._format_impl module; numpy.lib.format now only re-exports ARRAY_ALIGN via `from ._format_impl import ARRAY_ALIGN`, which binds a *separate* name in numpy.lib.format's namespace. Reassigning numpy.lib.format.ARRAY_ALIGN (this function's old approach) therefore no longer affects what _format_impl's header-writing code reads. Patch numpy.lib._format_impl.ARRAY_ALIGN when that module exists (numpy 2.1+), and numpy.lib.format.ARRAY_ALIGN otherwise (older numpy, where numpy.lib.format *is* the real implementation) - verified this produces the expected 4096-byte-aligned header on both numpy 1.26.4 and 2.5.2. Also wrapped the np.save() call in try/finally so a save failure can't leave ARRAY_ALIGN patched for later callers. 4. docker/Dockerfile.rocm, docker/Dockerfile.ci_gpu_rocm: add `cython` to the `pip install build wheel` step before `pip install -e .`. Fixes test_cython (`import dgl._ffi._cy3.core`): setup.py's config_cython() only builds that Cython extension when Cython is importable at `pip install` time, silently falling back to a ctypes-only FFI otherwise - this gap predates the ROCm 10.0.0 migration and isn't ROCm-specific. (test_to_pyg_data's torch_geometric ModuleNotFoundError needed no fix: torch_geometric is already a real setup.py test dependency and installs correctly via the existing `pip install -e .` step; it was only missing from this session's earlier ad-hoc minimal test venvs, not from a real Dockerfile-driven install.) Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm 10.0+ ships via pip across several packages (rocm-sdk-core, rocm-sdk-libraries, rocm-sdk-device-*), each with its own runtime .so files living inside that package's site-packages dir rather than on the default loader path. runUnitTests failed one missing lib at a time as the loader walked the NEEDED list (libamdhip64.so.7, then libhipsparse.so.4, ...). Instead of chasing each missing library individually, discover every directory across all rocm_sdk_* packages that contains a .so and add them all to LD_LIBRARY_PATH at once. Also fix pushd build assuming the caller's cwd was already the repo root; resolve the repo root relative to the script's own location so it works regardless of invocation directory. Co-authored-by: Cursor <cursoragent@cursor.com>
…blas lookup Avoids hardcoding a specific ROCm version (previously rocm-6.4.0), which broke after bumping the base image to ROCm 10.0.0. Matches the unversioned /opt/rocm path already used for CMAKE_INSTALL_PREFIX/CMAKE_PREFIX_PATH in CMakePresets.json. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This pull request contributes the AMD/ROCm integration work from the ROCm/dgl fork back to upstream DGL.
Source
Branch amd-integration on ROCm/dgl, created from develop.
Test plan
Made with Cursor