Skip to content

Commit 02db9dc

Browse files
wfaderhold21wfaderhold21
committed
TEST: MPI add user memh reg. (local + global)
Adds opt-in user memory-handle registration to the MPI collective tests so the mem_map code paths (e.g. TL/CUDA push) get real coverage. - Replace the per-collective inline local_registration blocks with a shared TestCase::register_memhs(sbuf, ssize, dbuf, dsize) helper. - Add a global handle-array path (register_memh_global): export each buffer's local handle, exchange serialized blobs across the team, and import a per-rank ucc_mem_map_mem_h[] into args.*_memh.global_memh with the *_MEMH_GLOBAL flags. Blobs use ucc_malloc so ucc_mem_unmap's ucc_free matches; dtor unmaps and frees them. - Replace the bool local_registration with ucc_test_memh_mode_t (NONE/LOCAL/GLOBAL) end to end (main.cc driver, TestCaseParams, UccTestMpi); extend --local_reg to 0..4 to select the modes. - Wire register_memhs into all buffered collectives; onesided alltoall /alltoallv skip it since their buffers are already mapped segments. Buffer sizing matches each collective's actual allocation, including the inverted inplace/non-inplace dst sizes for reduce_scatter (non-inplace rbuf is msgsize/comm_size) and reduce_scatterv (inplace rbuf is msgsize). TEST/MPI: add dispatch hook for test_mem_map.cc so mem_map tests execute TEST: fix UccJob to preserve caller-provided env vars for CUDA TL push tests UccJob::UccJob unconditionally appended UCC_TL_CUDA_TUNE=0 and other disabling variables after caller-provided values. Since the setenv loop at :312 is last-write-wins, this overwrote test-specific tuning like alltoall:cuda:@Push:0-inf:inf, disabling CUDA TL entirely and causing every push test to GTEST_SKIP(). Fix by only appending the default-disabling value when the caller has not already set that key. Uses C++11-compatible helper (no generic lambda) since Makefile.am pins -std=gnu++11. TEST: fix class redefinitions in test_mem_map.cc and clang-tidy unused parameters - Remove duplicate class declarations from test_mpi.h (TestMemMapExport, TestMemMapImport, TestMemMapStress, TestMemMapMultiSize) that were also fully defined in test_mem_map.cc, causing 'redefinition' build failures with --with-cuda. - Suppress unused-parameter warnings for 'triggered' and '_type' parameters in test_mem_map.cc run() methods and init_single() factory functions via (void) casts with explanatory comments. - Fix memory leaks on exchange_dst_memh error paths in test_tl_cuda_push.cc - Restore environment variables after UccJob destructor in test_ucc.cc Verified: build clean with --with-cuda=/opt/cuda, clang-tidy clean. TEST/MPI: fix global memh and mem-map test lifecycle
1 parent 70c9494 commit 02db9dc

29 files changed

Lines changed: 2310 additions & 73 deletions

‎bench/alltoall-global.out‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
faderholdt@clx-gaia-slurm-login-01:~/faderholdt/build-gpu/ucc-tlcuda$ mpirun -np 8 --map-by socket -H dgx-gaia-48:8 -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCC_CLS=hier,basic -x UCC_TLS=cuda,ucp -x UCC_TL_UCP_TUNE=alltoall:cuda:0 /swgwork/faderholdt/build-gpu/install.cuda/bin/ucc_perftest -c alltoall -b 1 -e $((512*1024*1024)) -m cuda -M global -F
2+
Rank 1 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:43:00.0
3+
Rank 2 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:52:00.0
4+
Rank 7 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:DF:00.0
5+
Rank 6 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:D1:00.0
6+
Rank 5 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:C3:00.0
7+
Rank 4 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:9D:00.0
8+
Rank 3 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:61:00.0
9+
Rank 0 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:1B:00.0
10+
Collective: Alltoall
11+
Memory type: cuda
12+
Datatype: float32
13+
Reduction: N/A
14+
Inplace: 0
15+
Warmup:
16+
small 100
17+
large 20
18+
Iterations:
19+
small 1000
20+
large 200
21+
22+
Count Size Time, us Bus Bandwidth, GB/s
23+
avg min max avg max min
24+
8 32 32.10 29.13 34.08 0.00 0.00 0.00
25+
16 64 32.19 29.13 34.25 0.00 0.00 0.00
26+
32 128 32.04 28.99 34.00 0.00 0.00 0.00
27+
64 256 32.29 29.76 34.98 0.01 0.01 0.01
28+
128 512 31.60 28.83 33.98 0.01 0.02 0.01
29+
256 1024 31.59 28.87 33.91 0.03 0.03 0.03
30+
512 2048 31.65 28.83 33.99 0.06 0.06 0.05
31+
1024 4096 31.65 28.83 34.02 0.11 0.12 0.11
32+
2048 8192 31.68 28.90 34.05 0.23 0.25 0.21
33+
4096 16384 31.76 29.00 34.10 0.45 0.49 0.42
34+
8192 32768 31.80 29.05 34.24 0.90 0.99 0.84
35+
16384 65536 31.94 29.14 34.27 1.80 1.97 1.67
36+
32768 131072 32.23 29.50 34.52 3.56 3.89 3.32
37+
65536 262144 31.98 29.48 34.67 7.17 7.78 6.62
38+
131072 524288 32.86 30.57 35.51 13.96 15.00 12.92
39+
262144 1048576 34.98 32.66 37.63 26.23 28.10 24.38
40+
524288 2097152 38.82 35.39 41.01 47.27 51.85 44.75
41+
1048576 4194304 44.78 41.51 47.17 81.96 88.41 77.80
42+
2097152 8388608 56.87 53.72 60.26 129.08 136.65 121.82
43+
4194304 16777216 81.33 78.82 84.35 180.49 186.25 174.04
44+
8388608 33554432 126.63 122.60 129.19 231.85 239.48 227.25
45+
16777216 67108864 204.43 201.99 207.42 287.24 290.72 283.10
46+
33554432 134217728 362.44 358.59 363.80 324.03 327.51 322.82
47+
67108864 268435456 692.96 690.64 694.15 338.95 340.09 338.37
48+
134217728 536870912 1341.49 1339.95 1342.80 350.18 350.58 349.84
49+
268435456 1073741824 2628.25 2627.24 2629.86 357.47 357.61 357.25
50+
536870912 2147483648 5220.45 5217.74 5225.16 359.94 360.13 359.62
51+
1073741824 4294967296 10395.49 10392.17 10399.85 361.51 361.63 361.36
52+
2147483648 8589934592 20739.16 20737.37 20741.37 362.42 362.45 362.38
53+
4294967296 17179869184 41426.86 41423.61 41431.95 362.87 362.89 362.82
54+
Total time: 83.9506 ms
55+
56+
faderholdt@clx-gaia-slurm-login-01:~/faderholdt/build-gpu/ucc-tlcuda$ mpirun -np 8 --map-by socket -H dgx-gaia-48:8 -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCC_CLS=hier,basic -x UCC_TLS=cuda,ucp -x UCC_TL_UCP_TUNE=alltoall:cuda:0 /swgwork/faderholdt/build-gpu/install.cuda/bin/ucc_perftest -c alltoall -b 1 -e $((512*1024*1024)) -m cuda -M local -F
57+
Rank 6 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:D1:00.0
58+
Rank 3 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:61:00.0
59+
Rank 5 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:C3:00.0
60+
Rank 2 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:52:00.0
61+
Rank 1 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:43:00.0
62+
Rank 0 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:1B:00.0
63+
Rank 7 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:DF:00.0
64+
Rank 4 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:9D:00.0
65+
Collective: Alltoall
66+
Memory type: cuda
67+
Datatype: float32
68+
Reduction: N/A
69+
Inplace: 0
70+
Warmup:
71+
small 100
72+
large 20
73+
Iterations:
74+
small 1000
75+
large 200
76+
77+
Count Size Time, us Bus Bandwidth, GB/s
78+
avg min max avg max min
79+
8 32 37.09 36.65 37.70 0.00 0.00 0.00
80+
16 64 37.04 36.59 37.60 0.00 0.00 0.00
81+
32 128 36.93 36.49 37.54 0.00 0.00 0.00
82+
64 256 36.94 36.47 37.54 0.01 0.01 0.01
83+
128 512 36.83 36.32 37.42 0.01 0.01 0.01
84+
256 1024 36.76 36.23 37.35 0.02 0.02 0.02
85+
512 2048 36.97 36.48 37.55 0.05 0.05 0.05
86+
1024 4096 36.85 36.34 37.43 0.10 0.10 0.10
87+
2048 8192 37.02 36.51 37.63 0.19 0.20 0.19
88+
4096 16384 36.98 36.42 37.62 0.39 0.39 0.38
89+
8192 32768 37.05 36.57 37.63 0.77 0.78 0.76
90+
16384 65536 37.25 36.70 37.87 1.54 1.56 1.51
91+
32768 131072 37.51 36.97 38.13 3.06 3.10 3.01
92+
65536 262144 37.85 37.30 38.51 6.06 6.15 5.96
93+
131072 524288 38.58 38.00 39.24 11.89 12.07 11.69
94+
262144 1048576 40.89 40.32 41.51 22.44 22.76 22.10
95+
524288 2097152 43.71 43.09 44.30 41.98 42.59 41.42
96+
1048576 4194304 50.48 49.84 51.09 72.70 73.63 71.84
97+
2097152 8388608 64.01 63.35 64.73 114.68 115.86 113.39
98+
4194304 16777216 89.60 88.97 90.25 163.85 165.01 162.66
99+
8388608 33554432 133.16 132.57 133.67 220.50 221.47 219.65
100+
16777216 67108864 212.99 212.44 213.59 275.69 276.40 274.92
101+
33554432 134217728 371.93 371.36 372.57 315.76 316.25 315.21
102+
67108864 268435456 698.48 697.95 698.88 336.27 336.53 336.08
103+
134217728 536870912 1347.51 1346.94 1348.22 348.62 348.76 348.43
104+
268435456 1073741824 2641.59 2641.20 2642.01 355.67 355.72 355.61
105+
536870912 2147483648 5227.31 5226.91 5227.67 359.47 359.49 359.44
106+
1073741824 4294967296 10410.81 10410.43 10411.15 360.98 360.99 360.97
107+
2147483648 8589934592 20762.13 20761.80 20762.45 362.01 362.02 362.01
108+
4294967296 17179869184 41445.11 41444.72 41445.60 362.71 362.71 362.70
109+
Total time: 84.1144 ms
110+
111+
faderholdt@clx-gaia-slurm-login-01:~/faderholdt/build-gpu/ucc-tlcuda$ mpirun -np 8 --map-by socket -H dgx-gaia-48:8 -x PATH=$PATH -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCC_CLS=hier,basic -x UCC_TLS=cuda,ucp -x UCC_TL_UCP_TUNE=alltoall:cuda:0 /swgwork/faderholdt/build-gpu/install.cuda/bin/ucc_perftest -c alltoall -b 1 -e $((512*1024*1024)) -m cuda -F
112+
Rank 6 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:D1:00.0
113+
Rank 5 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:C3:00.0
114+
Rank 0 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:1B:00.0
115+
Rank 1 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:43:00.0
116+
Rank 4 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:9D:00.0
117+
Rank 2 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:52:00.0
118+
Rank 3 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:61:00.0
119+
Rank 7 : dgx-gaia-48 - NVIDIA H100 80GB HBM3 0000:DF:00.0
120+
Collective: Alltoall
121+
Memory type: cuda
122+
Datatype: float32
123+
Reduction: N/A
124+
Inplace: 0
125+
Warmup:
126+
small 100
127+
large 20
128+
Iterations:
129+
small 1000
130+
large 200
131+
132+
Count Size Time, us Bus Bandwidth, GB/s
133+
avg min max avg max min
134+
8 32 30.48 30.00 30.96 0.00 0.00 0.00
135+
16 64 30.37 29.91 30.86 0.00 0.00 0.00
136+
32 128 30.30 29.81 30.82 0.00 0.00 0.00
137+
64 256 30.25 29.75 30.80 0.01 0.01 0.01
138+
128 512 30.36 29.88 30.88 0.01 0.01 0.01
139+
256 1024 30.33 29.84 30.86 0.03 0.03 0.03
140+
512 2048 30.34 29.81 30.83 0.06 0.06 0.06
141+
1024 4096 30.42 29.89 30.91 0.12 0.12 0.12
142+
2048 8192 30.47 29.93 30.95 0.24 0.24 0.23
143+
4096 16384 30.84 30.36 31.27 0.46 0.47 0.46
144+
8192 32768 30.54 30.01 31.04 0.94 0.96 0.92
145+
16384 65536 31.11 30.58 31.62 1.84 1.88 1.81
146+
32768 131072 30.97 30.43 31.49 3.70 3.77 3.64
147+
65536 262144 32.18 31.65 32.66 7.13 7.25 7.02
148+
131072 524288 32.56 32.05 33.05 14.09 14.31 13.88
149+
262144 1048576 35.27 34.84 35.67 26.01 26.34 25.72
150+
524288 2097152 39.12 38.56 39.64 46.91 47.59 46.29
151+
1048576 4194304 44.40 43.87 44.97 82.66 83.67 81.61
152+
2097152 8388608 53.84 53.42 54.33 136.32 137.39 135.11
153+
4194304 16777216 74.11 73.61 74.54 198.09 199.42 196.94
154+
8388608 33554432 114.87 114.38 115.35 255.60 256.70 254.53
155+
16777216 67108864 195.61 195.06 196.13 300.19 301.04 299.39
156+
33554432 134217728 360.81 360.36 361.26 325.49 325.90 325.09
157+
67108864 268435456 687.18 686.67 687.59 341.81 342.06 341.60
158+
134217728 536870912 1350.69 1350.18 1351.09 347.79 347.92 347.69
159+
268435456 1073741824 2635.59 2635.10 2635.96 356.48 356.54 356.43
160+
536870912 2147483648 5227.17 5226.73 5227.53 359.48 359.51 359.45
161+
1073741824 4294967296 10410.11 10409.66 10410.39 361.00 361.02 360.99
162+
2147483648 8589934592 20748.71 20748.26 20749.22 362.25 362.26 362.24
163+
4294967296 17179869184 41445.51 41445.04 41445.89 362.70 362.71 362.70
164+
Total time: 83.8985 ms

0 commit comments

Comments
 (0)