Repository navigation
Commit 91b2e90
authored
Let the CUDA allocator honor any power of two alignment (pytorch#23446)
## Problem
`CudaAllocator::allocate` refused alignments above 256 bytes. TensorRT
asks for 512 bytes when loading an engine, but CUDA guarantees only 256.
Accepting larger alignments must not depend on how a GPU packs its
allocations.
## Change
The allocator now reserves extra memory and rounds the returned pointer
up when the requested alignment exceeds 256 bytes.
| Requested alignment | Before | After |
| --- | --- | --- |
| Power of two, up to 256 bytes | Allocates | Allocates without padding
|
| Power of two, above 256 bytes | `NotSupported` | Allocates with
padding and rounding |
| Zero or not a power of two | `InvalidArgument` | `InvalidArgument` |
Requests above 256 reserve `alignment - 1` extra bytes. Rounding
consumes at most that many bytes. Size overflow returns
`InvalidArgument` before any CUDA call.
Only pointers changed by rounding are recorded. `deallocate` retrieves
and frees the original pointer. A mutex protects the map; an atomic
count keeps deallocation free of that lock when no rounded pointers are
live.
For alignments up to 256, the pointer check remains as a safety net. A
misaligned block is freed and rejected with `NotSupported`, instead of
`MemoryAllocationFailed`. Async allocation and memory pools are
unchanged. The large alignment tests no longer skip on ROCm.
## Validation
20 allocator tests passed on Linux x86_64 with an H100 and CUDA 12.8.
They cover 512 and 4096 byte round trips with mixed sizes live over 64
cycles, concurrent allocations, overflow boundaries, and original
pointer deallocation through weak CUDA stand ins. Compute Sanitizer
reported zero memory errors and zero leaked allocations. Intentional
invalid device API errors were excluded from its API error reporting.
The previous allocator fails four regression tests. Restoring this
implementation passes the full suite again. ROCm and Windows were not
built or run.1 parent 535dd92 commit 91b2e90
3 files changed
Lines changed: 240 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | | - | |
17 | | - | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
| 243 | + | |
248 | 244 | | |
| 245 | + | |
249 | 246 | | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
256 | 252 | | |
257 | 253 | | |
258 | 254 | | |
| |||
293 | 289 | | |
294 | 290 | | |
295 | 291 | | |
296 | | - | |
| 292 | + | |
297 | 293 | | |
298 | 294 | | |
299 | 295 | | |
| |||
309 | 305 | | |
310 | 306 | | |
311 | 307 | | |
312 | | - | |
313 | | - | |
314 | | - | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
315 | 319 | | |
316 | 320 | | |
317 | 321 | | |
318 | 322 | | |
319 | 323 | | |
320 | 324 | | |
321 | 325 | | |
322 | | - | |
| 326 | + | |
323 | 327 | | |
324 | 328 | | |
325 | 329 | | |
| |||
330 | 334 | | |
331 | 335 | | |
332 | 336 | | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
333 | 348 | | |
334 | 349 | | |
335 | 350 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
14 | 18 | | |
15 | 19 | | |
16 | 20 | | |
| |||
25 | 29 | | |
26 | 30 | | |
27 | 31 | | |
| 32 | + | |
28 | 33 | | |
29 | 34 | | |
30 | 35 | | |
| |||
125 | 130 | | |
126 | 131 | | |
127 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
128 | 138 | | |
129 | 139 | | |
130 | 140 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
17 | 22 | | |
18 | 23 | | |
19 | 24 | | |
| |||
23 | 28 | | |
24 | 29 | | |
25 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
26 | 75 | | |
27 | 76 | | |
28 | 77 | | |
| |||
153 | 202 | | |
154 | 203 | | |
155 | 204 | | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
156 | 353 | | |
157 | 354 | | |
158 | 355 | | |
| |||
0 commit comments