Repository navigation
Commit e4b18fa
authored
Load a method whose constant name collides with another method's (pytorch#21959)
Replaces pytorch#21706, whose branch history was rewritten by accident and
which GitHub then
closed. Same change, rebuilt on `main`.
## The problem
A program can hold several methods, for example one that fills a cache
from a prompt and
one that runs a single step. The CUDA delegate can share a constant
between them so one
copy of a weight serves every method that uses it. Sharing is keyed on
the constant's
name.
That key is not reliable on its own. AOTInductor names a lifted constant
by its position
within one compiled module, so two methods can each own a
`_tensor_constant2` holding
completely unrelated data. Sharing that pair points both methods at the
same storage, and
one of them then reads the wrong weights.
Until now the first such collision failed the whole load:
```
Constant '_tensor_constant2' in method 'decode' is incompatible with the cached
version from a previous method. Refusing to share.
```
So a valid program, two methods that happen to have same-named
constants, could not be
loaded at all.
## The change
Compare the cached tensor against the one the current method expects.
When they differ,
leave this method's own constant in place rather than failing the load,
and keep the
cached copy for whichever method it does match.
## What is covered, and what is not
The comparison happens where the weights blob is loaded, so it covers
any method that has
at least one constant the cache has not seen yet. That is the common
case.
**A method whose constant names are all already cached is not covered.**
Checking it would
mean re-uploading the weights blob to compare against, and that has to
happen for every
such method, which raises peak device memory enough to fail a large
model. An earlier
version of this change did exactly that and three of the four
`muse-glimmer` end to end
jobs died with `CUDA error: out of memory`. So that case still shares on
the name alone,
as it did before, and the comment at the cache lookup says so. Fixing it
properly needs a
way to read the cached constants' metadata without a blob round trip.
The comparison is also on tensor metadata only: dtype, rank, sizes,
strides, device type
and device index. It does not compare the data, and it cannot, because
both handles point
at device memory. Two constants that agree on all of that while holding
different values
would still be shared. This narrows the problem rather than eliminating
it, and it turns a
hard failure into a working load for the case it does cover.
## Test plan
- Built the delegate and loaded a two-method program whose constant
names collide where
one method has a constant the cache has not seen. Both methods load, and
the log names
the constant that was not shared.
- Loaded a two-method program with genuinely shared constants and
confirmed sharing still
happens, so the change does not disable the optimisation it guards.
- Ran the existing CUDA runtime unit tests.
There is no automated regression test for the collision case. Building
one needs a
two-method program with colliding constant names and a GPU to run it on,
which the unit
test targets in this directory do not have. Reverting this change would
not fail any test
in the repository today.1 parent f982cda commit e4b18fa
1 file changed
Lines changed: 28 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
1061 | 1062 | | |
1062 | 1063 | | |
1063 | 1064 | | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
1064 | 1069 | | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
1065 | 1076 | | |
1066 | 1077 | | |
1067 | 1078 | | |
| |||
1134 | 1145 | | |
1135 | 1146 | | |
1136 | 1147 | | |
1137 | | - | |
1138 | | - | |
1139 | | - | |
1140 | | - | |
1141 | | - | |
1142 | | - | |
1143 | | - | |
1144 | | - | |
1145 | | - | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
1146 | 1161 | | |
1147 | 1162 | | |
| 1163 | + | |
1148 | 1164 | | |
1149 | 1165 | | |
1150 | 1166 | | |
| |||
1173 | 1189 | | |
1174 | 1190 | | |
1175 | 1191 | | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
1176 | 1195 | | |
1177 | 1196 | | |
1178 | 1197 | | |
| |||
0 commit comments