Skip to content

Commit e4b18fa

Browse files
authored
Load a method whose constant name collides with another method's (pytorch#21959)
Replaces pytorch#21706, whose branch history was rewritten by accident and which GitHub then closed. Same change, rebuilt on `main`. ## The problem A program can hold several methods, for example one that fills a cache from a prompt and one that runs a single step. The CUDA delegate can share a constant between them so one copy of a weight serves every method that uses it. Sharing is keyed on the constant's name. That key is not reliable on its own. AOTInductor names a lifted constant by its position within one compiled module, so two methods can each own a `_tensor_constant2` holding completely unrelated data. Sharing that pair points both methods at the same storage, and one of them then reads the wrong weights. Until now the first such collision failed the whole load: ``` Constant '_tensor_constant2' in method 'decode' is incompatible with the cached version from a previous method. Refusing to share. ``` So a valid program, two methods that happen to have same-named constants, could not be loaded at all. ## The change Compare the cached tensor against the one the current method expects. When they differ, leave this method's own constant in place rather than failing the load, and keep the cached copy for whichever method it does match. ## What is covered, and what is not The comparison happens where the weights blob is loaded, so it covers any method that has at least one constant the cache has not seen yet. That is the common case. **A method whose constant names are all already cached is not covered.** Checking it would mean re-uploading the weights blob to compare against, and that has to happen for every such method, which raises peak device memory enough to fail a large model. An earlier version of this change did exactly that and three of the four `muse-glimmer` end to end jobs died with `CUDA error: out of memory`. So that case still shares on the name alone, as it did before, and the comment at the cache lookup says so. Fixing it properly needs a way to read the cached constants' metadata without a blob round trip. The comparison is also on tensor metadata only: dtype, rank, sizes, strides, device type and device index. It does not compare the data, and it cannot, because both handles point at device memory. Two constants that agree on all of that while holding different values would still be shared. This narrows the problem rather than eliminating it, and it turns a hard failure into a working load for the case it does cover. ## Test plan - Built the delegate and loaded a two-method program whose constant names collide where one method has a constant the cache has not seen. Both methods load, and the log names the constant that was not shared. - Loaded a two-method program with genuinely shared constants and confirmed sharing still happens, so the change does not disable the optimisation it guards. - Ran the existing CUDA runtime unit tests. There is no automated regression test for the collision case. Building one needs a two-method program with colliding constant names and a GPU to run it on, which the unit test targets in this directory do not have. Reverting this change would not fail any test in the repository today.
1 parent f982cda commit e4b18fa

1 file changed

Lines changed: 28 additions & 9 deletions

File tree

‎backends/cuda/runtime/cuda_backend.cpp‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include <string>
2727
#include <string_view>
2828
#include <unordered_map>
29+
#include <unordered_set>
2930
#include <vector>
3031

3132
// Include SlimTensor headers for CUDA backend
@@ -1061,7 +1062,17 @@ class ET_EXPERIMENTAL CudaBackend final
10611062
}
10621063
}
10631064

1065+
// Names this method must load itself because the cached tensor under the
1066+
// same name belongs to a different constant.
1067+
std::unordered_set<std::string> not_shared;
1068+
10641069
// Phase 2 (locked): pure cache lookup against shared_constant_tensors_.
1070+
// A cache hit here is provisional: the name matches, but whether the
1071+
// tensors match is only known after extraction, so a hit can still be
1072+
// rejected below. Only the path that loads the blob can check this. When
1073+
// every name is already cached there is nothing to compare against
1074+
// without re-uploading the blob, which costs too much device memory to
1075+
// do on every method, so that case still shares on the name alone.
10651076
{
10661077
std::lock_guard<std::mutex> guard(shared_constants_mutex_);
10671078
for (const auto& [fqn, _] : fqn_to_name) {
@@ -1134,17 +1145,22 @@ class ET_EXPERIMENTAL CudaBackend final
11341145
if (cached_it == shared_constant_tensors_.end()) {
11351146
// New constant — add to cache.
11361147
shared_constant_tensors_[fqn] = extracted_it->second;
1137-
} else {
1138-
// Same FQN seen before — verify the cached tensor is still
1139-
// compatible with what THIS method expects. On mismatch the
1140-
// helper logs the offending field and returns an error.
1141-
ET_CHECK_OK_OR_RETURN_ERROR(
1142-
check_cached_constant_match(
1143-
fqn, cached_it->second, extracted_it->second),
1144-
"Constant '%s' in method '%s' is incompatible with the "
1145-
"cached version from a previous method. Refusing to share.",
1148+
} else if (
1149+
check_cached_constant_match(
1150+
fqn, cached_it->second, extracted_it->second) != Error::Ok) {
1151+
// Same name, different tensor. AOTInductor names a lifted constant
1152+
// by position within one compiled module, so two methods each own a
1153+
// "_tensor_constant2" holding unrelated data, and a name is only
1154+
// evidence of sharing when the tensors agree. Sharing this pair
1155+
// would alias mismatched storage, so leave this method's own
1156+
// constant in place and keep the cached one for whoever matches it.
1157+
ET_LOG(
1158+
Info,
1159+
"Constant '%s' in method '%s' differs from the cached copy; "
1160+
"using this method's own copy instead of sharing",
11461161
fqn.c_str(),
11471162
method_name.c_str());
1163+
not_shared.insert(fqn);
11481164
}
11491165
}
11501166
ET_LOG(
@@ -1173,6 +1189,9 @@ class ET_EXPERIMENTAL CudaBackend final
11731189
{
11741190
std::lock_guard<std::mutex> guard(shared_constants_mutex_);
11751191
for (const auto& [fqn, internal_name] : fqn_to_name) {
1192+
if (not_shared.count(fqn) != 0) {
1193+
continue;
1194+
}
11761195
auto it = shared_constant_tensors_.find(fqn);
11771196
if (it != shared_constant_tensors_.end()) {
11781197
pairs.push_back({internal_name.c_str(), it->second});

0 commit comments

Comments
 (0)