Skip to content

Commit 6a315a4

Browse files
authored
Load an older PTD file in the Python reader instead of rejecting it (pytorch#22117)
### Summary The Python `FlatTensorSerializer.deserialize` refused any PTD file whose `version` did not **exactly** equal `_FLAT_TENSOR_VERSION`: ```python if flat_tensor.version != _FLAT_TENSOR_VERSION: raise NotImplementedError(...) ``` That rejects an **older** file too, which contradicts: - the append-only schema policy in `schema/README.md` (older files stay loadable), and - the C++ runtime readers, which accept anything `<=` their supported version and only refuse a file **newer** than they understand (`Program::load`, `FlatTensorDataMap::load`). This aligns the Python PTD reader with that policy: compare with `>` instead of `!=`, so an older or equal file loads and only a newer one is refused. The error message now says the file is newer than this reader supports. ### Context This was called out as a follow-up during review of pytorch#22114 (which adds the `<=` gate on the C++ side). It keeps the Python export/tooling path consistent so a future non-zero PTD version won't regress it. ### Test plan Two tests in `extension/flat_tensor/test/test_serialize.py`: - `test_deserialize_refuses_newer_version` — a bumped-version file is refused. - `test_deserialize_accepts_older_version` — a file older than the reader still loads (the case the old `!=` wrongly rejected). Both are behavior-only; no schema or format change. Version constants are unchanged (still `0`), so this is inert for existing files and only changes behavior once a non-zero version is ever stamped.
1 parent d181029 commit 6a315a4

2 files changed

Lines changed: 51 additions & 3 deletions

File tree

‎extension/flat_tensor/serialize/serialize.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -404,10 +404,12 @@ def deserialize(self, blob: Cord) -> DataPayload:
404404
flat_tensor_bytes = data[0 : header.flatbuffer_offset + header.flatbuffer_size]
405405
flat_tensor = _deserialize_to_flat_tensor(flat_tensor_bytes)
406406

407-
# Verify that this is a supported version.
408-
if flat_tensor.version != _FLAT_TENSOR_VERSION:
407+
# Verify that this is a supported version. Older files still load; only
408+
# a file newer than this reader understands is refused, matching the
409+
# runtime readers and the append-only schema policy in schema/README.md.
410+
if flat_tensor.version > _FLAT_TENSOR_VERSION:
409411
raise NotImplementedError(
410-
f"Flat tensor files reports unsupported version {flat_tensor.version}. Expected {_FLAT_TENSOR_VERSION}."
412+
f"Flat tensor file reports version {flat_tensor.version}, which is newer than this reader supports (max {_FLAT_TENSOR_VERSION})."
411413
)
412414

413415
# Extract the buffers.

‎extension/flat_tensor/test/test_serialize.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import unittest
1414

1515
from typing import Dict, List, Optional
16+
from unittest import mock
1617

1718
import torch
1819

@@ -31,6 +32,7 @@
3132

3233
from executorch.extension.flat_tensor.serialize.serialize import (
3334
_deserialize_to_flat_tensor,
35+
_FLAT_TENSOR_VERSION,
3436
_FLATBUFFER_ALIGNMENT,
3537
FlatTensorConfig,
3638
FlatTensorHeader,
@@ -282,6 +284,50 @@ def test_round_trip(self) -> None:
282284
TEST_DATA_PAYLOAD.named_data, deserialized_payload.named_data
283285
)
284286

287+
def test_deserialize_refuses_newer_version(self) -> None:
288+
# A file whose version is newer than this reader understands must be
289+
# refused, rather than silently misread. Serialize with a bumped
290+
# version, then deserialize with the real (lower) reader version.
291+
config = FlatTensorConfig()
292+
serializer: DataSerializer = FlatTensorSerializer(config)
293+
294+
with mock.patch(
295+
"executorch.extension.flat_tensor.serialize.serialize._FLAT_TENSOR_VERSION",
296+
_FLAT_TENSOR_VERSION + 1,
297+
):
298+
serialized_data = bytes(serializer.serialize(TEST_DATA_PAYLOAD))
299+
300+
with self.assertRaises(NotImplementedError):
301+
serializer.deserialize(Cord(serialized_data))
302+
303+
def test_deserialize_accepts_older_version(self) -> None:
304+
# Back-compat: a file older than the reader must still load, and load
305+
# correctly. Serialize at the current version, then deserialize with the
306+
# reader's version bumped higher, so the file looks older than the
307+
# reader. This is what `>` buys over the previous `!=`, which rejected
308+
# any mismatch including older.
309+
config = FlatTensorConfig()
310+
serializer: DataSerializer = FlatTensorSerializer(config)
311+
serialized_data = bytes(serializer.serialize(TEST_DATA_PAYLOAD))
312+
313+
with mock.patch(
314+
"executorch.extension.flat_tensor.serialize.serialize._FLAT_TENSOR_VERSION",
315+
_FLAT_TENSOR_VERSION + 1,
316+
):
317+
deserialized_payload = serializer.deserialize(Cord(serialized_data))
318+
319+
# The older file must not just load without raising; it must round-trip
320+
# intact, so a regression that loads but drops data can't pass silently.
321+
for i in range(len(deserialized_payload.buffers)):
322+
self.assertEqual(
323+
TEST_DATA_PAYLOAD.buffers[i],
324+
deserialized_payload.buffers[i],
325+
f"Buffer at index {i} does not match.",
326+
)
327+
self._check_named_data_entries(
328+
TEST_DATA_PAYLOAD.named_data, deserialized_payload.named_data
329+
)
330+
285331
def test_file_backed_data_matches_bytes(self) -> None:
286332
from executorch.exir._serialize._cord import FileBackedData
287333

0 commit comments

Comments
 (0)