Skip to content

Commit fd5226a

Browse files
authored
Sign the positive infinity token so flatc 1.12 accepts it
Summary: The non-finite float rewrite added in the parent diff spells positive infinity as a bare `inf`, which fails to lower under the `flatc` shipped in fbsource (1.12.0). `flatc` parses a `VkValue` union member's value before it knows the member's type, so a token that starts with a letter (`inf`, `nan`) is rejected as an unknown value: ``` schema.json:1: 128: error: cannot parse value starting with: inf ``` A signed token takes the numeric path and parses, so this emits `+inf` for positive infinity (`-inf` was already correct). `flatc` writes the bare `inf` back on decompile, which the existing reverse rewrite already maps to json's `Infinity`, so only the forward direction needed the fix. This unbreaks `test_serialize_deserialize_non_finite_scalars` and `test_serialize_deserialize_non_finite_floats_in_list`. Reviewed By: rascani Differential Revision: D119675660 Pull Request resolved: pytorch#22770
1 parent f6b333c commit fd5226a

1 file changed

Lines changed: 11 additions & 5 deletions

File tree

‎backends/vulkan/serialization/vulkan_graph_serialize.py‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,19 +29,25 @@
2929

3030

3131
# Python's json module spells the non-finite floats "Infinity" / "-Infinity" /
32-
# "NaN"; flatc spells the infinities "inf" / "-inf" and rejects Python's
32+
# "NaN"; flatc spells the infinities "+inf" / "-inf" and rejects Python's
3333
# spelling, so both directions need translating. A graph carries a non-finite
3434
# scalar whenever the model does -- the -inf fill value of a transformer
3535
# attention mask is the common case -- and without this the failure surfaces as
3636
# a flatc byte offset into a temporary file rather than anything pointing at
3737
# the graph.
3838
#
39+
# The infinity token must carry an explicit sign. flatc parses a union member's
40+
# value before it knows the member's type, so a bare "inf" -- a token that
41+
# starts with a letter -- is rejected as an unknown value, while "+inf" / "-inf"
42+
# take the numeric path and parse. flatc emits the bare "inf" back on decompile,
43+
# which the reverse rewrite maps to json's "Infinity".
44+
#
3945
# The rewrite runs over the serialized text rather than over the encoder's
4046
# chunks: json only emits a float as a chunk of its own inside an object, and
4147
# inside a list the chunk carries the delimiter with it ("[-Infinity"), so
4248
# matching whole chunks silently missed every DoubleList.
4349
_JSON_STRING_RE = re.compile(r'"(?:[^"\\]|\\.)*"')
44-
_PY_NONFINITE_RE = re.compile(r"(?<![\w.])(Infinity|NaN)(?![\w.])")
50+
_PY_NONFINITE_RE = re.compile(r"(?<![\w.])(-?)(Infinity|NaN)(?![\w.])")
4551
_FLATC_INF_RE = re.compile(r"(?<![\w.])inf(?![\w.])")
4652

4753

@@ -62,19 +68,19 @@ def _rewrite_outside_strings(text: str, sub) -> str:
6268

6369

6470
def _python_json_to_flatc_json(text: str) -> str:
65-
"""Rewrite json's ``Infinity`` tokens into the ``inf`` flatc accepts."""
71+
"""Rewrite json's ``Infinity`` tokens into the ``+inf`` / ``-inf`` flatc accepts."""
6672
if "Infinity" not in text and "NaN" not in text:
6773
return text
6874

6975
def replace(m: "re.Match[str]") -> str:
70-
if m.group(1) == "NaN":
76+
if m.group(2) == "NaN":
7177
raise ValueError(
7278
"Cannot serialize a NaN float value into a Vulkan graph: "
7379
"flatc rejects every spelling of NaN for a value inside a "
7480
"union, and every float in the Vulkan schema is a member of "
7581
"the VkValue union."
7682
)
77-
return "inf"
83+
return f"{m.group(1) or '+'}inf"
7884

7985
return _rewrite_outside_strings(
8086
text, lambda segment: _PY_NONFINITE_RE.sub(replace, segment)

0 commit comments

Comments
 (0)