Skip to content

Commit 3bc3f4b

Browse files
committed
fix: Keep the trailing zeros of a floatstring range element
Review on #342 pointed out that the string form of a <FloatRangeList> element exists in order to carry the scale it was written with: a range element written '2.50' is asking for `2.50` on a command line, and a renderer handed `2.5` instead has been given a different string. Normalizing the element to the number it denotes threw that away. Split the two kinds of zero, which are not the same thing: - Leading zeros are not part of the number and still go. '02' on an INT range is the task value 2, and forwarding the text renders `--frame 02`. - Trailing zeros are the author's chosen scale and are kept. '02.50' renders `2.50`, not `2.5`. The already-landed conformance fixture EXPR/jobs/expr1.3.4--float-passthrough pins the same rule for a FLOAT parameter default, so both fixtures can now be right at once. A <floatstring> therefore renders as its own text less any redundant leading zeros, which is a substring operation rather than a numeric one. That is what removes the machinery the two preceding commits added, because every problem they were solving came from having to choose a notation in which to re-render a Decimal: - normalize() and quantize() round to getcontext().prec, so an embedding application's decimal context changed what this library rendered. - str(Decimal) switches to exponent notation below 1e-6, so '0.0000001' rendered 1E-7. - format(value, 'f') is exact and plain but unbounded in the exponent, so '1e999999999' -- 11 characters of template text -- expanded to ~10**9 characters and needed a length bound to stay safe. Text needs none of those decisions. '1e999999999' now costs its own 11 characters, so the exponent bound and its boundary cases are gone with it, and an element can no longer silently exceed TaskParameterStringValueAsJob's 1024-character cap and fall through to a numeric member of the union. An <intstring> is unaffected: int() already discards leading zeros and there are no fractional digits to preserve. Verification. Model suite 5527 -> 5529 passed, 24 skipped, 3 xfailed; ruff, black and mypy clean at the CI-pinned versions. Three mutants confirm the tests pin both halves of the rule: dropping the leading-zero strip fails 7 tests, restoring the trailing-zero strip fails 7, and dropping the int() conversion fails 3. Conformance, run by file path against a venv built from this branch: base/jobs/3.4.1.1--int-range-intstring-elements-normalized passes, and base/jobs/3.4.1.2--float-range-floatstring-elements-normalized now emits `W:2.50` from '02.50'. That fixture asserts `W:2.5` and passes only because the runner matches substrings; openjd-specifications#179 carries it and is being corrected. The two controls this must not move -- a FLOAT `default: "3.500"` rendering 3.500, and <float> numeric literals [1.0, 2.5] rendering 1.0 and 2.5 -- both hold. Note that openjd-rs, which backs this package's `openjd.model._v1` API, still normalizes trailing zeros away: it stores a FLOAT range as Vec<f64>, so '02.50' renders 2.5 there. A matching fix is in flight. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
1 parent e7a17b3 commit 3bc3f4b

3 files changed

Lines changed: 86 additions & 132 deletions

File tree

‎src/openjd/model/v2023_09/_model.py‎

Lines changed: 39 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1245,74 +1245,44 @@ def _validate_target_runtime_seconds(cls, value: Any, info: ValidationInfo) -> A
12451245
return validate_int_fmtstring_field(value, ge=0, context=context)
12461246

12471247

1248+
# Redundant leading zeros in the integer digits of a numeric string, keeping any
1249+
# sign and always leaving one digit behind: '02.50' -> '2.50', '007' -> '7',
1250+
# '000' -> '0'. The lookahead is what leaves the last digit, and it also declines
1251+
# to touch '0.50', whose single zero is not redundant.
1252+
_REDUNDANT_LEADING_ZEROS = re.compile(r"^([+-]?)0+(?=[0-9])")
1253+
1254+
12481255
def _normalized_range_element(elem: str, to_int: bool) -> Any:
1249-
"""The number an ``<intstring>``/``<floatstring>`` range element denotes -- an
1250-
``int`` for ``<intstring>``, and for ``<floatstring>`` the plain-notation text
1251-
of the value, since ``str(Decimal)`` cannot render every magnitude without an
1252-
exponent. Returns the element unchanged when it does not denote a number, or
1253-
when its plain-notation form would not fit this field's character cap."""
1256+
"""The value an ``<intstring>``/``<floatstring>`` range element denotes.
1257+
1258+
An ``<intstring>`` becomes the ``int`` it denotes. A ``<floatstring>`` keeps
1259+
its source text, less any redundant leading zeros, because the number of
1260+
fractional digits it was written with is part of what it asks for -- an
1261+
element written ``'2.50'`` is asking for ``2.50`` on the command line, and a
1262+
renderer told ``2.5`` instead has been given a different string. Returns the
1263+
element unchanged when it does not denote a number.
1264+
"""
12541265
try:
12551266
if to_int:
1267+
# int() ignores leading zeros, and an <intstring> has no fractional
1268+
# digits to preserve, so the parsed value carries everything.
12561269
return int(elem)
1257-
value = Decimal(elem)
1258-
if not value.is_finite():
1259-
# Decimal accepts 'nan'/'inf', which are not <floatstring>s; leave
1260-
# them as text rather than changing how they render.
1261-
return elem
1262-
# Bound the expansion from the exponent before performing it. Decimal
1263-
# construction from a string is unbounded -- the context's Emax/Emin
1264-
# constrain arithmetic results, not construction, so Decimal('1e999999999')
1265-
# is finite and the guard above does not stop it -- and the plain-notation
1266-
# length is linear in the exponent, so 11 characters of template text
1267-
# would otherwise materialize ~10**9 characters.
1268-
#
1269-
# The bound is the character cap this field already enforces, because an
1270-
# expansion past it cannot be carried here as text at all: the value is
1271-
# stored in a TaskRangeList element, whose str member is
1272-
# TaskParameterStringValueAsJob, and pydantic's union falls through to
1273-
# the numeric members instead, rendering the element as an integer, or as
1274-
# inf, or as 0.0. Such an element keeps its source text -- what this
1275-
# function already does for anything it cannot normalize, and how it
1276-
# rendered before normalization was introduced.
1277-
exponent = cast(int, value.as_tuple().exponent) # finite: never 'n'/'N'/'F'
1278-
# Length of the '[-]<integer>.<fraction>' text returned below. Stripping
1279-
# redundant zeros can only shorten it, so this is an upper bound.
1280-
expansion_len = (
1281-
(1 if value.is_signed() else 0)
1282-
+ max(value.adjusted() + 1, 1) # integer digits, at least one
1283-
+ 1 # the decimal point
1284-
+ max(-exponent, 1) # fraction digits, at least one
1285-
)
1286-
if expansion_len > _MAX_TASK_PARAM_VALUE_LEN:
1287-
return elem
1288-
# format(value, 'f') is exact and plain at every magnitude: with no
1289-
# precision in the format spec it neither rounds to getcontext().prec
1290-
# nor switches to exponent notation, so an embedding app that sets a
1291-
# different context cannot change what this library renders. Neither
1292-
# holds of the obvious alternatives -- normalize() rounds to the
1293-
# ambient precision, str() switches to exponent notation below 1e-6,
1294-
# and quantize() raises InvalidOperation once the result needs more
1295-
# digits than the ambient precision allows.
1296-
text = format(value, "f")
1297-
negative = text.startswith("-")
1298-
integer, _, fraction = text.lstrip("-").partition(".")
1299-
# A <floatstring> denotes a number, so redundant zeros in its source
1300-
# text are not part of the value: '02.50' is 2.5. One fractional digit
1301-
# is always kept, matching openjd-rs, which renders the float 1.0 as
1302-
# `1.0` and never as `1`.
1303-
integer = integer.lstrip("0") or "0"
1304-
fraction = fraction.rstrip("0") or "0"
1305-
if negative and integer == "0" and fraction == "0":
1306-
# The number denoted by '-0.0' is zero, which has no sign; openjd-rs
1307-
# renders it `0.0`.
1308-
negative = False
1309-
return f"{'-' if negative else ''}{integer}.{fraction}"
1310-
except (ValueError, ArithmeticError):
1311-
# Not a number at all — e.g. a format string that resolved to
1270+
# A <floatstring> renders as text, so the parse is only a check that the
1271+
# element denotes a float at all. Deliberately not carried in a Decimal:
1272+
# rendering one requires choosing a notation, and every choice available
1273+
# is either context-sensitive (``normalize`` and ``quantize`` round to
1274+
# ``getcontext().prec``), notation-unstable (``str`` switches to an
1275+
# exponent below 1e-6), or unbounded in the exponent (``format(v, 'f')``
1276+
# expands '1e999999999' to ~10**9 characters). Text needs none of those
1277+
# decisions: it renders as the author wrote it.
1278+
float(elem)
1279+
except ValueError:
1280+
# Not a number at all -- e.g. a format string that resolved to
13121281
# non-numeric text. A literal range is already checked against its
13131282
# element type at template parse time, so leave such a value to the
13141283
# existing behaviour rather than adding a rejection path here.
13151284
return elem
1285+
return _REDUNDANT_LEADING_ZEROS.sub(r"\1", elem)
13161286

13171287

13181288
# Target model for task parameters when instantiating a job.
@@ -1330,10 +1300,15 @@ def _normalize_numeric_range_elements(cls, value: Any, info: ValidationInfo) ->
13301300
# §3.4.1.1/§3.4.1.2: an <IntRangeList> element is `<integer> |
13311301
# <intstring>` and a <FloatRangeList> element is `<float> |
13321302
# <floatstring>`, where an <intstring>/<floatstring> is "a string whose
1333-
# value is the string representation of" a number (§2.3, §2.4). Such an
1334-
# element therefore denotes that number, not its source text, so '02'
1335-
# is the task value 2 and '02.50' is 2.5. Python was keeping the source
1336-
# text, which reaches a task command line as `--frame 02`.
1303+
# value is the string representation of" a number (§2.3, §2.4). Leading
1304+
# zeros are not part of the number such an element denotes, and Python
1305+
# was keeping them, which reaches a task command line as `--frame 02`.
1306+
#
1307+
# Trailing zeros are kept. They are the reason the string form exists:
1308+
# a <floatstring> is how a template asks for a fixed number of decimal
1309+
# places, so '2.50' must render `2.50` and not `2.5`. The already-landed
1310+
# EXPR/jobs/expr1.3.4--float-passthrough fixture pins the same rule for
1311+
# a FLOAT parameter default.
13371312
#
13381313
# Applied on the instantiation target so every inbound range path is
13391314
# covered: a literal list, a range-expression expansion, and an RFC 0006

‎test/openjd/model_v0/test_step_param_space_iter.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -613,8 +613,9 @@ def test_int_range_intstring_elements(self) -> None:
613613

614614
def test_float_range_floatstring_elements(self) -> None:
615615
# WHEN a FLOAT range is written with <floatstring> elements
616-
# THEN each task gets the number the element denotes
617-
assert self._task_values("FLOAT", ["1.5", "02.50"]) == ["1.5", "2.5"]
616+
# THEN each task gets the element's own text, less redundant leading
617+
# zeros. The decimal places it was written with are what it asks for.
618+
assert self._task_values("FLOAT", ["1.5", "02.50", "3.500"]) == ["1.5", "2.50", "3.500"]
618619

619620
def test_float_range_numeric_literals_keep_their_scale(self) -> None:
620621
# A <float> literal is not a string representation, so it keeps the scale

‎test/openjd/model_v0/v2023_09/test_parameter_space.py‎

Lines changed: 44 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -444,10 +444,17 @@ class TestRangeListElementNormalization:
444444
"""§3.4.1.1/§3.4.1.2: an `<IntRangeList>` element is `<integer> | <intstring>`
445445
and a `<FloatRangeList>` element is `<float> | <floatstring>`. An
446446
`<intstring>`/`<floatstring>` is "a string whose value is the string
447-
representation of" a number (§2.3, §2.4), so it denotes that number and not
448-
its source text — a range of `['1', '02', '003']` is the task values 1, 2 and
449-
3. These values reach a task command line, so keeping the text renders
450-
`--frame 02`.
447+
representation of" a number (§2.3, §2.4).
448+
449+
Leading zeros are not part of the number, so they go: a range of
450+
`['1', '02', '003']` is the task values 1, 2 and 3, and keeping the text
451+
renders `--frame 02` on a command line.
452+
453+
Trailing zeros stay. They are why the string form exists — a `<floatstring>`
454+
is how a template asks for a fixed number of decimal places, so `'2.50'`
455+
renders `2.50`, and a renderer handed `2.5` has been given a different
456+
string. `EXPR/jobs/expr1.3.4--float-passthrough` pins the same rule for a
457+
FLOAT parameter default.
451458
"""
452459

453460
@pytest.mark.parametrize(
@@ -471,96 +478,67 @@ def test_intstring_elements_carry_their_value(self, param_type: str) -> None:
471478
@pytest.mark.parametrize(
472479
"element,expected",
473480
(
474-
pytest.param("1.5", "1.5", id="no normalization needed"),
475-
pytest.param("02.50", "2.5", id="leading and trailing zeros"),
476-
pytest.param("3.500", "3.5", id="trailing zeros"),
477-
pytest.param("-02.50", "-2.5", id="negative"),
478-
# An integral float keeps one fractional digit. openjd-rs renders
479-
# the <floatstring> '1.0' as `1.0` and '007' as `7.0`, so dropping
480-
# the fraction would make the same number render two ways depending
481-
# on whether it was written as a <float> or a <floatstring>.
482-
pytest.param("1.0", "1.0", id="integral float"),
481+
pytest.param("1.5", "1.5", id="nothing to strip"),
482+
# The rule, in both directions at once.
483+
pytest.param("02.50", "2.50", id="leading zero goes, trailing zero stays"),
484+
pytest.param("3.500", "3.500", id="trailing zeros stay"),
485+
pytest.param("-02.50", "-2.50", id="negative"),
486+
pytest.param("+02.50", "+2.50", id="explicit plus sign is kept"),
487+
pytest.param("007", "7", id="leading zeros on a whole number"),
488+
pytest.param("100", "100", id="a zero that is a significant digit"),
489+
# A single leading zero before the point is not redundant, and the
490+
# last digit of an all-zero integer part is not either.
491+
pytest.param("0.50", "0.50", id="the necessary leading zero stays"),
492+
pytest.param("000", "0", id="all zeros keeps one"),
493+
pytest.param("0", "0", id="bare zero"),
483494
pytest.param("0.0", "0.0", id="zero"),
484-
pytest.param("-0.0", "0.0", id="negative zero has no sign"),
485-
pytest.param("007", "7.0", id="leading zeros on a whole number"),
486-
pytest.param("100", "100.0", id="whole number"),
487-
# Exponent notation must never reach a task command line, at any
488-
# magnitude and in either direction.
489-
pytest.param("1E+2", "100.0", id="exponent notation in the source"),
490-
pytest.param(
491-
"0.0000001",
492-
"0.0000001",
493-
id="below 1e-6, where Decimal.__str__ would give 1E-7",
494-
),
495-
pytest.param(
496-
"1E+30",
497-
"1" + "0" * 30 + ".0",
498-
id="more digits than the default decimal precision",
499-
),
495+
pytest.param("0.00", "0.00", id="zero keeps its trailing zeros too"),
496+
pytest.param("1.0", "1.0", id="integral float keeps its point"),
497+
# Exponent notation is the author's own text and is forwarded. The
498+
# FLOAT parameter default path does the same for `default: "1E+2"`.
499+
pytest.param("1E+2", "1E+2", id="exponent notation"),
500+
pytest.param("01E+2", "1E+2", id="leading zero on an exponent form"),
501+
pytest.param("0.0000001", "0.0000001", id="below 1e-6"),
500502
pytest.param(
501503
"1.2345678901234567890123456789012345678901",
502504
"1.2345678901234567890123456789012345678901",
503-
id="more significant digits than the default decimal precision",
505+
id="more significant digits than a float can hold",
504506
),
505507
),
506508
)
507-
def test_floatstring_elements_carry_their_value(self, element: str, expected: str) -> None:
509+
def test_floatstring_elements_keep_their_scale(self, element: str, expected: str) -> None:
508510
# WHEN the instantiation target parses a float range holding a <floatstring>
509511
model = _parse_model(
510512
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
511513
)
512514

513-
# THEN the element renders as the number it denotes
515+
# THEN it renders as written, less any redundant leading zeros
514516
assert [str(v) for v in model.range] == [expected]
515517

516518
@pytest.mark.parametrize(
517519
"element",
518520
(
519-
# Decimal's context bounds arithmetic results, not construction from
520-
# a string, so these are finite: 11 characters of template text
521-
# whose plain-notation expansion is ~10**9 characters.
522521
pytest.param("1e999999999", id="huge positive exponent"),
523522
pytest.param("1e-999999999", id="huge negative exponent"),
524-
# One character past the cap in each direction: both expand to 1025.
525-
pytest.param("1E+1022", id="one past the cap, positive exponent"),
526-
pytest.param("1E-1023", id="one past the cap, negative exponent"),
523+
pytest.param("1E+1022", id="positive exponent past the field cap"),
524+
pytest.param("1E-1023", id="negative exponent past the field cap"),
527525
),
528526
)
529-
def test_element_too_long_to_expand_keeps_its_source_text(self, element: str) -> None:
527+
def test_a_huge_exponent_costs_only_its_own_characters(self, element: str) -> None:
530528
# WHEN the instantiation target parses a <floatstring> whose plain-notation
531-
# form would not fit TaskParameterStringValueAsJob's 1024-character cap
529+
# form would be enormous — 11 characters of template text denoting a
530+
# number that needs ~10**9 digits to write out without an exponent
532531
model = _parse_model(
533532
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
534533
)
535534

536-
# THEN it keeps its source text rather than being expanded. Expanding it
537-
# would allocate the whole expansion, and the result could not be carried
538-
# as text here anyway: it fails the str member of TaskRangeList's union,
539-
# and pydantic then falls through to the numeric members, rendering the
540-
# element as an integer, or as inf, or as 0.0.
535+
# THEN it renders as its own text. Nothing expands it, so no bound on the
536+
# exponent is needed to keep a template from allocating ~10**9 characters,
537+
# and the value cannot silently exceed TaskParameterStringValueAsJob's
538+
# 1024-character cap and fall through to a numeric member of the union.
541539
assert [str(v) for v in model.range] == [element]
542540

543-
@pytest.mark.parametrize(
544-
"element,expected",
545-
(
546-
pytest.param("1E+1021", "1" + "0" * 1021 + ".0", id="positive exponent"),
547-
pytest.param("1E-1022", "0." + "0" * 1021 + "1", id="negative exponent"),
548-
),
549-
)
550-
def test_element_at_the_cap_is_still_normalized(self, element: str, expected: str) -> None:
551-
# GIVEN a <floatstring> whose plain-notation form is exactly 1024
552-
# characters — at the cap, not past it
553-
assert len(expected) == 1024
554-
555-
# WHEN the instantiation target parses it
556-
model = _parse_model(
557-
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
558-
)
559-
560-
# THEN the length bound has not cost it its normalization
561-
assert [str(v) for v in model.range] == [expected]
562-
563-
def test_floatstring_normalization_ignores_the_decimal_context(self) -> None:
541+
def test_floatstring_rendering_ignores_the_decimal_context(self) -> None:
564542
# GIVEN an embedding application that has narrowed the process-wide
565543
# decimal context, and a <floatstring> with more significant digits
566544
# than that precision allows

0 commit comments

Comments
 (0)