Skip to content

Commit af89643

Browse files
committed
fix: Keep the trailing zeros of a floatstring range element
Review on #342 pointed out that the string form of a <FloatRangeList> element exists in order to carry the scale it was written with: a range element written '2.50' is asking for `2.50` on a command line, and a renderer handed `2.5` instead has been given a different string. It is also the only way a template can ask for a fixed number of decimal places, because a <float> literal of 2.50 is the same literal as 2.5 once parsed. #342 normalized the element to the number it denotes and threw that away. Split the two kinds of zero, which are not the same thing: - Leading zeros are not part of the number and still go. '02' on an INT range is the task value 2, and forwarding the text renders `--frame 02`. - Trailing zeros are the author's chosen scale and are kept. '02.50' renders `2.50`, not `2.5`. The already-landed conformance fixture EXPR/jobs/expr1.3.4--float-passthrough pins the same rule for a FLOAT parameter default, so both it and base/jobs/3.4.1.2 can now be right at once. openjd-specifications#180 states the rule as Template Schemas §7.5. A <floatstring> therefore renders as its own text less any redundant leading zeros, which is a substring operation rather than a numeric one. That is what removes the machinery the two preceding commits added, because every problem they were solving came from having to choose a notation in which to re-render a Decimal: - normalize() and quantize() round to getcontext().prec, so an embedding application's decimal context changed what this library rendered: measured, '1.2345678901234567890123456789012345678901' came back as '1.2346' under getcontext().prec = 5. - str(Decimal) switches to exponent notation once the adjusted exponent falls below -6, so '0.0000001' rendered 1E-7. - format(value, 'f') is exact and plain at every magnitude but unbounded in the exponent, so '1e999999999' -- 11 characters of template text -- expanded to ~10**9 characters and needed a length bound to stay safe. Text needs none of those decisions. '1e999999999' now costs its own 11 characters, so the exponent bound and its boundary cases are gone with it, and an element can no longer silently exceed TaskParameterStringValueAsJob's 1024-character cap and fall through to a numeric member of the union. An <intstring> is unaffected: int() already discards leading zeros and there are no fractional digits to preserve. Verification. Model suite 5527 -> 5529 passed, 24 skipped, 3 xfailed; ruff, black and mypy clean at the CI-pinned versions. Three mutants confirm the tests pin both halves of the rule: dropping the leading-zero strip fails 7 tests, restoring the trailing-zero strip fails 7, and dropping the int() conversion fails 3. Full 2023-09 conformance suite, run from source against openjd-cli mainline 7c7ece4 with openjd-sessions 0.10.14, on openjd-specifications#180's tree: 1162 passed, 0 failed. The baseline at e7a17b3 passes 1161 and fails only base/jobs/7.5--numeric-string-zeros-in-range-elements, so this closes that one fixture and moves nothing else. Both controls hold -- base/jobs/3.4--float-parameter and EXPR/jobs/expr1.3.4--float-passthrough. Relative to released 0.11.6 the only behaviour change here is the leading-zero trim: every element whose rendering differs from 0.11.6 differs by a removed leading zero. openjd-rs, which backs this package's `openjd.model._v1` API, needs the matching change; that is OpenJobDescription/openjd-rs#354. With both in place the two implementations render every range-element case tested identically, including the zero and exponent forms. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
1 parent e7a17b3 commit af89643

3 files changed

Lines changed: 68 additions & 153 deletions

File tree

‎src/openjd/model/v2023_09/_model.py‎

Lines changed: 26 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -1245,74 +1245,29 @@ def _validate_target_runtime_seconds(cls, value: Any, info: ValidationInfo) -> A
12451245
return validate_int_fmtstring_field(value, ge=0, context=context)
12461246

12471247

1248+
# '02.50' -> '2.50', '007' -> '7', '000' -> '0'. The lookahead leaves the last
1249+
# digit, so '0.50' keeps the zero that is its integer part.
1250+
_REDUNDANT_LEADING_ZEROS = re.compile(r"^([+-]?)0+(?=[0-9])")
1251+
1252+
12481253
def _normalized_range_element(elem: str, to_int: bool) -> Any:
1249-
"""The number an ``<intstring>``/``<floatstring>`` range element denotes -- an
1250-
``int`` for ``<intstring>``, and for ``<floatstring>`` the plain-notation text
1251-
of the value, since ``str(Decimal)`` cannot render every magnitude without an
1252-
exponent. Returns the element unchanged when it does not denote a number, or
1253-
when its plain-notation form would not fit this field's character cap."""
1254+
"""The value an ``<intstring>``/``<floatstring>`` range element denotes.
1255+
1256+
An ``<intstring>`` becomes an ``int``. A ``<floatstring>`` keeps its text less
1257+
redundant leading zeros, so the decimal places it was written with survive
1258+
(§7.5). Returns the element unchanged when it does not denote a number.
1259+
"""
12541260
try:
12551261
if to_int:
1256-
return int(elem)
1257-
value = Decimal(elem)
1258-
if not value.is_finite():
1259-
# Decimal accepts 'nan'/'inf', which are not <floatstring>s; leave
1260-
# them as text rather than changing how they render.
1261-
return elem
1262-
# Bound the expansion from the exponent before performing it. Decimal
1263-
# construction from a string is unbounded -- the context's Emax/Emin
1264-
# constrain arithmetic results, not construction, so Decimal('1e999999999')
1265-
# is finite and the guard above does not stop it -- and the plain-notation
1266-
# length is linear in the exponent, so 11 characters of template text
1267-
# would otherwise materialize ~10**9 characters.
1268-
#
1269-
# The bound is the character cap this field already enforces, because an
1270-
# expansion past it cannot be carried here as text at all: the value is
1271-
# stored in a TaskRangeList element, whose str member is
1272-
# TaskParameterStringValueAsJob, and pydantic's union falls through to
1273-
# the numeric members instead, rendering the element as an integer, or as
1274-
# inf, or as 0.0. Such an element keeps its source text -- what this
1275-
# function already does for anything it cannot normalize, and how it
1276-
# rendered before normalization was introduced.
1277-
exponent = cast(int, value.as_tuple().exponent) # finite: never 'n'/'N'/'F'
1278-
# Length of the '[-]<integer>.<fraction>' text returned below. Stripping
1279-
# redundant zeros can only shorten it, so this is an upper bound.
1280-
expansion_len = (
1281-
(1 if value.is_signed() else 0)
1282-
+ max(value.adjusted() + 1, 1) # integer digits, at least one
1283-
+ 1 # the decimal point
1284-
+ max(-exponent, 1) # fraction digits, at least one
1285-
)
1286-
if expansion_len > _MAX_TASK_PARAM_VALUE_LEN:
1287-
return elem
1288-
# format(value, 'f') is exact and plain at every magnitude: with no
1289-
# precision in the format spec it neither rounds to getcontext().prec
1290-
# nor switches to exponent notation, so an embedding app that sets a
1291-
# different context cannot change what this library renders. Neither
1292-
# holds of the obvious alternatives -- normalize() rounds to the
1293-
# ambient precision, str() switches to exponent notation below 1e-6,
1294-
# and quantize() raises InvalidOperation once the result needs more
1295-
# digits than the ambient precision allows.
1296-
text = format(value, "f")
1297-
negative = text.startswith("-")
1298-
integer, _, fraction = text.lstrip("-").partition(".")
1299-
# A <floatstring> denotes a number, so redundant zeros in its source
1300-
# text are not part of the value: '02.50' is 2.5. One fractional digit
1301-
# is always kept, matching openjd-rs, which renders the float 1.0 as
1302-
# `1.0` and never as `1`.
1303-
integer = integer.lstrip("0") or "0"
1304-
fraction = fraction.rstrip("0") or "0"
1305-
if negative and integer == "0" and fraction == "0":
1306-
# The number denoted by '-0.0' is zero, which has no sign; openjd-rs
1307-
# renders it `0.0`.
1308-
negative = False
1309-
return f"{'-' if negative else ''}{integer}.{fraction}"
1310-
except (ValueError, ArithmeticError):
1311-
# Not a number at all — e.g. a format string that resolved to
1312-
# non-numeric text. A literal range is already checked against its
1313-
# element type at template parse time, so leave such a value to the
1314-
# existing behaviour rather than adding a rejection path here.
1262+
return int(elem) # int() already drops leading zeros
1263+
# Parsed only to check it is a number; the text is what renders. Not a
1264+
# Decimal -- re-rendering one is context-sensitive and unbounded (§7.5).
1265+
float(elem)
1266+
except ValueError:
1267+
# A resolved format string can be non-numeric. Literals are checked at
1268+
# template parse time, so carry it through rather than rejecting here.
13151269
return elem
1270+
return _REDUNDANT_LEADING_ZEROS.sub(r"\1", elem)
13161271

13171272

13181273
# Target model for task parameters when instantiating a job.
@@ -1327,22 +1282,14 @@ class RangeListTaskParameterDefinition(OpenJDModel_v2023_09):
13271282
@field_validator("range", mode="before")
13281283
@classmethod
13291284
def _normalize_numeric_range_elements(cls, value: Any, info: ValidationInfo) -> Any:
1330-
# §3.4.1.1/§3.4.1.2: an <IntRangeList> element is `<integer> |
1331-
# <intstring>` and a <FloatRangeList> element is `<float> |
1332-
# <floatstring>`, where an <intstring>/<floatstring> is "a string whose
1333-
# value is the string representation of" a number (§2.3, §2.4). Such an
1334-
# element therefore denotes that number, not its source text, so '02'
1335-
# is the task value 2 and '02.50' is 2.5. Python was keeping the source
1336-
# text, which reaches a task command line as `--frame 02`.
1337-
#
1338-
# Applied on the instantiation target so every inbound range path is
1339-
# covered: a literal list, a range-expression expansion, and an RFC 0006
1340-
# typed whole-field resolution all funnel through this model.
1285+
# §7.5: a string-form range element loses redundant leading zeros and
1286+
# keeps its decimal places, so '02' is the value 2 and '02.50' renders
1287+
# `2.50`. Numeric literals carry no such request and are left as parsed;
1288+
# STRING and PATH ranges have no numeric form at all.
13411289
#
1342-
# Numeric elements are left exactly as parsed. A FLOAT range of [1.0]
1343-
# renders 1.0, which openjd-rs also does and the conformance suite pins
1344-
# (base/jobs/3.4--float-parameter). STRING and PATH ranges are text by
1345-
# definition and are never touched.
1290+
# On the instantiation target so all three inbound paths are covered: a
1291+
# literal list, a range-expression expansion, and RFC 0006 whole-field
1292+
# resolution.
13461293
param_type = info.data.get("type")
13471294
if not isinstance(value, list) or param_type not in (
13481295
TaskParameterType.INT,

‎test/openjd/model_v0/test_step_param_space_iter.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -613,8 +613,8 @@ def test_int_range_intstring_elements(self) -> None:
613613

614614
def test_float_range_floatstring_elements(self) -> None:
615615
# WHEN a FLOAT range is written with <floatstring> elements
616-
# THEN each task gets the number the element denotes
617-
assert self._task_values("FLOAT", ["1.5", "02.50"]) == ["1.5", "2.5"]
616+
# THEN each task gets the element's text, less redundant leading zeros
617+
assert self._task_values("FLOAT", ["1.5", "02.50", "3.500"]) == ["1.5", "2.50", "3.500"]
618618

619619
def test_float_range_numeric_literals_keep_their_scale(self) -> None:
620620
# A <float> literal is not a string representation, so it keeps the scale

‎test/openjd/model_v0/v2023_09/test_parameter_space.py‎

Lines changed: 40 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -441,13 +441,12 @@ def test_list_form_range_is_still_capped(self, model: Any, obj: dict[str, Any])
441441

442442

443443
class TestRangeListElementNormalization:
444-
"""§3.4.1.1/§3.4.1.2: an `<IntRangeList>` element is `<integer> | <intstring>`
445-
and a `<FloatRangeList>` element is `<float> | <floatstring>`. An
446-
`<intstring>`/`<floatstring>` is "a string whose value is the string
447-
representation of" a number (§2.3, §2.4), so it denotes that number and not
448-
its source text — a range of `['1', '02', '003']` is the task values 1, 2 and
449-
3. These values reach a task command line, so keeping the text renders
450-
`--frame 02`.
444+
"""§7.5 on a range element: leading zeros go, decimal places stay.
445+
446+
`['1', '02', '003']` is the task values 1, 2 and 3; keeping the text renders
447+
`--frame 02`. But `'2.50'` renders `2.50`, because the string form is how a
448+
template asks for a fixed number of decimal places —
449+
`EXPR/jobs/expr1.3.4--float-passthrough` pins that for a FLOAT default.
451450
"""
452451

453452
@pytest.mark.parametrize(
@@ -471,96 +470,65 @@ def test_intstring_elements_carry_their_value(self, param_type: str) -> None:
471470
@pytest.mark.parametrize(
472471
"element,expected",
473472
(
474-
pytest.param("1.5", "1.5", id="no normalization needed"),
475-
pytest.param("02.50", "2.5", id="leading and trailing zeros"),
476-
pytest.param("3.500", "3.5", id="trailing zeros"),
477-
pytest.param("-02.50", "-2.5", id="negative"),
478-
# An integral float keeps one fractional digit. openjd-rs renders
479-
# the <floatstring> '1.0' as `1.0` and '007' as `7.0`, so dropping
480-
# the fraction would make the same number render two ways depending
481-
# on whether it was written as a <float> or a <floatstring>.
482-
pytest.param("1.0", "1.0", id="integral float"),
473+
pytest.param("1.5", "1.5", id="nothing to strip"),
474+
# The rule, in both directions at once.
475+
pytest.param("02.50", "2.50", id="leading zero goes, trailing zero stays"),
476+
pytest.param("3.500", "3.500", id="trailing zeros stay"),
477+
pytest.param("-02.50", "-2.50", id="negative"),
478+
pytest.param("+02.50", "+2.50", id="explicit plus sign is kept"),
479+
pytest.param("007", "7", id="leading zeros on a whole number"),
480+
pytest.param("100", "100", id="a zero that is a significant digit"),
481+
# Neither the zero before the point nor the last of an all-zero
482+
# integer part is redundant.
483+
pytest.param("0.50", "0.50", id="the necessary leading zero stays"),
484+
pytest.param("000", "0", id="all zeros keeps one"),
485+
pytest.param("0", "0", id="bare zero"),
483486
pytest.param("0.0", "0.0", id="zero"),
484-
pytest.param("-0.0", "0.0", id="negative zero has no sign"),
485-
pytest.param("007", "7.0", id="leading zeros on a whole number"),
486-
pytest.param("100", "100.0", id="whole number"),
487-
# Exponent notation must never reach a task command line, at any
488-
# magnitude and in either direction.
489-
pytest.param("1E+2", "100.0", id="exponent notation in the source"),
490-
pytest.param(
491-
"0.0000001",
492-
"0.0000001",
493-
id="below 1e-6, where Decimal.__str__ would give 1E-7",
494-
),
495-
pytest.param(
496-
"1E+30",
497-
"1" + "0" * 30 + ".0",
498-
id="more digits than the default decimal precision",
499-
),
487+
pytest.param("0.00", "0.00", id="zero keeps its trailing zeros too"),
488+
pytest.param("1.0", "1.0", id="integral float keeps its point"),
489+
# Exponent notation is the author's own text and is forwarded. The
490+
# FLOAT parameter default path does the same for `default: "1E+2"`.
491+
pytest.param("1E+2", "1E+2", id="exponent notation"),
492+
pytest.param("01E+2", "1E+2", id="leading zero on an exponent form"),
493+
pytest.param("0.0000001", "0.0000001", id="below 1e-6"),
500494
pytest.param(
501495
"1.2345678901234567890123456789012345678901",
502496
"1.2345678901234567890123456789012345678901",
503-
id="more significant digits than the default decimal precision",
497+
id="more significant digits than a float can hold",
504498
),
505499
),
506500
)
507-
def test_floatstring_elements_carry_their_value(self, element: str, expected: str) -> None:
501+
def test_floatstring_elements_keep_their_scale(self, element: str, expected: str) -> None:
508502
# WHEN the instantiation target parses a float range holding a <floatstring>
509503
model = _parse_model(
510504
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
511505
)
512506

513-
# THEN the element renders as the number it denotes
507+
# THEN it renders as written, less any redundant leading zeros
514508
assert [str(v) for v in model.range] == [expected]
515509

516510
@pytest.mark.parametrize(
517511
"element",
518512
(
519-
# Decimal's context bounds arithmetic results, not construction from
520-
# a string, so these are finite: 11 characters of template text
521-
# whose plain-notation expansion is ~10**9 characters.
522513
pytest.param("1e999999999", id="huge positive exponent"),
523514
pytest.param("1e-999999999", id="huge negative exponent"),
524-
# One character past the cap in each direction: both expand to 1025.
525-
pytest.param("1E+1022", id="one past the cap, positive exponent"),
526-
pytest.param("1E-1023", id="one past the cap, negative exponent"),
515+
pytest.param("1E+1022", id="positive exponent past the field cap"),
516+
pytest.param("1E-1023", id="negative exponent past the field cap"),
527517
),
528518
)
529-
def test_element_too_long_to_expand_keeps_its_source_text(self, element: str) -> None:
530-
# WHEN the instantiation target parses a <floatstring> whose plain-notation
531-
# form would not fit TaskParameterStringValueAsJob's 1024-character cap
519+
def test_a_huge_exponent_costs_only_its_own_characters(self, element: str) -> None:
520+
# WHEN 11 characters of template text denote a number needing ~10**9
521+
# digits in plain notation
532522
model = _parse_model(
533523
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
534524
)
535525

536-
# THEN it keeps its source text rather than being expanded. Expanding it
537-
# would allocate the whole expansion, and the result could not be carried
538-
# as text here anyway: it fails the str member of TaskRangeList's union,
539-
# and pydantic then falls through to the numeric members, rendering the
540-
# element as an integer, or as inf, or as 0.0.
526+
# THEN it renders as its own text. Nothing expands it, so no exponent
527+
# bound is needed and the value cannot exceed the field's 1024-character
528+
# cap and fall through to a numeric member of the union.
541529
assert [str(v) for v in model.range] == [element]
542530

543-
@pytest.mark.parametrize(
544-
"element,expected",
545-
(
546-
pytest.param("1E+1021", "1" + "0" * 1021 + ".0", id="positive exponent"),
547-
pytest.param("1E-1022", "0." + "0" * 1021 + "1", id="negative exponent"),
548-
),
549-
)
550-
def test_element_at_the_cap_is_still_normalized(self, element: str, expected: str) -> None:
551-
# GIVEN a <floatstring> whose plain-notation form is exactly 1024
552-
# characters — at the cap, not past it
553-
assert len(expected) == 1024
554-
555-
# WHEN the instantiation target parses it
556-
model = _parse_model(
557-
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
558-
)
559-
560-
# THEN the length bound has not cost it its normalization
561-
assert [str(v) for v in model.range] == [expected]
562-
563-
def test_floatstring_normalization_ignores_the_decimal_context(self) -> None:
531+
def test_floatstring_rendering_ignores_the_decimal_context(self) -> None:
564532
# GIVEN an embedding application that has narrowed the process-wide
565533
# decimal context, and a <floatstring> with more significant digits
566534
# than that precision allows
@@ -573,8 +541,8 @@ def test_floatstring_normalization_ignores_the_decimal_context(self) -> None:
573541
model=RangeListTaskParameterDefinition, obj={"type": "FLOAT", "range": [element]}
574542
)
575543

576-
# THEN the value is unrounded: what this library renders is a property of
577-
# the template, not of the host application's decimal context
544+
# THEN it is unrounded: the rendering is a property of the template, not
545+
# of the host application's decimal context
578546
assert [str(v) for v in model.range] == [element]
579547

580548
@pytest.mark.parametrize(

0 commit comments

Comments
 (0)