Skip to content

Commit 8959e09

Browse files
authored
fix: apply the template extensions in the job parameter merge (#368)
fix: apply the template's extensions in the job parameter merge preprocess_job_parameters re-validates each merged job parameter definition through parse_model, and it passed no supported_extensions. The parsing context that reached NameIdentifierLengthMixin therefore carried an empty extension set, so the mixin enforced the base 64-character limit on a parameter name that decode had already accepted at up to 512 under FEATURE_BUNDLE_1. create_job calls preprocess_job_parameters itself, so both entry points rejected the name. Measured before the change, for a template declaring FEATURE_BUNDLE_1 with a 512-character STRING parameter name: decode_job_template OK preprocess_job_parameters DecodeValidationError: 1 validation errors for JobStringParameterDefinition name: name must be at most 64 characters long create_job same merge_job_parameter_definitions now derives the union of the extensions declared by the job template and any environment templates, and threads it to merge_job_parameter_definitions_for_one and on into parse_model. The union is sound because a definition was decoded under its own template's extension set and every definition of one job parameter carries the same name, so the union cannot admit a value decode did not already accept for the template that declared it. The public signature of merge_job_parameter_definitions is unchanged: it already receives the templates that carry the extensions, so the set is derived rather than passed. This follows the shape of #362, which fixed the sibling defect in instantiate_model, and it keeps a caller from declaring an extension the template did not. merge_job_parameter_definitions_for_one takes a new keyword-only supported_extensions argument that defaults to None, preserving its behaviour for any existing caller. Tests cover all four FEATURE_BUNDLE_1 length ceilings at job-creation time on both lanes. v0 gains group H in test_feature_bundle_1.py: the parameter name through the merge, preprocess_job_parameters and create_job for each of STRING, PATH, INT and FLOAT, a genuine two-source merge, the environment-template-only case, the EXPR-typed model_copy branch, and controls for the base 64 limit and the static 512 identifier ceiling. v1 gains an equivalent class in model_v1/test_create_job.py, which had no coverage of these ceilings at all; the Rust implementation validates in a single pass and already passes, so those tests are a ratchet against a re-validation pass being added upstream unnoticed. Every v0 test was mutation-checked. Seven mutants, each caught: dropping the threading at any of its three points, hardcoding FEATURE_BUNDLE_1 instead of deriving it, ignoring environment templates, applying the threading only to STRING, and applying it only to single-source merges. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
1 parent f6bf876 commit 8959e09

3 files changed

Lines changed: 534 additions & 7 deletions

File tree

‎src/openjd/model/_merge_job_parameter.py‎

Lines changed: 43 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
from collections import defaultdict
44
from decimal import Decimal
5-
from typing import Any, NamedTuple, Optional, Union, cast
5+
from typing import Any, Iterable, NamedTuple, Optional, Union, cast
66

77
from ._errors import CompatibilityError
88
from ._parse import parse_model
@@ -117,12 +117,21 @@ def merge_job_parameter_definitions(
117117
SourcedParamDefinition(source="JobTemplate", definition=param)
118118
)
119119

120+
# The merge re-validates each merged definition, so it must run under the same
121+
# extension set decode ran under or it rejects what decode accepted -- e.g. a
122+
# FEATURE_BUNDLE_1 parameter name between 65 and 512 characters.
123+
supported_extensions = _declared_extensions(job_template, environment_templates)
124+
120125
errors = list[str]()
121126
return_value = list[JobParameterDefinition]()
122127

123128
for name, source in collected_definitions.items():
124129
try:
125-
return_value.append(merge_job_parameter_definitions_for_one(source))
130+
return_value.append(
131+
merge_job_parameter_definitions_for_one(
132+
source, supported_extensions=supported_extensions
133+
)
134+
)
126135
except CompatibilityError as e:
127136
compat_errors = "\n\t".join(str(e).split("\n"))
128137
errors.append(
@@ -134,13 +143,38 @@ def merge_job_parameter_definitions(
134143
return return_value
135144

136145

146+
def _declared_extensions(
147+
job_template: Optional[JobTemplate],
148+
environment_templates: Optional[list[EnvironmentTemplate]],
149+
) -> list[str]:
150+
"""The union of the extensions declared by all of the given templates.
151+
152+
A definition was decoded under its own template's extension set, and every
153+
definition of one job parameter carries the same name, so the union cannot admit a
154+
value that decode did not already accept for the template that declared it.
155+
"""
156+
extensions: set[str] = set()
157+
for template in (*(environment_templates or []), job_template):
158+
if template is not None and template.extensions:
159+
extensions.update(template.extensions)
160+
return sorted(extensions)
161+
162+
137163
def merge_job_parameter_definitions_for_one(
138164
params: list[SourcedParamDefinition],
165+
*,
166+
supported_extensions: Optional[Iterable[str]] = None,
139167
) -> JobParameterDefinition:
140168
"""Given an ordered list of job parameter definitions of the *same* job parameter, this merges the definitions into a single
141169
job parameter definition. In the act of doing the merger, this performs checks to ensure that the job parameter definitions are
142170
compatible with one another.
143171
172+
Args:
173+
params: The definitions of one job parameter, in merge order.
174+
supported_extensions (optional): The extensions declared by the templates the definitions
175+
came from. The merged definition is re-validated under this set, so omitting it applies
176+
the base limits to a definition decode may have accepted under an extension.
177+
144178
Returns (JobParameterDefinition):
145179
The result of merging all of the given definitions in to a single definition.
146180
@@ -219,13 +253,16 @@ def merge_job_parameter_definitions_for_one(
219253
if errors:
220254
raise CompatibilityError("\n".join(errors))
221255

222-
return parse_model(model=params[0].definition.__class__, obj=merged_properties)
256+
return parse_model(
257+
model=params[0].definition.__class__,
258+
obj=merged_properties,
259+
supported_extensions=supported_extensions,
260+
)
223261

224262
# EXPR-extension types (BOOL, RANGE_EXPR, LIST[*]): not cross-merged. Return
225263
# the last-defined definition with the last-defined default applied.
226-
# ``model_copy`` avoids re-validation, which would otherwise re-trigger the
227-
# EXPR extension gate (the merge has no parsing context to satisfy it). But
228-
# because ``model_copy`` skips validators, re-validate the merged default
264+
# ``model_copy`` avoids re-validation, so this path does not re-trigger the EXPR
265+
# extension gate at all. But because ``model_copy`` skips validators, re-validate the merged default
229266
# against the base definition's own constraints explicitly so a default
230267
# carried over from another source that violates them is still rejected
231268
# (mirrors the ``parse_model`` re-validation the legacy path performs).

‎test/openjd/model_v0/v2023_09/test_feature_bundle_1.py‎

Lines changed: 229 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,23 @@
11
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
22

33

4+
from pathlib import Path
5+
from typing import Optional
6+
47
import pytest
58
from pydantic import TypeAdapter, ValidationError
69

7-
from openjd.model import create_job, decode_job_template
10+
from openjd.model import (
11+
create_job,
12+
decode_environment_template,
13+
decode_job_template,
14+
preprocess_job_parameters,
15+
)
816
from openjd.model._errors import DecodeValidationError
17+
from openjd.model._merge_job_parameter import (
18+
SourcedParamDefinition,
19+
merge_job_parameter_definitions_for_one,
20+
)
921
from openjd.model._format_strings import FormatString
1022
from openjd.model._parse import _parse_model
1123
from openjd.model._types import ParameterValue, ParameterValueType
@@ -1384,6 +1396,10 @@ class TestCreateJobPreservesFeatureBundle1Lengths:
13841396
create_job simultaneously.
13851397
G. Resolved job name (matrix): the resolved job name is checked against
13861398
the extension-aware 128/512 limit after format-string substitution.
1399+
H. Job parameter name: a 512-char parameter name survives the job-parameter
1400+
merge inside preprocess_job_parameters, which create_job also reaches, for
1401+
every type the merge re-validates and for both of its branches. Includes
1402+
the base-limit and static-ceiling controls.
13871403
"""
13881404

13891405
@staticmethod
@@ -1680,3 +1696,215 @@ def test_resolved_job_name_128_chars_no_extension_create_job_accepted(self) -> N
16801696
def test_resolved_job_name_129_chars_no_extension_create_job_rejected(self) -> None:
16811697
with pytest.raises(DecodeValidationError, match="128"):
16821698
self._create_with_resolved_job_name(False, 129)
1699+
1700+
# ---- Group H: job parameter name through the merge ----
1701+
#
1702+
# Groups A-G build templates with no parameterDefinitions, so none of them reaches
1703+
# the job-parameter merge inside preprocess_job_parameters. That merge re-validates
1704+
# each merged definition through parse_model, and it ran with an empty extension
1705+
# set, enforcing the base 64-character limit on a name decode had accepted at 512.
1706+
# create_job calls preprocess_job_parameters itself, so both were affected.
1707+
1708+
#: The four scalar types whose merged definition is re-validated through
1709+
#: ``parse_model``. The EXPR-extension types take the ``model_copy`` branch instead
1710+
#: and are covered separately at the end of the group.
1711+
_LEGACY_TYPE_DEFAULTS = {
1712+
"STRING": "v",
1713+
"PATH": "/tmp/v",
1714+
"INT": "1",
1715+
"FLOAT": "1.5",
1716+
}
1717+
1718+
def _template_with_param(
1719+
self,
1720+
param_name: str,
1721+
*,
1722+
param_type: str = "STRING",
1723+
default: object = "v",
1724+
declare_fb1: bool = True,
1725+
) -> dict:
1726+
"""A minimal template carrying one job parameter definition."""
1727+
template = self._template(declare_fb1=declare_fb1)
1728+
template["parameterDefinitions"] = [
1729+
{"name": param_name, "type": param_type, "default": default}
1730+
]
1731+
return template
1732+
1733+
@staticmethod
1734+
def _preprocess(
1735+
template: dict,
1736+
supported: list,
1737+
environment_templates: Optional[list[EnvironmentTemplate]] = None,
1738+
) -> dict:
1739+
"""Decode the template and run it through preprocess_job_parameters.
1740+
1741+
Paths do not matter to a name-length check, so this uses the walk-up form
1742+
rather than real directories.
1743+
"""
1744+
job_template = decode_job_template(template=template, supported_extensions=supported)
1745+
return preprocess_job_parameters(
1746+
job_template=job_template,
1747+
job_parameter_values={},
1748+
job_template_dir=Path(),
1749+
current_working_dir=Path(),
1750+
allow_job_template_dir_walk_up=True,
1751+
environment_templates=environment_templates,
1752+
)
1753+
1754+
@pytest.mark.parametrize("param_type", sorted(_LEGACY_TYPE_DEFAULTS))
1755+
def test_parameter_name_512_chars_fb1_preprocess_preserved(self, param_type: str) -> None:
1756+
"""A 512-char parameter name accepted at FEATURE_BUNDLE_1 decode must survive the
1757+
merge inside preprocess_job_parameters, for every type the merge re-validates."""
1758+
name = "a" * 512
1759+
template = self._template_with_param(
1760+
name, param_type=param_type, default=self._LEGACY_TYPE_DEFAULTS[param_type]
1761+
)
1762+
assert name in self._preprocess(template, _FB1_SUPPORTED)
1763+
1764+
@pytest.mark.parametrize("param_type", sorted(_LEGACY_TYPE_DEFAULTS))
1765+
def test_parameter_name_512_chars_fb1_create_job_preserved(self, param_type: str) -> None:
1766+
"""The same name must survive create_job, which reaches the merge through its own
1767+
preprocess_job_parameters call."""
1768+
name = "a" * 512
1769+
job = self._create(
1770+
self._template_with_param(
1771+
name, param_type=param_type, default=self._LEGACY_TYPE_DEFAULTS[param_type]
1772+
)
1773+
)
1774+
assert name in job.parameters
1775+
1776+
def test_parameter_name_512_chars_merged_from_two_sources_preserved(self) -> None:
1777+
"""The merge's actual job is combining definitions from more than one source. A
1778+
512-char name must survive when constraints are genuinely merged, not only when
1779+
a single definition is passed through."""
1780+
name = "a" * 512
1781+
env_template = decode_environment_template(
1782+
template={
1783+
"specificationVersion": "environment-2023-09",
1784+
"extensions": ["FEATURE_BUNDLE_1"],
1785+
"parameterDefinitions": [
1786+
{"name": name, "type": "STRING", "minLength": 1, "default": "v"}
1787+
],
1788+
"environment": {
1789+
"name": "Env",
1790+
"script": {"actions": {"onEnter": {"command": "echo enter"}}},
1791+
},
1792+
},
1793+
supported_extensions=_FB1_SUPPORTED,
1794+
)
1795+
template = self._template_with_param(name)
1796+
template["parameterDefinitions"][0]["maxLength"] = 8
1797+
values = self._preprocess(template, _FB1_SUPPORTED, environment_templates=[env_template])
1798+
assert name in values
1799+
1800+
def test_parameter_name_512_chars_from_environment_template_preserved(self) -> None:
1801+
"""A 512-char name declared by an environment template that enables
1802+
FEATURE_BUNDLE_1 survives even though the job template does not enable it,
1803+
because the merge honours every contributing template's extensions."""
1804+
name = "a" * 512
1805+
env_template = decode_environment_template(
1806+
template={
1807+
"specificationVersion": "environment-2023-09",
1808+
"extensions": ["FEATURE_BUNDLE_1"],
1809+
"parameterDefinitions": [{"name": name, "type": "STRING", "default": "v"}],
1810+
"environment": {
1811+
"name": "Env",
1812+
"script": {"actions": {"onEnter": {"command": "echo enter"}}},
1813+
},
1814+
},
1815+
supported_extensions=_FB1_SUPPORTED,
1816+
)
1817+
values = self._preprocess(
1818+
self._template(declare_fb1=False), [], environment_templates=[env_template]
1819+
)
1820+
assert name in values
1821+
1822+
def test_parameter_name_513_chars_rejected_by_static_ceiling(self) -> None:
1823+
"""512 is the hard ceiling for an identifier regardless of extension: at 513 the
1824+
``Identifier`` string constraint rejects before the extension-aware validator is
1825+
consulted, so this pins the type ceiling rather than the FEATURE_BUNDLE_1 branch.
1826+
"""
1827+
with pytest.raises(
1828+
DecodeValidationError, match=r"name:\n\tString should have at most 512 characters"
1829+
):
1830+
decode_job_template(
1831+
template=self._template_with_param("a" * 513),
1832+
supported_extensions=_FB1_SUPPORTED,
1833+
)
1834+
1835+
def test_parameter_name_65_chars_no_extension_decode_rejected(self) -> None:
1836+
"""Without the extension the base 64-character limit applies, and it is reached at
1837+
decode -- before the merge ever runs."""
1838+
with pytest.raises(
1839+
DecodeValidationError, match=r"name:\n\tname must be at most 64 characters long"
1840+
):
1841+
decode_job_template(
1842+
template=self._template_with_param("a" * 65, declare_fb1=False),
1843+
supported_extensions=[],
1844+
)
1845+
1846+
def test_parameter_name_64_chars_no_extension_create_job_preserved(self) -> None:
1847+
"""A parameter name at the base ceiling survives create_job with no extension."""
1848+
name = "a" * 64
1849+
job_template = decode_job_template(
1850+
template=self._template_with_param(name, declare_fb1=False),
1851+
supported_extensions=[],
1852+
)
1853+
job = create_job(job_template=job_template, job_parameter_values={})
1854+
assert name in job.parameters
1855+
1856+
def test_merge_without_extensions_enforces_base_identifier_limit(self) -> None:
1857+
"""The merge must apply the base limit when no extension is in play, rather than
1858+
skipping the length check altogether.
1859+
1860+
Decode rejects an over-length name before the merge is reachable through a
1861+
template, so this drives the merge directly with a definition decode accepted
1862+
under FEATURE_BUNDLE_1 and no extensions supplied to the merge.
1863+
"""
1864+
job_template = decode_job_template(
1865+
template=self._template_with_param("a" * 512),
1866+
supported_extensions=_FB1_SUPPORTED,
1867+
)
1868+
sourced = [
1869+
SourcedParamDefinition(
1870+
source="JobTemplate", definition=job_template.parameterDefinitions[0]
1871+
)
1872+
]
1873+
with pytest.raises(DecodeValidationError, match="name must be at most 64 characters long"):
1874+
merge_job_parameter_definitions_for_one(sourced)
1875+
1876+
# And it accepts the same definition when the extension is supplied.
1877+
merged = merge_job_parameter_definitions_for_one(
1878+
sourced, supported_extensions=["FEATURE_BUNDLE_1"]
1879+
)
1880+
assert len(merged.name) == 512
1881+
1882+
def test_all_five_fields_at_ceiling_fb1_create_job_preserved(self) -> None:
1883+
"""Group F plus the parameter name: every FEATURE_BUNDLE_1-lengthened field
1884+
survives create_job at once."""
1885+
param_name = "a" * 512
1886+
template = self._template(
1887+
job_name="a" * 512,
1888+
env_name="a" * 512,
1889+
ef_name="a" * 512,
1890+
ef_filename="a" * 256,
1891+
)
1892+
template["parameterDefinitions"] = [{"name": param_name, "type": "STRING", "default": "v"}]
1893+
job = self._create(template)
1894+
assert (
1895+
len(job.name),
1896+
len(job.jobEnvironments[0].name),
1897+
len(job.steps[0].script.embeddedFiles[0].name),
1898+
len(job.steps[0].script.embeddedFiles[0].filename),
1899+
len(next(iter(job.parameters))),
1900+
) == (512, 512, 512, 256, 512)
1901+
1902+
def test_parameter_name_512_chars_expr_typed_parameter_preserved(self) -> None:
1903+
"""The EXPR-extension parameter types take the merge's other branch, which copies
1904+
rather than re-validates. A 512-char name must survive there too, so that
1905+
unifying the two branches cannot silently reintroduce the base limit."""
1906+
name = "a" * 512
1907+
template = self._template_with_param(name, param_type="LIST[STRING]", default=["v"])
1908+
template["extensions"] = ["FEATURE_BUNDLE_1", "EXPR"]
1909+
values = self._preprocess(template, [ExtensionName.FEATURE_BUNDLE_1, ExtensionName.EXPR])
1910+
assert name in values

0 commit comments

Comments
 (0)