Skip to content

Commit 9cd6b77

Browse files
committed
fix: Render a PATH parameter in the host's path format
A session is host scope, so a `path` there takes the host operating system's semantics -- separators included (Expression Language 1.2.1: "In host contexts (SESSION and TASK scopes), the semantics match the host's operating system"). This package applied the host's separators only *inside* a matching path mapping rule, in `PathMappingRule.apply`, so that was the only thing that ever chose one. A PATH parameter that no rule matched reached the task in whatever form the submitter wrote it. Measured on a Windows host with no rules, `Task.Param.InputFile` for `/path/a.exr` was `/path/a.exr`, where conformance fixture 2023-09/base/jobs/3.4--path-parameter.test.yaml requires `\path\a.exr`; with a rule present but not matching, the same. That fixture has been failing on the Python conformance job's windows-latest leg continuously since 2026-05-22. openjd-rs applies the format unconditionally, wrapping both the mapped and the unmapped case in `ExprValue::new_path(mapped, PathFormat::host())` for `Param.<PATH>`, `Task.Param.<PATH>` and each element of the LIST[PATH] forms, while leaving `RawParam.*` raw. `processed_parameter_value` now mirrors that: `to_host_path_separators` wraps `apply_mapping` on both PATH branches, which is idempotent for a value a rule did match and so keeps the two paths agreeing. `to_host_path_separators` duplicates Rust's `normalize_path_separators` rather than calling it through `openjd.expr`. The fixture that pins this is a base-spec template, so the fix sits on the non-EXPR path that test/openjd/ test_import_purity.py protects, and the native extension must not become a load-time requirement there. The URI arm reuses this module's existing `_URI_SOURCE_RE`, so only Rust's three-way branch is restated. It lives in `_path_mapping` and reads that module's `os_name` deliberately: `apply` reads the same name, and deriving the host twice would let the two disagree. Separators and nothing else. Rendering through PureWindowsPath would satisfy the headline assertion and still be wrong -- measured, it turns `s3://bucket/key` into `s3:\bucket\key`, collapses `/a//b` to `\a\b`, drops the trailing separator that `apply` deliberately preserves, and renders `""` as `.`. 17 tests, written first and failing 7/17 against the unmodified source. Full suite unchanged apart from those 7 (15 failed/996 passed -> 8 failed/1003 passed, the 8 pre-existing and environmental). All 7 mutants caught, including one that formats `RawParam.*` too and one that swaps in PureWindowsPath. Windows is exercised through a patched seam rather than on Windows, so the end-to-end confirmation is 3.4--path-parameter's next windows-latest result. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
1 parent b3ee0b6 commit 9cd6b77

3 files changed

Lines changed: 325 additions & 3 deletions

File tree

‎src/openjd/sessions/_path_mapping.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,41 @@ def _ascii_lower(value: str) -> str:
2626
return value.translate(_ASCII_LOWER_TABLE)
2727

2828

29+
def to_host_path_separators(value: str) -> str:
30+
"""Render ``value`` with this host's path separators.
31+
32+
A session is host scope, so a ``path`` there takes the host operating
33+
system's semantics. openjd-rs applies that to every host-scope path value by
34+
constructing it through ``ExprValue::new_path(.., PathFormat::host())``,
35+
which calls ``normalize_path_separators``
36+
(``crates/openjd-expr/src/value.rs``). This is that function for the host's
37+
format, and the three cases are its three arms:
38+
39+
- a URI keeps forward slashes on every host, because its path portion is a
40+
set of opaque identifiers rather than a filesystem path;
41+
- a POSIX host changes nothing, because a backslash is a legal character in
42+
a POSIX filename and rewriting one would corrupt the path;
43+
- a Windows host replaces ``/`` with ``\\``.
44+
45+
Separators and nothing else. Rendering through ``PureWindowsPath`` would
46+
also collapse ``//`` to ``\\``, drop a trailing separator, and turn ``""``
47+
into ``"."`` -- and :meth:`PathMappingRule.apply` deliberately preserves a
48+
trailing separator, so that one is a behaviour this must not undo.
49+
50+
Duplicated from Rust rather than called through ``openjd.expr``: the native
51+
extension must not become a load-time requirement of a non-EXPR session (see
52+
``test/openjd/test_import_purity.py``), and a PATH parameter reaches this on
53+
the non-EXPR path. The URI test reuses :data:`_URI_SOURCE_RE`, which is this
54+
module's existing spelling of the same ``<scheme>://`` rule, so the
55+
duplication is of Rust's three-way branch only.
56+
"""
57+
if _URI_SOURCE_RE.match(value) is not None:
58+
return value
59+
if os_name == "posix":
60+
return value
61+
return value.replace("/", "\\")
62+
63+
2964
_URI_SOURCE_RE = re.compile(r"[A-Za-z][A-Za-z0-9+.\-]*://")
3065
"""A URI-format rule's ``source_path`` must start with ``<scheme>://``.
3166

‎src/openjd/sessions/_session.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141
from ._embedded_files import EmbeddedFiles, EmbeddedFilesScope, _FileRecord, write_file_for_user
4242
from ._logging import LOG, log_section_banner, LoggerAdapter, LogExtraInfo, LogContent
4343
from ._os_checker import is_posix, is_windows
44-
from ._path_mapping import PathMappingRule
44+
from ._path_mapping import PathMappingRule, to_host_path_separators
4545
from ._runner_base import (
4646
ScriptRunnerBase,
4747
apply_let_bindings,
@@ -1670,12 +1670,21 @@ def apply_mapping(path: str) -> str:
16701670
return path
16711671

16721672
def processed_parameter_value(param: ParameterValue) -> Any:
1673+
# The host format is applied to every PATH value, not only to a
1674+
# mapped one. `apply_mapping` chooses the host's separator for a
1675+
# rule's output, so before this it was the *only* thing that did:
1676+
# a value no rule matched reached the task in whatever form the
1677+
# submitter wrote it, and on a Windows host a POSIX-spelled
1678+
# parameter stayed POSIX-spelled. openjd-rs wraps both the mapped
1679+
# and the unmapped case in `ExprValue::new_path(.., host())`.
1680+
# Applying it after `apply_mapping` is idempotent for a value a rule
1681+
# did match, which is what keeps the two paths agreeing.
16731682
if param.type == ParameterValueType.PATH:
1674-
return apply_mapping(param.value)
1683+
return to_host_path_separators(apply_mapping(param.value))
16751684
if param.type == ParameterValueType.LIST_PATH and isinstance(param.value, list):
16761685
# openjd-rs maps each element of a LIST[PATH] parameter at
16771686
# session scope; mirror that element-wise mapping.
1678-
return [apply_mapping(p) for p in param.value]
1687+
return [to_host_path_separators(apply_mapping(p)) for p in param.value]
16791688
return param.value
16801689

16811690
def record_expr_types(
Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""A ``PATH`` parameter's ``Param.*``/``Task.Param.*`` value must render in the
5+
host's path format.
6+
7+
The session is host scope, and the Expression Language spec says a ``path`` there
8+
takes the host operating system's semantics -- separators included. openjd-rs
9+
does that unconditionally when it builds the session symbol table: every
10+
``Param.<PATH>`` and ``Task.Param.<PATH>`` is wrapped in
11+
``ExprValue::new_path(mapped, PathFormat::host())`` (``crates/openjd-sessions/
12+
src/session.rs``), so the format is applied whether or not a path mapping rule
13+
matched.
14+
15+
This package used to apply the host's separators only *inside*
16+
``PathMappingRule.apply``, so a parameter with no matching rule reached the task
17+
in whatever form the submitter wrote it. On a Windows host a POSIX-spelled
18+
parameter stayed POSIX-spelled, which is what conformance fixture
19+
``2023-09/base/jobs/3.4--path-parameter.test.yaml`` catches:
20+
21+
output_windows:
22+
- TASK:InputFile=\\path\\a.exr
23+
24+
On simulating a Windows host. A POSIX host renders both readings identically, so
25+
comparing values on this machine proves nothing -- the assertions would pass
26+
whatever the code did. ``_windows_host`` forces the other format.
27+
28+
It patches exactly one seam, ``openjd.sessions._path_mapping.os_name``, and that
29+
is deliberate rather than a simplification: that module-level name is the single
30+
place this package decides which separator a host-scope path uses. It is what
31+
``PathMappingRule.apply`` reads for the separator of a rule's output, and it is
32+
what ``host_path_format`` reads for the format of an unmapped value. Patching one
33+
and not the other would produce a self-inconsistent host -- mapped values
34+
rendering Windows while unmapped ones rendered POSIX -- and a test built on that
35+
would assert an arrangement that cannot occur in production. Keeping both readers
36+
on one seam is the reason the fix put ``host_path_format`` in ``_path_mapping``
37+
rather than deriving ``os.name`` a second time in ``_session``.
38+
"""
39+
40+
from __future__ import annotations
41+
42+
from contextlib import contextmanager
43+
from pathlib import PurePosixPath, PureWindowsPath
44+
from typing import Generator
45+
from unittest.mock import patch
46+
47+
import pytest
48+
49+
from openjd.model import (
50+
ParameterValue,
51+
ParameterValueType,
52+
SpecificationRevision,
53+
SymbolTable,
54+
)
55+
from openjd.sessions import PathFormat, PathMappingRule, Session
56+
57+
import openjd.sessions._path_mapping as path_mapping_impl_mod
58+
59+
# ---------------------------------------------------------------------------
60+
# Helpers
61+
# ---------------------------------------------------------------------------
62+
63+
_POSIX_TEXT = "/path/a.exr"
64+
"""The parameter as a submitter wrote it."""
65+
66+
_WINDOWS_TEXT = r"\path\a.exr"
67+
"""The same parameter as a Windows host must render it. Distinct from
68+
``_POSIX_TEXT``, which is what an unformatted value shows."""
69+
70+
71+
@contextmanager
72+
def _host(os_name: str) -> Generator[None, None, None]:
73+
"""Force the host path format. See this module's docstring for why one seam
74+
is enough, and why it is this one."""
75+
with patch.object(path_mapping_impl_mod, "os_name", os_name):
76+
yield
77+
78+
79+
def _symtab(
80+
params: dict[str, ParameterValue],
81+
*,
82+
os_name: str,
83+
rules: list[PathMappingRule] | None = None,
84+
) -> SymbolTable:
85+
"""Build a session symbol table the way a running session does."""
86+
with Session(
87+
session_id="test-path-format",
88+
job_parameter_values=params,
89+
path_mapping_rules=rules or [],
90+
) as session:
91+
with _host(os_name):
92+
return session._symbol_table(SpecificationRevision.v2023_09, params)
93+
94+
95+
def _path_param(value: str) -> dict[str, ParameterValue]:
96+
return {"InputFile": ParameterValue(type=ParameterValueType.PATH, value=value)}
97+
98+
99+
# ---------------------------------------------------------------------------
100+
# The defect
101+
# ---------------------------------------------------------------------------
102+
103+
104+
class TestPathParameterTakesTheHostFormat:
105+
"""``Param.*``/``Task.Param.*`` for a PATH parameter render in the host's
106+
format, with no path mapping rule involved."""
107+
108+
@pytest.mark.parametrize("key", ["Param.InputFile", "Task.Param.InputFile"])
109+
def test_windows_host_renders_backslashes(self, key: str) -> None:
110+
# GIVEN a POSIX-spelled PATH parameter and no path mapping rules
111+
# WHEN the session symbol table is built on a Windows host
112+
symtab = _symtab(_path_param(_POSIX_TEXT), os_name="nt")
113+
114+
# THEN the value carries the host's separators
115+
assert str(symtab[key]) == _WINDOWS_TEXT
116+
117+
@pytest.mark.parametrize("key", ["Param.InputFile", "Task.Param.InputFile"])
118+
def test_posix_host_leaves_the_value_alone(self, key: str) -> None:
119+
"""The negative control for the test above: a POSIX host must not
120+
rewrite anything, so a fix that always converts is caught here."""
121+
# GIVEN the same parameter
122+
# WHEN the table is built on a POSIX host
123+
symtab = _symtab(_path_param(_POSIX_TEXT), os_name="posix")
124+
125+
# THEN it is unchanged
126+
assert str(symtab[key]) == _POSIX_TEXT
127+
128+
@pytest.mark.parametrize("key", ["RawParam.InputFile", "Task.RawParam.InputFile"])
129+
def test_the_raw_form_is_not_reformatted(self, key: str) -> None:
130+
"""openjd-rs passes a PATH parameter's raw value through untouched
131+
(``JobParameterType::Path | ListPath => param.value.clone()``), so the
132+
fix must not reach ``RawParam.*``. Without this, a fix that normalizes
133+
both forms looks correct on every other assertion here."""
134+
# GIVEN a POSIX-spelled PATH parameter
135+
# WHEN the table is built on a Windows host
136+
symtab = _symtab(_path_param(_POSIX_TEXT), os_name="nt")
137+
138+
# THEN the raw form is still what the submitter wrote
139+
assert str(symtab[key]) == _POSIX_TEXT
140+
141+
142+
class TestPathParameterFormatAndPathMappingAgree:
143+
"""The host format is applied whether or not a rule matched. This is the
144+
property that ``PathMappingRule.apply`` alone cannot provide."""
145+
146+
def test_a_non_matching_rule_still_leaves_a_host_format_value(self) -> None:
147+
"""The defect in its narrowest form: a rule exists, so the mapping code
148+
runs, but it does not match, so its separator choice never applies."""
149+
# GIVEN a rule that cannot match the parameter
150+
rules = [
151+
PathMappingRule(
152+
source_path_format=PathFormat.POSIX,
153+
source_path=PurePosixPath("/nowhere"),
154+
destination_path=PurePosixPath("/elsewhere"),
155+
)
156+
]
157+
158+
# WHEN the table is built on a Windows host
159+
symtab = _symtab(_path_param(_POSIX_TEXT), os_name="nt", rules=rules)
160+
161+
# THEN the unmapped value still took the host's format
162+
assert str(symtab["Task.Param.InputFile"]) == _WINDOWS_TEXT
163+
164+
def test_a_matching_rule_is_unchanged_by_the_fix(self) -> None:
165+
"""A matching rule already emitted host separators, so normalizing after
166+
it must be a no-op rather than a second transformation."""
167+
# GIVEN a rule that matches
168+
rules = [
169+
PathMappingRule(
170+
source_path_format=PathFormat.POSIX,
171+
source_path=PurePosixPath("/path"),
172+
destination_path=PureWindowsPath(r"C:\dest"),
173+
)
174+
]
175+
176+
# WHEN the table is built on a Windows host
177+
symtab = _symtab(_path_param(_POSIX_TEXT), os_name="nt", rules=rules)
178+
179+
# THEN the mapped value is what path mapping produced, unaltered
180+
assert str(symtab["Task.Param.InputFile"]) == r"C:\dest\a.exr"
181+
182+
183+
class TestPathParameterFormatIsSeparatorsOnly:
184+
"""Guards the *shape* of the conversion. Rendering through
185+
``PureWindowsPath``/``PurePosixPath`` would satisfy the assertions above and
186+
still be wrong: it collapses duplicate separators, drops a trailing
187+
separator, rewrites ``s3://`` to ``s3:/``, and turns an empty value into
188+
``'.'``. openjd-rs replaces separators and nothing else
189+
(``normalize_path_separators``), so these pin that.
190+
"""
191+
192+
@pytest.mark.parametrize(
193+
"given,expected",
194+
[
195+
pytest.param("/a//b", r"\a\\b", id="duplicate separators survive"),
196+
pytest.param("/a/", "\\a\\", id="trailing separator survives"),
197+
pytest.param("relative/path", r"relative\path", id="relative path"),
198+
pytest.param("", "", id="empty value stays empty"),
199+
pytest.param(r"C:\already\windows", r"C:\already\windows", id="already windows"),
200+
],
201+
)
202+
def test_windows_host_replaces_separators_only(self, given: str, expected: str) -> None:
203+
# GIVEN a PATH parameter whose text a PurePath would rewrite
204+
# WHEN the table is built on a Windows host
205+
symtab = _symtab(_path_param(given), os_name="nt")
206+
207+
# THEN only the separators changed
208+
assert str(symtab["Task.Param.InputFile"]) == expected
209+
210+
def test_a_uri_is_left_alone(self) -> None:
211+
"""A ``path`` holding a URI keeps forward slashes on every host. The
212+
Expression Language spec makes URI paths exempt from the host's
213+
separator, and openjd-rs implements that in the same function the fix
214+
reaches (``is_uri`` short-circuits ``normalize_path_separators``)."""
215+
# GIVEN a PATH parameter holding a URI
216+
given = "s3://bucket/key/with/slashes"
217+
218+
# WHEN the table is built on a Windows host
219+
symtab = _symtab(_path_param(given), os_name="nt")
220+
221+
# THEN it is untouched
222+
assert str(symtab["Task.Param.InputFile"]) == given
223+
224+
225+
class TestListPathParameterTakesTheHostFormat:
226+
"""openjd-rs formats a ``LIST[PATH]`` element-wise. So must this."""
227+
228+
def test_windows_host_formats_every_element(self) -> None:
229+
# GIVEN a LIST[PATH] parameter of POSIX-spelled elements
230+
params = {
231+
"Inputs": ParameterValue(
232+
type=ParameterValueType.LIST_PATH,
233+
value=["/path/a.exr", "/other/b.exr"],
234+
)
235+
}
236+
237+
# WHEN the table is built on a Windows host
238+
symtab = _symtab(params, os_name="nt")
239+
240+
# THEN every element carries the host's separators
241+
assert [str(element) for element in symtab["Task.Param.Inputs"]] == [
242+
r"\path\a.exr",
243+
r"\other\b.exr",
244+
]
245+
246+
def test_posix_host_leaves_every_element_alone(self) -> None:
247+
# GIVEN the same parameter
248+
params = {
249+
"Inputs": ParameterValue(
250+
type=ParameterValueType.LIST_PATH,
251+
value=["/path/a.exr", "/other/b.exr"],
252+
)
253+
}
254+
255+
# WHEN the table is built on a POSIX host
256+
symtab = _symtab(params, os_name="posix")
257+
258+
# THEN nothing changed
259+
assert [str(element) for element in symtab["Task.Param.Inputs"]] == [
260+
"/path/a.exr",
261+
"/other/b.exr",
262+
]
263+
264+
265+
class TestOtherParameterTypesAreUntouched:
266+
"""The fix is scoped to PATH and LIST[PATH]. A STRING that happens to look
267+
like a path must not be reformatted -- openjd-rs coerces it as a string."""
268+
269+
def test_a_string_parameter_keeps_its_slashes_on_a_windows_host(self) -> None:
270+
# GIVEN a STRING parameter whose value looks like a POSIX path
271+
params = {"Text": ParameterValue(type=ParameterValueType.STRING, value=_POSIX_TEXT)}
272+
273+
# WHEN the table is built on a Windows host
274+
symtab = _symtab(params, os_name="nt")
275+
276+
# THEN it is still the string the submitter wrote
277+
assert str(symtab["Param.Text"]) == _POSIX_TEXT
278+
assert str(symtab["Task.Param.Text"]) == _POSIX_TEXT

0 commit comments

Comments
 (0)