Skip to content

Commit bcd09b5

Browse files
authored
fix: Forward the create-time step symbol table to openjd-sessions (#237)
* test: Pin declared-extension activation, Job.Name, Step.Name, and enter-failure cleanup Six behaviours the RFC 0007/0008 CLI work introduced were unpinned: reverting the template-declared-extensions fix entirely still passed all 307 tests. Each new test drives a real template through do_run and asserts on observed output. - Extension behaviours activate only when a template DECLARES the extension, never from the CLI's --extensions accept-list. A job template declaring no extensions must not get REDACTED_ENV_VARS even though --extensions defaults to every supported one. Kills: supported_extensions=extensions in _run_command (the pre-fix behaviour). - The declared set is the UNION over the job template and every external environment template. Only the environment template declares REDACTED_ENV_VARS, and redaction is active for the whole session. Kills: dropping the environment-template union loop. - Job.Name is seeded into the session symbol table, so a step-level EXPR `let` binding resolves it. Asserts the printed VALUE equals the job's name, so neither omitting job_name= from Session() nor passing an empty string satisfies it. - The step name reaches Session.run_task and feeds WrappedStep.Name inside an active onWrapTaskRun hook. Asserts the resolved value equals the step's real name. Kills: passing a wrong constant for step_name. - Step.Name is out of scope outside a step: a job environment referencing it is rejected. NOTE this is openjd-model static validation at template-read time, so it does NOT pin the CLI's non-seeding of step_name on job/external enters -- no loadable template can observe that, because the same rejection applies to every environment-template location including RFC 0008 wrap hooks. The docstring records the limit; the CLI's decision stays pinned only by test_localsession_step_env_enter_receives_step_name. - An enter that raises BEFORE the session registers the environment (two external wrap-hook environments hitting RFC 0008's "at most one") must drop it from the CLI's entered list, so the real error surfaces instead of a masking "Cannot exit unknown Environment". Kills: never popping on raise. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * docs: Correct three claims in the RFC 0007/0008 threading comments The enter-failure comment named two triggers that cannot reach it: at the pinned openjd-sessions floor (0.10.11) enter_environment has no `raise` after it registers the environment -- every failure it detects past that point routes through _fail_action_before_start() and returns normally (verified by AST-walking the function: raises at 780/782/793, append at 812, none after). The post-registration branch is therefore defensive, and now says so rather than implying a reachable path. The dependency-floor comment named run_task as what breaks below 0.10.11, but `job_name` is passed to Session() unconditionally, so construction raises TypeError first; `extra_let_bindings` is also 0.10.11-only. The forwarding comment explained that None is not passed without saying that pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthat pthat pteqthat pthat pthaion pins it. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * test: Widen the timing margins on two timeout fixtures Both tests asserted on a race against a tight action timeout and failed intermittently once this branch added 12 tests to the 12-worker parallel run: job_sleep_exit_normal left under 2s for interpreter startup before the timeout killed the action, so 'SLEEP' was never printed; and feature_bundle_1_timeout gave a bash action that sleeps 1s a 5s budget, which a loaded run exceeded, turning exit 0 into exit 1. Measured before: the full suite failed 2 of 3 runs on this branch while the base commit passed 3 of 3 (301 tests vs 313 -- the added contention, not a behaviour change; both tests pass 5/5 in isolation on both trees). After: 3 of 3 runs green. Neither test's subject changes. TaskTimeout still times out, because the sleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer than the timesleep is now an order of magnitude longer thanit asserts moves, 5s to 20s. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * fix: Forward the create-time step symbol table to sessions A step's template-scope `let` (RFC 0005 §3.6) is resolved once at job creation. openjd-model no longer merges the resolved bindings into `script.let` and openjd-sessions no longer re-evaluates the source expressions, so the only channel those values have into a session is the `resolved_symtab` argument. The CLI forwarded nothing, so a step-level `let` produced no bindings at all. It also still passed `extra_let_bindings`, which openjd-sessions removed in 0.12.0 in favour of `resolved_symtab`. That raised TypeError for any step with a `let`, which surfaced as "Cannot exit unknown Environment" once the failed enter left the environment unregistered. Take the tables from `create_job_with_symbol_tables` (same job as `create_job`, tables returned instead of discarded), carry them on LocalSession keyed by step name, and forward the right one at all three entry points: `run_task`, `enter_environment` for a step's environments, and `exit_environment` with the same table the enter used. The values are already `SerializedSymbolTable`, which is what the sessions API takes, so nothing is converted. `extra_let_bindings` is removed rather than kept beside the new channel. Raise both floors, since neither range contained a release with the API: openjd-sessions to >= 0.12.0 (first release with `resolved_symtab` on all three entry points; 0.10.11 through 0.11.0 have `extra_let_bindings` and no `resolved_symtab`, so no version satisfies both) and openjd-model to >= 0.11.4 (first release exporting `create_job_with_symbol_tables`). Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * docs: Point the conftest note at a real fixture The note in `sample_job_and_dirs` sent readers to a `sample_job_with_symtabs` fixture "below" that exists nowhere in the suite. The fixture that actually carries the per-step resolved symbol tables is `step_let_job` in test_step_symbol_tables.py, which builds them through a real `job_from_template` call. Comment only; no behaviour change. The CodeQL "unused import" report for `SerializedSymbolTable` in _run/_run_command.py is a false positive and is deliberately not acted on: the import is already guarded by `if TYPE_CHECKING`, and the quoted annotation on `_run_local_session`'s `step_symbol_tables` parameter needs it. Removing it makes mypy fail with `Name "SerializedSymbolTable" is not defined`. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * docs: Fix a stale version citation and two step-let docstring overclaims Three text corrections, no behaviour change. The enter_environment rationale cited openjd-sessions 0.10.11 for "nothing raises post-registration" while pyproject now pins >= 0.12.0. The claim still holds at 0.12.0 (verified against the released wheel: every raise in enter_environment precedes registration, and the post-registration resolved_symtab failure calls _fail_action_before_start and returns), so only the citation moves. test_do_run_job_name_in_step_let_binding claimed neither omitting job_name from the Session nor passing an empty one would satisfy it. Measured otherwise: deleting job_name=str(job.name) leaves the test passing, because the binding is step-level and openjd-model resolves it at job creation, so the value is already a literal in the resolved table before Session.__init__ runs. Reworded to the create-time forward path it does pin: model to step_symbol_tables to CLI to step-environment enter. test_do_run_step_name_in_step_environment had the same shape, calling itself "the end-to-end proof of the feature". Dropping step_name=step_name on the enter also leaves it passing (measured); that keyword is pinned by test_localsession_step_env_enter_receives_step_name. Scoped accordingly. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> --------- Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
1 parent a872650 commit bcd09b5

25 files changed

Lines changed: 789 additions & 79 deletions

‎pyproject.toml‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,18 @@ classifiers = [
2929
"Intended Audience :: End Users/Desktop"
3030
]
3131
dependencies = [
32-
# 0.10.11 is the first release with `step_name` on both Session.run_task and
33-
# Session.enter_environment (RFC 0007/0008). Below it, run_task() raises
34-
# TypeError on every task run, since this CLI passes the keyword unconditionally.
35-
"openjd-sessions >= 0.10.11,< 0.11",
36-
"openjd-model >= 0.9,< 0.12"
32+
# 0.12.0 is the first release carrying `resolved_symtab` on all three of
33+
# Session.enter_environment, Session.exit_environment and Session.run_task.
34+
# It is also the release that removed `extra_let_bindings` from
35+
# enter_environment: 0.10.11 through 0.11.0 have that keyword and no
36+
# `resolved_symtab`, so there is no version satisfying both channels and the
37+
# floor has to move rather than being feature-detected. Below 0.12.0 every
38+
# call site here raises TypeError on the unexpected `resolved_symtab`.
39+
"openjd-sessions >= 0.12.0,< 0.13",
40+
# 0.11.4 is the first release exporting `create_job_with_symbol_tables` and
41+
# `JobWithSymbolTables`. 0.11.3 and below have `create_job` only, so the
42+
# step-scope resolved symbol tables cannot be obtained at all.
43+
"openjd-model >= 0.11.4,< 0.12"
3744
]
3845

3946
[project.urls]

‎src/openjd/cli/_common/__init__.py‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
from dataclasses import asdict, dataclass
55
from enum import Enum
66
from pathlib import Path
7-
from typing import Callable, Literal
7+
from typing import TYPE_CHECKING, Callable, Literal
88
import json
99
import yaml
1010
import os
@@ -23,6 +23,10 @@
2323
)
2424
from openjd.model import DecodeValidationError, Job, JobParameterValues, EnvironmentTemplate
2525

26+
if TYPE_CHECKING:
27+
# Annotations only; see the note in _job_from_template.py.
28+
from openjd.expr import SerializedSymbolTable
29+
2630
__all__ = [
2731
"add_extensions_argument",
2832
"get_doc_type",
@@ -118,7 +122,10 @@ def generate_job(
118122
environments: list[EnvironmentTemplate] = [],
119123
*,
120124
supported_extensions: list[str],
121-
) -> tuple[Job, JobParameterValues]:
125+
) -> tuple[Job, JobParameterValues, dict[str, "SerializedSymbolTable"]]:
126+
"""Returns the job, its parameter values, and the per-step resolved symbol
127+
tables keyed by step name. See :func:`job_from_template` for why the tables
128+
must be carried alongside the job rather than discarded."""
122129
try:
123130
# Raises: RuntimeError, DecodeValidationError
124131
template = read_job_template(args.path, supported_extensions=supported_extensions)

‎src/openjd/cli/_common/_job_from_template.py‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import json
55
from pathlib import Path
66
import re
7-
from typing import Optional, Union
7+
from typing import TYPE_CHECKING, Optional, Union
88
import yaml
99

1010
from ._validation_utils import get_doc_type
@@ -15,10 +15,17 @@
1515
Job,
1616
JobParameterValues,
1717
JobTemplate,
18-
create_job,
18+
create_job_with_symbol_tables,
1919
preprocess_job_parameters,
2020
)
2121

22+
if TYPE_CHECKING:
23+
# Only for annotations: `openjd.expr` is a facade over the native
24+
# extension, and importing the CLI must not load it. The tables the model
25+
# hands back are already instances of this type, so nothing here
26+
# constructs one.
27+
from openjd.expr import SerializedSymbolTable
28+
2229

2330
def get_params_from_file(parameter_string: str) -> Union[dict, list]:
2431
"""
@@ -111,10 +118,18 @@ def job_from_template(
111118
parameter_args: list[str] | None,
112119
job_template_dir: Path,
113120
current_working_dir: Path,
114-
) -> tuple[Job, JobParameterValues]:
121+
) -> tuple[Job, JobParameterValues, dict[str, "SerializedSymbolTable"]]:
115122
"""
116123
Given a decoded Job Template and a user-input parameter dictionary,
117-
generates a Job object and the parameter values for running the job.
124+
generates a Job object, the parameter values for running the job, and the
125+
per-step resolved symbol tables, keyed by step name.
126+
127+
A step's template-scope `let` bindings (RFC 0005 §3.6) are evaluated once
128+
here, at job creation, and their resolved values reach a session only
129+
through those tables -- neither the model nor the session re-derives them
130+
from the source expressions. Sessions for a step must therefore be given
131+
`step_symbol_tables[step.name]`, or the step's `let` produces no bindings
132+
at all.
118133
119134
Raises: RuntimeError if parameters are an unsupported type or don't correspond to the template
120135
"""
@@ -132,11 +147,14 @@ def job_from_template(
132147
raise RuntimeError(str(ve))
133148

134149
try:
135-
job = create_job(
150+
# `create_job_with_symbol_tables` returns the same Job that
151+
# `create_job` does; it additionally returns the symbol tables that
152+
# instantiation built instead of discarding them.
153+
created = create_job_with_symbol_tables(
136154
job_template=template,
137155
job_parameter_values=parameter_values,
138156
environment_templates=environments,
139157
)
140-
return (job, parameter_values)
158+
return (created.job, parameter_values, created.step_symbol_tables)
141159
except DecodeValidationError as dve:
142160
raise RuntimeError(f"Could not generate Job from template and parameters: {str(dve)}")

‎src/openjd/cli/_run/_local_session/_actions.py‎

Lines changed: 66 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,18 @@
11
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
22

33
from enum import Enum
4-
from typing import Any, Optional
4+
from typing import TYPE_CHECKING, Optional
55

66
from openjd.model import Step, TaskParameterSet
77
from openjd.model.v2023_09 import Environment
88
from openjd.sessions import Session
99

10+
if TYPE_CHECKING:
11+
# Annotations only: `openjd.expr` is a facade over the native extension and
12+
# importing the CLI must not load it. These tables are produced by
13+
# `create_job_with_symbol_tables` and forwarded unchanged.
14+
from openjd.expr import SerializedSymbolTable
15+
1016

1117
class EnvironmentType(str, Enum):
1218
"""
@@ -40,11 +46,24 @@ def run(self):
4046
class RunTaskAction(SessionAction):
4147
_step: Step
4248
_parameters: TaskParameterSet
49+
_resolved_symtab: Optional["SerializedSymbolTable"]
4350

44-
def __init__(self, session: Session, step: Step, parameters: TaskParameterSet):
51+
def __init__(
52+
self,
53+
session: Session,
54+
step: Step,
55+
parameters: TaskParameterSet,
56+
resolved_symtab: Optional["SerializedSymbolTable"] = None,
57+
):
4558
super(RunTaskAction, self).__init__(session)
4659
self._step = step
4760
self._parameters = parameters
61+
# The step's create-time resolved symbol table
62+
# (`create_job_with_symbol_tables().step_symbol_tables[step.name]`).
63+
# It is the only channel for the step's template-scope `let` values:
64+
# the model does not merge them into `script.let`, and the session does
65+
# not re-evaluate the source expressions.
66+
self._resolved_symtab = resolved_symtab
4867

4968
def run(self):
5069
self._session.run_task(
@@ -53,6 +72,7 @@ def run(self):
5372
# RFC 0008: the step name feeds the WrappedStep.Name template
5473
# variable inside an active onWrapTaskRun hook.
5574
step_name=self._step.name,
75+
resolved_symtab=self._resolved_symtab,
5676
)
5777

5878
def __str__(self):
@@ -63,44 +83,53 @@ def __str__(self):
6383
class EnterEnvironmentAction(SessionAction):
6484
_environment: Environment
6585
_id: str
66-
_extra_let_bindings: Optional[list[str]]
86+
_resolved_symtab: Optional["SerializedSymbolTable"]
6787
_step_name: Optional[str]
6888

6989
def __init__(
7090
self,
7191
session: Session,
7292
environment: Environment,
7393
env_id: str,
74-
extra_let_bindings: Optional[list[str]] = None,
94+
resolved_symtab: Optional["SerializedSymbolTable"] = None,
7595
step_name: Optional[str] = None,
7696
):
7797
super(EnterEnvironmentAction, self).__init__(session)
7898
self._environment = environment
7999
self._id = env_id
80-
# RFC 0007: a step's environments are entered with the step-level
81-
# `let` bindings so their variables/actions can reference them.
82-
self._extra_let_bindings = extra_let_bindings
100+
# RFC 0005 §3.6: a step's environments are entered with the step's
101+
# create-time resolved symbol table, which carries the step's
102+
# template-scope `let` values so their variables/actions can reference
103+
# them. Only step-environment enters have one; job and external
104+
# environment enters leave it None.
105+
self._resolved_symtab = resolved_symtab
83106
# RFC 0007 §7.3.1 (EXPR): the owning step's name seeds Step.Name for
84107
# a step environment's `let` bindings, variables, and actions. Only
85108
# step-environment enters carry a step name; job/external enters
86109
# leave it None.
87110
self._step_name = step_name
88111

89112
def run(self):
90-
# Both keywords are guaranteed by this package's `openjd-sessions`
91-
# floor (>= 0.10.11), so neither is feature-detected. They are still
92-
# only forwarded when they carry something: a step with no `let`
93-
# bindings means "no extra bindings", and job/external environment
94-
# enters have no owning step, so `Step.Name` must stay undefined for
95-
# them rather than being seeded with None.
96-
optional_kwargs: dict[str, Any] = {}
97-
if self._extra_let_bindings:
98-
optional_kwargs["extra_let_bindings"] = self._extra_let_bindings
113+
# `step_name` is omitted rather than passed as None when this enter has
114+
# no owning step. Passing None would be equivalent today, since
115+
# enter_environment skips a None `step_name`, but omitting it keeps the
116+
# call site's intent legible at the boundary and it is what
117+
# test_localsession_step_env_enter_receives_step_name asserts — that
118+
# assertion is currently the only check that job and external enters
119+
# do not seed Step.Name, because a template referencing Step.Name
120+
# outside a step is rejected by static validation before any session
121+
# is built, so no loadable template can observe it.
122+
#
123+
# `resolved_symtab` is passed unconditionally: None is its documented
124+
# "no table" value and there is no analogous assertion keyed on its
125+
# absence.
126+
optional_kwargs: dict[str, str] = {}
99127
if self._step_name is not None:
100128
optional_kwargs["step_name"] = self._step_name
101129
self._session.enter_environment(
102130
environment=self._environment,
103131
identifier=self._id,
132+
resolved_symtab=self._resolved_symtab,
104133
**optional_kwargs,
105134
)
106135

@@ -111,15 +140,34 @@ def __str__(self):
111140
class ExitEnvironmentAction(SessionAction):
112141
_id: str
113142
_keep_session_running: bool
143+
_resolved_symtab: Optional["SerializedSymbolTable"]
114144

115-
def __init__(self, session: Session, id: str, keep_session_running: bool):
145+
def __init__(
146+
self,
147+
session: Session,
148+
id: str,
149+
keep_session_running: bool,
150+
resolved_symtab: Optional["SerializedSymbolTable"] = None,
151+
):
116152
super(ExitEnvironmentAction, self).__init__(session)
117153
self._id = id
118154
self._keep_session_running = keep_session_running
155+
# The same table the environment was entered with, so its onExit
156+
# resolves in the same scope as its onEnter (what
157+
# Session.exit_environment documents for this argument).
158+
#
159+
# Unlike the worker agent, the CLI holds the table as an object rather
160+
# than a JSON string served by the service, so there is no parse step
161+
# here that could fail and no wrapper degrading a parse failure to None
162+
# to keep teardown unconditional. An environment entered without a
163+
# table simply exits without one.
164+
self._resolved_symtab = resolved_symtab
119165

120166
def run(self):
121167
self._session.exit_environment(
122-
identifier=self._id, keep_session_running=self._keep_session_running
168+
identifier=self._id,
169+
keep_session_running=self._keep_session_running,
170+
resolved_symtab=self._resolved_symtab,
123171
)
124172

125173
def __str__(self):

0 commit comments

Comments
 (0)