Skip to content

Commit 5fe426d

Browse files
Rishabh0255Rishabh Devnani
andauthored
Feat/pipeline steps v2 backport (#6153)
* feat: Add Zimmer deployment and lineage step types (v2 backport) Backports the 8 Zimmer pipeline step classes from v3 (PR #6140) to v2: - EndpointConfigStep, EndpointStep (SageMaker inference deployment) - InferenceComponentStep (multi-model endpoint support) - BedrockCustomModelStep, BedrockCustomModelDeploymentStep, BedrockModelImportStep, BedrockProvisionedModelThroughputStep - LineageStep (ML governance tracking) Also adds 8 new values to StepTypeEnum. Design: each step accepts an opaque 'arguments: Dict[str, Any]' that is forwarded to the pipeline service verbatim, mirroring the existing SDK convention used by LambdaStep and CallbackStep. Refer to the corresponding AWS Create* API references for accepted fields. Retryability per Zimmer step contract: only EndpointConfigStep is retryable. Cacheability: EndpointConfigStep and EndpointStep are structurally cacheable via cache_config. sim: https://issues.amazon.com/issues/P424919850 --- X-AI-Prompt: Backport the Zimmer pipeline step types from the v3 PR (#6140) to the v2 SDK on master-v2 X-AI-Tool: kiro-cli * fix(pipeline): Add client-side argument validation for Zimmer steps Mirrors the v3 change (commit fbe6fce on feat/zimmer-pipeline-steps): validate top-level argument keys against the public AWS API input shape (botocore service model) at step construction and at serialization, and fail fast on fields SageMaker Pipelines rejects (DataCaptureConfig, ExplainerConfig on EndpointConfig; DeploymentConfig on Endpoint). Bedrock member names are PascalCase-converted before comparison. LineageStep now rejects unrecognized top-level keys. Values are intentionally not validated: they may be pipeline variables resolved at compile time. If the installed botocore does not know an operation, shape validation is skipped and the service remains the authority. Adds 8 validation unit tests (32 total). --- X-AI-Prompt: Mirror the Fortress-driven argument validation change to the v2 backport branch X-AI-Tool: kiro-cli * chore: empty commit to re-trigger CI Re-trigger the PR check suite (integ tests run issue). --- X-AI-Prompt: Make dummy commits on v2 and v3 branches to fix an integ tests run issue X-AI-Tool: kiro-cli --------- Co-authored-by: Rishabh Devnani <rdevnani@amazon.com>
1 parent eaa80b9 commit 5fe426d

8 files changed

Lines changed: 1441 additions & 0 deletions

File tree

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License"). You
4+
# may not use this file except in compliance with the License. A copy of
5+
# the License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# or in the "license" file accompanying this file. This file is
10+
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
11+
# ANY KIND, either express or implied. See the License for the specific
12+
# language governing permissions and limitations under the License.
13+
"""Client-side validation for pipeline step ``arguments`` blocks.
14+
15+
Validates the **top-level keys** of a step's ``arguments`` dict against
16+
the corresponding public AWS API input shape from botocore, and rejects
17+
fields that SageMaker Pipelines is known not to support. This fails fast
18+
at step construction with a clear error, instead of a server-side parse
19+
failure at ``CreatePipeline`` time.
20+
21+
Values are intentionally not validated: they may be pipeline variables
22+
(parameter references, step property references, ``Join``/``JsonGet``
23+
expressions) that only resolve at pipeline compile or execution time.
24+
25+
If the installed botocore release does not know the target operation
26+
(for example, a very old botocore without newer Bedrock APIs), shape
27+
validation is skipped and the service remains the authority.
28+
"""
29+
30+
from __future__ import absolute_import
31+
32+
import logging
33+
from typing import Any, Dict, FrozenSet, Optional, Sequence, Tuple
34+
35+
import botocore.session
36+
from botocore.exceptions import UnknownServiceError
37+
from botocore.model import OperationNotFoundError
38+
39+
logger = logging.getLogger(__name__)
40+
41+
# Cache of (service, operation, pascal_case) -> allowed top-level keys.
42+
# ``None`` means botocore does not know the operation; skip shape checks.
43+
_SHAPE_CACHE: Dict[Tuple[str, str, bool], Optional[FrozenSet[str]]] = {}
44+
45+
46+
def _allowed_top_level_keys(
47+
service_name: str, operation_name: str, pascal_case: bool
48+
) -> Optional[FrozenSet[str]]:
49+
"""Return the allowed top-level keys for an operation input shape.
50+
51+
Args:
52+
service_name (str): botocore service name (e.g. ``sagemaker``).
53+
operation_name (str): operation name (e.g. ``CreateEndpointConfig``).
54+
pascal_case (bool): If True, convert member names to PascalCase
55+
(used for Bedrock, whose JSON API members are camelCase but
56+
whose pipeline ``Arguments`` fields are PascalCase).
57+
58+
Returns:
59+
The allowed key set, or ``None`` if the installed botocore does
60+
not know the operation (validation should then be skipped).
61+
"""
62+
cache_key = (service_name, operation_name, pascal_case)
63+
if cache_key not in _SHAPE_CACHE:
64+
try:
65+
session = botocore.session.get_session()
66+
service_model = session.get_service_model(service_name)
67+
operation_model = service_model.operation_model(operation_name)
68+
members = operation_model.input_shape.members.keys()
69+
if pascal_case:
70+
members = [m[0].upper() + m[1:] for m in members]
71+
_SHAPE_CACHE[cache_key] = frozenset(members)
72+
except (UnknownServiceError, OperationNotFoundError):
73+
logger.warning(
74+
"Installed botocore does not know %s.%s; skipping "
75+
"client-side argument shape validation for this step.",
76+
service_name,
77+
operation_name,
78+
)
79+
_SHAPE_CACHE[cache_key] = None
80+
return _SHAPE_CACHE[cache_key]
81+
82+
83+
def validate_step_arguments(
84+
step_class_name: str,
85+
arguments: Dict[str, Any],
86+
service_name: str,
87+
operation_name: str,
88+
unsupported_fields: Sequence[str] = (),
89+
pascal_case: bool = False,
90+
) -> None:
91+
"""Validate the top-level keys of a step ``arguments`` dict.
92+
93+
Args:
94+
step_class_name (str): Step class name, used in error messages.
95+
arguments (Dict[str, Any]): The user-provided ``arguments`` dict.
96+
service_name (str): botocore service name of the wrapped API.
97+
operation_name (str): Operation whose input shape defines the
98+
allowed top-level fields.
99+
unsupported_fields (Sequence[str]): Fields that exist in the
100+
public API shape but are rejected by SageMaker Pipelines.
101+
pascal_case (bool): Convert botocore member names to PascalCase
102+
before comparison (Bedrock APIs).
103+
104+
Raises:
105+
ValueError: If ``arguments`` is not a non-empty dict with string
106+
keys, contains an unsupported field, or contains a key that
107+
is not part of the operation's input shape.
108+
"""
109+
if arguments is None:
110+
raise ValueError(f"arguments is required for {step_class_name}.")
111+
if not isinstance(arguments, dict) or not arguments:
112+
raise ValueError(f"{step_class_name}: arguments must be a non-empty dict.")
113+
non_string_keys = [key for key in arguments if not isinstance(key, str)]
114+
if non_string_keys:
115+
raise ValueError(
116+
f"{step_class_name}: argument keys must be strings; got {non_string_keys!r}."
117+
)
118+
rejected = sorted(field for field in unsupported_fields if field in arguments)
119+
if rejected:
120+
raise ValueError(
121+
f"{step_class_name}: field(s) {rejected} are not supported by "
122+
"SageMaker Pipelines and would be rejected at pipeline creation "
123+
"time. Remove them from arguments."
124+
)
125+
allowed = _allowed_top_level_keys(service_name, operation_name, pascal_case)
126+
if allowed is None:
127+
return
128+
unknown = sorted(set(arguments) - allowed)
129+
if unknown:
130+
raise ValueError(
131+
f"{step_class_name}: unknown argument field(s) {unknown}. "
132+
f"Allowed top-level fields (from {service_name}.{operation_name}): "
133+
f"{sorted(allowed)}."
134+
)

0 commit comments

Comments
 (0)