Skip to content

Commit 0c1faa3

Browse files
authored
Merge branch 'master' into fix/jumpstart-private-hub-modeltrainer
2 parents 6593984 + b981113 commit 0c1faa3

8 files changed

Lines changed: 264 additions & 117 deletions

File tree

‎.github/copilot-instructions.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# GitHub Copilot instructions — Amazon SageMaker Python SDK
2+
3+
Guidance for [GitHub Copilot](https://docs.github.com/en/copilot) working in
4+
**this repository** — the source of the Amazon SageMaker Python SDK.
5+
6+
This repository keeps its AI-agent guidance in a single source of truth,
7+
[`AGENTS.md`](../AGENTS.md), following the [AGENTS.md](https://agents.md) convention.
8+
Copilot instructions do not support file imports, so **read
9+
[`AGENTS.md`](../AGENTS.md) at the repository root and follow it**.
10+
11+
Key points from that file (see `AGENTS.md` for the authoritative, complete version):
12+
13+
- **v3 by default.** The current major version is v3 (`pip install sagemaker`). SDK v3 is a
14+
modular redesign and is **not** backward compatible with v2. Generate v3 patterns in all
15+
example code, docstrings, tests, and docs unless v2 is explicitly in scope.
16+
- **SDK-first.** Use the SageMaker Python SDK v3 as the primary interface (e.g.
17+
`sagemaker.train.ModelTrainer`, `sagemaker.serve.ModelBuilder`); do not drop to raw
18+
`boto3`, the AWS CLI, or hand-rolled scripts unless the SDK genuinely does not cover the
19+
task.
20+
- **No banned v2 patterns** (e.g. `sagemaker.estimator.Estimator`, `estimator.fit(...)`,
21+
framework estimator classes, `sagemaker.model.Model`) in new code. See the v2 → v3
22+
mapping table in `AGENTS.md` and [`migration.md`](../migration.md).
23+
- **Contributing:** add/update unit tests under `tests/unit/` for code changes, run the
24+
configured formatters/linters, and keep `migration.md` and docstrings consistent for
25+
public API changes. See [`CONTRIBUTING.md`](../CONTRIBUTING.md).

‎.github/workflows/pr-checks-master.yml‎

Lines changed: 34 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -246,18 +246,36 @@ jobs:
246246
# Additive: runs the shallow (submit-then-stop) suite for sagemaker-train
247247
# alongside the existing integ-tests job above, which is unchanged.
248248
#
249-
# Why a separate job rather than folding this into the CodeBuild suite: this
250-
# job's selection is reviewable in the PR that changes it, whereas the
251-
# sagemaker-train CodeBuild buildspec is CDK-managed outside this repo. It also
252-
# reports as its own check, so a shallow failure is distinguishable at a glance
253-
# from a deep-suite failure, and it finishes in minutes -- fast feedback that
254-
# does not wait on the 2XLARGE CodeBuild container.
249+
# Runs in CodeBuild, not on the runner. It began as a runner job, but
250+
# actions/checkout refuses to place a fork's head commit in a
251+
# pull_request_target job -- correctly, because the runner also holds the base
252+
# repo's GITHUB_TOKEN and assumes CI_AWS_ROLE_ARN, so a fork could edit
253+
# conftest.py and read those credentials out. That is the "pwn request" shape,
254+
# and on a public repo it is a live credential-exfiltration path, so the fix is
255+
# to move the execution rather than override the refusal with
256+
# allow-unsafe-pr-checkout. Nearly every PR here comes from a fork, so a
257+
# same-repo guard would have left the suite with almost no gate coverage.
258+
# source-version-override is how the three jobs above already run PR code: the
259+
# build never sees the runner's token, secrets or default-branch cache.
255260
#
256-
# What runs here: only tests/integ/train/shallow. The client-side tests
257-
# (recipe resolution, data utils, dry-run, log streaming) are deliberately NOT
258-
# repeated -- the CodeBuild suite already runs the whole tests/integ tree, so
259-
# widening this job's scope would duplicate them and double the job creation
260-
# the shallow suite performs.
261+
# Why its own project rather than folding this into the sagemaker-train
262+
# integ-tests project: it reports as its own check, so a shallow failure is
263+
# distinguishable at a glance from a deep-suite failure, and it runs
264+
# concurrently with the deep suite instead of queueing behind it.
265+
#
266+
# Tradeoff of moving off the runner: the pytest selection now lives in the
267+
# CDK's buildspecs.ts (createCIShallowIntegBuildSpec) instead of this file, so
268+
# changing which tests run is no longer reviewable in a PR to this repo. That
269+
# is the price of executing fork code safely, and it is the same place the
270+
# other three test jobs' selections already live.
271+
#
272+
# What runs there: only tests/integ/train/shallow, deselecting gpu_intensive
273+
# (the CPT and MTRL classes, which need a pre-provisioned HyperPod cluster and
274+
# an agent runtime plus an MLflow app) and us_east_1 (Nova cases, which run in
275+
# the integ-tests-us-east-1 project against the Nova account). The client-side
276+
# tests are deliberately not repeated -- the deep suite already runs the whole
277+
# tests/integ tree, so widening scope would duplicate them and double the job
278+
# creation the shallow suite performs.
261279
#
262280
# Why submit-then-stop is worth gating on: CreateTrainingJob returns a
263281
# TrainingJobArn only after the request has cleared public-model validation,
@@ -272,88 +290,22 @@ jobs:
272290
fast-integ-tests:
273291
runs-on: ubuntu-latest
274292
needs: [detect-changes]
293+
# No same-repo guard: nothing here checks out PR code, so fork PRs are gated
294+
# too. The suite only runs when sagemaker-train is in the change set.
275295
if: contains(fromJson(needs.detect-changes.outputs.submodules), 'sagemaker-train')
276296
steps:
277-
- uses: actions/checkout@v3
278-
with:
279-
# pull_request_target checks out the base ref by default; these tests
280-
# must run against the PR's code.
281-
ref: 'refs/pull/${{ github.event.pull_request.number }}/head'
282-
283-
- name: Set up Python
284-
uses: actions/setup-python@v5
285-
with:
286-
python-version: '3.12'
287-
288297
- name: Configure AWS Credentials
289298
uses: aws-actions/configure-aws-credentials@v4
290299
with:
291300
role-to-assume: ${{ secrets.CI_AWS_ROLE_ARN }}
292301
aws-region: us-west-2
293302
role-duration-seconds: 10800
294303

295-
- name: Install sagemaker-train and test dependencies
296-
run: |
297-
python -m pip install --upgrade pip
298-
pip install ./sagemaker-core
299-
pip install ./sagemaker-train
300-
pip install -r requirements/extras/test_requirements.txt
301-
302304
- name: Run shallow sagemaker-train integ tests
303-
working-directory: sagemaker-train
304-
env:
305-
AWS_DEFAULT_REGION: us-west-2
306-
# Role resolution goes through iam:SimulatePrincipalPolicy, which is
307-
# low-TPS; adaptive retries keep parallel workers from throttling each
308-
# other.
309-
AWS_RETRY_MODE: adaptive
310-
AWS_MAX_ATTEMPTS: '10'
311-
# Cap the training jobs the *service* counts against the concurrency
312-
# quota, across all xdist workers, so the suite stays inside the
313-
# "concurrent model customization serverless jobs per Region" quota
314-
# (20) with room for the deep integ-tests suite running the same
315-
# account concurrently. The harness holds each slot until the job is
316-
# terminal, not until stop() returns -- the service counts a job for
317-
# ~1-3 min after the stop -- so 10 means "at most 10 jobs counted at
318-
# once", the batches-of-10 behaviour, not "10 stops in flight".
319-
#
320-
# Not redundant with -n 8. -n caps worker processes; this caps what
321-
# the service counts, and with the slot held to terminal those diverge
322-
# sharply (each drain outlives the worker's stop() by minutes). It is
323-
# also what keeps the ceiling stable if -n is raised. A serverful job
324-
# counts one slot per instance.
325-
SHALLOW_MAX_CONCURRENT_JOBS: '10'
326-
run: |
327-
# Scoped to shallow/ only -- see the comment above this job for why the
328-
# rest of tests/integ/train is not repeated here.
329-
#
330-
# 84 of the suite's 100 tests run; the 16 deselected are:
331-
# us_east_1 (5) -- Nova cases; this job holds us-west-2
332-
# credentials only, so they run in the
333-
# integ-tests-us-east-1 job instead.
334-
# gpu_intensive (11) -- the CPT and MTRL classes. These are written in
335-
# the shallow style but cannot be made
336-
# self-contained: CPT submits only via HyperPod
337-
# (a pre-provisioned cluster, not
338-
# CreateTrainingJob) and MTRL needs an agent
339-
# runtime plus an MLflow app. Both become
340-
# gate-eligible by dropping one marker once those
341-
# prerequisites exist in the PR account.
342-
python -m pytest tests/integ/train/shallow \
343-
-m "not gpu_intensive and not us_east_1" \
344-
-n 8 \
345-
--dist loadfile \
346-
-v \
347-
--durations=15 \
348-
--junitxml=shallow-integ-results.xml
349-
350-
- name: Upload test results
351-
if: always()
352-
uses: actions/upload-artifact@v4
305+
uses: aws-actions/aws-codebuild-run-build@v1
353306
with:
354-
name: shallow-integ-test-results
355-
path: sagemaker-train/shallow-integ-results.xml
356-
if-no-files-found: warn
307+
project-name: ${{ github.event.repository.name }}-ci-sagemaker-train-fast-integ-tests
308+
source-version-override: 'refs/pull/${{ github.event.pull_request.number }}/head^{${{ github.event.pull_request.head.sha }}}'
357309

358310
integ-tests-us-east-1:
359311
runs-on: ubuntu-latest

‎CLAUDE.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# CLAUDE.md — Amazon SageMaker Python SDK
2+
3+
Guidance for [Claude Code](https://docs.anthropic.com/en/docs/claude-code) working in
4+
**this repository** — the source of the Amazon SageMaker Python SDK.
5+
6+
This repository follows the [AGENTS.md](https://agents.md) convention. To keep a single
7+
source of truth and avoid the two files drifting apart, all guidance lives in
8+
[`AGENTS.md`](./AGENTS.md) and is imported here:
9+
10+
@AGENTS.md
11+
12+
For the full guidance — project context, the **v3-by-default** golden rule, banned v2
13+
patterns and their v3 replacements, the SDK-first interface map, the contributing
14+
workflow, and a canonical v3 train + deploy example — read [`AGENTS.md`](./AGENTS.md)
15+
directly.

‎GEMINI.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# GEMINI.md — Amazon SageMaker Python SDK
2+
3+
Guidance for [Gemini CLI](https://github.com/google-gemini/gemini-cli) working in
4+
**this repository** — the source of the Amazon SageMaker Python SDK.
5+
6+
This repository follows the [AGENTS.md](https://agents.md) convention. To keep a single
7+
source of truth and avoid the files drifting apart, all guidance lives in
8+
[`AGENTS.md`](./AGENTS.md) and is imported here:
9+
10+
@AGENTS.md
11+
12+
For the full guidance — project context, the **v3-by-default** golden rule, banned v2
13+
patterns and their v3 replacements, the SDK-first interface map, the contributing
14+
workflow, and a canonical v3 train + deploy example — read [`AGENTS.md`](./AGENTS.md)
15+
directly.

‎sagemaker-train/src/sagemaker/train/rlaif_trainer.py‎

Lines changed: 67 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ class RLAIFTrainer(BaseTrainer):
4949
training_type=TrainingType.LORA,
5050
model_package_group="my-model-group",
5151
reward_model_id="reward-model-id",
52-
reward_prompt="Rate the helpfulness of this response on a scale of 1-10",
52+
reward_prompt="summarize",
5353
training_dataset="s3://bucket/rlaif_data.jsonl"
5454
)
5555
@@ -60,7 +60,7 @@ class RLAIFTrainer(BaseTrainer):
6060
model="meta-llama/Llama-2-7b-hf",
6161
model_package_group="my-rlaif-models",
6262
reward_model_id="reward-model-id",
63-
reward_prompt="Rate the helpfulness of this response on a scale of 1-10"
63+
reward_prompt="summarize"
6464
)
6565
6666
# Create training job (non-blocking)
@@ -396,8 +396,14 @@ def _process_hyperparameters(self):
396396
# Process reward_prompt parameter
397397
if hasattr(self, 'reward_prompt') and self.reward_prompt:
398398
if isinstance(self.reward_prompt, str):
399-
if self.reward_prompt.startswith("Builtin"):
400-
# Handle builtin reward prompts
399+
# Resolution order:
400+
# 1. Preset template name -> resolved locally against the recipe's
401+
# judge_prompt_template enum (no API call). Accepts "Builtin.Summarize",
402+
# "summarize", or "summarize.jinja".
403+
# 2. Evaluator ARN -> validated/assigned as-is.
404+
# 3. Otherwise -> HubContent name lookup (custom registered prompt),
405+
# which raises a clear error if not found.
406+
if self._is_preset_reward_prompt(self.reward_prompt):
401407
self._update_judge_prompt_template_direct(self.reward_prompt)
402408
else:
403409
# Handle evaluator ARN or hub content name
@@ -411,9 +417,44 @@ def _process_hyperparameters(self):
411417
evaluator_arn = _extract_evaluator_arn(self.reward_prompt, "reward_prompt")
412418
self._evaluator_arn = evaluator_arn
413419

420+
@staticmethod
421+
def _normalize_template_name(value: str) -> str:
422+
"""Normalize a preset name or enum path to a comparable key.
423+
424+
Handles an optional "Builtin." prefix, any path prefix, and an optional
425+
".jinja" suffix, case-insensitively. For example "Builtin.Summarize",
426+
"summarize", "summarize.jinja", and "/opt/ml/code/verl/summarize.jinja"
427+
all normalize to "summarize".
428+
"""
429+
name = (value or "").strip()
430+
if name.lower().startswith("builtin."):
431+
name = name.split(".", 1)[1]
432+
name = name.split("/")[-1] # basename
433+
if name.lower().endswith(".jinja"):
434+
name = name[: -len(".jinja")]
435+
return name.lower()
436+
437+
def _get_judge_prompt_template_enum(self):
438+
"""Return the recipe's judge_prompt_template enum values (already in memory)."""
439+
if not self.hyperparameters or not getattr(self.hyperparameters, "_specs", None):
440+
return []
441+
judge_prompt_spec = self.hyperparameters._specs.get("judge_prompt_template", {})
442+
return judge_prompt_spec.get("enum", []) or []
443+
444+
def _is_preset_reward_prompt(self, reward_prompt: str) -> bool:
445+
"""True if reward_prompt matches a recipe preset template (local, no API call).
446+
447+
An explicit "Builtin." prefix always routes to preset resolution so the
448+
user gets a clear "not available" error instead of a HubContent lookup.
449+
"""
450+
if reward_prompt.startswith("Builtin"):
451+
return True
452+
enum_keys = {self._normalize_template_name(e) for e in self._get_judge_prompt_template_enum()}
453+
return self._normalize_template_name(reward_prompt) in enum_keys
454+
414455
def _process_non_builtin_reward_prompt(self):
415-
"""Process non-builtin reward prompt (ARN or hub content name)."""
416-
# Remove judge_prompt_template for non-builtin prompts
456+
"""Process non-preset reward prompt (ARN or hub content name)."""
457+
# Remove judge_prompt_template for non-preset prompts
417458
if hasattr(self.hyperparameters, 'judge_prompt_template'):
418459
delattr(self.hyperparameters, 'judge_prompt_template')
419460
self.hyperparameters._specs.pop('judge_prompt_template', None)
@@ -442,37 +483,42 @@ def _process_non_builtin_reward_prompt(self):
442483

443484

444485
def _update_judge_prompt_template_direct(self, reward_prompt):
445-
"""Update judge_prompt_template based on Builtin reward function."""
486+
"""Resolve a preset reward prompt name to the recipe's judge_prompt_template value.
487+
488+
Accepts "Builtin.Summarize", "summarize", or "summarize.jinja" and matches
489+
it against the recipe's judge_prompt_template enum (normalized by basename,
490+
with an optional ".jinja" suffix). No API call is made.
491+
"""
446492
# Get available templates from hyperparameters specs
447-
judge_prompt_spec = self.hyperparameters._specs.get('judge_prompt_template', {})
448-
available_templates = judge_prompt_spec.get('enum', [])
449-
493+
available_templates = self._get_judge_prompt_template_enum()
494+
450495
if not available_templates:
451496
# If no enum found, use the current value as the only available option
452497
current_value = getattr(self.hyperparameters, 'judge_prompt_template', None)
453498
if current_value:
454499
available_templates = [current_value]
455500
else:
456501
return
457-
458-
# Extract template name after "Builtin." and convert to lowercase
459-
template_name = reward_prompt.split(".", 1)[1].lower()
460-
461-
# Find matching template by extracting filename without extension
502+
503+
# Normalize the requested name (strips optional "Builtin." prefix and ".jinja")
504+
template_name = self._normalize_template_name(reward_prompt)
505+
506+
# Find matching template by normalized basename
462507
matching_template = None
463508
for template in available_templates:
464-
template_filename = template.split("/")[-1].replace(".jinja", "").lower()
465-
if template_filename == template_name:
509+
if self._normalize_template_name(template) == template_name:
466510
matching_template = template
467511
break
468512

469513
if matching_template:
470514
self.hyperparameters.judge_prompt_template = matching_template
471515
else:
472-
available_options = [f"Builtin.{t.split('/')[-1].replace('.jinja', '')}" for t in available_templates]
516+
available_options = [self._normalize_template_name(t) for t in available_templates]
473517
raise ValueError(
474-
f"Selected reward function option '{reward_prompt}' is not available. "
475-
f"Choose one from the available options: {available_options}. "
476-
f"Example: reward_prompt='Builtin.summarize'"
518+
f"Selected reward prompt '{reward_prompt}' is not an available preset. "
519+
f"Choose one from the available options: {available_options} "
520+
f"(pass the name directly, e.g. reward_prompt='{available_options[0]}', "
521+
f"or with the 'Builtin.' prefix). "
522+
f"Alternatively pass an evaluator ARN or a registered HubContent prompt name."
477523
)
478524

‎sagemaker-train/tests/integ/train/shallow/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,23 @@ What this suite changes about the gate is not which job runs, but what the exist
88
one selects: the `gpu_intensive` marks added here deselect the deep tests that
99
submit a job and wait for it, and this suite covers those code paths instead.
1010

11+
> **Where this runs.** The `fast-integ-tests` job in `pr-checks-master.yml` does not
12+
> execute this suite on the GitHub runner — it starts the CodeBuild project
13+
> `sagemaker-python-sdk-ci-sagemaker-train-fast-integ-tests` via
14+
> `source-version-override`, the same way the `codestyle-doc-tests`, `unit-tests`
15+
> and `integ-tests` jobs run PR code. That is deliberate: the runner holds the base
16+
> repo's `GITHUB_TOKEN` and assumes `CI_AWS_ROLE_ARN`, so `actions/checkout` refuses
17+
> to place a fork's head commit there, and on a public repo overriding that refusal
18+
> would be a live credential-exfiltration path. Running in CodeBuild gates fork PRs
19+
> — which is nearly all of them — without exposing those credentials to PR code.
20+
>
21+
> **Consequence for editing this suite:** the marker selection above (`-n 8`,
22+
> `--dist loadfile`, `-m "not gpu_intensive and not us_east_1"`) lives in
23+
> `createCIShallowIntegBuildSpec` in the `SageMakerMLFPySDKInfraCDK` package, not in
24+
> this repo. Adding a file under `shallow/` is picked up automatically, but changing
25+
> *how* the suite is invoked means a change there, which deploys through a pipeline
26+
> rather than merging with your PR.
27+
1128
## What a passing test proves
1229

1330
Each test submits a real `CreateTrainingJob`, asserts the service returned a

0 commit comments

Comments
 (0)