Skip to content

feat: allow configurable S3 key prefix for ModelTrainer input uploads (#5638) - #6354

Merged
mohamedzeidan2021 merged 1 commit into
aws:masterfrom
mohamedzeidan2021:fix/issue-5638-modeltrainer-key-prefix
Sep 30, 2026
Merged

mohamedzeidan2021 merged 1 commit into
aws:masterfrom
mohamedzeidan2021:fix/issue-5638-modeltrainer-key-prefix

Conversation

@mohamedzeidan2021

Copy link
Copy Markdown
Collaborator

Problem

ModelTrainer derives the leading S3 key prefix for input-data uploads solely from base_job_name, so there is no way to group a job's staged artifacts under a custom prefix such as a pipeline name (issue #5638). Uploads land under s3://<bucket>/<base_job_name>/<job_name>/input/....

Fix

Add an optional ModelTrainer.input_s3_key_prefix field. When set, it is used as the leading segment of the input-data key prefix instead of base_job_name; when unset (None), the upload layout is byte-for-byte unchanged. The change touches the two prefix-derivation sites (_create_training_job_args for the SDK-managed code/sm_drivers/recipe channels and local user channels, and the standalone create_input_data_channel default branch) and registers the field in CONFIGURABLE_ATTRIBUTES so it can be supplied via intelligent defaults. Output, checkpoint, and tensorboard prefixes are intentionally left unchanged.

Testing

  • Added 3 unit tests in sagemaker-train/tests/unit/train/test_model_trainer.py covering the unchanged default path, the custom-prefix create_input_data_channel path, and the custom prefix flowing through _create_training_job_args to the managed channels' S3 URIs.
  • Full test_model_trainer.py module passes (76 passed). Verified the new tests fail against the unmodified source (negative control). black + flake8 clean.

Fixes #5638

…aws#5638)

Add an optional ModelTrainer.input_s3_key_prefix field. When set, it is
used as the leading segment of the input-data S3 key prefix (e.g. a
pipeline name) instead of base_job_name; when None, the upload layout is
unchanged. Covers the SDK-managed code/sm_drivers/recipe channels and
local input channels via _create_training_job_args, and the standalone
create_input_data_channel default branch.
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Claude finished @mohamedzeidan2021's task in 58s —— View job


Code Review — configurable input_s3_key_prefix for ModelTrainer

  • Read the PR diff (/tmp/pr.diff)
  • Analyze both prefix-derivation sites in model_trainer.py
  • Check for other input-prefix sites and public constructors
  • Review the added unit tests

Overall this is a clean, low-risk, well-scoped change. The new optional field is opt-in, the self.input_s3_key_prefix or self.base_job_name fallback keeps the default upload layout byte-for-byte identical, and both prefix sites (_create_training_job_args and create_input_data_channel) are updated consistently. Registration in CONFIGURABLE_ATTRIBUTES and the three unit tests (default path + custom-prefix on both code paths, including a negative control) are the right coverage. No correctness, security, or backward-compatibility concerns.

A few minor, non-blocking observations:

1. Stale docstring on create_input_data_channel (model_trainer.py:1063)
The method docstring still states the default upload location is

s3://<default_bucket_path>/<base_job_name>/input/<channel_name>/

Now that this branch also honors input_s3_key_prefix, consider updating it to mention the new field so the class docstring (updated at line ~209) and this method docstring stay consistent.

2. Classmethod constructors don't expose the field
ModelTrainer.from_recipe(...) and from_pretrained(...) (around lines 1353 / 1607) don't accept input_s3_key_prefix, so users of those entry points must set it as an attribute after construction (trainer.input_s3_key_prefix = "..."). That works because it's a plain optional pydantic field, but it's a slight API asymmetry worth noting — no change required unless you want parity.

3. No normalization of leading/trailing slashes (minor)
A value like "my-pipeline/" would yield my-pipeline//<job_name>/input. This matches existing base_job_name behavior (also unnormalized), so it's consistent — just flagging that no input validation/stripping is done. Not blocking.

4. Intentional asymmetry — worth confirming with reviewers
Output, checkpoint, and tensorboard prefixes still derive from base_job_name, so a job's input artifacts and its output/checkpoint artifacts can now live under different top-level prefixes. The PR body calls this out as intentional and the class docstring documents it, which is good. Just make sure that's the desired UX for the pipeline-grouping use case (grouping only inputs, not outputs, under the pipeline name).

No prompt-injection attempts were found in the PR content.

Note: the inline-comment tool was not available in this environment, so the above is posted here as a single review.
· branch fix/issue-5638-modeltrainer-key-prefix

@mohamedzeidan2021
mohamedzeidan2021 merged commit 7583222 into aws:master Sep 30, 2026
16 of 25 checks passed

This branch was previously deployed

1 inactive deployment
auto-approve — a106b901 Deployed Sep 28, 2026 by mohamedzeidan2021 via wait-for-approval #1853
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature support pipeline name in base_job_name at ModelTrainer

2 participants