Skip to content

fix(train): keep the timestamp when truncating long job names - #6371

Open
kwy404 wants to merge 2 commits into
aws:masterfrom
kwy404:fix-unique-name-timestamp
Open

kwy404 wants to merge 2 commits into
aws:masterfrom
kwy404:fix-unique-name-timestamp

Conversation

@kwy404

@kwy404 kwy404 commented Oct 1, 2026

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

Root cause: _get_unique_name in sagemaker-train appends a 14 digit timestamp to the base name and then cuts the whole string to max_length (63). When the base name is longer than 48 characters, that cut removes part or all of the timestamp, so repeated calls return the same name and the next job fails because the name is already in use.

Long bases come from the trainers' own defaults. For example RLVRTrainer(model="huggingface-reasoning-nvidia-nemotron-3-nano-30b-a3b-bf16") uses f"{model_name}-rlvr" (62 characters), and every call currently returns huggingface-reasoning-nvidia-nemotron-3-nano-30b-a3b-bf16-rlvr-, with no timestamp at all. That name also ends in a hyphen, which the TrainingJobName pattern does not allow.

Fix: trim the base instead of the final string, the same way name_from_base in sagemaker-core already does. The full timestamp is always kept and the result is still at most max_length characters.

Test: test_get_unique_name_keeps_timestamp_for_long_base_name in tests/unit/train/test_model_trainer.py pins the clock and checks that a 60 character base gives "a" * 48 + "-20260926123456". It fails on master (the name ends in -20) and passes with the fix. black and flake8 are clean on the changed files.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

_get_unique_name appended the timestamp and then cut the whole name to
max_length, so a base name longer than 48 characters lost part or all
of the timestamp and repeated calls produced the same job name. Trim
the base instead, like name_from_base in sagemaker-core, so the full
timestamp is always kept.

This branch is waiting to be deployed

1 waiting deployment
manual-approval — e92a7d21 Waiting Oct 27, 2026 by kwy404 via wait-for-approval #2127
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant