Repository navigation
feat: allow configurable S3 key prefix for ModelTrainer input uploads (#5638) - #6354
Conversation
…aws#5638) Add an optional ModelTrainer.input_s3_key_prefix field. When set, it is used as the leading segment of the input-data S3 key prefix (e.g. a pipeline name) instead of base_job_name; when None, the upload layout is unchanged. Covers the SDK-managed code/sm_drivers/recipe channels and local input channels via _create_training_job_args, and the standalone create_input_data_channel default branch.
|
Claude finished @mohamedzeidan2021's task in 58s —— View job Code Review — configurable
|
Problem
ModelTrainerderives the leading S3 key prefix for input-data uploads solely frombase_job_name, so there is no way to group a job's staged artifacts under a custom prefix such as a pipeline name (issue #5638). Uploads land unders3://<bucket>/<base_job_name>/<job_name>/input/....Fix
Add an optional
ModelTrainer.input_s3_key_prefixfield. When set, it is used as the leading segment of the input-data key prefix instead ofbase_job_name; when unset (None), the upload layout is byte-for-byte unchanged. The change touches the two prefix-derivation sites (_create_training_job_argsfor the SDK-managed code/sm_drivers/recipe channels and local user channels, and the standalonecreate_input_data_channeldefault branch) and registers the field inCONFIGURABLE_ATTRIBUTESso it can be supplied via intelligent defaults. Output, checkpoint, and tensorboard prefixes are intentionally left unchanged.Testing
sagemaker-train/tests/unit/train/test_model_trainer.pycovering the unchanged default path, the custom-prefixcreate_input_data_channelpath, and the custom prefix flowing through_create_training_job_argsto the managed channels' S3 URIs.test_model_trainer.pymodule passes (76 passed). Verified the new tests fail against the unmodified source (negative control).black+flake8clean.Fixes #5638