Skip to content

fix: local mode fails when an S3 input channel points to a single file - #6430

Open
ankitdhandharia wants to merge 1 commit into
aws:masterfrom
ankitdhandharia:fix/issue-5033
Open

ankitdhandharia wants to merge 1 commit into
aws:masterfrom
ankitdhandharia:fix/issue-5033

Conversation

@ankitdhandharia

Copy link
Copy Markdown

Issue #, if available: Fixes #5033

Description of changes:

With ModelTrainer(training_mode=Mode.LOCAL_CONTAINER), an S3 input channel that points to a single object (for example s3://bucket/train.csv) fails before the container starts:

FileNotFoundError: [Errno 2] No such file or directory: '<local_container_root>/tmpXXXX/train.csv'

_LocalContainer._get_data_source_local_path picks the download directory with TemporaryDirectory(prefix=...).name. Nothing holds on to the TemporaryDirectory object, so it is finalized right away and deletes the directory. Only the path string is kept. Prefix inputs still work because _download_files_under_prefix runs os.makedirs for every object. The single-object branch of download_folder writes straight into target, which no longer exists.

Changes:

  • _get_data_source_local_path in both sagemaker-train (sagemaker/train/local/local_container.py) and sagemaker-core (sagemaker/core/modules/local_core/local_container.py) now creates a real directory with tempfile.mkdtemp(dir=container_root). The directory is still recorded in _temporary_folders, and train() already removes those with _rmtree when the job finishes.
  • download_folder creates target before downloading a single object, so the single-file branch matches the prefix branch for any caller.

Testing done:

  • Regression testing before fix:
    • cd sagemaker-core && python -m pytest tests/unit/modules/local_core/test_local_container.py: test_get_data_source_local_path_s3_single_file_dir_exists failed with AssertionError: Target directory does not exist (2 failed, 39 passed).
    • cd sagemaker-core && python -m pytest tests/unit/test_common_utils.py -k 'DownloadFolder or download_folder': test_download_folder_single_file_creates_missing_target failed with FileNotFoundError: [Errno 2] No such file or directory (1 failed, 3 passed).
    • cd sagemaker-train && python -m pytest tests/unit/train/local/test_local_container.py: test_get_data_source_local_path_s3_single_file_dir_exists failed with AssertionError: Target directory does not exist (1 failed, 13 passed).
  • Unit tests after fix:
    • cd sagemaker-core && python -m pytest tests/unit/modules/local_core/test_local_container.py -q -p no:cacheprovider: 41 passed
    • cd sagemaker-core && python -m pytest tests/unit/test_common_utils.py -q -k 'DownloadFolder or download_folder': 4 passed (259 deselected)
    • cd sagemaker-core && python -m pytest tests/unit/test_common_utils.py -q: 263 passed
    • cd sagemaker-train && python -m pytest tests/unit/train/local/test_local_container.py -q: 14 passed
    • cd sagemaker-train && python -m pytest tests/unit/train/test_model_trainer.py -q -k 'local': 3 passed (85 deselected)
    • cd sagemaker-core && python -m pytest tests/unit/local/ -q: 176 passed
    • cd sagemaker-train && python -m pytest tests/unit/train/local/test_data.py tests/unit/train/local/test_entities.py -q: 43 passed (6 skipped)
  • Lint and format checks:
    • black --check -l 100 passed on all modified files
    • flake8 passed with 0 errors on all modified files

Merge Checklist

General

Tests

  • I have added tests that prove my fix is effective
  • Unit tests pass locally with my changes

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This branch is waiting to be deployed

1 waiting deployment
manual-approval — 9eb40fe7 Waiting Oct 11, 2026 by ankitdhandharia via wait-for-approval #1706
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(local) "File or directory not found", when input_data points to a single S3 file

1 participant