Repository navigation
Local mode fails to read serialized bytes from S3 alternative storage #5262
Description
Describe the bug
I encountered ServiceError: Failed to read serialized bytes from S3 alternative storage (MinIO).
To reproduce
- Run MinIO image with
docker, expose the server to port9000, the endpoint address ishttp://localhost:9000. - Utilize
LocalPipelineSessionwiths3_endpoint_url = 'http://localhost:9000'. - Start SageMaker local mode.
Expected behavior
Can read objects from MinIO without any problems.
Screenshots or logs
INFO [sagemaker.remote_function] Uploading serialized function code to s3://sagemaker-data/pipelines/step-name/function
INFO [sagemaker.remote_function] Uploading serialized function arguments to s3://sagemaker-data/pipelines/step-name/arguments
...
sagemaker.remote_function.errors.ServiceError: Failed to read serialized bytes from s3://sagemaker-data/pipelines/pipeline-name/step-name-xxx/xxx/function/metadata.json: ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden')System information
A description of your system. Please provide:
- SageMaker Python SDK version: 2.250.0
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): Not used
- Framework version: Not used
- Python version: 3.13.0
- CPU or GPU: CPU
- Custom Docker image (Y/N):
Additional context
- Successfully upload the serialized resources (function code, function arguments) to MinIO.
- Double-check on MinIO console to make sure the objects exist.
Reacted by HoangNguyen689
Activity
cuongvd23 commentedon Aug 17, 2025
The step invoke-function is currently using
boto3.session.Sessionwith the defaults3_endpoint_urlpointing to AWS S3.sagemaker-python-sdk/src/sagemaker/remote_function/invoke_function.py
Line 63 in 73bdd08
Setting
AWS_ENDPOINT_URL_S3environment variable would override the default value.However,
sagemaker.local.image.S3_ENDPOINT_URL_ENV_NAMEis set to"S3_ENDPOINT_URL"instead.sagemaker-python-sdk/src/sagemaker/local/image.py
Line 53 in 73bdd08
nmadan commentedon Sep 26, 2025
Thanks for catching and fixing this issue. The fix has been merged in #5264