Repository navigation
Batch transform fails for large files ( IP Insights Model ) #1951
Description
Describe the bug
I am using AWS inbuilt IP Insights model to run batch transform. This batch transform is failing with the below error -
2020-10-14T04:53:51.549:[sagemaker logs]: bucket-name/v1_2020-10-13.csv: Unable to get response from algorithm
To reproduce
Run the batch transform on this model and you will see the error.
Expected behavior
Batch transform should work without any issues.
Screenshots or logs
2020-10-14T04:53:51.549:[sagemaker logs]: bucket-name/v1_2020-10-13.csv: Unable to get response from algorithm
System information
A description of your system. Please provide:
- SageMaker Python SDK version:
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): IP Insights
- Framework version:
- Python version:
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
The CSV file I am using is 600MB and instance type I'm using for batch transform is ml.m5.4xlarge
Activity
ksachdeva11 commentedon Oct 14, 2020
@andremoeller @laurenyu Would you be able to help, please?
laurenyu commentedon Oct 15, 2020
cc @ChuyangDeng
chuyang-deng commentedon Oct 19, 2020
@andremoeller Could you help to confirm that 600MB is a safe size for batch transform?
andremoeller commentedon Oct 20, 2020
Hi @ksachdeva11
That message
Unable to get response from algorithmindicates that the IPInsights server couldn't respond to individual inference requests, but it's not quite definitive why.Would you be able to run a job with the following values? These are best guesses based on the instance type, the fact that your data is CSV -- together, these mean "split up the input files into 1MB requests by newline characters, and send 16 concurrent requests to the IPInsights server"
MaxConcurrentTransforms: 16
MaxPayloadInMB: 1
SplitType: "Line"
BatchStrategy: "MultiRecord"
ContentType: "text/csv"
Also, would you be able to share which values for these four fields you used in your request?
I've also notified the IPInsights owners as well, in case there's another root cause.
Thanks!