Training time on a p3.2xlarge instance suddenly increased 10 fold! Very slow. #1921

Activity

1. metrizable commented on Oct 6, 2020

   

   metrizable

   on Oct 6, 2020

   Contributor

   More actions

   @nectario

   Thank you for using Amazon SageMaker. Would you be able to provide the following to help us in troubleshooting your issue?

   • SageMaker Python SDK version
   • Custom Docker Image (Y/N)
   • Clear, step-by-step instructions to reproduce the bug
   • Code snippets that may be relevant for troubleshooting

   Look forward to hearing back from you.
2. metrizable

   added

   type: question

   on Oct 6, 2020
3. nectario commented on Oct 6, 2020

   

   nectario

   on Oct 6, 2020

   Author

   More actions

   Thank you. Here are answers to your questions:

   SageMaker Python SDK version: 2.13.0
   Custom Docker Image: Y

   This appears to be a problem with the TensorFlow Estimator. GPU utilization goes to zero. I am supplying you a notebook that demonstrates this by training my model with the estimator and without it.

   The below link is my codebase that recreates this issue.

   Please open the notebook: DeepTradingAI.ipynb using an AWS instance of ml.p3.2xlarge

   This notebook has two parts:

   1. Training as gpu_local using a TensorFlow Estimator which you will see the problem of every epoch taking about 40 seconds.
   2. Training the script directly with no TensorFlow estimator and you will see each epoch takes about 1 or 2 seconds.

   Note: You do not need to worry about the data as they are fetched from a database connection.

   Here is the download link:
   https://supportsagemaker.s3.us-east-2.amazonaws.com/sagemaker_gpu_training_issue_notebook.zip

   Thanks,

   Nektarios
4. nectario commented on Oct 7, 2020

   

   nectario

   on Oct 7, 2020

   Author

   More actions

   Even if I use my own custom Docker container, I get:

   2020-10-07 12:57:40.486419: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486514: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486609: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486685: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486779: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486868: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
   2020-10-07 12:57:40.486945: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
   
5. nectario commented on Oct 7, 2020

   

   nectario

   on Oct 7, 2020

   Author

   More actions

   I managed to get the above loaded, but still get:

   2020-10-07 16:17:42.690258: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

   This is my Dockerfile:

   FROM tensorflow/tensorflow:2.3.1-gpu
   
   RUN apt-get update && \
       apt-get upgrade -y && \
       apt-get install -y apt-utils
   
   RUN apt-get update && \
       apt-get upgrade -y && \
       apt-get install -y git
   
   RUN export LD_LIBRARY_PATH=/usr/local/nvidia/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
   
   RUN pip --no-cache-dir install --upgrade pip
   RUN pip install sagemaker-containers
   RUN pip --no-cache-dir install sagemaker-training
   
   RUN git clone https://dc9e2704c0415c0652e770ed7fd475d2b092bcb2:x-oauth-basic@github.com/nectario/deeptradingmodels.git /opt/ml/code/
   
   RUN pip --no-cache-dir install -r /opt/ml/code/requirements.txt
   
   WORKDIR /opt/ml/code/
   
   RUN cd $WORKDIR
   
   ENV SAGEMAKER_PROGRAM /opt/ml/code/Daily.py
   
6. nectario commented on Oct 7, 2020

   

   nectario

   on Oct 7, 2020

   Author

   More actions

   Do the boxes have CUDA installed properly?? This used to all work. All of a sudden things broke
7. bveeramani

   closed this as completedon May 20, 2021
8. aws

   locked and limited conversation to collaborators on May 20, 2021