Kernel dying when trying to train model on aws sagemaker #1922

Activity

1. icywang86rui commented on Sep 24, 2020

   

   icywang86rui

   on Sep 24, 2020

   Contributor

   More actions

   @shubhamsharma1609
   Sorry that you run into this issue.
   Could you show us your code which used to start the training job?
   Did you use the python sdk? If so which version?

   ml.p2.xlarge instance on conda_amazonei_tensorflow_p36 notebook

   Did you mean the training job was launched from the SageMaker hosted notebook instance.
   Did the kernel die when you run a specific line of code?
   Have you tried to restart the kernel?
2. shubham-scisar commented on Sep 24, 2020

   

   shubham-scisar

   on Sep 24, 2020

   Last edited by shubham-scisar

   Author

   More actions

   I did not use sagemaker pythonsdk. I installed keras-unet and with higher batch size i.e. 30 ,I don't get any error but the kernel dies. I am utilising the compute power of AWS sagemaker and nothing else from the sdk.

   history = model.fit_generator(
       train_gen,
       steps_per_epoch=50,
       epochs=20,
       
       validation_data=(x_val, y_val),
       callbacks=[callback_checkpoint]
   )
   
3. shubham-scisar commented on Sep 24, 2020

   

   shubham-scisar

   on Sep 24, 2020

   Author

   More actions

   I have somehow solved the problem by reducing the batch size, now my notebook is running for each epoch. I just want to ask that the Jupyter session is of 12 hours but my training job will take more than that, Can you please guide how can I increase the Jupyter session duration, so that it does not stop training. Also, can I close my browser but do not stop the instance and at a later time connect to the jupyter notebook and check the training is happening continuously ?
4. metrizable

   added

   type: question

   on Oct 6, 2020
5. bveeramani

   closed this as completedon May 20, 2021
6. aws

   locked and limited conversation to collaborators on May 20, 2021