Name of training data file  #552

Activity

1. ChoiByungWook commented on Dec 15, 2018

   

   ChoiByungWook

   on Dec 15, 2018

   Last edited by ChoiByungWook

   Contributor

   More actions

   Hello @NRauschmayr,

   Thank you for bringing this to our attention.

   If I understand this correctly, when calling fit(), the name of the dataset doesn't stay as originally attended and instead for the following examples, this happened:

   mxnet_estimator.fit({'train': 's3://rauscn-data/data/input_data.np') = /opt/ml/input/data/train/y
   mxnet_estimator.fit({'train': 's3://rauscn-data/data/input_data.npy') = /opt/ml/input/data/train/npy

   The dataset that was used to train was renamed to y and npy when referenced respectively for input_data.np and input_data.npy.

   Did this happen when training on SageMaker or through local mode?

   I was looking through our Python SDK and I don't think we modify the URI provided by the user.

   1. https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/job.py#L95
   2. https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/job.py#L117

   Would you be able to provide more information on how you are instantiating your estimator?

   Please let me know if there is anything I can clarify.

   Thanks!
2. ChoiByungWook

   added

   type: question

   on Dec 15, 2018
3. NRauschmayr commented on Jan 3, 2019

   

   NRauschmayr

   on Jan 3, 2019

   Author

   More actions

   Hello @ChoiByungWook,

   yes exactly that is what happened
   mxnet_estimator.fit({'train': 's3://rauscn-data/data/input_data.np') lead to /opt/ml/input/data/train/y

   I am creating the Estimator, like this:

   mxnet_estimator = MXNet('train.py',
                           role=role,
                           train_instance_type='local',#'ml.m5.xlarge',
                           train_instance_count=1,
                           output_path='s3://sagemaker-rauscn',
                           code_location='s3://sagemaker-rauscn',
                           framework_version='1.3.0', py_version='py2',
                           hyperparameters={'batch_size': 16,
                            'epochs': 10,
                            'learning_rate': 0.0001,
                            'wd': 0.0})
   
   mxnet_estimator.fit({'train': 's3://sagemaker-rauscn/data/input_data.npy'})
   
4. nadiaya commented on Jan 4, 2019

   

   nadiaya

   on Jan 4, 2019

   Contributor

   More actions

   If you go to the corresponding training job you ran in the sagemaker console what URI do you have specified in Input data configuration section?
   Also it would be really useful to know what values do you have for Channel name, S3 data type, S3 data distribution type in the same exact section?

   Thanks!
5. NRauschmayr commented on Jan 4, 2019

   

   NRauschmayr

   on Jan 4, 2019

   Author

   More actions

   Hello @nadiaya,

   When calling mxnet_estimator.fit({'train': 's3://sagemaker-rauscn/data/input_data.np'})
   URI : s3://sagemaker-rauscn/data/input_data.np
   Channel name: train
   S3 data type: S3Prefix
   S3 data distirbution type: FullyReplicated
6. nadiaya commented on Jan 4, 2019

   

   nadiaya

   on Jan 4, 2019

   Contributor

   More actions

   But inside of the container for this training job you got: /opt/ml/input/data/train/y instead of the expected file?

   Let me escalate this issue to the appropriate service team.
7. NRauschmayr commented on Jan 5, 2019

   

   NRauschmayr

   on Jan 5, 2019

   Author

   More actions

   Exactly. In my training script the following crashes: train = np.load("/opt/ml/input/data/train/input_data.npy")
   However this works: train = np.load("/opt/ml/input/data/train/y")

   From a user perspective, it is probably best to just get an error message if there is such a typo in the filename.
8. ChoiByungWook

   added a commit that references this issue on Dec 8, 2020

   fix: run UpdateTrainingJob tests only during allowed secondary status (…

   ...

   500bd36
9. bveeramani

   closed this as completedon May 19, 2021
10. aws

    locked and limited conversation to collaborators on May 19, 2021