Repository navigation
Sagemaker training smdebug/core/state_store.py FileNotFoundError #1791
Description
Hello,
I am trying to train a model using MXnet estimator. As soon as training starts I see following error when Sagemaker tries to upload checkpoints:
File "/opt/ml/code/translation.py", line 224, in __call__
return super(NMTModel, self).__call__(src_seq, tgt_seq, src_valid_length, tgt_valid_length)
File "/usr/local/lib/python3.6/site-packages/mxnet/gluon/block.py", line 756, in __call__
hook(self, args)
File "/usr/local/lib/python3.6/site-packages/smdebug/mxnet/hook.py", line 143, in forward_pre_hook
self._increment_step()
File "/usr/local/lib/python3.6/site-packages/smdebug/core/hook.py", line 511, in _increment_step
self._write_state()
File "/usr/local/lib/python3.6/site-packages/smdebug/core/hook.py", line 523, in _write_state
if self.state_store.is_checkpoint_updated():
File "/usr/local/lib/python3.6/site-packages/smdebug/core/state_store.py", line 112, in is_checkpoint_updated
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.6/site-packages/smdebug/core/state_store.py", line 112, in <listcomp>
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.6/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploading'
[2020-07-31 05:38:53.405 ip-10-2-236-41.ec2.internal:144 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.
2020-07-31 05:38:54,756 sagemaker-training-toolkit ERROR ExecuteUserScriptError:
Few details about the job:
- I am using Mxnet estimator with distributed setting
- Using 4 p3.16xlarge instances
Activity
laurallu commentedon Oct 29, 2020
Hello,
I encountered a similar issue using Tensorflow estimator. I used pre-built container image with framework version = 2.2 and py_version = py37. It has trained 3 epochs and threw the error in the 4th epoch.
guoren66 commentedon Nov 21, 2020
Hi
same issue with laurallu's.
I1121 13:42:35.783539 140148447449344 basic_session_run_hooks.py:260] loss = 0.65470797 (0.405 sec)
Traceback (most recent call last):
File "0-deepfm-local-tf2.py", line 380, in
tf.compat.v1.app.run()
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "0-deepfm-local-tf2.py", line 356, in main
tf.estimator.train_and_evaluate(DeepFM, train_spec, eval_spec)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 472, in train_and_evaluate
return executor.run()
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 350, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1186, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1219, in _train_model_default
saving_listeners)
File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1522, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 784, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1289, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1390, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1375, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1456, in run
run_metadata=run_metadata))
File "/usr/local/lib/python3.7/site-packages/smdebug/tensorflow/session.py", line 404, in after_run
self._increment_step()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 517, in _increment_step
self._write_state()
File "/usr/local/lib/python3.7/site-packages/smdebug/core/hook.py", line 529, in _write_state
if self.state_store.is_checkpoint_updated():
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 112, in is_checkpoint_updated
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/site-packages/smdebug/core/state_store.py", line 112, in
cp_file_sizes = [os.path.getsize(file) for file in checkpoint_files]
File "/usr/local/lib/python3.7/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploaded'