RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

[kirilllzaitsev]

I have a custom model checkpointing class:

class ModelSerializationCallback(lightning.pytorch.callbacks.ModelCheckpoint):

  def __init__(
      self,
      task,
      experiment_dir: str,
      every_n_train_steps: Optional[int],
      *args,
      **kwargs,
  ):
    self._task = task
    self._experiment_dir = experiment_dir
    self._every_n_train_steps = every_n_train_steps
    super().__init__(*args, **kwargs)

  def _save_checkpoint(self, trainer, filepath):
    if trainer.global_rank == 0:
      pl_module = trainer.lightning_module
      model_name = "model.pth"
      model_path = os.path.join(self._experiment_dir, model_name)
      os.path.makedirs(os.path.dirname(model_path), exist_ok=True)
      # pylint: disable=protected-access
      self._task.serialize_model(pl_module._model, model_path)
      # pylint: enable=protected-access

    self._last_global_step_saved = trainer.global_step
    self._last_checkpoint_saved = filepath

    # notify loggers
    if trainer.is_global_zero:
      for logger in trainer.loggers:
        logger.after_save_checkpoint(proxy(self))

In the end of a DDP training, sometimes it leads to the error RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory. This is an internal miniz error. If you are seeing this error, there is a high likelihood that your checkpoint file is corrupted. This can happen if the checkpoint was not saved properly, was transferred incorrectly, or the file was modified after saving.

Could you help to understand this problem?

[stef41]

This error typically means the checkpoint file is corrupted or was only partially written. Here are the most common causes and fixes:

1. Checkpoint saved only on rank 0, but all ranks try to load

Your custom callback has if trainer.global_rank == 0: — this means only rank 0 saves the checkpoint. But if you're using DDP and all ranks try to load from the same path, there's a race condition where non-zero ranks may read a partially-written file.

Also note: os.path.makedirs is the correct call, not os.path.dirname + makedirs. Your code has:

os.path.makedirs(os.path.dirname(model_path), exist_ok=True)

This should be:

os.makedirs(os.path.dirname(model_path), exist_ok=True)

(os.path.makedirs doesn't exist — this would raise AttributeError, which might be silently caught somewhere.)

2. The checkpoint file was truncated

Common causes:

• Disk full during save — check df -h on your training nodes
• Process killed mid-save (OOM, preemption, timeout)
• NFS/networked filesystem — writes may not be fully flushed before another process reads

Fix: Add a verification step after saving:

def _save_checkpoint(self, trainer, filepath):
    if trainer.global_rank == 0:
        # ... save ...
        # Verify the file is valid
        torch.load(model_path, weights_only=True, map_location="cpu")

3. File path collision

If model_name = "model.pth" is constant, you're overwriting the same file every time. If a save starts while the previous save's file is still being read for validation, you get a corrupt read.

Fix: Use unique filenames (e.g., include step/epoch) or save to a temp file first, then atomically rename:

import tempfile, shutil

tmp_path = model_path + ".tmp"
torch.save(state_dict, tmp_path)
shutil.move(tmp_path, model_path)  # atomic on same filesystem

4. Verify your checkpoint file

You can check if the file is a valid zip archive:

import zipfile
print(zipfile.is_zipfile("model.pth"))  # Should be True for PyTorch checkpoints