Behavior of on_epoch and sync_dist in self.log when logging raw tensors under DDP

[B-codes-core]

When using DDP/DDP Spawn strategies, I’d like to understand how sync_dist works for the following logging call:

self.log("metric", tensor, on_epoch=True, sync_dist=True)

Is the synchronization across GPUs performed at every step or does synchronization happen only once at the end of the epoch assuming on_step is False? Understanding this behavior would help me decide whether to use this, since per step synchronization could have significant overhead. I read the relevant code and it seems to me like the synchronization happens only at end of epoch, but I would appreciate confirmation from experts

Thank You!

[Samoilov2004]

Yes, if on_step=False and you are logging a raw tensor, synchronization is deferred until the epoch result is computed. Each rank accumulates its values locally, and at the end of the epoch Lightning synchronizes the accumulated value and batch count to compute the global result. Therefore, no collective operation is performed on every batch in this case.

The default value of on_step depends on the hook. It is False in validation_step and test_step, but True in training_step. For training, it should therefore be set explicitly:

self.log(
    "metric",
    tensor,
    on_step=False,
    on_epoch=True,
    sync_dist=True,
)

If on_step=True, the step value is synchronized on every update. This behavior applies to raw tensors; torchmetrics.Metric objects manage distributed synchronization through their own metric state.