Running out of memory during training

[Deathawaits4]

Hei, i noticed that for some reason the training runs oom at some point. It trained fine for 553 steps and then got an OOM. I did not run anything else apart from the training.

I think there is a memory leak happening somewhere

[bghira]

i have training going for 43,000 steps without an issue, you will have to do a bit more to help pin down the problem since it is not widespread. if you're using gradient checkpointing interval logic, this is probably the reason for the OOM. or the SOAP optimiser. but there's a lot that can go wrong.