CUDA out of memory

[Backlonhw2468]

I encountered an Error saying "CUDA out of memory. Tried to allocate 2.93 GiB (GPU 0; 23.80 GiB total capacity; 12.04 GiB already allocated; 2.82 GiB free; 19.69 GiB reserved in total by Pytorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF".

The error above occurs when training a GAT model. Is there any way to change the "max_split_size_mb " setting in torch?

The torch version is 0.13.0. The CUDA version is 11.7.

[cregouby]

Hello @Backlonhw2468

Cuda OoM is a pretty common error due to trying to feed too much in your GPU memory.
It is usually used to find the right value for your batch size ( through try-and-error).
The only advice you could get, not having any clue of how to reproduce your specific issue, is try to decrease your batch size and or try to decrease your network size.