How to set "accum-iter" in MAE to keep effective batch size = 4096 if limited GPU / memory?

[foolhard]

Hello,

In official MAE repo, accum_iter is used to maintain the effective batch size = 4096.

How could I maintain the effective batch size as 4096 in config, as I don't have enough resource as your setting (8xA100 80G)?

And, will the accuracy keep the same if effective batch size = 512 (256 batch size per gpu X 2 gpu )?

Thanks.

[YuanLiuuuuuu]

We did not use accum_iter to pre-train MAE, but theoretically, it should work the same as not using it. You can have a try and report the results here, if convenient. Thanks !

[YuanLiuuuuuu]

You can refer the this doc for the usage of accum_iter. Thanks!

[foolhard]

Understood. Thanks for your quick response.