inital bnb

[ewof]

u need to install bnb from source pip package doesn't work for me at least
https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md

[ewof]

8bit works fine
4bit i had working but cant get it to work again. there's a bug in transformers trainer.py line 499 needs to be

if self.place_model_on_device and not getattr(model, "is_loaded_in_8bit", False) and not getattr(model, "is_loaded_in_4bit", False):

instead of

if self.place_model_on_device and not getattr(model, "is_loaded_in_8bit", False):

but even with this i get weird matmul errors (which i didn't get for one run and 4bit worked fine but i can't recreate)
i made an issue on bnb repo