Multiple safetensor files, am I missing something?

[mousepixels]

The new llama-2 models have multiple safetensor files.

I get this error when running anything
!! Multiple files matching ../llama/transformed_70b/*.safetensors

I am assuming I am missing something, so i didn't raise this as an issue.

Thanks for any help!

[turboderp]

ExLlama expects a single .safetensors file and doesn't currently support sharding. I'm not aware of anyone releasing sharded GPTQ models, but if you have a link to where you found those files I could probably take a look.

[benburkhart1]

Hello @turboderp, Not @mousepixels, but I'm experiencing the same confusion here.

Downloading llama weights from meta results in multiple pth files, converting to hf results in multiple safetensors files.

Even the HF llama 2 has multiple safetensors files
https://huggingface.co/meta-llama/Llama-2-7b-hf/tree/main

What steps did you use for preparing the data from meta prior to using it, is there some guide or keywords I should be following?

[da-x]

Here's one with safetensors shards:
codellama/CodeLlama-7b-hf at main

[turboderp]

I'm not aware of any particular guides, but generally you want to look for GPTQ conversions. ExLlama only works on quantized weights, and those links seem to be to the original FP16 weights.

Here is a Llama2-7b conversion, and there are many other GPTQ models on HF that should work fine.