Minimal GGUF file

[mrh-chain]

While developing an application that uses llama.cpp, I ran into the issue of having to test model loading. Naturally, this requires an actual model to load, and for the time being I'm using TheBlokes TinyLlama Q2 GGUF model. That model was the smallest I could find, at around 482MB.

I seem to remember seeing a minimal GGUF model used during the testing of llama.cpp, but I can't for the life of me figure out if I'm just imagining it.

Does anyone know of barebones GGUF model that will read as a proper GGUF, but be small enough to use during testing?

[ggerganov]

You can generate small GGUF models like this:

make -j gguf

# write
./gguf dummy.gguf w

# read
./gguf dummy.gguf r

The code is in the examples/gguf folder

[KennanHunter]

Could you detail how to compile and run this example gguf script with the current repository's CMake?

[danbev]

The following commands should work:

$ cmake --build build --target llama-gguf
$ ./build/bin/llama-gguf --help
usage: ./build/bin/llama-gguf data.gguf r|w [n]
r: read data.gguf file
w: write data.gguf file
n: no check of tensor data

[mrh-chain]

That was quick, thank you!

[ericcurtin]

I use this one:

llama-run smollm:135m

it's < 100M. llama-run can dowload it from ollama repository.