Eval bug: Slow prompt processing with Q4_K_S

[Cynerva]

Name and Version

$ ./llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU, compute capability 8.6, VMM: yes
version: 4929 (3d82dbc)
built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 7 6800H + RTX 3050 Ti Laptop

Models

gemma-2-9b-it Q4_K_S
gemma-3-12b-it Q4_K_S

Self-built GGUFs

Problem description & steps to reproduce

I'm seeing very slow prompt processing speeds on Gemma 2 and 3 models at Q4_K_S. Seems to be a regression with a recent commit.

	Prompt processing	Generation
Before	420.10 T/s	6.44 T/s
After	37.07 T/s	6.54 T/s

The same models at IQ4_XS still work fine. Have not tried other models yet.

Running llama-server with:

$ build/bin/llama-server -m /path/to/gemma-2-9b-it.Q4_K_S.gguf -c 8192 -ngl 14 -sp

First Bad Commit

Git bisect: 3d82dbc

Relevant log output

none

[ggerganov]

@Srihari-mcw Could you take a look at this?

[slaren]

The weight repacking prevents offloading the prompt computation to the GPU. You can disable this with -DGGML_CPU_AARCH64=OFF.

[Cynerva]

I can confirm. Building with -DGGML_CUDA=ON -DGGML_CPU_AARCH64=OFF gets me back to normal prompt processing speeds (420.78 T/s).

[Srihari-mcw]

Hi @slaren ,

So should we then try to add changes to CMakeLists.txt for automating this : -DGGML_CPU_AARCH64=OFF whenever -DGGML_CUDA=ON. Thanks

[ggerganov]

The weight repacking prevents offloading the prompt computation to the GPU. You can disable this with -DGGML_CPU_AARCH64=OFF.

I see. I have missed this interaction when doing partial offload with repacked weights.

So should we then try to add changes to CMakeLists.txt for automating this : -DGGML_CPU_AARCH64=OFF whenever -DGGML_CUDA=ON.

The backends should be compiled independently from one another, so this is not a good solution. I guess we need some sort of runtime check to make the decision, although I am not very sure how it should be implemented.

[ggerganov]

Note that we have the same performance effect with the Q4_0 type since it also supports AVX repacking. The Arm repackings are less of an issue, because there aren't many Arm CPU + GPU machines, apart from Macs, but these usually do full GPU offload anyway.

I expect that there will be a lot of reports similar to this one very soon, so maybe we should revert the #12332 change temporarily, until we figure out how to handle this?

[0cc4m]

If a GPU is available it should not repack if that excludes GPU acceleration, since that is usually a lot faster.

[slaren]

If a GPU is available it should not repack if that excludes GPU acceleration, since that is usually a lot faster.

An easy way to implement this would be to modify make_cpu_buft_list to skip adding the extra buffer types if there is a GPU device.

[ggerganov]

If a GPU is available it should not repack if that excludes GPU acceleration, since that is usually a lot faster.

An easy way to implement this would be to modify make_cpu_buft_list to skip adding the extra buffer types if there is a GPU device.

Ok, will push a PR now.