Distributed inference GPU not being used and wrong memory value reported

[gngglobetech]

I am trying to do distributed Inference on two older computers, but the backend pc is only reporting Vram (11330MB) for one of the two gpu in that server. Nvidia-smi and NVCC work without errors. The command I use to start the rpc server is
“./rpc-server -p 50052 -H 0.0.0.0”
CUDA_VISIBLE_DEVICES=0,1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
WARNING: Host ('0.0.0.0') is != '127.0.0.1'
Never expose the RPC server to an open network!
This is an experimental feature and is not secure!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
create_backend: using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: Tesla M40, compute capability 5.2, VMM: yes
Device 1: Tesla M40, compute capability 5.2, VMM: yes
Starting RPC server on 0.0.0.0:50052, backend memory: 11330 MB

The command I use to start the main server is “./llama-cli -m /mnt/database/ds25/DeepSeek-V2.5-1210-Q6_K-00001-of-00005.gguf -p "Hello, my name is" -ngl 24 --rpc 192.168.1.46:50052” NVTOP shows all the gpus on the main server active but only one gpu being used on the backend pc. Is there any settings I can use to fix this?

Main server: Dell Poweredge C4130 - 4 X M40 (24GB)
Backend server: Dell Poweredge R720 - 2 X M40 (12GB)
Network: 10GBE

I also noticed the active gpu are only using their memory and the GPU% for them is stuck at zero.

[rgerganov]

You need to start a separate rpc-server for each GPU that you have:

$ CUDA_VISIBLE_DEVICES=0 bin/rpc-server -H 0.0.0.0 -p 50052
$ CUDA_VISIBLE_DEVICES=1 bin/rpc-server -H 0.0.0.0 -p 50053

Then on the main host:

$ llama-cli <other-params> --rpc <rpc-host>:50052 --rpc <rpc-host>:50053

[chenyihang1993]

+1

[gngglobetech]

Thank you I am now utilizing the gpu over RPC, one more question. How can i squeeze a bit more power from these cards. The M40-24GB is only using 12213 MIB, and the M40-12GB is using ~ 7106MIB?

print_info: max token length = 256
load_tensors: offloading 10 repeating layers to GPU
load_tensors: offloaded 10/62 layers to GPU
load_tensors: RPC[192.168.1.46:50052] model buffer size = 7106.67 MiB
load_tensors: RPC[192.168.1.46:50053] model buffer size = 6179.06 MiB
load_tensors: CUDA0 model buffer size = 14213.34 MiB
load_tensors: CUDA1 model buffer size = 14213.34 MiB
load_tensors: CUDA2 model buffer size = 14213.34 MiB
load_tensors: CUDA3 model buffer size = 14213.34 MiB
load_tensors: CPU_Mapped model buffer size = 41019.64 MiB
load_tensors: CPU_Mapped model buffer size = 41053.86 MiB
load_tensors: CPU_Mapped model buffer size = 41881.06 MiB
load_tensors: CPU_Mapped model buffer size = 40960.48 MiB
load_tensors: CPU_Mapped model buffer size = 41053.86 MiB
load_tensors: CPU_Mapped model buffer size = 40953.45 MiB
load_tensors: CPU_Mapped model buffer size = 40960.48 MiB
load_tensors: CPU_Mapped model buffer size = 27667.72 MiB
llama_init_from_model: n_seq_max = 1
llama_init_from_model: n_ctx = 4096
llama_init_from_model: n_ctx_per_seq = 4096
llama_init_from_model: n_batch = 2048
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 0

[gngglobetech]

I am not sure why the models I am loading are not using more of my GPU, Deepseek and llama3.3 hovers at 50%, what variable could I use to increase the GPU%

[slaren]

You can use the -ts parameter to configure this.

[gngglobetech]

-ts does not seems to be what is needed. The GPU% are not being utilized above 40 percent.