How to reclaim the memory by model loaded by a different application?

[eSPiYa]

I'm developing an application using LLamaSharp library, it is a .NET binding for llama.cpp. Usually, the model will clear from the memory after disposing LLamaWeights and LLamaContext, but one time I stopped the application during debugging before disposing the instances of these class, the VRAM is still full. I have to restart my machine just to reclaim the memory. I'm sure this will happen a few more times during development. Also, even though I mitigated this issue when it crashed by disposing the instances when it encounter an error, there are chances that this may still happen and can't be caught.

Not sure if this will matter, but I'm on CachyOS and using llama.cpp-vulkan. I got an iGPU of RX 780M and RX 9070 as an eGPU.

[bandoti]

one time I stopped the application during debugging before disposing the instances of these class, the VRAM is still full

One strategy might be for general testing use the CPU backend to ensure nothing goes to VRAM. This could accelerate the workflow to iron out any application-related issues. Then once those are figured out, moving to the GPU backend.

Here are a couple other possibilities (from Claude—your mileage may vary 😉):

1. Reset GPU Driver (Alternative to Full Restart)
   Instead of restarting your entire machine, you can try resetting just the GPU driver:

# For AMD GPUs on Linux
sudo modprobe -r amdgpu
sudo modprobe amdgpu

# Or use the GPU reset utility if available
sudo gpu-reset

2. Kill GPU Processes
   Check what's holding the GPU memory:

# Check GPU usage
rocm-smi  # for AMD GPUs
# or
nvidia-smi  # if you have NVIDIA tools installed for comparison

# Find and kill lingering processes
ps aux | grep llama
sudo kill -9 <process_id>

[bandoti]

It looks like rocm-smi is replaced by amd-smi