Whisper defaults to CPU instead of utilizing Nvidia GPU on Windows 11

[selfAndrewKB]

A warning upon first running the whisper model clued me in to it not using hardware acceleration:

UserWarning: FP16 is not supported on CPU; using FP32 instead

All I had to do in order to enable CUDA support was first uninstall Torch:
python -m pip3 uninstall torch

And reinstall with this command:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Confirm that CUDA is available in Python by running:
import torch
torch.cuda.is_available()

monkeyplug/whisper should now correctly use your GPU to significantly speed up operations. A youtube video with a runtime of 10:42 took 13 minutes and 42 seconds to process on my CPU with the medium.en model. After successfully enabling CUDA support, that same video took 3 minutes and 13 seconds to process on an RTX 3070. With noticeable accuracy over the default base.en.

I caught several warning messages that were raised during a job (might be related to generating timestamps?), but they don't seem to affect the operation at all:

C:\Users\username\AppData\Local\Programs\Python\Python312\Lib\site-packages\whisper\timing.py:42: UserWarning: Failed to launch Triton kernels, likely due to missing CUDA toolkit; falling back to a slower median kernel implementation...
warnings.warn(

C:\Users\username\AppData\Local\Programs\Python\Python312\Lib\site-packages\whisper\timing.py:146: UserWarning: Failed to launch Triton kernels, likely due to missing CUDA toolkit; falling back to a slower DTW implementation...
warnings.warn(

C:\Users\username\AppData\Local\Programs\Python\Python312\Lib\site-packages\whisper\timing.py:42: UserWarning: Failed to launch Triton kernels, likely due to missing CUDA toolkit; falling back to a slower median kernel implementation...`
warnings.warn(

C:\Users\username\AppData\Local\Programs\Python\Python312\Lib\site-packages\whisper\timing.py:146: UserWarning: Failed to launch Triton kernels, likely due to missing CUDA toolkit; falling back to a slower DTW implementation...
warnings.warn(

Noticed that #3 might be in the works, which might help, but I thought it could be wise/helpful to share my findings regardless in the meantime.

PS: Whisper really is another tier of accuracy and is much appreciated.

[mmguero]

Interesting, on my Linux machine it was using the GPU right out of the gate just with pip install openai-whisper without any other steps on my end (double-checked with nvidia-smi during processing).

[selfAndrewKB]

Oh and If it helps, this is a fresh install of Windows 11 and I actually used that very same command to install whisper following Python 3.12. Strange indeed.

[bradyj04]

Are you still having this issue any, I tried your steps and mine persisted.

[mmguero]

Right now I don't have access to a Windows machine with a GPU, so I don't have any way to confirm or look into this.

[selfAndrewKB]

Are you still having this issue any, I tried your steps and mine persisted.

Sorry to hear. It's been working just fine ever since. Could you provide more info about your setup? Operating system, whether you tried torch.cuda.is_available(), what it returns, any error messages you might've seen, etc.

[bradyj04]

Windows 11, getting the exact same error messages as you get in your original one. I'm currently just using a separate whisper program instead so no big deal, and yes torch returns true.

[therealmichaelberna]

@selfAndrewKB @bradyj04

For windows, I had to install an Nvidia triton windows compiler build from here : https://huggingface.co/madbuda/triton-windows-builds
Command:
pip install https://huggingface.co/madbuda/triton-windows-builds/resolve/main/triton-3.0.0-cp312-cp312-win_amd64.whl

If you have CUDA 12.6 or higher, this bugfix needs to be applied also.

https://github.com/triton-lang/triton/pull/4588/files (see the changed files tabs and note the added and removed lines)

For me, the file I had to edit was located in
C:\Users\User.conda\envs\monkeyplug_312\Lib\site-packages\triton\backends\nvidia\compiler.py

After this and doing the pytorch CUDA re-install, it worked for Windows.

Thanks for creating this and I hope this info can help someone.