[vad] Previous response is returned by server if no voice activity is detected in the sample

[tannisroot]

Hi, I'm using whisper.cpp in conjunction with whisper-api. I've tried the new VAD feature introduced into the server component of whisper.cpp by @danbev, and while it works well with samples that contain voice, there is an issue with audio that contains no voice.
It seems that if no voice is detected in the sample, the server just returns transcript of the last voice sample that was successfully transcribed
For example:

whisper-cpp-1  | whisper_vad_segments_from_probs: detecting speech timestamps using 145 probabilities
whisper-cpp-1  | whisper_vad_segments_from_probs: Final speech segments after filtering: 0
wyoming-api-1  | INFO:httpx:HTTP Request: POST http://whispercpp:8910/inference?temperature=0.0&temperature_inc=0.2&response_format=json "HTTP/1.1 200 OK"
wyoming-api-1  | INFO:wyoming_whisper_api_client.handler: set a timer for 30 minutes
wyoming-api-1  | 

The behavior I would expect instead is for the transcript to be empty, since no voice was detected.

[danbev]

Thanks for the report. Could you tell us which example you are using or a link to the implementation. For example, we are working on updating the stream example and it would currently do something like this as it adds part of the previous audio to mitigate would boundary issues, and perhaps something similar is happening in this case.

While looking at updating the stream example I noticed that for streaming audio the "built-in" VAD support might not be the best fit. It works well for whole audio samples but for streaming it might make sense to expose a whisper_vad function that returns a possibly VAD filtered audio sample, which could then be passed to whisper_full (without VAD enabled as that has already been done). I'm not completely sure about this solution but I'm planning on revisiting this next week. I've modified the example somewhat to work better with the new VAD support and I don't think exposing whisper_vad will be needed.

[Sebastinus-Schulze]

Hello everyone,

I hope my observations with Whipser.cpp and the VAD module fit thematically to the initial posting.

In the attached example, there is a 10 minute pause during a lecture recording.
Whisper.cpp in conjunction with VAD generates a continuous subtitle passage in this case, which refers to the speech section that follows the break.

I have tried to illustrate this with a few screenshots.
According to my observations, however, the problem only occurs with pauses or longer quiet phases in the middle of the subtitling-process, not if the pause or silence is present at the beginning or end of the audio track.

I tested this recording with different values for -vad-threshold N: 0.1 | 0.5 | 0.9 and it made no difference to the behavior.

[tannisroot]

Thanks for the report. Could you tell us which example you are using or a link to the implementation. For example, we are working on updating the stream example and it would currently do something like this as it adds part of the previous audio to mitigate would boundary issues, and perhaps something similar is happening in this case.

While looking at updating the stream example I noticed that for streaming audio the "built-in" VAD support might not be the best fit. It works well for whole audio samples but for streaming it might make sense to expose a whisper_vad function that returns a possibly VAD filtered audio sample, which could then be passed to whisper_full (without VAD enabled as that has already been done). I'm not completely sure about this solution but I'm planning on revisiting this next week. I've modified the example somewhat to work better with the new VAD support and I don't think exposing whisper_vad will be needed.

I'm launching whisper-cpp with the following parameters
build/bin/whisper-server -l en -bs 8 --best-of 8 -m /models/ggml-large-v3.bin --host 0.0.0.0 --port 8910 --suppress-nst --vad --vad-model /models/ggml-silero-v5.1.2.bin --prompt "redacted very long prompt" (in a container)
Audio samples are fed to it by https://github.com/ser/wyoming-whisper-api-client