Realtime STT with Bidirectional Streaming (gRPC)

[EngincanVaran]

Hi all!

I want to design a bidirectional streaming gRPC with OpenAI-Whisper model (for Turkish unfortunately there are not many models). I am struggling with streaming. I have tried faster-whisper also but still they seem not fast enough. I have used a Nvidia-V100 for gpu.

All in all, I want to ask how I can make my streaming faster and more efficient? Anybody heard of the Nvidia-Triton-Server for GPU virtualization? Is there a video about it, maybe I am missing something? Can I use it to increase my transcription speed? I can use up to 4 V100 GPU's if I need to.