WhisperFusion builds upon the capabilities of WhisperLive and WhisperSpeech to provide a seamless conversations with an AI.
  • Python 95%
  • Shell 4.5%
  • Dockerfile 0.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Marcus Edel 807d35dd13
Merge pull request #59 from makaveli10/fix_llm_service
Update chatml prompt formatting.
2024-07-31 09:28:41 -04:00
assets add mistral LLM 2023-12-18 17:54:12 +00:00
docker Update chatml prompt formatting 2024-07-31 04:53:05 -04:00
examples/chatbot/html Backport @DamianB-BitFlipper audio buffer fix on mac os. 2024-02-07 23:43:13 -05:00
whisper_live Pin SileroVAD to v4.0 2024-07-30 01:32:19 -04:00
docker-compose.yml Make Phi-3 the default model 2024-06-18 12:15:41 -04:00
llm_service.py Update chatml prompt formatting 2024-07-31 04:53:05 -04:00
main.py Update llm service for phi-3 2024-06-13 03:07:44 -04:00
README.md Update README 2024-06-13 04:39:55 -04:00
README.qmd Add sample video to the README. 2024-01-25 14:19:35 -05:00
requirements.txt Automate Docker building 2024-01-23 19:47:38 +01:00
tts_service.py add progress bar for warmup 2024-03-05 11:57:53 -05:00

WhisperFusion

WhisperFusion

Seamless conversations with AI (with ultra-low latency)

Welcome to WhisperFusion. WhisperFusion builds upon the capabilities of the WhisperLive and WhisperSpeech by integrating Mistral, a Large Language Model (LLM), on top of the real-time speech-to-text pipeline. Both LLM and Whisper are optimized to run efficiently as TensorRT engines, maximizing performance and real-time processing capabilities. While WhiperSpeech is optimized with torch.compile.

Features

  • Real-Time Speech-to-Text: Utilizes OpenAI WhisperLive to convert spoken language into text in real-time.

  • Large Language Model Integration: Adds Mistral, a Large Language Model, to enhance the understanding and context of the transcribed text.

  • TensorRT Optimization: Both LLM and Whisper are optimized to run as TensorRT engines, ensuring high-performance and low-latency processing.

  • torch.compile: WhisperSpeech uses torch.compile to speed up inference which makes PyTorch code run faster by JIT-compiling PyTorch code into optimized kernels.

Hardware Requirements

  • A GPU with at least 24GB of RAM
  • For optimal latency, the GPU should have a similar FP16 (half) TFLOPS as the RTX 4090. Here are the hardware specifications for the RTX 4090.

The demo was run on a single RTX 4090 GPU. WhisperFusion uses the Nvidia TensorRT-LLM library for CUDA optimized versions of popular LLM models. TensorRT-LLM supports multiple GPUs, so it should be possible to run WhisperFusion for even better performance on multiple GPUs.

Getting Started

We provide a Docker Compose setup to streamline the deployment of the pre-built TensorRT-LLM docker container. This setup includes both Whisper and Phi converted to TensorRT engines, and the WhisperSpeech model is pre-downloaded to quickly start interacting with WhisperFusion. Additionally, we include a simple web server for the Web GUI.

  • Build and Run with docker compose
mkdir docker/scratch-space
cp docker/scripts/build-* docker/scripts/run-whisperfusion.sh docker/scratch-space/

docker compose build
export MODEL=Phi-3-mini-4k-instruct    #Phi-3-mini-128k-instruct or phi-2, By default WhisperFusion uses phi-2
docker compose up
  • Start Web GUI on http://localhost:8000

NOTE

Contact Us

For questions or issues, please open an issue. Contact us at: marcus.edel@collabora.com, jpc@collabora.com, vineet.suryan@collabora.com