Pipecat — Real-Time Voice & Multimodal AI Agents
Pipecat is an open-source Python framework designed to help developers build real-time, voice-first and multimodal conversational agents. The project focuses on composing modular processors and services (speech-to-text, text-to-speech, LLMs, vision, transports, etc.) into pipelines that run with low latency for interactive use cases such as voice assistants, AI companions, meeting assistants, and interactive storytelling.
Key concepts and features
- Pluggable services: Pipecat provides adapters to many STT, TTS, and LLM providers so you can swap or mix services depending on latency, cost, or capability requirements.
- Composable pipelines: Behavior is defined as pipelines of small, testable components (processors), enabling complex dialog flows built from reusable parts.
- Real-time transports: Supports common real-time transports (WebSocket, WebRTC and others) so agents can stream audio/video and messages to clients with low latency.
- Voice-first design: Built-in handling for streaming audio, VAD, incremental transcription and streaming TTS for natural conversational experiences.
- Multimodal support: Integrates vision/image and video capabilities so agents can handle images and video-inference steps in addition to speech and text.
Ecosystem & tooling
Pipecat includes or links to multiple companion projects and SDKs:
- Official client SDKs: JavaScript, React, React Native, Swift (iOS), Kotlin (Android), C++ and even an ESP32 integration for embedded use.
- Utility and developer tools: Pipecat CLI to scaffold and deploy projects quickly, Whisker for real-time debugging, Tail for terminal dashboards, and example apps demonstrating common use cases.
- Example apps: A repository of example applications (simple chatbot, storytelling, translation, etc.) to help you get started quickly.
Supported services (examples)
Pipecat documents many built-in service adapters for common providers in categories such as:
- Speech-to-Text (AssemblyAI, Deepgram, OpenAI Whisper, Google, Azure, etc.)
- Text-to-Speech (ElevenLabs, Google, OpenAI, AWS, etc.)
- LLMs / models (OpenAI, Anthropic, Mistral, Gemini, Groq, Ollama, etc.)
- Video & vision (HeyGen, Tavus, vision services)
- Audio processing utilities (VAD, noise filters)
This broad set of integrations makes Pipecat suitable for experimenting with different provider mixes and for production deployments that require portability across vendors.
Getting started (high level)
- Install the Pipecat package (or add it to a project scaffolded with uv).
- Configure environment variables and add provider credentials for the services you want to use.
- Define a pipeline composed of processors (STT -> intent / LLM -> TTS, etc.) and configure transports for client connections.
- Run locally for development, use the CLI to monitor, and deploy agent processes to your cloud environment when ready.
Example (conceptual):
# pseudocode outline
from pipecat import Pipeline, STTService, LLMService, TTSService
pipeline = Pipeline([
STTService(...),
LLMService(...),
TTSService(...)
])
pipeline.serve(transport='webrtc')Contributing & community
Pipecat is open-source and welcomes contributions: bug reports, docs improvements, new service adapters, and example apps. The repository includes development setup instructions, testing guidelines, and a contributing guide. Community support channels include Discord and documentation site.
Suitable use cases
- Interactive voice assistants and conversational UIs
- Multimodal agents that combine speech, text and vision
- Rapid prototyping of voice-first workflows
- Deployable production agents with vendor-agnostic integrations
Overall, Pipecat aims to simplify building real-time conversational agents by providing a modular, extensible framework focused on voice and multimodal experiences.
