Voice, video and robots¶
Your agent can talk on a phone or browser call, see a camera or a shared screen, and move a robot, with the same approvals, budgets, cost and traces as every other tool. All of it comes in the next release; the rest of the SDK is on Features.
-
VoxCat, our own voice pipeline
VoxCat (
VoiceSession) handles phone, browser and laptop calls with no extra media server. It knows when the caller has finished on its own, starts answering before the turn is over, and stops when interrupted. A payment the caller talks over still happens only once.A voice call with no framework · VoxCat on its own · Benchmark
-
Or inside LiveKit and Pipecat
Already use LiveKit Agents or Pipecat? The same agent and the same safety rules plug into them. Use them for rooms with many people, SIP trunks and mobile apps.
-
Agents that can see
The caller turns on a camera or shares a screen, or a robot sends its camera. When the agent needs to, it takes one look and answers what it sees. Nothing is sent unless it looks.
-
Robots that stay safe
Every move is checked against where the robot may go, how fast and how hard, before it runs. An emergency stop latches until a person resets it, and every action is audited.
Start here¶
| I want to… | Read |
|---|---|
| answer phone calls (Twilio, Telnyx, Plivo, Vonage) | Phone calls |
| talk to the agent in a browser or on my laptop | A voice call with no framework, Transports |
| use hosted voices (OpenAI, Deepgram, ElevenLabs, Cartesia) or a realtime model | Hosted speech, Fast hosted voices, Realtime models |
| add idle prompts, recording, voicemail detection or a noise filter | Voice call features |
| run on open models I host (Whisper, Parakeet, Kyutai, Chatterbox, Orpheus) or add a talker | Open speech models |
| keep payments and bookings safe when the caller interrupts | Build a voice agent |
| answer sooner: speculative replies, early read-only tools, acknowledgements, cached answers | Answer sooner |
| be fast with any STT and TTS, set a call up from one YAML file | VoxCat on its own |
| compare VoxCat with LiveKit and Pipecat | VoxCat benchmark, Voice stacks compared |
| choose between our stack, LiveKit and Pipecat | Voice stacks compared |
| let the agent see a camera or a screen | Let a voice agent see |
| drive a robot or a simulator safely | Gate a robot |
Measured, not claimed¶
All on one laptop, offline, with no account or key:
- First reply: 2.4 s on our stack against 4.3 s on LiveKit and Pipecat, on the same scripted call (the race).
- Charged once: 20 of 20 calls where the caller talks over a payment.
- Voicemail: 6 of 6 answering-machine greetings found, 0 of 8 people mistaken for one.
- "Stop!" by voice: a simulated arm halted in 470 ms (median of 5).
- Seeing: a small offline vision model sorted 6 of 6 coloured blocks with a simulated arm.
- ROS 2: a turtlesim robot moved, stopped and refused to move after the stop, 5 of 5.
Everything in it¶
| Feature | Guide |
|---|---|
| Interruption-safe tools: a started payment or booking finishes once when the caller cuts in; a repeat window; spoken yes/no approvals bound to the call | Build a voice agent |
VoiceSession: our own call pipeline with no media server; mic and speakers, WebSocket, browser WebRTC, Twilio, Telnyx, Plivo and Vonage; Silero VAD and Smart Turn; an echo guard |
A voice call with no framework, Phone calls, Transports |
VoxCat on its own: EagerTurns (its own end of turn, no model), replies on the STT's interim words, first words before the sentence ends, VoiceConfig and agent-adapter voice serve; a head-to-head benchmark with LiveKit and Pipecat |
VoxCat on its own, Benchmark |
| Speed tricks: a speculative reply on the early transcript, opted-in read-only tools run early (gated in the executor), pre-rendered acknowledgements for slow replies, an answer cache with audio, warm-up at call start, per-stage turn timings | Answer sooner |
Speech: Moonshine offline, OpenAI, Deepgram (Nova, Flux), ElevenLabs, Cartesia, Fish Audio, AssemblyAI, Soniox, Rime, any provider via CallableSTT / CallableTTS; early end of turn and text streamed into speech; OpenAI Realtime and Gemini Live |
Hosted speech, Fast hosted voices, Realtime models |
| Call features: idle prompt and hang-up, lines the caller cannot cut, mute while tools run, an end-call tool, typed input, recording, a thinking sound, voicemail detection, a noise filter | Voice call features |
| LiveKit Agents and Pipecat connectors (preview): the same agent and safety rules inside their rooms and pipelines | LiveKit, Pipecat, Voice stacks compared |
Video: VideoFeed keeps the latest frame from a browser camera or screen share, a local camera, the screen or a robot camera; the read-only look_tool hands it to the model |
Let a voice agent see |
Robot gate: workspace, speed and force limits checked before every move, a simulator shadow (MuJoCo), a latched emergency stop, an audit log; SimRobot and a ROS 2 port |
Gate a robot |
Not built yet: phone menu (IVR) navigation is next on the Roadmap. Avatars, multi-person rooms, SIP trunks and mobile apps come through the LiveKit and Pipecat connectors.