Skip to content

Voice, video and robots

Your agent can talk on a phone or browser call, see a camera or a shared screen, and move a robot, with the same approvals, budgets, cost and traces as every other tool. All of it comes in the next release; the rest of the SDK is on Features.

  • VoxCat, our own voice pipeline


    VoxCat (VoiceSession) handles phone, browser and laptop calls with no extra media server. It knows when the caller has finished on its own, starts answering before the turn is over, and stops when interrupted. A payment the caller talks over still happens only once.

    A voice call with no framework · VoxCat on its own · Benchmark

  • Or inside LiveKit and Pipecat


    Already use LiveKit Agents or Pipecat? The same agent and the same safety rules plug into them. Use them for rooms with many people, SIP trunks and mobile apps.

    Build a voice agent · LiveKit · Pipecat

  • Agents that can see


    The caller turns on a camera or shares a screen, or a robot sends its camera. When the agent needs to, it takes one look and answers what it sees. Nothing is sent unless it looks.

    Let a voice agent see

  • Robots that stay safe


    Every move is checked against where the robot may go, how fast and how hard, before it runs. An emergency stop latches until a person resets it, and every action is audited.

    Gate a robot

Start here

I want to… Read
answer phone calls (Twilio, Telnyx, Plivo, Vonage) Phone calls
talk to the agent in a browser or on my laptop A voice call with no framework, Transports
use hosted voices (OpenAI, Deepgram, ElevenLabs, Cartesia) or a realtime model Hosted speech, Fast hosted voices, Realtime models
add idle prompts, recording, voicemail detection or a noise filter Voice call features
run on open models I host (Whisper, Parakeet, Kyutai, Chatterbox, Orpheus) or add a talker Open speech models
keep payments and bookings safe when the caller interrupts Build a voice agent
answer sooner: speculative replies, early read-only tools, acknowledgements, cached answers Answer sooner
be fast with any STT and TTS, set a call up from one YAML file VoxCat on its own
compare VoxCat with LiveKit and Pipecat VoxCat benchmark, Voice stacks compared
choose between our stack, LiveKit and Pipecat Voice stacks compared
let the agent see a camera or a screen Let a voice agent see
drive a robot or a simulator safely Gate a robot

Measured, not claimed

All on one laptop, offline, with no account or key:

  • First reply: 2.4 s on our stack against 4.3 s on LiveKit and Pipecat, on the same scripted call (the race).
  • Charged once: 20 of 20 calls where the caller talks over a payment.
  • Voicemail: 6 of 6 answering-machine greetings found, 0 of 8 people mistaken for one.
  • "Stop!" by voice: a simulated arm halted in 470 ms (median of 5).
  • Seeing: a small offline vision model sorted 6 of 6 coloured blocks with a simulated arm.
  • ROS 2: a turtlesim robot moved, stopped and refused to move after the stop, 5 of 5.

Everything in it

Feature Guide
Interruption-safe tools: a started payment or booking finishes once when the caller cuts in; a repeat window; spoken yes/no approvals bound to the call Build a voice agent
VoiceSession: our own call pipeline with no media server; mic and speakers, WebSocket, browser WebRTC, Twilio, Telnyx, Plivo and Vonage; Silero VAD and Smart Turn; an echo guard A voice call with no framework, Phone calls, Transports
VoxCat on its own: EagerTurns (its own end of turn, no model), replies on the STT's interim words, first words before the sentence ends, VoiceConfig and agent-adapter voice serve; a head-to-head benchmark with LiveKit and Pipecat VoxCat on its own, Benchmark
Speed tricks: a speculative reply on the early transcript, opted-in read-only tools run early (gated in the executor), pre-rendered acknowledgements for slow replies, an answer cache with audio, warm-up at call start, per-stage turn timings Answer sooner
Speech: Moonshine offline, OpenAI, Deepgram (Nova, Flux), ElevenLabs, Cartesia, Fish Audio, AssemblyAI, Soniox, Rime, any provider via CallableSTT / CallableTTS; early end of turn and text streamed into speech; OpenAI Realtime and Gemini Live Hosted speech, Fast hosted voices, Realtime models
Call features: idle prompt and hang-up, lines the caller cannot cut, mute while tools run, an end-call tool, typed input, recording, a thinking sound, voicemail detection, a noise filter Voice call features
LiveKit Agents and Pipecat connectors (preview): the same agent and safety rules inside their rooms and pipelines LiveKit, Pipecat, Voice stacks compared
Video: VideoFeed keeps the latest frame from a browser camera or screen share, a local camera, the screen or a robot camera; the read-only look_tool hands it to the model Let a voice agent see
Robot gate: workspace, speed and force limits checked before every move, a simulator shadow (MuJoCo), a latched emergency stop, an audit log; SimRobot and a ROS 2 port Gate a robot

Not built yet: phone menu (IVR) navigation is next on the Roadmap. Avatars, multi-person rooms, SIP trunks and mobile apps come through the LiveKit and Pipecat connectors.