Run a voice call with no framework¶
VoiceSession is a whole voice pipeline in this package. It takes raw audio from a
transport, finds the turns, transcribes them, runs your VoiceAgent,
speaks with any text-to-speech and plays the reply. No Pipecat, no LiveKit, no media server.
flowchart LR
T["Transport: mic, WebSocket, Twilio, WebRTC"] --> D[Turn detector]
D --> S[Speech-to-text]
S --> A["VoiceAgent + executor"]
A --> P["Player: TTS + playout buffer"]
P --> T
Start a call¶
from agent_adapter.voice import LocalAudio, VoiceAgent, VoiceSession
from agent_adapter.voice.moonshine import MoonshineSTT, MoonshineTTS
from agent_adapter.voice.silero import SileroTurns
from agent_adapter.voice.smart_turn import SmartTurn
session = VoiceSession(
LocalAudio(),
stt=MoonshineSTT(),
tts=MoonshineTTS(),
turns=SmartTurn("smart-turn-v3.2-cpu.onnx", SileroTurns("silero_vad.onnx")),
)
agent = VoiceAgent(model, executor, session.speaker, instructions="You book shows.")
await session.run(agent) # until the transport ends; then the agent is closed
Moonshine (MIT, Kokoro voices Apache-2.0) runs on the CPU with no keys:
pip install "agent-adapter[voice-local,vad]". Any other provider fits CallableSTT(fn)
and CallableTTS(fn, sample_rate=); an STT with open() (StreamingSTT) hears each
turn as it is spoken.
Deepgram, OpenAI, ElevenLabs and Cartesia are ready-made: see
Hosted speech, echo and phone calls.
Pick a transport¶
| Transport | For | Notes |
|---|---|---|
LocalAudio() |
a laptop | sounddevice; headphones, or speakers with an EchoGuard |
AudioSocket(socket) |
a browser, app or gateway | binary PCM16 both ways; we send {"type": "clear"} on a cut |
TwilioStream(socket) |
phone calls | Twilio Media Streams, 8 kHz μ-law; clear on a cut, mark per line; keypad, hang up, transfer |
WebRTCAudio() |
a browser, peer to peer | pip install "agent-adapter[webrtc]" (aiortc, BSD) |
Every transport drains the session's playout buffer at the pace audio plays, so a pause
or a cut takes effect within one block (20 ms), or lead_ms (60 to 80 ms) over a socket.
On Twilio a line counts as played only when Twilio echoes its mark.
from agent_adapter.voice import TwilioStream
from agent_adapter.voice.socket import text_socket
@app.websocket("/twilio") # your <Connect><Stream url="wss://.../twilio"/> TwiML
async def twilio(ws):
await ws.accept()
phone = TwilioStream(text_socket(ws.send_text, ws.iter_text()))
session = VoiceSession(phone, stt=stt, tts=tts, turns=turns)
await session.run(VoiceAgent(model, executor, session.speaker))
For WebRTC, answer = await WebRTCAudio().accept(offer) answers the browser's SDP offer from
any route you own; webrtc_page("/offer") is a one-page client to try it. The browser
cancels echo, so speakers work. With no ice_servers it contacts no STUN server and offers
local addresses only; pass your STUN or TURN servers for calls across networks.
What it does on a call¶
| When | What happens |
|---|---|
| The caller starts talking | playback pauses within one audio block |
The speech lasts confirm_ms (300) or has words |
the turn is cut: thinking stops; a started booking finishes once |
| A cough, or "uh-huh" while the agent speaks | playback resumes where it paused; nothing is cut |
The caller goes quiet for early_ms (200) |
the final transcription starts while the detector still decides |
| The caller goes on after all | that early pass is dropped and the whole turn is heard |
| A line is cut half-way | the history keeps the words that played, by the share of audio heard |
| The agent has more to say | the next sentence is synthesised while this one plays |
| Each turn | a custom.voice_turn event with voice_to_voice_ms and cut_ms, never the words |
Test all of it offline with agent_adapter.voice.testing: MemoryTransport (you send the
caller's audio and pull the agent's), ScriptedSTT and WordTTS.
Idle prompts, voicemail, recording, a thinking sound, hang-up and a noise filter are on Voice call features. How this stack compares with LiveKit and Pipecat, feature by feature: Voice stacks compared.
The same call on three stacks¶
scripts/voice_race.py runs one scripted call on our pipeline, on Pipecat with
voice_processor() and on LiveKit with attach_voice_agent: the same caller, the same
Silero and Smart Turn detectors, the same Moonshine models and the same scripted model. The
caller asks for a booking, then cuts in 0.3, 0.8, 1.5 or 2.5 s into it. Each stack is timed
from what the caller hears. 20 runs per stack (5 at each cut point) on a Windows laptop, CPU
only (Pipecat and LiveKit re-run after the fix to the quiet measure below); medians, with
the range in brackets:
| Our pipeline | Pipecat 1.16 | LiveKit 1.8 | |
|---|---|---|---|
| Charged once | 20 of 20 | 20 of 20 | 20 of 20 |
| First reply heard, after the caller stops | 2.4 s (2.1 to 2.7) | 4.3 s (4.0 to 4.4) | 4.3 s (4.2 to 8.7) |
| Agent goes quiet, after the caller cuts in | 54 ms (0 to 78), 10 runs | 54 ms (31 to 79), 10 runs | 484 ms (453 to 485), 5 runs |
| The VoiceAgent is told of the cut | 352 ms | 63 ms | 391 ms |
| CPU time for the call | 25 s (16 to 27) | 20 s (20 to 28) | 32 s (27 to 33) |
How to read it:
- First reply is taken from the runs that cut in at 1.5 and 2.5 s, where nothing was cut
before the agent first spoke. We are faster because speech-to-text streams and starts its
last pass early, and the voice plays from its first chunk. Pipecat runs its own
MoonshineSTTService, which transcribes once the turn ends. LiveKit has no Moonshine service, so both use the same batch adapters we wrote for them. - Goes quiet counts only runs where the agent was talking when the caller cut in. We tie with Pipecat; LiveKit is slower because its turn detection hears the cut later. A gap in the audio counts as quiet: Pipecat writes nothing while silent, and a first version of this script missed that and showed Pipecat taking 5.5 s.
- Told of the cut is later for us by design: playback stops at once, but the turn is cut only after 300 ms of speech, so a cough does not throw away the agent's work.
- CPU is counted from the moment the call starts, after models load, in a later run of 8 calls per stack. We use about a quarter more than Pipecat: speech-to-text and the voice both stream, which is what makes the first reply twice as fast. A first version of this script started our clock before the models loaded and showed 47 s.
- The caller is a second Kokoro voice sent as audio frames, not a person or a microphone. The cut line was heard as "Wait till that go through" in some runs on every stack. The same audio gives "did" or "till" depending only on the silence before it.