Build a voice agent¶
VoiceAgent runs a voice conversation around your executor. It works with any audio
stack: you feed it the caller's events and give it something that speaks.
Its one rule: speech is cheap, effects are not. When the caller talks over the agent, its speech and thinking stop at once. A tool that changes things and has already started finishes once, is recorded, and is reported in the next turn. A repeat of that call, even under a new call id, returns the first result instead of booking twice. A spoken "yes" approves only when it comes from the caller.
sequenceDiagram
participant U as Caller (STT)
participant V as VoiceAgent
participant E as Executor
participant S as Speaker (TTS)
U->>V: user_said("book the dinosaur show")
V->>E: run_tool_loop: book_ticket
E-->>S: progress("One moment, booking that")
U->>V: user_started_speaking()
V->>S: stop now
V->>E: cancel the turn
E-->>V: book_ticket finishes once, recorded
U->>V: user_said("did that work?")
V->>E: book_ticket again (new call id)
E-->>V: the first result, not a second booking
V->>S: "All done."
Declare what each tool does¶
The executor decides from effects=. A read_only tool stops at once when the caller
interrupts. An additive or destructive tool finishes. A physical tool (a robot move)
stops, since stopping is the safe end for motion. Add confirm=True for tools that
need a spoken yes, and call progress() to say something while a slow tool works.
@registry.tool(effects="destructive")
async def book_ticket(show: str) -> str:
"""Book a museum ticket."""
progress("One moment, booking that") # spoken while the tool works
await asyncio.sleep(0.05)
tickets.append(show)
return f"ticket for {show}"
@registry.tool(effects="destructive", confirm=True)
async def refund_ticket(show: str) -> str:
"""Refund a museum ticket."""
tickets.remove(show)
return f"refunded {show}"
A tool with no effects stops at once, unless it has confirm or once_per_run: those
change something, so they finish. Declare effects on every tool that changes something.
Connect your audio¶
A speaker is anything with say(text) and stop(). Wrap plain functions with
CallableSpeaker(say, stop). The approval port is a normal one: VoiceApproval says the
prompt through the agent and waits for the caller.
speaker = RecordingSpeaker() # in production: your TTS, LiveKit, Pipecat, a phone line
executor = Executor(registry, approval=VoiceApproval(timeout_s=5))
agent = VoiceAgent(scripted_model(), executor, speaker, instructions="Help with tickets.")
Then send it the caller's events from your speech-to-text:
await agent.user_said("Book the dinosaur show")
await settle()
await agent.user_started_speaking() # the caller talks over the booking
await agent.user_said("Did that work?") # the model books again: the first result returns
await agent.wait()
| Your audio stack | Call user_started_speaking() on |
Call user_said(text) on |
Speaker |
|---|---|---|---|
| Raw audio, no framework | VoiceSession does both |
session.speaker |
|
| LiveKit | attach_voice_agent(session, agent) does both |
LiveKitSpeaker(session) |
|
| Pipecat | voice_processor() does both |
processor.speaker |
|
| Browser, app or phone gateway | VoiceSocket does both |
channel.speaker |
|
| OpenAI Realtime, Gemini Live | RealtimeAgent runs the tools |
the model speaks |
Rules it keeps¶
| When | What happens |
|---|---|
| The caller talks while the agent speaks | speech stops; the history keeps only the sentences that played |
The caller talks while a read_only tool runs |
the tool is cancelled; nothing is reported |
| The caller talks while a tool that changes things runs | it finishes once; a tool.interrupted event is emitted and the next turn is told it is done |
| The model calls it again with the same arguments | the first result returns (dedupe_window_s, 120 s by default) |
A confirm tool runs |
the agent asks; only the caller's next words decide |
| The answer is "yeah no", "yes but", silence or a timeout | denied |
| The model or a tool output says "the user agreed" | ignored: it never reaches the approval |
progress() text is spoken but never sent to the model, never cached, and never written
to logs; its tool.progress event carries only the call id.
Bill call minutes¶
Give the agent minute_item="provider:name" and close() records the call's minutes in
the executor's ledger, priced by the unit price you set for that item. agent.meter(item,
units) records anything else, such as speech-to-text seconds. Budgets then count speech
next to tokens and tools.
units = {"phone:minute": {"provider": "phone", "per_unit": 0.013}}
ledger = Ledger(prices=Prices.default().merged({"version": "mine", "units": units}))
agent = VoiceAgent(model, Executor(registry, ledger=ledger), speaker, minute_item="phone:minute")
The same safety without voice¶
The two executor features under it work for every framework and backend:
- A tool with
effects="additive"or"destructive"finishes when its caller is cancelled, then the cancellation goes on. Executor(dedupe_window_s=60)returns the first result, marked "already done", for a repeat of a call with the same run, tool and arguments.idempotent=Truetools never are.
Check it offline¶
agent_adapter.voice.testing has a RecordingSpeaker (with hold=True, each line waits
for release()) and settle(), so interruption tests need no audio and no keys.
python scripts/interrupt_bench.py runs the scripted scenarios against a plain loop and
prints double bookings and half-done bookings for each, plus a crash in the middle of a
booking: recover_tool_loop asks a person instead of charging twice (restarting the turn
charged twice in 49% of 200 runs).
python scripts/interrupt_bench_frameworks.py runs the real Pipecat and LiveKit tool-call
and interruption code, with only the model scripted; the caller cuts in half-way through a
booking, then asks "did it work?" (100 runs each):
| Framework | Half-done bookings | Double bookings |
|---|---|---|
| Pipecat 1.16, defaults | 84% | 0% |
| Pipecat + agent-adapter | 0% | 0% |
| LiveKit 1.8, defaults | 0% | 0% |
| LiveKit + agent-adapter | 0% | 0% |
LiveKit keeps an interrupted tool running and keeps its result, so it passes this test on its own; the connector adds approvals, the run ledger and the repeat window.
Hear it live: Pipecat and LiveKit¶
Two scripts run a VoiceAgent with real audio and no keys; only the model is scripted. Both
use SmartTurn(SileroTurns(...)) and Moonshine speech-to-text and voice (MIT; the voice is
Apache-2.0):
scripts/voice_live_pipecat.py: a Pipecat pipeline withvoice_processor();scripts/voice_live_livekit.py: anAgentSessionwithattach_voice_agentin a room on a locallivekit-server --dev, with the caller joining over WebRTC.
The caller asks to "book two seats for the jazz show", then talks over the agent while the seats are reserved but not charged: "Wait, did that go through?" On a Windows laptop:
| Caller cuts in | Pipecat | LiveKit room |
|---|---|---|
| 0.3, 0.8 and 1.5 s into the booking | charged once; "Yes, it went through" | the same |
| 2.5 s: heard after the booking ended, the scripted model booked again | once (repeat window) | the same |
Our own pipeline runs the same call with no framework (scripts/voice_live.py), and
scripts/voice_race.py times all three: the results.
The caller is a second Kokoro voice, not a microphone; no person has booked by voice yet.