Voice agents¶
Turns, safe interruptions and spoken approval around the executor, our own audio pipeline and transports, plus offline fakes. Guide: Build a voice agent.
agent_adapter.voice¶
Voice agents: safe interruptions, spoken approval and one ledger, on any audio stack.
VoiceSession is our own pipeline (audio, turns, speech-to-text, text-to-speech); the
transports are LocalAudio, AudioSocket, TwilioStream, TelnyxStream, PlivoStream,
VonageStream and WebRTCAudio.
YES_WORDS = frozenset({'yes', 'yeah', 'yep', 'yup', 'sure', 'ok', 'okay', 'confirm', 'confirmed', 'correct', 'right', 'absolutely', 'definitely', 'please', 'go', 'ahead', 'do', 'it', "that's", 'thats', 'fine', 'affirmative', 'proceed'})
module-attribute
¶
AudioSocket
¶
An AudioTransport over a BinarySocket.
AudioTransport
¶
Bases: Protocol
Where the call's audio comes from and goes to: a sound card, a socket, a phone line.
frames() yields the caller's audio as 16-bit mono PCM at in_rate. start(buffer)
makes the transport drain buffer (PCM16 at out_rate) in real time.
CallObserver
¶
Override what you need; every method is optional.
agent_audio is called as the agent's audio plays, possibly on the sound card's
thread: keep it quick. caller_said returning True keeps that text from the agent.
agent_audio(pcm)
¶
The agent's audio as it plays, PCM16 mono at the transport's out_rate.
caller_audio(pcm)
¶
The caller's audio: 16 kHz PCM16 mono, after the session's filters.
caller_said(text)
¶
A final transcript of the caller; return True to keep it from the agent.
ended()
async
¶
The call ended.
started(session)
¶
The call started.
CallRecorder
¶
Bases: CallObserver
Writes path when the call ends: 16 kHz, 16-bit, two channels.
Each side is placed at the time it was heard, so the two line up. Recording stops
after max_minutes. Tell callers they are recorded where the law asks you to.
CallableSTT
¶
An STT from fn(pcm, sample_rate) -> str; a sync function runs in a thread.
CallableSpeaker
¶
A Speaker from plain functions: say(text) and an optional stop(), sync or async.
CallableTTS
¶
A TTS from fn(text) returning PCM16 bytes, or an iterable or async iterable of
chunks; a sync function runs in a thread.
EchoGuard
dataclass
¶
When speech counts as a barge-in while the agent talks on speakers with no echo
cancelling: loud_ms of audio at level RMS or above (a fraction of full scale; the
agent's voice through the mic is quieter than a caller at it), or speech_ms of speech.
Tune level on your hardware: above the agent's echo, below your voice.
EventPump
¶
Hands caller events to a VoiceAgent one at a time, in order.
Transport callbacks must return at once, but a cut can wait for a booking to finish and an approval waits for the caller's next words; the pump queues events so the transport never waits (and never deadlocks on its own next message) and order is kept.
aclose()
async
¶
Stop handing events over (queued ones are dropped).
drop_speculation()
¶
The caller went on after the early transcript.
idle()
async
¶
Wait until every queued event has been handed over.
said(text, timing=None)
¶
A final transcript; blank text is ignored.
speculate(text)
¶
An early transcript: the agent may start its reply (VoiceAgent.speculate).
started_speaking()
¶
The caller started talking (VAD).
turn_ended()
¶
The caller finished a turn of real speech (VoiceAgent.turn_ended).
IdlePolicy
¶
Bases: CallObserver
When nobody has spoken for after_s, ask prompt; after hang_up_after unanswered
prompts, say goodbye and hang up. Any caller words start the count again.
LocalAudio
¶
Microphone in at in_rate, speakers out at out_rate, in frame_ms blocks.
The speaker callback pulls straight from the playout buffer, so a pause or a cut is heard within one block.
hang_up()
¶
End the call: frames() stops.
NoiseGate
¶
Learns the steady background noise in each frequency band and turns it down by up to
reduction_db, leaving speech alone. 16 kHz PCM16 in and out, same length, about
frame / 16 ms later. Steady noise (fans, hum, road) it handles; other voices it cannot.
Any callable pcm -> pcm works as a filter, so a stronger denoiser plugs in the same way.
Player
¶
Speaks lines through a PlayoutBuffer and knows how much of each the caller heard.
prepare(text) starts synthesis early, so the next line is ready when this one ends.
After stop(), heard_of_cut holds the words of the cut line that played (estimated
from the share of its audio that played), so the history keeps only what was heard.
The last cache_lines short lines are kept as audio, so a repeated line ("One
moment.") plays at once; 0 turns that off. text_hooks rewrite each line just before
synthesis (a SpokenForm from agent_adapter.voice.quality); heard_of_cut stays in
the original words.
speaking
property
¶
Audio is queued or a line is being spoken.
pause()
¶
Hold playback (the caller may be cutting in).
prepare(text)
¶
Start synthesising text now; say(text) will use it.
say(text)
async
¶
Speak text; return when it has played, or was cut.
stop()
async
¶
Cut now: drop queued audio and pending synthesis; record what was heard.
warm(lines)
async
¶
Synthesise lines into the cache now, so they play at once later.
PlivoStream
¶
Bases: PhoneStream
An AudioTransport over a Plivo Audio Streaming WebSocket (any TextSocket).
content_type is one of Plivo's: audio/x-l16;rate=8000 (Plivo's default),
audio/x-l16;rate=16000 or audio/x-mulaw;rate=8000. The identity fields come from
the first start message only; trust them only if the host verified
X-Plivo-Signature-V3 on the upgrade.
STT
¶
Bases: Protocol
Speech-to-text for one finished turn of 16-bit mono PCM.
Speaker
¶
TTS
¶
Bases: Protocol
Text-to-speech: 16-bit mono PCM chunks at sample_rate, as soon as each is ready.
TelnyxStream
¶
Bases: PhoneStream
An AudioTransport over a Telnyx Media Streaming WebSocket (any TextSocket).
The identity fields come from the first start message only; trust them only if the
host verified the telnyx-signature-ed25519 header on the upgrade.
ThinkingSound
¶
Bases: CallObserver
Loops pcm (16-bit mono at sample_rate) at gain once a turn has gone delay_s
with nothing to say, and stops as soon as the agent speaks or the caller talks.
It is queued chunk_ms at a time, so speech waits at most that long behind it. It
never counts as the agent speaking: no cut, no echo guard, no voice-to-voice time.
TurnTiming
dataclass
¶
When each step of one caller turn happened, in seconds on the monotonic clock.
stages()
¶
Each stage that happened, in ms after the caller's last speech (negative if before).
TwilioCalls
¶
A small async client for the Calls resource; client takes your httpx.AsyncClient.
aclose()
async
¶
Close the HTTP client this object made (one you passed in stays open).
call(to, from_, *, twiml=None, url=None, status_callback=None, timeout_s=None)
async
¶
Place an outbound call that runs twiml (or fetches it from url) on answer.
hang_up(call_sid)
async
¶
End a live call.
transfer(call_sid, to, *, caller_id=None)
async
¶
Hand a live call to a person at to; the media stream ends as Twilio dials.
update(call_sid, **form)
async
¶
Change a live call (Twiml, Url or Status, as Twilio names them).
TwilioStream
¶
Bases: PhoneStream
An AudioTransport over a Twilio Media Streams WebSocket (any TextSocket).
The identity fields (call_sid, stream_sid, parameters) come from the first start
message only; trust them only if the host verified X-Twilio-Signature on the upgrade.
hang_up()
async
¶
End the call: through the Calls API when calls is set, and end frames().
With no calls, only our side ends; Twilio then runs the TwiML after <Connect>,
which hangs up when there is none.
transfer(to, *, caller_id=None)
async
¶
Hand the caller to a person at to; Twilio ends this stream as it dials.
VideoFeed
¶
The latest frame of one video source, for look_tool. Start sources in async code.
latest
property
¶
The latest kept frame, or None.
age()
¶
Seconds since the latest frame was kept, or None with no frame.
camera(device=None)
¶
Read a local camera; device defaults to the first one this platform finds.
capture(file, *, format=None, options=None)
¶
Read any FFmpeg input (a device, a file, a stream URL) through aiortc's MediaPlayer.
clear()
¶
Forget the latest frame (the camera was switched off).
close()
async
¶
Stop every source this feed started.
follow(track)
¶
Read a video track (aiortc MediaStreamTrack) until it ends.
jpeg()
¶
The latest frame as JPEG, encoded once per frame; None with no frame.
poll(source)
¶
Call source (sync or async) fps times a second; it returns a frame or None.
A sync source runs on the event loop, so keep it quick (a MuJoCo render is); give a
slow one as an async function, e.g. one that awaits asyncio.to_thread.
push(frame)
¶
Offer a frame; it is kept unless the last one was kept under 1/fps seconds ago.
screen(display=None)
¶
Read the screen (Windows, macOS, X11; not Wayland).
VoiceAgent
¶
A voice agent from a model, an executor and a speaker; works with any audio stack.
Feed it the caller's events: user_started_speaking() when they start talking and
user_said(text) with each final transcript. Speech is cheap, effects are not:
talking over the agent stops its speech and its thinking at once, but a tool that
changes things (effects=) and has started finishes once and is reported in the next
turn. Repeats of a finished call within dedupe_window_s return the first result.
With minute_item ("provider:name"), close() bills the call's minutes at its unit price.
With a VoiceApproval, a yes counts only if user_started_speaking() came after the
question finished playing. With mute_during_tools, the caller cannot cut in while a
tool runs (a VoiceSession drops that speech). speed adds speculative replies,
acknowledgements, an answer cache and warm-up (Speed).
busy
property
¶
A turn is running: the model is thinking, a tool runs, or the reply plays.
interruptible
property
¶
Whether the caller may cut in now.
close()
async
¶
End the call: cut any turn, stop speaking and bill the call minutes (once).
drop_speculation()
¶
The caller went on: drop the speculative reply.
meter(item, units)
¶
Bill units of item ("provider:name", e.g. STT seconds) to this call's run.
say(text, *, interruptible=True, record=True)
¶
Speak text now, outside a model turn: a greeting, a notice, a goodbye.
With record, what the caller heard of it joins the history. With
interruptible=False, the caller cannot cut it (a VoiceSession drops that speech).
speculate(text)
¶
Start replying to text, an early transcript, while the caller may still go on.
Kept if user_said brings the same words and nothing was said since; needs speed.
turn_ended()
¶
The caller finished a turn of real speech; its transcript is on the way.
user_said(text, *, timing=None)
async
¶
A final transcript: an answer to a waiting question, or a new turn.
With speed, the stages of the reply are noted in timing.
user_started_speaking()
async
¶
The caller talks over the agent: stop speech now; cut the turn unless it asks yes/no.
wait()
async
¶
Wait until the current turn has finished speaking.
wait_spoken()
async
¶
Wait until everything queued to speak has played.
warm_up(providers=())
async
¶
Get speed's connections, lines and prefetches ready; VoiceSession calls it.
VoiceApproval
¶
Approval port for voice: asks the caller aloud, then waits for their own answer.
Each call binds its say under its run id (a VoiceAgent or RealtimeAgent does
this), so many calls can share one executor: a question goes only to the caller whose
run asked, and only that caller's words answer it. A run with no bound call is denied.
One question is open per call at a time, and an answer counts only when the caller
began speaking after the prompt finished playing.
Only text passed to answer (the caller's transcript) can approve; model text and tool
output never reach it. Silence until timeout_s, or any answer that is not a clear yes,
denies. prompts maps a tool name to its own prompt function. readback may put a line
before the prompt, such as "Did you say 4,500?" for words the caller's speech-to-text
was unsure of (UnsureWords.readback in agent_adapter.voice.quality); the one yes
then answers both.
waiting
property
¶
True while any call has a question open.
answer(text, *, run_id=None, question=None)
¶
Give the caller's final words to their open question; False when none is open.
With run_id, the words count only if question is the id armed_question gave
when the caller began them; others are taken but dropped. True means the words
belonged to the open question, used or not.
armed_question(run_id)
¶
The id of the run's question once its prompt has played; None before that.
Read it when the caller starts speaking and hand it to answer with their words:
speech that began before the question was fully asked never answers it.
bind(say, *, run_id=None)
¶
Speak this run's prompts with say. With no run_id, say serves every run that
has no call of its own: only for a process that handles one call at a time.
request(request)
async
¶
Ask the run's caller aloud and decide from their answer; anything unclear denies.
unbind(run_id=None, say=None)
¶
Forget the run's call (when say is given, only if it is still the bound one);
its open question is denied at once.
waiting_for(run_id)
¶
True while the run's call has a question playing or waiting for its answer.
VoiceSession
¶
Runs a VoiceAgent on an AudioTransport; give the agent session.speaker.
turns gets 16 kHz audio. confirm_ms of speech confirms a cut before the transcript
is ready; backchannels are words that never cut the agent while it speaks.
Transcription starts early: a StreamingSTT hears the turn as it is spoken, and the
final pass starts after early_ms of quiet, while the detector is still deciding
whether the turn is over. If the caller goes on, that pass is dropped. With an agent
Speed, the early transcript also starts the reply (VoiceAgent.speculate), first on
the stream's interim words (a partial attribute) if it has them, and the agent's
warm-up runs as the call starts. With EagerTurns, those words also move the turn's end.
Keypad digits from a transport with on_digit (TwilioStream) reach the agent as
keypad_text, once keypad_wait_ms passes with no new digit, at once on #, or
after max_digits. A turn that never pauses ends after max_turn_s.
filters transform the caller's 16 kHz audio first (a noise filter); observers
(CallObserver) add call features such as CallRecorder, IdlePolicy and
VoicemailDetector.
agent
property
¶
The agent this session runs, once run started.
caller_talking
property
¶
A caller turn is open.
end_after_reply()
¶
Hang up once the agent's current reply has played (an end_call tool).
hang_up(*, after_speech=False)
async
¶
End the call; with after_speech, once the current reply has played.
run(agent)
async
¶
Run the call until the transport ends; then close the agent and the transport.
send_text(text)
¶
A typed message from the caller, handled like a spoken turn.
VoicemailDetector
¶
Bases: CallObserver
Listens to the first window_s of a call for a machine.
Two offline signals: a greeting that says what machines say (PHRASES, or your own
classify(text)), and the beep (a steady tone of 400 to 2,500 Hz for beep_ms).
Once the greeting is heard, the machine's words are kept from the agent. With
message, the agent waits for the beep (or beep_wait_s of quiet after the
greeting), speaks it and hangs up; with on_detect(session, reason), you decide.
VonageStream
¶
Bases: PhoneStream
An AudioTransport over a Vonage Voice API WebSocket (any BinarySocket).
rate must match the NCCO endpoint's content-type. headers holds the first
websocket:connected message's fields (your NCCO headers); trust them only if the
host checked the Authorization header on the upgrade.
binary_socket(send, messages)
¶
A BinarySocket from a send function and an async iterator of received messages.
end_call_tool(session, *, name='end_call', description='End the phone call once your reply has been spoken.')
¶
A tool for the agent to hang up after its goodbye: registry.add(end_call_tool(session)).
is_yes(text, words=YES_WORDS)
¶
True only when every word is a yes word: "yeah no", "yes but" and "" are all no.
look_tool(feed, name='look', *, description=None, max_age=None)
¶
A read-only tool that returns the feed's latest frame as a JPEG image.
With max_age, a frame older than that many seconds counts as no frame: a video track
that stopped sending (a camera switched off) stops being shown as the present.
agent_adapter.voice.session¶
VoiceSession: our own voice pipeline, from raw audio to a VoiceAgent and back.
transport -> turn detector -> speech-to-text -> VoiceAgent -> text-to-speech -> transport
When the caller starts talking, playback pauses within one audio frame. The turn is cut
only once the speech is real: confirm_ms of speech, or words in the transcript. A cough,
or an "uh-huh" while the agent speaks, resumes playback where it paused.
With an EchoGuard (speakers with no echo cancelling), speech that starts while the agent
talks pauses nothing until it is loud enough or long enough; one that never is was the
agent's own voice in the mic and is dropped.
While the agent asks a spoken yes/no question, only a turn that began after the question
finished playing can answer it; earlier speech, an echo of the prompt included, is dropped.
Speech that starts while the agent cannot be cut (VoiceAgent.interruptible) is dropped too.
Also exported here, documented under the canonical path: TurnTiming, VoiceSession.
agent_adapter.voice.timing¶
Also exported here, documented under the canonical path: TurnTiming.
agent_adapter.voice.speed¶
Speed: the speed tricks of a VoiceAgent, and the per-call fast path that runs them.
FastPath
¶
One call's speed state: the running speculation, the gate and the turn plans.
Speed
¶
What a VoiceAgent does to answer sooner; each part is optional.
speculate: start the reply on the early transcript; kept only if the final one has the same words and nothing was said since, dropped if the caller goes on.tools: the read-only tools that reply may run before the turn is final.acks: short pre-rendered lines while a slow reply is on its way.answerswithtenant: replies to repeated questions, with their audio.warmup: connections, lines and read-only prefetches at call start.first_words: speak a reply's first clause once it has this many words, not its first whole sentence (None: sentence by sentence).
agent_adapter.voice.speculate¶
Speculative replies: start answering on the early transcript, keep it only if it was final.
A Speculation runs the agent's tool loop before the turn is over. Its text and progress
lines are held back until the agent adopts it (the final transcript matched) and dropped if
the caller goes on. Tools wait for adoption at the executor, in SpeculationGate, which runs
before guards and approval: only tools named in SpeculativeTools that are declared
effects="read_only" and have no confirm may run early, within their own rate limits.
SPECULATIVE_STAGE = 'voice_speculation'
module-attribute
¶
TOOL_ITEM = 'voxcat:speculative_tool'
module-attribute
¶
ReplyOut
¶
Bases: Protocol
Where a turn's words go: the speech queue, or a tap in front of it.
Speculation
¶
One reply started on the early transcript; its output waits until it is adopted.
ready
property
¶
Some reply text is already waiting.
adopt(out)
¶
Keep it: replay the held output into out and pass the rest straight through.
bind()
¶
Mark the running task as this speculation (for the gate); returns a reset token.
drop()
¶
The caller went on: cancel it, once any early tool call has landed in the cache.
kept()
async
¶
Return once adopted; a dropped speculation is cancelled here.
matches(text, history)
¶
The final transcript is this one, and nothing was said since it started.
SpeculationGate
¶
Bases: Hook
Holds every tool call of an unadopted speculation, except allowed read-only ones.
It runs first among the executor's hooks, so nothing (guards, budget, approval) sees a held call until the turn is final; a dropped speculation is cancelled while it waits.
SpeculativeTools
¶
Tools a speculative reply may run before the caller has finished: opt-in by name.
Each must be declared effects="read_only" with no confirm; the executor checks this
on every call, not only here. At most max_per_turn run early per speculation and
max_per_minute per call; past either, a call waits for the turn to be final. Each early
call is billed as one unit of budget_item and traced as custom.voice.speculative_tool.
Give them cache_ttl, so a dropped speculation's call is reused by the next one.
check(registry)
¶
Raise ToolUsageError if a named tool is missing or may change something.
can_speculate(spec)
¶
A tool declared read-only, with no confirm rule: it may run before the turn is final.
current()
¶
same_words(text)
¶
text reduced for an exact match: lower case, no punctuation, single spaces.
agent_adapter.voice.ack¶
Acknowledgements: a short pre-rendered line while a slow reply is on its way.
Acknowledgements
¶
Plays a short line ("Mm-hm.") when the reply will be slow, and never before a fast one.
The agent learns how long its replies take to start (a moving average from expect_ms).
When a turn is handed over and the reply is expected later than fast_ms, a line plays at
once; otherwise it waits fast_ms and plays only if the reply has still not started. A
reply that is already waiting (a kept speculation, a cached answer) gets none.
Lines play only once rendered at call start (VoiceAgent.warm_up, which a VoiceSession
runs), so an acknowledgement is never slower than the reply it stands in for.
delay_ms(thought_ms)
¶
How long to wait before acknowledging, thought_ms into the thinking.
learn(reply_ms)
¶
A reply started reply_ms after its thinking began.
pick()
¶
The next rendered line, in turn; None before any is rendered.
run(thought_ms, ready, say)
async
¶
Wait as planned, then say a line unless the reply is ready; returns the line.
agent_adapter.voice.answers¶
AnswerCache: replies to questions heard before, with their audio, so they play at once.
Answer
dataclass
¶
AnswerCache
¶
Word-for-word repeats of a question get the same reply, audio included, for ttl_s.
Keys are the tenant plus the question's words (case and punctuation ignored), so two
tenants never share an entry. A VoiceAgent with Speed(answers=..., tenant=...) stores
a reply only when it needed no tool call and opened the call (it depends on the question
and the instructions alone); put seeds known answers. Questions under min_words words
("yes", "the second one") depend on the call and are never stored.
Wrap the session's TTS with voice(tts): it keeps the audio of each stored answer and
plays it back with no synthesis. VoiceAgent.warm_up renders seeded answers' audio.
forget(tenant=None)
¶
Drop every entry, or one tenant's.
get(tenant, question)
¶
The stored reply to question for tenant, or None.
has(tenant, question)
¶
Whether question has a live answer for tenant (not counted as a hit).
lines()
¶
Lines of live answers with no audio yet (rendered at call start).
put(tenant, question, reply)
¶
Store reply for question; None when the question is too short to store.
voice(tts)
¶
tts that replays stored answers' audio and keeps the audio of new ones.
agent_adapter.voice.warm¶
Warm-up at call start: connections, rendered lines and read-only prefetches, all at once.
WarmReport
dataclass
¶
How long each warm-up step took, in ms, and the steps that failed (error type).
Warmup
¶
What to get ready when a call starts, so its first turn is as fast as the rest.
connections are objects with an async warm() (a provider's connection pool) or plain
functions, sync or async (open a socket, send a HEAD request). The call's STT, TTS and
model are warmed too when they have warm(). lines are rendered to audio now (a
greeting); acknowledgements and cached answers are always rendered. tools are read-only
calls, (name, arguments), run now through the executor: with cache_ttl on the tool,
the turn that asks for one gets the cached result. Each step has timeout_s; a failure
is logged and never stops the call.
check(registry)
¶
Raise ToolUsageError if a prefetch tool is missing or may change something.
warm_up(agent, providers=())
async
¶
Run every warm-up step of agent.speed concurrently; never raises.
agent_adapter.voice.media¶
The ports of our own voice pipeline: audio transports, speech-to-text and text-to-speech.
Every port is a small protocol, so any provider fits: a local model, a hosted API, or a
function you already have (wrap it with CallableSTT or CallableTTS).
Also exported here, documented under the canonical path: AudioTransport, CallableSTT, CallableTTS, STT, TTS.
agent_adapter.voice.ports¶
What a voice agent needs from the outside: something that speaks text, and can stop.
Two optional speed ports sit beside the media ports: TurnSignals, a speech-to-text
stream that says early when the provider thinks the turn ended, and StreamingTTS,
text-to-speech that takes text while the model is still writing it.
TurnEvent = Literal['eager_end', 'resumed', 'end']
module-attribute
¶
StreamingTTS
¶
Bases: Protocol
A TTS that can also take text in pieces: open() starts one TTSStream.
TTSStream
¶
Bases: Protocol
One reply spoken while its text is still being written.
push text as it comes (flush=True speaks what is buffered now, at the end of a
clause), then end(). audio() yields 16-bit mono PCM at the TTS sample_rate.
words fills with WordTimes when the provider sends timestamps.
TurnSignal
dataclass
¶
The provider's view of the turn: eager_end (probably done: start a reply now,
but keep it unspoken), resumed (the caller went on: drop that reply) or end
(done). text is the transcript of the turn so far.
TurnSignals
¶
Bases: Protocol
An STTStream that also reports TurnSignals as they arrive.
The callback runs on the event loop and must return at once. A stream without this method still works; the session then decides the end of the turn on its own.
WordTime
dataclass
¶
A spoken word and when it plays, in seconds from the start of the stream's audio.
agent_adapter.voice.player¶
agent_adapter.voice.player re-exports, documented under the canonical path: Player.
agent_adapter.voice.playout¶
Audio waiting to play, drained in real time, so we always know what the caller heard.
A transport drains the PlayoutBuffer: a sound card callback, a WebRTC track, or a
Pacer that sends it over a socket at the pace it plays. Because nothing is handed over
before its time, a pause or a cut takes effect within one frame, and the count of bytes
pulled for each line is what was heard of it.
Pacer
¶
Sends a buffer's audio over a socket at the pace it plays, lead_ms ahead.
The far end buffers what it receives, so audio is never sent more than lead_ms
early: a pause or a cut stops what the caller hears within that much.
PlayoutBuffer
dataclass
¶
Thread-safe queue of PCM16 mono audio at sample_rate, pulled in real time.
With confirm_playout, a line counts as played only when the transport calls
confirm(id) (a phone network reporting a mark), not when its last byte is pulled.
busy
property
¶
Speech is queued (background sound does not count).
idle
property
¶
Nothing is queued to play.
clear()
¶
Drop everything queued, unpause, and end every unfinished line as cut.
confirm(segment_id)
¶
The far end reports segment_id played (with confirm_playout).
finish(segment)
¶
No more audio for segment; it is done once all of it has played.
open(*, ambient=False)
¶
Start a new line; write its audio, then finish it and await done.
pause()
¶
Hold playback where it is (silence until resume).
pull(size, *, pad=True)
¶
Up to size bytes to play now; padded with silence when pad (a sound card).
Paused or empty: silence, or b"" without pad.
Segment
dataclass
¶
One line of speech in the buffer.
heard
property
¶
The share of the written audio that played, 0 to 1.
fire(coro)
¶
Run coro in the background, held until it ends so it is not collected mid-send.
agent_adapter.voice.pcm¶
16-bit PCM helpers with no dependencies: streaming resampling and G.711 μ-law.
audioop left the standard library in Python 3.13, so these are written out here.
Resampler
¶
Changes the sample rate of a PCM16 mono stream, chunk by chunk, with no seams.
Linear interpolation; when the rate drops by 2x or more, a box filter first removes most of what would alias. Good enough for speech models and phone audio.
bytes_of(samples)
¶
Native 16-bit integers as little-endian PCM.
rms(pcm)
¶
Root-mean-square level as a fraction of full scale (0 for no audio).
samples_of(pcm)
¶
16-bit little-endian PCM as native integers (a trailing odd byte is dropped).
ulaw_decode(data)
¶
G.711 μ-law to PCM16.
ulaw_encode(pcm)
¶
PCM16 to G.711 μ-law, one byte per sample.
agent_adapter.voice.turns¶
Turn detection from raw audio: when the caller starts talking and when they have finished.
TurnDetector is the port; Silero VAD, Smart Turn or your STT's own end-of-utterance
signal plug in behind it. SilenceTurns is a dependency-free default for 16-bit mono PCM:
loud for start_ms starts a turn, quiet for end_ms ends it.
TurnEvent = Literal['started', 'ended']
module-attribute
¶
SilenceTurns
¶
Energy-based turns over 16-bit little-endian mono PCM.
threshold is the RMS level, as a fraction of full scale, above which a frame is
speech. Short noises under start_ms never start a turn, and pauses under end_ms
never end one.
TurnDetector
¶
Bases: Protocol
Takes audio frames as they arrive; returns the turn events each frame caused.
TurnTimer
¶
Turns from per-frame speech decisions: start_ms of speech starts one, end_ms of
quiet ends it. Any VAD (an energy level, Silero, your STT's flags) can drive it.
step(speech, ms)
¶
Advance by one frame of ms milliseconds.
pcm16(data)
¶
16-bit little-endian PCM as native integers.
Also exported here, documented under the canonical path: EchoGuard.
agent_adapter.voice.endpoint¶
EagerTurns: end a turn sooner when the caller's words look finished, later when not.
No model and no provider feature: it reads the early transcript VoiceSession already
makes (any STT) and moves the end-of-turn wait of the detector it wraps.
EagerTurns
¶
Wraps a detector with a TurnTimer (SilenceTurns, SileroTurns, or one SmartTurn
wraps). After hint(text) the turn ends done_ms after speech when the words look
finished, or waits hold_ms when they trail off ("and", "the", a comma).
It learns each caller: mid-turn pauses and turns it ended too soon (the caller went on
within regret_ms) raise the wait for finished-looking words, up to the detector's own.
hint(text)
¶
The transcript so far; moves this turn's end. Returns looks_done(text).
looks_done(text)
¶
True if text reads as a finished turn, False if it trails off, None if unsure.
agent_adapter.voice.config¶
VoiceConfig: a whole VoxCat call from one YAML file or dict.
stt: deepgram # a name below, or "package.module:Class"
tts: {use: cartesia, voice: "a0e99841-438c-4a64-b679-ae501e7d6091"}
model: {use: openai, model: gpt-4.1-mini}
instructions: You book tables. Keep answers short.
tools: mypkg.tools:registry # optional
turns: {end_ms: 600, eager: true} # eager: EagerTurns on SilenceTurns
early_ms: 100
speed: {first_words: 4, acks: ["One moment."]}
Keys stay in the environment: a secret-looking option (api_key, token) is refused.
MODEL_NAMES = {'openai': 'agent_adapter.models:OpenAIModel', 'anthropic': 'agent_adapter.models:AnthropicModel', 'gemini': 'agent_adapter.models:GeminiModel'}
module-attribute
¶
STT_NAMES = {'deepgram': _V + 'deepgram:DeepgramSTT', 'deepgram-flux': _V + 'deepgram:DeepgramFluxSTT', 'openai': _V + 'cloud:OpenAISTT', 'cartesia': _V + 'cartesia:CartesiaInkSTT', 'elevenlabs': _V + 'elevenlabs:ElevenLabsScribeSTT', 'assemblyai': _V + 'assemblyai:AssemblyAISTT', 'soniox': _V + 'soniox:SonioxSTT', 'moonshine': _V + 'moonshine:MoonshineSTT', 'whisper': _V + 'whisper:FasterWhisperSTT', 'parakeet': _V + 'parakeet:ParakeetSTT', 'kyutai': _V + 'kyutai:KyutaiSTT'}
module-attribute
¶
TTS_NAMES = {'cartesia': _V + 'cartesia:CartesiaLiveTTS', 'elevenlabs': _V + 'elevenlabs:ElevenLabsLiveTTS', 'openai': _V + 'cloud:OpenAITTS', 'fish': _V + 'fish:FishAudioTTS', 'rime': _V + 'rime:RimeTTS', 'moonshine': _V + 'moonshine:MoonshineTTS', 'kyutai': _V + 'kyutai:KyutaiTTS', 'chatterbox': _V + 'chatterbox_tts:ChatterboxTTS', 'orpheus': _V + 'orpheus:OrpheusTTS'}
module-attribute
¶
Part
dataclass
¶
One piece of the call: use (a short name or "module:Name") and its options.
VoiceConfig
dataclass
¶
build(transport, executor=None)
¶
The session and agent for one call on transport; run session.run(agent).
Pass an executor with an approval port for confirm=True tools; the default one
has none, so those calls are refused.
load(source)
classmethod
¶
From a dict, or a .yaml/.yml/.json file. Raises ConfigError on a bad setting.
load(ref)
¶
package.module:Name -> the object.
agent_adapter.voice.silero¶
Turns from Silero VAD, run locally with ONNX Runtime: speech, not just loudness.
Needs pip install "agent-adapter[vad]" and the model file: silero_vad.onnx from the
snakers4/silero-vad release (MIT), or the copy Pipecat ships under pipecat/audio/vad/data.
SileroTurns
¶
A TurnDetector for 16-bit mono PCM at 16 or 8 kHz.
A window with speech probability >= threshold counts as speech; once speaking, it takes
a drop below threshold - 0.15 to count as quiet, so trailing syllables don't end a turn.
agent_adapter.voice.smart_turn¶
End of turn from what the caller said, not just how long they paused: Smart Turn v3.
A pause after "I'd like to book a seat for" is not the end of a turn. SmartTurn wraps any
TurnDetector; when it hears a pause, the Smart Turn model looks at the turn's audio and the
turn ends only if the model says the caller has finished (or after max_wait_ms more quiet).
Needs pip install "agent-adapter[vad]" and smart-turn-v3.*.onnx from the
pipecat-ai/smart-turn-v3 model page (BSD-2-Clause), or the copy Pipecat ships under
pipecat/audio/turn/smart_turn/data.
RATE = 16000
module-attribute
¶
SmartTurn
¶
A TurnDetector for 16 kHz 16-bit mono PCM: detector hears pauses, the model
decides whether a pause ends the turn.
probability(audio)
¶
How sure the model is that a turn ending with audio (float32, 16 kHz) is done.
log_mel(audio, np)
¶
Whisper's log-mel features of 8 s of 16 kHz float32 audio: an (80, 800) array.
agent_adapter.voice.realtime¶
Speech-to-speech models (OpenAI Realtime, Gemini Live) with the executor's safety.
The model hears and speaks on its own socket; RealtimeAgent runs the tools it calls.
Speech is cheap, effects are not: when the provider cancels calls because the caller cut
in, read-only tools stop at once, but a tool that changes things finishes and the model is
told it is done. Repeats within the window return the first result. A confirm tool asks
aloud and only the caller's own transcribed words can approve it, and only words they began
after the provider reported the prompt finished (PromptDone). Limit: the agent arms on the
first PromptDone after it sends the prompt, so a codec that cannot tell its lines apart may
arm a little early; the answer must still start after that.
A codec turns one provider's JSON messages into the neutral events below and back.
RealtimeEvent = ToolCalled | SpeechStarted | CallerSaid | CallsCancelled | PromptDone
module-attribute
¶
CallerSaid
dataclass
¶
CallsCancelled
dataclass
¶
PromptDone
dataclass
¶
The model finished speaking a line the app asked it to say.
RealtimeAgent
¶
Runs a realtime model's tool calls through executor over a socket you opened.
socket is any TextSocket (a websockets connection works as it is) already
connected and authenticated to the provider. With minute_item ("provider:name"),
run bills the call's minutes when it ends.
run()
async
¶
Send the setup, then handle the provider's messages until the socket closes.
RealtimeCodec
¶
Bases: Protocol
One provider's wire format: JSON objects in and out.
A codec that emits PromptDone sets prompt_done_events = True.
SpeechStarted
dataclass
¶
The caller started talking (over the model, or a new utterance).
ToolCalled
dataclass
¶
agent_adapter.voice.openai_realtime¶
OpenAI Realtime (GA) wire format for RealtimeAgent.
Connect with websockets.connect("wss://api.openai.com/v1/realtime?model=gpt-realtime",
additional_headers={"Authorization": f"Bearer {key}"}) and pass the connection in.
The server's own VAD stays on: it cancels the model's speech when the caller cuts in, and
this side keeps the tools safe. Each line the app asks for carries metadata.aa_say, so its
response.done tells the agent when an approval prompt has played.
OpenAIRealtime
¶
RealtimeCodec for the OpenAI Realtime API; session adds fields to session.update.
agent_adapter.voice.gemini_live¶
Gemini Live (BidiGenerateContent) wire format for RealtimeAgent.
Connect with websockets.connect("wss://generativelanguage.googleapis.com/ws/google.ai.
generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=...") and pass the
connection in. Gemini cancels pending calls when the caller cuts in (toolCallCancellation);
read-only ones stop, and one that changes things finishes and is reported as done.
Gemini sends the caller's words in pieces; they count as one answer once the model starts
its reply or the turn ends, and the first piece marks when the caller began. Gemini has no
"say exactly" message, so an approval prompt is a text turn asking the model to read it; the
next turnComplete counts as the prompt having played (Gemini does not tag turns).
GeminiLive
¶
RealtimeCodec for the Gemini Live API; setup adds fields to the setup message.
agent_adapter.voice.testing¶
Offline fakes for voice agents: test interruptions and approvals with no audio and no keys.
MemoryTransport
¶
An AudioTransport a test drives: speak(pcm) sends caller audio, play(ms) plays
ms of the agent's audio into heard, and hang_up() ends the call.
RecordingSpeaker
¶
A Speaker that records what it said; with hold=True each line waits for release.
release(lines=1)
¶
Let lines more held lines play to the end.
ScriptedSTT
¶
An STT that returns the given transcripts in order, then empty strings.
WordTTS
¶
A TTS with ms_per_word of tone per word, in one chunk per word.
settle(rounds=50)
async
¶
Let every ready task run (a few event-loop turns).
silence(ms, *, rate=16000)
¶
tone(ms, *, level=8000, rate=16000)
¶
ms of a loud square wave as PCM16: a stand-in for speech.