Skip to content

Voice agents

Turns, safe interruptions and spoken approval around the executor, our own audio pipeline and transports, plus offline fakes. Guide: Build a voice agent.

agent_adapter.voice

Voice agents: safe interruptions, spoken approval and one ledger, on any audio stack.

VoiceSession is our own pipeline (audio, turns, speech-to-text, text-to-speech); the transports are LocalAudio, AudioSocket, TwilioStream, TelnyxStream, PlivoStream, VonageStream and WebRTCAudio.

YES_WORDS = frozenset({'yes', 'yeah', 'yep', 'yup', 'sure', 'ok', 'okay', 'confirm', 'confirmed', 'correct', 'right', 'absolutely', 'definitely', 'please', 'go', 'ahead', 'do', 'it', "that's", 'thats', 'fine', 'affirmative', 'proceed'}) module-attribute

AudioSocket

An AudioTransport over a BinarySocket.

AudioTransport

Bases: Protocol

Where the call's audio comes from and goes to: a sound card, a socket, a phone line.

frames() yields the caller's audio as 16-bit mono PCM at in_rate. start(buffer) makes the transport drain buffer (PCM16 at out_rate) in real time.

CallObserver

Override what you need; every method is optional.

agent_audio is called as the agent's audio plays, possibly on the sound card's thread: keep it quick. caller_said returning True keeps that text from the agent.

agent_audio(pcm)

The agent's audio as it plays, PCM16 mono at the transport's out_rate.

caller_audio(pcm)

The caller's audio: 16 kHz PCM16 mono, after the session's filters.

caller_said(text)

A final transcript of the caller; return True to keep it from the agent.

ended() async

The call ended.

started(session)

The call started.

CallRecorder

Bases: CallObserver

Writes path when the call ends: 16 kHz, 16-bit, two channels.

Each side is placed at the time it was heard, so the two line up. Recording stops after max_minutes. Tell callers they are recorded where the law asks you to.

CallableSTT

An STT from fn(pcm, sample_rate) -> str; a sync function runs in a thread.

CallableSpeaker

A Speaker from plain functions: say(text) and an optional stop(), sync or async.

CallableTTS

A TTS from fn(text) returning PCM16 bytes, or an iterable or async iterable of chunks; a sync function runs in a thread.

EchoGuard dataclass

When speech counts as a barge-in while the agent talks on speakers with no echo cancelling: loud_ms of audio at level RMS or above (a fraction of full scale; the agent's voice through the mic is quieter than a caller at it), or speech_ms of speech.

Tune level on your hardware: above the agent's echo, below your voice.

EventPump

Hands caller events to a VoiceAgent one at a time, in order.

Transport callbacks must return at once, but a cut can wait for a booking to finish and an approval waits for the caller's next words; the pump queues events so the transport never waits (and never deadlocks on its own next message) and order is kept.

aclose() async

Stop handing events over (queued ones are dropped).

drop_speculation()

The caller went on after the early transcript.

idle() async

Wait until every queued event has been handed over.

said(text, timing=None)

A final transcript; blank text is ignored.

speculate(text)

An early transcript: the agent may start its reply (VoiceAgent.speculate).

started_speaking()

The caller started talking (VAD).

turn_ended()

The caller finished a turn of real speech (VoiceAgent.turn_ended).

IdlePolicy

Bases: CallObserver

When nobody has spoken for after_s, ask prompt; after hang_up_after unanswered prompts, say goodbye and hang up. Any caller words start the count again.

LocalAudio

Microphone in at in_rate, speakers out at out_rate, in frame_ms blocks.

The speaker callback pulls straight from the playout buffer, so a pause or a cut is heard within one block.

hang_up()

End the call: frames() stops.

NoiseGate

Learns the steady background noise in each frequency band and turns it down by up to reduction_db, leaving speech alone. 16 kHz PCM16 in and out, same length, about frame / 16 ms later. Steady noise (fans, hum, road) it handles; other voices it cannot.

Any callable pcm -> pcm works as a filter, so a stronger denoiser plugs in the same way.

Player

Speaks lines through a PlayoutBuffer and knows how much of each the caller heard.

prepare(text) starts synthesis early, so the next line is ready when this one ends. After stop(), heard_of_cut holds the words of the cut line that played (estimated from the share of its audio that played), so the history keeps only what was heard. The last cache_lines short lines are kept as audio, so a repeated line ("One moment.") plays at once; 0 turns that off. text_hooks rewrite each line just before synthesis (a SpokenForm from agent_adapter.voice.quality); heard_of_cut stays in the original words.

speaking property

Audio is queued or a line is being spoken.

pause()

Hold playback (the caller may be cutting in).

prepare(text)

Start synthesising text now; say(text) will use it.

say(text) async

Speak text; return when it has played, or was cut.

stop() async

Cut now: drop queued audio and pending synthesis; record what was heard.

warm(lines) async

Synthesise lines into the cache now, so they play at once later.

PlivoStream

Bases: PhoneStream

An AudioTransport over a Plivo Audio Streaming WebSocket (any TextSocket).

content_type is one of Plivo's: audio/x-l16;rate=8000 (Plivo's default), audio/x-l16;rate=16000 or audio/x-mulaw;rate=8000. The identity fields come from the first start message only; trust them only if the host verified X-Plivo-Signature-V3 on the upgrade.

STT

Bases: Protocol

Speech-to-text for one finished turn of 16-bit mono PCM.

Speaker

Bases: Protocol

Speaks text aloud: LiveKit, Pipecat, a phone line, a browser, any TTS.

say(text) async

Speak text; return when it has played (or was stopped).

stop() async

Cut the audio now (the caller started talking).

TTS

Bases: Protocol

Text-to-speech: 16-bit mono PCM chunks at sample_rate, as soon as each is ready.

TelnyxStream

Bases: PhoneStream

An AudioTransport over a Telnyx Media Streaming WebSocket (any TextSocket).

The identity fields come from the first start message only; trust them only if the host verified the telnyx-signature-ed25519 header on the upgrade.

ThinkingSound

Bases: CallObserver

Loops pcm (16-bit mono at sample_rate) at gain once a turn has gone delay_s with nothing to say, and stops as soon as the agent speaks or the caller talks.

It is queued chunk_ms at a time, so speech waits at most that long behind it. It never counts as the agent speaking: no cut, no echo guard, no voice-to-voice time.

from_wav(path, **options) classmethod

Load a 16-bit mono WAV file.

ticks(*, every_s=0.5, **options) classmethod

A soft tick, like a clock, built in so no file is needed.

TurnTiming dataclass

When each step of one caller turn happened, in seconds on the monotonic clock.

stages()

Each stage that happened, in ms after the caller's last speech (negative if before).

TwilioCalls

A small async client for the Calls resource; client takes your httpx.AsyncClient.

aclose() async

Close the HTTP client this object made (one you passed in stays open).

call(to, from_, *, twiml=None, url=None, status_callback=None, timeout_s=None) async

Place an outbound call that runs twiml (or fetches it from url) on answer.

hang_up(call_sid) async

End a live call.

transfer(call_sid, to, *, caller_id=None) async

Hand a live call to a person at to; the media stream ends as Twilio dials.

update(call_sid, **form) async

Change a live call (Twiml, Url or Status, as Twilio names them).

TwilioStream

Bases: PhoneStream

An AudioTransport over a Twilio Media Streams WebSocket (any TextSocket).

The identity fields (call_sid, stream_sid, parameters) come from the first start message only; trust them only if the host verified X-Twilio-Signature on the upgrade.

hang_up() async

End the call: through the Calls API when calls is set, and end frames().

With no calls, only our side ends; Twilio then runs the TwiML after <Connect>, which hangs up when there is none.

transfer(to, *, caller_id=None) async

Hand the caller to a person at to; Twilio ends this stream as it dials.

VideoFeed

The latest frame of one video source, for look_tool. Start sources in async code.

latest property

The latest kept frame, or None.

age()

Seconds since the latest frame was kept, or None with no frame.

camera(device=None)

Read a local camera; device defaults to the first one this platform finds.

capture(file, *, format=None, options=None)

Read any FFmpeg input (a device, a file, a stream URL) through aiortc's MediaPlayer.

clear()

Forget the latest frame (the camera was switched off).

close() async

Stop every source this feed started.

follow(track)

Read a video track (aiortc MediaStreamTrack) until it ends.

jpeg()

The latest frame as JPEG, encoded once per frame; None with no frame.

poll(source)

Call source (sync or async) fps times a second; it returns a frame or None.

A sync source runs on the event loop, so keep it quick (a MuJoCo render is); give a slow one as an async function, e.g. one that awaits asyncio.to_thread.

push(frame)

Offer a frame; it is kept unless the last one was kept under 1/fps seconds ago.

screen(display=None)

Read the screen (Windows, macOS, X11; not Wayland).

VoiceAgent

A voice agent from a model, an executor and a speaker; works with any audio stack.

Feed it the caller's events: user_started_speaking() when they start talking and user_said(text) with each final transcript. Speech is cheap, effects are not: talking over the agent stops its speech and its thinking at once, but a tool that changes things (effects=) and has started finishes once and is reported in the next turn. Repeats of a finished call within dedupe_window_s return the first result. With minute_item ("provider:name"), close() bills the call's minutes at its unit price. With a VoiceApproval, a yes counts only if user_started_speaking() came after the question finished playing. With mute_during_tools, the caller cannot cut in while a tool runs (a VoiceSession drops that speech). speed adds speculative replies, acknowledgements, an answer cache and warm-up (Speed).

busy property

A turn is running: the model is thinking, a tool runs, or the reply plays.

interruptible property

Whether the caller may cut in now.

close() async

End the call: cut any turn, stop speaking and bill the call minutes (once).

drop_speculation()

The caller went on: drop the speculative reply.

meter(item, units)

Bill units of item ("provider:name", e.g. STT seconds) to this call's run.

say(text, *, interruptible=True, record=True)

Speak text now, outside a model turn: a greeting, a notice, a goodbye.

With record, what the caller heard of it joins the history. With interruptible=False, the caller cannot cut it (a VoiceSession drops that speech).

speculate(text)

Start replying to text, an early transcript, while the caller may still go on.

Kept if user_said brings the same words and nothing was said since; needs speed.

turn_ended()

The caller finished a turn of real speech; its transcript is on the way.

user_said(text, *, timing=None) async

A final transcript: an answer to a waiting question, or a new turn.

With speed, the stages of the reply are noted in timing.

user_started_speaking() async

The caller talks over the agent: stop speech now; cut the turn unless it asks yes/no.

wait() async

Wait until the current turn has finished speaking.

wait_spoken() async

Wait until everything queued to speak has played.

warm_up(providers=()) async

Get speed's connections, lines and prefetches ready; VoiceSession calls it.

VoiceApproval

Approval port for voice: asks the caller aloud, then waits for their own answer.

Each call binds its say under its run id (a VoiceAgent or RealtimeAgent does this), so many calls can share one executor: a question goes only to the caller whose run asked, and only that caller's words answer it. A run with no bound call is denied. One question is open per call at a time, and an answer counts only when the caller began speaking after the prompt finished playing.

Only text passed to answer (the caller's transcript) can approve; model text and tool output never reach it. Silence until timeout_s, or any answer that is not a clear yes, denies. prompts maps a tool name to its own prompt function. readback may put a line before the prompt, such as "Did you say 4,500?" for words the caller's speech-to-text was unsure of (UnsureWords.readback in agent_adapter.voice.quality); the one yes then answers both.

waiting property

True while any call has a question open.

answer(text, *, run_id=None, question=None)

Give the caller's final words to their open question; False when none is open.

With run_id, the words count only if question is the id armed_question gave when the caller began them; others are taken but dropped. True means the words belonged to the open question, used or not.

armed_question(run_id)

The id of the run's question once its prompt has played; None before that.

Read it when the caller starts speaking and hand it to answer with their words: speech that began before the question was fully asked never answers it.

bind(say, *, run_id=None)

Speak this run's prompts with say. With no run_id, say serves every run that has no call of its own: only for a process that handles one call at a time.

request(request) async

Ask the run's caller aloud and decide from their answer; anything unclear denies.

unbind(run_id=None, say=None)

Forget the run's call (when say is given, only if it is still the bound one); its open question is denied at once.

waiting_for(run_id)

True while the run's call has a question playing or waiting for its answer.

VoiceSession

Runs a VoiceAgent on an AudioTransport; give the agent session.speaker.

turns gets 16 kHz audio. confirm_ms of speech confirms a cut before the transcript is ready; backchannels are words that never cut the agent while it speaks.

Transcription starts early: a StreamingSTT hears the turn as it is spoken, and the final pass starts after early_ms of quiet, while the detector is still deciding whether the turn is over. If the caller goes on, that pass is dropped. With an agent Speed, the early transcript also starts the reply (VoiceAgent.speculate), first on the stream's interim words (a partial attribute) if it has them, and the agent's warm-up runs as the call starts. With EagerTurns, those words also move the turn's end.

Keypad digits from a transport with on_digit (TwilioStream) reach the agent as keypad_text, once keypad_wait_ms passes with no new digit, at once on #, or after max_digits. A turn that never pauses ends after max_turn_s.

filters transform the caller's 16 kHz audio first (a noise filter); observers (CallObserver) add call features such as CallRecorder, IdlePolicy and VoicemailDetector.

agent property

The agent this session runs, once run started.

caller_talking property

A caller turn is open.

end_after_reply()

Hang up once the agent's current reply has played (an end_call tool).

hang_up(*, after_speech=False) async

End the call; with after_speech, once the current reply has played.

run(agent) async

Run the call until the transport ends; then close the agent and the transport.

send_text(text)

A typed message from the caller, handled like a spoken turn.

VoicemailDetector

Bases: CallObserver

Listens to the first window_s of a call for a machine.

Two offline signals: a greeting that says what machines say (PHRASES, or your own classify(text)), and the beep (a steady tone of 400 to 2,500 Hz for beep_ms). Once the greeting is heard, the machine's words are kept from the agent. With message, the agent waits for the beep (or beep_wait_s of quiet after the greeting), speaks it and hangs up; with on_detect(session, reason), you decide.

VonageStream

Bases: PhoneStream

An AudioTransport over a Vonage Voice API WebSocket (any BinarySocket).

rate must match the NCCO endpoint's content-type. headers holds the first websocket:connected message's fields (your NCCO headers); trust them only if the host checked the Authorization header on the upgrade.

binary_socket(send, messages)

A BinarySocket from a send function and an async iterator of received messages.

end_call_tool(session, *, name='end_call', description='End the phone call once your reply has been spoken.')

A tool for the agent to hang up after its goodbye: registry.add(end_call_tool(session)).

is_yes(text, words=YES_WORDS)

True only when every word is a yes word: "yeah no", "yes but" and "" are all no.

look_tool(feed, name='look', *, description=None, max_age=None)

A read-only tool that returns the feed's latest frame as a JPEG image.

With max_age, a frame older than that many seconds counts as no frame: a video track that stopped sending (a camera switched off) stops being shown as the present.

agent_adapter.voice.session

VoiceSession: our own voice pipeline, from raw audio to a VoiceAgent and back.

transport -> turn detector -> speech-to-text -> VoiceAgent -> text-to-speech -> transport

When the caller starts talking, playback pauses within one audio frame. The turn is cut only once the speech is real: confirm_ms of speech, or words in the transcript. A cough, or an "uh-huh" while the agent speaks, resumes playback where it paused.

With an EchoGuard (speakers with no echo cancelling), speech that starts while the agent talks pauses nothing until it is loud enough or long enough; one that never is was the agent's own voice in the mic and is dropped.

While the agent asks a spoken yes/no question, only a turn that began after the question finished playing can answer it; earlier speech, an echo of the prompt included, is dropped. Speech that starts while the agent cannot be cut (VoiceAgent.interruptible) is dropped too.

BACKCHANNELS = frozenset({'uh', 'huh', 'uh-huh', 'mhm', 'mm', 'hmm', 'mm-hmm', 'mmhmm', 'hm', 'yeah', 'yep', 'yes'} | {'ok', 'okay', 'right', 'sure', 'got', 'it', 'i', 'see'}) module-attribute

RATE = 16000 module-attribute

Also exported here, documented under the canonical path: TurnTiming, VoiceSession.

agent_adapter.voice.timing

TurnTiming: when each stage of one caller turn happened, for traces and benchmarks.

STAGES = ('early', 'ended', 'transcribed', 'speculated', 'replied', 'acked', 'first_audio') module-attribute

event_data(timing)

Timings only: the transcript is the caller's words and stays out of traces.

Also exported here, documented under the canonical path: TurnTiming.

agent_adapter.voice.speed

Speed: the speed tricks of a VoiceAgent, and the per-call fast path that runs them.

FastPath

One call's speed state: the running speculation, the gate and the turn plans.

acked_early property

An acknowledgement played at the end of the turn, before its transcript.

turn(text, timing)

The task that answers text: a cached answer, the kept speculation, or a new turn.

turn_ended()

The caller finished a real turn: acknowledge now if the reply will be slow.

Speed

What a VoiceAgent does to answer sooner; each part is optional.

  • speculate: start the reply on the early transcript; kept only if the final one has the same words and nothing was said since, dropped if the caller goes on.
  • tools: the read-only tools that reply may run before the turn is final.
  • acks: short pre-rendered lines while a slow reply is on its way.
  • answers with tenant: replies to repeated questions, with their audio.
  • warmup: connections, lines and read-only prefetches at call start.
  • first_words: speak a reply's first clause once it has this many words, not its first whole sentence (None: sentence by sentence).

agent_adapter.voice.speculate

Speculative replies: start answering on the early transcript, keep it only if it was final.

A Speculation runs the agent's tool loop before the turn is over. Its text and progress lines are held back until the agent adopts it (the final transcript matched) and dropped if the caller goes on. Tools wait for adoption at the executor, in SpeculationGate, which runs before guards and approval: only tools named in SpeculativeTools that are declared effects="read_only" and have no confirm may run early, within their own rate limits.

SPECULATIVE_STAGE = 'voice_speculation' module-attribute

TOOL_ITEM = 'voxcat:speculative_tool' module-attribute

ReplyOut

Bases: Protocol

Where a turn's words go: the speech queue, or a tap in front of it.

Speculation

One reply started on the early transcript; its output waits until it is adopted.

ready property

Some reply text is already waiting.

adopt(out)

Keep it: replay the held output into out and pass the rest straight through.

bind()

Mark the running task as this speculation (for the gate); returns a reset token.

drop()

The caller went on: cancel it, once any early tool call has landed in the cache.

kept() async

Return once adopted; a dropped speculation is cancelled here.

matches(text, history)

The final transcript is this one, and nothing was said since it started.

SpeculationGate

Bases: Hook

Holds every tool call of an unadopted speculation, except allowed read-only ones.

It runs first among the executor's hooks, so nothing (guards, budget, approval) sees a held call until the turn is final; a dropped speculation is cancelled while it waits.

SpeculativeTools

Tools a speculative reply may run before the caller has finished: opt-in by name.

Each must be declared effects="read_only" with no confirm; the executor checks this on every call, not only here. At most max_per_turn run early per speculation and max_per_minute per call; past either, a call waits for the turn to be final. Each early call is billed as one unit of budget_item and traced as custom.voice.speculative_tool. Give them cache_ttl, so a dropped speculation's call is reused by the next one.

check(registry)

Raise ToolUsageError if a named tool is missing or may change something.

can_speculate(spec)

A tool declared read-only, with no confirm rule: it may run before the turn is final.

current()

same_words(text)

text reduced for an exact match: lower case, no punctuation, single spaces.

agent_adapter.voice.ack

Acknowledgements: a short pre-rendered line while a slow reply is on its way.

Acknowledgements

Plays a short line ("Mm-hm.") when the reply will be slow, and never before a fast one.

The agent learns how long its replies take to start (a moving average from expect_ms). When a turn is handed over and the reply is expected later than fast_ms, a line plays at once; otherwise it waits fast_ms and plays only if the reply has still not started. A reply that is already waiting (a kept speculation, a cached answer) gets none.

Lines play only once rendered at call start (VoiceAgent.warm_up, which a VoiceSession runs), so an acknowledgement is never slower than the reply it stands in for.

delay_ms(thought_ms)

How long to wait before acknowledging, thought_ms into the thinking.

learn(reply_ms)

A reply started reply_ms after its thinking began.

pick()

The next rendered line, in turn; None before any is rendered.

run(thought_ms, ready, say) async

Wait as planned, then say a line unless the reply is ready; returns the line.

agent_adapter.voice.answers

AnswerCache: replies to questions heard before, with their audio, so they play at once.

Answer dataclass

AnswerCache

Word-for-word repeats of a question get the same reply, audio included, for ttl_s.

Keys are the tenant plus the question's words (case and punctuation ignored), so two tenants never share an entry. A VoiceAgent with Speed(answers=..., tenant=...) stores a reply only when it needed no tool call and opened the call (it depends on the question and the instructions alone); put seeds known answers. Questions under min_words words ("yes", "the second one") depend on the call and are never stored.

Wrap the session's TTS with voice(tts): it keeps the audio of each stored answer and plays it back with no synthesis. VoiceAgent.warm_up renders seeded answers' audio.

forget(tenant=None)

Drop every entry, or one tenant's.

get(tenant, question)

The stored reply to question for tenant, or None.

has(tenant, question)

Whether question has a live answer for tenant (not counted as a hit).

lines()

Lines of live answers with no audio yet (rendered at call start).

put(tenant, question, reply)

Store reply for question; None when the question is too short to store.

voice(tts)

tts that replays stored answers' audio and keeps the audio of new ones.

agent_adapter.voice.warm

Warm-up at call start: connections, rendered lines and read-only prefetches, all at once.

WarmReport dataclass

How long each warm-up step took, in ms, and the steps that failed (error type).

Warmup

What to get ready when a call starts, so its first turn is as fast as the rest.

connections are objects with an async warm() (a provider's connection pool) or plain functions, sync or async (open a socket, send a HEAD request). The call's STT, TTS and model are warmed too when they have warm(). lines are rendered to audio now (a greeting); acknowledgements and cached answers are always rendered. tools are read-only calls, (name, arguments), run now through the executor: with cache_ttl on the tool, the turn that asks for one gets the cached result. Each step has timeout_s; a failure is logged and never stops the call.

check(registry)

Raise ToolUsageError if a prefetch tool is missing or may change something.

warm_up(agent, providers=()) async

Run every warm-up step of agent.speed concurrently; never raises.

agent_adapter.voice.media

The ports of our own voice pipeline: audio transports, speech-to-text and text-to-speech.

Every port is a small protocol, so any provider fits: a local model, a hosted API, or a function you already have (wrap it with CallableSTT or CallableTTS).

STTStream

Bases: Protocol

One turn transcribed while the caller talks; finish returns the final text.

feed(pcm)

Add audio; must return at once (do the work in a thread or a task).

StreamingSTT

Bases: STT, Protocol

An STT that can also transcribe as the caller speaks: at the end of the turn only the last moment is left to do, instead of the whole turn.

Also exported here, documented under the canonical path: AudioTransport, CallableSTT, CallableTTS, STT, TTS.

agent_adapter.voice.ports

What a voice agent needs from the outside: something that speaks text, and can stop.

Two optional speed ports sit beside the media ports: TurnSignals, a speech-to-text stream that says early when the provider thinks the turn ended, and StreamingTTS, text-to-speech that takes text while the model is still writing it.

TurnEvent = Literal['eager_end', 'resumed', 'end'] module-attribute

StreamingTTS

Bases: Protocol

A TTS that can also take text in pieces: open() starts one TTSStream.

TTSStream

Bases: Protocol

One reply spoken while its text is still being written.

push text as it comes (flush=True speaks what is buffered now, at the end of a clause), then end(). audio() yields 16-bit mono PCM at the TTS sample_rate. words fills with WordTimes when the provider sends timestamps.

TurnSignal dataclass

The provider's view of the turn: eager_end (probably done: start a reply now, but keep it unspoken), resumed (the caller went on: drop that reply) or end (done). text is the transcript of the turn so far.

TurnSignals

Bases: Protocol

An STTStream that also reports TurnSignals as they arrive.

The callback runs on the event loop and must return at once. A stream without this method still works; the session then decides the end of the turn on its own.

WordTime dataclass

A spoken word and when it plays, in seconds from the start of the stream's audio.

agent_adapter.voice.player

agent_adapter.voice.player re-exports, documented under the canonical path: Player.

agent_adapter.voice.playout

Audio waiting to play, drained in real time, so we always know what the caller heard.

A transport drains the PlayoutBuffer: a sound card callback, a WebRTC track, or a Pacer that sends it over a socket at the pace it plays. Because nothing is handed over before its time, a pause or a cut takes effect within one frame, and the count of bytes pulled for each line is what was heard of it.

Pacer

Sends a buffer's audio over a socket at the pace it plays, lead_ms ahead.

The far end buffers what it receives, so audio is never sent more than lead_ms early: a pause or a cut stops what the caller hears within that much.

PlayoutBuffer dataclass

Thread-safe queue of PCM16 mono audio at sample_rate, pulled in real time.

With confirm_playout, a line counts as played only when the transport calls confirm(id) (a phone network reporting a mark), not when its last byte is pulled.

busy property

Speech is queued (background sound does not count).

idle property

Nothing is queued to play.

clear()

Drop everything queued, unpause, and end every unfinished line as cut.

confirm(segment_id)

The far end reports segment_id played (with confirm_playout).

finish(segment)

No more audio for segment; it is done once all of it has played.

open(*, ambient=False)

Start a new line; write its audio, then finish it and await done.

pause()

Hold playback where it is (silence until resume).

pull(size, *, pad=True)

Up to size bytes to play now; padded with silence when pad (a sound card).

Paused or empty: silence, or b"" without pad.

Segment dataclass

One line of speech in the buffer.

heard property

The share of the written audio that played, 0 to 1.

fire(coro)

Run coro in the background, held until it ends so it is not collected mid-send.

agent_adapter.voice.pcm

16-bit PCM helpers with no dependencies: streaming resampling and G.711 μ-law.

audioop left the standard library in Python 3.13, so these are written out here.

Resampler

Changes the sample rate of a PCM16 mono stream, chunk by chunk, with no seams.

Linear interpolation; when the rate drops by 2x or more, a box filter first removes most of what would alias. Good enough for speech models and phone audio.

bytes_of(samples)

Native 16-bit integers as little-endian PCM.

rms(pcm)

Root-mean-square level as a fraction of full scale (0 for no audio).

samples_of(pcm)

16-bit little-endian PCM as native integers (a trailing odd byte is dropped).

ulaw_decode(data)

G.711 μ-law to PCM16.

ulaw_encode(pcm)

PCM16 to G.711 μ-law, one byte per sample.

agent_adapter.voice.turns

Turn detection from raw audio: when the caller starts talking and when they have finished.

TurnDetector is the port; Silero VAD, Smart Turn or your STT's own end-of-utterance signal plug in behind it. SilenceTurns is a dependency-free default for 16-bit mono PCM: loud for start_ms starts a turn, quiet for end_ms ends it.

TurnEvent = Literal['started', 'ended'] module-attribute

SilenceTurns

Energy-based turns over 16-bit little-endian mono PCM.

threshold is the RMS level, as a fraction of full scale, above which a frame is speech. Short noises under start_ms never start a turn, and pauses under end_ms never end one.

TurnDetector

Bases: Protocol

Takes audio frames as they arrive; returns the turn events each frame caused.

TurnTimer

Turns from per-frame speech decisions: start_ms of speech starts one, end_ms of quiet ends it. Any VAD (an energy level, Silero, your STT's flags) can drive it.

step(speech, ms)

Advance by one frame of ms milliseconds.

pcm16(data)

16-bit little-endian PCM as native integers.

Also exported here, documented under the canonical path: EchoGuard.

agent_adapter.voice.endpoint

EagerTurns: end a turn sooner when the caller's words look finished, later when not.

No model and no provider feature: it reads the early transcript VoiceSession already makes (any STT) and moves the end-of-turn wait of the detector it wraps.

EagerTurns

Wraps a detector with a TurnTimer (SilenceTurns, SileroTurns, or one SmartTurn wraps). After hint(text) the turn ends done_ms after speech when the words look finished, or waits hold_ms when they trail off ("and", "the", a comma).

It learns each caller: mid-turn pauses and turns it ended too soon (the caller went on within regret_ms) raise the wait for finished-looking words, up to the detector's own.

hint(text)

The transcript so far; moves this turn's end. Returns looks_done(text).

looks_done(text)

True if text reads as a finished turn, False if it trails off, None if unsure.

agent_adapter.voice.config

VoiceConfig: a whole VoxCat call from one YAML file or dict.

stt: deepgram                      # a name below, or "package.module:Class"
tts: {use: cartesia, voice: "a0e99841-438c-4a64-b679-ae501e7d6091"}
model: {use: openai, model: gpt-4.1-mini}
instructions: You book tables. Keep answers short.
tools: mypkg.tools:registry        # optional
turns: {end_ms: 600, eager: true}  # eager: EagerTurns on SilenceTurns
early_ms: 100
speed: {first_words: 4, acks: ["One moment."]}

Keys stay in the environment: a secret-looking option (api_key, token) is refused.

MODEL_NAMES = {'openai': 'agent_adapter.models:OpenAIModel', 'anthropic': 'agent_adapter.models:AnthropicModel', 'gemini': 'agent_adapter.models:GeminiModel'} module-attribute

STT_NAMES = {'deepgram': _V + 'deepgram:DeepgramSTT', 'deepgram-flux': _V + 'deepgram:DeepgramFluxSTT', 'openai': _V + 'cloud:OpenAISTT', 'cartesia': _V + 'cartesia:CartesiaInkSTT', 'elevenlabs': _V + 'elevenlabs:ElevenLabsScribeSTT', 'assemblyai': _V + 'assemblyai:AssemblyAISTT', 'soniox': _V + 'soniox:SonioxSTT', 'moonshine': _V + 'moonshine:MoonshineSTT', 'whisper': _V + 'whisper:FasterWhisperSTT', 'parakeet': _V + 'parakeet:ParakeetSTT', 'kyutai': _V + 'kyutai:KyutaiSTT'} module-attribute

TTS_NAMES = {'cartesia': _V + 'cartesia:CartesiaLiveTTS', 'elevenlabs': _V + 'elevenlabs:ElevenLabsLiveTTS', 'openai': _V + 'cloud:OpenAITTS', 'fish': _V + 'fish:FishAudioTTS', 'rime': _V + 'rime:RimeTTS', 'moonshine': _V + 'moonshine:MoonshineTTS', 'kyutai': _V + 'kyutai:KyutaiTTS', 'chatterbox': _V + 'chatterbox_tts:ChatterboxTTS', 'orpheus': _V + 'orpheus:OrpheusTTS'} module-attribute

Part dataclass

One piece of the call: use (a short name or "module:Name") and its options.

VoiceConfig dataclass

build(transport, executor=None)

The session and agent for one call on transport; run session.run(agent).

Pass an executor with an approval port for confirm=True tools; the default one has none, so those calls are refused.

load(source) classmethod

From a dict, or a .yaml/.yml/.json file. Raises ConfigError on a bad setting.

load(ref)

package.module:Name -> the object.

agent_adapter.voice.silero

Turns from Silero VAD, run locally with ONNX Runtime: speech, not just loudness.

Needs pip install "agent-adapter[vad]" and the model file: silero_vad.onnx from the snakers4/silero-vad release (MIT), or the copy Pipecat ships under pipecat/audio/vad/data.

SileroTurns

A TurnDetector for 16-bit mono PCM at 16 or 8 kHz.

A window with speech probability >= threshold counts as speech; once speaking, it takes a drop below threshold - 0.15 to count as quiet, so trailing syllables don't end a turn.

probability(window)

Speech probability of one window of float32 samples in [-1, 1].

reset()

Forget the model's memory of earlier audio.

agent_adapter.voice.smart_turn

End of turn from what the caller said, not just how long they paused: Smart Turn v3.

A pause after "I'd like to book a seat for" is not the end of a turn. SmartTurn wraps any TurnDetector; when it hears a pause, the Smart Turn model looks at the turn's audio and the turn ends only if the model says the caller has finished (or after max_wait_ms more quiet).

Needs pip install "agent-adapter[vad]" and smart-turn-v3.*.onnx from the pipecat-ai/smart-turn-v3 model page (BSD-2-Clause), or the copy Pipecat ships under pipecat/audio/turn/smart_turn/data.

RATE = 16000 module-attribute

SmartTurn

A TurnDetector for 16 kHz 16-bit mono PCM: detector hears pauses, the model decides whether a pause ends the turn.

probability(audio)

How sure the model is that a turn ending with audio (float32, 16 kHz) is done.

log_mel(audio, np)

Whisper's log-mel features of 8 s of 16 kHz float32 audio: an (80, 800) array.

agent_adapter.voice.realtime

Speech-to-speech models (OpenAI Realtime, Gemini Live) with the executor's safety.

The model hears and speaks on its own socket; RealtimeAgent runs the tools it calls. Speech is cheap, effects are not: when the provider cancels calls because the caller cut in, read-only tools stop at once, but a tool that changes things finishes and the model is told it is done. Repeats within the window return the first result. A confirm tool asks aloud and only the caller's own transcribed words can approve it, and only words they began after the provider reported the prompt finished (PromptDone). Limit: the agent arms on the first PromptDone after it sends the prompt, so a codec that cannot tell its lines apart may arm a little early; the answer must still start after that.

A codec turns one provider's JSON messages into the neutral events below and back.

RealtimeEvent = ToolCalled | SpeechStarted | CallerSaid | CallsCancelled | PromptDone module-attribute

CallerSaid dataclass

CallsCancelled dataclass

PromptDone dataclass

The model finished speaking a line the app asked it to say.

RealtimeAgent

Runs a realtime model's tool calls through executor over a socket you opened.

socket is any TextSocket (a websockets connection works as it is) already connected and authenticated to the provider. With minute_item ("provider:name"), run bills the call's minutes when it ends.

run() async

Send the setup, then handle the provider's messages until the socket closes.

RealtimeCodec

Bases: Protocol

One provider's wire format: JSON objects in and out.

A codec that emits PromptDone sets prompt_done_events = True.

SpeechStarted dataclass

The caller started talking (over the model, or a new utterance).

ToolCalled dataclass

agent_adapter.voice.openai_realtime

OpenAI Realtime (GA) wire format for RealtimeAgent.

Connect with websockets.connect("wss://api.openai.com/v1/realtime?model=gpt-realtime", additional_headers={"Authorization": f"Bearer {key}"}) and pass the connection in. The server's own VAD stays on: it cancels the model's speech when the caller cuts in, and this side keeps the tools safe. Each line the app asks for carries metadata.aa_say, so its response.done tells the agent when an approval prompt has played.

OpenAIRealtime

RealtimeCodec for the OpenAI Realtime API; session adds fields to session.update.

agent_adapter.voice.gemini_live

Gemini Live (BidiGenerateContent) wire format for RealtimeAgent.

Connect with websockets.connect("wss://generativelanguage.googleapis.com/ws/google.ai. generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=...") and pass the connection in. Gemini cancels pending calls when the caller cuts in (toolCallCancellation); read-only ones stop, and one that changes things finishes and is reported as done.

Gemini sends the caller's words in pieces; they count as one answer once the model starts its reply or the turn ends, and the first piece marks when the caller began. Gemini has no "say exactly" message, so an approval prompt is a text turn asking the model to read it; the next turnComplete counts as the prompt having played (Gemini does not tag turns).

GeminiLive

RealtimeCodec for the Gemini Live API; setup adds fields to the setup message.

agent_adapter.voice.testing

Offline fakes for voice agents: test interruptions and approvals with no audio and no keys.

MemoryTransport

An AudioTransport a test drives: speak(pcm) sends caller audio, play(ms) plays ms of the agent's audio into heard, and hang_up() ends the call.

RecordingSpeaker

A Speaker that records what it said; with hold=True each line waits for release.

release(lines=1)

Let lines more held lines play to the end.

ScriptedSTT

An STT that returns the given transcripts in order, then empty strings.

WordTTS

A TTS with ms_per_word of tone per word, in one chunk per word.

settle(rounds=50) async

Let every ready task run (a few event-loop turns).

silence(ms, *, rate=16000)

tone(ms, *, level=8000, rate=16000)

ms of a loud square wave as PCM16: a stand-in for speech.