Skip to content

Voice calls and transports

Transports (sound card, WebSocket, phone networks, WebRTC), hosted and local speech, and call features such as recording, idle prompts and voicemail. Guide: Voice call features.

agent_adapter.voice.observers

agent_adapter.voice.observers re-exports, documented under the canonical path: CallObserver.

agent_adapter.voice.ending

agent_adapter.voice.ending re-exports, documented under the canonical path: IdlePolicy, end_call_tool.

agent_adapter.voice.recorder

agent_adapter.voice.recorder re-exports, documented under the canonical path: CallRecorder.

agent_adapter.voice.ambience

agent_adapter.voice.ambience re-exports, documented under the canonical path: ThinkingSound.

agent_adapter.voice.voicemail

VoicemailDetector: tell an answering machine from a person on an outbound call, offline.

PHRASES = ('leave a message', 'leave your message', 'after the tone', 'after the beep', 'not available', 'unavailable', "can't take your call", 'cannot take your call', "can't come to the phone", 'voicemail', 'voice mail', 'mailbox', 'record your message') module-attribute

tone_of(pcm, min_level=0.02)

The frequency of a steady tone in pcm (16 kHz), or 0.0 if it is not one.

A pure tone crosses zero at even intervals; speech and noise do not.

Also exported here, documented under the canonical path: VoicemailDetector.

agent_adapter.voice.denoise

agent_adapter.voice.denoise re-exports, documented under the canonical path: NoiseGate.

agent_adapter.voice.quality

VoxCat quality checks: fast, model-free guards on a call's audio and words.

Each check is pure Python and costs well under a millisecond a frame or a sentence: InputCheck (the caller's audio), OutputCheck (the agent's audio), KnownNames (fix misheard names), UnsureWords (read back unsure numbers, dates and names before a confirm=True tool) and SpokenForm (money, numbers and dates as a voice says them).

CheckedSTT

Bases: CallObserver

An STT that runs fixes (such as KnownNames) on every transcript and gives the words to unsure. Make one with check_stt, and pass it as an observer too, so the words land under the call's run id: VoiceSession(stt=checked, observers=[checked]).

Word confidence comes from the STT, or its stream, setting last_words (a list of Word or provider word dicts) with each transcript; without it every word counts as sure, though a name the fixes guessed is still read back.

check(text, words=None)

Run the fixes on text and note its words; returns the fixed text.

Currency

Bases: NamedTuple

How to say an amount: "rupee", "rupees", "paisa", "paise", read in lakh and crore.

FrameStats dataclass

What one frame of audio looks like.

InputCheck

Bases: CallObserver

Watches the caller's audio for clipping, dropouts, a high noise floor or a voice too quiet to hear. Pass it as both a filter and an observer: VoiceSession(filters=[check], observers=[check]).

As a filter, frames below gate_db become silence (hiss stops waking the turn detector), holding open hold_ms after speech so word endings stay. The noise floor is the quietest of the last floor_frames frames. A turn has a problem when clip_share of its sound clipped, its dropouts (exact silence for under max_gap_ms with speech on both sides) add up to dropout_ms, its loudest speech stays under quiet_db, or the floor sits above noisy_db. After such a turn it keeps the transcript from the agent and says repeat_prompt instead, at most max_repeats times in a row; ask_again=False only measures. Each frame costs well under a millisecond, in pure Python.

__call__(pcm)

The filter: measure the frame, then gate it.

feed(pcm)

Measure one 16 kHz frame and add it to the current turn.

judge()

Close the current turn: name its problems and start a new one.

Issue dataclass

One problem heard in the agent's audio.

KnownNames

Fixes words that sound like a known name: "I'm Rahool from Acmy Corp" -> "I'm Rahul from Acme Corp", for KnownNames(["Rahul", "Acme Corp"]).

A word changes when its spelling is at least cutoff alike (1 minus edit distance over length), or sound_cutoff alike and it sounds the same. Words shorter than min_len and common English words never change; exact matches only get the name's capitals. last_fixes lists the guesses of the latest call; guessed names deserve a read-back.

fix(text)

text with misheard names replaced; last_fixes says what changed.

NameFix dataclass

One change: what was heard, the name it became, and how close they were (0 to 1).

OutputCheck

Bases: CallObserver

Checks the agent's audio as it plays: a jump of click_jump (share of full scale) between two samples is a click, clip_share of a chunk at full scale is clipping, and silence_s of near-silent audio inside a line is dead air from a broken voice.

It runs on the playout thread, so it only counts: counts, the last keep issues, and on_issue(issue) if given (keep that quick too). At the end of the call the counts go to the trace as a custom.voice_quality event. Pure Python, well under a millisecond a chunk.

feed(pcm)

Check one chunk of the agent's audio; returns the issues it found.

SpokenForm

Bases: CallObserver

Rewrites text before text-to-speech: "Rs 4,50,000" -> "four lakh fifty thousand rupees".

Handles money (rupees with lakh and crore, dollars, euros, pounds), plain and decimal numbers, ranges, percentages, ordinals, dates, times, phone numbers (digit by digit), units and short forms. locale="en-IN" reads plain numbers in lakh and crore and slash dates day first; "en-US" reads millions and month first. pronounce maps a word (case sensitive) to how to say it: {"VoxCat": "vox cat", "SQL": "sequel"}. currencies adds symbols or codes: {"AED": Currency("dirham", "dirhams", "fils", "fils")}; a code of capital letters also reads after the amount ("30 USD").

Call it on a string, or pass it as a VoiceSession observer: it then rewrites every line the session's player speaks. Pure Python; well under a millisecond a sentence.

TurnAudio dataclass

The caller's audio since the last transcript, and what was wrong with it.

Unsure dataclass

A number, date or name heard with low confidence.

UnsureWords

Keeps the low-confidence numbers, dates and names of each call's last turns turns, and reads them back before a confirm=True tool runs: "Did you say 4,500?".

Feed it what the speech-to-text heard with hear(words, run_id=...) (a check_stt wrapper does this when the STT reports word confidence). Give readback to the spoken approval: VoiceApproval(readback=unsure.readback). A word below threshold counts; with match_arguments, only the ones that show up in the tool's arguments are read back. Each is read back once. Words with no confidence are taken as sure.

hear(heard, *, run_id=None, guessed=())

Note one turn's words; guessed are (name, score) a names fix put in. Returns the unsure items found.

readback(request)

The read-back line for this approval, or None when nothing needs one.

unsure(run_id=None)

The unsure items still waiting for a read-back, oldest first.

Word dataclass

One heard word and the speech-to-text's confidence in it (0 to 1; None if not given).

check_stt(stt, *, fixes=(), unsure=None)

Wrap stt so its transcripts get fixes and unsure sees its words; a streaming STT stays streaming.

measure(pcm, *, clip_level=32000, previous=None)

Measure one frame; previous is the last sample before it, so a click at the seam counts.

number_words(n, *, indian=False)

4500 -> "four thousand five hundred"; with indian, 450000 -> "four lakh fifty thousand".

agent_adapter.voice.local

agent_adapter.voice.local re-exports, documented under the canonical path: LocalAudio.

agent_adapter.voice.audio_socket

AudioSocket: raw audio over one WebSocket, for browsers, apps and phone gateways.

From the client Meaning
binary message the caller's audio: 16-bit little-endian mono PCM at in_rate
{"type": "hangup"} the call is over
To the client Meaning
binary message the agent's audio: PCM16 at out_rate; play it as it arrives
{"type": "clear"} drop any audio not yet played (the caller cut in)

Audio goes out at the pace it plays, lead_ms ahead, so the client needs only a small jitter buffer and a cut stops what the caller hears within that much.

BinarySocket

Bases: Protocol

A WebSocket that sends and receives both text and binary messages.

A websockets connection is one as it is; wrap other servers with binary_socket.

starlette_messages(websocket) async

The messages of a Starlette or FastAPI WebSocket, text or binary, until it closes.

Also exported here, documented under the canonical path: AudioSocket, binary_socket.

agent_adapter.voice.socket

A voice agent over one WebSocket, at the transcript level: the client hears and speaks.

The client (a browser with the Web Speech API, a phone gateway, a native app) does speech to text and text to speech; this side decides. Messages are JSON text:

From the client Meaning
{"type": "speech_started"} the caller started talking
{"type": "transcript", "text": "...", "final": true} a transcript; only final ones count
{"type": "played", "id": 3} line 3 played to the end
{"type": "hangup"} the call is over
To the client Meaning
{"type": "say", "id": 3, "text": "..."} speak this line, then send played
{"type": "stop"} stop speaking now and drop queued lines

A line counts as played only after its played message, so the history after a cut holds exactly what the caller heard. A client that never acknowledges is trusted after played_timeout_s. Send speech_started as each utterance begins: an answer to a spoken yes/no question counts only if its speech started after the question played.

TextSocket

Bases: Protocol

A text WebSocket: send(text) and async iteration over received texts.

The websockets client and server connections are one as they are; wrap Starlette or FastAPI sockets with text_socket(ws.send_text, ws.iter_text()).

VoiceSocket

Runs a VoiceAgent over a TextSocket; give the agent voice_socket.speaker.

run(agent) async

Read the client until it hangs up or disconnects; then close the agent.

text_socket(send, messages)

A TextSocket from a send function and an async iterator of received texts.

agent_adapter.voice.twilio

agent_adapter.voice.twilio re-exports, documented under the canonical path: TwilioStream.

agent_adapter.voice.twilio_calls

TwilioCalls: place, transfer and end phone calls through Twilio's Calls REST API.

Credentials come from the constructor or the environment: TWILIO_ACCOUNT_SID with TWILIO_AUTH_TOKEN, or an API key (TWILIO_API_KEY and TWILIO_API_SECRET). They are sent only as HTTP Basic auth and never logged. Each call here has a side effect: when an agent may trigger one, wrap it in a tool marked confirm=True.

dial_twiml(to, *, caller_id=None)

TwiML that dials to (a phone number, or a sip: address) from the live call.

stream_twiml(url, parameters=None)

TwiML that connects a call to your TwilioStream WebSocket at url (wss://...).

Also exported here, documented under the canonical path: TwilioCalls.

agent_adapter.voice.telnyx

agent_adapter.voice.telnyx re-exports, documented under the canonical path: TelnyxStream.

agent_adapter.voice.plivo

agent_adapter.voice.plivo re-exports, documented under the canonical path: PlivoStream.

agent_adapter.voice.vonage

agent_adapter.voice.vonage re-exports, documented under the canonical path: VonageStream.

agent_adapter.voice.webrtc

WebRTCAudio: a browser call over WebRTC, peer to peer, with no media server (aiortc).

The browser sends an SDP offer to any HTTP route you own; accept(offer) returns the answer. Audio is Opus at 48 kHz both ways. The browser's own echo cancellation keeps the agent's voice out of the microphone, so speakers work, not only headphones.

With video=VideoFeed(...), the browser's camera or screen share feeds it (see agent_adapter.voice.video); without one, incoming video is declined.

Our answer never asks the browser for DTX: with DTX, the browser sends nothing in quiet, and the first syllable after a pause (a "Wait") arrives clipped.

OUT_RATE = 48000 module-attribute

WebRTCAudio

An AudioTransport for one browser call; accept the browser's offer first.

accept(offer) async

Answer the browser's {"sdp", "type"} offer; returns the answer to send back.

without_dtx(sdp)

sdp with any Opus usedtx=1 removed.

agent_adapter.voice.webrtc_page

A one-page browser client for WebRTCAudio: talk to your agent, show it a camera or screen.

webrtc_page(offer_url='/offer')

The HTML page; it POSTs its SDP offer as JSON to offer_url and plays the answer.

agent_adapter.voice.video

VideoFeed and look_tool: let an agent see a browser camera, a screen or a robot camera.

The feed keeps only the latest frame, sampled at most fps times a second, and encodes it to JPEG only when the agent looks. A frame is an av.VideoFrame, a uint8 RGB, RGBA or grey array (numpy; a MuJoCo render is one), a PIL image, or encoded image bytes. Sources: a WebRTC video track (follow, or WebRTCAudio(video=feed)), a local camera or the screen (camera, screen; PyAV through aiortc), or any function that returns a frame (poll).

device_input(kind, device, platform, fps)

The FFmpeg (file, format, options) for a camera or the screen on platform.

to_jpeg(frame, *, max_side=1024, quality=80)

frame as JPEG bytes, scaled down so its longest side is at most max_side.

JPEG bytes are returned as they are; other encoded images are decoded and re-encoded.

Also exported here, documented under the canonical path: VideoFeed, look_tool.

agent_adapter.voice.moonshine

Local speech models with permissive licences: Moonshine speech-to-text and Kokoro voices.

moonshine-voice is MIT; its Kokoro voices are Apache-2.0. Models download once (about 360 MB; set MOONSHINE_VOICE_CACHE to choose where) and then run offline on the CPU.

Both stream: speech is transcribed while the caller talks, so the text is ready as the turn ends, and a line starts playing from its first chunk, not after the whole line.

MoonshineSTT

Speech-to-text on the CPU; model is a Moonshine architecture name.

A stream decodes what it has heard every update_interval seconds of audio. Only the final text is used, so a longer interval saves CPU; past about 1.5 s the final pass has more left to decode and the reply starts later.

MoonshineTTS

Text-to-speech with a Kokoro voice at 24 kHz, streamed chunk by chunk.