Voice calls and transports¶
Transports (sound card, WebSocket, phone networks, WebRTC), hosted and local speech, and call features such as recording, idle prompts and voicemail. Guide: Voice call features.
agent_adapter.voice.observers¶
agent_adapter.voice.observers re-exports, documented under the canonical path: CallObserver.
agent_adapter.voice.ending¶
agent_adapter.voice.ending re-exports, documented under the canonical path: IdlePolicy, end_call_tool.
agent_adapter.voice.recorder¶
agent_adapter.voice.recorder re-exports, documented under the canonical path: CallRecorder.
agent_adapter.voice.ambience¶
agent_adapter.voice.ambience re-exports, documented under the canonical path: ThinkingSound.
agent_adapter.voice.voicemail¶
VoicemailDetector: tell an answering machine from a person on an outbound call, offline.
PHRASES = ('leave a message', 'leave your message', 'after the tone', 'after the beep', 'not available', 'unavailable', "can't take your call", 'cannot take your call', "can't come to the phone", 'voicemail', 'voice mail', 'mailbox', 'record your message')
module-attribute
¶
tone_of(pcm, min_level=0.02)
¶
The frequency of a steady tone in pcm (16 kHz), or 0.0 if it is not one.
A pure tone crosses zero at even intervals; speech and noise do not.
Also exported here, documented under the canonical path: VoicemailDetector.
agent_adapter.voice.denoise¶
agent_adapter.voice.denoise re-exports, documented under the canonical path: NoiseGate.
agent_adapter.voice.quality¶
VoxCat quality checks: fast, model-free guards on a call's audio and words.
Each check is pure Python and costs well under a millisecond a frame or a sentence:
InputCheck (the caller's audio), OutputCheck (the agent's audio), KnownNames (fix
misheard names), UnsureWords (read back unsure numbers, dates and names before a
confirm=True tool) and SpokenForm (money, numbers and dates as a voice says them).
CheckedSTT
¶
Bases: CallObserver
An STT that runs fixes (such as KnownNames) on every transcript and gives the words
to unsure. Make one with check_stt, and pass it as an observer too, so the words land
under the call's run id: VoiceSession(stt=checked, observers=[checked]).
Word confidence comes from the STT, or its stream, setting last_words (a list of Word
or provider word dicts) with each transcript; without it every word counts as sure,
though a name the fixes guessed is still read back.
check(text, words=None)
¶
Run the fixes on text and note its words; returns the fixed text.
Currency
¶
Bases: NamedTuple
How to say an amount: "rupee", "rupees", "paisa", "paise", read in lakh and crore.
FrameStats
dataclass
¶
What one frame of audio looks like.
InputCheck
¶
Bases: CallObserver
Watches the caller's audio for clipping, dropouts, a high noise floor or a voice too
quiet to hear. Pass it as both a filter and an observer:
VoiceSession(filters=[check], observers=[check]).
As a filter, frames below gate_db become silence (hiss stops waking the turn detector),
holding open hold_ms after speech so word endings stay. The noise floor is the
quietest of the last floor_frames frames. A turn has a problem when clip_share of
its sound clipped, its dropouts (exact silence for under max_gap_ms with speech on both
sides) add up to dropout_ms, its loudest speech stays under quiet_db, or the floor
sits above noisy_db. After such a turn it keeps the transcript from the agent and says
repeat_prompt instead, at most max_repeats times in a row; ask_again=False only
measures. Each frame costs well under a millisecond, in pure Python.
Issue
dataclass
¶
One problem heard in the agent's audio.
KnownNames
¶
Fixes words that sound like a known name: "I'm Rahool from Acmy Corp" ->
"I'm Rahul from Acme Corp", for KnownNames(["Rahul", "Acme Corp"]).
A word changes when its spelling is at least cutoff alike (1 minus edit distance over
length), or sound_cutoff alike and it sounds the same. Words shorter than min_len
and common English words never change; exact matches only get the name's capitals.
last_fixes lists the guesses of the latest call; guessed names deserve a read-back.
fix(text)
¶
text with misheard names replaced; last_fixes says what changed.
NameFix
dataclass
¶
One change: what was heard, the name it became, and how close they were (0 to 1).
OutputCheck
¶
Bases: CallObserver
Checks the agent's audio as it plays: a jump of click_jump (share of full scale)
between two samples is a click, clip_share of a chunk at full scale is clipping, and
silence_s of near-silent audio inside a line is dead air from a broken voice.
It runs on the playout thread, so it only counts: counts, the last keep issues,
and on_issue(issue) if given (keep that quick too). At the end of the call the counts go
to the trace as a custom.voice_quality event. Pure Python, well under a millisecond a chunk.
feed(pcm)
¶
Check one chunk of the agent's audio; returns the issues it found.
SpokenForm
¶
Bases: CallObserver
Rewrites text before text-to-speech: "Rs 4,50,000" -> "four lakh fifty thousand rupees".
Handles money (rupees with lakh and crore, dollars, euros, pounds), plain and decimal
numbers, ranges, percentages, ordinals, dates, times, phone numbers (digit by digit),
units and short forms. locale="en-IN" reads plain numbers in lakh and crore and slash
dates day first; "en-US" reads millions and month first. pronounce maps a word (case
sensitive) to how to say it: {"VoxCat": "vox cat", "SQL": "sequel"}. currencies adds
symbols or codes: {"AED": Currency("dirham", "dirhams", "fils", "fils")}; a code of
capital letters also reads after the amount ("30 USD").
Call it on a string, or pass it as a VoiceSession observer: it then rewrites every line
the session's player speaks. Pure Python; well under a millisecond a sentence.
TurnAudio
dataclass
¶
The caller's audio since the last transcript, and what was wrong with it.
Unsure
dataclass
¶
A number, date or name heard with low confidence.
UnsureWords
¶
Keeps the low-confidence numbers, dates and names of each call's last turns turns,
and reads them back before a confirm=True tool runs: "Did you say 4,500?".
Feed it what the speech-to-text heard with hear(words, run_id=...) (a check_stt
wrapper does this when the STT reports word confidence). Give readback to the spoken
approval: VoiceApproval(readback=unsure.readback). A word below threshold counts;
with match_arguments, only the ones that show up in the tool's arguments are read back.
Each is read back once. Words with no confidence are taken as sure.
hear(heard, *, run_id=None, guessed=())
¶
Note one turn's words; guessed are (name, score) a names fix put in. Returns the
unsure items found.
readback(request)
¶
The read-back line for this approval, or None when nothing needs one.
unsure(run_id=None)
¶
The unsure items still waiting for a read-back, oldest first.
Word
dataclass
¶
One heard word and the speech-to-text's confidence in it (0 to 1; None if not given).
check_stt(stt, *, fixes=(), unsure=None)
¶
Wrap stt so its transcripts get fixes and unsure sees its words; a streaming
STT stays streaming.
measure(pcm, *, clip_level=32000, previous=None)
¶
Measure one frame; previous is the last sample before it, so a click at the seam counts.
number_words(n, *, indian=False)
¶
4500 -> "four thousand five hundred"; with indian, 450000 -> "four lakh fifty thousand".
agent_adapter.voice.local¶
agent_adapter.voice.local re-exports, documented under the canonical path: LocalAudio.
agent_adapter.voice.audio_socket¶
AudioSocket: raw audio over one WebSocket, for browsers, apps and phone gateways.
| From the client | Meaning |
|---|---|
| binary message | the caller's audio: 16-bit little-endian mono PCM at in_rate |
{"type": "hangup"} |
the call is over |
| To the client | Meaning |
|---|---|
| binary message | the agent's audio: PCM16 at out_rate; play it as it arrives |
{"type": "clear"} |
drop any audio not yet played (the caller cut in) |
Audio goes out at the pace it plays, lead_ms ahead, so the client needs only a small
jitter buffer and a cut stops what the caller hears within that much.
Also exported here, documented under the canonical path: AudioSocket, binary_socket.
agent_adapter.voice.socket¶
A voice agent over one WebSocket, at the transcript level: the client hears and speaks.
The client (a browser with the Web Speech API, a phone gateway, a native app) does speech to text and text to speech; this side decides. Messages are JSON text:
| From the client | Meaning |
|---|---|
{"type": "speech_started"} |
the caller started talking |
{"type": "transcript", "text": "...", "final": true} |
a transcript; only final ones count |
{"type": "played", "id": 3} |
line 3 played to the end |
{"type": "hangup"} |
the call is over |
| To the client | Meaning |
|---|---|
{"type": "say", "id": 3, "text": "..."} |
speak this line, then send played |
{"type": "stop"} |
stop speaking now and drop queued lines |
A line counts as played only after its played message, so the history after a cut holds
exactly what the caller heard. A client that never acknowledges is trusted after
played_timeout_s. Send speech_started as each utterance begins: an answer to a spoken
yes/no question counts only if its speech started after the question played.
TextSocket
¶
Bases: Protocol
A text WebSocket: send(text) and async iteration over received texts.
The websockets client and server connections are one as they are; wrap Starlette or
FastAPI sockets with text_socket(ws.send_text, ws.iter_text()).
VoiceSocket
¶
Runs a VoiceAgent over a TextSocket; give the agent voice_socket.speaker.
run(agent)
async
¶
Read the client until it hangs up or disconnects; then close the agent.
text_socket(send, messages)
¶
A TextSocket from a send function and an async iterator of received texts.
agent_adapter.voice.twilio¶
agent_adapter.voice.twilio re-exports, documented under the canonical path: TwilioStream.
agent_adapter.voice.twilio_calls¶
TwilioCalls: place, transfer and end phone calls through Twilio's Calls REST API.
Credentials come from the constructor or the environment: TWILIO_ACCOUNT_SID with
TWILIO_AUTH_TOKEN, or an API key (TWILIO_API_KEY and TWILIO_API_SECRET). They are
sent only as HTTP Basic auth and never logged. Each call here has a side effect: when an
agent may trigger one, wrap it in a tool marked confirm=True.
Also exported here, documented under the canonical path: TwilioCalls.
agent_adapter.voice.telnyx¶
agent_adapter.voice.telnyx re-exports, documented under the canonical path: TelnyxStream.
agent_adapter.voice.plivo¶
agent_adapter.voice.plivo re-exports, documented under the canonical path: PlivoStream.
agent_adapter.voice.vonage¶
agent_adapter.voice.vonage re-exports, documented under the canonical path: VonageStream.
agent_adapter.voice.webrtc¶
WebRTCAudio: a browser call over WebRTC, peer to peer, with no media server (aiortc).
The browser sends an SDP offer to any HTTP route you own; accept(offer) returns the
answer. Audio is Opus at 48 kHz both ways. The browser's own echo cancellation keeps the
agent's voice out of the microphone, so speakers work, not only headphones.
With video=VideoFeed(...), the browser's camera or screen share feeds it (see
agent_adapter.voice.video); without one, incoming video is declined.
Our answer never asks the browser for DTX: with DTX, the browser sends nothing in quiet, and the first syllable after a pause (a "Wait") arrives clipped.
agent_adapter.voice.webrtc_page¶
A one-page browser client for WebRTCAudio: talk to your agent, show it a camera or screen.
webrtc_page(offer_url='/offer')
¶
The HTML page; it POSTs its SDP offer as JSON to offer_url and plays the answer.
agent_adapter.voice.video¶
VideoFeed and look_tool: let an agent see a browser camera, a screen or a robot camera.
The feed keeps only the latest frame, sampled at most fps times a second, and encodes it
to JPEG only when the agent looks. A frame is an av.VideoFrame, a uint8 RGB, RGBA or grey
array (numpy; a MuJoCo render is one), a PIL image, or encoded image bytes. Sources: a
WebRTC video track (follow, or WebRTCAudio(video=feed)), a local camera or the screen
(camera, screen; PyAV through aiortc), or any function that returns a frame (poll).
device_input(kind, device, platform, fps)
¶
The FFmpeg (file, format, options) for a camera or the screen on platform.
to_jpeg(frame, *, max_side=1024, quality=80)
¶
frame as JPEG bytes, scaled down so its longest side is at most max_side.
JPEG bytes are returned as they are; other encoded images are decoded and re-encoded.
Also exported here, documented under the canonical path: VideoFeed, look_tool.
agent_adapter.voice.moonshine¶
Local speech models with permissive licences: Moonshine speech-to-text and Kokoro voices.
moonshine-voice is MIT; its Kokoro voices are Apache-2.0. Models download once (about
360 MB; set MOONSHINE_VOICE_CACHE to choose where) and then run offline on the CPU.
Both stream: speech is transcribed while the caller talks, so the text is ready as the turn ends, and a line starts playing from its first chunk, not after the whole line.
MoonshineSTT
¶
Speech-to-text on the CPU; model is a Moonshine architecture name.
A stream decodes what it has heard every update_interval seconds of audio. Only the
final text is used, so a longer interval saves CPU; past about 1.5 s the final pass has
more left to decode and the reply starts later.
MoonshineTTS
¶
Text-to-speech with a Kokoro voice at 24 kHz, streamed chunk by chunk.