Hosted speech, echo and phone calls¶
This page adds three things to our own voice pipeline: hosted speech providers, a guard against the agent hearing itself on speakers, and the phone basics on Twilio.
Use a hosted speech provider¶
Each class fits the STT, StreamingSTT or TTS port, so it drops into VoiceSession:
from agent_adapter.voice import VoiceSession
from agent_adapter.voice.cloud import CartesiaTTS, ElevenLabsTTS, OpenAISTT, OpenAITTS
from agent_adapter.voice.deepgram import DeepgramSTT
session = VoiceSession(
transport,
stt=DeepgramSTT("nova-3"), # streams each turn as it is spoken
tts=ElevenLabsTTS("your-voice-id", sample_rate=24_000),
turns=turns,
)
# or: stt=OpenAISTT(), tts=OpenAITTS(voice="coral"), tts=CartesiaTTS("your-voice-id")
| Class | Key from | What it calls |
|---|---|---|
DeepgramSTT |
DEEPGRAM_API_KEY |
the live WebSocket per turn (Finalize, then CloseStream at its end); one POST /v1/listen for a whole turn |
OpenAISTT |
OPENAI_API_KEY |
POST /v1/audio/transcriptions with the turn as WAV |
OpenAITTS |
OPENAI_API_KEY |
POST /v1/audio/speech, raw 24 kHz PCM as it streams |
ElevenLabsTTS |
ELEVENLABS_API_KEY |
POST /v1/text-to-speech/{voice}/stream, output_format=pcm_<rate> |
CartesiaTTS |
CARTESIA_API_KEY |
POST /tts/bytes, raw pcm_s16le |
For the WebSocket speed features (Deepgram Flux early end of turn, text streamed into Cartesia, ElevenLabs, Fish Audio or Rime, and AssemblyAI, Soniox, Cartesia Ink or Scribe speech-to-text), see Fast hosted voices.
- Pass
api_key=to skip the environment. The key travels only in a request header; it never reaches a log or an error message. - Failures raise the same typed errors as the model ports:
ModelAuthError,ModelRateLimited(withretry_after),ModelUnavailableandModelTimeout. A failed speech-to-text pass resumes playback; a broken Deepgram stream falls back to the whole turn. OpenAISTTandOpenAITTStakebase_url=for any OpenAI-compatible server.client=takes your ownhttpx.AsyncClient(proxies, retries, tests);aclose()closes only a client the object made itself.
Play on speakers without cutting yourself off¶
With no echo cancelling, the mic hears the agent and the agent stops for its own voice.
EchoGuard raises the bar only while the agent is speaking:
from agent_adapter.voice import EchoGuard, LocalAudio, VoiceSession
guard = EchoGuard(level=0.1, loud_ms=200, speech_ms=1200)
session = VoiceSession(LocalAudio(), stt=stt, tts=tts, turns=turns, echo_guard=guard)
| While the agent speaks | What happens |
|---|---|
Speech under level RMS that ends before speech_ms |
nothing: playback goes on and nothing is transcribed |
loud_ms of audio at level or above |
a normal barge-in: playback pauses and the turn is heard |
Speech that lasts speech_ms |
a barge-in too, however quiet |
When the agent is quiet the guard does nothing. Set level between the agent's echo and
your voice at the mic; headphones or a browser (which cancels echo) need no guard. A real
echo canceller that installs from wheels on Windows and Linux exists in LiveKit's
livekit package (Apache-2.0, AudioProcessingModule); the pipeline does not use it yet.
Phone basics on Twilio¶
from agent_adapter.voice import TwilioCalls, TwilioStream, VoiceSession
from agent_adapter.voice.socket import text_socket
calls = TwilioCalls() # TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN (or an API key)
@app.websocket("/twilio")
async def twilio(ws):
await ws.accept()
phone = TwilioStream(text_socket(ws.send_text, ws.iter_text()), calls=calls)
session = VoiceSession(phone, stt=stt, tts=tts, turns=turns)
await session.run(VoiceAgent(model, executor, session.speaker))
- Keypad: each
dtmfdigit goes tophone.on_digit.VoiceSessionstops the agent at the first digit and hands it(keypad) 1234afterkeypad_wait_ms(1500) with no new digit, or at once on#. Change the words withkeypad_text=. - Hang up:
await phone.hang_up()ends the call through the REST API (Status=completed) and endsframes(). With nocalls, only our side ends and Twilio runs the TwiML after<Connect>, which hangs up when there is none. - Transfer:
await phone.transfer("+15558675310", caller_id="+15551112222")dials a person from the live call; Twilio ends the stream as it dials. - Call out:
await calls.call(to, from_, twiml=stream_twiml("wss://you.example/twilio"))places a call that connects to the same route when answered (stream_twimlis inagent_adapter.voice.twilio_calls).
Each of these has a side effect. When the agent may trigger one, wrap it in a tool marked
confirm=True, so a person (or the caller, with VoiceApproval) approves it first:
@registry.tool(confirm=True)
async def transfer_to_desk(reason: str) -> str:
"""Hand the caller to a person at the front desk."""
await current_call().transfer("+15558675310") # look up this call's TwilioStream
return "transferring"
Twilio's own errors raise the same typed errors (provider="twilio"); the credentials go
only in HTTP Basic auth. Test all of it offline: pass client= an
httpx.AsyncClient(transport=httpx.MockTransport(handler)), and DeepgramSTT(connect=) a
fake socket.
Other phone networks¶
TelnyxStream, PlivoStream and VonageStream work like TwilioStream: paced playout,
a cut clears what is queued, each line is confirmed by the network, keypad presses reach
the agent and hang_up() ends our side.
| Network | Transport | Start it with | Audio |
|---|---|---|---|
| Telnyx | TelnyxStream(text_socket(...)) |
Call Control stream_url with stream_bidirectional_mode="rtp", or TeXML <Stream bidirectionalMode="rtp"> |
8 kHz μ-law; mark per line |
| Plivo | PlivoStream(text_socket(...), content_type=...) |
<Stream bidirectional="true"> |
8 or 16 kHz PCM, or μ-law; checkpoint per line |
| Vonage | VonageStream(socket, rate=16000) |
NCCO connect to a WebSocket, audio/l16;rate=16000 |
8, 16 or 24 kHz PCM; notify per line |
These are tested against each network's documented message formats with fake sockets, not on live calls: run one test call on your account before going live.