Hosted speech providers¶
Speech-to-text and text-to-speech services for our voice pipeline, each optional: Deepgram, OpenAI, Cartesia, ElevenLabs, Fish Audio, Rime, AssemblyAI and Soniox. Guide: Fast hosted voices.
agent_adapter.voice.deepgram¶
Deepgram speech-to-text that hears each turn as it is spoken: DeepgramSTT (Nova) and
DeepgramFluxSTT (Flux, which also says when the turn ends, early).
DeepgramSTT.open() streams the turn over the live WebSocket; at its end we send
Finalize and CloseStream, so only the last moment is left to transcribe.
transcribe() sends a finished turn in one HTTP request. The key comes from api_key=
or DEEPGRAM_API_KEY and travels only in the Authorization header.
Connect = Callable[[str, dict[str, str]], Awaitable[LiveSocket]]
module-attribute
¶
DeepgramFluxSTT
¶
Bases: _Deepgram
A StreamingSTT on Deepgram Flux (flux-general-en) whose streams send TurnSignals.
Flux decides the end of the turn itself. eot_threshold is how sure it must be to
send EndOfTurn; with eager_eot_threshold set (lower, such as 0.5) it first sends
EagerEndOfTurn (our eager_end), so a reply can start early, and TurnResumed
(resumed) if the caller goes on. Once EndOfTurn is in, finish() returns at once;
before that it sends ForceEndTurn, so Flux closes the turn without waiting for silence.
DeepgramSTT
¶
Bases: _Deepgram
A StreamingSTT on Deepgram (model such as nova-3).
finish_timeout_s bounds the wait for the final words after the turn ends; past it
the session falls back to transcribe() on the whole turn.
aclose()
async
¶
Close the HTTP client this object made (one you passed in stays open).
LiveSocket
¶
Bases: Protocol
The part of a WebSocket client connection the providers use.
agent_adapter.voice.cloud¶
Hosted speech providers for VoiceSession: OpenAI, ElevenLabs and Cartesia.
Each class fits the STT or TTS port and streams over plain HTTP (httpx). Keys come
from the constructor or the provider's usual environment variable, read on the first
call, and never reach a log or an error message. Every voice is 16-bit mono PCM at
sample_rate. Deepgram streaming speech-to-text is in agent_adapter.voice.deepgram.
CartesiaTTS
¶
Bases: _Hosted
Text-to-speech with Cartesia Sonic over its bytes endpoint (CARTESIA_API_KEY).
voice_id is a Cartesia voice id; version is the Cartesia-Version header.
ElevenLabsTTS
¶
Bases: _Hosted
Text-to-speech with ElevenLabs' streaming endpoint (ELEVENLABS_API_KEY).
voice_id is a voice from your ElevenLabs library.
OpenAISTT
¶
Bases: _Hosted
Speech-to-text with OpenAI's transcription endpoint (OPENAI_API_KEY).
base_url points it at any OpenAI-compatible server instead.
OpenAITTS
¶
Bases: _Hosted
Text-to-speech with OpenAI's speech endpoint, streamed as raw 24 kHz PCM.
agent_adapter.voice.cartesia¶
Cartesia over WebSocket: CartesiaLiveTTS (Sonic) and CartesiaInkSTT (Ink).
CartesiaLiveTTS keeps one socket open and speaks each reply on its own context_id:
text is sent as the model writes it with continue: true, and a last empty request
with continue: false closes the context. Word timestamps come back with the audio.
The key comes from api_key= or CARTESIA_API_KEY and travels only in the X-API-Key
header. Cartesia over plain HTTP is agent_adapter.voice.cloud.CartesiaTTS.
CartesiaInkSTT
¶
Bases: LiveSTT
A StreamingSTT on Cartesia Ink (ink-2, or ink-whisper with language).
At the end of the turn we send finalize and stop at its flush_done.
CartesiaLiveTTS
¶
Bases: LiveTTS
A StreamingTTS on Cartesia Sonic (sonic-3) over one shared WebSocket.
speed (0.6 to 1.5), volume (0.5 to 2.0) and emotion (neutral, calm,
angry, content, sad) go in generation_config. max_buffer_delay_ms is how
long Cartesia may wait for more text before it speaks (lower is sooner).
agent_adapter.voice.elevenlabs¶
ElevenLabs over WebSocket: ElevenLabsLiveTTS (stream-input) and
ElevenLabsScribeSTT (Scribe realtime).
ElevenLabsLiveTTS opens one socket per reply and sends text as the model writes it;
chunk_length_schedule sets how many characters ElevenLabs waits for before each piece
of audio, and flush speaks what is buffered now. warm() opens the next socket ahead.
The key comes from api_key= or ELEVENLABS_API_KEY and travels only in the
xi-api-key header. ElevenLabs over plain HTTP is agent_adapter.voice.cloud.ElevenLabsTTS.
ElevenLabsLiveTTS
¶
Bases: LiveTTS
A StreamingTTS on ElevenLabs' stream-input WebSocket (eleven_flash_v2_5).
chunk_length_schedule (characters, 50 to 500 each) trades a sooner first sound for
smoother speech; voice_settings is ElevenLabs' object (stability, speed, ...).
ElevenLabsScribeSTT
¶
Bases: LiveSTT
A StreamingSTT on ElevenLabs Scribe realtime (scribe_v2_realtime).
With commit_strategy="vad" (or "turn_prediction" on scribe_v3_live) Scribe
commits a segment when the speaker pauses, and the stream passes that on as an end
TurnSignal; with "manual" only the end of our turn commits.
agent_adapter.voice.fish¶
FishAudioTTS: Fish Audio live text-to-speech over wss://api.fish.audio/v1/tts/live.
Every frame both ways is a MessagePack map: start once, text as the model writes it,
flush to speak what is buffered, stop at the end; audio comes back in audio
events until finish. The voice is a saved model (reference_id) or a clip with its
transcript (reference=(audio, text)). The key comes from api_key= or
FISH_AUDIO_API_KEY and travels only in the Authorization header.
FishAudioTTS
¶
Bases: LiveTTS
A StreamingTTS on Fish Audio (model such as s2.1-pro), one socket per reply.
latency is low, normal or balanced; speed (0.5 to 2) and volume (dB,
-20 to 20) go in prosody; chunk_length (100 to 300) is how much text Fish waits
for before it speaks. warm() opens the next socket ahead.
agent_adapter.voice.rime¶
RimeTTS: Rime text-to-speech over its WebSocket v1 protocol (Coda or Mist v3).
One socket stays open; each reply is a contextId that starts, takes text as the
model writes it and ends, with audio and word spans coming back on the same context.
Rime greets each socket with ready before it takes requests. The key comes from
api_key= or RIME_API_KEY and travels only in the Authorization header.
RimeTTS
¶
Bases: LiveTTS
A StreamingTTS on Rime (model coda or mistv3), one shared socket.
speaker is a Rime voice of that model; time_scale above 1 speaks slower.
Text is grouped into sentences as it arrives (SPLIT_STRATEGY_ACCUMULATE_SENTENCE).
agent_adapter.voice.assemblyai¶
AssemblyAISTT: AssemblyAI Universal-Streaming (v3) speech-to-text, one socket per turn.
The model decides the end of the turn itself and says so with end_of_turn; the stream
passes that on as an end TurnSignal. The key comes from api_key= or
ASSEMBLYAI_API_KEY and travels only in the Authorization header.
AssemblyAISTT
¶
Bases: LiveSTT
A StreamingSTT on AssemblyAI (speech_model, such as universal-3-6-pro or
assemblyai-stt-3-7-realtime-agent).
mode is the API's preset (min_latency, balanced, max_accuracy);
min_turn_silence and max_turn_silence (ms) tune the end of the turn. AssemblyAI
takes 20 to 1000 ms chunks no faster than real time, so the stream regroups audio
into 50 to 1000 ms chunks and transcribe() sends at the pace it plays.
agent_adapter.voice.soniox¶
SonioxSTT: Soniox real-time speech-to-text, one socket per turn.
With endpoint detection on, Soniox sends an <end> token when the speaker is done; the
stream passes it on as an end TurnSignal. At the end of the turn we send finalize
and stop at its <fin> token. The key comes from api_key= or SONIOX_API_KEY and
travels only in the Authorization header (never in the config message).
SonioxSTT
¶
Bases: LiveSTT
A StreamingSTT on Soniox (stt-rt-v6).
language_hints (such as ("en",)) and context (Soniox's context object: terms,
text) go in the config message; max_endpoint_delay_ms (500 to 3000) bounds how long
Soniox waits before it calls the end of speech.