Skip to content

Hosted speech providers

Speech-to-text and text-to-speech services for our voice pipeline, each optional: Deepgram, OpenAI, Cartesia, ElevenLabs, Fish Audio, Rime, AssemblyAI and Soniox. Guide: Fast hosted voices.

agent_adapter.voice.deepgram

Deepgram speech-to-text that hears each turn as it is spoken: DeepgramSTT (Nova) and DeepgramFluxSTT (Flux, which also says when the turn ends, early).

DeepgramSTT.open() streams the turn over the live WebSocket; at its end we send Finalize and CloseStream, so only the last moment is left to transcribe. transcribe() sends a finished turn in one HTTP request. The key comes from api_key= or DEEPGRAM_API_KEY and travels only in the Authorization header.

Connect = Callable[[str, dict[str, str]], Awaitable[LiveSocket]] module-attribute

DeepgramFluxSTT

Bases: _Deepgram

A StreamingSTT on Deepgram Flux (flux-general-en) whose streams send TurnSignals.

Flux decides the end of the turn itself. eot_threshold is how sure it must be to send EndOfTurn; with eager_eot_threshold set (lower, such as 0.5) it first sends EagerEndOfTurn (our eager_end), so a reply can start early, and TurnResumed (resumed) if the caller goes on. Once EndOfTurn is in, finish() returns at once; before that it sends ForceEndTurn, so Flux closes the turn without waiting for silence.

DeepgramSTT

Bases: _Deepgram

A StreamingSTT on Deepgram (model such as nova-3).

finish_timeout_s bounds the wait for the final words after the turn ends; past it the session falls back to transcribe() on the whole turn.

aclose() async

Close the HTTP client this object made (one you passed in stays open).

LiveSocket

Bases: Protocol

The part of a WebSocket client connection the providers use.

agent_adapter.voice.cloud

Hosted speech providers for VoiceSession: OpenAI, ElevenLabs and Cartesia.

Each class fits the STT or TTS port and streams over plain HTTP (httpx). Keys come from the constructor or the provider's usual environment variable, read on the first call, and never reach a log or an error message. Every voice is 16-bit mono PCM at sample_rate. Deepgram streaming speech-to-text is in agent_adapter.voice.deepgram.

CartesiaTTS

Bases: _Hosted

Text-to-speech with Cartesia Sonic over its bytes endpoint (CARTESIA_API_KEY).

voice_id is a Cartesia voice id; version is the Cartesia-Version header.

ElevenLabsTTS

Bases: _Hosted

Text-to-speech with ElevenLabs' streaming endpoint (ELEVENLABS_API_KEY).

voice_id is a voice from your ElevenLabs library.

OpenAISTT

Bases: _Hosted

Speech-to-text with OpenAI's transcription endpoint (OPENAI_API_KEY).

base_url points it at any OpenAI-compatible server instead.

OpenAITTS

Bases: _Hosted

Text-to-speech with OpenAI's speech endpoint, streamed as raw 24 kHz PCM.

agent_adapter.voice.cartesia

Cartesia over WebSocket: CartesiaLiveTTS (Sonic) and CartesiaInkSTT (Ink).

CartesiaLiveTTS keeps one socket open and speaks each reply on its own context_id: text is sent as the model writes it with continue: true, and a last empty request with continue: false closes the context. Word timestamps come back with the audio. The key comes from api_key= or CARTESIA_API_KEY and travels only in the X-API-Key header. Cartesia over plain HTTP is agent_adapter.voice.cloud.CartesiaTTS.

CartesiaInkSTT

Bases: LiveSTT

A StreamingSTT on Cartesia Ink (ink-2, or ink-whisper with language).

At the end of the turn we send finalize and stop at its flush_done.

CartesiaLiveTTS

Bases: LiveTTS

A StreamingTTS on Cartesia Sonic (sonic-3) over one shared WebSocket.

speed (0.6 to 1.5), volume (0.5 to 2.0) and emotion (neutral, calm, angry, content, sad) go in generation_config. max_buffer_delay_ms is how long Cartesia may wait for more text before it speaks (lower is sooner).

agent_adapter.voice.elevenlabs

ElevenLabs over WebSocket: ElevenLabsLiveTTS (stream-input) and ElevenLabsScribeSTT (Scribe realtime).

ElevenLabsLiveTTS opens one socket per reply and sends text as the model writes it; chunk_length_schedule sets how many characters ElevenLabs waits for before each piece of audio, and flush speaks what is buffered now. warm() opens the next socket ahead. The key comes from api_key= or ELEVENLABS_API_KEY and travels only in the xi-api-key header. ElevenLabs over plain HTTP is agent_adapter.voice.cloud.ElevenLabsTTS.

ElevenLabsLiveTTS

Bases: LiveTTS

A StreamingTTS on ElevenLabs' stream-input WebSocket (eleven_flash_v2_5).

chunk_length_schedule (characters, 50 to 500 each) trades a sooner first sound for smoother speech; voice_settings is ElevenLabs' object (stability, speed, ...).

ElevenLabsScribeSTT

Bases: LiveSTT

A StreamingSTT on ElevenLabs Scribe realtime (scribe_v2_realtime).

With commit_strategy="vad" (or "turn_prediction" on scribe_v3_live) Scribe commits a segment when the speaker pauses, and the stream passes that on as an end TurnSignal; with "manual" only the end of our turn commits.

agent_adapter.voice.fish

FishAudioTTS: Fish Audio live text-to-speech over wss://api.fish.audio/v1/tts/live.

Every frame both ways is a MessagePack map: start once, text as the model writes it, flush to speak what is buffered, stop at the end; audio comes back in audio events until finish. The voice is a saved model (reference_id) or a clip with its transcript (reference=(audio, text)). The key comes from api_key= or FISH_AUDIO_API_KEY and travels only in the Authorization header.

FishAudioTTS

Bases: LiveTTS

A StreamingTTS on Fish Audio (model such as s2.1-pro), one socket per reply.

latency is low, normal or balanced; speed (0.5 to 2) and volume (dB, -20 to 20) go in prosody; chunk_length (100 to 300) is how much text Fish waits for before it speaks. warm() opens the next socket ahead.

agent_adapter.voice.rime

RimeTTS: Rime text-to-speech over its WebSocket v1 protocol (Coda or Mist v3).

One socket stays open; each reply is a contextId that starts, takes text as the model writes it and ends, with audio and word spans coming back on the same context. Rime greets each socket with ready before it takes requests. The key comes from api_key= or RIME_API_KEY and travels only in the Authorization header.

RimeTTS

Bases: LiveTTS

A StreamingTTS on Rime (model coda or mistv3), one shared socket.

speaker is a Rime voice of that model; time_scale above 1 speaks slower. Text is grouped into sentences as it arrives (SPLIT_STRATEGY_ACCUMULATE_SENTENCE).

agent_adapter.voice.assemblyai

AssemblyAISTT: AssemblyAI Universal-Streaming (v3) speech-to-text, one socket per turn.

The model decides the end of the turn itself and says so with end_of_turn; the stream passes that on as an end TurnSignal. The key comes from api_key= or ASSEMBLYAI_API_KEY and travels only in the Authorization header.

AssemblyAISTT

Bases: LiveSTT

A StreamingSTT on AssemblyAI (speech_model, such as universal-3-6-pro or assemblyai-stt-3-7-realtime-agent).

mode is the API's preset (min_latency, balanced, max_accuracy); min_turn_silence and max_turn_silence (ms) tune the end of the turn. AssemblyAI takes 20 to 1000 ms chunks no faster than real time, so the stream regroups audio into 50 to 1000 ms chunks and transcribe() sends at the pace it plays.

agent_adapter.voice.soniox

SonioxSTT: Soniox real-time speech-to-text, one socket per turn.

With endpoint detection on, Soniox sends an <end> token when the speaker is done; the stream passes it on as an end TurnSignal. At the end of the turn we send finalize and stop at its <fin> token. The key comes from api_key= or SONIOX_API_KEY and travels only in the Authorization header (never in the config message).

SonioxSTT

Bases: LiveSTT

A StreamingSTT on Soniox (stt-rt-v6).

language_hints (such as ("en",)) and context (Soniox's context object: terms, text) go in the config message; max_endpoint_delay_ms (500 to 3000) bounds how long Soniox waits before it calls the end of speech.