Open speech models¶
Open speech-to-text and text-to-speech models for our voice pipeline, and the talker port for full-duplex models next to the agent. Guide: Open speech models.
agent_adapter.voice.whisper¶
Whisper speech-to-text on CTranslate2 (faster-whisper, MIT; Whisper weights MIT).
The model loads once per process and is shared by every FasterWhisperSTT with the same
settings. Greedy decoding (beam_size=1) and a fixed language are the fast defaults for
short spoken turns; raise beam_size for harder audio.
FasterWhisperSTT
¶
Speech-to-text for one finished turn; model is a size name, a path or a Hub id.
Extra keyword options go to WhisperModel.transcribe as they are (for example
vad_filter=True or hotwords="Acme").
warm()
async
¶
Load the model and decode half a second of silence, so the first turn is fast.
agent_adapter.voice.parakeet¶
NVIDIA Parakeet speech-to-text through NeMo (weights CC-BY-4.0; NeMo Apache-2.0).
parakeet-tdt-0.6b-v3 covers 25 European languages; -v2 is English only. NeMo needs
Python 3.12 or newer and PyTorch; give it its own environment.
agent_adapter.voice.kyutai¶
Kyutai streaming speech models through the moshi package (code MIT; weights CC-BY-4.0).
KyutaiSTT decodes every 80 ms frame as it arrives, so at the end of a turn only the
model's own delay is left (0.5 s of audio for stt-1b-en_fr, 2.5 s for stt-2.6b-en, run
faster than real time). KyutaiTTS plays each frame as soon as it is decoded. One loaded
model serves one stream at a time; the others wait for it.
agent_adapter.voice.chatterbox_tts¶
Chatterbox text-to-speech from Resemble AI (code and weights MIT; archived in 2026).
Chatterbox renders a whole clip per call, so a line is split at sentence ends and each sentence plays as soon as it is ready: the caller hears the first sentence while the rest renders. Every clip carries Resemble's PerTh watermark.
ChatterboxTTS
¶
Chatterbox voices at 24 kHz; voice is a reference clip (a path) to clone.
variant="turbo" (350M, one-step decoder) is the fastest and needs voice;
nano=True picks its 110M model for the CPU. variant="multilingual" needs
language (for example "fr"). exaggeration and cfg_weight apply to the standard
and multilingual models.
warm()
async
¶
Load the model and render one short word, so the first line is fast.
agent_adapter.voice.orpheus¶
Orpheus text-to-speech from Canopy Labs (code Apache-2.0; weights Apache-2.0 on a Llama 3.2 base). It runs on vLLM, so it needs a CUDA GPU on Linux.
Orpheus streams: the first chunk arrives in about 200 ms and plays while the rest renders.
Emotive tags such as <laugh> and <sigh> in the text are spoken as sounds.
OrpheusTTS
¶
Orpheus voices as PCM16 at 24 kHz; English voices: tara, leah, jess, leo, dan, mia,
zac, zoe. Extra keyword options go to generate_speech (for example
repetition_penalty=1.1, which the Orpheus README says must stay at 1.1 or more).
warm()
async
¶
Load the model and render one short word, so the first line is fast.
agent_adapter.voice.talker¶
The talker port: a full-duplex speech model that keeps the caller company while the agent thinks.
A talker (Haru-4B, Moshi) hears the caller frame by frame and answers in its own voice at the same time: "mm-hm", "right", "let me check". It gives no facts and runs no tools. The agent (the thinker) still owns the turn: it calls tools, decides and speaks the answer. Play talker cues only while the agent is silent, and drop them once it speaks.
Cue
dataclass
¶
A short piece of the talker's speech: PCM16 mono at the talker's sample_rate,
and the text it said, when the model gives one.
Talker
¶
Bases: Protocol
A full-duplex model. warm() loads it ahead of the call; open(rate) starts one.
TalkerStream
¶
backchannel(stream, play, *, may_speak)
async
¶
Play each cue while may_speak() is true (the agent is not speaking); drop the rest.
Run it as a task next to the call; it ends when the stream closes.
agent_adapter.voice.haru¶
Haru-4B from higurashi as a talker (code and weights Apache-2.0; a research release).
Haru listens and speaks at once in 133 ms frames (3200 samples at 24 kHz). It is a pretrained base model: use it for backchannels and short acknowledgements next to the agent, never for facts. Clone a voice only with the right to use it.