Skip to content

Open speech models

Open speech-to-text and text-to-speech models for our voice pipeline, and the talker port for full-duplex models next to the agent. Guide: Open speech models.

agent_adapter.voice.whisper

Whisper speech-to-text on CTranslate2 (faster-whisper, MIT; Whisper weights MIT).

The model loads once per process and is shared by every FasterWhisperSTT with the same settings. Greedy decoding (beam_size=1) and a fixed language are the fast defaults for short spoken turns; raise beam_size for harder audio.

FasterWhisperSTT

Speech-to-text for one finished turn; model is a size name, a path or a Hub id.

Extra keyword options go to WhisperModel.transcribe as they are (for example vad_filter=True or hotwords="Acme").

warm() async

Load the model and decode half a second of silence, so the first turn is fast.

agent_adapter.voice.parakeet

NVIDIA Parakeet speech-to-text through NeMo (weights CC-BY-4.0; NeMo Apache-2.0).

parakeet-tdt-0.6b-v3 covers 25 European languages; -v2 is English only. NeMo needs Python 3.12 or newer and PyTorch; give it its own environment.

ParakeetSTT

Speech-to-text for one finished turn, on a GPU (device="cuda") or the CPU.

warm() async

Load the model and decode half a second of silence, so the first turn is fast.

agent_adapter.voice.kyutai

Kyutai streaming speech models through the moshi package (code MIT; weights CC-BY-4.0).

KyutaiSTT decodes every 80 ms frame as it arrives, so at the end of a turn only the model's own delay is left (0.5 s of audio for stt-1b-en_fr, 2.5 s for stt-2.6b-en, run faster than real time). KyutaiTTS plays each frame as soon as it is decoded. One loaded model serves one stream at a time; the others wait for it.

KyutaiSTT

Streaming speech-to-text; model is a Kyutai STT repo in the moshi format.

warm() async

Load the model and decode half a second of silence, so the first turn is fast.

KyutaiTTS

Streaming text-to-speech at 24 kHz; voice is a name in kyutai/tts-voices or a .safetensors voice embedding. The voice is prepared once and reused.

warm() async

Load the model and render one short word, so the first line is fast.

agent_adapter.voice.chatterbox_tts

Chatterbox text-to-speech from Resemble AI (code and weights MIT; archived in 2026).

Chatterbox renders a whole clip per call, so a line is split at sentence ends and each sentence plays as soon as it is ready: the caller hears the first sentence while the rest renders. Every clip carries Resemble's PerTh watermark.

ChatterboxTTS

Chatterbox voices at 24 kHz; voice is a reference clip (a path) to clone.

variant="turbo" (350M, one-step decoder) is the fastest and needs voice; nano=True picks its 110M model for the CPU. variant="multilingual" needs language (for example "fr"). exaggeration and cfg_weight apply to the standard and multilingual models.

warm() async

Load the model and render one short word, so the first line is fast.

agent_adapter.voice.orpheus

Orpheus text-to-speech from Canopy Labs (code Apache-2.0; weights Apache-2.0 on a Llama 3.2 base). It runs on vLLM, so it needs a CUDA GPU on Linux.

Orpheus streams: the first chunk arrives in about 200 ms and plays while the rest renders. Emotive tags such as <laugh> and <sigh> in the text are spoken as sounds.

OrpheusTTS

Orpheus voices as PCM16 at 24 kHz; English voices: tara, leah, jess, leo, dan, mia, zac, zoe. Extra keyword options go to generate_speech (for example repetition_penalty=1.1, which the Orpheus README says must stay at 1.1 or more).

warm() async

Load the model and render one short word, so the first line is fast.

agent_adapter.voice.talker

The talker port: a full-duplex speech model that keeps the caller company while the agent thinks.

A talker (Haru-4B, Moshi) hears the caller frame by frame and answers in its own voice at the same time: "mm-hm", "right", "let me check". It gives no facts and runs no tools. The agent (the thinker) still owns the turn: it calls tools, decides and speaks the answer. Play talker cues only while the agent is silent, and drop them once it speaks.

Cue dataclass

A short piece of the talker's speech: PCM16 mono at the talker's sample_rate, and the text it said, when the model gives one.

Talker

Bases: Protocol

A full-duplex model. warm() loads it ahead of the call; open(rate) starts one.

TalkerStream

Bases: Protocol

One call's duplex link to a talker.

cues()

The talker's speech as it comes; silence between cues is left out.

hear(pcm)

Add the caller's audio (PCM16 at the rate given to open); returns at once.

backchannel(stream, play, *, may_speak) async

Play each cue while may_speak() is true (the agent is not speaking); drop the rest.

Run it as a task next to the call; it ends when the stream closes.

agent_adapter.voice.haru

Haru-4B from higurashi as a talker (code and weights Apache-2.0; a research release).

Haru listens and speaks at once in 133 ms frames (3200 samples at 24 kHz). It is a pretrained base model: use it for backchannels and short acknowledgements next to the agent, never for facts. Clone a voice only with the right to use it.

HaruTalker

A Talker on Haru-4B. system_prompt is Haru's character card (haru.prompt.render_card); voice is a reference clip to speak in, prepared once.

warm() async

Load the model (and the voice) ahead of the first call.

agent_adapter.voice.moshi_talker

Moshi from Kyutai as a talker (code MIT; weights CC-BY-4.0). Its model card says it is for research only, so the docs list it as research only.

Moshi listens and speaks at once in 80 ms frames (1920 samples at 24 kHz).

MoshiTalker

A Talker on Moshi; model is a Moshi repo (moshiko male, moshika female).

warm() async

Load the model ahead of the first call.