Use open speech models¶
VoxCat, our own voice pipeline (agent_adapter.voice), runs on open models you host as
well as on hosted APIs. Each adapter fits the same ports as the
Moonshine models, so you swap one line. A library is imported only
when you build its adapter; if it is missing you get MissingDependency with the pip
command.
Models and licences¶
| Model | Task | Class | Licence | Use | Extra | How tested |
|---|---|---|---|---|---|---|
| faster-whisper (Whisper) | STT | FasterWhisperSTT |
MIT code and weights | commercial | voice-whisper |
fake WhisperModel from the README |
| Parakeet TDT 0.6B v3, v2 | STT | ParakeetSTT |
weights CC-BY-4.0, NeMo Apache-2.0 | commercial, credit NVIDIA | voice-parakeet |
fake ASRModel from the model card |
| Kyutai STT 1B, 2.6B | streaming STT | KyutaiSTT |
weights CC-BY-4.0, code MIT | commercial, credit Kyutai | voice-kyutai |
fake moshi from stt_from_file_pytorch.py |
| Chatterbox Turbo, multilingual | TTS | ChatterboxTTS |
MIT code and weights | commercial; archived, watermarked | voice-chatterbox |
fake models from the README |
| Orpheus 3B | streaming TTS | OrpheusTTS |
Apache-2.0 code and weights | commercial; Llama 3.2 base | voice-orpheus |
fake OrpheusModel from the README |
| Kyutai TTS 1.6B | streaming TTS | KyutaiTTS |
weights CC-BY-4.0, code MIT | commercial, credit Kyutai | voice-kyutai |
fake TTSModel from tts_pytorch.py |
| Haru-4B | talker | HaruTalker |
Apache-2.0 code and weights | research release | none (see below) | fake DuplexSession from the README |
| Moshi | talker | MoshiTalker |
weights CC-BY-4.0, code MIT | research only | voice-kyutai |
fake moshi from its README |
Licences were read from each model's official page on 2026-11-18. Check them again before you ship.
Research only
Moshi's model card says the model is for research only, so we list it as research only: do not put it in a product. Haru-4B is Apache-2.0, but higurashi releases it as a research base model; use it for short acknowledgements, never for facts.
Notes on the rest: CC-BY-4.0 lets you use the weights commercially if you credit the author. Orpheus is a fine-tune of Llama 3.2, so read Meta's Llama 3.2 licence as well. Chatterbox was retired in August 2026: the code stays public under MIT with no updates, and every clip carries Resemble's PerTh watermark.
Install¶
Each family has its own extra. All but voice-whisper pull PyTorch, so give each one its
own virtual environment.
voice-parakeetneeds Python 3.12 or newer (NeMo's floor).voice-orpheusruns on vLLM: a CUDA GPU on Linux.- Haru is not on PyPI: clone
github.com/higurashi-ai/Haru, thenpip install -e ./haru.
Models download from Hugging Face on first use, not at install.
Speech-to-text¶
from agent_adapter.voice.kyutai import KyutaiSTT
from agent_adapter.voice.parakeet import ParakeetSTT
from agent_adapter.voice.whisper import FasterWhisperSTT
stt = FasterWhisperSTT("large-v3-turbo", device="cuda", compute_type="float16")
stt = ParakeetSTT("nvidia/parakeet-tdt-0.6b-v3", device="cuda")
stt = KyutaiSTT("kyutai/stt-1b-en_fr", device="cuda") # transcribes while the caller talks
text = await stt.transcribe(pcm, 16_000)
Whisper and Parakeet transcribe the finished turn. FasterWhisperSTT decodes greedily
(beam_size=1) with a fixed language by default, the fast setting for short turns. Kyutai
STT decodes each 80 ms frame as it arrives, so when the turn ends only the model's delay
is left: pass it to VoiceSession as a streaming STT.
Text-to-speech¶
from agent_adapter.voice.chatterbox_tts import ChatterboxTTS
from agent_adapter.voice.kyutai import KyutaiTTS
from agent_adapter.voice.orpheus import OrpheusTTS
tts = ChatterboxTTS(variant="turbo", voice="agent_voice.wav") # a clip you may clone
tts = OrpheusTTS(voice="tara")
tts = KyutaiTTS(voice="expresso/ex03-ex01_happy_001_channel1_334s.wav")
async for chunk in tts.synthesize("Your order ships today."):
... # PCM16 at 24 kHz, as soon as each chunk is ready
Orpheus and Kyutai stream frame by frame. Chatterbox renders whole clips, so the adapter splits a line at sentence ends and plays the first sentence while the next one renders. When the caller cuts in, rendering stops at the next chunk.
Keep models warm¶
A model loads once per process and is shared by every adapter built with the same
settings, so a new call does not reload it. Loading happens in a worker thread on first
use; call warm() at start-up to load it and run one tiny inference ahead of the first
caller:
One loaded model runs one inference at a time. Build a second adapter on another device
(device="cuda:1") to serve calls side by side.
Add a talker next to the agent¶
A talker is a full-duplex model: it hears the caller and speaks at the same time. It fills the gap while the agent (the thinker) calls tools: "mm-hm", "right", "one moment". It never answers for the agent. The port is small:
Talker.open(rate)starts aTalkerStream;warm()loads the model ahead of time.stream.hear(pcm)passes the caller's audio; it returns at once.stream.cues()yields eachCue(audio, text)the talker says; silence is left out.backchannel(stream, play, may_speak=...)plays cues only whilemay_speak()is true.
import asyncio
from agent_adapter.voice.haru import HaruTalker
from agent_adapter.voice.talker import backchannel
talker = HaruTalker(system_prompt=card, voice="agent_voice.wav") # MoshiTalker() also fits
await talker.warm()
stream = talker.open(16_000)
nods = asyncio.create_task(backchannel(stream, play, may_speak=lambda: not agent_speaking))
stream.hear(caller_frame) # every frame the caller sends
card is Haru's character card (haru.prompt.render_card). The agent owns the turn: once
its answer starts, may_speak turns false and the talker's cues are dropped.
Try it offline¶
examples/voxcat_open_models.py
runs a call with stand-ins for each part, so it needs no model, GPU or download. On a GPU
machine with voice-whisper and voice-kyutai, --real swaps in faster-whisper, Kyutai
TTS and Moshi.
How these are tested¶
We do not download any model in CI. Each test puts a stand-in for the library in place, shaped after the calls in its README or model card, and checks what the adapter sends and how it turns the result into audio or text: resampling, sentence splitting, streaming, barge-in, one load per process and the missing-library message.