Skip to content

Answer sooner on a voice call

Speed makes a VoiceAgent answer sooner. Each part is optional:

Part What it does
speculative reply starts the reply on the early transcript; keeps it if the final one has the same words
SpeculativeTools lets that reply run opted-in read-only tools before the turn is final
Acknowledgements plays a short pre-rendered line ("Mm-hm.") when the reply will be slow
AnswerCache replays the reply to a repeated question, audio included
Warmup connects, renders lines and prefetches read-only tools at call start
from agent_adapter.voice import VoiceAgent, VoiceSession
from agent_adapter.voice.ack import Acknowledgements
from agent_adapter.voice.answers import AnswerCache
from agent_adapter.voice.speculate import SpeculativeTools
from agent_adapter.voice.speed import Speed
from agent_adapter.voice.warm import Warmup

answers = AnswerCache(ttl_s=300)  # one per process, shared by its calls
session = VoiceSession(transport, stt=stt, tts=answers.voice(tts), turns=turns, early_ms=200)
speed = Speed(
    tools=SpeculativeTools(["weather"]),
    acks=Acknowledgements(["Mm-hm.", "Okay."]),
    answers=answers,
    tenant="acme",
    warmup=Warmup(connections=[pool], tools=[("account", {"caller": caller_id})]),
)
agent = VoiceAgent(model, executor, session.speaker, speed=speed)
await session.run(agent)  # warm-up starts with the call

Run the whole thing offline: python examples/voxcat_speed.py.

Start the reply before the turn is over

After early_ms of quiet, VoiceSession already starts the final transcription while the turn detector decides. With Speed, that early transcript also starts the reply. Its words are held back. When the turn ends with the same words and nothing was said in between, the reply is kept and plays at once; if the caller goes on, it is dropped.

speed = Speed(tools=SpeculativeTools(["weather"], max_per_turn=2, max_per_minute=30))
agent = VoiceAgent(model(), Executor(registry, cache=MemoryCache()), RecordingSpeaker(),
                   speed=speed)  # fmt: skip
agent.speculate("weather in oslo")  # the early transcript: the reply starts, held back
await settle()
print("ran before the turn was final:", calls)
timing = TurnTiming(started=0.0)
await agent.user_said("Weather in Oslo?", timing=timing)  # same words: kept
await agent.wait()
print("kept the early reply:", timing.adopted)

Its model calls bill under the ledger stage voice_speculation, and each one emits a custom.voice.speculation event (started, adopted, dropped). It saves the time the turn detector waits after the transcript: much with a silence end-of-turn, little with a fast semantic one.

Run read-only tools early, and nothing else

The gate sits in the executor, before guards, budgets and approval. While the reply is speculative, a tool call runs only if the tool is named in SpeculativeTools, declared effects="read_only" and has no confirm. It is checked when the agent is built and again on every call. Any other call waits until the turn is final, and never runs if the caller goes on. The model's words cannot change this.

@registry.tool(effects="read_only", cache_ttl=60)  # may run before the caller has finished
async def weather(city: str) -> str:
    """Today's weather for a city."""
    calls.append(f"weather {city}")
    return f"{city}: sunny, 21 C"


@registry.tool(effects="destructive")  # never runs until the turn is final
async def book_table(time: str) -> str:
    """Book a table."""
    calls.append(f"book {time}")
    return f"booked {time}"
  • max_per_turn (2) and max_per_minute (30) cap early calls; past them, calls wait.
  • Each early call is one unit of the ledger item voxcat:speculative_tool; price it like any metered item. Each emits custom.voice.speculative_tool with the tool name.
  • Give these tools cache_ttl: a dropped reply's call finishes into the cache, and the next reply shares it while it runs (in-flight sharing), so the tool runs once.

Acknowledge a slow reply, never a fast one

The agent learns how long its replies take to start. When a turn ends on real speech and the reply is expected later than fast_ms, a line plays at once, before the transcript is back. Otherwise it waits fast_ms and plays only if the reply has not started. A kept speculation or a cached answer gets none.

acks = Acknowledgements(["Mm-hm.", "Okay."], fast_ms=300, expect_ms=700)
speaker = RecordingSpeaker()
agent = VoiceAgent(model(delay_s=0.4), Executor(registry), speaker, speed=Speed(acks=acks))
await agent.warm_up()  # renders the lines; a VoiceSession does this at call start
await agent.user_said("What is on the menu?")
await agent.wait()
print("heard:", speaker.said)

Lines play only after they are rendered at call start, so an acknowledgement is never slower than the reply it covers. A speaker with no warm() (your own audio stack) speaks them its own way.

Replay answers to repeated questions

AnswerCache matches a question word for word (case and punctuation ignored), per tenant, for ttl_s. The agent stores a reply only when it opened the call and used no tool, so it depends on the question and the instructions alone. Questions under min_words (3) are never stored. put seeds known answers. answers.voice(tts) keeps each stored reply's audio and replays it with no synthesis; warm-up renders seeded answers.

answers = AnswerCache(ttl_s=300)  # one per process; wrap your TTS with answers.voice(tts)
answers.put("acme", "When do you open?", "We open at nine.")
speed = Speed(answers=answers, tenant="acme", warmup=Warmup(lines=["Hello, Acme here."]))
speaker = RecordingSpeaker()
agent = VoiceAgent(model(delay_s=1.0), Executor(registry), speaker, speed=speed)
print("warm-up steps:", sorted((await agent.warm_up()).ms))
await agent.user_said("when do you open")  # no model call: the stored reply plays
await agent.wait()
print("heard:", speaker.said, "cache hits:", answers.hits)

Use one tenant per agent persona: two agents with different instructions should not share.

Warm up at call start

VoiceSession.run calls agent.warm_up() as the call starts. It calls warm() on the STT, TTS, model and each of connections (or calls a plain function), renders the acknowledgements, lines and cached answers, and runs the prefetch tools through the executor, all at once. A failed step is logged and listed in the WarmReport; the call goes on. Prefetch tools follow the same rule as early tools: read-only, no confirm.

Read the stage timings

Each TurnTiming in session.timings has the stages of its turn, and the custom.voice_turn event carries them in stages_ms, in ms after the caller's last speech: early, ended, transcribed, speculated, replied, acked and first_audio, plus adopted and answer_cached.

Measured

python scripts/voxcat_speed.py runs the real pipeline with fake STT, LLM and TTS on a virtual clock, with approximate vendor-stated delays: STT final words 100 ms, first token 350 ms (900 ms for the slow model) then 40 tokens a second, first TTS audio 90 ms, a lookup tool 300 ms. Flags change them. Times are ms after the caller's last speech.

Scenario 800 ms silence EagerTurns (600 ms, 250 ms when done) 160 ms smart
plain question, no tricks 1395 845 755
plain question, speculative reply 895 695 655
tool question, no tricks 2045 1495 1405
tool question, speculative reply, tool waits 1695 1345 1305
tool question, speculative reply and tool 1445 1345 1305
slow model: first sound with an acknowledgement 805 255 165
repeated question: answer cache 805 255 165

With an 800 ms silence end-of-turn, the speculative reply saves 500 ms and early tools 250 ms more on a tool question. EagerTurns needs no model and no provider feature and lands close to a 160 ms semantic end-of-turn. The pipeline's own CPU time per 20 ms frame was about 25 µs at 16 kHz and 0.2 ms for 8 kHz phone audio (most of it resampling) on a laptop. Against LiveKit and Pipecat on the same parts: VoxCat benchmark.