Cache model answers across workers

A FAQ bot gets the same question all day. Pay for it once, and share the answer between the web workers and the batch job.

flowchart LR
  W1["web worker<br/>executor.model(port, cache_ttl)"] --> K["key: provider, model, request"]
  W2["batch worker"] --> K
  K --> R[("Redis<br/>prefix llm:")]
  R -- hit --> H["cached=True, $0"]
  R -- miss --> P["paid call, then store"]

Workers

def worker() -> Executor:
    """One process's executor; in real code RedisCache("redis://cache:6379/0", prefix=...)."""
    cache = RedisCache(client_factory=lambda: FakeAsyncRedis(server=server), prefix="llm:")
    return Executor(Registry(), cache=cache)


model = CallableModel(answer, provider="host", model="house")
web, batch = worker(), worker()

RedisCache takes a URL or a client_factory; here the factory returns fakeredis, so the example runs offline. A Redis error is logged at WARNING and counts as a miss.

Ask

ask = LLMRequest(
    messages=(Message(role="user", content="What is the returns policy?"),), stage="faq"
)
first = await web.model(model, cache_ttl=86_400).generate(ask)  # a miss: pays, stores
again = await batch.model(model, cache_ttl=86_400).generate(ask)  # another worker: a hit
other_stage = ask.model_copy(update={"stage": "triage"})  # stage is not part of the key
third = await batch.model(model, cache_ttl=86_400).generate(other_stage)
fresh = ask.model_copy(update={"response_cache": False})  # opt out for this request
fourth = await web.model(model, cache_ttl=86_400).generate(fresh)
print([r.cached for r in (first, again, third, fourth)], paid_calls["n"])

[False, True, True, False] 2

Part of the request	In the key?
provider, model, messages, tools, options	yes
stage, stream, metadata, response_cache	no

Pitfalls

Pitfall	Do instead
Answers that depend on who asks (hidden user data, tools with side effects)	put the user in the request, or opt out with response_cache=False
No cache_ttl	entries never expire: set one shorter than your content's lifetime
Expecting a stream to be cached	streams are never stored
Confusing it with the provider's prefix cache	that is a discount on repeated prompt prefixes (Cache)

Files

File	Does
examples/cookbook_llm_cache.py	two workers sharing one Redis, four requests
tests/docs/test_cookbook.py	runs it offline and checks the output