Retry a flaky tool

A billing API drops connections now and then. Retry those failures with backoff. Do not retry bad input: the model must fix it instead.

flowchart LR
  C["call"] --> A["attempt n<br/>under timeout"]
  A -- ok --> OK["ToolResult ok"]
  A -- "error" --> R{"retryable?<br/>attempts left?"}
  R -- yes --> W["wait Backoff.delay(n)<br/>tool.retry event"] --> A
  R -- no --> E["ToolResult error"]

The tool

class UpstreamBusy(ToolError):
    """The upstream said "try later": a ToolError the executor may retry."""

    retryable = True


@registry.tool(retries=3, timeout=2)  # up to 4 attempts, each at most 2 s
def fetch_invoice(number: str) -> dict[str, str]:
    """Fetch an invoice from the billing API."""
    if not number.startswith("INV-"):
        raise ToolError("invoice numbers start with INV-")  # not retryable: the model fixes it
    if failures["left"]:
        failures["left"] -= 1
        if failures["left"] % 2:
            raise ConnectionError("reset by peer")  # a foreign exception: retried
        raise UpstreamBusy("billing API busy")  # retryable = True: retried
    return {"number": number, "status": "paid"}

Raised	Retried?	Why
a foreign exception (ConnectionError, httpx.ReadError, ...)	yes	the adapter cannot know it is permanent
ToolTimeout (the timeout= expired)	yes	retryable = True
ToolError	no	the message goes to the model, which can change its arguments
your ToolError subclass with retryable = True	yes	"try later" from an upstream
ToolArgsInvalid, approval, budget and guard errors	no	another attempt gives the same answer

The executor

events = EventLog()
executor = Executor(
    registry,
    backoff=Backoff(base_s=0.5, max_s=8.0, multiplier=2.0, jitter=0.25),
    sink=events,
    sleep=no_wait,  # leave out in real code; tests inject it
    rng=lambda: 0.0,  # no jitter, so the delays below are exact
)

The delay after attempt n is min(max_s, base_s × multiplier^(n-1)) × (1 + rng() × jitter). The defaults are Backoff(base_s=0.5, max_s=30, multiplier=2, jitter=0.25).

Run it

failures["left"] = 3  # three failures, then success
ok = await executor.run_call(ToolCall(name="fetch_invoice", arguments={"number": "INV-7"}))
print(ok.status, ok.attempts, [e.data["delay_s"] for e in events.of_type("tool.retry")])

bad = await executor.run_call(ToolCall(name="fetch_invoice", arguments={"number": "7"}))
print(bad.error.kind if bad.error else "", bad.attempts)  # ToolError: one attempt only

failures["left"] = 9  # never recovers
down = await executor.run_call(ToolCall(name="fetch_invoice", arguments={"number": "INV-8"}))
assert down.error is not None  # every attempt failed
print(down.error.kind, down.attempts, down.error.retryable)

ok 4 [0.5, 1.0, 2.0]
tool-error 1
tool-failed 4 False

When every attempt fails with a foreign exception, the result is tool-failed. Its message names the tool and the attempts, never the exception text (that is logged at DEBUG only).

Pitfalls

Pitfall	Do instead
Retrying a tool with side effects (a payment, an email)	retry only idempotent tools, or pass an idempotency key to the upstream
No timeout= on a network tool	one hung call holds the run forever; a timeout also makes the hang retryable
Catching the exception inside the tool and returning None	raise it: the executor retries it, logs it and reports it
Real sleeps in tests	inject sleep= and rng= into Executor

Files

File	Does
examples/cookbook_retry.py	the tool, a retryable ToolError, the backoff and three runs
tests/docs/test_cookbook.py	runs it offline and checks the output