Optimise your agent¶

The optimiser tunes the texts around any agent you can call: a plain function, or a LangGraph, OpenAI Agents or Pydantic AI agent. It runs your eval cases, tries edits, keeps what helped, and writes a patch for you to review.

Tuned: the texts you put on the surface (prompt/system, prompt/layout/<section>, tool/<name>/description, tool/<name>/prompt_hint). Never tuned: your code, the model or its settings, tool behaviour, guardrails.

The loop¶

flowchart LR
  C["cases.jsonl<br/>load_cases"] --> H["CallableHost<br/>run_case(tree, case, model)"]
  H --> B["probe base"] --> P["propose<br/>procedure + knowledge"] --> R["probe candidates"]
  R --> M["compose: admit what helped,<br/>drop what did not"] --> F["probe final"] --> D{"beats base?"}
  D -- yes --> O["harness.patch<br/>report.md"] --> A["you review<br/>load_harness → apply"]
  D -- no --> K["KEEP_BASE"]
  G["grader → Score"] -.-> B & R & F

Worked example (offline)	Does
`examples/harness_cases.jsonl`	8 eval cases: a topic in, the help-center article id expected
`examples/harness_agent.py`	the agent being optimised: one tool, no framework
`examples/harness_fakes.py`	fake agent and proposer models; delete it and pass real ones
`examples/harness_optimise.py`	host, grader, run, then apply the chosen tree
`examples/harness_frameworks.py`	the same agent on LangGraph, OpenAI Agents, Pydantic AI

1. Write your cases¶

One JSON object per line: id and input are required, expected and tags optional.

{"id": "returns", "input": "return an item", "expected": "HC-101", "tags": ["orders"]}
{"id": "refund", "input": "refund status", "expected": "HC-102", "tags": ["billing"]}
{"id": "shipping", "input": "shipping times", "expected": "HC-201", "tags": ["orders"]}
{"id": "tracking", "input": "track a parcel", "expected": "HC-202", "tags": ["orders"]}
{"id": "warranty", "input": "warranty claim", "expected": "HC-301", "tags": ["orders"]}
{"id": "password", "input": "reset my password", "expected": "HC-401", "tags": ["account"]}
{"id": "invoice", "input": "download an invoice", "expected": "HC-501", "tags": ["billing"]}
{"id": "cancel", "input": "cancel a subscription", "expected": "HC-502", "tags": ["billing"]}

load_cases(path) reads .jsonl, .json (a list) or .csv (a header row). A missing field, a duplicate id or bad JSON raises ConfigError naming the line.

2. Pick a grader¶

judge = contains()  # 1.0 when the expected article id is in the answer

Grader (`agent_adapter.optimiser.graders`)	Passes when
`exact()`	the answer equals `expected` (case and spacing ignored)
`contains()`	`expected` appears in the answer
`regex()`	the pattern in `expected` matches the answer
`code_check(fn)`	`fn(case, run)` returns `True`, a float in [0, 1] or a `Score`
`llm_judge(model, rubric)`	your judge model scores the answer against the rubric
`all_of(a, b, weights=...)`	the weighted mean of the graders that graded

expected may be a list. A grader never raises: an error logs a WARNING and leaves the case ungraded. llm_judge scrubs expected from its notes, which the proposer reads.

3. Wrap your agent¶

Name the tunable texts with their base values, then write run_case(tree, case, model), which reads each tuned text from tree.files:

SURFACE = {  # the texts the optimiser may tune, and their base values
    "prompt/system": "You answer customer-support questions.",
    "tool/lookup_article/description": "Look up the help-center article for a topic.",
    "tool/lookup_article/prompt_hint": "Use it when you are unsure.",
}

async def run_case(tree: HarnessTree, case: EvalCase, model: LLMPort) -> str:
    """One answer. Swap in your own agent here: read the texts you want tuned from `tree`."""
    files = tree.files
    system = files["prompt/system"]
    if hint := files.get("tool/lookup_article/prompt_hint"):
        system += f"\n\nWhen to use lookup_article: {hint}"
    tool = ToolDef(
        name="lookup_article",
        description=files["tool/lookup_article/description"],
        parameters=PARAMETERS,
    )
    messages = [
        Message(role="system", content=system),
        Message(role="user", content=str(case.input)),
    ]
    for _ in range(2):  # one tool round, then the answer
        request = LLMRequest(messages=tuple(messages), tools=(tool,), stage="agent")
        reply = await model.generate(request)  # use `model`: its calls are metered and cached
        if not reply.tool_calls:
            return reply.text
        messages.append(reply.message)
        for call in reply.tool_calls:
            result = TOOLS[call.name](**call.arguments)
            messages.append(
                Message(role="tool", content=result, tool_call_id=call.id, name=call.name)
            )
    raise RuntimeError("the agent asked for tools twice without answering")

host = CallableHost(
    run_case,  # your agent: (tree, case, model) -> answer text
    surface=SURFACE,  # the texts it may tune
    cases=load_cases(CASES),
    judge=judge,
    domain="support",
    required=["prompt/system"],
)

run_case returns the answer text, or a CaseRun with your own transcript and metrics. If it raises, the case counts as failed and the run goes on. Only paths on the surface can be edited; required paths can never be deleted or emptied.

4. Run offline first, then for real¶

config = OptimiserConfig(
    k=1, out_dir=out, lessons_path="", budget=OptimiserBudget(max_case_runs=40)
)
result = await optimise(
    host,
    config,
    proposer=ScriptedProposer(),  # swap in your proposer model here
    model=FakeFaqModel(),  # swap in your agent's model here
    ledger=Ledger(prices=FREE),  # your ledger and price table; optional
)

python examples/harness_optimise.py prints:

base 0.25 -> agg1 1.00  (chosen: agg1)
kept    prompt/system
dropped tool/lookup_article/prompt_hint
--- a/prompt/system
You answer customer-support questions. Always call lookup_article first, then cite the article id.

For real, pass your models in place of the fakes, or run the CLI inside examples/ (it imports --host from the current folder); then swap --dry-run for --model openai:gpt-5, which needs pip install "agent-adapter[openai]" and OPENAI_API_KEY:

cd examples
python -m agent_adapter.optimiser --host harness_optimise:build_host --dry-run

5. Read the decisions¶

The system-prompt edit was kept: it took the score from 2/8 to 8/8. The tool-hint edit was dropped: the agent looked every topic up but never cited the article, so no case improved. report.md in out_dir has the counts behind each line.

6. Apply what you accept¶

tree = load_harness(out)  # the tree you reviewed in harness.patch
print(tree.files["prompt/system"])  # feed it to your agent, as run_case does

Your agent	Apply the tree with
your own `run_case`	pass `load_harness(out_dir)` as `tree`
a `Registry`	`apply_tree(registry, tree)` for descriptions and hints
a `PromptLayout`	`layout_from_tree(layout, tree)` for the prompt and sections

7. Agents from other frameworks¶

The framework agent brings its own model, so pass NullModel() as model (CLI: --model none --proposer-model ...). Rebuild the agent per case from the tree:

# LangGraph
from langgraph.prebuilt import create_react_agent

from agent_adapter.integrations.langchain import structured_tools

tuned = apply_tree(registry, tree)  # tuned tool descriptions
tools = structured_tools(tuned, executor=Executor(tuned))
agent = create_react_agent(chat_model, tools, prompt=system_prompt(tree))
state = await agent.ainvoke({"messages": [("user", str(case.input))]})
return str(state["messages"][-1].content)

# OpenAI Agents
from agents import Agent, Runner

from agent_adapter.integrations.openai_agents import function_tools

tuned = apply_tree(registry, tree)
tools = function_tools(tuned, executor=Executor(tuned))
agent = Agent(
    name="support", instructions=system_prompt(tree), tools=tools, model=agents_model
)
result = await Runner.run(agent, str(case.input))
return str(result.final_output)

# Pydantic AI
from pydantic_ai import Agent

from agent_adapter.integrations.pydantic_ai import pydantic_ai_tools

tuned = apply_tree(registry, tree)
tools = pydantic_ai_tools(tuned, executor=Executor(tuned))
agent = Agent(pai_model, system_prompt=system_prompt(tree), tools=tools)
result = await agent.run(str(case.input))
return str(result.output)

Tools run through the Executor (approval, hooks, ledger). Each build_*_host raises MissingDependency when its SDK is not installed.

8. Budget and safety¶

OptimiserBudget(max_usd, max_case_runs, max_wall_s) caps the run; with k=1 and N cases it makes 4 × N case runs and 2 proposer calls.
Model calls through the model you are given are priced and counted. If your agent pays for its own model, return a CaseRun with metrics={"cost_usd": ...}: it counts toward max_usd. Report the case's total: only the part the ledger did not see is added.
case_timeout_s cancels a slow case (it counts as failed). Pass your confirm tools as withhold=[...]: transcripts then leave out their arguments.
Candidates are text only; no candidate code ever runs. Nothing is applied for you.