Optimiser

Improve tool descriptions and prompts against eval cases (Optimiser).

agent_adapter.optimiser

Offline harness optimiser: Propose -> Probe -> Compose over a host's eval cases.

It writes a reviewable patch and report; nothing is applied to the host.

Judge = Callable[[EvalCase, CaseRun], Awaitable[Score | None]] module-attribute

RunCase = Callable[[HarnessTree, EvalCase, LLMPort], Awaitable[str | CaseRun]] module-attribute

BudgetUsage

Bases: BaseModel

What a run spent; stopped names the limit that ended it early ("" when none did).

CallableHost

Optimise any agent you can call: run_case reads the tunable texts from tree.

surface maps component paths to their base texts; only those paths can be edited. required paths may not be deleted or emptied. Model calls made through the model passed to run_case are metered and recorded; withhold names tools whose call arguments transcripts leave out (pass your confirm tools). Report metrics["cost_usd"] in a returned CaseRun when your agent pays for its own model.

Candidate

Bases: BaseModel

A harness proposal (or the base): its edits, the tree they produce and how it was made.

CaseRun

Bases: BaseModel

One eval case run under one harness: output, transcript, metrics and the judge's view.

Edit

Bases: BaseModel

One proposed change: content None deletes the path; reason cites the evidence.

FlowHost

Runs a flow with FlowRunner; edits node prompts (Flow.with_prompts) and hook configs.

Nodes read their prompt with node_prompt(ctx) and the model with node_model(ctx); hooks are tunable as in RegistryHost.

HarnessTree

Bases: BaseModel

Every editable harness component as path -> text.

diff(other)

What other adds, removes and modifies relative to this tree.

unified_diff(other, *, a='base', b='candidate')

A unified diff per changed path (--- a/<path>, +++ b/<path>); JSON pretty-printed.

with_edits(edits)

A copy with edits applied: str sets a path, None deletes it, omitted is inherited.

Host

Bases: Protocol

What the optimiser needs from an application: its harness, its cases, and a way to run.

Lesson

Bases: BaseModel

A transferable cue from one run, stored in lessons.json for later runs.

LessonsStore

lessons.json: lessons of every run, rewritten atomically on each add.

add(lessons)

Append lessons whose id is new; re-reads the file first so other writers are kept.

load()

Every stored lesson, oldest first; [] when the file does not exist.

Raises:

Type	Description
OptimiserError	the file is not a valid lessons file (it is never overwritten).

relevant(*, domain, tree, limit)

Newest first: lessons of domain whose components exist in tree (or name none).

NullModel

The agent model for hosts whose agent brings its own: every call raises ConfigError.

generate(request) async

Never answers.

Raises:

Type	Description
ConfigError	always.

stream(request) async

Never streams.

Raises:

Type	Description
ConfigError	always.

OptimiseResult

Bases: BaseModel

Everything one optimiser run produced; nothing in it has been applied to the host.

OptimiserBudget

Bases: BaseModel

Limits for one optimiser run; None means unlimited.

OptimiserConfig

Bases: BaseModel

Settings of one optimiser run (k is the main cost knob: (1 + 3k) * N case runs).

RegistryHost

Runs run_tool_loop over a registry; edits descriptions, hints, prompts and hook configs.

hooks run on tool calls after the executor's own hooks; tunable ones (with a config_schema()) are on the surface, safety hooks only with allow_safety_tuning=True. Put guardrails on executor=, which is never tuned. Tool hints are rendered into the system prompt.

layout_for(tree, registry)

The prompt layout the tree describes, with tool hints as a final section.

SurfaceCaps

Bases: BaseModel

Character caps per path pattern, the editable kinds, and JSON schemas of hook configs.

apply_tree(registry, tree)

A copy of registry with the tree's descriptions and prompt hints; tools without paths stay.

Raises:

Type	Description
OptimiserError	a tool path names a tool the registry does not hold.

layout_from_tree(base, tree)

The tree's system prompt, sections and tool hints; only base.dynamic is kept.

load_harness(out_dir)

The tree an optimiser run chose (or its base), read from <out_dir>/harness.after.json.

Raises:

Type	Description
ConfigError	the file is missing, not JSON, of another version, or holds a bad path.

optimise(host, config, *, proposer, model, composer=None, cache=None, ledger=None) async

Run Propose -> Probe -> Compose over the host's cases and write a reviewable patch.

model is the host's agent model; proposer writes edits; composer (default: the proposer) merges conflicts, reflects and selects without a judge. Every call goes through an InstrumentedModel with the response cache (namespace config.namespace), the ledger and a budget hook. Outputs land in config.out_dir; nothing is applied to the host. A run with the same inputs and out_dir resumes from state.json.

Raises:

Type	Description
ConfigError	invalid config values.
OptimiserError	the host has no cases, duplicate case ids or an invalid surface; the lessons file is invalid; a model error is fatal (bad key, rejected request); no base case ran; or every proposer of a stage failed on a model error.

Also exported here, documented under the canonical path: EvalCase, Score, load_cases.

agent_adapter.optimiser.graders

agent_adapter.optimiser.graders re-exports, documented under the canonical path: Check, JUDGE_SCHEMA, all_of, code_check, contains, exact, llm_judge, regex.