Evals

Run eval cases on any agent, grade them and report (Evaluate your agent).

agent_adapter.evals

Evals: run cases on any agent, grade the runs, and report for people and CI.

Names load on first use, so the pytest plugin can import this package cheaply.

CASE_EVENT = 'custom.eval.case' module-attribute

DEFAULT_EVALS_FOLDER = '.agent_adapter/evals' module-attribute

Grader = Callable[[EvalCase, Any], Awaitable[Score | None]] module-attribute

CallableTarget

fn(input) or fn(input, ctx=ctx), sync or async; take ctx so tool calls join the run.

__init__(fn, *, name='')

Raises: ConfigError: fn is not callable.

run_case(case, *, executor, ctx) async

The function's output for case.input (streamed text is joined).

CaseChange

Bases: BaseModel

One case that got worse or better between two reports.

CaseResult

Bases: BaseModel

One case: its run id (open it in the viewer), verdict, grades, cost and time.

ChatTarget

A chat-page agent fn(message, session); each case is a new one-turn conversation.

__init__(agent, *, name='')

Raises: ConfigError: agent is not callable.

run_case(case, *, executor, ctx) async

The agent's reply text.

Raises:

Type	Description
ToolUsageError	the reply is neither text nor an async iterator of text.

EvalCase

Bases: BaseModel

One evaluation input; expected is for the graders only.

EvalComparison

Bases: BaseModel

How a report differs from a baseline: regressions, improvements, added and removed cases.

regressed property

True when a shared case got worse, or a baseline case is missing now.

EvalReport

Bases: BaseModel

Every case result of one eval, with totals; render it as JUnit, Markdown or JSON.

by_grader property

Totals per grader over the cases it saw; an ungraded case counts as 0.

by_tag property

Totals per tag; a case counts once under each of its tags.

cost_usd property

Total cost of every case run.

failed property

The cases that did not pass.

mean_score property

Mean case score; errors and ungraded cases count as 0.

n_cases property

How many cases ran.

n_errors property

How many cases failed to run or did not finish.

n_passed property

How many cases passed.

pass_rate property

Share of cases that passed.

compare(baseline, *, tolerance=0.0)

This report against baseline: a case regresses when it stops passing or loses more than tolerance of score.

from_json(text) classmethod

A report from to_json output; the totals are recomputed.

Raises:

Type	Description
ValidationError	not a report.

to_json(*, indent=2)

The report, with its totals, as JSON (EvalReport.from_json reads it back).

to_junit()

JUnit XML: one test case per eval case, failures and errors with their reasons.

to_markdown()

A Markdown summary: totals, per-tag table and the failing cases.

EvalRun dataclass

What graders see of one case run; tool_calls and cost_usd are None when unobserved.

FlowTarget

A Flow run by FlowRunner; initial_state(case) builds the state, output reads it.

__init__(flow, *, initial_state, output=None, model=None, name='')

Raises: ConfigError: flow is not a Flow or initial_state is not callable.

run_case(case, *, executor, ctx) async

output(final state) (the state itself without output).

GradeResult

Bases: BaseModel

One grader's verdict on one case; score None means it did not grade.

GroupSummary

Bases: BaseModel

Totals of a group of cases (one tag, or one grader).

LLMJudge

A grader that asks model to score each answer against rubric; it never raises.

Case data and the answer are escaped, so they cannot close their tags. The judge's own passed is ignored unless trust_passed=True: the case passes on value >= pass_at. Expected values are scrubbed from the notes. Any LLMPort works.

__call__(case, run) async

The score, or None (logged) when the model or its reply fails.

__init__(model, rubric, *, stage='judge', show_expected=True, trust_passed=False)

Raises: ConfigError: an empty rubric.

grade(case, run) async

The model's score for run.

Raises:

Type	Description
ModelError	the model call failed.
ValidationError	the reply is not a valid score.

Outcome dataclass

A target's output, and why it did not finish ("" when it did); unfinished is an error.

RecordedRun dataclass

One recorded run: its key, id, status, whether content was kept, events and tags.

RunCase dataclass

An eval case made from one recorded run; source_run is that run's id.

record()

The JSONL object (load_cases reads id, input, expected, tags).

RunCases dataclass

The cases made from a runs folder, and the runs skipped.

Score

Bases: BaseModel

A grader's verdict on one run; value is in [0, 1] and notes is the reason.

reason property

The grader's reason (the same text as notes).

SkippedRun dataclass

A run that gave no case, and why.

Target

Bases: Protocol

Runs one case in the given run (ctx) on executor; returns the output or an Outcome.

ToolLoopTarget

run_tool_loop over registry with an optional system prompt; output is the answer.

__init__(model, registry, *, system='', max_steps=8, output_schema=None, name='tool_loop')

Raises: ConfigError: registry is not a Registry.

run_case(case, *, executor, ctx) async

The final text (or parsed output with output_schema).

Raises:

Type	Description
ApprovalPending	a tool call is waiting for an approval.

as_target(target)

target as a Target: a Target as is, a 2-argument callable as a chat agent, else a callable.

Raises:

Type	Description
ConfigError	a Flow (wrap it in FlowTarget), or not a callable of one or two arguments.

assert_eval(target, case, graders, *, executor=None, timeout_s=300.0, pass_at=1.0)

Run one case (in a test, say); the result, or AssertionError with the reasons.

Raises:

Type	Description
AssertionError	the case errored or did not pass.
ConfigError	a bad target, grader or option.

cases_from_runs(folder, *, run_keys=None, status=None, tags=(), expected_from='output')

Eval cases from the recorded runs in folder (a runs folder or one run file).

run_keys picks runs; status keeps runs with that status (ok, failed, paused, unfinished); tags keeps runs carrying every one of these run tags. Runs without recorded content are skipped with a reason.

Raises:

Type	Description
ConfigError	a missing folder, a bad expected_from, or a file that is not a run.
OSError	the folder cannot be listed.

eval_executor()

The default eval executor: an empty registry (no stray tools) and a fresh ledger.

load_cases(path, *, id_field='id', input_field='input', expected_field='expected', tags_field='tags')

Cases from JSONL (one object per line), JSON (a list of objects) or CSV (a header row).

id and input are required; expected and tags are optional. Tags are a list, or a string split on , or ;. CSV cells are text; an empty expected cell means none. Other fields are ignored.

Raises:

Type	Description
ConfigError	unreadable file, unknown suffix, bad JSON, a missing field, duplicate ids, or no cases.

run_eval(target, cases, graders, *, concurrency=4, executor=None, name='', timeout_s=300.0, pass_at=1.0, save_to=None, capture_content=True) async

Run each case on target, grade it with every grader, and return the report.

Each case is its own run on executor (default: eval_executor()), so its hooks, approvals, budgets, ledger and sinks apply, and a recorder attached to it records the run. A grader without its own verdict passes at score >= pass_at; a grader that errors or does not grade fails the case and scores 0. A failed grade adds at most pass_at to the case score. save_to also writes the report into that folder (DEFAULT_EVALS_FOLDER is the one the viewer's Evals view reads). With capture_content=False the report keeps no input, expected value, output or judge notes.

Raises:

Type	Description
ConfigError	no cases, duplicate case ids, no graders, or a bad option.
OSError	save_to cannot be written.

run_eval_sync(target, cases, graders, **options)

run_eval for sync code (scripts, tests).

Raises:

Type	Description
ConfigError	as run_eval.

save_report(report, folder=DEFAULT_EVALS_FOLDER)

Write report as <folder>/<key>.json (atomically) and return the path.

Raises:

Type	Description
OSError	the folder cannot be created or the file cannot be written.

write_cases(path, cases)

Append cases to a .jsonl file, skipping ids it already holds; (written, skipped ids).

Raises:

Type	Description
ConfigError	not a .jsonl path, or an existing file that cannot be read.
OSError	the file cannot be written.

agent_adapter.evals.advisor

Cost advisor: find a cheaper model per flow node or stage that keeps the eval score.

COMBINED = 'combined' module-attribute

AdvisorReport

Bases: BaseModel

Per-step model advice from evals and the ledger; apply mapping yourself, never auto.

advised property

{step: candidate} per step on its own, verified together or not.

mapping property

The advice to apply: advised once verified together, else empty (see note).

from_json(text) classmethod

A report from to_json output.

Raises:

Type	Description
ValidationError	not an advisor report.

to_json(*, indent=2)

The report as JSON (AdvisorReport.from_json reads it back).

to_markdown()

A Markdown summary: the baseline, one row per step, and the mapping to apply.

StepAdvice

Bases: BaseModel

The advice for one step: the cheapest candidate that keeps the score, if any.

StepModels

An LLMPort that answers each step with steps[step], else with baseline.

Steps are node:<name> (any call made inside a flow node) or stage:<stage> (a tool loop, agent or other stage); the key "*" covers every step not listed. Pass it as FlowRunner(model=...), ToolLoopTarget(model, ...) or run_tool_loop(model, ...). A routed port gets the request with model=None, so it uses its own model.

Raises:

Type	Description
ConfigError	baseline or a step model is not an LLMPort, or a step key is empty.

cache_key_fields()

Every routed port's key material, plus the flow node running now (if any).

generate(request) async

The reply of the port mapped to the current step.

Raises:

Type	Description
ModelError	as the chosen port.

port_for(step)

The port that answers step.

stream(request) async

The stream of the port mapped to the current step.

Raises:

Type	Description
ModelError	as the chosen port.

StepUse

Bases: BaseModel

What one step's model calls cost in one eval run, summed over its cases.

Trial

Bases: BaseModel

One eval run: model answers step and every other step stays on the baseline.

advise(target_factory, cases, graders, *, candidates, baseline, per_step=True, min_score_ratio=0.98, prices=None, max_spend=None, max_output_tokens=DEFAULT_MAX_OUTPUT_TOKENS, executor=None, concurrency=4, pass_at=1.0, verify=True, name='') async

Run the evals on baseline, then each cheaper candidate per step; report the advice.

target_factory(model) builds the eval target around the given port (pass it on as FlowRunner(model=...), ToolLoopTarget(model, ...), ...). Steps are the flow nodes and stages that called it. Each trial swaps one step's model and keeps the rest on the baseline; with per_step=False a candidate replaces every step at once. prices is a Prices or a table mapping (default: the executor ledger's, else the built-in table). max_spend (USD) caps every model call the advisor makes, LLMJudge graders included: each holds its worst-case cost (output limited to max_output_tokens) before it runs and settles the real cost after; the advisor stops early when a hold does not fit. verify reruns the evals with all advice at once; report.mapping holds the advice only when that run kept the score (else it is empty and report.note says why). Nothing is applied: pass report.mapping to apply_mapping yourself.

Raises:

Type	Description
ConfigError	a bad option, an unknown baseline, a non-port candidate, or, with max_spend, a candidate or judge model without a token price.

advise_sync(target_factory, cases, graders, **options)

advise for sync code (scripts, tests).

Raises:

Type	Description
ConfigError	as advise.

apply_mapping(baseline, mapping, candidates)

A StepModels from an advisor mapping {step: candidate name}.

Raises:

Type	Description
ConfigError	a mapped name is not in candidates.

current_step(request)

The step of a model call running now.

entry_step(entry)

The step of a ledger entry, from its metadata["node"] and stage.

step_of(stage, node='')

The step of a model call: node:<node> inside a flow node, else stage:<stage>.

agent_adapter.evals.from_runs

Turn recorded runs into eval cases: production runs become regression cases for CI.

ExpectedFrom = Literal['output', 'none'] module-attribute

case_from_run(run, *, expected_from='output', case_id='', extra_tags=())

A case from one run's recorded content, or why there is none (never invented data).

Raises:

Type	Description
ConfigError	a bad expected_from.

Also exported here, documented under the canonical path: RecordedRun, RunCase, RunCases, SkippedRun, cases_from_runs, write_cases.

agent_adapter.evals.graders

Ready-made graders async (case, run) -> Score | None; they never raise (an error is None).

Check = Callable[[EvalCase, Any], CheckResult | Awaitable[CheckResult]] module-attribute

JUDGE_SCHEMA = OutputSchema(name='judge_score', json_schema={'type': 'object', 'properties': {'value': {'type': 'number', 'description': '0 (fails) to 1 (fully meets the rubric)'}, 'passed': {'type': 'boolean'}, 'notes': {'type': 'string', 'description': 'one short reason'}}, 'required': ['value', 'passed', 'notes'], 'additionalProperties': False}) module-attribute

all_of(*graders, weights=None)

Weighted mean of the graders that graded; passed only if every one graded and none failed.

A sub-grader that errors or does not grade makes passed False (fail closed); passed is None only when every sub-grader gave a value without a verdict.

Raises:

Type	Description
ConfigError	no graders, or weights of the wrong length, negative or summing to 0.

code_check(fn)

Your check: fn(case, run), sync or async, returns a bool, a float in [0, 1] or a Score.

contains(*, normalise=True)

1.0 when the expected text (or one of a list of them) appears in the output.

exact(*, normalise=True)

1.0 when the output equals the expected text (or one of a list of them).

json_schema(schema)

1.0 when the output (JSON text, a dict or a model) matches schema.

Checked keywords: type, enum, const, bounds, lengths, properties, required, additionalProperties, items, anyOf and oneOf; others are ignored.

Raises:

Type	Description
ConfigError	schema is not a JSON object.

llm_judge(model, rubric, *, stage='judge', show_expected=True, trust_passed=True)

The optimiser's form of LLMJudge: it keeps the judge's own passed by default.

Raises:

Type	Description
ConfigError	an empty rubric.

max_cost(usd)

Passes when the run cost at most usd (from the ledger, else the events).

Raises:

Type	Description
ConfigError	usd is not a finite number >= 0.

max_latency(seconds)

Passes when the run took at most seconds of wall time.

Raises:

Type	Description
ConfigError	seconds is not a finite number >= 0.

regex(pattern=None, *, flags=re.IGNORECASE)

1.0 when pattern (default: the expected pattern or list of them) matches the output.

Raises:

Type	Description
ConfigError	pattern is not a valid regular expression.

tools_called(*names, forbid=(), only=False)

Every tool in names was called, none in forbid was, and with only no other one.

value is the share of names called (0 when a forbidden or extra tool ran).

Raises:

Type	Description
ConfigError	no names and nothing forbidden, or a name that is not text.

Also exported here, documented under the canonical path: Grader, LLMJudge.