Evals

An eval is a test case for an agent. It lives in evals/, sends the agent one or more messages, and checks what came back.

Assertions are graded in two tiers. A bullet that names a known key is checked in code — free, exact, identical on every run. Everything else is prose and goes to an LLM judge. Prefer the first tier: a judge costs a model call, varies between runs, and can only check what you thought to ask it about.

Layout

One directory per eval, mirroring agents/<name>/ and tools/<name>/:

evals/
├── refund-over-cap/
│   ├── eval.json          # manifest — metadata, never assertions
│   └── eval.md            # the case
├── ingest-opord/
│   ├── eval.json
│   ├── eval.ts            # code-authored instead of Markdown
│   └── opord.pdf          # a fixture the case sends
├── triage.md              # flat shorthand — fine for a one-off
└── README.md              # skipped, not run

The eval's id is its path relative to evals/, minus the extension — refund-over-cap, ingest-opord. Ids key baselines and trends, so they are derived rather than authored and a layout change must not move them.

Use the flat triage.md shorthand for a single-file case. Use a directory when the eval owns anything else: a fixture, a longer manifest, or a .ts body.

Nest as deeply as you like. Discovery is recursive and the id is just the path, so evals/journeys/onboarding/first-run/eval.md is the id journeys/onboarding/first-run. How you group — by feature, by risk, by test level, not at all — is yours to decide; nothing in the runtime depends on it. Select by path or by tag when you want a slice.

eval.json — metadata only

Structured config lives here so a report, a CI gate, or the optimizer can read it without parsing prose or executing code.

{
  "tags": ["refusal", "write-path"],
  "objectives": { "turns": { "max": 4 }, "hitlRequests": { "min": 1 } },
  "attachments": [{ "path": "opord.pdf" }],
  "runs": 5,
  "passRate": 0.8,
  "judgeModel": "claude-sonnet-5"
}

A flat file can carry the same fields in YAML front matter. Declaring one field in both front matter and eval.json is an error rather than a silent precedence rule — config that needs two files to agree will eventually disagree.

eval.md — the case

---
tags: [happy-path]
---
 
# Eval: Looks the order up before answering
 
The floor on `toolCalls` is the point: answering from memory would satisfy a
ceiling perfectly while being exactly the failure this checks for.
 
## Setup
 
Context: The operator is viewing order ORD-1001.
 
## Query
 
"What's the status of ORD-1001?"
 
## Assertions
 
- tool_called_with: lookup_order {"order_id": "ORD-1001"}
- contains: delivered
- no_failed_actions
- Should state the status without hedging.
- Should NOT invent a delivery date the tool did not return.

There is no ## Objectives section: objectives are metadata and belong in the manifest. Writing one is a load error that says so.

Repeating an eval

An agent is not deterministic. A case that fails one run in five is not the same as one that fails every run, and grading a single sample cannot tell them apart — the suite reports whichever way the coin landed.

{ "runs": 5, "passRate": 0.8 }

Five runs, four of which must pass. passRate defaults to 1, so runs: 5 alone means all five must pass — the strictest setting, and the right one for a safety invariant where an intermittent violation must not be averaged away.

The required count is rounded up: 0.8 over 5 runs needs 4, never 3.99 rounded down into a weaker gate than you wrote.

Repeats stop as soon as the verdict is settled. With 5 runs needing 4, a second failure decides it — no arrangement of the rest reaches 4 — so the remaining runs are not paid for. A case that fails immediately costs two model calls, not five.

Repeats run inside the eval's timeoutMs, not around it, so runs: 5 cannot outlive the budget the case declares: five runs share one ceiling rather than getting five of them. If the deadline cuts the repeats short, the verdict is judged against the declared total, so a partial sample cannot pass.

A failed eval reports its first failing run, not a passing one — surfacing the run that went wrong is the only reason to repeat. The tally rides along:

FAIL  refund-over-cap   4/5  3/5 runs  8210ms

The session an eval runs in

By default an eval runs the way the default chat does. These fields let it run the way a specific surface does — which is the only way to test surfaces that are reached by rooting a session in them rather than by asking nicely.

{
  "agent": "request-intake",
  "scopeId": "user-a",
  "maxSessionTokens": 4000,
  "scope": { "mission_id": "m-42" }
}

All of them work on EvalDefinition too, so one .ts file can target several surfaces.

There is no per-eval answer model, on purpose. Which model runs is a run-wide choice (amodal eval --model …), because a suite exists to be compared — across prompts, across models — and a case that pinned its own model would opt out of the comparison it belongs to. A case whose result depends on the model is a finding about the model, and should surface when the suite runs on it. judgeModel is the exception: pinning the grader stabilises grading without changing what is under test.

There is no per-eval skill, on purpose. Pinning one does not activate a skill — it removes every other skill from the prompt, a state no user reaches. What is worth testing is whether the agent loads the right skill on its own, and skill_loaded: asserts exactly that without pre-deciding the answer. An eval that force-injected a skill could pass on content the agent never reaches in production, which is worse than no coverage at all.

Assertions

Deterministic keys

A bullet shaped key: value where key is known is graded in code against facts derived from the run's event stream.

Negating a key

Any deterministic key can be negated with a not prefix. It passes exactly when the plain form fails.

- not tool_called_with: issue_refund {"amount": 940}
- not tool_returned: check_conflicts {"ok": false}
- not skill_loaded: escalation

This matters most for argument-level negatives — "the tool ran, but never with those arguments" — which no other key expresses. tool_not_called bans the tool outright, which is wrong when the tool legitimately runs for a different subject:

- tool_called_with: find_clear_frequency {"emitter_id": "EM-16"}
- not tool_called_with: find_clear_frequency {"emitter_id": "EM-11"}

That pair says retune the one, never the other — a rule a whole-tool ban cannot state and a judged bullet grades unreliably.

Should NOT <key>: … works too and means the same thing. It used to not: the key lookup saw Should NOT tool_called, missed, and quietly routed a deterministic assertion to the LLM judge. That is the historical reason tool_not_called exists as its own key. It stays — - tool_not_called: present reads better than the negated form — but it is now an alias rather than the only way to say no.

no_failed_actions is worth putting on almost every eval. Without it, a run where the agent's tools were broken and it improvised an answer gets graded on the improvisation.

tool_succeeded says a tool did not throw; tool_returned says what it produced. On a model-free tool: step the difference is the whole test — there is no reply for contains: to read, so without tool_returned a composite that returns the wrong number passes everything.

- tool_returned: combine {"sum": 14}

Prefer it to tool_result_contains for structured returns: 14 is a substring of 1400, and a subset match on the parsed object is not.

A handler that returns something other than an object is wrapped as {result: <value>}, so a tool returning a plain string is still reachable by subset — under result, not by matching the bare string:

- tool_returned: summarize {"result": "2 emitters remain in mission m-42."}

Use tool_result_contains when you want a substring of the output rather than a whole value — a long report where only one phrase matters, or output whose exact wording is not the thing under test.

A bullet that looks like a key (snake_case:) but names an unknown one is a load error, not prose. A typo would otherwise become a judged assertion that usually passes, quietly degrading the suite from deterministic to judged.

Judged prose

Anything else goes to the LLM judge.

• - Should … — required.
• - Should NOT … — required to be absent.
• - May … — permitted, not required. Graded and reported, but excluded from the pass/fail verdict, so an optional behaviour cannot fail a run.

Failures quote the observed state back, not just "failed":

✗ tool_called: issue_refund
  tool "issue_refund" was never called; observed: [lookup_order, load_skill]

Conversation

A run is not always one message. Answering a confirmation is part of the scenario, and an eval that cannot answer one can never exercise a write path.

## Conversation
 
- user: "Refund ORD-1002 in full, $940."
- deny: issue_refund
- user: "What would it take to approve it?"

Confirmations default to denied. A script that does not say otherwise should not be silently approving destructive writes.

## Setup context is folded into the first user turn only — restating it every turn repeats a premise the agent already has.

Calling a tool directly

A composite tool is a pipeline of code: it fans out to subagents and nested tools along a fixed path. Reaching it through a prompt tests two things at once, and when it fails the eval cannot say which one broke — a bad result might be a broken pipeline or a model that never called it.

- tool: invokes one directly. No model is involved.

## Conversation
 
- tool: build_from_packet {"packet_id": "PK-1"}

Arguments are a JSON object, and optional if the tool takes none. The tool's events — tool_call_start, its nested subagent cards, tool_call_result — land on the stream exactly as if the model had called it, so every assertion works unchanged:

- subagent_called: package-reviewer
- tool_called: check_conflicts
- no_failed_actions

A script may be tool steps only; it does not need a user turn. And because the call lands in session history, a tool step composes as a setup step for a conversation that follows:

- tool: seed_mission {"id": "m-42"}
- user: "What is in mission m-42?"

In a code eval the same lane is t.callTool, which returns a turn like send:

const built = await t.callTool('build_from_packet', {packet_id: 'PK-1'});
built.calledSubagent('package-reviewer');
built.calledTool('build_from_packet', {output: {status: 'staged'}});

calledTool takes an output matcher alongside input — a JSON subset, or a predicate over the raw output string for prose returns.

This lane is ungated

Permission checks, preToolUse / postToolUse hooks and requiresConfirmation all gate what the model chose to do. A direct invocation chose nothing, so none of them run — a tool that would normally stop for approval executes here without stopping.

So a direct call proves the pipeline works. It cannot prove a gate holds, and an eval that asserts one from a tool: step is asserting against a path the gate was never on. Writing deny: after a tool: step is a load error for that reason. Test a gate by driving the model — user: plus deny:.

Arguments are checked against the tool's declared parameters before it runs: a missing required argument or a name the tool does not declare is an error, not a call that passes undefined. Hand-written JSON has nothing upstream constraining it the way the provider constrains the model, and a typo would otherwise send the handler down its "nothing was asked for" branch — which the eval would then grade as though it were the real one.

Attachments

An eval whose scenario is "the operator uploaded an OPORD" has to be able to send one. Declare files in the manifest, reference them by name:

{ "attachments": [{ "path": "opord.pdf" }, { "name": "annex", "path": "fixtures/annex-b.pdf" }] }

- user: "Draft the mission record from this." [attach: opord.pdf, annex]

Paths resolve against the eval's own directory, which is why an eval that sends files wants the directory layout. The name defaults to the basename.

Supported: PDF, PNG, JPEG, GIF, WebP. Anything else needs an explicit mimeType — guessing changes how a provider reads the bytes.

It is a path, never inline base64: a 2 MB document pasted into a manifest stops the eval being reviewable in a diff. Three things fail at load rather than at grading, because each would otherwise produce a plausible-looking run against an empty-handed turn — a name the manifest does not declare, a file that cannot be read, and an unknown extension. Only a user turn may attach.

A useful habit: assert on a string that appears only inside the file. Then the eval proves the document arrived instead of assuming it.

Objectives

Soft bounds on the run, used for ranking rather than pass/fail.

{ "objectives": { "turns": { "max": 4 }, "hitlRequests": { "min": 1 } } }

Metrics: turns, userTurns, toolCalls, redundantToolCalls, subagentCalls, hitlRequests, failedActions, replyChars, latencyMs, costMicros.

min matters as much as max. An optimizer told to reduce interruptions is rewarded for an agent that stops asking before it refunds — a floor on hitlRequests is what stops that. An unknown metric is a load error, so a typo'd objective is never silently ignored.

Code-authored evals

Use eval.ts when the case needs control flow — a condition between turns, a predicate over the reply, or an assertion scoped to one turn rather than the whole run. One file may define several evals; a named export becomes part of the id (ingest/withAnnex).

// Imports nothing — the loader introspects the export.
export const disclosesAsDocumentsArrive = {
  description: 'Findings appear only once the document supporting them arrives.',
  objectives: { turns: { max: 8 } },
 
  async test(t) {
    const first = await t.send('Here is the OPORD.', { attachments: ['opord.pdf'] });
 
    // Everything after this measures the wrong thing if the file did not land.
    t.require('the OPORD reached the model', () => /EASTERN ANVIL/.test(first.reply ?? ''));
    first.satisfies('does not yet know the annex frequency', () => !/1572/.test(first.reply ?? ''));
 
    const second = await t.send('And the annex.', { attachments: ['annex.pdf'] });
    second.satisfies('reads the frequency from the annex', () => /1572/.test(second.reply ?? ''));
 
    t.noFailedActions();
    t.notCalledTool('present');
  },
};

t carries the same vocabulary as the Markdown keys — calledTool, notCalledTool, calledSubagent, loadedSkill, toolOrder, messageIncludes, maxTurns, maxToolCalls, noFailedActions, succeeded, plus satisfies(label, predicate) for anything else. The same methods exist on the object returned by send, where they are scoped to that turn.

• t.send(msg, { approve, attachments }) — drive one turn.
• t.require(label, predicate) — a precondition. Records like any assertion, then stops the script, because everything after a failed precondition fails for the same reason and the cascade hides which one mattered. It is a failure, not a skip.
• t.skip(reason) — abandon as not-applicable.
• t.log(message) — a note in the run's output.

Assertions are collected, never thrown. One failed check must not hide the four behind it.

Scorers

A scorer is a reusable, named check — deterministic, or a narrowly-scoped model call. Put it in scorers/<name>/scorer.ts and reference it as - scorer: <name>.

export default {
  description: 'Every section number the reply cites exists in the loaded corpus.',
  requires: ['finalMessage'],
 
  preprocess: ({ facts }) => {
    const cited = [...(facts.finalMessage ?? '').matchAll(/§\s*([\d.]+)/g)].map((m) => m[1]);
    return { cited, missing: cited.filter((id) => !KNOWN.has(id)) };
  },
 
  score: ({ pre }) => (pre.cited.length === 0 ? 1 : 1 - pre.missing.length / pre.cited.length),
  reason: ({ pre }) => (pre.missing.length ? `not in the corpus: ${pre.missing.join(', ')}` : 'all cited sections exist'),
};

Scorers earn their keep where a property is important, checkable, and would otherwise be graded by a model that shares the agent's blind spots — "did it fabricate a citation" being the clearest example. requires declares the facts it reads, so a scorer needing an unavailable field is rejected at registration rather than silently scoring wrong.

Running

amodal eval                       # every eval in the agent
amodal eval refund                # ids containing "refund"
amodal eval journeys/             # everything under a directory, any depth
amodal eval '/^(smoke|journeys)\//'   # a regex — slashes make it one
amodal eval --tag safety          # evals carrying a manifest tag
amodal eval --tag safety --tag p0 # several tags are OR'd
amodal eval journeys/ --tag safety    # path AND tag
amodal eval --json                # machine-readable
amodal eval --record-dir out/     # write training records for the optimizer
amodal eval --url https://…       # target a deployed runtime instead of booting one

The positional filter is an id, a directory prefix, a substring, or — wrapped in slashes — a regex, the same convention regex: assertions use. A malformed regex is an error rather than a silent zero-match, which would otherwise read as "nothing to run".

Without --url it boots the same local runtime amodal dev uses, on an ephemeral port. It exits non-zero if any eval fails or if zero evals ran — a filter that matches nothing is a mistake, not a pass.

A suite that exceeds --timeout keeps the results it already produced and reports how many of how many finished, rather than discarding the run.