Evals

An eval runs an agent scenario and checks the result. Define cases in evals/. Deterministic assertions inspect replies, tool events, and run metrics. Prose assertions are graded by an LLM judge or TypeSafe Jev.

Layout

evals/
├── refund-over-cap/
│   ├── eval.json
│   └── eval.md
├── ingest-report/
│   ├── eval.json
│   ├── eval.ts
│   └── report.pdf
├── triage.md
└── README.md

A directory case uses eval.md or eval.ts, with an optional eval.json manifest and fixtures. A flat Markdown file works for a case without supporting files. README.md is skipped.

Discovery is recursive. The ID is the path relative to evals/, without the file extension or trailing /eval: refund-over-cap or journeys/onboarding/first-run. Moving a case changes its ID, which identifies baselines and trends.

Top-level evals/traces/ is reserved for online trace evaluators and excluded from eval discovery.

eval.json: metadata only

{
  "tags": ["refusal", "write-path"],
  "objectives": { "turns": { "max": 4 }, "hitlRequests": { "min": 1 } },
  "runs": 5,
  "passRate": 0.8,
  "timeoutMs": 60000
}

The same fields can be written in YAML frontmatter. Declaring a field in both frontmatter and eval.json is a load error.

eval.md: the case

# Eval: Looks up the order before answering
 
Check that the answer comes from the order system.
 
## Setup
 
Context: The operator is viewing order ORD-1001.
 
## Query
 
"What's the status of ORD-1001?"
 
## Assertions
 
- tool_called_with: lookup_order {"order_id": "ORD-1001"}
- contains: delivered
- no_failed_actions
- Should state the order status clearly.
- Should NOT invent a delivery date the tool did not return.

Set repetitions with runs in metadata, not a Runs: line. Set scopeId and agent in metadata, not in ## Setup. A ## Objectives section is a load error.

Grading with TypeSafe Jev

Prose assertions use an LLM judge unless the case or the run selects TypeSafe Jev. Save grading settings in eval.json or frontmatter:

{
  "grader": {
    "kind": "typesafe",
    "model": "jev-latest",
    "passAbove": 0.7,
    "fallback": {
      "model": "anthropic/claude-haiku-4-5-20251001",
      "uncertain": { "low": 0.3, "high": 0.7 }
    }
  }
}

kind is typesafe or llm_judge; both accept an optional model. passAbove passes each prose assertion at or above a probability. For scored grading, use rubric: { "levels": ["Not satisfied", "Partly satisfied", "Fully satisfied"], "passAt": 2 } instead. Levels are numbered from zero, and a rubric accepts 2 to 10 nonempty levels.

The optional fallback sends uncertain answers to an LLM. Statement probabilities within uncertain, including both endpoints, use the fallback. A rubric's fallback uses confidenceBelow, such as 0.6, instead of uncertain. A failed grading call is reported as Not graded.

Deterministic checks, named scorers, and code-authored evals keep their own rules. In Studio, Run settings on the Evals page can override the saved grader for a run; Saved with each case uses the saved grader.

Standalone runtimes require TYPESAFE_API_KEY; hosted runs use the platform's TypeSafe configuration. Each Jev request has a 30-second ceiling. Responses over 100,000 characters require an LLM judge.

Repeating an eval

With "runs": 5, "passRate": 0.8, at least four of the five runs must pass. The required count is ceil(runs × passRate). Execution stops when the verdict is settled; here a second failure makes a pass impossible, so the remaining attempts are skipped.

Each attempt gets its own timeoutMs, bounded by the time left in the suite's --timeout. A deadline that stops the repeats does not reduce the number needed to pass. The result reports attempted, passed, required, and total runs, and includes the first failing attempt when the case fails.

The session an eval runs in

An eval without agent uses the chat route's default selection: agents/default/ if present, otherwise unscoped chat.

{
  "agent": "request-intake",
  "scopeId": "user-a",
  "maxSessionTokens": 4000,
  "scope": { "mission_id": "m-42" }
}

Each attempt creates a fresh session. Setting scopeId does not supply authenticated user claims.

The answer model is selected for the whole suite: amodal eval --model sets it when the CLI starts the runtime, and a --url target uses its own model. A case cannot pin an answer model or force a skill. Use skill_loaded: to check whether the model chose the expected skill.

Assertions

Deterministic keys

An unknown key shaped like snake_case: is a load error. Other text is judged prose.

Prefix a key with not or Should NOT to invert it. Use an argument-level negative when the tool may run for other records:

- tool_called_with: update_request {"request_id": "R-16"}
- not tool_called_with: update_request {"request_id": "R-11"}

Pair tool_succeeded with tool_returned when the value matters. A JSON subset such as tool_returned: combine {"sum": 14} distinguishes 14 from 1400; a substring check does not.

Judged prose

Conversation

## Conversation
 
- user: "Refund ORD-1002 in full, $940."
- deny: issue_refund
- user: "What would it take to approve it?"

Confirmations default to denied. The first approve: or deny: after a user message controls all of that turn's confirmation requests. Its tool name is a label, not a selector. Use separate user turns for actions needing different decisions, and assert the actual operation and arguments.

For a native OpenAPI write, denial happens before execution:

## Conversation
 
- user: "Record that Lee Park is adopting Biscuit."
- deny: shelter\_\_adoptResident
 
## Assertions
 
- hitl_requested: /residents/{residentId}/adopt
- not tool_succeeded: shelter\_\_adoptResident
- no_failed_actions

This assumes a shelter connection with confirmation required on the adoption operation. A confirmation question in ordinary prose does not exercise the runtime's approval gate.

Calling a tool directly

A tool: step tests a handler without relying on the model to choose it:

## Conversation
 
- tool: build_from_packet {"packet_id": "PK-1"}
- user: "What did the packet build?"
 
## Assertions
 
- subagent_called: package-reviewer
- tool_returned: build_from_packet {"status": "staged"}

Arguments must be a JSON object and can be omitted for a tool with no parameters. Missing required arguments fail validation before execution. Nested tool and subagent events are recorded for assertions. A script can contain only tool steps.

Direct invocation skips the outer permission checker, preToolUse and postToolUse hooks, and requiresConfirmation, so it can execute a write without asking. The handler's own checks still run. Test approval with user: and deny:. An approval step directly after a tool: step is a load error.

Attachments

Declare fixture files in the manifest and attach them to user turns by name:

{
  "attachments": [{ "path": "report.pdf" }, { "name": "annex", "path": "fixtures/annex-b.pdf" }]
}

- user: "Summarize this report and its annex." [attach: report.pdf, annex]

Paths resolve against the eval directory. A name defaults to the filename. Only user turns can attach files.

MIME type is inferred for .pdf, .png, .jpg, .jpeg, .gif, .webp, .txt, .md, .csv, .docx, .pptx, .xlsx, .odt, .odp, .ods, and .rtf. Other extensions require mimeType. Office and OpenDocument files are converted to text at ingestion.

An undeclared attachment name fails loading. An unreadable file or an unknown extension without a MIME type fails the eval when it runs, before the first model call. Assert a fact found only inside the fixture to verify that the document reached the model.

Objectives

Objectives are soft bounds used for ranking, not pass/fail. Supported metrics are turns, userTurns, toolCalls, redundantToolCalls, subagentCalls, hitlRequests, failedActions, replyChars, latencyMs, and costMicros. Each takes a min, a max, or "minimize". An unknown metric is a load error.

Use an assertion when a bound must fail the case: max_turns: 4 is a gate, while an objective on turns is a ranking signal.

Code-authored evals

Use eval.ts for branching conversations, custom predicates, or assertions scoped to one turn:

export const readsBothDocuments = {
  description: 'Read facts from documents as they arrive.',
  async test(t) {
    const first = await t.send('Read this report.', { attachments: ['report.pdf'] });
    t.require('report reached the model', () => /Project Cedar/.test(first.reply ?? ''));
    first.satisfies('does not invent the annex amount', () => !/1572/.test(first.reply ?? ''));
 
    const second = await t.send('Read its annex.', { attachments: ['annex.pdf'] });
    second.satisfies('reads the annex amount', () => /1572/.test(second.reply ?? ''));
    t.noFailedActions();
  },
};

Declare both attachments in the sibling manifest. One file can export several cases. A named export contributes to the case ID, such as ingest/readsBothDocuments. An exported object can set tags, objectives, runs, passRate, timeoutMs, judgeModel, agent, scopeId, and maxSessionTokens, which override the file's eval.json. Set scope, attachments, and grader in eval.json; t.send can also pass scope per turn.

The context t and returned turn objects expose calledTool, notCalledTool, calledSubagent, loadedSkill, toolOrder, messageIncludes, maxTurns, maxToolCalls, noFailedActions, succeeded, and satisfies(label, predicate). Methods on a returned turn inspect only that turn. calledTool(name, {input, output}) accepts a JSON subset or a predicate for each matcher.

Ordinary assertions collect failures and continue.

Scorers

A scorer is a reusable check in scorers/<name>/scorer.ts. Reference it with scorer: <name>.

const known = new Set(['1', '2.1', '2.2']);
 
export default {
  description: 'Every cited section exists in the reference index.',
  requires: ['finalMessage'],
  preprocess: ({ facts }) => {
    const cited = [...(facts.finalMessage ?? '').matchAll(/§\s*([\d.]+)/g)].map((m) => m[1]);
    return { cited, missing: cited.filter((id) => !known.has(id)) };
  },
  score: ({ pre }) => (pre.cited.length === 0 ? 1 : 1 - pre.missing.length / pre.cited.length),
  reason: ({ pre }) =>
    pre.missing.length ? `Unknown sections: ${pre.missing.join(', ')}` : 'All cited sections exist',
};

A scorer: assertion passes only when the score is 1. Scores are clamped to the range 0 to 1, and a scorer that throws scores 0.

requires declares the facts a scorer reads: finalMessage, toolCalls, toolArguments, subagentCalls, skillsActivated, hitlRequests, or turns. A scorer that requires a fact the run cannot supply, or an unknown fact name, scores 0 and names the missing facts.

Running

amodal eval
amodal eval journeys/
amodal eval '/^(smoke|journeys)\//'
amodal eval journeys/ --tag safety --tag p0
amodal eval --json
amodal eval --url https://your-runtime.example.com

The positional filter matches an ID, directory prefix, substring, or slash-delimited regex. Multiple tags are OR'd; the tag filter and positional filter are AND'd.

Without --url, the CLI starts the runtime that amodal dev runs, on an ephemeral port. With --url, it runs the evals provided by that runtime, with its configured providers. See CLI for the other flags.

The command exits nonzero if any case fails or no cases run. When the suite exceeds --timeout, the report retains completed results and identifies the incomplete run.

Running in Amodal

On the agent's Evals page, run one case or Run all. Filtering the list turns Run all into Run N shown. Selecting several models under Run settings compares them on the same cases.

Runs execute from the browser tab. Refreshing, closing the tab, signing out, or leaving the agent workspace aborts unfinished runs; navigating within the workspace does not. Stop run in the progress banner stops the active case and skips the queued ones. Completed results saved to the agent's history appear under Run history on the case.

Hosted eval sessions are not saved to conversation history. Agent and judge calls still produce traces and incur model usage charges.

Deployment triggers

Deployment evals are opt-in. For GitHub-connected agents, set ci.evals.runOn in amodal.json to run the suite after a successful deployment:

{
  "ci": {
    "evals": {
      "runOn": "production"
    }
  }
}

ci.evals also accepts runs, passRate, and judgeModel as suite defaults. A case that sets its own runs or passRate keeps both of its values, and a case's judgeModel overrides the suite's.

Cloud reads the configuration at the deployed commit. Missing or unreadable configuration, invalid JSON, and an unrecognized runOn value skip evals. A triggered run posts an amodal/evals commit status, which gates promotion; see Deployment API. Skipped evals post no status.

Amodal-hosted source has no deployment trigger; run its evals from the Evals page or with amodal eval.