> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# How replay scoring works

> How a scoring config picks the points a run replays, scores each reply, and turns those scores into each trace's.

How a scoring config picks the points a run replays, scores each reply, and turns those scores into each trace's.

## How a score is built

* **Points.** `points` says where in each trace a reply is asked for, in one step: `tools` says which of the recorded
  assistant turns can be replayed, by what production called there, and `take` which of those are. At each point,
  each subject gives its reply there.

  | `points` | Replays, in each trace, by production's recorded turn |
  | - | - |
  | `{ take: all }` | every assistant turn |
  | `{ take: last }` | the last assistant turn |
  | `{ take: { type: sample, k: 8 } }` | the first and last turns, and up to 8 between them, the same ones every run |
  | `{ tools: any, take: all }` | every turn that calls a tool |
  | `{ tools: none, take: last }` | the last turn that calls no tool: an answer in text, or an empty turn |
  | `{ tools: [escalate_to_human], take: all }` | every turn that calls `escalate_to_human` |
  | `{ tools: [escalate_to_human], take: last }` | the last turn that calls it, wherever in the trace that is |

  Picking by `tools` replays only those turns, so a run costs only their replies and judges, and exports only them.
  `points.tools` sees only the recorded turn, since nothing has replied before the replay: a subject that would
  escalate where production didn't is never asked there. To catch that, ask at every turn,
  `points: { tools: null, take: all }`, with a group whose `turns` match the call on the reply and whose criterion says
  when the call is due (as the [`ticket-agent`](https://docs.ambral.ai/examples/ticket-agent) example's `escalation` group does: its `escalates_to_right_team`
  answers `needless_escalation`; that example samples its points), or with a scorer that grades every reply.
* **A reply's score.** The `taskset` scorers score each reply. `fromBaseline` compares it with the baseline's, the reply
  the recorded trace holds at that point (did it make the same tool calls, does it say the same thing); criteria grade
  it on its own terms. A reply's score is the weighted mean of the scorers that count toward it (each `rewards` entry,
  judge and criteria group, by its `weight`); `metrics` are recorded beside it without counting, unless `required` names
  one. A scorer that doesn't apply at a point records nothing there and doesn't count.
* **`required` for what must never fail.** A scorer, a group or a criterion (`<group>/<criterion>`) named in
  `required` that scores below 1 makes its reply score 0; a criterion answered n/a doesn't. So a required criterion
  is yes or no: a middle choice scores below 1, and would fail its reply too.
* **The trace's score** (`episode`). With no `episode.rewards`, a trace scores the mean of its replies' scores, and
  one none of whose replies has a score has no trace score.
  `episode.rewards.replies` combines its replies' scores by `mode` (`mean`, `min`, `all-pass`: 1 only when every scored
  reply scores 1, or `any`: 1 when one does); `episode.rewards.trajectory` is one judge call over the whole session;
  each of `episode.rewards.checks` scores one rule once per trace (see "Score a rule that applies only sometimes, once
  per trace"). Once `episode.rewards` names any, the trace scores the mean of those alone, so reply scorers count toward
  it only through `replies` or a check.
* **Production's score.** Production is scored by criteria groups alone: every other scorer, `fromBaseline` and the
  trajectory judge, compares a reply with the baseline's, which production's is. So where the config has other
  scorers that count, production's per-trace score is the mean of its criteria alone, while a candidate's includes the
  rest too. Compare production with a candidate on the criteria rows of a run's summary, not the per-trace row,
  unless criteria groups are all that count.
* **What `ambral tasksets scoring set <workload>/<name> --file <file> --check` warns about.** A scorer that is recorded
  but counts toward nothing: when `episode.rewards` names a scorer but not `replies`, each reply scorer that no check
  there combines, and each `required` entry, which can't reach the trace's score (a criteria group counts when checks
  there combine it or each of its criteria; with only some checked, the warning names the rest); and otherwise a reward,
  judge or criteria group of weight 0 that `required` doesn't name, and no check under `episode.rewards` combines. Also
  a group's `turns` that `points.tools` keeps from running, and a check that a subject escapes by never making a call
  (see "When only some moments matter"). They are warnings, not mistakes: the config still saves.

## Score a rule that applies only sometimes, once per trace

A criterion with a `notApplicable`, or a group whose `turns` narrow it, scores only some replies. Per point, its mean
weights a model by how often it triggered the rule: one that touched refunds 40 times counts 8 times as much as one that
touched them 5 times, and a trace where the rule never applied disappears. A check scores the rule once per trace
instead. The [`chat-agent`](https://docs.ambral.ai/examples/chat-agent), [`coding-agent`](https://docs.ambral.ai/examples/coding-agent), [`ticket-agent`](https://docs.ambral.ai/examples/ticket-agent) and [`state-writes`](https://docs.ambral.ai/examples/state-writes) examples show checks of each kind.

* **When to add one.** For each criterion with a `notApplicable`, and each group with `turns`, that states a rule
  ("every time X, Y"), when the run replays more than one point per trace. With one point per trace, a check is its
  scorer's score there, and adds nothing (the [`retrieval`](https://docs.ambral.ai/examples/retrieval) example). Under `take: sample` it sees only the sampled
  replies.
* **n/a or a filter.** A filter makes no judge call, but sees only the tools a turn called (by name, `any` or `none`)
  and whether it is the last. A condition that takes judgment ("the reply speaks of a refund") is the criterion's
  `notApplicable`, which costs a judge call at each point. A check n/a in most traces is a sign a filter could save
  those calls.
* **`mode`.** `all-pass` for a rule: one breach fails the trace. `min` for a graded criterion, where a trace is as good
  as its worst reply. `mean` for a degree of quality, where one weak reply among good ones lowers the trace instead of
  failing it. `any` for at least once.
* **`whenNone`.** Keep `null` unless every trace must contain the action; then `fail`, as the [`coding-agent`](https://docs.ambral.ai/examples/coding-agent) example's
  `done_means_tested` does. Never `pass`: a model trained on the export learns that never acting always passes. `fail`
  works with a filter too. A rule about how a session ends ("every session ends with the ticket closed") can be met only
  at the trace's last point, where the subject replies to production's history: for a candidate, it measures whether it
  ends the session where production did, not whether it would finish its own.
* **Where it counts.** Under `episode.rewards`, it counts in the trace's score; add `replies` beside it to keep the
  replies counting too; a trace where it has no score leaves it out of the trace's mean, not 0, and a trace where no
  reward under `episode.rewards` has a score has no trace score. Under `episode.metrics`, it is recorded and counts for
  nothing. A check combines its scorer's own scores, which `required` doesn't change: `required` zeroes a reply's score,
  which reaches the trace only through `replies` (or, with no `episode.rewards`, the mean of its replies). So with
  checks alone under `episode.rewards`, `required` changes no trace's score; an `all-pass` check of the rule fails the
  trace instead.
* **A check or `episode.replies`.** `replies` combines each reply's whole score, every scorer together; a check, one
  scorer's. With one scorer the two nearly coincide: write the check, which names the rule a trace broke.

## When only some moments matter

Most configs grade every sampled turn. Where what matters happens at a few turns of a long session (a write to a
record, a message sent, an escalation), a targeted run beside the main config scores only those turns.
The [`state-writes`](https://docs.ambral.ai/examples/state-writes) example puts the steps together:

* **Select the turns** with `points: { tools: [<tool>, …], take: all }`, naming only the tools whose calls are the
  moments you grade: a run replays, pays for and exports only the turns where production called one. A config has one
  `points`, so asks that need different selections (only the writes, and every turn) are separate runs, each with its
  own `--scoring` file.
* **Grade each action where the reply makes it,** with a group per tool whose `turns` name it on the reply: no judge
  call where the reply doesn't make the call.
* **Score each rule once per trace** with a check over its group (see "Score a rule that applies only sometimes, once
  per trace").
* **Fail a reply that skips the call** with a `calls` check beside each such check, `{ calls: [<tool>] }`, under the
  same `episode.rewards` or `episode.metrics` (a metric backs no reward). `mode: any` passes a trace whose reply made
  the call at one point where production did; `all-pass` needs it at every such point. Without one, a subject that
  never makes the call has no score from the group's check, so the check can't fail it; `--check` warns at such a
  check. A rule that leaves replies out by `notApplicable` rather than by `turns` needs its `calls`
  check just the same, and `--check` won't flag it.
* **See where it ran.** A run's summary shows how often each group ran and each check scored, and how a check reads
  beside its `calls` check.
* **Run broadly now and then.** The same groups under `points: { take: all }` ask at every turn, so they see a subject
  that acts where production didn't, which the targeted run never asks.

**A `calls` check, `match` or `sensible_step`.** Use a `calls` check where only whether the action happens matters;
`match` or `same_tools` where the reply's calls must be production's, whole or by tool; and `sensible_step` where
acting a turn later, after a lookup or a question, can be right, at a judge call per point.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.