> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# Build and run a replay eval

> Build a taskset from your agent's recorded traces, set how its replies are scored, run candidate models on it, and read the results.

> [!NOTE]
>
> **Your coding agent can do everything on this page for you: see the [Quickstart](https://docs.ambral.ai/quickstart.md).** This page says
> what it does, and why.

A replay eval is a **taskset** of your agent's recorded sessions, with a **scoring config** that says where a run
asks for replies and how it scores them. A **replay run** replays each session to each model you name: at each
point the config picks, the model writes the next reply to the session as production lived it, and the config
scores that reply. It needs no environment, no tools and no test suite, only
[recorded traces](https://docs.ambral.ai/guides/connect-an-agent.md).

A workload usually covers several kinds of request: a support agent handles refunds, shipping questions and
account changes. Give each kind that matters its own taskset, named for it, such as `support-agent/refunds`, so
each score says how a model does on that kind.

## Build a taskset from recorded traces

List the workload's sessions, and read the promising ones whole:

```sh theme={null}
ambral traces list support-agent --label env=prod
ambral traces get tr_…
```

The list shows each trace's first user message, its tools and its steps. `--label env=prod` keeps real traffic,
when your agent [labels its sessions](https://docs.ambral.ai/guides/connect-an-agent.md#end-users-and-labels), so a test harness's
sessions don't end up in the taskset.

Choose traces of one kind of request, with variety within it: different requests, different tools, short and
long.

* **Pick the hard sessions, not only the common ones.** A request just past a limit of your policy beside one
  just inside it, and the sessions built to mislead: an instruction planted in a message, a request from someone
  without the authority, a third contact about the same problem. That is where models differ. A taskset of easy
  sessions scores them all alike.
* **Pick sessions the recorded model got right,** when the scoring will compare candidates with the baseline, the
  reply each trace recorded. Those replies are the answers every candidate is graded against, mistakes included,
  so read each reply before you pick its trace.

Then create the taskset from their ids:

```sh theme={null}
ambral tasksets create support-agent/refunds --traces tr_…,tr_…
```

A trace id takes the trace's latest branch with a reply. `tr_…:<node>`, as `ambral traces get` names a branch,
takes another. The branches are fixed when the taskset is created, so every run replays the same sessions. A
taskset's name is spelled as a workload's is.

### When the traffic is thin

Real traffic is mostly easy sessions, and a new workload has little of it. The sessions that tell models apart
sit on the lines your agent's policy draws: each limit, each rule that sends a case to a person, who may ask for
what. Where the traces lack them, record them on purpose:

1. Write inputs on both sides of each line, and the ones built to mislead, with each input's right outcome.
2. Run your agent on each as a scenario of an [offline run](https://docs.ambral.ai/guides/offline-runs.md), on a strong model of the
   workload's pool. Its sessions are recorded as the run's, apart from real traffic, and the run says which
   inputs the agent got right.
3. Build tasksets from the run's sessions as from any others: `ambral runs trace <run>` lists each one's `tr_…`.

### When to make a new taskset

A taskset's traces are fixed, and its scoring is versioned. When only how you grade changes, save the next
scoring version on the same taskset. Make a new taskset, such as `support-agent/refunds-v2`, from traces recorded
since, when what makes a good session has changed: a new policy, new tools, a different flow. Shipping a new
system prompt is such a point too, since `production` should be the agent as it runs today.

## Set the scoring

See [How replay scoring works](https://docs.ambral.ai/guides/scoring.md), and the examples, each a scoring config for one goal, starting with
[A support agent working tickets](https://docs.ambral.ai/examples/ticket-agent.md).

A scoring config is a YAML file you keep in your repo, beside your agent's code. Start it from a preset:

| Preset | When it fits |
| - | - |
| `strict` | A workflow whose tool calls must be the baseline's. At each point where either calls a tool, the reply must make the same calls with the same arguments. No judge calls. |
| `balanced` | Comparing turn by turn with the baseline, where a reply may differ in wording or approach and still be right. A judge decides at each scored point. |
| `trajectory` | Long, open-ended tasks where many paths are valid and their order varies from run to run. One judge call per trace and subject asks whether the subject would have handled the whole session. |
| `criteria` | What a good reply must do, in your own words, whatever the baseline did. It suits evals where the baseline isn't a bar to match. |

`ambral tasksets scoring preset` lists them, each with when to use it, and prints one to start from. A preset
with judges takes `--judge`, a model from `ambral models list`:

```sh theme={null}
ambral tasksets scoring preset balanced --judge <model> > evals/support-agent/refunds.yaml
```

Every field of the file is explained in a comment above it. Edit it: remove the scorers the eval doesn't need,
and add the ones it does. Most evals need a few criteria of their own, statements you write that a judge model
grades each reply against.

Then check the file, and save it once it passes:

```sh theme={null}
ambral tasksets scoring set support-agent/refunds --file evals/support-agent/refunds.yaml --check
ambral tasksets scoring set support-agent/refunds --file evals/support-agent/refunds.yaml
```

`--check` lists the mistakes it finds, each at its field, and saves nothing. Each save is the taskset's next
scoring version, so save once the file is right, not after each small change. To change the scoring later, edit
the same file, then check and save it again. `ambral tasksets scoring get` writes the saved version back out,
every default filled in.

## Run it

A run scores a taskset for its **subjects**, which you name with `--subjects`:

* **`production`** is what your agent sent: the replies the traces recorded. No model is asked for them, so it
  costs only the judges' calls. Include it whenever you compare models: a score of 0.88 means little alone, and
  four points below production means something. Scorers that compare a reply with the baseline's record nothing
  for it, since they would compare production with itself.
* **A model of the workload's pool** writes its own reply at each point. `ambral workloads get` lists the pool,
  and `ambral workloads pool add` adds a model to it.
* **A model under a prompt**, `<model>@<prompt>`, is asked with that prompt in place of the system prompt each
  trace opens with. The prompt is a text file, or the `pr_…` id that `ambral prompts put` printed for one. It is
  scored in a column of its own, so `production,<model>,<model>@pr_…` reads the prompt's effect beside the
  model's own.

Start the run, and wait for it:

```sh theme={null}
ambral runs start support-agent/refunds --subjects production,<model>,<model> --wait --budget <usd>
```

Each trace is replayed once per subject. A candidate's replay calls its model at every point the config scores,
then each of the config's judges scores the reply. So a run's time grows with its subjects, its traces and the
points in each: three subjects over ten short traces can take more than ten minutes.

* **`--wait`** stays until the run ends, then prints its summary. It gives up after 30 minutes, or after
  `--timeout`, and the run goes on. Without `--wait`, the command prints the run's id at once: read the run with
  `ambral runs show <run>`, and find its id again with `ambral runs list --taskset support-agent/refunds`.
* **`--budget`** caps what the run's calls may cost, in USD. Once they cost that much the run is cancelled, and
  keeps what it scored before then. Calls already in flight still cost, so size it with room to spare. Nothing
  prices a run before it runs: to size a large one, run 2 or 3 of its traces first, and multiply the cost per
  finished trial that `ambral runs show` prints by the large run's trials.
* **`--scoring <file>`** scores this run alone under that file, without saving it to the taskset, so you can
  compare a draft with the saved version.
* **`--label`** labels the run, such as `--label experiment=shorter-prompt`, so that
  `ambral runs list --label experiment=shorter-prompt` finds an experiment's runs together.

While a run runs, `ambral runs show` prints its health: `active`, `queued` and `retrying` all mean it is on its
way. `stalled` means nothing has happened for an hour, and it won't end by itself.

`ambral runs cancel <run>` stops a run in flight, and `--subject` stops one subject while the others finish.
`ambral runs resume <run>` scores the trials a cancelled run left unfinished, under the same config. With
`--budget`, it finishes a run its budget stopped, within a larger one that counts what the run has already spent.
The trials it finished keep their results and aren't paid for again.

While a run is in flight, wait for it rather than starting it again, which pays for the same replies twice. Once
it has ended, running it again is worth it to check that a result holds, since models and judges answer a little
differently each time.

## Read the results

Read a run from its summary down to single replies.

```sh theme={null}
ambral runs show <run>
```

The summary prints each subject's mean scores side by side, production's column first:

* **Per trace.** Each trace counts once, so compare models on this row.
* **Per scorer**, by its name: each reward, metric, judge, criteria group and criterion. The count in brackets is
  how many scores are behind a mean: a criterion that is n/a at most points has few, so a 1.00 over 3 points says
  little.
* **Cost per trace** is what a subject would cost to serve. A candidate's is an estimate, and production's is
  what production really cost.

Scores compare only within one scoring version, so compare runs whose header shows the same `scoring vN`.

Then find where a subject fell short:

```sh theme={null}
ambral runs verdicts <run> --below 1
ambral runs verdicts <run> --check <group>/<criterion> --verdict no --subject <model>
```

Each line is one scorer's result on one reply: the trace, the point, the subject, the scorer, the judge's
verdict, the score and the judge's reason. A scorer whose verdicts all agree with what you would say is working.
One whose reasons don't hold up is misfiring, and its config needs changing.

Read one trace reply by reply:

```sh theme={null}
ambral runs trace <run> tr_…
```

At each point it shows what led up to it, the baseline's reply, and each subject's reply, with the model call
that made it and each scorer's verdict and reason. Read it before you trust a scorer or change it.

### Change the scoring, and score again

When a scorer grades a reply differently from how you would, change the scoring file and rescore the run:

```sh theme={null}
ambral runs rescore <run> --scoring evals/support-agent/refunds.yaml --wait
```

It judges the same replies again and calls only the judges. Save the file as the taskset's next scoring version
once its verdicts read the way you mean them.

### Work on the failures

The traces where a scorer failed are the cases to work on. `ambral runs verdicts` with `--check` and `--verdict`
prints them on its last lines, as `Their traces: tr_…:<node>,…`, which `ambral tasksets create` takes as they
are. Run candidates on that focused taskset beside `production`, under the same scoring.

### Promote a candidate

A candidate that beats `production`, on a result that holds when you run it again, may be worth serving. A replay
run scores one reply at a time, so try the candidate on whole sessions first: with your own suite, as an
[offline run](https://docs.ambral.ai/guides/offline-runs.md), or on part of production. For that, create a second workload that serves the
candidate, send it a small share of your agent's sessions, and score both workloads' traces the same way.

Then make the candidate the model the workload serves:

```sh theme={null}
ambral workloads set-primary support-agent <model>
```

The Gateway serves it within about a minute, with no change to your agent. To roll back, run the same command
with the model the workload served before.

### Export a run

`ambral runs export <run> > traces.jsonl` writes every result as one line of JSON in verifiers' episode format,
to analyse the results or train on them outside Ambral. Export a run once it has ended: the file holds the whole
run only when it had, and the command exited 0.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.