> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# Report and read an offline run

> Wire your suite to report each time it runs, check its first run, and read its runs across commits and models.

> [!NOTE]
>
> **Your coding agent can do everything on this page for you: see the [Quickstart](https://docs.ambral.ai/quickstart.md).** This page says
> what it does, and why.

## What an offline run is

A team that tests its agent with a suite of its own (Vitest, pytest, verifiers, Harbor, a script) keeps running
it where it runs today, on CI or a laptop, and it reports to Ambral as it runs. The suite runs each scenario and
scores it. The agent's model calls go through the Gateway, which serves them with the model the run chose and
records them. Each time the suite runs is an **offline run**.

An offline run holds two things for each run of a scenario:

* its **result**: what the suite reported, which is whether it passed, its rewards and the reasons for them;
* its **calls**: the agent's model calls the Gateway recorded, as a trace.

A [replay run](https://docs.ambral.ai/guides/replay-evals.md) scores one reply at a time on your agent's recorded production sessions. An
offline run reports what your own suite scores on whole sessions, each on the candidate's own path. They answer
different questions, and most agents should have both: see [Concepts](https://docs.ambral.ai/concepts.md#replay-runs-and-offline-runs).
The commands that read runs answer for both.

## Before you wire it

* **The suite's agent has to be on the Gateway.** An offline run records only the calls that go through it. A
  suite whose agent calls a provider with a key of its own, or answers from a mocked model or recorded
  cassettes, can't report one as it is: [connect its agent](https://docs.ambral.ai/guides/connect-an-agent.md) first, as you would for
  production.
* **The suite needs a key.** It reports with an Ambral key, the same kind its agent calls the Gateway with, read
  from `AMBRAL_API_KEY`. Set it where the suite runs: your shell or a `.env` file that git ignores on a laptop,
  and the CI's secrets on CI.
  [The SDK's settings](https://docs.ambral.ai/guides/offline-results.md#the-sdk%E2%80%99s-settings) lists the rest of what it reads from
  the environment.
* **Note each client that calls a model in a scenario.** Besides the agent's, a suite often has a judge, a
  simulated user, or a second agent the first one calls. Only some of them
  [get the run's headers](#which-clients-get-the-headers).

## Name the workload, the taskset and the scenarios

* **The workload** is the agent's, the one its production calls already name. Its pool, which
  `ambral workloads get` lists, is the models a run may choose. Add one with `ambral workloads pool add`.
* **The taskset** is the suite's name in Ambral, `<workload>/<taskset>`. Its first run creates it. Lead with the
  kind of case and end with `-scenarios`, so the suite sorts beside the replay tasksets of the same kind:
  `support-agent/refunds` and `support-agent/refunds-scenarios`. Every run of the suite, on every model and every
  commit, reports to the same taskset, which is what lets its runs be lined up. A taskset a typo created can't be
  deleted, so check the name before the first run.
* **A scenario's id** is what lines a scenario up across runs, so keep it stable: the same case keeps its id from
  run to run and commit to commit. Write it as a path, most general first, such as `refunds/over-the-limit`,
  since the commands that read runs roll scenarios up by the segments of their ids. Never put a run's index, a
  timestamp or a random id in it: every run would then be a suite of new scenarios, and no scenario would have a
  history. An id is at most 512 characters.
* **A check's name**, a reward's or a metric's, is how checks are lined up across runs. Name a check by the rule
  it tests, such as `policy_followed` or `solved`.

## Pick a harness

* [TypeScript: `@ambral/evals`](https://docs.ambral.ai/sdks/typescript): a Vitest suite, a script or another test runner.
* [Python: `ambral-evals`](https://docs.ambral.ai/sdks/python): a pytest suite, a script, a verifiers eval or a Harbor job.

Each does the same four things: it opens a run, hands each scenario's agent its headers, reports each scenario's
result, and closes the run.

### Which clients get the headers

Every model call of the system under test carries the scenario's headers, and nothing else does.

* **The agent's client: always.** Its calls are served the run's model and recorded as the scenario's session.
  The agent keeps sending `ambral/auto` as its model: the headers say which workload, run and model serve it.
* **A judge's, or a simulated user's: never.** Give them a client of their own, without the headers. A call that
  carries them is served the run's model, so the judge or the user changes with the model being tested, and its
  messages are recorded in the agent's conversation.
* **Another workload the agent calls**, such as a router or a sub-agent: give its client the headers too. A call
  that names another workload is served that workload's primary model, as in production, and recorded under the
  run beside the scenario's session.

When the suite calls the agent over HTTP, or starts it in a container, the SDK can't reach its model client. A
scenario's headers are two plain headers: hand them over by what the harness has, such as a request header the
agent's service forwards or an environment variable the container reads, and have the agent's client send both on
each model call.

## What a result holds

See [What a result holds](https://docs.ambral.ai/guides/offline-results.md).

## Repeats, labels and the model

* **Repeats.** A model answers a little differently each time, so a scenario that ran once can pass or fail by
  luck. Run each scenario 3 times or more where you can afford it. Each run of a scenario in one run is a repeat,
  and the commands that read runs count the repeats passed, as `2/3`. In Vitest, set `repeats` on the test, not
  `retry`, which runs a test again only after it failed. In verifiers, set `num_rollouts`, in Harbor
  `--n-attempts`, and in a script, loop. pytest has no repeats of its own: run the suite again under the same
  `AMBRAL_EVAL_KEY`, which reopens the run and adds to its repeats.
* **Labels.** Label each run with what you will look for it by: the commit and the branch it ran at, from the
  CI's own variables. `ambral tasksets results` prints each run's `commit`, and
  `ambral runs list --label branch=main` finds a branch's runs.
* **The model.** A run is served one model: the one the code names, else `AMBRAL_EVAL_MODEL`, else the workload's
  primary when the run opens. It must be in the workload's pool. To compare models, run the suite once per model,
  with no change to the code: a CI matrix that sets `AMBRAL_EVAL_MODEL` per job, or one invocation per model on
  a laptop.

## Cost and concurrency

A suite's model calls are paid for with the org's credits, at the same prices as production's, and they draw on
the same providers' rate limits. With automatic top-up on, a suite's calls can take the balance under its
threshold and charge the saved card, as production's calls do. An offline run has no budget. One run costs
roughly its scenarios, times its repeats, times the agent's calls in each: work that out before the first large
run, and read what a run cost with `ambral runs show <run>`.

Cap how many scenarios run at once, so a suite doesn't slow production's calls or hit a provider's limits:
Vitest's `maxConcurrency` and `maxWorkers`, pytest-xdist's `-n`, verifiers' `max_concurrent`, Harbor's
`--n-concurrent`. A few dozen at once is plenty for most suites.

## Where the suite runs

A run records where it ran, its **source**: `ci` when the `CI` environment variable is set, to anything but `0`
or `false`, and `local` otherwise. A suite's history leaves local runs out and counts them, so a developer trying
a change on a laptop doesn't move the suite's record. `--source all` reads them too.

* **On GitHub Actions, GitLab CI, CircleCI or Buildkite** there is nothing to set.
* **On Jenkins and Azure Pipelines**, which don't set `CI`, set `AMBRAL_EVAL_SOURCE=ci`, and `AMBRAL_EVAL_KEY`
  to the pipeline run's id, so its jobs share one run.
* **On laptops only, or a scheduled machine**, the history is empty until the runs of record say they are, with
  `AMBRAL_EVAL_SOURCE=ci` where they run.

The shards, workers and matrix jobs of one CI run report to one run. Each opens it by the same key, taskset and
model, and it completes once each has closed it. The key is the CI's id for the workflow run, or
`AMBRAL_EVAL_KEY`. A process that crashes without closing leaves a shared run open:
`ambral runs list --status open` lists the open runs.

## Check the first run

Run three scenarios before the whole suite, on a laptop or with `AMBRAL_EVAL_SOURCE=local`. A wiring mistake
shows in the first run, and costs less there.

**1. Run them**

The SDK's last line names the run and the command that reads it:

```text theme={null}
Ambral: closed run <run>, with 3 results from here. Read it with `ambral runs show <run>`.
```

pytest shows the line only with `-rP`. A report Ambral refused fails the suite with its reason: fix what it
says first.

**2. Wait half a minute, then read the run**

```sh theme={null}
ambral runs show <run>
```

The agent's calls reach Ambral about 30 seconds after its last one. Read the lines under the scores:

* **calls**: `3 of 3 results have their agent's calls recorded`. A result with no calls, 2 minutes after it
  was reported, means the agent's model client didn't send the run's headers, or the agent names its own
  sessions and the suite didn't pass the SDK its id.
* **results**: every session that made calls with the run's headers has a result. When some have none, which
  `ambral runs trace <run> --no-result` lists, the suite stopped before reporting them, or a client that
  isn't the agent's has the headers.
* **ids**: from the suite's second run on, how many scenario ids are new to the suite. All of them new means
  the ids aren't stable.

**3. Read one run of a scenario whole**

```sh theme={null}
ambral runs trace <run> 'refunds.eval.ts > over the limit' --repeat 0
```

Name it by the scenario's id, quoted, or by a `tr_…` that `ambral runs trace <run>` lists. Its conversation
should hold the agent's calls only: no judge's prompt, and no simulated user's turn.

**4. Check the scores read as the suite meant them**

`ambral runs show <run>` prints each reward's mean and the reason given its lowest score, and
`ambral runs verdicts <run>` each result's.

Once all four hold, run the whole suite.

## Read a suite's runs

A suite that reports its runs builds a history: a run per commit on CI, and a run per model when you compare
them. `ambral tasksets results` lines them up by scenario:

```sh theme={null}
ambral tasksets results support-agent/refunds-scenarios --by run --only fell
ambral tasksets results support-agent/refunds-scenarios --by model --only differ
```

| Question | What answers it |
| - | - |
| How did one run go? | `ambral runs show <run>`, then `ambral runs verdicts <run> --failed` for each failed reward and its reason. |
| What did a change do? | `--by run`: the newest runs of one model, side by side. `--only fell` keeps the scenarios that passed fewer repeats than in the run before. |
| Where does each model break? | `--by model`: each model's newest completed run. `--only differ` keeps the scenarios they split on. |
| Where to look first? | `--depth 1` rolls the scenarios up by the first segment of their ids. `--scenario` narrows to one area. |
| Is it noise? | `--only mixed` keeps the scenarios whose repeats disagreed. |

Columns compare only on what they share, so check that a comparison is sound before you read it: that the
columns ran the same scenarios, at the same commit, with enough repeats to tell a failure from luck. Then read
the scenarios that split, not the averages, since a mean score hides which cases moved.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.