> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ambral.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set Ambral up in a project, run `npx ambral@latest skill install` in the project's root. It installs Ambral's skill, matched to the CLI's version, and the skill says how to do everything these pages describe with the `ambral` CLI. Prefer it to working from these pages. Never ask anyone for an Ambral API key, and never handle one: the person creates it in the web app and stores it where their app reads its secrets.

# ambral runs

> Start score runs of a taskset, read what they found, rescore those that completed, cancel those still running, and resume those cancelled

Start score runs of a taskset, read what they found, rescore those that completed, cancel those still running, and resume those cancelled

Every ambral command also takes the [global flags](https://docs.ambral.ai/cli/reference/global-flags.md).

## ambral runs start

```text theme={null}
ambral runs start [flags] <taskset>
```

Start a score run of a taskset: each of its traces is replayed for each subject, and each reply scored. It scores under the taskset's current scoring config, or `--scoring`'s. Production's replies are the ones the taskset's traces recorded, so scorers that compare a reply with those record nothing for it. A run takes minutes, longer with more subjects, traces and points, and judge calls at each: three subjects over ten short traces can take more than ten minutes. Without `--wait` it prints the run's id at once; read it with `ambral runs show <run> --json` until its `run.status` is no longer `running` (it ends `completed`, `failed` or `cancelled`), and find it again with `ambral runs list --taskset <workload>/<taskset>`, or by a `--label` it was started with. A rescore of it keeps its labels

| Argument | Takes | Description |
| - | - | - |
| `<taskset>` | string | The taskset, as `<workload>/<taskset>`: its workload's name, then its own. A taskset's name is spelled as a workload's: lowercase letters, digits, `.`, `_` and `-`, starting with a letter or digit, at most 63 characters |

| Flag | Takes | Description |
| - | - | - |
| `--subjects` | string; required | Whose replies to judge, separated by commas, each once: `production` for the replies the taskset's traces recorded, which asks no model for replies, and models of the workload's pool, as `ambral workloads get` lists them. `<model>@<prompt>` asks a model under a prompt: `pr_…`, an id `ambral prompts put` printed, or a file, which is put first (a path with a comma can't be named here: put it with `ambral prompts put`). The prompt replaces the system prompt each trace opens with (one of several content parts whole: its later parts aren't sent), or is added in front of one with none, and the judges still see the trace's own. A run that asks a model under a prompt scores every subject, production included, only on the traces that open with the one original its prompts replace (`--replaces`) |
| `--replaces` | string | The system prompt the taskset's traces open with, by the `pr_…` id `ambral tasksets prompts list` lists, that the prompts in `--subjects` replace: every subject, production included, is then scored only on the traces that open with it. Needed only when they open with several; a run that asks no model under a prompt takes none |
| `--scoring` | file | Score this run under this config in place of the taskset's: a .yaml, .yml or .json file, as `ambral tasksets scoring preset` and `get` print one. Nothing is saved to the taskset |
| `--label` | string; repeatable | A label, `k=v`, or `k` for a bare tag; repeat it for more, each key once |
| `--budget` | string | The most the run's calls may cost, in USD, as `25` or `0.50`: once they cost this much, the run is cancelled and its trials already scoring are stopped within seconds. Calls already in flight, and those its trials make in those seconds, still cost, so it can go past the budget by that much. `ambral runs show` reads what it has spent |
| `--wait` | | Stay until the run ends, then print it as `ambral runs show` does. A run that ends failed exits 1; a trial whose request a model's provider refused scores 0 and doesn't fail the run. A run that stalls (nothing happened for an hour, so it won't end by itself) exits 1 too. A run can outlast a shell's command timeout: without `--wait`, follow it with `ambral runs show <run> --json` |
| `--timeout` | string; default 30m | With `--wait`, how long to wait before giving up, as `30 minutes` or `2 hours`; the run goes on |

**Examples**

```sh theme={null}
# Score one pool model on the taskset beside production's own replies, and print the summary once it ends
ambral runs start support-agent/refunds --subjects production,openai/gpt-6-luna --wait

# Start it and return at once, for a shell that stops long commands; then poll `ambral runs show <run> --json` for `run.status`
ambral runs start support-agent/refunds --subjects production,openai/gpt-6-luna

# Score a pool model under an edited prompt beside the same model under the agent's own: the file is put first, and the run names it by its pr_… id
ambral runs start support-agent/refunds --subjects production,openai/gpt-6-luna,openai/gpt-6-luna@prompts/shorter.txt --wait

# Score a prompt edited from one of several the taskset's traces open with, on the traces that open with that one
ambral runs start support-agent/refunds --subjects production,openai/gpt-6-luna,openai/gpt-6-luna@pr_9d2e7a1c04b3f8e2 --replaces pr_3fa9c1e04d7b2a91 --wait

# Score only production's replies: no model is asked to reply, so only judges are called
ambral runs start support-agent/refunds --subjects production --wait

# Compare two pool models under a config that isn't saved yet
ambral runs start support-agent/refunds --subjects openai/gpt-6-luna,anthropic/claude-haiku-5.5 --scoring scoring.yaml

# Label the run with the experiment it belongs to, to list that experiment's runs later
ambral runs start support-agent/refunds --subjects production,openai/gpt-6-luna --label experiment=shorter-prompt
```

## ambral runs rescore

```text theme={null}
ambral runs rescore [flags] <run>
```

Rescore a completed run: a new run judges its replies again, every subject's on every trace, under the taskset's current scoring config or `--scoring`'s, without asking any model for a new reply. Only its judges are called and paid for; each subject's reply cost is what the rescored run paid for those replies. Use it to try a changed config on fixed replies, or to see how far a judge agrees with itself under an unchanged one. The new run keeps the run's labels, so `ambral runs list --label` finds it beside it. Its `points` must be ones the run replied at: under the same `tools`, `take` `last`, then `sample` by its `k`, then `all`, each covering those before it; under a run that took `all`, `tools` that match only turns its own matched

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

| Flag | Takes | Description |
| - | - | - |
| `--scoring` | file | Judge the replies under this config in place of the taskset's current one: a .yaml, .yml or .json file, as `ambral tasksets scoring preset` and `get` print one. Its `points` must pick only points the run replied at. Nothing is saved to the taskset |
| `--budget` | string | The most the run's calls may cost, in USD, as `25` or `0.50`: once they cost this much, the run is cancelled and its trials already scoring are stopped within seconds. Calls already in flight, and those its trials make in those seconds, still cost, so it can go past the budget by that much. `ambral runs show` reads what it has spent |
| `--wait` | | Stay until the run ends, then print it as `ambral runs show` does. A run that ends failed exits 1; a trial whose request a model's provider refused scores 0 and doesn't fail the run. A run that stalls (nothing happened for an hour, so it won't end by itself) exits 1 too. A run can outlast a shell's command timeout: without `--wait`, follow it with `ambral runs show <run> --json` |
| `--timeout` | string; default 30m | With `--wait`, how long to wait before giving up, as `30 minutes` or `2 hours`; the run goes on |

**Examples**

```sh theme={null}
# Judge a run's replies under an edited config, and print the summary once it ends
ambral runs rescore <run> --scoring scoring.yaml --wait

# Judge them again under the taskset's current config: run twice, it shows how far the judges agree with themselves
ambral runs rescore <run>
```

## ambral runs list

```text theme={null}
ambral runs list [flags]
```

List the org's score runs, newest first, a page at a time with how many there are, or a workload's, a taskset's, a label's, a kind's or a model's, each with how it went: a replay run's trials done, then its best candidate beside the serving model; an offline run's pass count and mean score, and where its suite ran

| Flag | Takes | Description |
| - | - | - |
| `--workload` | string | Only this workload's runs |
| `--taskset` | string | Only this taskset's runs, as `<workload>/<taskset>` |
| `--cursor` | string | Start after this run: the `next` a page printed, for the runs started before it |
| `--label` | string; repeatable | Only those labelled so: `k=v` for that value, `k` for any value; repeat it, and each must hold |
| `--status` | `open` \| `running` \| `completed` \| `failed` \| `cancelled` | Only runs in this status: an offline run is `open` while its suite reports to it |
| `--source` | `ci` \| `local` | Only offline runs whose suite said it ran on a CI (`ci`) or elsewhere, such as a laptop (`local`) |
| `--kind` | `replay` \| `offline` | Only runs of this kind: `replay`, which Ambral scores, or `offline`, which a suite reports |
| `--model` | string | Only runs with a subject of this model, by its id, or `production` |
| `--since` | date | Only runs started at or after this date or time, UTC (`2026-11-01`, `2026-11-01T09:00`) |
| `--sort` | `newest` \| `oldest` | The order: `newest` first (the default) or `oldest` first |

**Examples**

```sh theme={null}
# The runs of support-agent's refunds taskset
ambral runs list --taskset support-agent/refunds

# The runs started with --label experiment=shorter-prompt, and their rescores
ambral runs list --label experiment=shorter-prompt

# The completed runs of a suite that ran on a CI, each with its result
ambral runs list --taskset support-agent/refunds-scenarios --source ci --status completed
```

## ambral runs show

```text theme={null}
ambral runs show [flags] <run>
```

Show a run: while it runs, how many of its trials are scored, its health (`active`, `queued`, `retrying`, or `stalled` when nothing has happened for an hour, so it won't end by itself), each subject's trials by where they stand, and each trial waiting to be retried; once it ends, each subject's mean scores side by side, per trace, per scorer (each reward, metric, judge, criteria group and criterion) and per kind of point, what its replies cost and how many of its points were scored, what the judges cost, and each trial that failed. An offline run, which a suite reports: how many scenarios and runs of them it reported, how many passed by the suite's own verdict, and their scores, each reward's beside them

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

**Examples**

```sh theme={null}
# Every summary, and each trial that failed, as JSON
ambral runs show <run> --json
```

## ambral runs verdicts

```text theme={null}
ambral runs verdicts [flags] <run>
```

List a run's verdicts: each check's result on each subject's reply, and on each whole trace, with the judge's verdict and reason where a judge answered, by trace, subject and point, a page at a time. Where its judge asks twice (`askTwice`), a verdict shows how many of the calls gave it (`no 2/3`), and `--disagreed` lists only those whose calls differed. Trials that failed are left out, as the summary's per-point scores leave them out. Each trace is printed as `tr_…:<node>`, the branch the run replayed, which `ambral tasksets create` takes. An offline run's: each reward its suite reported of each run of a scenario, with the reason it gave, by scenario, repeat and check, filtered by `--check`, `--below`, `--scenario` and `--failed`, each row with the trace its session was recorded under

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

| Flag | Takes | Description |
| - | - | - |
| `--check` | string | Only this check's: a reward, metric or judge, a criteria group, a criterion as `<group>/<criterion>`, `replies`, `trajectory`, or an `episode` check, by the name the config gives it |
| `--verdict` | string | Only those a judge answered with this: one of a criterion's choices, or `n/a`; `yes` or `no` for a judge that compares with the baseline's reply, and for the trajectory judge |
| `--below` | number | Only those that scored below this score, from 0 to 1, whatever answered them: `--below 1` lists every check that fell short. A check that scored nothing (n/a) is left out |
| `--subject` | string | Only this subject's: `production`, a model's id, or `<model>@pr_…` for a model asked under a prompt |
| `--trace` | string | Only this trace's, `tr_…`, or one branch's, `tr_…:<node>`, as this command prints them |
| `--scenario` | string | An offline run's: only scenarios whose id starts with this, as `refunds/`, or a whole id |
| `--failed` | | An offline run's: only the runs of scenarios its suite said failed. A suite that gives no verdict of its own has none: use `--below` 1 |
| `--disagreed` | | Only those whose judge, asked twice (`askTwice`), gave answers that differed: the verdicts it can't decide |
| `--cursor` | string | Start after this: the `next` a page printed, for the verdicts after it |

**Examples**

```sh theme={null}
# The replies where a criterion wasn't met, and why the judge said so
ambral runs verdicts <run> --check manners/confirms_refund --verdict no

# Where a check fell short, a scoring function's included, which gives no verdict
ambral runs verdicts <run> --check same_call --below 1

# Where a criterion asked twice got answers that differed: one its judge can't decide
ambral runs verdicts <run> --check manners/confirms_refund --disagreed

# Every verdict on production's replies, as JSON
ambral runs verdicts <run> --subject production --json

# An offline run's scenarios under refunds/ its suite said failed, each reward's score and reason
ambral runs verdicts <run> --failed --scenario refunds/
```

## ambral runs trace

```text theme={null}
ambral runs trace [flags] <run> [<trace>]
```

Show a run's traces reply by reply, one or a page of those it scored: each subject's trace score and its checks of the whole trace, then at each point what led up to it, the baseline's reply, and each subject's reply, with the model call that made it and each check's result and reason. `--subject` keeps one subject's replies. With `--json`, each check also carries the judge's whole call: the prompt it was sent and its reply. Of an offline run: one run of a scenario, by its `tr_…` or the scenario's id (with `--repeat` when it ran more than once), its task, each reward its suite reported with its reason, and its session's conversation and traces under other workloads; without one, a page of its runs of scenarios; with `--no-result`, its sessions that made calls with its headers but have no result

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |
| `<trace>` | string; optional | A trace the run scored, `tr_…`, or one branch of it, `tr_…:<node>`, as `ambral runs verdicts` prints them; of an offline run, also a scenario's id. Without one, a page of the traces a replay run scored, or of an offline run's runs of scenarios |

| Flag | Takes | Description |
| - | - | - |
| `--repeat` | integer | Of a scenario that ran more than once in an offline run, which of its runs, from 0 |
| `--no-result` | | An offline run's sessions that made calls with its headers, under any workload, but have no result: no report named them. Each with its traces |
| `--cursor` | string | Start after this: the `next` a page of a run's traces, an offline run's scenarios, or `--no-result` printed |
| `--subject` | string | Only this subject's: `production`, a model's id, or `<model>@pr_…` for a model asked under a prompt |

**Examples**

```sh theme={null}
# Why each check scored what it did
ambral runs trace <run> tr_3f9a0c1b7e22a5d4

# Every reply a model gave under a prompt, trace by trace
ambral runs trace <run> --subject <model>@pr_…

# The same, with what each judge was shown and answered
ambral runs trace <run> tr_3f9a0c1b7e22a5d4 --json

# An offline run's second run of a scenario: its rewards, reasons and conversation
ambral runs trace <run> refunds/over-the-limit --repeat 1

# An offline run's sessions that made calls but have no result
ambral runs trace <run> --no-result
```

## ambral runs export

```text theme={null}
ambral runs export [flags] <run>
```

Print a run's results in verifiers' episode format, one JSON object a line, which verifiers and prime-rl load from `traces.jsonl`. It reads them a page at a time, and prints a page only once it holds every episode the server counted, asking again for one that came back short; if one still does, it stops with an error on stderr, with `--json` too, that says how many it printed, and exits 1. A run still running exports the trials finished when each page was read, with a warning on stderr. So a file it wrote holds the whole run only when the run had ended and the command exits 0

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

**Examples**

```sh theme={null}
# Save them to a file
ambral runs export <run> > traces.jsonl
```

## ambral runs cancel

```text theme={null}
ambral runs cancel [flags] <run>
```

Cancel a run that is still running: it ends `cancelled`, and its trials not yet scoring are skipped, so they call no model. A trial already scoring is stopped within seconds, its model calls cut off; one that finished first keeps its result. Cancelling a cancelled run again changes nothing; a run that completed or failed can't be cancelled. `--subject` cancels one subject the same way, and the others carry on: the run completes once they finish, and the subject's results cover only the trials it finished. Cancelling the last subject still scoring cancels the run

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

| Flag | Takes | Description |
| - | - | - |
| `--subject` | string | Only this subject's: `production`, a model's id, or `<model>@pr_…` for a model asked under a prompt |

**Examples**

```sh theme={null}
# Stop a run started by mistake, or one no longer needed
ambral runs cancel <run>

# Stop paying for a model that is clearly losing, and let the others finish
ambral runs cancel <run> --subject <model>
```

## ambral runs resume

```text theme={null}
ambral runs resume [flags] <run>
```

Resume a cancelled run, whether someone or its budget cancelled it: it runs again, scoring the trials it left unfinished, each with the retries it had left, and those it finished keep their results. It scores under the same config, and its cancelled subjects stay cancelled. It keeps its budget unless `--budget` gives another, which must be above what it has spent, or its next call would cancel it again. A run cancelled before Ambral kept a cancelled run's unfinished trials can't be resumed

| Argument | Takes | Description |
| - | - | - |
| `<run>` | string | The run's id, as `ambral runs start` and `ambral runs list` print it |

| Flag | Takes | Description |
| - | - | - |
| `--budget` | string | The most all the run's calls may cost, those before the resume included, in USD, as `25` or `0.50`: once they cost this much, the run is cancelled and its trials already scoring are stopped within seconds. Calls already in flight, and those its trials make in those seconds, still cost, so it can go past the budget by that much. `ambral runs show` reads what it has spent. It replaces the run's own budget, and must be above what the run has spent |
| `--wait` | | Stay until the run ends, then print it as `ambral runs show` does. A run that ends failed exits 1; a trial whose request a model's provider refused scores 0 and doesn't fail the run. A run that stalls (nothing happened for an hour, so it won't end by itself) exits 1 too. A run can outlast a shell's command timeout: without `--wait`, follow it with `ambral runs show <run> --json` |
| `--timeout` | string; default 30m | With `--wait`, how long to wait before giving up, as `30 minutes` or `2 hours`; the run goes on |

**Examples**

```sh theme={null}
# Finish a run its budget stopped, within a larger one, and print the summary once it ends
ambral runs resume <run> --budget 40 --wait

# Finish a run cancelled by mistake
ambral runs resume <run>
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.