Skip to main content
Your coding agent can do everything on this page for you: see the Quickstart. This page says what it does, and why.

What an offline run is

A team that tests its agent with a suite of its own (Vitest, pytest, verifiers, Harbor, a script) keeps running it where it runs today, on CI or a laptop, and it reports to Ambral as it runs. The suite runs each scenario and scores it. The agent’s model calls go through the Gateway, which serves them with the model the run chose and records them. Each time the suite runs is an offline run. An offline run holds two things for each run of a scenario:
  • its result: what the suite reported, which is whether it passed, its rewards and the reasons for them;
  • its calls: the agent’s model calls the Gateway recorded, as a trace.
A replay run scores one reply at a time on your agent’s recorded production sessions. An offline run reports what your own suite scores on whole sessions, each on the candidate’s own path. They answer different questions, and most agents should have both: see Concepts. The commands that read runs answer for both.

Before you wire it

  • The suite’s agent has to be on the Gateway. An offline run records only the calls that go through it. A suite whose agent calls a provider with a key of its own, or answers from a mocked model or recorded cassettes, can’t report one as it is: connect its agent first, as you would for production.
  • The suite needs a key. It reports with an Ambral key, the same kind its agent calls the Gateway with, read from AMBRAL_API_KEY. Set it where the suite runs: your shell or a .env file that git ignores on a laptop, and the CI’s secrets on CI. The SDK’s settings lists the rest of what it reads from the environment.
  • Note each client that calls a model in a scenario. Besides the agent’s, a suite often has a judge, a simulated user, or a second agent the first one calls. Only some of them get the run’s headers.

Name the workload, the taskset and the scenarios

  • The workload is the agent’s, the one its production calls already name. Its pool, which ambral workloads get lists, is the models a run may choose. Add one with ambral workloads pool add.
  • The taskset is the suite’s name in Ambral, <workload>/<taskset>. Its first run creates it. Lead with the kind of case and end with -scenarios, so the suite sorts beside the replay tasksets of the same kind: support-agent/refunds and support-agent/refunds-scenarios. Every run of the suite, on every model and every commit, reports to the same taskset, which is what lets its runs be lined up. A taskset a typo created can’t be deleted, so check the name before the first run.
  • A scenario’s id is what lines a scenario up across runs, so keep it stable: the same case keeps its id from run to run and commit to commit. Write it as a path, most general first, such as refunds/over-the-limit, since the commands that read runs roll scenarios up by the segments of their ids. Never put a run’s index, a timestamp or a random id in it: every run would then be a suite of new scenarios, and no scenario would have a history. An id is at most 512 characters.
  • A check’s name, a reward’s or a metric’s, is how checks are lined up across runs. Name a check by the rule it tests, such as policy_followed or solved.

Pick a harness

Each does the same four things: it opens a run, hands each scenario’s agent its headers, reports each scenario’s result, and closes the run.

Which clients get the headers

Every model call of the system under test carries the scenario’s headers, and nothing else does.
  • The agent’s client: always. Its calls are served the run’s model and recorded as the scenario’s session. The agent keeps sending ambral/auto as its model: the headers say which workload, run and model serve it.
  • A judge’s, or a simulated user’s: never. Give them a client of their own, without the headers. A call that carries them is served the run’s model, so the judge or the user changes with the model being tested, and its messages are recorded in the agent’s conversation.
  • Another workload the agent calls, such as a router or a sub-agent: give its client the headers too. A call that names another workload is served that workload’s primary model, as in production, and recorded under the run beside the scenario’s session.
When the suite calls the agent over HTTP, or starts it in a container, the SDK can’t reach its model client. A scenario’s headers are two plain headers: hand them over by what the harness has, such as a request header the agent’s service forwards or an environment variable the container reads, and have the agent’s client send both on each model call.

What a result holds

See What a result holds.

Repeats, labels and the model

  • Repeats. A model answers a little differently each time, so a scenario that ran once can pass or fail by luck. Run each scenario 3 times or more where you can afford it. Each run of a scenario in one run is a repeat, and the commands that read runs count the repeats passed, as 2/3. In Vitest, set repeats on the test, not retry, which runs a test again only after it failed. In verifiers, set num_rollouts, in Harbor --n-attempts, and in a script, loop. pytest has no repeats of its own: run the suite again under the same AMBRAL_EVAL_KEY, which reopens the run and adds to its repeats.
  • Labels. Label each run with what you will look for it by: the commit and the branch it ran at, from the CI’s own variables. ambral tasksets results prints each run’s commit, and ambral runs list --label branch=main finds a branch’s runs.
  • The model. A run is served one model: the one the code names, else AMBRAL_EVAL_MODEL, else the workload’s primary when the run opens. It must be in the workload’s pool. To compare models, run the suite once per model, with no change to the code: a CI matrix that sets AMBRAL_EVAL_MODEL per job, or one invocation per model on a laptop.

Cost and concurrency

A suite’s model calls are paid for with the org’s credits, at the same prices as production’s, and they draw on the same providers’ rate limits. With automatic top-up on, a suite’s calls can take the balance under its threshold and charge the saved card, as production’s calls do. An offline run has no budget. One run costs roughly its scenarios, times its repeats, times the agent’s calls in each: work that out before the first large run, and read what a run cost with ambral runs show <run>. Cap how many scenarios run at once, so a suite doesn’t slow production’s calls or hit a provider’s limits: Vitest’s maxConcurrency and maxWorkers, pytest-xdist’s -n, verifiers’ max_concurrent, Harbor’s --n-concurrent. A few dozen at once is plenty for most suites.

Where the suite runs

A run records where it ran, its source: ci when the CI environment variable is set, to anything but 0 or false, and local otherwise. A suite’s history leaves local runs out and counts them, so a developer trying a change on a laptop doesn’t move the suite’s record. --source all reads them too.
  • On GitHub Actions, GitLab CI, CircleCI or Buildkite there is nothing to set.
  • On Jenkins and Azure Pipelines, which don’t set CI, set AMBRAL_EVAL_SOURCE=ci, and AMBRAL_EVAL_KEY to the pipeline run’s id, so its jobs share one run.
  • On laptops only, or a scheduled machine, the history is empty until the runs of record say they are, with AMBRAL_EVAL_SOURCE=ci where they run.
The shards, workers and matrix jobs of one CI run report to one run. Each opens it by the same key, taskset and model, and it completes once each has closed it. The key is the CI’s id for the workflow run, or AMBRAL_EVAL_KEY. A process that crashes without closing leaves a shared run open: ambral runs list --status open lists the open runs.

Check the first run

Run three scenarios before the whole suite, on a laptop or with AMBRAL_EVAL_SOURCE=local. A wiring mistake shows in the first run, and costs less there.
1

Run them

The SDK’s last line names the run and the command that reads it:
pytest shows the line only with -rP. A report Ambral refused fails the suite with its reason: fix what it says first.
2

Wait half a minute, then read the run

The agent’s calls reach Ambral about 30 seconds after its last one. Read the lines under the scores:
  • calls: 3 of 3 results have their agent's calls recorded. A result with no calls, 2 minutes after it was reported, means the agent’s model client didn’t send the run’s headers, or the agent names its own sessions and the suite didn’t pass the SDK its id.
  • results: every session that made calls with the run’s headers has a result. When some have none, which ambral runs trace <run> --no-result lists, the suite stopped before reporting them, or a client that isn’t the agent’s has the headers.
  • ids: from the suite’s second run on, how many scenario ids are new to the suite. All of them new means the ids aren’t stable.
3

Read one run of a scenario whole

Name it by the scenario’s id, quoted, or by a tr_… that ambral runs trace <run> lists. Its conversation should hold the agent’s calls only: no judge’s prompt, and no simulated user’s turn.
4

Check the scores read as the suite meant them

ambral runs show <run> prints each reward’s mean and the reason given its lowest score, and ambral runs verdicts <run> each result’s.
Once all four hold, run the whole suite.

Read a suite’s runs

A suite that reports its runs builds a history: a run per commit on CI, and a run per model when you compare them. ambral tasksets results lines them up by scenario:
Columns compare only on what they share, so check that a comparison is sound before you read it: that the columns ran the same scenarios, at the same commit, with enough repeats to tell a failure from luck. Then read the scenarios that split, not the averages, since a mean score hides which cases moved.