Your coding agent can do everything on this page for you: see the Quickstart. This page says
what it does, and why.
support-agent/refunds, so
each score says how a model does on that kind.
Build a taskset from recorded traces
List the workload’s sessions, and read the promising ones whole:--label env=prod keeps real traffic,
when your agent labels its sessions, so a test harness’s
sessions don’t end up in the taskset.
Choose traces of one kind of request, with variety within it: different requests, different tools, short and
long.
- Pick the hard sessions, not only the common ones. A request just past a limit of your policy beside one just inside it, and the sessions built to mislead: an instruction planted in a message, a request from someone without the authority, a third contact about the same problem. That is where models differ. A taskset of easy sessions scores them all alike.
- Pick sessions the recorded model got right, when the scoring will compare candidates with the baseline, the reply each trace recorded. Those replies are the answers every candidate is graded against, mistakes included, so read each reply before you pick its trace.
tr_…:<node>, as ambral traces get names a branch,
takes another. The branches are fixed when the taskset is created, so every run replays the same sessions. A
taskset’s name is spelled as a workload’s is.
When the traffic is thin
Real traffic is mostly easy sessions, and a new workload has little of it. The sessions that tell models apart sit on the lines your agent’s policy draws: each limit, each rule that sends a case to a person, who may ask for what. Where the traces lack them, record them on purpose:- Write inputs on both sides of each line, and the ones built to mislead, with each input’s right outcome.
- Run your agent on each as a scenario of an offline run, on a strong model of the workload’s pool. Its sessions are recorded as the run’s, apart from real traffic, and the run says which inputs the agent got right.
- Build tasksets from the run’s sessions as from any others:
ambral runs trace <run>lists each one’str_….
When to make a new taskset
A taskset’s traces are fixed, and its scoring is versioned. When only how you grade changes, save the next scoring version on the same taskset. Make a new taskset, such assupport-agent/refunds-v2, from traces recorded
since, when what makes a good session has changed: a new policy, new tools, a different flow. Shipping a new
system prompt is such a point too, since production should be the agent as it runs today.
Set the scoring
See How replay scoring works, and the examples, each a scoring config for one goal, starting with A support agent working tickets. A scoring config is a YAML file you keep in your repo, beside your agent’s code. Start it from a preset:ambral tasksets scoring preset lists them, each with when to use it, and prints one to start from. A preset
with judges takes --judge, a model from ambral models list:
--check lists the mistakes it finds, each at its field, and saves nothing. Each save is the taskset’s next
scoring version, so save once the file is right, not after each small change. To change the scoring later, edit
the same file, then check and save it again. ambral tasksets scoring get writes the saved version back out,
every default filled in.
Run it
A run scores a taskset for its subjects, which you name with--subjects:
productionis what your agent sent: the replies the traces recorded. No model is asked for them, so it costs only the judges’ calls. Include it whenever you compare models: a score of 0.88 means little alone, and four points below production means something. Scorers that compare a reply with the baseline’s record nothing for it, since they would compare production with itself.- A model of the workload’s pool writes its own reply at each point.
ambral workloads getlists the pool, andambral workloads pool addadds a model to it. - A model under a prompt,
<model>@<prompt>, is asked with that prompt in place of the system prompt each trace opens with. The prompt is a text file, or thepr_…id thatambral prompts putprinted for one. It is scored in a column of its own, soproduction,<model>,<model>@pr_…reads the prompt’s effect beside the model’s own.
--waitstays until the run ends, then prints its summary. It gives up after 30 minutes, or after--timeout, and the run goes on. Without--wait, the command prints the run’s id at once: read the run withambral runs show <run>, and find its id again withambral runs list --taskset support-agent/refunds.--budgetcaps what the run’s calls may cost, in USD. Once they cost that much the run is cancelled, and keeps what it scored before then. Calls already in flight still cost, so size it with room to spare. Nothing prices a run before it runs: to size a large one, run 2 or 3 of its traces first, and multiply the cost per finished trial thatambral runs showprints by the large run’s trials.--scoring <file>scores this run alone under that file, without saving it to the taskset, so you can compare a draft with the saved version.--labellabels the run, such as--label experiment=shorter-prompt, so thatambral runs list --label experiment=shorter-promptfinds an experiment’s runs together.
ambral runs show prints its health: active, queued and retrying all mean it is on its
way. stalled means nothing has happened for an hour, and it won’t end by itself.
ambral runs cancel <run> stops a run in flight, and --subject stops one subject while the others finish.
ambral runs resume <run> scores the trials a cancelled run left unfinished, under the same config. With
--budget, it finishes a run its budget stopped, within a larger one that counts what the run has already spent.
The trials it finished keep their results and aren’t paid for again.
While a run is in flight, wait for it rather than starting it again, which pays for the same replies twice. Once
it has ended, running it again is worth it to check that a result holds, since models and judges answer a little
differently each time.
Read the results
Read a run from its summary down to single replies.- Per trace. Each trace counts once, so compare models on this row.
- Per scorer, by its name: each reward, metric, judge, criteria group and criterion. The count in brackets is how many scores are behind a mean: a criterion that is n/a at most points has few, so a 1.00 over 3 points says little.
- Cost per trace is what a subject would cost to serve. A candidate’s is an estimate, and production’s is what production really cost.
scoring vN.
Then find where a subject fell short:
Change the scoring, and score again
When a scorer grades a reply differently from how you would, change the scoring file and rescore the run:Work on the failures
The traces where a scorer failed are the cases to work on.ambral runs verdicts with --check and --verdict
prints them on its last lines, as Their traces: tr_…:<node>,…, which ambral tasksets create takes as they
are. Run candidates on that focused taskset beside production, under the same scoring.
Promote a candidate
A candidate that beatsproduction, on a result that holds when you run it again, may be worth serving. A replay
run scores one reply at a time, so try the candidate on whole sessions first: with your own suite, as an
offline run, or on part of production. For that, create a second workload that serves the
candidate, send it a small share of your agent’s sessions, and score both workloads’ traces the same way.
Then make the candidate the model the workload serves:
Export a run
ambral runs export <run> > traces.jsonl writes every result as one line of JSON in verifiers’ episode format,
to analyse the results or train on them outside Ambral. Export a run once it has ended: the file holds the whole
run only when it had, and the command exited 0.