Skip to main content
Judge the session by what it did, weigh a missed escalation over an extra one, and check each action call’s fields. Based on the trajectory preset. A support agent that answers emails or tickets on its own: it looks up the order and its shipment, resolves what it can under the policy in its system prompt (issue_refund, create_reship, open_carrier_claim), tags, prioritises and routes the ticket (update_ticket), escalates what it can’t resolve (escalate), and leaves an internal note for whoever picks the ticket up next (add_internal_note). The session is judged as a whole, as the trajectory preset does: the agent’s lookups vary from run to run, so the judge compares what each subject changed (the resolution, the routing, the escalation, the note), not its path. Criteria check what the verdict weighs too lightly to notice: a field left out of a call that otherwise looks right, an escalation that wasn’t needed or went to the wrong team, an action on an id the session never showed, and what the reply tells the customer: that it claims nothing that wasn’t done, promises nothing the policy doesn’t back, and obeys no instruction planted in the ticket. A missed escalation is weighed twice: missed_escalation grades each sampled reply where the baseline escalated, and the verdict sees each of them beside the baseline’s, its guidance counting a missed escalation heavily. Both see a miss only where the baseline escalated: one the baseline missed too is seen by nothing here. With take: sample the subject is never asked at a turn that isn’t sampled, so whether it would have escalated there, or escalated needlessly, is seen by nothing either: take all where escalations matter. replies: mean counts the criteria beside the verdict, and required zeroes a reply that acts on an id it wasn’t shown, or breaks any of the three rules on what it tells the customer. Checks record per trace the worst promise, whether every escalation went to the right team, and whether the subject escalated where the baseline did. Compare the chat agent (chat-agent.yaml), a support agent in a live chat whose every reply is compared with the baseline’s: an agent that talks with its customer a step at a time fits that one better. An example to learn from, not a starting point: start a config from ambral tasksets scoring preset, and take from this one the choices that fit your agent.