Skip to main content
A shadow eval, also called a replay eval, scores models on your agent’s recorded production sessions. It is a taskset of those sessions with a scoring config, and each run of it is a replay run: at each point the config picks, each candidate writes the next reply to the session as production lived it, and the config scores that reply beside the one production gave. It needs no environment, no tools and no test suite, only recorded traces. So it is where to start, and the fastest way to find failure modes, try a prompt, and compare models on real traffic. Its signal is directional. A candidate writes one reply at a time to production’s history, never to its own earlier replies, so anything that shows only on the candidate’s own path is invisible to it. Before a candidate serves everything, try it on whole sessions too, with your own suite or on part of production: see Concepts.

Build and run a replay eval

Build a taskset from your agent's recorded traces, set how its replies are scored, run candidate models on it, and read the results.

How replay scoring works

How a scoring config picks the points a run replays, scores each reply, and turns those scores into each trace's.

Examples

A support agent in a live chat

Match the baseline turn by turn, keep to a policy it must never break, and hand to a person what the policy says to.

A coding agent

Judge the path, with rules that must hold at every reply.

A research agent

Judge the path and the report, half each.

A retrieval agent

Grade the answer, not how it got there, and cheaply.

An agent that writes records

Grade only the moments that change something, each message to the tenant and each write to the record, and don't let a model pass by never making them. It is one targeted run beside the workload's main scoring, not all of it.

A support agent working tickets

Judge the session by what it did, weigh a missed escalation over an extra one, and check each action call's fields.

An agent with destructive tools

One rule at every reply that must never fail.