Shadow evals
Shadow evals
A shadow eval, also called a replay eval, replays your agent’s recorded sessions to candidate models and scores each reply.
A shadow eval, also called a replay eval, scores models on your agent’s recorded production sessions. It
is a taskset of those sessions with a scoring config, and each run of it is a replay run: at each point the
config picks, each candidate writes the next reply to the session as production lived it, and the config scores
that reply beside the one production gave.
It needs no environment, no tools and no test suite, only recorded traces. So it is
where to start, and the fastest way to find failure modes, try a prompt, and compare models on real traffic.
Its signal is directional. A candidate writes one reply at a time to production’s history, never to its own
earlier replies, so anything that shows only on the candidate’s own path is invisible to it. Before a candidate
serves everything, try it on whole sessions too, with your own suite or on part of
production: see Concepts.
Build and run a replay eval
Build a taskset from your agent's recorded traces, set how its replies are scored, run candidate models on it, and read the results.
How replay scoring works
How a scoring config picks the points a run replays, scores each reply, and turns those scores into each trace's.
Examples
A support agent in a live chat
Match the baseline turn by turn, keep to a policy it must never break, and hand to a person what the policy says to.
A coding agent
Judge the path, with rules that must hold at every reply.
A research agent
Judge the path and the report, half each.
A retrieval agent
Grade the answer, not how it got there, and cheaply.
An agent that writes records
Grade only the moments that change something, each message to the tenant and each write to the record, and don't let a model pass by never making them. It is one targeted run beside the workload's main scoring, not all of it.
A support agent working tickets
Judge the session by what it did, weigh a missed escalation over an extra one, and check each action call's fields.
An agent with destructive tools
One rule at every reply that must never fail.