Skip to main content
Grade the answer, not how it got there, and cheaply. Based on the criteria preset. A retrieval agent: it looks up documents, then answers the user’s question from them. Only the final answer matters, so only it is asked for and graded (take: "last"), and every score comes from criteria grading it on its own terms, as the criteria preset does: nothing compares it with the baseline’s. Its three checks are quick and read the same answer, so they’re asked in one judge call, and grounding weighs most. Compare the research agent, which judges the path as well as the report. With no scorer comparing with the baseline, production is graded on every criterion as a candidate is: run it with --subjects production,<model> to read a candidate’s answers beside the agent’s own on the same criteria, or --subjects production alone to grade what the agent sends today, at the cost of the judge calls alone. An example to learn from, not a starting point: start a config from ambral tasksets scoring preset, and take from this one the choices that fit your agent.