connection.baseUrl from ambral workloads get <workload> --json,
under whatever name the agent reads it (AMBRAL_GATEWAY_URL below).
Vitest
npm install --save-dev @ambral/evals, which needs Node 22 or later and Vitest 5.0.4 or later. The suite gets a
Vitest config of its own, apart from the unit tests’, which takes only its eval files and names the suite:
test:
- Run it with
vitest run --config vitest.evals.config.ts. Plainvitestwatches, and each rerun opens a run of its own. Never spreadambral()into the unit tests’ config: itsglobalSetupandprovidereplace theirs, and every unit run would open a run and needAMBRAL_API_KEY. ./agent.tsstands for the suite’s own agent. Its model client sendsscenario.headerson each call: as its default headers (new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, defaultHeaders: scenario.headers })), or on each request.- Each test of the SDK’s
testis a scenario. Its id is the test’s file and full name (refunds.eval.ts > over the limit), so renaming a test or its file starts a new scenario. - A test that passes reports
passed, and one that fails reports its errors.scenario.score(name, score, reason)adds a reward (vitest-evals’ scores are read too), andscenario.task(data)sets the scenario’s task. - An agent that names its own sessions takes
scenario.sessionas its session id. ambral({ … })also takesmodel,labels,keyandsource. A config of several projects spreads one into each project’stest, and each opens a run of its own suite.
A TypeScript script, or another test runner
scenario.report(result)resolves once Ambral stored it, and reports go together, about once a second: collect them and wait for them together, not one after another.run.close()sends what is left, prints the run’s id and the command that reads it, and rejects naming each scenario this process ran that has no result, a refused report’s reason included. Close in afinally: a run nobody closed stays open.run.scenario(id, { session })takes the agent’s own session id.
A model client built once
A client built once, at module scope, can’t take each scenario’s headers as its defaults. Build it with the SDK’s transport instead, which adds the current scenario’s headers to each call, and refuses a call made outside a scenario rather than send it as production’s. In TypeScript, the transport isfetch from @ambral/evals:
new OpenAI({ baseURL: process.env.AMBRAL_GATEWAY_URL, fetch }). In a test of @ambral/evals/vitest, the test’s
scenario is the current one. A script runs each scenario’s agent within scenario.run(() => …).