httpx.Client(transport=Transport()), Transport from
ambral_evals.httpx (AsyncTransport for an httpx.AsyncClient), and run each scenario’s agent within
with scenario.current():.
pytest
pip install ambral-evals, which needs Python 3.10 or later. The Python SDK has no pytest plugin: this conftest.py
reports each test that takes the scenario fixture, its id the test’s node id
(test_refunds.py::test_refund[over-the-limit]). Copy it into the suite’s directory and set WORKLOAD and TASKSET:
scenario.headers, sets the scenario’s task, and scores it:
- Give each parametrized case an id of its own (
ids=), not the default index, so the scenario’s id stays stable. A node id is relative to pytest’s rootdir: run the suite from the same directory each time. - Under pytest-xdist (
pytest -n 4), the workers share one key, so they report to one run. On GitHub Actions, GitLab CI, CircleCI and Buildkite that is the CI’s id for the workflow run, which every job shares; elsewhere the controller mints one per invocation (on Jenkins or Azure Pipelines, setAMBRAL_EVAL_KEYto the pipeline run’s id to share one run across jobs). - A skipped test, or one whose setup failed, reports nothing. The test file imports
Caseonly for its types, so it runs under--import-mode=importlibtoo. The fixture needs pytest 8 or later. - pytest captures the line the SDK prints as the run closes, which names the run. Run with
-rPto read it, underCaptured stdout teardown, once per pytest-xdist worker; or find the run withambral runs list --taskset <workload>/<taskset>, which lists the newest first.
A Python script, or another harness
scenario.send(result)queues the result and returns at once;scenario.report(result)waits until it is stored. The end of thewithblock closes the run, and raises naming each scenario with no result.run.scenario(id, session=...)takes the agent’s own session id.
verifiers
Run the eval withambral-verifiers, naming the suite, then vf-eval’s own arguments:
base_url, the Gateway’s /v1 base URL), naming the run’s header in its own
client:
- Keep
type = "eval": verifiers refuses a seat’s client that doesn’t name its type. - The model is
AMBRAL_EVAL_MODEL. verifiers sends each rollout’s id as its session, and reports the episode’s task, so nothing else is needed. - verifiers has no verdict of its own, so its runs read by their scores.
- A scenario’s id is its task’s
id, else itsname, else its index in the taskset, which holds only while the taskset’s order does: give each task anid. - Only the agent’s seat names the header, in its own client. An episode in which more than one seat sends it isn’t reported.
- Repeats are
num_rollouts(-r); concurrency ismax_concurrent(-c), 128 unless set.
Harbor
headers(self.context_id), from ambral_evals.harbor, on each of its model calls. Each trial is
one run of its task, whose name is the scenario’s id, and the plugin reports the task’s checksum, so a changed task is
a new revision. Harbor has no verdict of its own, so its runs read by their scores. The run’s key is the job’s id,
which a resumed job keeps. Repeats are --n-attempts; concurrency --n-concurrent. The plugin also takes model,
key and source as --plugin-kwargs.