Skip to main content
Report a pytest suite, a script, a verifiers eval or a Harbor job to Ambral as an offline run. In Python, give a model client built once httpx.Client(transport=Transport()), Transport from ambral_evals.httpx (AsyncTransport for an httpx.AsyncClient), and run each scenario’s agent within with scenario.current():.

pytest

pip install ambral-evals, which needs Python 3.10 or later. The Python SDK has no pytest plugin: this conftest.py reports each test that takes the scenario fixture, its id the test’s node id (test_refunds.py::test_refund[over-the-limit]). Copy it into the suite’s directory and set WORKLOAD and TASKSET:
A test hands the agent scenario.headers, sets the scenario’s task, and scores it:
  • Give each parametrized case an id of its own (ids=), not the default index, so the scenario’s id stays stable. A node id is relative to pytest’s rootdir: run the suite from the same directory each time.
  • Under pytest-xdist (pytest -n 4), the workers share one key, so they report to one run. On GitHub Actions, GitLab CI, CircleCI and Buildkite that is the CI’s id for the workflow run, which every job shares; elsewhere the controller mints one per invocation (on Jenkins or Azure Pipelines, set AMBRAL_EVAL_KEY to the pipeline run’s id to share one run across jobs).
  • A skipped test, or one whose setup failed, reports nothing. The test file imports Case only for its types, so it runs under --import-mode=importlib too. The fixture needs pytest 8 or later.
  • pytest captures the line the SDK prints as the run closes, which names the run. Run with -rP to read it, under Captured stdout teardown, once per pytest-xdist worker; or find the run with ambral runs list --taskset <workload>/<taskset>, which lists the newest first.

A Python script, or another harness

  • scenario.send(result) queues the result and returns at once; scenario.report(result) waits until it is stored. The end of the with block closes the run, and raises naming each scenario with no result.
  • run.scenario(id, session=...) takes the agent’s own session id.

verifiers

Run the eval with ambral-verifiers, naming the suite, then vf-eval’s own arguments:
The eval’s agent seat calls the Gateway (base_url, the Gateway’s /v1 base URL), naming the run’s header in its own client:
  • Keep type = "eval": verifiers refuses a seat’s client that doesn’t name its type.
  • The model is AMBRAL_EVAL_MODEL. verifiers sends each rollout’s id as its session, and reports the episode’s task, so nothing else is needed.
  • verifiers has no verdict of its own, so its runs read by their scores.
  • A scenario’s id is its task’s id, else its name, else its index in the taskset, which holds only while the taskset’s order does: give each task an id.
  • Only the agent’s seat names the header, in its own client. An episode in which more than one seat sends it isn’t reported.
  • Repeats are num_rollouts (-r); concurrency is max_concurrent (-c), 128 unless set.

Harbor

The agent class sends headers(self.context_id), from ambral_evals.harbor, on each of its model calls. Each trial is one run of its task, whose name is the scenario’s id, and the plugin reports the task’s checksum, so a changed task is a new revision. Harbor has no verdict of its own, so its runs read by their scores. The run’s key is the job’s id, which a resumed job keeps. Repeats are --n-attempts; concurrency --n-concurrent. The plugin also takes model, key and source as --plugin-kwargs.