Agent tests

Run agent benchmark

Run a multi-model benchmark on an agent’s linked tests as a background job.

calibrate agent-tests benchmark [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to benchmark
`-m, --models`	stringArray	Required. Model names to benchmark
`-t, --test-uuids`	string	A subset of the agent’s linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests

Examples

calibrate agent-tests benchmark --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --models '["openai/gpt-4.1","anthropic/claude-sonnet-4"]'

Get benchmark status

Get the results of a benchmark run

calibrate agent-tests get-benchmark [flags]

Options

Option	Type	Description
`-c, --compact`	model_results.test_results	Return a compact response that omits heavy detail fields (model_results.test_results, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail
`--only-failed`	—	Return only failing test cases for each model. Omit to return every case
`-t, --task-id`	string	Required. Benchmark run to poll for status and results

Examples

calibrate agent-tests get-benchmark --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890

Get test run status

Poll a test run for its status and evaluation results.

calibrate agent-tests get-run [flags]

Options

Option	Type	Description
`-c, --compact`	results.output	Return a compact response that omits heavy detail fields (results.output, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail
`--only-failed`	—	Return only failing test cases. Omit to return every case
`-t, --task-id`	string	Required. Test run to poll for status and results

Examples

calibrate agent-tests get-run --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890

Link tests to agent

Link one or more tests to an agent. Tests that are already linked are skipped.

calibrate agent-tests link [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to link tests to
`-t, --test-uuids`	stringArray	Required. Tests to link. Any that are already linked are skipped

Examples

calibrate agent-tests link --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'

List tests for agent

List the tests linked to an agent.

calibrate agent-tests list-for-agent [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent whose linked tests to list
`-l, --limit`	string	Maximum number of items to return. Omit for no limit (all items)
`--offset`	int	Number of items to skip before returning results
`--q`	name	Case-insensitive substring search on name. Blank is a no-op

Examples

calibrate agent-tests list-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

List test runs for agent

List an agent’s test runs with their results

calibrate agent-tests list-runs-for-agent [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent whose test runs to list
`--has-failures`	true	Filter by whether the run has any failing test case or model. true returns only runs with failures (or errors), `false` only clean runs. Omit for both
`-l, --limit`	string	Maximum number of items to return. Omit for no limit (all items)
`--offset`	int	Number of items to skip before returning results
`-s, --status`	string	Filter by run status. Omit for all statuses (options: queued, in_progress, cancelled, done, failed)
`-t, --type`	llm-unit-test	Filter by run type. Omit to return both:

Examples

calibrate agent-tests list-runs-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Run agent tests

Run an agent’s linked tests as a background job, returning a task ID to poll.

calibrate agent-tests run [flags]

Options

Option	Type	Description
`-a, --agent-uuid`	string	Required. Agent to test
`-t, --test-uuids`	string	Tests to run. Omit to run all tests linked to the agent

Examples

calibrate agent-tests run --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479

Run agent tests in batch

Run agent tests for every agent, or for a selected set.

calibrate agent-tests run-batch [flags]

Options

Option	Type	Description
`--body`	string	Request body as JSON (alternative to individual flags). Can also be provided via stdin.
`-b, --body-param`	string	JSON object

Examples

calibrate agent-tests run-batch

Getting started

Guides

Run agent benchmark

Get benchmark status

Get test run status

Link tests to agent

List tests for agent

List test runs for agent

Run agent tests

Run agent tests in batch

​Run agent benchmark

​Get benchmark status

​Get test run status

​Link tests to agent

​List tests for agent

​List test runs for agent

​Run agent tests

​Run agent tests in batch

Run agent benchmark

Get benchmark status

Get test run status

Link tests to agent

List tests for agent

List test runs for agent

Run agent tests

Run agent tests in batch