Tools - Calibrate

Each tool maps 1-to-1 to a public API operation. Tools are grouped by resource below.

Agent tests

list-agent-test-runs

List an agent’s test and benchmark runs, most recently created first, with their status and results. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	Agent whose test runs to list
`type`	string	No	Filter by run type. Omit to return both:
`status`	TaskStatus	No	Filter by run status. Omit for all statuses
`has_failures`	boolean	No	Filter by whether the run has any failing test case or model. `true` returns only runs with failures (or errors), `false` only clean runs. Omit for both
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /agent-tests/agent/{agent_uuid}/runs in the API reference tab for the full schema.

list-agent-tests

List the tests linked to an agent. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	Agent whose linked tests to list
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /agent-tests/agent/{agent_uuid}/tests in the API reference tab for the full schema.

get-agent-benchmark

Get the results of a benchmark run, including per-model scores and the leaderboard. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_id`	string	Yes	Benchmark run to poll for status and results
`only_failed`	boolean	No	Return only failing test cases for each model. Omit to return every case
`compact`	boolean	No	Return a compact response that omits heavy detail fields (`model_results.test_results`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail

See GET /agent-tests/benchmark/{task_id} in the API reference tab for the full schema.

get-agent-test-run

Poll the status and results of an agent test run by task_id. Returns queued, in_progress, completed, failed, or aborted status. When completed, includes per-test-case pass/fail results and evaluator judgments. Use the task_id from run-agent-tests or run-agent-tests-batch. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_id`	string	Yes	Test run to poll for status and results
`only_failed`	boolean	No	Return only failing test cases. Omit to return every case
`compact`	boolean	No	Return a compact response that omits heavy detail fields (`results.output`, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail

See GET /agent-tests/run/{task_id} in the API reference tab for the full schema.

run-agent-benchmark

Start a multi-model benchmark over an agent’s linked tests as a background job. Optionally restrict to a subset of linked tests. Returns a task_id to poll with get-agent-benchmark. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	Agent to benchmark
`models`	array of string	Yes	Model names to benchmark
`test_uuids`	array of string	No	A subset of the agent’s linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests

See POST /agent-tests/agent/{agent_uuid}/benchmark in the API reference tab for the full schema.

run-agent-tests

Start a background LLM unit-test job for one agent. Runs all tests linked to the agent unless test_uuids is provided. Returns a task_id to poll with get-agent-test-run. The agent must belong to the caller’s org and have a verified connection; otherwise the call fails with 400 or 404. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	Agent to test
`test_uuids`	array of string	No	Tests to run. Omit to run all tests linked to the agent

See POST /agent-tests/agent/{agent_uuid}/run in the API reference tab for the full schema.

run-agent-tests-batch

Launch one test job per agent in the caller’s org. Optionally restrict to agent_names; omit or pass an empty list to run every org agent. Agents with no linked tests or an unverified connection are skipped (reported under skipped) rather than failing the batch. Returns task_id values to poll with get-agent-test-run. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_names`	array of string	No	Agents to run. Omit to run every agent in your workspace

See POST /agent-tests/run in the API reference tab for the full schema.

link-tests-to-agent

Link one or more existing tests to an agent so they run together. Tests already linked are skipped. The agent and every test must be in your workspace. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	Agent to link tests to
`test_uuids`	array of string	Yes	Tests to link. Any that are already linked are skipped

See POST /agent-tests in the API reference tab for the full schema.

Agents

list-agent-evaluators

List the evaluators linked to an agent, most recently linked first. Returns each evaluator’s UUID, name, type, and live version summary. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	The agent whose evaluators to list
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /agents/{agent_uuid}/evaluators in the API reference tab for the full schema.

list-agents

List every agent in the caller’s organization. Returns each agent’s UUID, name, type, and config. Use this to discover which agents exist before resolving names or launching test runs. Requires a Calibrate API key (CALIBRATE_API_KEY). Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /agents in the API reference tab for the full schema.

get-agent

Fetch a single agent by UUID within the caller’s org. Returns the agent’s name, type, and config. Fails with 404 if the agent does not exist or belongs to another org. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	The agent to retrieve

See GET /agents/{agent_uuid} in the API reference tab for the full schema.

resolve-agent-names

Map human-friendly agent names to UUIDs within the caller’s org. Each name resolves to at most one agent; unknown names are returned under not_found. Use this before run-agent-tests when you have names instead of UUIDs. Does not create or modify agents. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`names`	array of string	Yes	Agent names to resolve to IDs

See POST /agents/resolve in the API reference tab for the full schema.

create-agent

Create a new agent in the caller’s organization. Provide the agent’s name, type, and config. Returns the created agent including its UUID. Requires a Calibrate API key (CALIBRATE_API_KEY). Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`name`	string	Yes	Agent name, unique within the workspace
`type`	string	No	- `agent`: built inside Calibrate
`config`	object	No	Agent behavioral config. The keys depend on `type`.

Examples Agent within Calibrate Build a voice/chat agent inside Calibrate. This config is the managed defaults spelled out. Override only the keys you want to change; omitted keys still inherit the defaults.

{
  "name": "create-agent",
  "arguments": {
    "name": "Support Agent",
    "type": "agent",
    "config": {
      "system_prompt": "You are a helpful assistant.",
      "llm": {
        "model": "google/gemini-2.5-flash"
      },
      "stt": {
        "provider": "google"
      },
      "tts": {
        "provider": "google"
      },
      "settings": {
        "agent_speaks_first": false,
        "max_assistant_turns": 10
      }
    }
  }
}

Connect OpenAI-compatible agent Connect your own agent over an OpenAI-compatible HTTP endpoint. config.agent_url is required; agent_headers carries the auth token the endpoint expects.

{
  "name": "create-agent",
  "arguments": {
    "name": "My Hosted Agent",
    "type": "connection",
    "config": {
      "agent_url": "https://api.example.com/v1/chat/completions",
      "agent_headers": {
        "Authorization": "Bearer <token>"
      },
      "benchmark_provider": "openrouter"
    }
  }
}

For more details, see Agent connections. See POST /agents in the API reference tab for the full schema.

update-agent

Update an existing agent by UUID within the caller’s org. Supply the fields to change (name, type, config). Returns the updated agent. Fails with 404 if the agent does not exist or belongs to another org. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	The agent to update
`name`	string	No	New agent name. Omit to leave the name unchanged
`config`	object	No	Agent behavioral config. The keys depend on `type`.

See PUT /agents/{agent_uuid} in the API reference tab for the full schema.

link-evaluators-to-agent

Link one or more evaluators to an agent, skipping any already linked. Each evaluator must be one you created or a built-in default, and the agent must be in your workspace. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	The agent to link the evaluators to
`evaluator_ids`	array of string	Yes	The evaluators to link to the agent. Ones that are already linked are skipped. Each must be one you created or a built-in default

See POST /agents/{agent_uuid}/evaluators in the API reference tab for the full schema.

verify-agent-connection

Verify that a connection-type agent responds. Optionally pass a model to verify it before benchmarking. The endpoint and headers come from the agent’s stored config; a successful check is recorded on the agent. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`agent_uuid`	string	Yes	The agent whose connection to verify
`model`	string	No	Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model
`messages`	array of object	No	Sample chat messages to send during verification. Omit to use the default probe

See POST /agents/{agent_uuid}/verify-connection in the API reference tab for the full schema.

Annotation tasks

list-annotation-tasks

List your annotation tasks with their linked evaluators. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /annotation-tasks in the API reference tab for the full schema.

get-annotation-agreement

Get human-vs-human and human-vs-evaluator agreement metrics for a task, with a trend series. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`bucket`	string	No	How to bucket points in the trend series
`days`	integer	No	Trailing window in days for the trend series

See GET /annotation-tasks/{task_uuid}/agreement in the API reference tab for the full schema.

get-annotation-evaluator-run

Get one evaluator-run job with its results and human-agreement summary. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`job_uuid`	string	Yes	The evaluator run to act on

See GET /annotation-tasks/{task_uuid}/evaluator-runs/{job_uuid} in the API reference tab for the full schema.

get-annotation-summary

Get a paginated summary table for a task, with each item’s evaluator values and human labels. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`item_id`	string	No	Filter rows to a single item. The full task-wide annotator union is still returned in `annotators`
`live_only`	boolean	No	When true, emit only one row for each (item, evaluator) pair using the evaluator’s live version. Versions other than the live one that have runs are excluded
`disagreement_only`	boolean	No	When true, keep only rows where the evaluator disagreed with at least one annotator
`q`	string	No	Case-insensitive substring search on `payload.name`. Blank is a no-op
`sort_by`	string	No	Sort key for the results
`order`	string	No	Sort direction
`limit`	integer	No	Maximum number of items to return
`offset`	integer	No	Number of items to skip before returning results
`compact`	boolean	No	Return a compact response that omits heavy detail fields (`rows.payload`, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail

See GET /annotation-tasks/{task_uuid}/summary in the API reference tab for the full schema.

get-annotation-task

Get one annotation task with its evaluators, items, and labelling jobs. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Task to retrieve

See GET /annotation-tasks/{task_uuid} in the API reference tab for the full schema.

create-annotation-task

Create an annotation task for labelling items against evaluators. Optionally link evaluators in order. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`name`	string	Yes	Task name, unique within your workspace
`type`	string	Yes	Task type. Determines the shape of each item’s payload.
`description`	string	No	A description for the task. Omit for none
`evaluator_ids`	array of string	No	IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators

See POST /annotation-tasks in the API reference tab for the full schema.

run-annotation-evaluators

Run one or more of a task’s evaluators over its items as a background job. Returns a job_uuid to poll with get-annotation-evaluator-run. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`evaluators`	array of EvaluatorRunRequestEntry	Yes	The evaluators to run. Each must be linked to the task
`item_ids`	array of string	No	Item IDs to run on. Required when `select_all=false`. Ignored when `select_all=true`
`select_all`	boolean	No	When `true`, run on every item in the task. Set `q` to run only items whose name matches it
`q`	string	No	Case-insensitive substring filter on `payload.name`. Applies only when `select_all=true`

See POST /annotation-tasks/{task_uuid}/evaluator-runs in the API reference tab for the full schema.

update-annotation-items

Bulk-update item payloads in an annotation task. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`updates`	array of ItemUpdatePayload	Yes	The new payload for each item you’re updating. Entries not in this task, or referencing deleted items, are skipped

See PUT /annotation-tasks/{task_uuid}/items in the API reference tab for the full schema.

add-annotation-items

Bulk-create items in an annotation task, optionally seeding human annotations. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`items`	array of AnnotationItemPayload	Yes	Items to insert. Insertion order is preserved
`annotator_id`	string	No	Annotator these initial annotations belong to. Required when any item carries annotations

See POST /annotation-tasks/{task_uuid}/items in the API reference tab for the full schema.

set-task-evaluators

Replace the task’s linked evaluators with exactly the given ordered set: new ids are linked, omitted ids are unlinked, and the list order sets their display position. Send an empty list to unlink all. Each evaluator must be one you created or a built-in default, and the task must be in your workspace. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`task_uuid`	string	Yes	Annotation task to act on
`evaluator_ids`	array of string	Yes	The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default

See PUT /annotation-tasks/{task_uuid}/evaluators in the API reference tab for the full schema.

Evaluators

list-evaluators

List your evaluators. Optionally filter by what they judge or their modality, and include the built-in defaults. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`evaluator_type`	string	No	Filter by what the evaluator judges. Omit for all types
`data_type`	string	No	Filter by modality. Omit for all
`include_defaults`	boolean	No	Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /evaluators in the API reference tab for the full schema.

get-evaluator

Get one evaluator with its full version history. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`evaluator_uuid`	string	Yes	Evaluator to retrieve
`compact`	boolean	No	Return a compact response that omits heavy detail fields (`versions.system_prompt`, `versions.output_config`, `versions.variables`), keeping only the lightweight decision fields. Omit for full detail

See GET /evaluators/{evaluator_uuid} in the API reference tab for the full schema.

create-evaluator

Create an evaluator and its first version, which is set live. An evaluator judges an output against a prompt and rubric. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`name`	string	Yes	Evaluator name, unique within your workspace
`description`	string	No	Description. Omit to leave blank
`evaluator_type`	string	No	What the evaluator judges:
`data_type`	string	No	The modality the judge reads:
`output_type`	string	No	How the evaluator scores:
`version`	EvaluatorVersionCreate	Yes	The evaluator’s first version. Set as live when you create the evaluator

See POST /evaluators in the API reference tab for the full schema.

create-evaluator-version

Add a new version to an evaluator you created, optionally making it live. The variable names must match the previous version. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`evaluator_uuid`	string	Yes	Evaluator to add a version to
`judge_model`	string	Yes	The model that runs the judge, named the way its provider does, for example `openai/gpt-4.1` or `anthropic/claude-sonnet-4`
`system_prompt`	string	Yes	Judge system prompt. May contain `{{variable}}` placeholders
`output_config`	OutputConfig	No	The scale points and their labels. Required for a `rating` evaluator. A `binary` evaluator uses the default Correct/Wrong labels unless you set your own
`variables`	array of VariableSpec	No	Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable’s description or default, but not add, remove, or rename one
`make_live`	boolean	No	When `true`, immediately point the evaluator’s live version at this new version

See POST /evaluators/{evaluator_uuid}/versions in the API reference tab for the full schema.

Tests

list-tests

List every test in the caller’s organization. Returns each test’s UUID, name, type, and config. Use this to discover which tests exist before creating runs. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`q`	string	No	Case-insensitive substring search on `name`. Blank is a no-op
`limit`	integer	No	Maximum number of items to return. Omit for no limit (all items)
`offset`	integer	No	Number of items to skip before returning results

See GET /tests in the API reference tab for the full schema.

get-test

Fetch a single test by UUID within the caller’s org. Returns the test’s name, type, and config. Fails with 404 if the test does not exist or belongs to another org. Scope: read · Access: Read-only Parameters

Parameter	Type	Required	Description
`test_uuid`	string	Yes	Test to retrieve

See GET /tests/{test_uuid} in the API reference tab for the full schema.

create-test

Create a new test in the caller’s organization. Provide the test’s name, type, and config. Returns the created test including its UUID. Requires a Calibrate API key (CALIBRATE_API_KEY). Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`name`	string	Yes	Name of the test, unique within the workspace
`type`	string	Yes	What the test judges:
`config`	object	No	The calibrate test config. Three top-level keys.
`evaluators`	array of routers__tests__EvaluatorRef	No	Evaluators to link. Used by `response` and `conversation` tests

See POST /tests in the API reference tab for the full schema.

bulk-create-tests

Create multiple tests in the caller’s organization in one call. Provide a list of test definitions (name, type, config). Returns the created tests including their UUIDs. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`type`	string	Yes	What the test judges:
`tests`	array of BulkTestItem	Yes	Test items to create, at most 500 per request, with names unique within the batch
`agent_uuids`	array of string	No	IDs of agents to link every created test to. Omit to link none
`language`	string	No	Language written to each test’s `config.settings.language`. Omit to leave unset

See POST /tests/bulk in the API reference tab for the full schema.

update-test

Update an existing test by UUID within the caller’s org. Supply the fields to change (name, config). Returns the updated test. Fails with 404 if the test does not exist or belongs to another org. Scope: write · Access: Write Parameters

Parameter	Type	Required	Description
`test_uuid`	string	Yes	Test to update
`name`	string	No	New test name. Omit to leave unchanged
`type`	string	No	What the test judges:
`config`	object	No	The calibrate test config. Three top-level keys.
`evaluators`	array of routers__tests__EvaluatorRef	No	New evaluator links for the test. Omit to leave unchanged. An empty list clears them, except on `conversation` tests, which must keep at least one

See PUT /tests/{test_uuid} in the API reference tab for the full schema.

​Agent tests

​list-agent-test-runs

​list-agent-tests

​get-agent-benchmark

​get-agent-test-run

​run-agent-benchmark

​run-agent-tests

​run-agent-tests-batch

​link-tests-to-agent

​Agents

​list-agent-evaluators

​list-agents

​get-agent

​resolve-agent-names

​create-agent

​update-agent

​link-evaluators-to-agent

​verify-agent-connection

​Annotation tasks

​list-annotation-tasks

​get-annotation-agreement

​get-annotation-evaluator-run

​get-annotation-summary

​get-annotation-task

​create-annotation-task

​run-annotation-evaluators

​update-annotation-items

​add-annotation-items

​set-task-evaluators

​Evaluators

​list-evaluators

​get-evaluator

​create-evaluator

​create-evaluator-version

​Tests

​list-tests

​get-test

​create-test

​bulk-create-tests

​update-test