> ## Documentation Index > Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt > Use this file to discover all available pages before exploring further. # Tools > Every tool the Calibrate MCP server exposes Each tool maps 1-to-1 to a [public API](/docs/api-reference/introduction) operation. Tools are grouped by resource below. ## Agent tests ### list-agent-test-runs List an agent's test and benchmark runs, most recently created first, with their status and results. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | -------------- | ---------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agent_uuid` | string | Yes | Agent whose test runs to list | | `type` | string | No | Filter by run type. Omit to return both: | | `status` | TaskStatus | No | Filter by run status. Omit for all statuses | | `has_failures` | boolean | No | Filter by whether the run has any failing test case or model. `true` returns only runs with failures (or errors), `false` only clean runs. Omit for both | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /agent-tests/agent/\{agent\_uuid}/runs** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### list-agent-tests List the tests linked to an agent. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------- | | `agent_uuid` | string | Yes | Agent whose linked tests to list | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /agent-tests/agent/\{agent\_uuid}/tests** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-agent-benchmark Get the results of a benchmark run, including per-model scores and the leaderboard. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------- | ------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_id` | string | Yes | Benchmark run to poll for status and results | | `only_failed` | boolean | No | Return only failing test cases for each model. Omit to return every case | | `compact` | boolean | No | Return a compact response that omits heavy detail fields (`model_results.test_results`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail | See **GET /agent-tests/benchmark/\{task\_id}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-agent-test-run Poll the status and results of an agent test run by task\_id. Returns queued, in\_progress, completed, failed, or aborted status. When completed, includes per-test-case pass/fail results and evaluator judgments. Use the task\_id from run-agent-tests or run-agent-tests-batch. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_id` | string | Yes | Test run to poll for status and results | | `only_failed` | boolean | No | Return only failing test cases. Omit to return every case | | `compact` | boolean | No | Return a compact response that omits heavy detail fields (`results.output`, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail | See **GET /agent-tests/run/\{task\_id}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### run-agent-benchmark Start a multi-model benchmark over an agent's linked tests as a background job. Optionally restrict to a subset of linked tests. Returns a task\_id to poll with get-agent-benchmark. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------- | | `agent_uuid` | string | Yes | Agent to benchmark | | `models` | array of string | Yes | Model names to benchmark | | `test_uuids` | array of string | No | A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests | See **POST /agent-tests/agent/\{agent\_uuid}/benchmark** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### run-agent-tests Start a background LLM unit-test job for one agent. Runs all tests linked to the agent unless test\_uuids is provided. Returns a task\_id to poll with get-agent-test-run. The agent must belong to the caller's org and have a verified connection; otherwise the call fails with 400 or 404. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | --------------- | -------- | ------------------------------------------------------- | | `agent_uuid` | string | Yes | Agent to test | | `test_uuids` | array of string | No | Tests to run. Omit to run all tests linked to the agent | See **POST /agent-tests/agent/\{agent\_uuid}/run** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### run-agent-tests-batch Launch one test job per agent in the caller's org. Optionally restrict to agent\_names; omit or pass an empty list to run every org agent. Agents with no linked tests or an unverified connection are skipped (reported under skipped) rather than failing the batch. Returns task\_id values to poll with get-agent-test-run. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------- | --------------- | -------- | -------------------------------------------------------- | | `agent_names` | array of string | No | Agents to run. Omit to run every agent in your workspace | See **POST /agent-tests/run** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### link-tests-to-agent Link one or more existing tests to an agent so they run together. Tests already linked are skipped. The agent and every test must be in your workspace. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | --------------- | -------- | ------------------------------------------------------ | | `agent_uuid` | string | Yes | Agent to link tests to | | `test_uuids` | array of string | Yes | Tests to link. Any that are already linked are skipped | See **POST /agent-tests** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ## Agents ### list-agent-evaluators List the evaluators linked to an agent, most recently linked first. Returns each evaluator's UUID, name, type, and live version summary. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------- | | `agent_uuid` | string | Yes | The agent whose evaluators to list | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /agents/\{agent\_uuid}/evaluators** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### list-agents List every agent in the caller's organization. Returns each agent's UUID, name, type, and config. Use this to discover which agents exist before resolving names or launching test runs. Requires a Calibrate API key (CALIBRATE\_API\_KEY). **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------------------------------------- | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /agents** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-agent Fetch a single agent by UUID within the caller's org. Returns the agent's name, type, and config. Fails with 404 if the agent does not exist or belongs to another org. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | --------------------- | | `agent_uuid` | string | Yes | The agent to retrieve | See **GET /agents/\{agent\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### resolve-agent-names Map human-friendly agent names to UUIDs within the caller's org. Each name resolves to at most one agent; unknown names are returned under not\_found. Use this before run-agent-tests when you have names instead of UUIDs. Does not create or modify agents. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | --------- | --------------- | -------- | ----------------------------- | | `names` | array of string | Yes | Agent names to resolve to IDs | See **POST /agents/resolve** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### create-agent Create a new agent in the caller's organization. Provide the agent's name, type, and config. Returns the created agent including its UUID. Requires a Calibrate API key (CALIBRATE\_API\_KEY). **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | --------- | ------ | -------- | --------------------------------------------------- | | `name` | string | Yes | Agent name, unique within the workspace | | `type` | string | No | - `agent`: built inside Calibrate | | `config` | object | No | Agent behavioral config. The keys depend on `type`. | **Examples** **Agent within Calibrate** Build a voice/chat agent inside Calibrate. This config is the managed defaults spelled out. Override only the keys you want to change; omitted keys still inherit the defaults. ```json theme={null} { "name": "create-agent", "arguments": { "name": "Support Agent", "type": "agent", "config": { "system_prompt": "You are a helpful assistant.", "llm": { "model": "google/gemini-2.5-flash" }, "stt": { "provider": "google" }, "tts": { "provider": "google" }, "settings": { "agent_speaks_first": false, "max_assistant_turns": 10 } } } } ``` **Connect OpenAI-compatible agent** Connect your own agent over an OpenAI-compatible HTTP endpoint. `config.agent_url` is required; `agent_headers` carries the auth token the endpoint expects. ```json theme={null} { "name": "create-agent", "arguments": { "name": "My Hosted Agent", "type": "connection", "config": { "agent_url": "https://api.example.com/v1/chat/completions", "agent_headers": { "Authorization": "Bearer " }, "benchmark_provider": "openrouter" } } } ``` For more details, see [Agent connections](/docs/core-concepts/agent-connections). See **POST /agents** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### update-agent Update an existing agent by UUID within the caller's org. Supply the fields to change (name, type, config). Returns the updated agent. Fails with 404 if the agent does not exist or belongs to another org. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | ------ | -------- | --------------------------------------------------- | | `agent_uuid` | string | Yes | The agent to update | | `name` | string | No | New agent name. Omit to leave the name unchanged | | `config` | object | No | Agent behavioral config. The keys depend on `type`. | See **PUT /agents/\{agent\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### link-evaluators-to-agent Link one or more evaluators to an agent, skipping any already linked. Each evaluator must be one you created or a built-in default, and the agent must be in your workspace. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | --------------- | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- | | `agent_uuid` | string | Yes | The agent to link the evaluators to | | `evaluator_ids` | array of string | Yes | The evaluators to link to the agent. Ones that are already linked are skipped. Each must be one you created or a built-in default | See **POST /agents/\{agent\_uuid}/evaluators** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### verify-agent-connection Verify that a connection-type agent responds. Optionally pass a model to verify it before benchmarking. The endpoint and headers come from the agent's stored config; a successful check is recorded on the agent. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------ | | `agent_uuid` | string | Yes | The agent whose connection to verify | | `model` | string | No | Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model | | `messages` | array of object | No | Sample chat messages to send during verification. Omit to use the default probe | See **POST /agents/\{agent\_uuid}/verify-connection** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ## Annotation tasks ### list-annotation-tasks List your annotation tasks with their linked evaluators. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------------------------------------- | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /annotation-tasks** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-annotation-agreement Get human-vs-human and human-vs-evaluator agreement metrics for a task, with a trend series. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ----------- | ------- | -------- | -------------------------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `bucket` | string | No | How to bucket points in the trend series | | `days` | integer | No | Trailing window in days for the trend series | See **GET /annotation-tasks/\{task\_uuid}/agreement** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-annotation-evaluator-run Get one evaluator-run job with its results and human-agreement summary. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | --------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `job_uuid` | string | Yes | The evaluator run to act on | See **GET /annotation-tasks/\{task\_uuid}/evaluator-runs/\{job\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-annotation-summary Get a paginated summary table for a task, with each item's evaluator values and human labels. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `item_id` | string | No | Filter rows to a single item. The full task-wide annotator union is still returned in `annotators` | | `live_only` | boolean | No | When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded | | `disagreement_only` | boolean | No | When true, keep only rows where the evaluator disagreed with at least one annotator | | `q` | string | No | Case-insensitive substring search on `payload.name`. Blank is a no-op | | `sort_by` | string | No | Sort key for the results | | `order` | string | No | Sort direction | | `limit` | integer | No | Maximum number of items to return | | `offset` | integer | No | Number of items to skip before returning results | | `compact` | boolean | No | Return a compact response that omits heavy detail fields (`rows.payload`, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail | See **GET /annotation-tasks/\{task\_uuid}/summary** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-annotation-task Get one annotation task with its evaluators, items, and labelling jobs. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | ---------------- | | `task_uuid` | string | Yes | Task to retrieve | See **GET /annotation-tasks/\{task\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### create-annotation-task Create an annotation task for labelling items against evaluators. Optionally link evaluators in order. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | --------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | Yes | Task name, unique within your workspace | | `type` | string | Yes | Task type. Determines the shape of each item's payload. | | `description` | string | No | A description for the task. Omit for none | | `evaluator_ids` | array of string | No | IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators | See **POST /annotation-tasks** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### run-annotation-evaluators Run one or more of a task's evaluators over its items as a background job. Returns a job\_uuid to poll with get-annotation-evaluator-run. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | --------------------------------- | -------- | ------------------------------------------------------------------------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `evaluators` | array of EvaluatorRunRequestEntry | Yes | The evaluators to run. Each must be linked to the task | | `item_ids` | array of string | No | Item IDs to run on. **Required when `select_all=false`**. Ignored when `select_all=true` | | `select_all` | boolean | No | When `true`, run on every item in the task. Set `q` to run only items whose name matches it | | `q` | string | No | Case-insensitive substring filter on `payload.name`. Applies only when `select_all=true` | See **POST /annotation-tasks/\{task\_uuid}/evaluator-runs** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### update-annotation-items Bulk-update item payloads in an annotation task. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ----------- | -------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | | `task_uuid` | string | Yes | Annotation task to act on | | `updates` | array of ItemUpdatePayload | Yes | The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped | See **PUT /annotation-tasks/\{task\_uuid}/items** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### add-annotation-items Bulk-create items in an annotation task, optionally seeding human annotations. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | -------------- | ------------------------------ | -------- | ----------------------------------------------------------------------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `items` | array of AnnotationItemPayload | Yes | Items to insert. Insertion order is preserved | | `annotator_id` | string | No | Annotator these initial annotations belong to. Required when any item carries annotations | See **POST /annotation-tasks/\{task\_uuid}/items** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### set-task-evaluators Replace the task's linked evaluators with exactly the given ordered set: new ids are linked, omitted ids are unlinked, and the list order sets their display position. Send an empty list to unlink all. Each evaluator must be one you created or a built-in default, and the task must be in your workspace. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | --------------- | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_uuid` | string | Yes | Annotation task to act on | | `evaluator_ids` | array of string | Yes | The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default | See **PUT /annotation-tasks/\{task\_uuid}/evaluators** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ## Evaluators ### list-evaluators List your evaluators. Optionally filter by what they judge or their modality, and include the built-in defaults. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ------------------ | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `evaluator_type` | string | No | Filter by what the evaluator judges. Omit for all types | | `data_type` | string | No | Filter by modality. Omit for all | | `include_defaults` | boolean | No | Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /evaluators** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-evaluator Get one evaluator with its full version history. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ---------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `evaluator_uuid` | string | Yes | Evaluator to retrieve | | `compact` | boolean | No | Return a compact response that omits heavy detail fields (`versions.system_prompt`, `versions.output_config`, `versions.variables`), keeping only the lightweight decision fields. Omit for full detail | See **GET /evaluators/\{evaluator\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### create-evaluator Create an evaluator and its first version, which is set live. An evaluator judges an output against a prompt and rubric. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ---------------- | ---------------------- | -------- | ------------------------------------------------------------------------ | | `name` | string | Yes | Evaluator name, unique within your workspace | | `description` | string | No | Description. Omit to leave blank | | `evaluator_type` | string | No | What the evaluator judges: | | `data_type` | string | No | The modality the judge reads: | | `output_type` | string | No | How the evaluator scores: | | `version` | EvaluatorVersionCreate | Yes | The evaluator's first version. Set as live when you create the evaluator | See **POST /evaluators** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### create-evaluator-version Add a new version to an evaluator you created, optionally making it live. The variable names must match the previous version. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ---------------- | --------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `evaluator_uuid` | string | Yes | Evaluator to add a version to | | `judge_model` | string | Yes | The model that runs the judge, named the way its provider does, for example `openai/gpt-4.1` or `anthropic/claude-sonnet-4` | | `system_prompt` | string | Yes | Judge system prompt. May contain `{{variable}}` placeholders | | `output_config` | OutputConfig | No | The scale points and their labels. Required for a `rating` evaluator. A `binary` evaluator uses the default Correct/Wrong labels unless you set your own | | `variables` | array of VariableSpec | No | Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable's description or default, but not add, remove, or rename one | | `make_live` | boolean | No | When `true`, immediately point the evaluator's live version at this new version | See **POST /evaluators/\{evaluator\_uuid}/versions** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ## Tests ### list-tests List every test in the caller's organization. Returns each test's UUID, name, type, and config. Use this to discover which tests exist before creating runs. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------------------------------------- | | `q` | string | No | Case-insensitive substring search on `name`. Blank is a no-op | | `limit` | integer | No | Maximum number of items to return. Omit for no limit (all items) | | `offset` | integer | No | Number of items to skip before returning results | See **GET /tests** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### get-test Fetch a single test by UUID within the caller's org. Returns the test's name, type, and config. Fails with 404 if the test does not exist or belongs to another org. **Scope:** `read` · **Access:** Read-only **Parameters** | Parameter | Type | Required | Description | | ----------- | ------ | -------- | ---------------- | | `test_uuid` | string | Yes | Test to retrieve | See **GET /tests/\{test\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### create-test Create a new test in the caller's organization. Provide the test's name, type, and config. Returns the created test including its UUID. Requires a Calibrate API key (CALIBRATE\_API\_KEY). **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | ----------------------------------------- | -------- | --------------------------------------------------------------- | | `name` | string | Yes | Name of the test, unique within the workspace | | `type` | string | Yes | What the test judges: | | `config` | object | No | The calibrate test config. Three top-level keys. | | `evaluators` | array of routers\_\_tests\_\_EvaluatorRef | No | Evaluators to link. Used by `response` and `conversation` tests | See **POST /tests** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### bulk-create-tests Create multiple tests in the caller's organization in one call. Provide a list of test definitions (name, type, config). Returns the created tests including their UUIDs. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------- | --------------------- | -------- | --------------------------------------------------------------------------------- | | `type` | string | Yes | What the test judges: | | `tests` | array of BulkTestItem | Yes | Test items to create, at most 500 per request, with names unique within the batch | | `agent_uuids` | array of string | No | IDs of agents to link every created test to. Omit to link none | | `language` | string | No | Language written to each test's `config.settings.language`. Omit to leave unset | See **POST /tests/bulk** in the [API reference](/docs/api-reference/introduction) tab for the full schema. ### update-test Update an existing test by UUID within the caller's org. Supply the fields to change (name, config). Returns the updated test. Fails with 404 if the test does not exist or belongs to another org. **Scope:** `write` · **Access:** Write **Parameters** | Parameter | Type | Required | Description | | ------------ | ----------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `test_uuid` | string | Yes | Test to update | | `name` | string | No | New test name. Omit to leave unchanged | | `type` | string | No | What the test judges: | | `config` | object | No | The calibrate test config. Three top-level keys. | | `evaluators` | array of routers\_\_tests\_\_EvaluatorRef | No | New evaluator links for the test. Omit to leave unchanged. An empty list clears them, except on `conversation` tests, which must keep at least one | See **PUT /tests/\{test\_uuid}** in the [API reference](/docs/api-reference/introduction) tab for the full schema.