> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent tests

> Operations for agent-tests

## Run agent benchmark

Run a multi-model benchmark on an agent's linked tests as a background job.

```bash theme={null}
calibrate agent-tests benchmark [flags]
```

**Options**

| Option             | Type        | Description                                                                                                          |
| ------------------ | ----------- | -------------------------------------------------------------------------------------------------------------------- |
| `-a, --agent-uuid` | string      | **Required.** Agent to benchmark                                                                                     |
| `-m, --models`     | stringArray | **Required.** Model names to benchmark                                                                               |
| `-t, --test-uuids` | string      | A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests |

**Examples**

```bash theme={null}
calibrate agent-tests benchmark --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --models '["openai/gpt-4.1","anthropic/claude-sonnet-4"]'
```

## Get benchmark status

Get the results of a benchmark run

```bash theme={null}
calibrate agent-tests get-benchmark [flags]
```

**Options**

| Option          | Type                         | Description                                                                                                                                                                             |
| --------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-c, --compact` | model\_results.test\_results | Return a compact response that omits heavy detail fields (model\_results.test\_results, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
| `--only-failed` | —                            | Return only failing test cases for each model. Omit to return every case                                                                                                                |
| `-t, --task-id` | string                       | **Required.** Benchmark run to poll for status and results                                                                                                                              |

**Examples**

```bash theme={null}
calibrate agent-tests get-benchmark --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890
```

## Get test run status

Poll a test run for its status and evaluation results.

```bash theme={null}
calibrate agent-tests get-run [flags]
```

**Options**

| Option          | Type           | Description                                                                                                                                                                                                                                  |
| --------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-c, --compact` | results.output | Return a compact response that omits heavy detail fields (results.output, `results.test_case`, `results.judge_results`, `results.reasoning`, `evaluators.output_config`), keeping only the lightweight decision fields. Omit for full detail |
| `--only-failed` | —              | Return only failing test cases. Omit to return every case                                                                                                                                                                                    |
| `-t, --task-id` | string         | **Required.** Test run to poll for status and results                                                                                                                                                                                        |

**Examples**

```bash theme={null}
calibrate agent-tests get-run --task-id a3b2c1d0-e5f4-3210-abcd-ef1234567890
```

## Link tests to agent

Link one or more tests to an agent. Tests that are already linked are skipped.

```bash theme={null}
calibrate agent-tests link [flags]
```

**Options**

| Option             | Type        | Description                                                          |
| ------------------ | ----------- | -------------------------------------------------------------------- |
| `-a, --agent-uuid` | string      | **Required.** Agent to link tests to                                 |
| `-t, --test-uuids` | stringArray | **Required.** Tests to link. Any that are already linked are skipped |

**Examples**

```bash theme={null}
calibrate agent-tests link --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --test-uuids '["b1c2d3e4-f5a6-7890-bcde-f12345678901"]'
```

## List tests for agent

List the tests linked to an agent.

```bash theme={null}
calibrate agent-tests list-for-agent [flags]
```

**Options**

| Option             | Type   | Description                                                      |
| ------------------ | ------ | ---------------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** Agent whose linked tests to list                   |
| `-l, --limit`      | string | Maximum number of items to return. Omit for no limit (all items) |
| `--offset`         | int    | Number of items to skip before returning results                 |
| `--q`              | name   | Case-insensitive substring search on name. Blank is a no-op      |

**Examples**

```bash theme={null}
calibrate agent-tests list-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```

## List test runs for agent

List an agent's test runs with their results

```bash theme={null}
calibrate agent-tests list-runs-for-agent [flags]
```

**Options**

| Option             | Type          | Description                                                                                                                                            |
| ------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-a, --agent-uuid` | string        | **Required.** Agent whose test runs to list                                                                                                            |
| `--has-failures`   | true          | Filter by whether the run has any failing test case or model. true returns only runs with failures (or errors), `false` only clean runs. Omit for both |
| `-l, --limit`      | string        | Maximum number of items to return. Omit for no limit (all items)                                                                                       |
| `--offset`         | int           | Number of items to skip before returning results                                                                                                       |
| `-s, --status`     | string        | Filter by run status. Omit for all statuses (options: queued, in\_progress, cancelled, done, failed)                                                   |
| `-t, --type`       | llm-unit-test | Filter by run type. Omit to return both:                                                                                                               |

**Examples**

```bash theme={null}
calibrate agent-tests list-runs-for-agent --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```

## Run agent tests

Run an agent's linked tests as a background job, returning a task ID to poll.

```bash theme={null}
calibrate agent-tests run [flags]
```

**Options**

| Option             | Type   | Description                                             |
| ------------------ | ------ | ------------------------------------------------------- |
| `-a, --agent-uuid` | string | **Required.** Agent to test                             |
| `-t, --test-uuids` | string | Tests to run. Omit to run all tests linked to the agent |

**Examples**

```bash theme={null}
calibrate agent-tests run --agent-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
```

## Run agent tests in batch

Run agent tests for every agent, or for a selected set.

```bash theme={null}
calibrate agent-tests run-batch [flags]
```

**Options**

| Option             | Type   | Description                                                                             |
| ------------------ | ------ | --------------------------------------------------------------------------------------- |
| `--body`           | string | Request body as JSON (alternative to individual flags). Can also be provided via stdin. |
| `-b, --body-param` | string | JSON object                                                                             |

**Examples**

```bash theme={null}
calibrate agent-tests run-batch
```