> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Command-line interface for the Calibrate AI evaluation platform

The Calibrate CLI brings the full evaluation loop to your terminal — create and
connect agents, author tests and evaluators, launch runs and benchmarks, poll
results, and set up annotation tasks — for scripting, automation, and CI/CD
integration. It's [agent-mode](/docs/cli/calibrate/agent-mode) aware, emitting
structured errors and compact TOON output when an AI coding agent drives it.

<Card title="Calibrate CLI on GitHub" icon="github" href="https://github.com/dalmia/calibrate-cli">
  View source, releases, and contribute
</Card>

## What you can do

* **Manage agents** — create, update, list, and verify-connect agents; resolve friendly names to UUIDs
* **Build test suites** — author test cases individually or in bulk, update them, and link them to agents
* **Configure evaluators** — create and version the judges that score runs
* **Run evaluations** — launch test runs and benchmarks for one agent or many, then poll status and per-test-case results
* **Review with humans** — create annotation tasks, run evaluators over them, and measure reviewer agreement
* **Script anything** — JSON, YAML, table, or TOON output with `jq` filtering, plus a non-interactive mode built for CI/CD

## Quick start

<Steps>
  <Step title="Install the CLI">
    ```bash theme={null}
    brew install dalmia/tap/calibrate
    ```
  </Step>

  <Step title="Authenticate">
    ```bash theme={null}
    calibrate login
    ```

    Create an API key under [**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) — see the [API keys guide](/docs/reference/api-keys).
  </Step>

  <Step title="Run agent tests">
    ```bash theme={null}
    calibrate agent-tests run \
      --agent-uuid <agent_uuid> \
      --test-uuids <test_uuids>
    ```

    Returns the evaluation `task_id`.
  </Step>

  <Step title="Poll for the evaluation results">
    ```bash theme={null}
    calibrate agent-tests get-run --task-id <task_id>
    ```
  </Step>
</Steps>

## Command reference

<CardGroup cols={2}>
  <Card title="Agents" icon="robot" href="/docs/cli/calibrate/agents">
    List agents and resolve names to UUIDs.
  </Card>

  <Card title="Agent tests" icon="flask" href="/docs/cli/calibrate/agent-tests">
    Link tests, launch runs and benchmarks, and poll their results.
  </Card>

  <Card title="Tests" icon="list-check" href="/docs/cli/calibrate/tests">
    Create test cases individually or in bulk, and update them.
  </Card>

  <Card title="Evaluators" icon="scale-balanced" href="/docs/cli/calibrate/evaluators">
    Create and version the judges that score runs.
  </Card>

  <Card title="Annotation tasks" icon="user-check" href="/docs/cli/calibrate/annotation-tasks">
    Route runs to human reviewers and measure agreement.
  </Card>
</CardGroup>

## Global flags

All commands support these flags:

| Option                | Type        | Default  | Description                                                                                                                                                                                                     |
| --------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--agent-mode`        | —           | —        | Enable structured errors and default TOON output for AI coding agents. Automatically enabled when a known agent environment is detected (CLAUDE\_CODE, CURSOR\_AGENT, etc.). Use --agent-mode=false to disable. |
| `--api-key-auth`      | string      | —        | Workspace API key. Create one under Workspace settings → API keys.                                                                                                                                              |
| `--color`             | string      | `auto`   | Control colored output: auto (color when output is a TTY), always, or never. Respects NO\_COLOR and FORCE\_COLOR env vars.                                                                                      |
| `-d, --debug`         | —           | —        | Log request and response diagnostics to stderr                                                                                                                                                                  |
| `--dry-run`           | —           | —        | Preview the request that would be sent without executing it (output to stderr)                                                                                                                                  |
| `-H, --header`        | stringArray | —        | Set a custom HTTP request header (format: "Key: Value"). Can be specified multiple times.                                                                                                                       |
| `--include-headers`   | —           | —        | Include HTTP response headers in the output                                                                                                                                                                     |
| `-q, --jq`            | string      | —        | Filter and transform output using a jq expression (e.g., '.name', '.items\[] \| .id')                                                                                                                           |
| `--no-interactive`    | —           | —        | Disable all interactive features (auto-prompting, explorer auto-launch, TUI forms)                                                                                                                              |
| `-o, --output-format` | string      | `pretty` | Specify the output format. Options: pretty, json, yaml, table, toon.                                                                                                                                            |
| `--server`            | string      | —        | Select a server by index (for indexed servers) or name (for named servers)                                                                                                                                      |
| `--server-url`        | string      | —        | Override the default server URL                                                                                                                                                                                 |
| `--timeout`           | string      | —        | HTTP request timeout (e.g., 30s, 5m, 100ms)                                                                                                                                                                     |
| `--usage`             | —           | —        | Print the CLI Usage schema in KDL format                                                                                                                                                                        |

## JSON output

Use `--output-format json` to get machine-readable output:

```bash theme={null}
# List agent UUIDs
calibrate agents list --output-format json | jq '.[].uuid'

# Get an evaluation run's status
calibrate agent-tests get-run --task-id <task_id> --output-format json | jq '.status'
```

## Requirements

* macOS or Linux