Overview

The Calibrate CLI brings the full evaluation loop to your terminal — create and connect agents, author tests and evaluators, launch runs and benchmarks, poll results, and set up annotation tasks — for scripting, automation, and CI/CD integration. It’s agent-mode aware, emitting structured errors and compact TOON output when an AI coding agent drives it.

Calibrate CLI on GitHub

View source, releases, and contribute

What you can do

Manage agents — create, update, list, and verify-connect agents; resolve friendly names to UUIDs
Build test suites — author test cases individually or in bulk, update them, and link them to agents
Configure evaluators — create and version the judges that score runs
Run evaluations — launch test runs and benchmarks for one agent or many, then poll status and per-test-case results
Review with humans — create annotation tasks, run evaluators over them, and measure reviewer agreement
Script anything — JSON, YAML, table, or TOON output with jq filtering, plus a non-interactive mode built for CI/CD

Quick start

Install the CLI

brew install dalmia/tap/calibrate

Authenticate

calibrate login

Create an API key under Workspace settings → API keys — see the API keys guide.

Run agent tests

calibrate agent-tests run \
  --agent-uuid <agent_uuid> \
  --test-uuids <test_uuids>

Returns the evaluation task_id.

Poll for the evaluation results

calibrate agent-tests get-run --task-id <task_id>

Command reference

Agents

List agents and resolve names to UUIDs.

Agent tests

Link tests, launch runs and benchmarks, and poll their results.

Tests

Create test cases individually or in bulk, and update them.

Evaluators

Create and version the judges that score runs.

Annotation tasks

Route runs to human reviewers and measure agreement.

Global flags

All commands support these flags:

Option	Type	Default	Description
`--agent-mode`	—	—	Enable structured errors and default TOON output for AI coding agents. Automatically enabled when a known agent environment is detected (CLAUDE_CODE, CURSOR_AGENT, etc.). Use —agent-mode=false to disable.
`--api-key-auth`	string	—	Workspace API key. Create one under Workspace settings → API keys.
`--color`	string	`auto`	Control colored output: auto (color when output is a TTY), always, or never. Respects NO_COLOR and FORCE_COLOR env vars.
`-d, --debug`	—	—	Log request and response diagnostics to stderr
`--dry-run`	—	—	Preview the request that would be sent without executing it (output to stderr)
`-H, --header`	stringArray	—	Set a custom HTTP request header (format: “Key: Value”). Can be specified multiple times.
`--include-headers`	—	—	Include HTTP response headers in the output
`-q, --jq`	string	—	Filter and transform output using a jq expression (e.g., ‘.name’, ‘.items[] \| .id’)
`--no-interactive`	—	—	Disable all interactive features (auto-prompting, explorer auto-launch, TUI forms)
`-o, --output-format`	string	`pretty`	Specify the output format. Options: pretty, json, yaml, table, toon.
`--server`	string	—	Select a server by index (for indexed servers) or name (for named servers)
`--server-url`	string	—	Override the default server URL
`--timeout`	string	—	HTTP request timeout (e.g., 30s, 5m, 100ms)
`--usage`	—	—	Print the CLI Usage schema in KDL format

JSON output

Use --output-format json to get machine-readable output:

# List agent UUIDs
calibrate agents list --output-format json | jq '.[].uuid'

# Get an evaluation run's status
calibrate agent-tests get-run --task-id <task_id> --output-format json | jq '.status'

Requirements

macOS or Linux

Getting started

Guides

Calibrate CLI on GitHub

What you can do

Quick start

Command reference

Agents

Agent tests

Tests

Evaluators

Annotation tasks

Global flags

JSON output

Requirements

Calibrate CLI on GitHub

​What you can do

​Quick start

​Command reference

Agents

Agent tests

Tests

Evaluators

Annotation tasks

​Global flags

​JSON output

​Requirements

What you can do

Quick start

Command reference

Global flags

JSON output

Requirements