Overview - Calibrate

The Calibrate MCP server lets AI assistants like Claude Code, Cursor, and Codex work with Calibrate’s evaluation API through the Model Context Protocol — so they can manage agents, tests, and evaluators, launch runs and benchmarks, and set up annotation tasks without leaving your editor. Every tool maps 1-to-1 to a public API operation.

What you can do

Ask your coding agent to:

Create and connect agents, and resolve friendly names to UUIDs
Author, bulk-upload, and update test cases
Create and version evaluators
Link tests, launch runs and benchmarks, and poll per-test-case results
Create annotation tasks, run evaluators over them, and check reviewer agreement

Quick start

Get your API key

Create one under Workspace settings → API keys — see the API keys guide.

Configure your MCP client

Add the Calibrate MCP server to your client’s config:

Hosted (recommended)
Local (npx)

Connects to Calibrate’s managed MCP endpoint via mcp-remote — no local server to maintain:

claude mcp add --transport http calibrate \
  https://mcp.calibrate.artpark.ai/mcp \
  --header "X-API-Key: your_api_key"

Runs the server on your own machine:

npx -y @dalmia/calibrate-mcp start --api-key-auth your_api_key

See Installation for config file locations per client and platform.

Restart your client

Fully quit and reopen your MCP-compatible app so it picks up the new server.

Start asking

Query your agent — “list my agents”, “run tests for support-bot”, “what’s the status of run abc123”.

Available tools

The Calibrate MCP server exposes 35 tools across 5 categories:

Category	Tools	Purpose
Agent tests	`list-agent-test-runs`, `list-agent-tests`, `get-agent-benchmark`, `get-agent-test-run`, `run-agent-benchmark`, `run-agent-tests`, `run-agent-tests-batch`, `link-tests-to-agent`	Launch and monitor agent test runs
Agents	`list-agent-evaluators`, `list-agents`, `get-agent`, `resolve-agent-names`, `create-agent`, `update-agent`, `link-evaluators-to-agent`, `verify-agent-connection`	Manage the agents in your workspace
Annotation tasks	`list-annotation-tasks`, `get-annotation-agreement`, `get-annotation-evaluator-run`, `get-annotation-summary`, `get-annotation-task`, `create-annotation-task`, `run-annotation-evaluators`, `update-annotation-items`, `add-annotation-items`, `set-task-evaluators`	Create and run annotation tasks
Evaluators	`list-evaluators`, `get-evaluator`, `create-evaluator`, `create-evaluator-version`	Create and version evaluators
Tests	`list-tests`, `get-test`, `create-test`, `bulk-create-tests`, `update-test`	Create and manage tests

See the Tools reference for each tool’s parameters and the API operation it calls.

Example queries

Once connected, you can ask your coding agent things like:

“List all my agents”
“Resolve the names support-bot and billing-bot to UUIDs”
“Create a test case for support-bot and link it”
“Run the tests for support-bot and give me the task id”
“Run tests for every agent in my workspace”
“What’s the status and results of run abc123?”
“Create an annotation task from that run and check reviewer agreement”

Requirements

Node.js 20+
A Calibrate API key
An MCP-compatible client (Claude Code, Cursor, Codex, Claude Desktop)

Support

Source

calibrate-mcp

API keys

Create and manage keys

​What you can do

​Quick start

​Available tools

​Example queries

​Requirements

​Support