> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Expose Calibrate to coding agents over the Model Context Protocol

The Calibrate [MCP server](https://modelcontextprotocol.io) lets AI assistants
like Claude Code, Cursor, and Codex work with Calibrate's evaluation API through
the Model Context Protocol — so they can manage agents, tests, and evaluators,
launch runs and benchmarks, and set up annotation tasks without leaving your
editor. Every tool maps 1-to-1 to a
[public API](/docs/api-reference/introduction) operation.

## What you can do

Ask your coding agent to:

* Create and connect agents, and resolve friendly names to UUIDs
* Author, bulk-upload, and update test cases
* Create and version evaluators
* Link tests, launch runs and benchmarks, and poll per-test-case results
* Create annotation tasks, run evaluators over them, and check reviewer agreement

## Quick start

<Steps>
  <Step title="Get your API key">
    Create one under [**Workspace settings → API keys**](https://calibrate.artpark.ai/workspace-settings?tab=api-keys) — see the [API keys guide](/docs/reference/api-keys).
  </Step>

  <Step title="Configure your MCP client">
    Add the Calibrate MCP server to your client's config:

    <Tabs>
      <Tab title="Hosted (recommended)">
        Connects to Calibrate's managed MCP endpoint via `mcp-remote` — no local server to maintain:

        ```bash theme={null}
        claude mcp add --transport http calibrate \
          https://mcp.calibrate.artpark.ai/mcp \
          --header "X-API-Key: your_api_key"
        ```
      </Tab>

      <Tab title="Local (npx)">
        Runs the server on your own machine:

        ```bash theme={null}
        npx -y @dalmia/calibrate-mcp start --api-key-auth your_api_key
        ```
      </Tab>
    </Tabs>

    See [Installation](/docs/mcp/installation) for config file locations per client and platform.
  </Step>

  <Step title="Restart your client">
    Fully quit and reopen your MCP-compatible app so it picks up the new server.
  </Step>

  <Step title="Start asking">
    Query your agent — "list my agents", "run tests for support-bot", "what's the status of run abc123".
  </Step>
</Steps>

## Available tools

The Calibrate MCP server exposes 35 tools across 5 categories:

| Category                                        | Tools                                                                                                                                                                                                                                                                 | Purpose                             |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| [Agent tests](/docs/mcp/tools#agent-tests)           | `list-agent-test-runs`, `list-agent-tests`, `get-agent-benchmark`, `get-agent-test-run`, `run-agent-benchmark`, `run-agent-tests`, `run-agent-tests-batch`, `link-tests-to-agent`                                                                                     | Launch and monitor agent test runs  |
| [Agents](/docs/mcp/tools#agents)                     | `list-agent-evaluators`, `list-agents`, `get-agent`, `resolve-agent-names`, `create-agent`, `update-agent`, `link-evaluators-to-agent`, `verify-agent-connection`                                                                                                     | Manage the agents in your workspace |
| [Annotation tasks](/docs/mcp/tools#annotation-tasks) | `list-annotation-tasks`, `get-annotation-agreement`, `get-annotation-evaluator-run`, `get-annotation-summary`, `get-annotation-task`, `create-annotation-task`, `run-annotation-evaluators`, `update-annotation-items`, `add-annotation-items`, `set-task-evaluators` | Create and run annotation tasks     |
| [Evaluators](/docs/mcp/tools#evaluators)             | `list-evaluators`, `get-evaluator`, `create-evaluator`, `create-evaluator-version`                                                                                                                                                                                    | Create and version evaluators       |
| [Tests](/docs/mcp/tools#tests)                       | `list-tests`, `get-test`, `create-test`, `bulk-create-tests`, `update-test`                                                                                                                                                                                           | Create and manage tests             |

See the [Tools reference](/docs/mcp/tools) for each tool's parameters and the API
operation it calls.

## Example queries

Once connected, you can ask your coding agent things like:

* "List all my agents"
* "Resolve the names support-bot and billing-bot to UUIDs"
* "Create a test case for support-bot and link it"
* "Run the tests for support-bot and give me the task id"
* "Run tests for every agent in my workspace"
* "What's the status and results of run abc123?"
* "Create an annotation task from that run and check reviewer agreement"

## Requirements

* Node.js 20+
* A Calibrate API key
* An MCP-compatible client (Claude Code, Cursor, Codex, Claude Desktop)

## Support

<CardGroup cols={2}>
  <Card title="Source" icon="github" href="https://github.com/dalmia/calibrate-mcp">
    calibrate-mcp
  </Card>

  <Card title="API keys" icon="key" href="/docs/reference/api-keys">
    Create and manage keys
  </Card>
</CardGroup>