Beginner's guide

The Model Context Protocol (MCP) lets an AI assistant like Claude Code or Cursor call external tools. The Calibrate MCP server exposes your Calibrate workspace as tools, so you can create and run evaluations just by describing what you want — no clicking through the UI or writing API calls by hand. If you’ve never set up an MCP server before, this page walks you through it end to end.

​

What you can do

Once connected, ask your coding agent in plain English:

​

Manage agents

“List my agents”, “create an agent called support-bot”, or “update support-bot’s model” — the agent calls the right tool and shows you the result.

​

Build and update tests

“Create a test for the billing flow” or “bulk-upload these test cases” — add and edit the tests your agents are evaluated against.

​

Launch and monitor runs

“Run the tests for support-bot” or “run tests for every agent in my workspace” — kick off evaluation runs and get back a task id.

​

Check results and debug

“What’s the status of run abc123?” — poll a run and read per-test-case pass/fail results and evaluator judgments without leaving your editor.

​

How to get set up

First, create a Calibrate API key under Workspace settings → API keys. The snippets below use the hosted server (nothing to install); for the local npx server and other clients see Installation. Then add the server to your client one of two ways.

​

Option 1: Via the app

Some clients let you edit MCP config from their UI:

1. Open your client’s settings — in Claude Desktop, Settings → Developer → Edit Config.
2. Paste the Calibrate server block (below) into the mcpServers object.
3. Save, then fully quit and reopen the client.

​

Option 2: Via terminal

Edit the config file directly:

1. Open your client’s config file — e.g. on macOS:
   open "$HOME/Library/Application Support/Claude/claude_desktop_config.json"
   
2. Add the Calibrate server, pasting in your API key:
   {
     "mcpServers": {
       "calibrate": {
         "command": "npx",
         "args": [
           "-y",
           "mcp-remote",
           "https://mcp.calibrate.artpark.ai/mcp",
           "--header",
           "X-API-Key: ${CALIBRATE_API_KEY}"
         ],
         "env": {
           "CALIBRATE_API_KEY": "your_api_key"
         }
       }
     }
   }
   
3. Save, then fully quit and reopen the client.

Once it restarts, ask “list my agents” — if you get your agents back, you’re connected.

​

Your first queries

• “List all my agents”
• “Resolve the names support-bot and billing-bot to UUIDs”
• “Run the tests for support-bot and give me the task id”
• “What’s the status and results of run abc123?”

When you’re ready for the full list of what the agent can call, see the Tools reference.