Agent Connections

If you already have a deployed agent, you can connect it to Calibrate and run evaluations, benchmarks, and simulations against it — without rebuilding your agent inside Calibrate.

If you don’t have an existing agent and want to build one from scratch within Calibrate, see Core Concepts: Agents instead.

How the connection works

Calibrate does not need to understand your whole agent. It needs a single HTTP endpoint that follows one contract:

Request — Calibrate sends a POST request with the conversation history as a messages array in the OpenAI chat message format.
Response — your endpoint returns a JSON object with the agent’s reply as a response string, and/or its tool calls as a tool_calls array.

Request format

Calibrate sends a POST request to your endpoint. The body carries the full conversation history in chronological order, using the OpenAI chat message format:

{
  "messages": [
    {
      "role": "assistant",
      "content": "Namaste! Main aapki kaise madad kar sakti hoon?"
    },
    { "role": "user", "content": "Meri beti ka vaccination schedule kya hai?" }
  ]
}

The request may also carry an optional model field. Calibrate adds it only when you benchmark across models, so your endpoint knows which model to run for that request. Handle it only if your agent is set up to switch models from that input; otherwise ignore it.

Response format

By default, your endpoint returns a response field containing the agent’s text reply:

{
  "response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye."
}

If your agent also returns tool calls, expand the body to include a tool_calls array — each item has a tool name and an arguments object:

{
  "response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye.",
  "tool_calls": [
    { "tool": "get_schedule", "arguments": { "child_age_weeks": 14 } }
  ]
}

Each tool call may also include an optional output field — the result the tool returned when your agent ran it. It can be any JSON value, is shown in the results view for review only, and never affects whether a test passes or fails:

{
  "tool_calls": [
    {
      "tool": "get_schedule",
      "arguments": { "child_age_weeks": 14 },
      "output": { "next_visit_weeks": 14, "vaccines": ["OPV", "DPT"] }
    }
  ]
}

Your endpoint’s output must include either a response field or a tool_calls field.

Add a Calibrate endpoint

The change to your codebase: pull the model call out of your agent handler into a reusable function, then expose a new route that calls it directly. Before — the LLM call is buried inside your request handler:

def inference(x1, x2, x3):
    y1 = preprocess(x1)
    y2 = preprocess(x2)
    y3 = preprocess(x3)

    messages = [
        {"role": "user", "content": "..."},
        {"role": "assistant", "content": "..."},
        {"role": "user", "content": "..."},
    ]

    response = client.chat.completions.create(messages=messages)  # buried here
    return transform(response)

@app.get("/agent")
def agent():
    return inference(x1, x2, x3)

After — the LLM call is modularized and re-used by a thin Calibrate route:

def llm_inference(messages):
    """The core model call — reused by both your agent and Calibrate."""
    response = client.chat.completions.create(messages=messages)
    return response.choices[0].message.content

def inference(x1, x2, x3):
    y1 = preprocess(x1)
    y2 = preprocess(x2)
    y3 = preprocess(x3)

    messages = [
        {"role": "user", "content": "..."},
        {"role": "assistant", "content": "..."},
        {"role": "user", "content": "..."},
    ]

    response = llm_inference(messages)   # same call, now shared
    return transform(response)

@app.get("/agent")
def agent():
    return inference(x1, x2, x3)

@app.post("/calibrate/test")           # the endpoint you connect to Calibrate
def calibrate_test(body):
    return {"response": llm_inference(body["messages"])}

Point Calibrate at the new endpoint POST /calibrate/test, add any auth headers your agent needs, and verify the connection as shown below.

Create an agent connection

From the sidebar, click Agents → New agent. Select Connect your existing agent, give it a name, and click Create.

You will be taken to the new agent page where you configure the connection details.

Configure your connection

The Connection tab lets you set up everything Calibrate needs to communicate with your agent:

Fill in the following:

Agent URL (required) — The HTTPS endpoint where your agent is deployed (e.g. https://your-agent.example.com/chat). Calibrate will send a POST request to this URL with conversation messages.
Headers (optional) — Add any headers your agent needs for authentication or custom metadata (e.g. Authorization: Bearer YOUR_API_KEY). Click + Add header to add multiple headers.
Support benchmarking different models (optional) — Turn this on if your agent can switch LLMs from the request model field, then pick your Model provider. The provider sets the format Calibrate sends in model: for OpenRouter it’s provider-prefixed (e.g. openai/gpt-4.1), for OpenAI it’s bare (e.g. gpt-4.1). In the API/SDK this is the benchmark_provider field on the agent config. See Enable benchmarking across models for the full flow.

The request and response formats your endpoint must follow are described in How the connection works above.

Verify your connection

Before running any tests, we need to verify we can connect to your agent and that it returns the output in the correct format. Click the Verify button in the Connection check panel. A dialog will open where you can customize the sample request that will be sent to your agent:

Edit the Messages to set the conversation history for the test request.
The Request body preview on the right updates live so you can see the exact JSON that will be sent.
Click Send & Verify to send the request to your agent.

After verification:

✅ Verified — Your agent is reachable and returns the correct format
❌ Failed — The panel will show the error along with the actual output received from your agent so you can debug and try again.

Connection verification is required before running LLM tests, benchmarks or simulations

Even after your agent is verified, if you make any changes to your agent you can come back here and verify the connection again.

Enable benchmarking across models

Calibrate supports benchmarking different LLMs on your dataset to find the best model for your agent. For this, you need to instrument your agent API to support a model field in the request body that Calibrate will send along with the conversation history.

Your agent is responsible for reading the model parameter and setting the right model to perform the actual inference. Calibrate sends this field but does not control which model your agent uses.

To get started, turn on Support benchmarking different models from the Connection tab.

Agent Connection with Benchmarking Enabled

When enabled:

Select your Model provider from the dropdown (OpenRouter, OpenAI, Anthropic, Google, etc.). This represents the provider you have configured for your agent on your system.
Calibrate will include a model field in the request body:

{
  "messages": [...],
  "model": "openai/gpt-4.1"
}

Depending on the provider you have selected, the format of the model field will be different. For example, if you have selected OpenRouter, the model field is provider-prefixed (e.g. openai/gpt-4.1). If you have selected OpenAI, the model field will just be gpt-4.1.

When running a benchmark via Compare models, Calibrate will verify the connection for each selected model before starting the evaluation. See this for details.

Next steps

Run LLM Tests

Create and run test cases against your agent

Find the best LLM

Compare the performance of different LLMs on your agent

Run Simulations

Simulate conversations with realistic personas

Get Started

Quickstart

Core Concepts

Reference

How the connection works

Request format

Response format

Add a Calibrate endpoint

Create an agent connection

Configure your connection

Verify your connection

Enable benchmarking across models

Next steps

Run LLM Tests

Find the best LLM

Run Simulations

​How the connection works

​Request format

​Response format

​Add a Calibrate endpoint

​Create an agent connection

​Configure your connection

​Verify your connection

​Enable benchmarking across models

​Next steps

Run LLM Tests

Find the best LLM

Run Simulations

How the connection works

Request format

Response format

Add a Calibrate endpoint

Create an agent connection

Configure your connection

Verify your connection

Enable benchmarking across models

Next steps