> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Agent Connections
> Connect your existing deployed agent to Calibrate
If you already have a deployed agent, you can connect it to Calibrate and run evaluations, benchmarks, and simulations against it — without rebuilding your agent inside Calibrate.
If you don't have an existing agent and want to build one from scratch within
Calibrate, see [Core Concepts: Agents](/docs/core-concepts/agents) instead.
## How the connection works
Calibrate does not need to understand your whole agent. It needs a single HTTP
endpoint that follows one contract:
* **Request** — Calibrate sends a `POST` request with the conversation history as a `messages`
array in the OpenAI chat message format.
* **Response** — your endpoint returns a JSON object with the agent's reply as a
`response` string, and/or its tool calls as a `tool_calls` array.
### Request format
Calibrate sends a `POST` request to your endpoint. The body carries the full
conversation history in chronological order, using the OpenAI chat message
format:
```json theme={null}
{
"messages": [
{
"role": "assistant",
"content": "Namaste! Main aapki kaise madad kar sakti hoon?"
},
{ "role": "user", "content": "Meri beti ka vaccination schedule kya hai?" }
]
}
```
The request may also carry an optional `model` field. Calibrate adds it only when you [benchmark across models](#enable-benchmarking-across-models), so your endpoint knows which model to run for that request. Handle it only if your agent is set up to switch models from that input; otherwise ignore it.
### Response format
By default, your endpoint returns a `response` field containing the agent's text
reply:
```json theme={null}
{
"response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye."
}
```
If your agent also returns tool calls, expand the body to include a `tool_calls`
array — each item has a `tool` name and an `arguments` object:
```json theme={null}
{
"response": "Aapki beti ka agla vaccination 14 weeks pe hai — OPV aur DPT ke liye.",
"tool_calls": [
{ "tool": "get_schedule", "arguments": { "child_age_weeks": 14 } }
]
}
```
Each tool call may also include an optional `output` field — the result the tool
returned when your agent ran it. It can be any JSON value, is shown in the
results view for review only, and never affects whether a test passes or fails:
```json theme={null}
{
"tool_calls": [
{
"tool": "get_schedule",
"arguments": { "child_age_weeks": 14 },
"output": { "next_visit_weeks": 14, "vaccines": ["OPV", "DPT"] }
}
]
}
```
Your endpoint's output **must** include either a `response` field or a
`tool_calls` field.
### Add a Calibrate endpoint
The change to your codebase: pull the model call out of your agent handler into a
reusable function, then expose a new route that calls it directly.
**Before** — the LLM call is buried inside your request handler:
```python theme={null}
def inference(x1, x2, x3):
y1 = preprocess(x1)
y2 = preprocess(x2)
y3 = preprocess(x3)
messages = [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."},
]
response = client.chat.completions.create(messages=messages) # buried here
return transform(response)
@app.get("/agent")
def agent():
return inference(x1, x2, x3)
```
**After** — the LLM call is modularized and re-used by a thin Calibrate route:
```python theme={null}
def llm_inference(messages):
"""The core model call — reused by both your agent and Calibrate."""
response = client.chat.completions.create(messages=messages)
return response.choices[0].message.content
def inference(x1, x2, x3):
y1 = preprocess(x1)
y2 = preprocess(x2)
y3 = preprocess(x3)
messages = [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."},
]
response = llm_inference(messages) # same call, now shared
return transform(response)
@app.get("/agent")
def agent():
return inference(x1, x2, x3)
@app.post("/calibrate/test") # the endpoint you connect to Calibrate
def calibrate_test(body):
return {"response": llm_inference(body["messages"])}
```
Point Calibrate at the new endpoint `POST /calibrate/test`, add any auth headers your agent needs, and verify the connection as shown below.
## Create an agent connection
From the sidebar, click **Agents** → **New agent**. Select **Connect your existing agent**, give it a name, and click **Create**.
You will be taken to the new agent page where you configure the connection details.
## Configure your connection
The **Connection** tab lets you set up everything Calibrate needs to communicate with your agent:
Fill in the following:
1. **Agent URL** (required) — The HTTPS endpoint where your agent is deployed (e.g. `https://your-agent.example.com/chat`). Calibrate will send a `POST` request to this URL with conversation messages.
2. **Headers** (optional) — Add any headers your agent needs for authentication or custom metadata (e.g. `Authorization: Bearer YOUR_API_KEY`). Click **+ Add header** to add multiple headers.
3. **Support benchmarking different models** (optional) — Turn this on if your agent can switch LLMs from the request `model` field, then pick your **Model provider**. The provider sets the format Calibrate sends in `model`: for OpenRouter it's provider-prefixed (e.g. `openai/gpt-4.1`), for OpenAI it's bare (e.g. `gpt-4.1`). In the API/SDK this is the `benchmark_provider` field on the agent config. See [Enable benchmarking across models](#enable-benchmarking-across-models) for the full flow.
The request and response formats your endpoint must follow are described in
[How the connection works](#how-the-connection-works) above.
## Verify your connection
Before running any tests, we need to verify we can connect to your agent and that it returns the output in the correct format.
Click the **Verify** button in the **Connection check** panel. A dialog will open where you can customize the sample request that will be sent to your agent:
* Edit the **Messages** to set the conversation history for the test request.
* The **Request body preview** on the right updates live so you can see the exact JSON that will be sent.
* Click **Send & Verify** to send the request to your agent.
After verification:
* ✅ **Verified** — Your agent is reachable and returns the correct format
* ❌ **Failed** — The panel will show the error along with the actual output received from your agent so you can debug and try again.
Connection verification is required before running LLM tests, benchmarks or
simulations
Even after your agent is verified, if you make any changes to your agent you can come back here and verify the connection again.
## Enable benchmarking across models
Calibrate supports [benchmarking](/docs/quickstart/text-to-text#find-the-best-llm-for-your-agent) different LLMs on your dataset to find the best model for your agent. For this, you need to instrument your agent API to support a `model` field in the request body that Calibrate will send along with the conversation history.
Your agent is responsible for reading the `model` parameter and setting the
right model to perform the actual inference. Calibrate sends this field but
does not control which model your agent uses.
To get started, turn on **Support benchmarking different models** from the **Connection** tab.
When enabled:
1. Select your **Model provider** from the dropdown (OpenRouter, OpenAI, Anthropic, Google, etc.). This represents the provider you have configured for your agent on your system.
2. Calibrate will include a `model` field in the request body:
```json theme={null}
{
"messages": [...],
"model": "openai/gpt-4.1"
}
```
Depending on the provider you have selected, the format of the `model` field will be different. For example, if you have selected `OpenRouter`, the `model` field is provider-prefixed (e.g. `openai/gpt-4.1`). If you have selected OpenAI, the `model` field will just be `gpt-4.1`.
3. When running a benchmark via **Compare models**, Calibrate will verify the connection for each selected model before starting the evaluation. See [this](/docs/quickstart/text-to-text#verifying-agent-connection) for details.
## Next steps
Create and run test cases against your agent
Compare the performance of different LLMs on your agent
Simulate conversations with realistic personas