LLM tests

Set up your agent

There are two ways to run LLM evaluations:

Calibrate agent

Define your agent completely within Calibrate

Agent connection

Connect your existing text or voice agent

Test cases

An array of test cases to evaluate your agent on. Each test case has:

Key	Type	Description
`id`	string	Optional unique test-case id. When provided, results include `test_case_id`.
`history`	array	Conversation history as context for the agent to generate the next output
`evaluation`	object	The criteria for evaluating the agent — a tool call, a response, or a whole conversation

Tool call test

Verifies the agent calls the correct tools with the expected arguments:

{
  // tool_call test
  "id": "test-id-1",
  "history": [
    { "role": "assistant", "content": "Hello! How can I help you today?" },
    {
      "role": "user",
      "content": "I want to check my order status. My order ID is ORD-12345.",
    },
  ],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "check_order",
        "arguments": {
          "order_id": "ORD-12345",
        },
      },
    ],
  },
}

Passing "arguments": null will make the agent simply check if the tool is called without checking the arguments.

Judging an argument against a criteria

If exact matching is too strict, you can also pass a criteria that an LLM judge uses to evaluate if the argument is correct. Each argument is matched independently, letting you mix exact matches and criteria-based evaluations in the same tool call:

{
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "send_sms",
        "arguments": {
          // exact match
          "phone_number": {
            "match_type": "exact",
            "value": "555-123-4567",
          },
          // judged by an LLM against the criteria
          "message": {
            "match_type": "llm_judge",
            "criteria": "A friendly appointment reminder that includes the date",
          },
        },
      },
    ],
  },
}

A plain literal value (e.g. "555-123-4567") is matched exactly, as usual. To opt an argument out of exact matching, give it a spec with one of these match_type values:

`match_type`	Shape	Behavior
`"exact"`	`{"match_type": "exact", "value": <literal>}`	Check if the produced value is equal to the `<literal>` value
`"llm_judge"`	`{"match_type": "llm_judge", "criteria": "..."}`	An LLM judge checks the produced value against `criteria`
`"any"`	`{"match_type": "any"}`	Wildcard — the argument is ignored entirely (see below)

To leave an argument completely unconstrained, give it an "any" spec. This is a wildcard: the parameter is ignored entirely — it need not appear in the produced tool call, and whatever value it has is accepted. Use it to pin down only the arguments that matter and leave free-form or non-deterministic ones unchecked.

{
  "tool": "send_sms",
  "arguments": {
    "phone_number": "555-123-4567", // exact match
    "message": { "match_type": "any" }, // ignored — not checked
  },
}

Works for nested arguments as well where each sub-parameter is graded independently:

{
  "tool": "book_appointment",
  "arguments": {
    "patient": {
      "name": "John Doe", // nested exact match
      "note": {
        // nested LLM-judged
        "match_type": "llm_judge",
        "criteria": "Describes a medical symptom or complaint",
      },
    },
  },
}

For agent connection, your agent owns its tools internally — there is no need to define them in the config. The history and evaluation keys work identically for both setup options.

Response test

Given a conversation history, this test verifies that the agent’s response meets the evaluation criteria defined by you (evaluated by an LLM judge):

{
  // response test — implicit default correctness evaluator (string criteria)
  "id": "test-id-1",
  "history": [
    { "role": "assistant", "content": "Hello! How can I help you today?" },
    { "role": "user", "content": "What's the weather like?" },
  ],
  "evaluation": {
    "type": "response",
    "criteria": "The assistant should politely redirect the conversation back to store-related topics.",
  },
}

Conversation test

Like a response test, the agent is run on the history to produce its next reply — but instead of grading only that reply, the entire resulting conversation (history + the new reply) is graded by the simulation judge. Use this when correctness depends on the whole exchange, not just the latest turn. As with response tests, history ends at the user’s turn that the agent should answer.

{
  // conversation test — agent replies, then the whole conversation is judged
  "id": "test-id-1",
  "history": [
    { "role": "assistant", "content": "Hello! How can I help you today?" },
    { "role": "user", "content": "I'd like to check order ORD-12345." },
  ],
  "evaluation": {
    "type": "conversation",
    "criteria": [
      { "name": "resolution", "arguments": { "topic": "order status" } },
      { "name": "tone" },
    ],
  },
}

See examples/llm/config-conversation.json for a full template.

Evaluators

Response test cases can be evaluated by one or more text LLM judges (routed through OpenRouter, set OPENROUTER_API_KEY in your environment). Each evaluator’s system_prompt is sent to its own dedicated LLM judge call (one call per evaluator, run in parallel) with the conversation history and the agent’s last response as the inputs.

Defining custom evaluators

Define one or more evaluators at the top level and reference them by name from each test case. Each evaluator is an independent LLM call and produces its own column in the leaderboard:

{
  "evaluators": [
    {
      "id": "accuracy-id",
      "name": "accuracy",
      "system_prompt": "You are a highly accurate evaluator. You will be given a conversation and the agent's final response. Mark True if the response is factually correct and relevant.",
      "judge_model": "openai/gpt-4.1"
    },
    {
      "id": "tone-id",
      "name": "tone",
      "system_prompt": "You are a highly accurate evaluator. You will be given a conversation and the agent's final response. Rate the tone on a 1-5 scale: 1 = rude/robotic, 3 = neutral, 5 = warm and helpful.",
      "judge_model": "openai/gpt-4.1",
      "type": "rating",
      "scale_min": 1,
      "scale_max": 5
    }
  ],
  "test_cases": [
    {
      // response — custom named evaluators (accuracy + tone)
      "id": "test-id-1",
      "history": [...],
      "evaluation": {
        "type": "response",
        "criteria": [
          { "name": "accuracy" },
          { "name": "tone" }
        ]
      }
    }
  ]
}

Using the default evaluator

The simplest setup needs no top-level evaluators. Pass criteria as a string and the implicit correctness evaluator runs. The string is substituted into the default evaluator’s system prompt as the {{criteria}} variable:

{
  // response — implicit default correctness evaluator (string criteria only)
  "evaluation": {
    "type": "response",
    "criteria": "The assistant should politely redirect to store-related topics.",
  },
}

correctness (default evaluator system prompt)

When you omit top-level evaluators and pass a string criteria, this is the implicit evaluator from DEFAULT_LLM_TEST_EVALUATOR that is used by default. Your criteria string is substituted for {{criteria}}.

You are a highly accurate evaluator evaluating the response of an agent to a user's message.

You will be given a conversation between a user and an agent along with the response of the agent to the final user message.

You need to evaluate if the response adheres to the evaluation criteria:

{{criteria}}

Key	Type	Description
`evaluators`	array	Top-level list of evaluators. Each one becomes its own LLM call per test case.
`evaluators[].id`	string	Optional unique id. Results echo it as `evaluator_id`; output `config.json` includes raw `evaluators` and an `evaluators_map`.
`evaluators[].name`	string	Unique evaluator name. Test cases reference it via `evaluation.criteria[].name`.
`evaluators[].system_prompt`	string	Full system prompt for this evaluator. Supports `{{variable}}` placeholders that are substituted from the test case’s `arguments`.
`evaluators[].judge_model`	string	OpenRouter model id for this evaluator (default: `openai/gpt-5.4-mini`).
`evaluators[].type`	string	`"binary"` (default) or `"rating"`.
`evaluators[].scale_min`	integer	Required when `type` is `"rating"`. Lowest allowed score.
`evaluators[].scale_max`	integer	Required when `type` is `"rating"`. Highest allowed score.

Variable substitution

A test case can pass arguments to an evaluator. Each {{variable}} placeholder in the evaluator’s system_prompt is replaced with the matching value before the judge call:

{
  "evaluators": [
    {
      "id": "correctness-id",
      "name": "correctness",
      "system_prompt": "You are a highly accurate evaluator. Evaluate whether the agent's response satisfies: {{criteria}}",
      "judge_model": "openai/gpt-4.1"
    }
  ],
  "test_cases": [
    {
      // response — custom correctness evaluator with {{criteria}} substituted from arguments
      "history": [...],
      "evaluation": {
        "type": "response",
        "criteria": [
          { "name": "correctness", "arguments": { "criteria": "Politely redirect off-topic questions." } }
        ]
      }
    }
  ]
}

The string-form criteria (shown in the quickstart above) is just a shortcut for the implicit correctness evaluator with { "arguments": { "criteria": "<your string>" } }.

Binary vs rating

Binary evaluators produce per-row pass/fail and a mean pass-rate in aggregates. Rating evaluators report mean/min/max scores per model on the leaderboard. At test-case pass/fail time, every referenced evaluator must pass: binary evaluators require match: true, and rating evaluators require the numeric score to equal scale_max (anything lower fails that evaluator). So on a 1–5 scale (scale_min 1, scale_max 5), only a judge score of 5 counts as pass — intermediate scores fail unless they hit the top of your declared scale. If you want thresholds different from “top-of-scale” (for example “tone at least 4”), encode that as a binary evaluator whose system_prompt asks the judge explicitly (for example: “Mark True if and only if tone is at least 4 on a 1–5 scale”).

Full examples

Calibrate agent example

Same structure as examples/llm/config-internal-agent.json, with documentation-only // comments before each test case (jsonc — strip comments if you paste into a strict JSON parser).

{
  "system_prompt": "You are a helpful customer support assistant for an online store.\n\nYou help customers with:\n1. Checking order status\n2. Processing returns\n3. Answering product questions\n\nWhen a customer asks about their order, call the `check_order` webhook tool with their order ID to fetch the status.\n\nWhen a customer provides feedback or sentiment during the conversation, call the `log_interaction` tool to record the interaction type, sentiment, and a short free-text summary of the interaction.\n\nBe friendly, helpful, and concise in your responses.",
  "tools": [
    {
      "type": "client",
      "name": "log_interaction",
      "description": "Log customer interaction details including the type of inquiry and customer sentiment",
      "parameters": [
        {
          "id": "interaction_type",
          "type": "string",
          "description": "Type of customer interaction: order_inquiry, return_request, product_question, complaint, or general",
          "required": true
        },
        {
          "id": "sentiment",
          "type": "string",
          "description": "Customer sentiment: positive, neutral, or negative",
          "required": true
        },
        {
          "id": "summary",
          "type": "string",
          "description": "A short free-text summary of what the customer said",
          "required": true
        }
      ]
    },
    {
      "type": "webhook",
      "name": "check_order",
      "description": "Check the status of a customer order by order ID",
      "parameters": [ ],
      "webhook": {
        "method": "GET",
        "url": "https://api.example.com/orders",
        "timeout": 10,
        "headers": [ { "name": "Authorization", "value": "Bearer API_KEY" } ],
        "queryParameters": [
          {
            "id": "order_id",
            "type": "string",
            "description": "The customer's order ID",
            "required": true
          }
        ]
      }
    }
  ],
  "evaluators": [
    {
      "id": "relevance-id",
      "name": "relevance",
      "system_prompt": "You are a highly accurate evaluator evaluating the response to a conversation.\n\nYou will be given a conversation between a user and a human agent along with the response of the human agent to the final user message.\n\nMark True if the assistant's response is about product availability or asks for more details about the laptop.",
      "judge_model": "openai/gpt-4.1"
    },
    {
      "id": "tone-id",
      "name": "tone",
      "system_prompt": "You are a highly accurate evaluator evaluating the response to a conversation.\n\nYou will be given a conversation between a user and a human agent along with the response of the human agent to the final user message.\n\nRate the tone on a 1-5 scale: 1 = rude/robotic, 3 = neutral, 5 = warm and helpful.",
      "judge_model": "openai/gpt-4.1",
      "type": "rating",
      "scale_min": 1,
      "scale_max": 5
    }
  ],
  "test_cases": [
    {
      // tool_call — expect check_order (webhook)
      "id": "test-tool-call-1",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "I want to check my order status. My order ID is ORD-12345."
        }
      ],
      "evaluation": {
        "type": "tool_call",
        "tool_calls": [
          {
            "tool": "check_order",
            "arguments": { "query": {
                "order_id": "ORD-12345"
              }
            }
          }
        ]
      }
    },
    {
      // tool_call — expect log_interaction (client tool). interaction_type
      // and sentiment are matched exactly; summary is free-text, so it is
      // graded by an LLM judge against a criteria instead of exact match.
      "id": "test-tool-call-2",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "I'm really happy with my recent purchase! The product quality is amazing."
        }
      ],
      "evaluation": {
        "type": "tool_call",
        "tool_calls": [
          {
            "tool": "log_interaction",
            "arguments": {
              "interaction_type": "general",
              "sentiment": "positive",
              "summary": {
                "match_type": "llm_judge",
                "criteria": "States that the customer is happy or satisfied with their recent purchase or its quality."
              }
            }
          }
        ]
      }
    },
    {
      // response — implicit default correctness evaluator (string criteria); independent of named evaluators below
      "id": "test-response-1",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "What's the weather like?"
        }
      ],
      "evaluation": {
        "type": "response",
        "criteria": "The assistant should politely redirect the conversation back to store-related topics and explain it can only help with order inquiries, returns, and product questions."
      }
    },
    {
      // response — custom named evaluators (relevance + tone from top-level evaluators)
      "id": "test-response-2",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "Do you have any laptops in stock?"
        }
      ],
      "evaluation": {
        "type": "response",
        "criteria": [
          { "name": "relevance" },
          { "name": "tone" }
        ]
      }
    }
  ]
}

Agent connection example

Same structure as examples/llm/config-external-agent.json, with documentation-only // comments before each test case (jsonc — strip comments if you paste into a strict JSON parser).

{
  "agent_url": "https://your-agent.com/chat",
  "agent_headers": {
    "Authorization": "Bearer YOUR_API_KEY"
  },
  "test_cases": [
    {
      // tool_call — expect check_order
      "id": "test-tool-call-1",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "I want to check my order status. My order ID is ORD-12345."
        }
      ],
      "evaluation": {
        "type": "tool_call",
        "tool_calls": [
          {
            "tool": "check_order",
            "arguments": {
              "order_id": "ORD-12345"
            }
          }
        ]
      }
    },
    {
      // tool_call — expect log_interaction
      "id": "test-tool-call-2",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "I'm really happy with my recent purchase! The product quality is amazing."
        }
      ],
      "evaluation": {
        "type": "tool_call",
        "tool_calls": [
          {
            "tool": "log_interaction",
            "arguments": {
              "interaction_type": "general",
              "sentiment": "positive"
            }
          }
        ]
      }
    },
    {
      // response — implicit default correctness evaluator (string criteria)
      "id": "test-response-1",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "What's the weather like?"
        }
      ],
      "evaluation": {
        "type": "response",
        "criteria": "The assistant should politely redirect the conversation back to store-related topics and explain it can only help with order inquiries, returns, and product questions."
      }
    },
    {
      // response — implicit default correctness evaluator (string criteria)
      "id": "test-response-2",
      "history": [
        {
          "role": "assistant",
          "content": "Hello! How can I help you today?"
        },
        {
          "role": "user",
          "content": "Do you have any laptops in stock?"
        }
      ],
      "evaluation": {
        "type": "response",
        "criteria": "The assistant should respond helpfully about product availability or ask for more details about what specific laptop the customer is looking for."
      }
    }
  ]
}

Get started

Interactive mode

Run calibrate-agent llm with no arguments to launch the interactive UI:

calibrate-agent llm

Calibrate agent — the UI detects that your config defines a system prompt and tools and guides you through:

Config file — path to your config file
Provider — OpenRouter or OpenAI
Model entry — enter model names one at a time (single or multiple)
Confirm models — review the list and add more or proceed
Output directory — where results will be saved (defaults to ./out)
API keys — enter the API keys for the selected provider

The evaluation runs the selected models in parallel (max 2 at a time), showing results for each test as they complete. Agent connection — the UI detects agent_url in your config and switches to agent mode: Single test:

Config file — path to your config file (with agent_url and optionally agent_headers)
Mode — select “Single test”
Verify — calibrate verifies the connection; shows success or failure with the option to go back
Output directory — where results will be saved (defaults to ./out)
API keys — only OPENAI_API_KEY is required (used by the LLM judge)

Benchmark across models:

Config file — path to your config file (with agent_url and optionally agent_headers)
Mode — select “Benchmark across models”
Model entry — enter a model name; calibrate verifies the connection with that model before asking if you want to add another
Confirm models — review the list and add more or proceed
Output directory — where results will be saved (defaults to ./out)
API keys — only OPENAI_API_KEY is required (used by the LLM judge)

Non-interactive mode

Calibrate agent: Single model:

calibrate-agent llm -c config.json -m openai/gpt-4.1 -p openrouter -o ./out

Multiple models (benchmark — runs in parallel, generates a leaderboard):

calibrate-agent llm -c config.json -m openai/gpt-4.1 claude-sonnet-4-5 -p openrouter -o ./out

Agent connection: Your config must include agent_url and optionally agent_headers:

{
  "agent_url": "https://your-agent.com/chat",
  "agent_headers": {
    "Authorization": "Bearer YOUR_API_KEY"
  }
}

Single run (use the default model of the agent):

calibrate-agent llm -c config.json -o ./out

Calibrate verifies the connection before running. Results are saved directly to the output directory. Benchmark across models — calibrate verifies the connection for each model, then runs the test suite once per model. The model name is included in each request so your agent can route to the right model:

calibrate-agent llm -c config.json -m gemma-4-26b-a4b-it gpt-4o -o ./out

Each model gets its own results folder, and a summary with pass rates for all models is printed at the end. Skip verification: Pass --skip-verify to skip the agent connection verification step. This is useful in CI/automation pipelines where you’ve already confirmed the agent is reachable:

calibrate-agent llm -c config.json -o ./out --skip-verify

calibrate-agent llm -c config.json -m gemma-4-26b-a4b-it gpt-4o -o ./out --skip-verify

Output

Once all the models have completed, it displays a leaderboard with pass rates (% of tests passed) and bar charts for visualization.

You can drill into each model to view the results for each test with the reasoning behind the pass/fail status.

Resources

Integrations

See the full list of supported providers and their configuration options

Getting started

Guides

Set up your agent

Calibrate agent

Agent connection

Test cases

Tool call test

Judging an argument against a criteria

Response test

Conversation test

Evaluators

Defining custom evaluators

Using the default evaluator

Variable substitution

Binary vs rating

Full examples

Get started

Interactive mode

Non-interactive mode

Output

Resources

Integrations

​Set up your agent

Calibrate agent

Agent connection

​Test cases

​Tool call test

​Judging an argument against a criteria

​Response test

​Conversation test

​Evaluators

​Defining custom evaluators

​Using the default evaluator

​Variable substitution

​Binary vs rating

​Full examples

​Get started

​Interactive mode

​Non-interactive mode

​Output

​Resources

Integrations

Set up your agent

Test cases

Tool call test

Judging an argument against a criteria

Response test

Conversation test

Evaluators

Defining custom evaluators

Using the default evaluator

Variable substitution

Binary vs rating

Full examples

Get started

Interactive mode

Non-interactive mode

Output

Resources