Skip to main content
Calibrate is an open-source evaluation platform for AI agents — text or voice.
Testing AI agents manually is slow, inconsistent, and doesn’t scale. Every agent has to get several things right, turn after turn:
  • Reasoning about the user’s intent against the context and purpose of the conversation
  • Calling the right tools with the right arguments in the right order
  • Generating the right response
  • Handling multiple languages and code-switching
It can be hard to even define what “the right response” means. If you’re building a voice agent, there are further challenges on top:
  • Transcribing the user’s speech accurately and efficiently
  • Producing natural-sounding speech back to the user
  • Detecting when the user has finished speaking or has only paused
  • Handling interruptions while the agent is speaking
  • Keeping the turnaround time low
  • Handling varying accents and dialects
and a lot more. There is no simple way to evaluate each of these across every provider on your own dataset. Even when individual pieces work well, it is hard to tell whether the agent as a whole will behave in production. This leads to many teams deploying with a lot of uncertainty and risk, and their users having to deal with poor experiences. Calibrate solves this by letting you evaluate the whole agent, whichever pieces it uses:
  • Text agents (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations, define custom evaluators, and find the best LLM for your agent
  • Voice agents (STT & TTS): Benchmark speech-to-text and text-to-speech providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages, using metrics optimised for agentic use cases
  • Simulations: Simulate realistic conversations using user personas and scenarios to surface failure modes — including interruptions — before you deploy
  • Human alignment: Collect human labels and iteratively align your LLM judges to them, so you can automate monitoring reliably
Calibrate helps you continuously improve your agent, ensure a bug never repeats itself, and deploy with confidence.

​
Get started

LLM tests

Create test suites that verify model responses and tool calling behavior

Simulations

Simulate agent conversations with customizable personas and scenarios

Speech to Text

Compare transcription accuracy across multiple providers on your dataset

Text to Speech

Benchmark generated audio quality across multiple providers

​
Learn more

Core Concepts

Understand how Calibrate helps you evaluate effectively

Agents

Give your AI coding agent the skills to evaluate on Calibrate

CLI

Drive the full evaluation loop from your terminal

Python SDK

Script agents, tests, runs, and benchmarks from Python

MCP

Give your coding agent native Calibrate tools over MCP

Integrations

See all supported providers for different components