Introduction - Calibrate

Calibrate is an open-source framework for automated, repeatable and systematic evaluation of voice agents at scale. Testing voice agents manually is slow, inconsistent, and doesn’t scale. There is no simple way to compare different providers for each component. Even if individual components are working, it’s not clear if the agent as a whole will work as expected when you deploy it to production. Calibrate solves this in two ways:

Component level testing: Evaluate individual components (Speech to Text, LLM, Text to Speech) across multiple providers on your dataset and specific edge cases your agent is likely to face in production.
End-to-end testing: Simulate conversations with your agent using realistic scenarios and user personas to identify critical pathways where your agent fails before deploying it to production.

This way, Calibrate lets you continuously improve your agent, ensure a bug never repeats itself and deploy your agent with confidence.