Learning resources

A step-by-step walkthrough of all the core features on Calibrate using a form filling voice agent as a case study: evaluating LLMs using structured tests, finding the best model across latency, cost and quality, aligning LLM judges with human experts, continuously monitoring agent performance, evaluating speech-to-text and text-to-speech models, and running simulated conversations with agents to evaluate them end-to-end.

Using the Calibrate command line tool and skills, you can connect your preferred AI tool with Calibrate so that it can create the evals automatically for your use case, upload and run them on Calibrate, analyse what went wrong, and suggest fixes. It can also analyse where your LLM judges disagree with humans and iterate on the judge on its own until it is aligned, without requiring you to intervene at each step.

Why AI systems need to be evaluated, what evaluation means, how to get started, creating a golden dataset, minimum viable evaluation, and how to keep improving your AI system. We also answer questions asked by the community during the live workshop, and we have written up a summary of the insights.

A tutorial on how Calibrate helps evaluate AI agents using a form-filling voice agent as a case study. It explains the eval-driven approach to building agents, where evals are not considered an afterthought but closely inform what needs to be built and help ensure agent quality. We show how to convert each failure mode into a test case and comprehensively discuss how to evaluate open-ended responses using LLM judges. We also share best practices and the pitfalls to avoid.

Office hours is a space for nonprofits to get their questions on AI evals answered. We covered how to evaluate when a team has few engineers, how domain experts can do the work without engineering help, how to write test cases that cover many situations, and how to watch quality over time. Teams asked about models that keep changing, splitting the work between engineers and domain experts, the cost of building and testing, building a trusted set of correct answers, and setting a baseline for correctness in public health and community health worker programmes.

A detailed walkthrough of the 4-level framework for evaluating AI products in the social sector:

• Level 1, model evaluationDoes the AI system perform as intended?
• Level 2, product evaluationDoes the overall product engage and retain users?
• Level 3, user evaluationDoes the product change users' thoughts, feelings, knowledge and behaviour towards the development outcome?
• Level 4, impact evaluationDo users with access to the product improve development outcomes?

A longer version of the slides that come with the Calibrate demo at the top of this page, additionally covering: How LLMs work, why AI fails in social-sector contexts, how evaluation helps deploy AI responsibly.