Building a reliable AI coaching assistant for last-mile communities in rural Uganda

This post was first published on The Agency Fund’s Substack.

At a glance:

• Evaluating a new AI tool: Raising The Village (RTV) built an AI coaching assistant to help field staff deliver consistent, tailored coaching to rural households, but they needed to evaluate message quality before deployment.
• Combining automated and human review: Automated AI evaluations using LLM judges, supplemented by multiple rounds of expert review, helped identify errors and improve coaching messages.
• Lessons for the social sector: LLM judges struggle with evaluating subjective criteria, making expert human feedback essential. Organizations should choose human reviewers carefully, learn from their disagreements, and iteratively test the evaluation workflow on a small scale to manage costs.

Using AI to support human coaches

Raising The Village (RTV) delivers a 24-month, village-wide ultra-poverty graduation program that has reached more than 2.4 million people across Sub-Saharan Africa. It supports last-mile rural communities with the tools, skills, and systems to help them increase incomes, build assets, and sustain pathways out of ultra-poverty.

Coaching is core to the program. Group trainings introduce households to improved agricultural techniques, livelihood practices, and water, sanitation and hygiene (WASH) measures. Follow-up 1:1 coaching reinforces sustained adoption, supporting long-term economic and health outcomes. During each visit, a field agent (FA) works directly with households to understand barriers to adoption and provide tailored advice – focusing on questions like:

As RTV expands across more geographies and grows its team of FAs, maintaining consistent coaching quality becomes harder. To address this challenge, RTV built an AI coaching assistant to support FAs in real time – using information about each household to generate customized coaching messages based on RTV’s program guidelines, helping even new FAs deliver more consistent guidance. For example:

Before deploying at scale, the tool needed to be evaluated – a process that yielded valuable lessons for both RTV and the social sector at large.

The evaluation challenge

Building an AI coaching assistant for the remote communities where RTV works was challenging. FAs cover large territories quickly (since coaching for practices like composting or planting must be timed to the growing season) and the AI tool must work offline in areas with little or no internet connectivity. So, the team planned to generate 130,000 coaching messages in advance, covering a wide range of scenarios, at a cost of nearly $1,000 (depending on the model used). Because the messages were bundled into an app, any corrections after launch require a new version that FAs would need to download and install.

To ensure message quality before deployment, RTV and The Agency Fund partnered to develop automated AI evaluations supplemented by human expert review.

We decided to assess each message on four dimensions:

• Accuracy: Technically correct, practical in the household context, and consistent with program guidelines
• Encouragement: Motivates households to practice rather than simply instructing them
• Respectfulness: Engages households with deep consideration for their social-cultural norms and behaviors, maintaining objective and non-judgmental tone
• Clarity: Gives households clear, easy-to-follow instructions they can immediately act on

When FAs rated sample messages, their reviews were unsurprisingly inconsistent; the AI tool is meant to address that variation. To obtain expert feedback, RTV recruited five experienced, top-performing former FAs for a four-day evaluation workshop in Mbarara, Uganda.

Automated evaluations using LLM judges

Before the next human expert workshop, we used Calibrate – an open-source AI agent evaluation tool for non-profits – to check message accuracy, the only objective evaluation dimension. An LLM judge checked: Is the message fully grounded in RTV guidelines, or does any part of it go beyond them? (The prompt is available here.)

We generated a sample of 350 coaching messages. The LLM judged messages grounded in RTV’s knowledge base as correct, and it flagged others as incorrect with an explanation:

Only 48% of messages passed the accuracy test. Human review later found that some failed messages were in fact accurate, but nearly 150 had genuine mistakes that could have been deployed to real users without this evaluation step.

We then classified the errors by type. One common error was applying guidance for one farm animal to others. For example, the knowledge base warns against spreading fresh pig manure in kitchen gardens, but an AI-generated message applied the guideline to guinea-pig droppings. This reinforced the need to ground every part of each message in RTV’s knowledge base, reducing the risk of unsupported or inappropriate guidance.

We also compared different OpenAI models. After finding that accuracy increased significantly with newer models, we used GPT 5.6 sol to generate messages for the human expert labeling workshop:

Refining the AI assistant with human experts

Five experienced, high-performing former FAs reviewed, rated, and helped refine the 350 generated messages across all four dimensions in the evaluation rubric.

In an earlier workshop, the same group had rated each message on a 1-5 scale. They gave messages an average score of 4.75 out of 5, but often disagreed; the 1–5 scale required fine distinctions on subjective dimensions like clarity and respectfulness, where ratings may reflect individual preferences rather than objective correctness. For the new workshop, we simplified the review task and asked a single binary question: Do you believe this message is helpful for household coaching?

We held a couple of dry runs to train the group on the rating tool and calibrate their ratings to reduce disagreements. On Day 1, the reviewers found the messages generally sound and agreement was 96.3%, but they found the binary choice of “Yes” and “No” too restrictive. They wanted a way to indicate when they were unsure. So, we added a “Not sure” option (see below), and agreement fell to 64% on Day 2, revealing differences the binary option had obscured. This gave the team a chance to identify issues, revise messages, and recalibrate.

In the final round, all five reviewers rated the same 50 messages. The average quality score was 2.82 out of 3 and agreement rose to 75%. The LLM judge agreed with reviewers on accuracy 68% of the time, close to their agreement with one another.

One consistent finding was that a message can satisfy all four dimensions of the evaluation rubric, yet still fail to address a household’s specific barrier. For example, reviewers rejected a soil-management message that simply restated the practice’s benefits instead of addressing the household’s land-access barrier. They also rejected messages that were technically correct but not practical. For example, an organic-pesticide message recommended neem, which is scarce in RTV’s operating districts.

Most messages required only one revision. Reviewers’ suggestions also gave the team useful feedback for the next cycle of message generation.

Broader lessons

Our experience evaluating RTV’s AI assistant highlights several lessons for teams building and testing AI tools in the social sector:

• Expert feedback can be slow and costly, but it’s irreplaceable. Experienced reviewers identified nuances that were otherwise hard for the team to catch, providing a clearer picture of where the model worked and where it needed improvement.
• Choose evaluators wisely and learn from their disagreements. Feedback from the staff who will use the tool was inconsistent initially and unhelpful for assessing message quality. Having each expert reviewer assess a unique set of messages would have produced a larger dataset, but having multiple experts review the same messages revealed where they disagreed. The process of resolving those disagreements brought greater clarity.
• LLM judges struggle with subjective criteria. LLM judges can help scale AI evals when faced with limited human review capacity, and surface critical issues to inform decisions. But they perform poorly on subjective criteria. We used them only to evaluate accuracy, our most objective dimension – this is a “minimum viable evaluation” approach that resolves easy to identify issues before human review.
• Account for evaluation costs. To manage costs, we evaluated a small sample of messages instead of the entire set. Once confident in the system, we generated all 130,000 messages using GPT 5.6 sol for nearly $900.

The refined messages are now in the app for the upcoming planting season, when RTV will A/B test whether AI-assisted coaching improves household practices and knowledge. If your organization is tackling similar challenges, we’d love to hear from you – contact us at tech@agency.fund.

If you are struggling to evaluate your AI tool systematically and want your entire team, including domain experts, to be involved, check out Calibrate, the open-source AI evaluation tool mentioned in this post.

Aman Dalmia is Lead AI Engineer at The Agency Fund.

Cosmas Wamozo is a Senior Data Scientist at Raising The Village.

Oscar Katahweire is a Learning and Knowledge Management Coordinator at Raising The Village.