A growing consensus: iterative experimentation is gaining ground

Across global education, a familiar challenge is becoming more acute: how to deliver greater impact in education systems with constrained resources. As funding landscapes shift and systems come under pressure to do more with less, questions of cost-effectiveness are increasingly central to decision-making. At the same time, many organisations are grappling with a practical gap. While large-scale evaluations remain essential for understanding whether programmes work, they are often too slow, infrequent, or rigid to guide the day-to-day decisions that shape implementation at scale.

A recent webinar brought together perspectives from funders, researchers, implementers, and government to explore how iterative A/B testing, a structured approach to testing small changes to make existing programmes more cost-effective and scalable, can help address this gap. The webinar included perspectives from the What Works Hub for Global Education, Youth Impact, the Gates Foundation, Innovations for Poverty Action, the Agency Fund and South Africa’s Department of Basic Education. The webinar covered key lessons from a newly released Iterative A/B Testing Toolkit. A broad convergence across sectors emerged around the need for more continuous, embedded approaches to learning.

From evaluation to optimisation

One of the clearest themes across speakers was a shift in emphasis: from proving impact to improving it. As webinar chair Jonathan Stern of the Gates Foundation put it:

‘Ultimately, A/B testing doesn’t replace the need for long-term evaluations, but it’s a much-needed complement in order to rapidly understand what works, what doesn’t, and how variations in implementation can lead to important shifts in the cost-effectiveness of programming.’

When education programmes are already operating at scale, the key question is often focused on refining delivery in ways that increase impact or reduce cost without compromising quality. Iterative testing offers a way to generate actionable evidence on shorter timelines, enabling organisations to make incremental improvements based on data rather than intuition alone.

An approach travelling across sectors

A particularly striking point in the discussion was the range of institutional contexts in which this approach is taking hold. Originating in the technology world, Noam Angrist (What Works Hub for Global Education/Youth Impact) described a growing momentum:

‘It’s clear A/B testing is really building as a movement… It’s happening in multiple organisations and sectors around the world: research orgs, implementing orgs, governments increasingly, funders.’

This cross-sector uptake reflects a shared set of pressures to improve programme efficiency at scale, and also a growing recognition that experimentation need not be confined to academic settings. Instead, it can be embedded within routine programme delivery.

From a funder perspective, the appeal is closely tied to value for money. As Stern put it:

‘As a person responsible for ensuring that we do everything we can to maximise the impact of each dollar we spend, I think this toolkit marks a really important shift for the sector.’

For implementers, the relevance lies in improving operational decisions. Amanda Beatty (Youth Impact/Pratham) emphasised that the most valuable questions often emerge from those closest to delivery:

‘One thing we find really important to highlight is that the questions come from the implementation team… the better questions might come from your programme team.’

This grounding in real-world constraints helps ensure that testing focuses on changes that are both feasible and relevant on the ground.

What it looks like in practice

Across examples discussed in the webinar, iterative A/B testing was used to refine the details that shape both cost and effectiveness.

In digital-first programmes, for instance, Andrés Parrado (Innovations for Poverty Action – Right-Fit Evidence Unit) highlighted how low marginal costs create opportunities for rapid iteration:

‘In digital-first programming, it’s a lot easier to reach more users with very limited marginal cost. So that means there are tweaks you might want to make in the beginning, to improve cost-effectiveness.’

In government contexts, the approach can help de-risk larger investments. Nompumelelo Mohohlwane (South Africa’s Department of Basic Education) noted the potential of using A/B testing early in the evidence cycle in public education systems, where decisions affect thousands of schools:

‘Often the lesson can be very expensive to find out at the end of your RCT that [a solution] didn’t work… So an A/B test upfront would be a quick and easy way to figure out which version would be effective.’

At the same time, practical challenges remain, particularly when it comes to moving from intuitive adaptation to more structured approaches to learning. As Mohohlwane noted:

‘Implementers are really good at implementing their programme, but not so good at articulating small changes and tracking and measuring them in a way that goes beyond intuition.’

Addressing this gap is central to making iterative testing accessible at scale. Emerging resources — including the Iterative A/B Testing Toolkit and related tools — bridge the divide between implementation and evidence, while helping organisations build the muscle to become learning organisations: working in closer collaboration across teams, developing a regular rhythm of testing, and becoming more comfortable with rapid, low-stakes iteration.

Building the conditions for experimentation

A recurring theme throughout the discussion was that successful A/B testing is not just a technical exercise, but an organisational one.

Temina Madon (the Agency Fund) emphasised the importance of culture:

‘Running A/B tests becomes part of an organisation’s culture, and it needs to for the practice of A/B testing to succeed.’

This includes creating shared ownership of experiments across teams, rather than confining them to monitoring and evaluation functions. Simple practices such as inviting staff to predict the outcomes of tests can help build engagement and shift mindsets around learning and uncertainty.

It also requires clear frameworks for prioritising what to test. Madon suggested using a programme’s theory of change as a guide:

‘Think about where your assumptions are strongest and have the weakest support; you probably want to make tweaks around that space.’

Questions from participants reflected strong interest in how to operationalise this approach in practice. Common themes included how to select meaningful short-term indicators, how to ensure sufficient statistical power in clustered settings such as classrooms, and how to build the internal systems needed to run tests consistently.

A growing movement with more to learn

While enthusiasm for iterative testing is growing, the discussion also highlighted the practical questions organisations face when putting it into practice. There was a shared sense that this is only the beginning. As Angrist reflected:

‘When have you run enough A/B tests? Never. There’s always more to learn, and you can always improve.’

What emerged most clearly from the discussion was not a single method, but a shared direction of travel. As education systems face increasing pressure to improve outcomes with limited resources, the ability to test, learn, and adapt in real time is becoming less of an innovation and more of a necessity.

A growing set of resources is beginning to support this shift — including materials shared during the webinar, and concrete examples of A/B test case studies. For organisations looking to take the next step, these provide practical entry points into building more structured, routine approaches to learning. Upcoming opportunities to engage further include hands-on workshops and a planned virtual session, which aim to support implementers in applying these ideas in practice.

Further resources:

• Webinar event page
• Webinar recording
• Amanda Beatty’s slides
• Andrés Parrado’s slides
• Iterative A/B Testing Toolkit
• IPA A/B testing guidance
• IPA Insight note: A practical approach to developing A/B testing systems for digital-first organisations
• Working paper: Cheaper (and more effective) by the dozen: Evidence from 12 randomised A/B tests optimising tutoring for scale
• Evidential: A free, open-source experiment engine
• Iterative A/B Testing for Social Impact: Rigorous, Rapid, Regular