Home > Evidence & resources >Implementation matters: Generalising treatment effects in education

Working paper

22 November 2023

Implementation matters: Generalising treatment effects in education

Authors:

Noam Angrist, Rachael Meager

Targeted instruction (that is, teaching to the level of a child's understanding, rather than the prescribed level for their age) is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. What explains the difference?

Abstract

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation by aggregating evidence from prior randomised trials across five contexts, and use the results to inform a new randomised trial.

We find two factors explain most of the heterogeneity in effects across contexts: the degree of implementation (intention-to-treat or treatment-on-the-treated) and program delivery model (teachers or volunteers). Accounting for these implementation factors yields high generalisability, with similar effect sizes across studies. Thus, reporting treatment-on-the-treated effects, a practice which remains limited, can enhance external validity.

We also introduce a new Bayesian framework to formally incorporate implementation metrics into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.42 SD when taken up and 0.85 SD when implemented with high fidelity. To investigate how implementation can be improved in future settings, we run a new randomised trial of a targeted instruction program in Botswana. Results demonstrate that implementation can be improved in the context of a scaling program with large causal effects on learning. While research on implementation has been limited to date, our findings and framework reveal its importance for impact evaluation and generalisability.

https://doi.org/10.35489/BSG-WhatWorksHubforGlobalEducation-WP_2023/03

Peer review status:

Peer-reviewed by academic leadership team

Discover more

Young female student with notebook. Photo by Apex 360, Unsplash.

What we do

Our work will directly affect up to 3 million children, and reach up to 17 million more through its influence.

Teacher sits on the floor with group of students. Photo by Husniati Salma, Unsplash.

Who we are

A group of strategic partners, consortium partners, researchers, policymakers, practitioners and professionals working together.

Children reading. Photo by Andrwe Ebrahim, Unsplash.

Get involved

Share our goal of literacy, numeracy and other key skills for all children? Follow us, work with us or join us at an event.

Loading...