22 November 2023

Authors:

Noam Angrist, Rachael Meager

Targeted instruction (that is, teaching to the level of a child's understanding, rather than the prescribed level for their age) is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. What explains the difference?

Abstract

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation by aggregating evidence from prior randomised trials across five contexts, and use the results to inform a new randomised trial.

We find two factors explain most of the heterogeneity in effects across contexts: the degree of implementation (intention-to-treat or treatment-on-the-treated) and program delivery model (teachers or volunteers). Accounting for these implementation factors yields high generalisability, with similar effect sizes across studies. Thus, reporting treatment-on-the-treated effects, a practice which remains limited, can enhance external validity.

We also introduce a new Bayesian framework to formally incorporate implementation metrics into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.42 SD when taken up and 0.85 SD when implemented with high fidelity. To investigate how implementation can be improved in future settings, we run a new randomised trial of a targeted instruction program in Botswana. Results demonstrate that implementation can be improved in the context of a scaling program with large causal effects on learning. While research on implementation has been limited to date, our findings and framework reveal its importance for impact evaluation and generalisability.

https://doi.org/10.35489/BSG-WhatWorksHubforGlobalEducation-WP_2023/03