Implementation matters: Generalising treatment effects in education

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation by aggregating evidence from prior randomised trials across five contexts, and use the results to inform a new randomised trial.

We find two factors explain most of the heterogeneity in effects across contexts: the degree of implementation (intention-to-treat or treatment-on-the-treated) and program delivery model (teachers or volunteers). Accounting for these implementation factors yields high generalisability, with similar effect sizes across studies. Thus, reporting treatment-on-the-treated effects, a practice which remains limited, can enhance external validity.

We also introduce a new Bayesian framework to formally incorporate implementation metrics into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.42 SD when taken up and 0.85 SD when implemented with high fidelity. To investigate how implementation can be improved in future settings, we run a new randomised trial of a targeted instruction program in Botswana. Results demonstrate that implementation can be improved in the context of a scaling program with large causal effects on learning. While research on implementation has been limited to date, our findings and framework reveal its importance for impact evaluation and generalisability.

References

Andrews, Isaiah, and Maximilian Kasy. 2019. “Identification of and correction for publication bias.” American Economic Review 109, no. 8: 2766–94.

Andrews, Isaiah, and Emily Oster. 2019. “A simple approximation for evaluating external validity bias.” Economics Letters 178: 58–62.

Angrist, Noam, Simeon Djankov, Pinelopi K. Goldberg, and Harry A. Patrinos. 2021. “Measuring human capital using global learning data.” Nature 592, no. 7854: 403–408.

Angrist, Noam, Peter Bergman, and Moitshepi Matsheng. 2022. “Experimental evidence on learning using low-tech when school is out.” Nature Human Behaviour 6, no. 7: 941–950.

Angrist, Noam, David K. Evans, Deon Filmer, Rachel Glennerster, F. Halsey Rogers, and Shwetlena Sabarwal. 2020. “How to improve education outcomes most efficiently? A comparison of 150 interventions using the new Learning-Adjusted Years of Schooling metric.” The World Bank.

Angrist, Joshua, and Peter Hull. 2023. “IV Methods Reconcile Intention-to-Screen Effects Across Pragmatic Cancer Screening Trials.” No. w31443. National Bureau of Economic Research,

Bandiera, Oriana, Greg Fischer, Andrea Prat, and Erina Ytsma. 2021. “Do women respond less to performance pay? Building evidence from multiple experiments.” American Economic Review: Insights 3, no. 4: 435–454.

Bando, Rosangela, Emma Näslund-Hadley, and Paul Gertler. 2019. “Effect of inquiry and problem-based pedagogy on learning: Evidence from 10 field experiments in four countries.” No. w26280. National Bureau of Economic Research.

Banerjee, Abhijit, Banerji, Rukmini, Berry, James, Duflo, Esther, Kannan, Harini, Mukherji, Shobhini, Shotland, Marc and Walton, Michael. 2017. “From proof of concept to scalable policies: Challenges and solutions, with an application.” Journal of Economic Perspectives 31, no. (4), pp. 73–102.

Banerjee, A. V., Hanna, R., Kreindler, G.E. and Olken, B.A., 2017b. “Debunking the stereotype of the lazy welfare recipient: Evidence from cash transfer programs.” World Bank Research Observer 32, no. 2, pp. 155–184.

Banerjee, A. V., Banerji, R., Duflo, E., Glennerster, R., and Khemani, S. 2010. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy (Vol. 2, Issue 1, pp. 1–30. American Economic Association.

Banerjee, Abhijit V., Shawn Cole, Esther Duflo, and Leigh Linden. 2007. “Remedying education: Evidence from two randomized experiments in India.” Quarterly Journal of Economics 122, no. 3: 1235–1264.

Banerjee, Abhijit, Esther Duflo, Nathanael Goldberg, Dean Karlan, Robert Osei, William Pariente, Jeremy Shapiro, Bram Thuysbaert, and Christopher Udry. 2015. “A multifaceted program causes lasting progress for the very poor: Evidence from six countries.” Science 348, no. 6236.

Banerji, Rukmini, and Chavan. 2016. “Improving literacy and math instruction at scale in India’s primary schools: The case of Pratham’s Read India program.” Journal of Educational Change 17(4), pp. 453–475.

Bauer, Mark S., Laura Damschroder, Hildi Hagedorn, Jeffrey Smith, and Amy M. Kilbourne. 2015. “An introduction to implementation science for the non-specialist.” BMC Psychology 3, no. 1: 1–12.

Beuermann, Diether W., Julian Cristia, Santiago Cueto, Ofer Malamud, and Yyannu Cruz-Aguayo. 2015. “One laptop per child at home: Short-term impacts from a randomized experiment in Peru.” American Economic Journal: Applied Economics 7, no. 2: 53–80.

Card, David, and Alan B. Krueger. 1995. “Time-series minimum-wage studies: a meta-analysis.” The American Economic Review 85, no. 2: 238–243.

Chung, Y., Gelman, A, Rabe-Hesketh, S., Liu, J., & Dorie, V. 2015. “Weakly informative prior for point estimation of covariance matrices in hierarchical models.” Journal of Educational and Behavioral Statistics 40(2), 136–157.

Chung, Y., Rabe-Hesketh, S., Dorie, V., Gelman, A., & Liu, J., 2013. “A non-degenerate penalized likelihood estimator for variance parameters in multilevel models.” Psychometrika 78, 685–709.

Cunha, Flavio, and James Heckman. 2007. “The technology of skill formation.” American Economic Review 97, no. 2: 31–47.

Deaton, Angus, and Nancy Cartwright. 2018. “Understanding and misunderstanding randomized controlled trials.” Social Science & Medicine 210: 2–21.

Duflo, Esther, Pascaline Dupas, and Michael Kremer. 2011. “Peer effects, teacher incentives, and the impact of tracking: Evidence from a randomized evaluation in Kenya.” American Economic Review 101, no. 5: 1739–1774.

Duflo, Esther. 2017. “The economist as plumber.” American Economic Review 107, no. 5: 1–26.

Duflo, Annie, Jessica Kiessel, and Adrienne Lucas. 2020. “Experimental Evidence on Alternative Policies to Increase Learning at Scale.” No. w27298. National Bureau of Economic Research,

Evans, David K., and Anna Popova. 2016. “What really works to improve learning in developing countries?” World Bank Research Observer 31, no. 2: 242–270.

Evans, David K., and Fei Yuan. 2020. “How big are effect sizes in international education studies?” Educational Evaluation and Policy Analysis: 01623737221079646.

Ganimian, Alejandro J., and Richard J. Murnane. 2016. “Improving education in developing countries: Lessons from rigorous impact evaluations.” Review of Educational Research 86: 719–755.

Glewwe, Paul, Michael Kremer, and Sylvie Moulin. 2009. “Many children left behind? Textbooks and test scores in Kenya.” American Economic Journal: Applied Economics 1, no. 1: 112–135.

Global Education Evidence Advisory Panel. 2020. “Cost-Effective Approaches to Improve Global Learning: What. Does Recent Evidence Tell Us Are “Smart Buys” for Improving Learning in Low- and Middle-Income Countries? Recommendations from the Global Education Evidence Advisor Panel. The World Bank.

Gechter, Michael. 2023. “Generalizing the Results from Social Experiments: Theory and Evidence from India.” Working manuscript, Pennsylvania State University.

Gelman, A., John B. Carlin, Hal S. Stern, & Donald B. Rubin. 2004. “Bayesian Data Analysis: Second Edition”, Taylor & Francis.

Gelman, Andrew, and Jennifer Hill. 2007. “Data analysis using regression and multilevel hierarchical models.” Cambridge University Press.

Gelman, A., and Pardoe, I.,  2006. “Bayesian measures of explained variance and pooling in multilevel models (hierarchical) models.” Technometrics, 48(2), 241–251.

Hanushek, E. A. 1995. “Interpreting Recent Research on Schooling in Developing Countries.” World Bank Research Observer, Vol 10, No.2.

Higgins, J. P. T., Green S (editors). 2011. “Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. [updated March 2011]. ”The Cochrane Collaboration. Available from www.handbook.cochrane.org.

Imbens, Guido W., and Joshua D. Angrist. 1994. “Identification and estimation of local average treatment effects.” Econometrica: journal of the Econometric Society: 467-475.

Innovations for Poverty Action. 2018. “Evaluating the Teacher Community Assistant Initiative.” Accessed July 19, 2018. https://www.poverty-action.org/study/evaluating-teacher-communityassistant-initiative-ghana
J-PAL. 2013. “Improving learning by increasing motivation, targeting instruction, and addressing school governance.”J-PAL Policy Insights.

Kraft, Matthew A. 2020. “Interpreting effect sizes of education interventions.” Educational Researcher 49, no. 4: 241–253.

Kremer, Michael, Stephen P. Luby, Ricardo Maertens, Brandon Tan, and Witold Więcek. 2023. “Water Treatment And Child Mortality: A Meta-Analysis And Cost-effectiveness Analysis.” No. w30835.National Bureau of Economic Research.

Kremer, Michael, Conner Brannen, and Rachel Glennerster. 2013. “The challenge of education and learning in the developing world.” Science 340, no. 6130: 297–300.

Lewbel, Arthur. 2019. “The identification zoo: Meanings of identification in econometrics.” Journal of Economic Literature 57, no. 4: 835–903.

Lockheed, Marlaine E., and Adriaan Verspoor. 1991. “Improving Primary Education in Developing Countries.” Oxford University Press.

Lund, Crick, Kate Orkin, Marc Witte, Thandi Davies, John Walker, Johannes Haushofer, Sarah Murray, Judy Bass, Laura Murray, and Vikram Patel. 2022. “Treating Mental Health Conditions Improves Labor Market and Other Economic Outcomes in Low and Middle-Income Countries.” University of Oxford. Working Paper.

Meager, Rachael. 2019. “Understanding the average impact of microcredit expansions: A Bayesian hierarchical analysis of seven randomized experiments.” American Economic Journal: Applied Economics 11, no. 1: 57-91.

Meager, Rachael. 2022. “Aggregating distributional treatment effects: A Bayesian hierarchical

analysis of the microcredit literature.” American Economic Review 112, no. 6: 1818-47.

Muralidharan, Karthik, Abhijeet Singh, and Alejandro J. Ganimian. 2019. “Disrupting education? Experimental evidence on technology-aided instruction in India.” American Economic Review 109, no. 4: 1426–1460.

Piper, Benjamin, Stephanie Simmons Zuilkowski, and Abel Mugenda. 2014. “Improving reading outcomes in Kenya: First-year effects of the PRIMR Initiative.” International Journal of Educational Development 37: 11–21.

Pritchett, Lant. 2013. “The Rebirth of Education: Schooling Ain’t Learning.” CGD Books.

Pritchett, Lant, and Justin Sandefur. 2015. “Learning from experiments when context matters.” American Economic Review 105, no. 5: 471–475.

Rubin, D. B., 1981. “Estimation in parallel randomized experiments.” Journal of Educational Statistics 6 (4), pp. 377–401.

Smale-Jacobse, Annemieke E., Anna Meijer, Michelle Helms-Lorenz, and Ridwan Maulana. 2019. “Differentiated instruction in secondary education: A systematic review of research evidence.” Frontiers in Psychology 10: 2366.

Snilstveit, Birte, Jennifer Stevenson, Radhika Menon, Daniel Phillips, Emma Gallagher, Maisie Geleen, Hannah Jobse, Tanja Schmidt, and Emmanuel Jimenez. 2016. “The impact of education programmes on learning and school participation in low- and middle-income countries.”

Tomlinson, Carol Ann. 2014. “The Differentiated Classroom: Responding to the Needs of All Learners.”

UNESCO. 2017. “More Than One-Half of Children and Adolescents Are Not Learning Worldwide.” UIS Fact Sheet No. 46.

Vespoor, Adriaan. “Pathways to Change: Improving the Quality of Education in Developing Countries.” World Bank Discussion Papers 53.

Vigneri, Marcella, Edoardo Masset, Mike Clarke, Josephine Exley, Peter Tugwell, Vivian Welch, and Howard White. 2018. “Economics and Epidemiology: Two Sides of the Same Coin or Different Currencies for Evaluating Impact?”. CEDIL Inception Paper 10.

Vivalt, Eva. 2020. “How much can we generalize from impact evaluations?” Journal of the European Economic Association 18, no. 6: 3045–3089.

World Bank. 2018. “World Development Report 2018: Learning to Realize Education’s Promise.” Washington, DC.