Working paper
Building resilient education systems: Experimental evidence across five countries
Noam Angrist, Micheal Ainomugisha, Sai Pramod Bathena, Peter Bergman, Colin Crossley, Claire Cullen, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal and Tim Sullivan
Abstract
Education systems need to withstand frequent shocks, including disease and climate events,which close schools for over 222 million children. During these emergencies, alternative models are needed to deliver education. However, rigorous evaluation of effective educational approaches in these settings is challenging and rare, especially across multiple countries. We present results from five randomised trials in India, Kenya, Nepal, Philippines, and Uganda to evaluate the provision of education in emergency settings. These multi-site coordinated trials enable us to uniquely inform internal and external validity simultaneously. We also test multiple scalable models of remote targeted tutoring instruction, comparing government and NGO delivery. While SMS messages have mixed results, phone call tutorials show consistent effectiveness across diverse settings. We find large and robust effect sizes on learning, with average effects of 0.30-0.35 standard deviations. These effects are highly cost-effective, delivering up to four years of high-quality instruction per $100 spent. Results also show similar effects when delivered by government teachers as NGO instructors and sustained improvements to teacher beliefs and practices. We further evaluate effects during a government scale-up with 25,000 teachers in India and an additional weather shock in the Philippines. Altogether, our results reveal it is possible to strengthen the resilience of education systems, enabling education provision amidst disruptions, and to deliver cost-effective learning gains across contexts and with governments.
References
Aker, Jenny C., and Isaac M. Mbiti. “Mobile phones and economic development in Africa.” Journal of economic Perspectives 24, no. 3 (2010): 207-32.
Aker, Jenny C., Christopher Ksoll, and Travis J. Lybbert. “Can mobile phones improve learning? Evidence from a field experiment in Niger.” American Economic Journal: Applied Economics 4, no. 4 (2012): 94-120.
Alan, Sule, Teodora Boneva, and Seda Ertac. “Ever failed, try again, succeed better: Results from a randomized educational intervention on grit.” The Quarterly Journal of Economics 134, no. 3 (2019): 1121-1162.
Anderson, Michael L. “Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects.” Journal of the American statistical Association 103, no. 484 (2008): 1481-1495.
Andrabi, Tahir, Benjamin Daniels, and Jishnu Das. “Human capital accumulation and disasters: Evidence from the Pakistan earthquake of 2005.” Journal of Human Resources (2021): 0520-10887R1.
Angrist, Noam, Peter Bergman, and Moitshepi Matsheng. “Experimental evidence on learning using low-tech when school is out.” Nature human behaviour (2022): 1-10.
Angrist, Noam, Simeon Djankov, Pinelopi K. Goldberg, and Harry A. Patrinos. “Measuring human capital using global learning data.” Nature 592, no. 7854 (2021): 403-408.
Angrist, Noam, Michael Ainomugisha, Saipramod Bathena, Peter Bergman, Colin Crossley, Claire Cullen, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal, and Tim Sullivan. 2023. “Learning Curve: Progress in the Replication Crisis.” AEA Papers and Proceedings.
Angrist, Noam, and Rachael Meager. “Implementation Matters: Generalizing Treatment Effects in Education.” Available at SSRN 4487496 (2023).
Angrist, Noam, David K. Evans, Deon Filmer, Rachel Glennerster, Halsey Rogers, and Shwetlena Sabarwal. “How to improve education outcomes most efficiently? A review of the evidence using a unified metric.” Journal of Development Economics (2024): 103382.
Araujo, M. Caridad, Marta Rubio-Codina, and Norbert Schady. “70 to 700 to 70,000: Lessons from the Jamaica Experiment.” In The Scale-Up Effect in Early Childhood and Public Policy, pp. 211-232. Routledge, 2021.
Azevedo, João Pedro, Amer Hasan, Diana Goldemberg, Koen Geven, and Syedah Aroob Iqbal. “Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates.” The World Bank Research Observer 36, no. 1 (2021): 1-40.
Bacher-Hicks, Andrew, Joshua Goodman, and Christine Mulhern. “Inequality in household adaptation to schooling shocks: Covid-induced online learning engagement in real time.” Journal of Public Economics 193 (2021): 104345.
Bando, Rosangela, Emma Näslund-Hadley, and Paul Gertler. Effect of inquiry and problem based pedagogy on learning: Evidence from 10 field experiments in four countries. No. w26280. National Bureau of Economic Research, 2019.
Banerjee, Abhijit V., Shawn Cole, Esther Duflo, and Leigh Linden. “Remedying education: Evidence from two randomized experiments in India.” The Quarterly Journal of Economics 122, no. 3 (2007): 1235-1264.
Banerjee, Abhijit, Esther Duflo, Nathanael Goldberg, Dean Karlan, Robert Osei, William Parienté, Jeremy Shapiro, Bram Thuysbaert, and Christopher Udry. “A multifaceted program causes lasting progress for the very poor: Evidence from six countries.” Science 348, no. 6236 (2015): 1260799.
Banerjee, Abhijit, Rukmini Banerji, James Berry, Esther Duflo, Harini Kannan, Shobhini Mukerji, Marc Shotland, and Michael Walton. “From proof of concept to scalable policies: Challenges and solutions, with an application.” Journal of Economic Perspectives 31, no. 4 (2017): 73-102.
Bergman, Peter. “Parent-child information frictions and human capital investment: Evidence from a field experiment.” Journal of Political Economy 129, no. 1 (2021): 286-322.
Bold, Tessa, Mwangi Kimenyi, Germano Mwabu, and Justin Sandefur. “Experimental evidence on scaling up education reforms in Kenya.” Journal of Public Economics 168 (2018): 1-20.
Burde, Dana, and Leigh L. Linden. “Bringing education to Afghan girls: A randomized controlled trial of village-based schools.” American Economic Journal: Applied Economics 5, no. 3 (2013): 27-40.
Burde, Dana, Ozen Guven, Jo Kelcey, Heddy Lahmann, and Khaled Al-Abbadi. “What works to promote children’s educational access, quality of learning, and wellbeing in crisis-affected contexts.” Education Rigorous Literature Review, Department for International Development. London: Department for International Development (2015).
Callen, Michael, Miguel Fajardo-Steinhäuser, Michael G. Findley, and Tarek Ghani.“Can digital aid deliver during humanitarian crises?.” Management Science (2025).
Carlana, Michela, and Eliana La Ferrara. “Apart but connected: Online tutoring and student outcomes during the COVID-19 pandemic.” (2021).
Carlana, Michela, Eliana La Ferrara, and Carolina Lopez. “Exacerbated Inequalities: the Learning Loss from COVID-19 in Italy.” AEA Papers and Proceedings (2023).
Carvalho, S. & Crawfurd, L. School’s Out: Now What? (Center for Global Development, 2020); https://www.cgdev.org/blog/schools-out-now-what
Cooper, Harris, Barbara Nye, Kelly Charlton, James Lindsay, and Scott Greathouse. “The effects of summer vacation on achievement test scores: A narrative and meta-analytic review.” Review of educational research 66, no. 3 (1996): 227-268.
Crawfurd, Lee, David K. Evans, Susannah Hares, and Justin Sandefur. Teaching and testing by phone in a pandemic. No. 591. Center for Global Development, 2021.
Christensen, Darin, Oeindrila Dube, Johannes Haushofer, Bilal Siddiqi, and Maarten Voors. “Building resilient health systems: Experimental evidence from sierra leone and the 2014 ebola outbreak.” The Quarterly Journal of Economics 136, no. 2 (2021): 1145-1198.
Cunha, Flavio, and James Heckman. “The technology of skill formation.” American economic review 97, no. 2 (2007): 31-47.
Dehejia, Rajeev, Cristian Pop-Eleches, and Cyrus Samii. “From local to global: External validity in a fertility natural experiment.” Journal of Business & Economic Statistics 39, no. 1 (2021): 217-243.
Duflo, Annie, Jessica Kiessel, and Adrienne Lucas. External Validity: Four Models of Improving Student Achievement. No. w27298. Cambridge: National Bureau of Economic Research, 2020.
Eble, Alex, Chris Frost, Alpha Camara, Baboucarr Bouy, Momodou Bah, Maitri Sivaraman, Pei-Tseng Jenny Hsieh et al. “How much can we remedy very low learning levels in rural parts of low-income countries? Impact and generalizability of a multi-pronged para-teacher intervention from a cluster-randomized trial in the Gambia.” Journal of Development Economics 148 (2021): 102539.
Evans, David K., Pamela Jakiela, and Heather A. Knauer. “The impact of early childhood interventions on mothers.” Science 372, no. 6544 (2021): 794-796.
Evans, David K., and Fei Yuan. “How big are effect sizes in international education studies?.” Educational Evaluation and Policy Analysis 44, no. 3 (2022): 532-540.
Ganimian, Alejandro J., and Sharnic Djaker. “How can developing countries address heterogeneity in students’ preparation for school? A review of the challenge and potential solutions.” Steinhardt School of Culture, Education, and Human Development, New York University. New York, NY (2022).
Ganimian, Alejandro J., Emiliana Vegas, and Frederick M. Hess. “Realizing the promise:, How can education technology improve learning for all?.” (2020).
Glewwe, Paul, and Karthik Muralidharan. “Improving education outcomes in developing countries: Evidence, knowledge gaps, and policy implications.” In Handbook of the Economics of Education, vol. 5, pp. 653-743. Elsevier, 2016.
Gneezy, Uri, John A. List, Jeffrey A. Livingston, Xiangdong Qin, Sally Sadoff, and Yang Xu. “Measuring success in education: The role of effort on the test itself.” American Economic Review: Insights 1, no. 3 (2019): 291-308.
Gupta, Saloni, Kumar Satyam, Niharika Gupta and Rahul Ahluwalia (2022). “Can Phone Assessments Generate Reliable Student Learning Data?” Central Square Foundation.
Hanna, Rema, Sendhil Mullainathan, and Joshua Schwartzstein. “Learning through noticing: Theory and evidence from a field experiment.” The Quarterly Journal of Economics 129, no. 3 (2014): 1311-1353.
Hevia, Felipe J., Miguel Székely, Tamara Vinacur, and Pablo Zoido. “Tutorías remotas: revisión de la literatura.” (2022). Inter-American Development Bank.
Hotz, V. Joseph, Guido W. Imbens, and Julie H. Mortimer. “Predicting the efficacy of future training programs using past experiences at other locations.” Journal of econometrics 125, no. 1-2 (2005): 241-270.
Jackson, C. Kirabo. “What do test scores miss? The importance of teacher effects on non–test score outcomes.” Journal of Political Economy 126, no. 5 (2018): 2072-2107.
Jack, Rebecca, Clare Halloran, James Okun, and Emily Oster. “Pandemic Schooling Mode and Student Test Scores: Evidence from US School Districts.” American Economic Review: Insights (2021)
Jaume, David, and Alexander Willén. “The long-run effects of teacher strikes: evidence from Argentina.” Journal of Labor Economics 37, no. 4 (2019): 1097-1139.
Kremer, Michael, Conner Brannen, and Rachel Glennerster. “The challenge of education and learning in the developing world.” Science 340, no. 6130 (2013): 297-300.
List, John A.“Optimally generate policy-based evidence before scaling.” Nature 626, no. 7999 (2024): 491-499.
List, John A. The voltage effect: How to make good ideas great and great ideas scale. 2022.
Lichand, Guilherme, Carlos Alberto Doria, Onicio Leal-Neto, and João Paulo Cossi Fernandes. “The impacts of remote learning in secondary education during the pandemic in Brazil.” Nature Human Behaviour 6, no. 8 (2022): 1079-1086.
Lucas, Adrienne M., Patrick J. McEwan, Moses Ngware, and Moses Oketch. “Improving early-grade literacy in East Africa: Experimental evidence from Kenya and Uganda.” Journal of Policy Analysis and Management 33, no. 4 (2014): 950-976.
Maniadis, Zacharias, Fabio Tufano, and John A. List. ”To replicate or not to replicate? Exploring reproducibility in economics through the lens of a model and a pilot study.” The Economic Journal (2017).
Mobarak, Ahmed Mushfiq. 2022. “Assessing social aid: the scale-up process needs evidence, too.” Nature: 892-894.
Moscoviz, Laura, and David K. Evans. “Learning loss and student dropouts during the covid-19 pandemic: A review of the evidence two years after schools shut down.” Center for Global Development, Working Paper 609 (2022).
Muralidharan, Karthik, Abhijeet Singh, and Alejandro J. Ganimian. “Disrupting education? Experimental evidence on technology-aided instruction in India.” American Economic Review 109, no. 4 (2019): 1426-60.
Nickow, Andre, Philip Oreopoulos, and Vincent Quan. 2020. “The impressive effects of tutoring on prek-12 learning: A systematic review and meta-analysis of the experimental evidence.”
Patrinos, Harry Anthony, Emiliana Vegas, and Rohan Carter-Rau. “An Analysis of COVID-19 Student Learning Loss.” (2022).
Pritchett, Lant. The rebirth of education: Schooling ain’t learning. CGD Books, 2013.
Pritchett, Lant, and Justin Sandefur. “Learning from experiments when context matters.” American Economic Review 105, no. 5 (2015): 471-475.
Radhakrishnan, Karthika; Angrist, Noam; Bergman, Peter; Cullen, Claire; Matsheng, Moitshepi; Ramakrishnan, Anusha; Sabarwal, Shwetlena; Sharma, Uttam. Learning in the Time of COVID-19 : Insights from Nepal (2021). World Bank, Washington, DC.
Robinson, Carly D., and Susanna Loeb. “High-impact tutoring: State of the research and priorities for future learning.” National Student Support Accelerator 21, no. 284 (2021): 1-53.
Rodriguez-Segura, Daniel, and Beth E. Schueler. “Can learning be measured by phone? Evidence from Kenya.” Economics of Education Review 90 (2022): 102309.
Sabarwal, Shwetlena, Sergio Venegas Marin, Marla Hillary Spivack, and Diego Ambasz. “Choosing Our Future-Education for Climate Action.” (2024).
Schaffner, Julie, Paul Glewwe, and Uttam Sharma. Why Programs Fail: Lessons for Improving Public Service Quality from a Mixed-Methods Evaluation of an Unsuccessful Teacher Training Program in Nepal. No. 1701-2021-3437. 2021.
Schueler, Beth E., and Daniel Rodriguez-Segura. “A Cautionary Tale of Tutoring Hard-to-Reach Students in Kenya. EdWorkingPaper No. 21-432.” Annenberg Institute for School Reform at Brown University (2021).
Slade, Timothy S., Benjamin Piper, Zikani Kaunda, Simon King, and Hibatalla Ibrahim. “Is ‘summer’ reading loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from grade-transition breaks.” Research in Comparative and International Education 12, no. 4 (2017): 461-485.
UNESCO. Sustainable Development Goal (SDG) 4 Kenya Country Profile (2016).
UNESCO. COVID-19 Education Response (2023). https://covid19.uis.unesco.org/global-monitoring-school-closures-covid19/country-dashboard/
Uwezo. Are Our Children Learning? Kenya Sixth Learning Assessment Report (2016). Nairobi.
Uwezo. Are our Children Learning? Illuminating the Covid-19 Learning Losses and Gains in Uganda. Uwezo National Learning Assessment Report, 2021 (2021). Kampala.
World Bank. 2018. World Development Report 2018: Learning to realize education’s promise.
World Bank. 2019. Uganda Economic Update, 13th Edition, May 2019; Uganda Economic Update, 13th Edition, May 2019 : Economic Development and Human Capital in Uganda – A Case for Investing More in Education. World Bank, Washington, DC.
World Bank. 2021. Mobile cellular subscriptions (per 100 people).
https://data.worldbank.org/indicator/IT.CEL.SETS.P2
World Bank. Supporting Student Learning at Home with Phone-based Formative Assessments: Landscape Review. World Bank, 2022.
Discover more
What we do
Our work will directly affect up to 3 million children, and reach up to 17 million more through its influence.
Who we are
A group of strategic partners, consortium partners, researchers, policymakers, practitioners and professionals working together.
Get involved
Share our goal of literacy, numeracy and other key skills for all children? Follow us, work with us or join us at an event.