A simple text message intervention can help improve the effectiveness of government programmes

James Habyarimana and Jacobus Cilliers

Well-designed programmes with prior evidence of success often fail when scaled up by government. This is perhaps not surprising given the complicated incentive structures of large bureaucracies, limited resources available to the public sector in developing countries, and the complexity of implementing new programmes. Given these challenges, one approach has been to outsource public service delivery to non-state actors, but there are concerns that this hollows out the public sector and undermines government legitimacy. Another approach is the titanic task of trying to reform the public sector: changing the incentives, organisational structures, resources, or personnel. But there is a third way: work within the existing government systems and find bespoke, low-cost ways to improve the functioning of government.

In this blog we identify and test for a low-cost way to improve the implementation of a school governance reform programme in Tanzania, which was rolled out nationwide to over 20,000 schools between 2017 and 2019. School Quality Assurance Officers, previously known as school inspectors, conduct a three-day Whole School Visit (WSV) that produces detailed diagnostic information on school quality and recommendations for improvement. This information is shared with the school community at an ‘exit meeting’, and a report is sent to the director of local government. The report provides a long, detailed list of recommendations along six different domains of school quality, and also a summary of the main recommendations that should be prioritised. In theory, the local government leadership would direct the close-to-school monitors, called Ward Education Officers, to follow up with schools to ensure compliance.  

We hypothesised that a key challenge to implementation was that the monitors do not have direct and timely access to the WSV reports, nor are they authorised to act without specific instructions from their supervisors. District level bureaucrats receive the reports but may not provide school-specific instructions to the school monitors because: (a) of the high costs of processing many (roughly 100) reports; or (b) required recommendation follow-up efforts would crowd out efforts for existing district-specific priorities. This relates to a broader challenge of implementing programmes that require coordination between two different agencies, each with their own existing set of tasks and targets and lacking the additional resources to achieve the goals of the joint programme. The school monitors and their supervisors fall under the Ministry of Regional and Local Government, but the school inspectors under the Ministry of Education.  

To address this challenge, the research team implemented a low-cost information intervention, sending the main recommendations of a WSV report directly to the personal phone of the relevant school monitor. We secured permission for the district Chief School Quality Assurance Officers to email us all their completed WSV reports. A research staff member then collated the main recommendations, summarised them, and sent them to the monitor. Text messages were sent roughly every two months.  

The intervention was officially approved by the Ministry of Regional and Local Government at inter-agency meetings that brought together all the relevant bureaucrats, and where the monitors’ superiors publicly endorsed the messages. To this end, messages were signed as coming from their superiors, legitimising readily accessible, customised, actionable, and storable information, and empowering them to act on it without additional direction. The text messages thus improve the flow of information between and within agencies and reduce the costs of adequately responding to it. 

In our study, we exploit a randomised phased-in approach, selecting a nationally representative sample of about 400 schools from across Tanzania—one from each Ward. From this group, the research team randomly assigned half to treatment. Government agreed to prioritize these schools, and only visited the remaining schools on the list two years later. To test for the implementation failures above, we sent text summaries of the visit’s main recommendations to half of the early WSV schools.  

Results 

Our main results are the following:  

1. Good implementation, but weak coordination with the school monitors. Overall, the programme was very well implemented. The school inspectors all received training, visited the schools, and conducted all the activities required under the new programme. Furthermore, the diagnostic information and ratings found in the reports predict growth in student learning, and the head teachers who received WSVs indicated that they learnt something new about their school. But this information rarely reached the school monitors: Only a fifth could show a copy of the report, only half of them attended the ‘exit meeting’, and only a fifth discussed the results of the WSV with their superior. 

2. Text messages substantially improved programme impact. The school governance programme by itself (denoted as Visit) did not improve school management, teaching practices, nor student learning. But when combined with the text messages to school monitors (denoted as Visit&Text), teaching practices improved by about 0.27 standard deviations (SDs) after 6-12 months. Moreover, after about 18 months student learning was 0.11SDs higher in the Visit&Text arm, relative to Visit only arm.   

3. Highly cost-effective intervention. Although the impacts on student learning were modest, they were highly cost effective, because it required limited additional human resources. The main cost driver was organising the regional workshops to get approval from local authorities (74%), followed by the cost of paying someone to manage the process (25%). The costs of airtime and a data management system to automate the sending of text messages was nominal.       

Implications and policy recommendations 

This study informs how governments can put into practice education policy reforms at scale, which is a key area of the What Works Hub for Global Education framework. It underscores the efficacy of low-cost, scalable interventions like targeted text messages to improve the implementation of government programmes. By simply improving coordination through a reduction in the cost of processing and sharing information, such interventions can significantly enhance the efficiency and effectiveness of public sector initiatives. Small tweaks can make a big difference, as long as one can identify and fix the weak links in government implementation.  

But the programme did not solve the deeper problems related to public service delivery and inter-agency coordination in Tanzania.  If anything, the SMS intervention just highlights the cost of weak coordination, by showing the benefits of addressing it. As a metaphor, our booster programme was the application of tape to plug a leaking pipe. It might be too expensive to make wholesale improvements to your home plumbing. But if you can correctly identify the source of the leak, some tape can save you a lot of money.  

In our view, the results of this study have the following policy recommendations:  

• Make bureaucrats’ work easier, not harder. Too many programmes add additional burdens on already over-burdened civil servants. But interventions targeted at making it easier to implement existing programmes could improve implementation in a sustainable way. In our case, it was unrealistic to expect district supervisors, who are each responsible for roughly 100 schools, to read all the reports and communicate specific tasks to the school monitors. And it was very difficult for the school monitors to get access to the reports themselves. The text intervention circumvented all of that.  

• Identify the weak links in the long chain of implementation. Government programmes typically require involvement of multiple stakeholders, agencies, and levels of government. In each step there is risk of a breakdown of implementation. In our case, a deep understanding of the planned implementation strategy, the institutional context and the de facto roles of all stakeholders involved, allowed us to identify one potential weak link. This can be (should be) done with every government programme, especially complex ones that require coordination across different ministries.  

• Augment rather than circumvent existing incentive structures. It is likely that the text messages would have been ineffective, if they were not signed as coming directly from the district supervisors. The school monitors will have limited incentives to reallocate tasks away from what is expected from their superiors. Although the programme was externally implemented, it operated within the existing government structures and lines of authority. This required substantial upfront engagement, with multiple levels of government, so that the programme was officially approved and endorsed. 

Future research in this area could help inform the specific mechanisms by which the SMS are effective through A/B test of different contents and dosage (frequency, timing) of SMSs, and studying their long-run effects to help governments design the best timed and targeted SMS campaigns.   

The czar of plugging leaking pipes? 

On the face of it, it seems obvious that governments should implement such cost-effective innovations. But the incentives to do so are limited. Improving implementation requires either an increase or a re-allocation of resources. But, given short political cycles, central governments have incentives to announce new programmes, rather than allocate more resources towards improving the functioning of existing programmes. Moreover, separate agencies are unlikely to divert resources if it undermines their own existing performance benchmarks. Bureaucracies are typically rewarded for meeting easily observable output benchmarks (e.g., number of schools visited), rather than outcomes (e.g., improvement in school quality).  

One idea would be to create a dedicated unit responsible for improving implementation of specific programmes. Its tasks should include (a) identifying the weak links in the implementation of specific programmes, (b) making specific recommendations for low-cost ways to improve the programme, and (c) allocating its own internal resources towards fixing it. Given the fact that many programmes require coordination between different agencies, this unit should be autonomous. But it should work closely with all the agencies responsible for implementing a similar programme.  

This idea is different to the deliverology approach–where there is a dedicated unit for service delivery–in its (limited) ambition. The unit should be relatively small, with a limited number of staff and discretionary budget, yet sufficiently independent. This will help overcome political challenges in setting up and sustaining a new unit, given incentives for agencies to exert control and extract resources.  

Habyarimana, J., and Cilliers, J. 2024. A simple text message intervention can help improve the effectiveness of government programmes. What Works Hub for Global Education. 2024/009.

https://doi.org/10.35489/BSG-WhatWorksHubforGlobalEducation-BL_2024/009