In the last couple of years I have bee involved in several major evaluation projects, which are detailed below.
- In Burkina Faso we evaluated the impact of SMS reminders on retention and adherence of people living with HIV. The project was financed by the International Initiative for Impact Evaluation (3ie).
- In The Philippines we evaluated the long-term effects of expanded health insurance targeted to young children in comparison to those of a pay-for-performance (P4P) scheme for hospital staff designed to improve the quality of care. The project was financed by the Rotterdam Global Health Initiative.
- In Uganda we evaluated a police accountability project and the local government councils’ score card initiative. The project was financed by the Evaluation Department (IOB) of the Netherlands Ministry of Foreign Affairs.
- In Tanzania we evaluated the impact of a rural electrification program, implemented by the Facility for Infrastructure Development ORIO and financed by the Netherlands Ministry of Foreign Affairs.
The projects are further detailed below.
The rapid uptake of mobile phones across the developing world in recent years has inspired a host of innovative concepts for how these devices can be harnessed to promote public health. In these contexts, the potential of mobile devices to promote healthy behaviors and facilitate health service delivery looms large. Yet, to date, few rigorous evaluations have been conducted demonstrating the impact of “mobile health” (mHealth) interventions.
Concomitantly, the treatment of people living with HIV is still a major public health challenge for developing countries. In recent years, it has been observed that many patients who begin antiretroviral therapy do not continue beyond the first two years. This is despite the fact that ART is a lifelong treatment course and that stoppages or breaks in treatment will likely lead to a rapid deterioration in health. In Sub-Saharan Africa, rates of attrition from ART programs are particularly high. One systematic review found that 40% of patients are lost after two years, 21% of these in just the first six months of treatment (Rosen and Fox, 2007).
The intervention that was implemented as randomized controlled trial took aim at these high rates of patient attrition and employed an everyday technology, the mobile phone, to promote healthy behaviors. Specifically, the intervention was an SMS support system for patients under antiretroviral treatment all across Burkina Faso. Randomly selected participants recruited from health centers across the country received weekly SMS reminders. These SMS were intended to encourage patients to take their pills and to remind them of the importance of ART for their health.
As immediate intended outcomes we identified enhanced retention and adherence (i.e. pill-taking) to antiretroviral therapy. Improvements in patient health and psycho-social wellbeing were also desired. Primary outcomes were: (1) retention, (2) adherence, and (3) physical health. Secondary outcomes were: (1) subjective health, and (2) mental health. The intervention was implemented for a period of two years. The study was officially registered as trail with ISRCTN. The trial reference N° is ISRCTN16558614.
Patients on antiretroviral regimen had been enrolled in the study from February to March 2015 when we carried out the baseline survey among more than 3,800 people living with HIV all across Burkina. We then randomly assigned the participants to an SMS intervention or control group. For those participants enrolled in the intervention, SMS were sent on a weekly basis, reminding them to take their antiretroviral medications. Four treatment groups were set up varying by message type and frequency:
Treatment 0: Control group
Treatment 1: SMS text with varying content, low frequency
Treatment 2: SMS text with varying content, high frequency
Treatment 3: SMS text with varying content and ASCII image
Treatment 4: ASCII image, low frequency
Fieldwork has been concluded. Data collection was done at baseline and during three follow-up surveys at 6, 12 and 24 months after the start of the intervention.
In short, our findings suggest the following: Based on the intent-to-treat (ITT) estimates, patients benefitting from any version of the intervention did not improve retention, adherence or physical health but did improve mental health outcomes (current and future happiness, and faith in life). The largest impact occurs in the first year of the intervention but the impact dissipates in the second year suggesting that the intervention is most effective in the short run and should not necessarily be implemented as part of long-term health care provision. We find no difference in impacts as consequence of the type or frequency of the SMS messages indicating that a low level intervention is sufficient to reap benefits. Moreover, our study shows that lack of (ICT) literacy combined with a large variety of local languages can considerably undermine the benefits of SMS interventions if not addressed alongside the intervention. Notably, while physical health outcomes might only be improved to a small extent, SMS messages are valuable since they have a strong and persistent impact on psychosocial and mental health over time. To date most research focuses on improvements in biomarkers without considering other dimensions of health. Our findings suggest that future research should investigate psychosocial health outcomes including preferences and choices of PLHIV in more detail.
The following publications have already been realized in the context of the study:
- Geldof, M., B.A. Thiombiano & N. Wagner (2020). When I receive the SMS, it is a sign of love: Symbolic Connotations of SMS messages for People living with HIV in Burkina Faso. AIDS Care. doi: 10.1080/09540121.2020.1769832.
- Artavia-Mora, L., N. Wagner, B.A. Thiombiano & A. Bedi (2020). Adaptation and biomedical transition of people living with HIV to antiretroviral treatment in Burkina Faso. Global Public Health: An International Journal for Research, Policy and Practice. 15(5): 638-653. doi: 10.1080/17441692.2019.1704819.
- Demena, B.A., L. Artavia-Mora, D. Ouedraogo, B.A. Thiombiano & N. Wagner (2020). A Systematic Review of Mobile Phone Interventions (SMS/IVR/calls) to improve Adherence and Retention to Antiretroviral Treatment in Low and Middle-Income Countries. Aids Patient Care and STDs. 34(2): 59-71. doi: 10.1089/apc.2019.0181.
- Wagner, N., D. Ouedraogo, L.D. Artavia-Mora, A.S. Bedi & B.A. Thiombiano (2016). Protocol for a Randomized Controlled Trial Evaluating Mobile Text Messaging to Promote Retention and Adherence to Antiretroviral Therapy for People Living With HIV in Burkina Faso. JMIR Research Protocols, 5(3): e170. doi: 10.2196/resprot.5823.
Impact Evaluation: Long-term effects of a health reform to improve hospital quality
with Stella Quimbo
In this project we built on the Quality Improvement Demonstration Study (QIDS), a randomized policy experiment conducted in the Philippines from 2003 to 2008. We analyze the long-term effects of a health reform that had two different intervention arms. The first intervention is an expanded health insurance program targeted to young children. The intervention provided full insurance support for children under 5 years of age. The second intervention is a pay-for-performance (P4P) scheme for hospital staff designed to improve the quality of care. We examined the long-term effects of this randomized policy experiment. Five years after the end of the original QIDS program we resurveyed the doctors who initially participated in the project. Field work was carried out in the first half of 2013, we achived tracking rates of more than 94% and remeasured the quality of care provided by the doctors. During the three years of the intervention, quality—as measured by clinical performance and value vignettes—improved and was sustained in both intervention sites compared with controls. Five years after the interventions were discontinued, we found that the intervention sites continued to have significantly higher quality compared with the control sites. The previously documented quality improvement in intervention sites appears to be sustained; moreover, it was subject to a very low (less than 1% per year) rate of decay in quality scores. The project was funded by the Rotterdam Global Health Initiative.
The following publication has already been realized in the context of the study:
- Quimbo, S., N. Wagner, J. Floremtino, O. Solon & J. Peabody (2016). Do Health Reforms To Improve Quality Have Long-term Effects? Results of a Follow-up on a Randomized Policy Experiment in the Philippines. Health Economics. 25(2): 165-177. doi: 10.1002/hec.3129.
Impact Evaluation: Democratization, promotion of rule of law and control of corruption
with Wil Hout and Rose Namara
Project 1: Police Accountability and Reform Project
The Ugandan Police Accountability and Reform Project (PARP) was implemented between 2010 and 2012. The project aimed at
- Enhanced police accountability and
- More effective public order management.
Enhanced police accountability should be achieved by implementing the following steps:
- More effective civilian oversight of the Ugandan police force;
- More effective engagement between citizens and the police in order to enhance public safety and security;
- Attainment of public order management with greater respect for human rights.
We assessed the impact of PARP on police integrity by conducting a survey among 600 police officers that were interviewed about police integrity by means of twelve hypothetical vignette cases depicting context-specific, undesirable behaviour of varying degrees of severity. The assessments of the cases by the police officers were analysed using propensity score matching, inverse probability weighting and seemingly unrelated regression techniques. We showed that self-selection of police officers into the program was unlikely to drive the results. The results suggest that officers participating in PARP activities (i) judge the presented cases of misconduct more severely, (ii) are more inclined to report misconduct and (iii) also expect their colleagues to judge misbehaviour at the level of the police more critically although the latter two coefficient estimates are smaller in size. This suggests that PARP activities have affected the perception of police officers but only encouraged them moderately to actually take action against bad practices.
The following publications have already been realized in the context of the study:
- Hout, W., N. Wagner & R. Namara (forthcoming). Holding Ugandan Police to Account: Case study of the Police Accountability and Reform Project. Edited Volume “The State of Accountability in the Global South: Challenges and Responses” by S. Bergh, C. Sathyamala & S. Pellissery (Eds.). Publisher: Edward Elgar.
- Wagner, N., W. Hout & R. Namara (2020). Improving Police Integrity in Uganda: Impact Assessment of the Police Accountability and Reform Project. Review of Development Economics. 24(1): 62-83. doi: 10.1111/rode.12643.
- Wagner, N. & W. Hout (2019). Police integrity and the perceived e ectiveness of policing: Evidence from a survey among Ugandan police officers. Edited Volume “Exploring Police Integrity” by S. Kutnjak Ivkovic & M. Haberfeld (Eds.). Publisher: Springer.
- Wagner, N., M. Rieger, A.S. Bedi & W. Hout (2017). Gender and Policing Norms: Evidence from Survey Experiments among Police Officers in Uganda. Journal of African Economies. 26(4): 492-515. doi: 10.1093/jae/ejx014.
Project 2: Local Governemnt Councils’ Score Card Initiative (LGCSCI)
The Local Government Councils’ Score Card Initiative (LGCSCI) was set up in 2009 by the Advocates Coalition for Development and Environment (ACODE). The local scoreard initiative is an ongoing endeavor to strengthen citizens’ demand for effective public service delivery and accountability. More specifically, it aims to ‘strengthen the weak political accountability mechanisms between citizens and elected leaders that prevent citizens from receiving adequate public services, mainly by overcoming information-related barriers and boosting Councilor professionalisation and performance’ (VNG International 2014: 1). The local scorecard initiative has the following three objectives:
- A citizenry that increasingly demands for better accountability from political leaders and public officers;
- A citizenry that increasingly demands for better performance and efficient service delivery and,
- Enhanced demand for improved accountability and services at the local level, which moves in a vertical trajectory and translates into improved governance, accountability and efficiency at national and sub-national levels (VNG International 2014: 7-8).
The local scorecard initiative’s three main activities are: 1) an annual assessment of district councils’ and councilors’ performance based on a scorecard; 2) feedback reports on assessments, covering the included districts and a synthesis report providing a comparative analysis; and 3) capacity building activities aimed at increasing the effectiveness of councils and councilors on the one hand and citizens’ demand for accountability on the other hand. Increased capacity of the local councilors is expected to result in better oversight of local service providers and thus in better service delivery.
We conducted an impact assessment of the local scorecard initiative. To this end, we collected district level administrative data and combineed that information with Afrobarometer data on the citizens’ perceptions of local council performance. We applied a mixed-methods approach combining the analysis of the district-level panel dataset (2005-2016) with qualitative information from interviews and focus group discussions. By contrasting scorecard and non-scorecard districts we expand the evidence base since existing studies focus exclusively on scorecard districts. Tumushabe et al. (2013) studied trends in the coverage of public services in scorecard districts and Grossman and Michelitch (2018) assessed the impact of the scorecards when randomly distributed to the constituents. Thus, the latter are interested in the effect of the dissemination of the performance results and treat the scorecard ratings as an outcome whereas we consider the scorecards as the intervention. Employing a rigorous difference-in-difference framework with district-specific trends we show that the identifying assumptions hold, supporting our choice of analysis, but we only find a limited measurable impact of the scorecards on public services. At first sight and if only employing a simple difference in means test we seem to identify large and significant impacts of the scorecard intervention on public service delivery. Yet, results are driven by omitted variables. When employing our rigorous difference-in-difference specification with district specific time trends, we only identify an effect on the share of the budget spent. Note that in the Ugandan system unused budget from the centre needs to be returned. The finding that districts with the scorecard initiative spent on average 3.8 percent less suggests that the local councils are more careful in their decision making over the local budget. It indicates that the accountability aspect of the scorecard initiative results in less waste of public money and less ad hoc spending. It further points at an accountability-investment trade-off indicating that increasing the number of rules and regulations embedded in accountability mechanisms might result in under-spending. It is clearly desirable that spending’ decisions are carefully done but if lack of spending is equivalent to foregone budget, an accountability mechanism might even result in less investment in local projects. It seems that from the point of view of the councillors the scorecards impose an additional set of constraints and regulations limiting their scope for decision making. The question therefore is whether a reduction in public spending is an actually intended outcome of the scorecard initiative or rather an unintended by-product of the increased oversight of civil society over the work of the local councillors. At the same time, we have to relate the result to the overall and per capita public spending. But we show that even if there is a slight reduction in the budget spent it does not have implications on the per capita and total public budget spent.
The analysis for the Afrobarometer data indicates that councillors in scorecard districts are perceived as being less corrupt. The effect of -0.20 is sizable compared to the average of 1.38. Thus, the citizens’ perceptions about corruption of the local councillors seem to be reduced as a consequence of the scorecard initiative. This finding is in line with our argument that the initiative can be seen as an additional layer of monitoring. Yet, the overall performance of the local government councillors is not perceived as any better in response to the scorecard intervention.
Impact Evaluation: Electrifying Rural Tanzania
with Rheinisch-Westfälisches Institut für Wirtschaftsforschung (RWI) and Georg-August University Göttingen
Access to reliable modern energy is one of 17 Sustainable Development Goals. The project aimed at providing electricity to 133,000 people that would be without electricity in the absence of the project. The project targeted three remote Tanzanian districts. Two of them, Biharamulo and Ngara district, are located in northern Tanzania at the border to Rwanda (Kagera region) and the third, Mpanda district, is situated in the west of the country at the Congolese border (Rukwa region). All are located too far away to be covered by the existing national grid in the eastern parts of the country. All three generators were successfully set up.
Taking into account the set-up of the electrification project, we applied a three-tier evaluation approach:
Tier 1 examined the socio-economic effects on the domestic user level and comprised a large household survey using a structured questionnaire.
Tier 2 examined the effects on small businesses. It comprised a smaller survey of micro-enterprises using both quantitative and qualitative techniques.
Tier 3 examined the effects on stakeholders like the Ministry of Energy and Minerals, the Rural Energy Agency and, most importantly, TANESCO, the Tanzanian parastatal utility provider we were working with. It comprised a profound qualitative institutional analysis and concentrated on the effects of the project’s capacity building activities and the effectiveness of the O&M phase.
The evaluation was based on a series of field visits and surveys. An inception mission was conducted in 2014, followed by two rounds of surveys: a baseline survey in 2014/2015 before project implementation and a follow-up survey in 2018/2019. Additional intermediate data collection specific to the institutional assessment took place in 2016. We employed a mix of methods to determine the impacts of the project. In total, we surveyed 1155 households in rural areas as well as 300 households and 595 enterprises in towns. For rural enterprises, we conducted a smaller survey using qualitative interviews. We additionally used willingness-to-pay methods to assess the degree to which connected firms and households value changes in the electricity service quality in towns after implementation of the project.
We find that 38 percent of households in newly grid-covered areas are connected to the grid at the time of follow-up. One of the most striking findings is the massive rise of solar panel use among off-grid households. In control communities, 60 percent of households satisfy their electricity demand through solar panels at follow-up, which is three times more than four years earlier. Because of this rise of solar panel usage, the project does not achieve an increase in access to electricity according to our difference-in-differences estimations. Quality and power of the service, though, are higher for grid-connected households. This is reflected in more grid-connected households using a TV set, line-powered radios and electric irons. Also the usage of artificial light is considerably higher. Outages remain a problem in particular for urban firms. Strategies to cope with outages are an important question in the context of electricity reliability, especially for enterprises. Outages affect firms in different sectors differently. The majority of trade enterprises (around 70 percent) but only half of service and a third of manufacturing firms continue normal operations in case of outages. Half of manufacturing firms even have to stop their business.
As an additional impact measure, we elicited the hypothetical willingness to pay (WTP) of urban enterprises for a service improvement. Those firms that are more dependent on reliable engergy access, i.e. manufacturing and service firms, state a higher WTP than trade firms.
What are the lessons learned and policy recommendations?
- Electrification interventions are no longer about providing mere access to electricity. The solar market is dynamic, fast moving and working even in remote rural areas like those under evaluation to cater for basic electricity demand.
- The adequacy of electricity generation is only one supply-side factor that affects reliability. Similarly important, reliability depends on and can be further enhanced through conditions of the power system infrastructure.
- The recurrent problem with setting up O&M components calls for a clearer definition of modalities on how to contract out critical maintenance tasks.