«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
In sum, as discussed further below, if the project had been designed to permit rigorous impact evaluation rather than monitoring, a plan for gathering data on control units would have been created as part of the initial project design. Ideally, one would have compared treated and untreated municipalities inside the seven regions. In the absence of untreated municipalities inside the regions, data could have been gathered on appropriately selected municipalities outside the region.14 Surveys should have included residents of untreated municipalities, and evaluations of municipal capacity (such as the GRADE study) should have included pre- and postmeasures on municipalities with which USAID/Peru’s contractor was not assigned to work.
13 Interviews, Ayacucho, June 27, 2007.
14 However, as discussed below, without assignment, data on controls may also not help with the inferential issues mentioned in the previous paragraph.
0 IMPROVING DEMOCRACY ASSISTANCE An Alternatie Ealuation Design It is possible, looking backward, to describe an ideal randomized impact evaluation design for the decentralization project that could have been implemented in 2002. Assume that the decision to implement the decentralization project in the seven nonrandomly chosen regions was not negotiable; inferences about the effect of the intervention would then be made to the districts and provinces that comprise these regions.
The simplest design would involve randomization of treatment at the district level. Districts in the treatment group would be invited to receive the full bundle of interventions associated with the decentralization project (e.g., training in participatory budgeting, assistance for civil society groups); control districts would receive no interventions.
There are two disadvantages to randomizing at the district level, however. One is that some of the relevant interventions in fact take place at the provincial level.15 Another is that district mayors and other actors may more easily become aware of treatments in neighboring districts.
For both of these reasons it would be useful to randomize instead at the provincial level. Then all districts in a province that is randomly selected for treatment would be invited to receive the bundle of interventions.
Several different kinds of outcome measures could be gathered. Survey evidence on citizens’ perceptions of local government responsiveness would be useful, as would information on participation in local government and evaluations of municipal governance capacity taken across all municipalities in the seven regions (both treated and untreated).
A difference in average outcomes across groups at the end of the project—for example, differences in the percentage of residents who say government services are “good” or “very good,” or the percentage who say the government responds “almost always” or “on the majority of occasions” to what the people want—could then be reliably attributed to the effect of the bundle of interventions, if the difference is bigger than might reasonably arise by chance.
One feature of this design that may be perceived as disadvantageous is the fact that treated municipalities are subject to a bundle of interventions. Thus, if a difference is observed across treated and untreated groups, it may not be known which particular intervention was responsible (or most responsible) for the difference: Did training in participatory budgeting matter most? Assistance to CSOs? Or some other aspect of the bundle of interventions? This problem arises as well in some medical trials and other experiments involving complex treatments, where 15 Some interventions also occurred at the regional level, particularly toward the end of the
it may not be clear exactly what aspect of treatment is responsible for differences in average outcomes across treatment and control groups.
Despite this drawback, it seems preferable to design an evaluation plan that would allow USAID to know with some confidence whether a project it financed made any difference. Bundling the interventions may provide the best chance to estimate a causal effect of treatment. Once this question is answered, one might then want to ask what aspect of the bundle of interventions made a difference, using further experimental designs.
However, another possibility discussed below is to implement a more complex design in which different municipalities would be randomized to different bundles of interventions.
USAID/Peru is preparing to roll out a second five-year phase of the decentralization project, possibly again in the seven regions in which it typically works. At this point, all municipalities in the seven regions were already treated (or at least targeted for treatment) in the first phase. This may raise some special considerations for the second-phase design. The committee’s understanding is that there are several possibilities for the actual implementation of the second phase of the project; which option is chosen will depend on the available budget and other factors. One is that all 536 municipalities are again targeted for treatment. As in the firstphase design, this would not allow the possibility of partitioning municipalities in the seven regions into a treatment group and controls.
In this case the best option for an experimental design may be to randomly assign different treatments—bundles of interventions—to different municipalities. While such an approach would not allow comparison of treated and untreated cases, it would allow us to assess the relative effects of different bundles of interventions. This may be quite useful, particularly for assessing the question raised above about which aspect of a given bundle of interventions has the most impact on outcomes. Do workshops on participatory budgeting matter more than training CSOs? Randomly assigning workshops to some municipalities and training to others would allow us to find out.
A second possibility for the second phase of the project is to reduce the number of municipalities treated, for budgetary reasons. Suppose the number of municipalities were reduced by half. The best option in this case is probably to randomize the control municipalities out of treatment, leaving half of the universe assigned to treatment and the other half as the control. Those municipalities assigned to treatment would be offered the full menu of interventions in the decentralization program.
Of course, randomizing some municipalities out of treatment is sure to displease authorities in control municipalities as well as USAID officials who would want to choose municipalities where they believe they have the greatest chances for success. Yet if the budget only allows for 268 IMPROVING DEMOCRACY ASSISTANCE municipalities assigned to treatment and 268 to control, this displeasure will arise whether or not the allocation of continued treatment is randomized. In fact, as discussed below, it may be that using a lottery to determine which municipalities are invited to stay in the program is perceived as the fairest method of allocating scarce resources.16 The preceding discussion of USAID/Peru’s past and present support of decentralization projects suggests that impact evaluations could be achieved by incorporating techniques of randomized evaluation. Current monitoring efforts do not give USAID evidence about the impact of investments in local government, yet such decentralization and local government strengthening projects are a staple in the USAID DG toolbox. The good news is that the committee’s field team concluded that a randomized evaluation of key aspects of the Peru decentralization project would be feasible with only modest adjustments in project design.
Supporting Multiparty Democracy in Uganda In 2007, USAID/Uganda finalized plans for two comprehensive multiyear DG initiatives in response to the changing political dynamics in the country, especially the reintroduction of multiparty politics. One project, entitled Linkages, aims to strengthen democratic linkages within and among the Ugandan Parliament, selected local governments, and CSOs, building on the mission’s longstanding support of legislative strengthening. The Linkages project is intended to “assist civil society groups, local government, and Parliament to demand transparency, accountability, and more effective leadership at both the local and national levels that will ultimately result in increased and improved essential service delivery and effective democratic representation” (USAID/Uganda 2007a). The guiding hypothesis of this project is that investments in citizen participation will drive growing demands for responsiveness and thus increase the overall quality of participation, representation, and interaction across all levels of government.
The second project, comprising a set of activities to strengthen multiparty democracy in Uganda, has the goal of “increasing democratic participation, transparency, and accountability in Uganda by supporting peaceful political competition, consensus building, and capacity building of major parties” (USAID/Uganda 2007b). This effort is driven by the hypothesis that increasing citizen participation in the development of political parties will improve the overall quality of political participation, representation, response, and interactions.
16 For reasons discussed above, it may also be useful to conduct the randomization at the
Both projects are multifaceted. They involve a wide range of interventions at different levels, from support for party development at the national and local levels, to continued legislative-strengthening activities in Parliament, to capacity-building efforts with CSOs and local governments. Recognizing the complexity of these programs, the field team worked with the mission to identify a subset of distinct interventions that would be amenable to randomized evaluation. Three specific designs are described here; additional details are included in Appendix E. As with the Peru programs discussed above, the goal of this exercise is to assess the feasibility of this approach for the programs under consideration by the Uganda mission and to highlight improvements it could afford for making causal inferences about program success. As in the Peruvian case, the evaluation designs were shaped in consultation with mission staff.
Support for CSOs One of the core activities envisioned in the Linkages project is a capacity-building program with grants to CSOs to enable them to monitor local governments and help improve representation and service delivery at the local level (USAID/Uganda 2007a). These grants are thought to have two main impacts: (1) to develop a more robust civil society by increasing the capacity of the CSOs that are awarded the grants and (2) to improve the performance of government service delivery by increasing civic input and oversight of government officials. Whether such grants indeed have these effects is a question that can be addressed using randomized evaluation.
The best possible strategy for measuring impact would involve a large N randomized evaluation. Because a large N study would require providing grants to a large number of CSOs (more than 50) and additional monitoring and measurement, the costs are greater than what is currently envisioned for CSO grants in the Linkages project. However, this design offers substantial benefits over a small N comparison and is of general interest to USAID (especially given the results of the Gugerty and Kremer (2006) study on assistance to women’s self-help CSOs described in Chapter 5).
In the proposed design, across carefully matched subcounties, large grants, small grants, and no grants would be allocated by lottery to local CSOs working on HIV/AIDS.17 One goal would be to compare the impact 17An additional benefit of focusing on HIV/AIDS is that USAID/Uganda is receiving a very large infusion of funds from the President’s Emergency Plan for AIDS Relief program, so information about the effectiveness of CSOs doing HIV/AIDS service delivery would serve broader mission interests.