«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
In this case, the best option for an experimental design may be to randomly assign different treatments—bundles of interventions—to different municipalities. While such an approach will not allow us to compare treated and untreated cases, it will allow us to assess the relative effects of different bundles of interventions. This may be quite useful, particularly for assessing the question raised above about which aspect of a given bundle of interventions has the most impact on outcomes. Do workshops on participatory budgeting matter more than training civil society organizations (CSOs)? Randomly assigning workshops to some municipalities and training to others would allow us to find out.
A second possibility for the second phase of the program is to reduce the number of municipalities treated, for budgetary reasons. Suppose the number of municipalities were to be reduced by half. The best option in this case is probably to randomize the control municipalities out of treatment, leaving half assigned to treatment and the other half in control.
Those municipalities assigned to treatment would be offered the full menu of interventions in the decentralization program.
Of course, randomizing some municipalities out of treatment is sure to encounter displeasure among authorities in control municipalities. Yet if the budget only allows for 268 municipalities assigned to treatment and 268 to control, this displeasure will arise whether or not the allocation of 00 APPENDIX E continued treatment is randomized. In fact, as discussed below, it may be that using a lottery to determine which municipalities are invited to stay in the program is perceived as the fairest method of allocating scarce resources.9 Cost of evaluation under the best-possible designs. The need to gather outcome measures on control units—both through surveys of residents in untreated municipalities and through independent evaluations of municipal capacity in control districts—will mean an additional cost of program evaluation.
However, it is worth bearing in mind that such additional costs would have likely represented only a small fraction of the cost of the overall program as well as of the portion of overall costs going to evaluation.
For example, adding 500 respondents from appropriately chosen control municipalities would likely cost no more than $10,000, a small amount compared to the overall program budget.
In addition, with appropriate design modifications, there might be substantial net savings. One possibility for cost savings would involve substantially limiting the volume of output/outcome indicators gathered by each of the local subcontractors. For example, measures could be sampled across local jurisdictions, rather than gathered quarterly on each of 536 municipalities. A related idea is that local subcontractors could be asked to gather the indicators and report on them each quarter with some positive probability; but they would not actually have to do so in each quarter.
Other Examples: Rule of Law, Political Parties, and Extractie Industries Several of the programs planned under the new Peru strategic assessment might also be amenable to randomized designs. In this section, we briefly review possibilities for experimental designs afforded by programs related to the rule of law, political parties, and extractive industries.
Rule of law. Most of the interventions under the rule of law programs implemented were not amenable to randomization across units. However, there were one or two interventions that could in principle have been randomized. For example, after the passage of a new penal code, some judges in district courts were switched to the new system of judging cases while others were left to clear the backlog of cases that had already entered the courts under the old system. Under the observational (nonexperimental) evaluation plan that was actually adopted, cases administered by judges 9 For reasons discussed above, it may also be useful to conduct the randomization at the
under the new system were compared to cases administered under the old system. Comparisons were made across groups with respect to variables such as the average time to disposition of the court cases.
This nonexperimental design represented a valuable evaluation plan:
There was a comparison made across treated and untreated units on an outcome measure of interest. In this and similar examples, the data seemed to show a substantial effect of treatment.
However, judges were nonrandomly assigned to stay in the old system or migrate to the new one (the chief judge apparently decided who would move). This raises the possibility that characteristics of judges who stayed or migrated are partially or wholly responsible for differences in the average time to disposition.10 In principle, it would have been possible to assign district court judges to the old and new systems at random.
While the research design idea is straightforward, however, it was likely to be politically difficult: Chief judges may not want to relinquish power over these assignments.
Political parties. One idea under the new political parties program is to provide assistance to the major national-level parties in opening or strengthening local offices in selected municipalities. At this point, however, the parties themselves would choose where to open offices, so the design is nonexperimental.
Moreover, if outcomes are not tracked in municipalities in which USAID partners do not support local party offices (i.e., controls), inferences may be especially misleading. Suppose measures are taken today and in five years of local party strengthening and an increase is found. Is this due to the effect of local-party-strengthening activities supported by USAID? Perhaps. Yet it could be due to some other factor, like a change from an electoral system with preferential voting to closed party lists, which would tend to strengthen party discipline and, perhaps, local parties; such a change is currently being considered in Peru.11 The point is 10 While data were not available, it would have been helpful to compare the difference in time to resolution, before and after the switch of systems, among judges who switched and judges who did not; this could have required pre- and postswitch data on both groups of judges. While still nonexperimental, this comparison would lend greater confidence to the claim that the switch in systems had a causal effect on the time to resolution of court cases.
11 In the current electoral system, there is proportional representation at the department level, and voters vote for party lists but can indicate which candidate on the list they prefer;
according to a range of research on the topic, this can create incentives for candidates to cultivate personal reputations and also makes the party label less important to candidates.
Under a closed-list system, voters simply vote for the party ticket, and party leaders may decide the order of candidates on the list. This may tend to increase party discipline and cohesion (as well as the internal power of party elites).
0 APPENDIX E that without data on controls, it will be impossible to separate the effect of USAID local activities from the effect of the law.
At a minimum, then, it would be advisable to consider gathering data on control municipalities. In addition, while an experimental approach may not be deemed feasible in this instance, it is possible in principle, and it would provide a stronger basis for impact attribution than a nonexperimental approach.
Under an experimental design, USAID or the local implementer would select municipalities in which to establish or strengthen local parties randomly, from a set of acceptable municipalities. Local parties would have to accept that USAID or the contractor would select the municipalities. There may be ways to overcome any resistance to such a plan, however; for instance, a party such as Unidad Nacional (the rightist party whose candidate in the 2001 and 2006 presidential elections was Lourdes Flores Nano) has almost no base outside Lima and might accept any help it can get to broaden that base. Another obstacle is that parties may want to target certain kinds of municipalities, for example, those where they already have some support. It may be helpful for this purpose to stratify municipalities—for example, by past levels of electoral support for each party—and conduct the randomization within strata.
Outcome indicators might include the municipal vote share of each party in subsequent elections, with comparisons being made across treated and untreated municipalities; there may be other, harder-to-measure outcomes of interest, too.
Inferences may be complicated if more than one party opens or strengthens an office in the same municipality (i.e., if there are two parties and both are strengthened locally, party vote shares may be unchanged).
This concern may be lessened by the fragmentation of the party system and by the current local dominance of regional parties. In recent regional elections, for example, 23 different regional parties won office across Peru’s 24 departments; these regional parties differ from the national parties whose local roots USAID seeks to strengthen.
Extractive industries. There is currently a very small pilot program that seeks to promote dialogue in two mining communities among the State, companies, and local citizens, with the larger goal of “decreasing the probability of social conflict.” This program has the advantage of possessing a relatively easyto-measure outcome variable, social conflict (compared to, say, transparency). For example, this variable might conceivably be proxied by the annual number of local marches/demonstrations. However, without comparing mining communities with which USAID works to those with which it does not, it will be difficult to evaluate the causal impact of the program on decreasing the probability of social conflict.
0 APPENDIX E In a future rollout of the program, mining communities with which USAID might work could be randomly selected from the set of eligible mining communities. This would provide the most secure basis for attaching a causal interpretation to a finding that, for example, there were fewer marches and demonstrations in communities in which USAID worked than in those in which it did not work.
Selected Designs from Uganda: Civil Society, Parliamentary Strengthening, and Anticorruption Large and Small Grants to CSOs In the proposed project for Strengthening Democratic Linkages in Uganda Program, USAID proposes to provide at least $100,000 per year for grants to CSOs to enable them to monitor local governments and help improve representation and service delivery at the local level. 12 These grants are thought to have two main effects: (1) to develop a more robust civil society by increasing the capacity of the CSOs who are awarded the grants, and (2) to improve the performance of government service delivery by increasing civic input and oversight of government officials.
Across carefully matched subcounties, large grants, small grants, and no grants will be allocated randomly to local CSOs working on HIV/ AIDS. The goal is to compare the effects of large grants to CSOs (treatment group) versus small grants to CSOs (partial control group) in order to determine the effects of increases in CSO funding. Providing small grants to the partial control group allows USAID to assess independently the effect of greater monetary resources, while controlling for the nonmonetary effects of receiving a USAID grant (such as public recognition, special accounting requirements, and outside monitoring). It also facilitates the collection of equivalent data from CSOs in both the treatment and partial control groups. Both the treatment group and the partial control group will also be compared to CSOs in matching sub-counties where no grants are awarded (full control group) to evaluate the total effect of awarding a grant.
Carefully matched groups of three subcounties will be purposively selected so that the subcounties within each group are similar along a number of dimensions that are measurable and likely to be associated with CSO capacity and government service delivery for HIV/AIDS programs. Selection criteria might include the type, size, budget, and experience of the HIV/AIDS-related CSOs already working in the subcounties, as well as the subcounties’ size, urban population, wealth, voting patThe Strengthening Multi-Party Democracy in Uganda program also provides for $100,000