«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
IMPROVING DEMOCRACY ASSISTANCE of large grants to CSOs (treatment group 1) versus small grants to CSOs (treatment group 2) in order to determine the impact of increases in CSO funding. Providing small grants to a second treatment group would allow USAID to assess independently the effect of greater monetary resources, while controlling for the nonmonetary effects of receiving a USAID grant (such as public recognition, special accounting requirements, and outside monitoring). Both treatment groups could then be compared to CSOs in matching subcounties where no grants are awarded (control group) to evaluate the total impact of awarding a grant.
Carefully matched groups of three subcounties would be selected purposively so that the subcounties in each group are similar along a number of dimensions that are measurable and likely to be associated with CSO capacity and government service delivery for HIV/AIDS programs. Selection criteria might include the type, size, budget, and experience of the HIV/AIDS-related CSOs already working in the subcounties, as well as the subcounties’ size, urban population, wealth, voting patterns, background of key officials, location, ethnic composition, number and type of health care facilities, and infection rates. The most important criteria to ensure comparability should be determined in consultations with experts.
Grouped subcounties might be next to each other, but immediate proximity is not necessary (or even desirable).18 In each subcounty one CSO working in HIV/AIDS would be selected with the aim of finding similar CSOs across three subcounties in the group. One subcounty in each group would be randomly assigned to receive a large CSO grant to monitor HIV/AIDS services in the subcounty.
Another subcounty in the group would be randomly selected to receive a small CSO grant for HIV/AIDS. The remaining subcounty in the group would act as the control and receive no grant. This would be repeated for at least 50 groups, preferably more.19 It is important to ensure that (1) the large grant provides a significant increase to the existing budget of the CSOs and that the small grants do not and (2) that the CSOs spend their grants entirely on HIV/AIDS activities within the selected subcounty and that there is no contamination (sharing of resources or expertise) across subcounties. It would probably work best to select CSOs that work only in a single subcounty to prevent supplementing or siphoning of funds to the treatment sites. CSOs in both treatment groups should receive equivalent 18 Instead of grouping subcounties in sets of three, it might be more feasible to use an alternative stratified sampling procedure whereby all subcounties in the sample are stratified into types according to key factors and then subcounties within each stratum are randomly assigned into each of the three categories.
19 Depending on the districts chosen for Linkages, it may be possible to randomly select all the treatment and control subcounties from within the 10 districts.
IMPLEMENTING IMPACT EVALUATIONS IN THE FIELDtechnical assistance and training on how to use the grant money and how to monitor and improve service delivery.
Data would be collected before the grants are awarded, after the money is given (or at several points during the grant period), and two years after the end of the grant in order to assess both short-term and medium-term impacts. Equivalent data would be needed about CSOs and service delivery in the both the treatment and the control subcounties. To study the effect of grants and increased resources on the organizational capacity of the CSOs, data would be collected on the budget, activities, operations, and planning of the CSOs. In addition, pre- and postintervention surveys could be conducted with CSO employees, volunteers, government officials and employees, and stakeholders to evaluate changes in the activities, effectiveness, and reputation of the CSOs.
To evaluate the effectiveness of CSO grants on the delivery of government services, data could be collected on HIV/AIDS services and outcomes within each subcounty. Much of these data may already be collected by the government (such as the periodic National Service Delivery Survey conducted by the Uganda Bureau of Statistics—though perhaps USAID would need to fund an oversampling of respondents in treatment and control subcounties) or perhaps they could be collected in collaboration with other donor projects. Special attention should be given during the research design stage to determine the government activities likely to be affected by greater CSO involvement and how those activities might be accurately measured. Additional data collection could be done through surveys of service recipients or randomized checks on facilities and services. In addition, money-tracking studies of local government and government agencies could be conducted to evaluate the level of corruption in HIV/AIDS projects in the selected subcounties.20 Local Goernment Support One objective of USAID/Uganda’s Linkages program is to increase the capacity of local governments to demand better services and representation and the accountability of local governments to their own constituents. USAID calls for actions that build the knowledge and efficacy of local government leaders, strengthen public and CSO participation, and increase local government involvement in fighting corruption (USAID/ Uganda 2007a:21-22). USAID also notes that, “due to over-absorption of development programs in many district centers and severe underabsorption at the sub-district level, the Contractor should propose methFor more information on Public Expenditure Tracking Surveys and Quantitative Service
ods of working with selected local governments and civil society groups at the sub-county levels in the identified districts” (USAID/Uganda 2007a:16). Although the specific interventions were as yet undefined, the Request for Proposals suggested working with elected and appointed leaders, traditional leaders, women, youth, constituents, and CSOs at a subcounty level. Most likely, the program will consist of a bundle of interventions rather than a single activity.
The fact that USAID plans to work with a sample of subcounties (within 10 preselected districts) makes this activity an excellent candidate for randomized evaluation. The number of subcounties within the 10 districts will almost certainly be enough to provide for a large N randomized evaluation. Therefore, in planning interventions at the subcounty level, provision would be made for the random selection of treatment and control subcounties. One approach would be to randomly select half the subcounties within the 10 districts to be in the treatment group and receive the full bundle of interventions. The remainder of the subcounties would receive no interventions and thus serve as a control group. Alternatively, subcounties could be stratified along district boundaries or other criteria, and random selection could take place within strata to facilitate equivalence on important dimensions.
It is difficult to determine the most appropriate measurement tools without a better understanding of the exact interventions and the goals of the program. Regardless of the measurement approach, equivalent data would need to be collected in the subcounties in the control group as well as those in the treatment group. Ideally, baseline data would be collected before implementation of the program and then again during and after.
USAID could also investigate the possibility of contributing to ongoing data collection efforts by the government or other agencies (such as the yearly school census, the service delivery survey, the Afrobarometer public opinion survey, and public expenditure tracking surveys) in order to provide the necessary funds for oversampling in the 10 selected districts.
In most cases, oversampling will be necessary to obtain data that are representative at the subcounty level.
Interparty Debates In an effort to support multiparty democracy, USAID envisions interventions to “foster discussion and dialogue among the political parties so that difficult decisions can be achieved through compromise and negotiation before they result in conflict and stalemate” (USAID/Uganda 2007b:18). Building on successful interparty dialogues during the campaign before the 2006 presidential elections, USAID is considering sponsoring local-level political debates at the district level and below to engage
IMPLEMENTING IMPACT EVALUATIONS IN THE FIELDcitizens in multiparty politics more effectively. In thinking about how to evaluate such activities, it is natural to ask: How does exposure to interparty debates impact citizen knowledge and attitudes about politics, voter turnout, voting outcomes, and political conflict at the local level?
Randomized evaluation offers a powerful tool for assessing the impact of interparty dialogues. Five voting precincts could be randomly selected to be in the treatment group for each of 14 different parliamentary constituencies. Remaining precincts in the 14 constituencies would make up the control group. In each of the 70 treatment precincts, interparty debates would be held between candidates for Parliament in advance of the next election. Specifically, a given group of candidates vying for a single parliamentary seat would participate in interparty debates in five different precincts within their own constituency. This would take place across 14 different groups of candidates in 14 different constituencies.
Many outcomes of interest are already collected by the electoral commission—voter registration, voter turnout, and the percent vote for each candidate. If interparty candidate debates help mobilize candidates, there should be higher registration and turnout rates in treatment precincts. One might imagine also that debates inform citizens about lesser-known candidates and thus increase the vote for nonincumbent candidates or parties.
Therefore, if debates create a more informed citizenry, there should be a smaller share of the vote for incumbents in treatment precincts. If, instead, debates remind voters of the greater experience and access to largess possessed by the incumbent, the opposite effect would be evident. To gain greater power, a difference in difference estimation strategy21 could be used to evaluate changes from the last election in turnout and vote outcomes (assuming that the boundaries of the voting precincts are relatively stable since the last election and polling-station-level data are available for the last election). An analysis of the distance of control precincts from treatment precincts can also be performed to account for the fact that citizens in neighboring constituencies in the control group may attend or learn about debates in the treatment precincts.
To assess the impact of interparty debates on local conflict, one could also compare measures of election-day violence and intimidation gathered by DEMGroup, party observers, or outside monitors.
If resources were available to conduct surveys in treatment and control precincts, the evaluation would provide an even richer perspective on citizen knowledge, attitudes, political tolerance, and behaviors, enabling a better understanding of the causal pathways linking debates with registration, turnout, and vote choice. Ideally, pre- and posttreatment panel surveys would be carried out in treatment and control sites. Of course, 21 See Chapter 5 for a description of this evaluation design.
IMPROVING DEMOCRACY ASSISTANCE care must be taken to ensure that the population surveyed in the treatment sites is comparable to those surveyed in the control sites. For example, it would be misleading to survey only those individuals who attended the debates in the treatment sites but to survey a random sample of individuals in the control group (including those who would have attended if the debate were held in their area and those who would not have). A random sample of all adult citizens in both treatment and control groups would be more informative.