«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
Senge, P. 2006. The Fifth Discipline: The Art and Practice of the Learning Organization, Rev. Ed.
New York: Doubleday.
Skocpol, T. 2003. Diminished Democracy: From Membership to Management in American Civic Life. Julian T. Rothbaum Lecture Series, Vol. 8. Norman: University of Oklahoma Press.
Shadish, W.R., Cook, T.D., and Campbell, D.T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.
Trochim, W., and Donnelly, J.P. 2007. The Research Methods Knowledge Base, 3rd ed. Cincinnati, OH: Atomic Dog Publishing.
Vermeersch, C., and Kremer, M. 2004. School Meals, Educational Achievement, and School Competition: Evidence from a Randomized Evaluation. World Bank Policy Research Working Paper No. 3523. Washington, DC: The World Bank.
Wholey, J.S., Hatry, H.P., and Newcomer, K.E., eds. 2004. Handbook of Practical Program Evaluation, 2nd ed. San Francisco: Jossey-Bass.
Wilson, E.O. 1998. Consilience: The Unity of Knowledge. New York: Knopf.
World Bank. 2004. Monitoring & Ealuation: Some Tools, Methods & Approaches. Washington, DC: International Bank for Reconstruction, The World Bank.
Implementing Impact Evaluations in the Field
To find out, the committee commissioned three expert teams to visit USAID missions overseas in an effort to assess the viability of impact evaluations for past and present DG programming. The key task for each team was to talk with implementers, local partners, and USAID mission IMPROVING DEMOCRACY ASSISTANCE personnel on the ground to assess the feasibility of actually implementing in practice the evaluation methodologies outlined in the previous chapter.
The first part of this chapter presents the results of those field visits. The second part provides responses to the most commonly raised objections that the committee and its field teams heard expressed about the use of randomized evaluations in DG programs.
Before turning to the details of what the field teams found, it is important to highlight a clear and consistent message that came through from all three field visits. All three teams concluded, first, that the introduction of randomized ealuations into USAID project ealuation was both feasible and cost-effectie in many of the contexts they inestigated. They were unanimous that, where possible, adopting such methods would represent an improvement over current practices. Second, they reported that, for projects where randomized ealuations were not possible, other improements to USAID ealuation—for example, improed measurement, systematic collection of baseline data, and comparisons across treated and untreated units—also hae the potential to yield significant improements in the agency’s ability to attribute project impact.
These issues are discussed in Chapter 7. Finally, the teams returned from the field energized by their interactions with mission staff and confident that a willingness, and even excitement, exists about improving the quality of project evaluations. The teams were also impressed with some of the work already being done as part of current project monitoring, in particular in the broadening of measurement strategies beyond project outputs to include an assessment of outcomes.
FIELD vISITS TO USAID MISSIONSAs a complement to the deliberations in Washington and extensive engagement with USAID staff and implementers, the committee felt strongly that its recommendations should be informed by a set of extended field visits to USAID missions. The committee therefore identified a set of missions, representing a diversity of regions, that were engaged in substantial programming on DG issues and were in the process of designing large, new projects in one of USAID’s core DG areas (rule of law, elections and political processes, civil society, and governance). From the list of missions provided, USAID explored the willingness of the missions to host the team and consider new approaches to project evaluation. After negotiating issues of timing and access, USAID and the committee agreed to send field teams to Albania, Peru, and Uganda. The field visits were
intended to accomplish three main goals:
2. to explore the feasibility of introducing impact evaluations in the future, including (but not limited to) randomized evaluation; and
3. to obtain the perspectives of mission personnel and USAID implementers regarding the possibilities for, and impediments to, new approaches to evaluation.
The committee encouraged the field teams to explore the range of DG activities currently under way in each mission, assess the adequacy of current evaluation approaches, and provide concrete examples of how existing approaches could be improved. In addition, the field teams were directed to focus particular attention on the development of an impact evaluation design in one specific area in each mission. The teams focused on local government/decentralization in Albania and Peru and support for multiparty democracy in Uganda.1 Each field team was composed of methodological consultants, academic or other experts with relevant experience in research design or program evaluation and DG issues, and country or regional expertise;
a Washington-based USAID staff member who was familiar with the mission, the committee’s work, and USAID policies and practices; and National Research Council professional staff, who assisted the consultants in meeting the team’s objectives and coordinated the logistics of the field visits.
In evaluating the findings of the three field teams, it is important to keep in mind that the field teams visited missions that had expressed an interest in improving their evaluation strategies. The field teams’ conclusions about the applicability of impact evaluations, especially its sense that standard objections to these designs can be addressed, thus reflect the experiences gleaned from this (nonrandom) sample of missions. It is not known if other missions, especially smaller ones with leaner budgets or those in countries experiencing violent conflicts or particularly rapid
political change, would be as amenable to new approaches to evaluation:
The committee has no control group of non-self-selecting missions with which to compare its findings. Yet the committee believes it unlikely that missions that did not invite the committee to send a field team would have offered novel additional objections. Over the 15 months of the study period, the committee talked with numerous USAID staff and implementers from a variety of areas and with backgrounds and experience with DG programming in a great many countries, and the set of objections that are taken up in the second part of this chapter dominated the responses of everyone with whom the committee spoke.
1 Key results of the field visits are discussed in this chapter and the next. Additional infor
EMPLOyINg RANDOMIzED IMPACT EvALUATIONS
FOR USAID Dg PROJECTS IN THE FIELDRandomized evaluations are widely considered the best method for determining the causal effects of treatment in a broad range of areas, including public health, education, microfinance, and agriculture. As the Olken (2007) and Gugerty and Kremer (2006) studies described in Chapter 5 show, such methodologies are also beginning to be applied to evaluate the effectiveness of projects in the area of democratic governance.
Nonetheless, the committee learned from its consultations with USAID staff and implementers that there is a general feeling that randomized evaluation was not an option for many of the projects that USAID carries out. Even in those cases where randomized evaluations might be possible theoretically, the assumption among USAID staff seemed to be that such approaches would be too difficult to implement in practice, owing to an inability to select treatment groups by lottery, the difficulty of preserving a control group, the difficulty of identifying good indicators for key outcomes, the high cost of the extensive data collection that would be required, or the tension between the flexibility staff believe they need to respond to opportunities and challenges as projects go forward and the need to minimize changes to ensure an effective evaluation.
These are legitimate concerns. To address them, this section discusses how randomized evaluations could be used in current USAID projects, drawing on examples gleaned from the field visits and consultations with practitioners. We begin with a decentralization project in Peru that has already been implemented, outlining how the project monitoring strategy that was employed could have been adjusted to accommodate a randomized component that would have made it an impact evaluation design and showing how such an adjustment would have permitted the mission to generate much stronger inferences about project impact.2 Then a planned multipronged effort to support multiparty democracy in Uganda is described, emphasizing how pieces of the existing project might be amenable to randomized evaluation and showing how adopting such an evaluation method would improve USAID’s ability to assess the project’s effects.3 The committee’s goal is to use these projects as illustrations of the potential payoffs that could accrue from improved evaluation strategies.
2 The discussion here of decentralization in Peru is drawn from the report of a field team led by Thad Dunning, assistant professor of political science, Yale University.
3 These designs were developed by a team led by Devra Moehler, assistant professor of
Decentralization in Peru USAID/Peru launched a project in 2002 to support national decentralization policies initiated by the Peruvian government. Over a five-year
period, the Pro-Decentralization (PRODES) program was intended to:
• support the implementation of mechanisms for citizen participation with subnational governments (such as “participatory budgeting”),
• strengthen the management skills of subnational governments in selected regions of Peru, and
• increase the capacity of nongovernmental organizations in these same regions to interact with their local government (USAID/Peru 2002).