«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
set of agency-university partnerships designed to facilitate high-quality evaluations and research in particular sectors or issues. These partners should also be involved in designing and implementing a range of discussion/learning activities for DG officers in regard to evaluations and other research on democracy. Possible models include the “centers of excellence” funded by the U.S. Department of Homeland Security or the National Institutes of Health. In addition to providing expertise to advise programming and research to advance knowledge, such agencyuniversity centers could assist DG—and USAID more broadly—in developing a standardized training module on evaluation techniques for DG program staff.
AgENDA FOR USAID AND SORAAs part of its charge from USAID, the committee was asked to recommend a “refined and clear overall research and analytic design that integrates the various research projects under SORA into a coherent whole in order to produce valid and useful findings and recommendations for democracy program improvements.”5 Various parts of this design have been dealt with in depth in earlier chapters and will not be repeated here. But the committee does want to summarize the essential elements it believes could enable SORA to continue to serve as a major resource for USAID in studying the effectiveness of its programs and providing knowledge to guide policy planning.
Retrospective Studies SORA began its work by exploring how USAID might mine the wealth of its experience with DG programs around the world to inform its future work. Based on the study by Bollen et al (2005) and its own investigations, the committee found that the records and evaluations of past USAID DG projects could not provide the requisite baseline, outcome, and comparison group data needed to do retrospective impact evaluations of those projects. Therefore the committee recommends that the most useful retrospective studies that USAID could support, if it chooses to, would be long-term comparative case studies that examine the role of democracy assistance in a variety of trajectories and contexts of democratic development. A diverse and theoretically structured set 5As discussed in Chapter 1, in 2000 the Office of Democracy and Governance in the Bureau for Democracy, Conflict, and Humanitarian Assistance created SORA, which consists of a number of research activities. SORA’s goal is to improve the quality of U.S. government DG programs and strategic approaches by (1) analyzing the effectiveness and impact of USAID DG programs since their inception and (2) developing specific findings and recommendations useful to democracy practitioners and policymakers.
AN EVALUATION INITIATIVE TO SUPPORT LEARNINGof case studies could provide insights into overall patterns of democratization that could improve strategic assessment and planning (see Chapter 4). If USAID chooses first to take advantage of current research in the academic and policy communities, it could undertake an effort to engage systematically with those producing research and serve as a vital bridge to accumulate and disseminate evidence and findings in the most policy-friendly format possible. If USAID chooses to support case study research of its own, the committee has suggested some key characteristics for a successful research design.
Strategic Assessment Chapter 3 made the case for a significant effort by USAID, if possible in cooperation with other donors, to support the development of a set of “meso-level” indicators that would be the best focus for USAID’s efforts to track and assess countries’ progress or problems with democratization.
This would be a long-term and expensive effort, but there are already substantial numbers of candidate indicators that could potentially contribute to such an index (see, e.g., the review by Landman 2003). If the United States and other donors are going to continue to support the development of democracy worldwide, the committee strongly believes that it is time to invest the resources needed to provide high-quality indicators comparable to those that have been developed over time in other economic and social fields. Whether or not SORA or the Office of Democracy and Governance became the home for such an effort, its recent experience with a major quantitative assessment of the impact of U.S. democracy assistance (Finkel et al 2007, 2008) and its understanding of the needs of DG officers in Washington and in the field would make it a logical place from which such an initiative could be developed.
Improving Monitoring and Evaluation This chapter has outlined the proposed evaluation initiative the committee believes should be the core of the effort to improve USAID’s ability to assess the effectiveness of its projects in the future. The committee’s recommendations for high-level leadership would support day-to-day implementation of the initiative and provide a central focus. One of the frequent comments that the field teams heard from DG officers was the desire for advice and assistance in understanding and developing impact evaluations, and this is a role SORA could readily play. It would also be a logical starting point if the recommendations for a wider effort to restore USAID’s evaluation capacity were implemented (NRC 2006:90SORA could also be given responsibility for developing the social sciences advisory group and the broader partnerships with universities 0 IMPROVING DEMOCRACY ASSISTANCE that the committee recommends. These could both contribute to the work of the evaluation initiative and support learning from retrospective case studies.
Active Learning While it will take time for the results of the evaluation initiative to mount and provide evidence for the positive or negative impact of various USAID DG projects, USAID can and should take advantage of other avenues to learn about DG assistance. The case studies and other analyses recommended in this report would be an essential part of this effort, as would regular opportunities to discuss DG officers’ experiences and academic research on democratization. Active organizational learning means much more than simply having such research materials available for DG staff to peruse or view on the Web. As discussed in Chapter 8, it means having DG staff actively engaged with such materials through discussions and meetings with the authors of such research, probing to seek the lessons contained in the research. The continuing pilot effort for the “Voices from the Field” project discussed in Chapter 8 could over time become a key instrument in acquiring and disseminating insights from active practitioners as another element in this commitment to learning.
The committee thus recommends that part of the agenda for the Office of Democracy and Governance and the final part of the DG evaluation initiative should be a provision for active learning through regular meetings of DG staff with academics, NGOs, and think tank researchers who are exploring such issues as trajectories of democracy, the progress of democracy in various regions or nations, and the reception of DG programs in various settings. These need not all be in Washington but could include meetings in the field focused on regional issues or certain types of DG programs (e.g., having a conference in Africa on anticorruption programs that draws in regional DG staff). The planning for such meetings could involve partnerships with academics, think tanks, local partners, or other DG assistance donors.
Taken together and supported by the leadership of USAID, the SORA program and the wider efforts of the DG office and USAID that are more broadly discussed throughout this report would provide USAID with the capacity to effectively evaluate and continuously improve its work to support democratic development.
uneasy relationship with Congress and uncertainty regarding its evolving relationship with the other parts of the Executive Branch.
Across the world, and across the U.S. government, there are efforts to improve results, accountability, and organizational knowledge of foreign assistance. The committee hopes that the efforts of SORA and the recommendations in this report will form part of this broad movement to reform foreign aid.
However, such improvement will only come with a commitment to learning what works and what does not, in a spirit that avoids blame and offers credit for learning that advances the effectiveness of aid. Military and medical institutions have learned that simply punishing failures leads to efforts to hide or cover up problems and thus to those problems being prolonged. Greater progress toward the overall goals is obtained when people are encouraged to report unintended problems or setbacks and are not penalized for them. Congress and the Executive Brach must take a position on foreign aid that learning of a program’s ineffectiveness, although it may lead to ending that particular program, will not be used to undermine foreign aid in general or those who worked on that program. Indeed, given the currently uncertain knowledge and difficult challenge of advancing democracy in diverse conditions, learning that half or two-thirds of USAID’s DG programs have real and significant effects in helping countries advance should be seen as fundamentally positive and evidence of success, while learning which half or one-third of programs are not effective should be seen as an important step in advancing the targeting and effectiveness of democracy assistance. Unrealistic expectations for universal success or rapid advances, given USAID’s modest budgets for DG assistance and the complexities and many countervailing forces that prevail in the real world of democracy assistance, will not help the necessary learning—which will involve some incremental advances and some cases of learning from setbacks—that would lead to meaningful advances in the field of foreign assistance.
Congress, of course, is ultimately responsible for seeing that the public’s money is used wisely, and it should be helped to understand that rigorous impact evaluations are an important tool in seeking that end. But more than that, the committee hopes for a renewed partnership between USAID and other branches of the federal government. Congress and Executive Branch policymakers should recognize that USAID DG programs cannot be held responsible for the successes or failures of democratic development in any given country. Even U.S. foreign policy as a whole with all of its instruments, of which USAID DG assistance is only a small part, may be unable to have a substantial impact. In turn, USAID should be held accountable for determining the success or failure of the DG projects it undertakes and for making a systematic effort to document and learn from what works and what does not. USAID should not fear IMPROVING DEMOCRACY ASSISTANCE this process; repeated studies have now shown that, overall, democracy assistance is effectie (Finkel et al 2007, 2008). What needs to be done next to improve such assistance is to learn more about which specific projects are being most effective and in what contexts. This simply cannot be done accurately without a strong commitment in both Congress and USAID to making sound impact evaluations a significant part of the agency’s overall M&E and learning activities.