«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
For the majority of USAID Dg projects, however, the goal should be more modest. Where USAID mission directors request them, the initiative should provide support and advice to help the missions request and oversee good-quality impact evaluations that pay attention to all three elements of good evaluation practices: a focus on outcomes, good baseline measurements, and comparison with untreated groups. Evaluations should include pre- and postintervention outcome measures, along with, where possible, an analysis of outcomes in a relevant control group. As Chapters 5, 6, and 7 demonstrated, a wide variety of evaluation designs aside from randomized assignments are available to help USAID accumulate systematic evidence of the efficacy of particular approaches in order to guide its decision making as new investments are planned.
IMPROVING DEMOCRACY ASSISTANCE To assist in the effort, the committee recommends that the USAID administrator consider establishing a social sciences advisory group for the agency. This group could play a useful role in advising on the design of the evaluation initiative, helping work through issues that arise during implementation, and developing a peer review process for assessing the evaluations undertaken during the initiative.
Resources The five-year Dg evaluation initiative should be supported with special, dedicated resources outside the usual project structure. Supporting the initiative with special funds would be another signal of the strong commitment to change. The committee is not able to provide an estimate of the likely cost of the initiative, in part because the difficulties in obtaining estimates of what USAID currently spends on M&E provide no basis for comparison. Some of the essential components are discussed below to provide a rough basis for making an estimate. But the important point is that the funds not come out of current mission program budgets that are already stretched thin.
It is also important that the resources be used to support both the special impact evaluations chosen as the chief task of the DG evaluation initiative and efforts by country missions to improve their evaluations or conduct their own impact evaluations on chosen projects. The initiative should thus make its resources and expertise available to mission directors who want its support in conducting impact evaluations or otherwise changing their mix of M&E activities, in order to make the initiative an asset to the Dg officers in the field rather than an additional unfunded burden.
Capacity One of the biggest challenges facing the initiative relates to capacity.
Over the past four decades, the structure of USAID has been transformed, moving away from an in-house professional staff of development experts with a significant and substantive role in projects toward an arrangement in which development projects are prioritized, solicited, approved, and overseen by USAID officers, but projects are largely designed, carried out, and evaluated by outside implementers (NRC 2006). This shift has led to an increasing focus on time-consuming issues of grant and contract management rather than project design and evaluation. This long-term shift has taken place in parallel with the more recent changes in agency policy described in Chapter 2 toward an increased emphasis on project monitoring and the use of evaluations to respond periodically to management needs, rather than systematically assess project impacts.
AN EVALUATION INITIATIVE TO SUPPORT LEARNINGOne consequence of these changes in policy and in the responsibilities of USAID staff has been the erosion of the skill base and expertise required to design and oversee impact evaluations for a variety of programs and contexts. The DG officers the committee encountered were experienced and knowledgeable in substantive matters, but even if they had training in general social sciences research methods, they rarely had training or experience with impact evaluation design. The evaluation capacity of USAID’s DG programs, like other capabilities, has thus increasingly shifted to the implementers who design and carry out the projects. Although the committee found in its own field visits that DG officers were, in general, quite willing to work with the committee’s consultants who were evaluation experts and that the DG officers were open to considering new approaches to testing the efficacy of their programs, few of the officers thought they were capable of judging and overseeing varied impact evaluation designs without additional assistance and resources.
The expertise needed among USAID professionals and, in particular, DG officers to support the initiative deserves particular attention. USAID’s past hiring in the DG field has stressed bringing in individuals with practical or theoretical training in democratic processes and institutions.
This will continue to be the main area for DG expertise, but it is clearly distinct from, and not sufficient for, providing expertise in the full range of project evaluation strategies. The World Bank, health care agencies, and other foreign assistance organizations regularly hire Ph.D.-level researchers whose advanced training focused on carrying out experimental and statistical evaluation analyses to support their subject matter experts. To increase its in-house capacity to support improved evaluations, USAID will need to hire more individuals with Ph.D.s in the social sciences whose training was strong in techniques of experimental and statistical analysis that can be applied to DG projects. The committee recommends that USAID acquire sufficient internal expertise in this area to both guide an initiative at USAID headquarters and provide advice and support to field missions as a key element of the initiative.
The DG office, like other parts of USAID, has made use of short-term appointments to augment its expertise. In the committee’s judgment, however, if the recommended evaluation initiative is accepted, the practice of having an occasional Ph.D.-trained experimental analyst as a fellow in the DG office can be helpful but will probably not be sufficient. As discussed further below, valuable assistance could be provided by outside experts through USAID’s various contracting mechanisms, but the leadership and confidence that come with in-house knowledge will be important to the success of the proposed initiative.
For many years the lack of staff capacity was offset by an active agency-wide centralized evaluation office (as in most bilateral and multiIMPROVING DEMOCRACY ASSISTANCE lateral development agencies)—the Center for Development Information and Evaluation (CDIE). The DG office in particular was the subject of many detailed CDIE evaluations, including substantial comparative studies of DG projects (see, e.g., Blair and Hanson 1994, de Zeeuw and Kumar 2006). As already discussed, these were generally process evaluations and not formal impact evaluations, but they did provide systematic research intended to gather lessons and compare experiences. With the increased emphasis on project monitoring, however, CDIE had grown gradually weaker in recent years and was recently absorbed into the office of the new director of foreign assistance in the State Department.
Whether or not an independent central evaluation office should be restored is beyond this committee’s charge, but the committee believes the capacity of USAID headquarters to provide significant resources and expertise to assist DG officers in the field (and perhaps other USAID programs as well) who wish to develop impact evaluations of their programs would be a valuable augmentation of USAID’s in-house resources.
Partnerships to Add Capacity from Outside USAID While the committee believes that a substantial augmentation of USAID’s internal capacity for evaluation design is necessary for the proposed evaluation initiative to be effective, there is no reason that USAID’s efforts to improve evaluation must be purely an in-house affair. The need for supplemental outside capacity is particularly acute with regard to impact evaluations and broad-based learning. There is no need to keep on staff sufficient experts on evaluation design to provide all the assistance requested by country missions in that regard, if USAID can find other means to deliver the required technical support to field staff at critical moments of project design, implementation, and evaluation. And many of USAID’s organizational learning activities can and should be enriched by partnerships with academic institutions and think tanks exploring similar issues.
USAID has a number of options through its current contracting mechanisms to acquire this support. The committee’s discussions in Washington and during its field visits suggest that a significant number of implementers already have or could readily add the necessary expertise in impact evaluations; the problem has been a lack of demand for impact evaluations as parts of calls for proposals, rather than a lack of capacity among implementers.3 As discussed earlier, the committee believes that 3 Local grantees, such as NGOs, pose a different problem. Although it was found in the
it is important to maintain independence between those implementing a program and those responsible for its evaluation, but this could be achieved in a number of ways.
Universities also offer a major source of expertise related to highquality impact evaluations. Many university-based scholars already serve as consultants to USAID implementers on a range of DG issues. Increasingly, scholars are also partnering directly with international development agencies and NGOs to design and undertake systematic program evaluations. Mechanisms such as the Democracy Fellows program allow USAID to bring scholars onto its staff for short-term appointments.
Moreover, there is ample precedent in USAID for drawing on the expertise and resources of universities rather than individual scholars.
Over several decades USAID established itself as a pioneer in research leading to development in the field of agriculture. The agency accomplished this through a wide array of partnerships (usually constructed in the form of “cooperative agreements”) with U.S. land grant colleges and universities. These were institutions that had long been carrying out the research needed to achieve better agricultural outcomes. Land grant officials were accustomed to working with state agricultural extension services, for example, providing them with technical support to detect, diagnose, and cure outbreaks of diseases and infestations threatening crops and livestock. The research was not limited to agricultural production itself but dealt with a wide range of issues, including rural credit, in which Ohio State University played a key role, or land tenure, in which the Land Tenure Center at the University of Wisconsin became the world leader. Those partnerships expanded beyond the borders of the United States into international networks of research centers dedicated to agricultural research and extension. A prime illustration is Zamorano, in Honduras, but there are many others.
When USAID embarked on democracy programs as a major effort distinct from its other programs, it did not make a comparable investment in basic research partnerships with universities to provide additional knowledge and intellectual capacity. In most cases the focus was and remains on doing democracy rather than studying how to do democracy.
There were and are important exceptions, and in addition some universities are major implementers of USAID DG programs, such as SUNY Albany’s long-term efforts at legislative strengthening, or the work of the IRIS Center at the University of Maryland on issues related to economic development and governance.4 Although not necessary for the initial DG evaluation initiative, for the longer term USAID might consider investing resources to develop a 4 Further information about the IRIS program may be found at http://www.iris.umd.edu/ and