«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
Multipronged approaches to evaluation are being employed by the German Agency for Technical Cooperation (Deutsche Gesellschaft für Technische Zusammenarbeit, GTZ). The range of instruments currently being employed is based on elements of self-evaluation as well as independent and external evaluations. Evaluations aim to address questions of relevance, effectiveness, impact, efficiency, and sustainability.3 These questions are addressed throughout the project’s life span as a means of better understanding the links between inputs and outcome. Commitment by the GTZ to evaluations is demonstrated by the agency’s increased spending on these activities, spending “roughly 1.2 percent of its public benefit turnover on worldwide evaluations—some EUR 9 million a year” (Schmid 2007).
The Swedish Agency for International Development Cooperation (SIDA) is also actively considering ways to improve its evaluation tools.
Since 2005, SIDA has shifted from post-hoc project evaluations to a focus on underlying assumptions and theories; specifically, SIDA is currently conducting a project that “looks at the program theory of a number of different projects in the area. This evaluation focuses on the theoretical constructs that underpin these projects and tries to discern patterns of 3 For further information, see “Working on Sustainable Results: Evaluation at GTZ.” Avail
ideas and assumptions that recur across projects and contexts.” 4 Building on these initial efforts, SIDA hopes to combine the results of this study with others to “make an overall assessment of the field.” The Norwegian Agency for Development Cooperation (NORAD) has also initiated a new strategy for evaluating the effectiveness of its programs in the area of development assistance. The intent of this new strategy, undertaken in 2006, is to “help Norwegian aid administrators learn from experience by systematizing knowledge, whether it is developed by (themselves), in conjunction with others, or entirely by others. Additionally, the evaluation work has a control function to assess the quality of the development cooperation and determine whether resources applied are commensurate with results achieved.”5 Additional attention is being paid to communicating the results of such evaluations with other agencies and stakeholders; this emphasis on communicating results is widely shared in the donor community.
The Danish Ministry of Foreign Affairs has embarked on an extensive study of both its own and multilateral agencies’ evaluations of development and democracy assistance (Danish Ministry of Foreign Affairs 2005).
It has found that evaluations vary greatly in method and value, with many evaluations failing to provide unambiguous determinations of program results. In regard to the United Nations Development Program’s central evaluation office, “its potential for helping strengthen accountability and performance assessment is being underexploited, both for the purpose of accountability and as an essential basis for learning” (Danish Ministry of Foreign Affairs 2005:4).
Finally, the Canadian International Development Agency (CIDA) has been involved in recent efforts to improve evaluation and learning from collective experiences at international assistance in the area of democracy and governance.
In April 1996, as part of its commitment to becoming more resultsoriented, CIDA’s President issued the “Results-Based Management in CIDA—Policy Statement.” This statement consolidated the agency’s experience in implementing Results-Based Management (RBM) and established some of the key terms, basic concepts and implementation principles. It has since served as the basis for the development of a variety of management tools, frameworks, and training programs. The Agency Accountability Framework, approved in July 1998, is another 4 For more information on this project, see SIDA, “Sida’s Work with Democracy and Human Rights.” Available at: http://www.sida.se/sida/jsp/sida.jsp?d=0&a=0&language=en _US. Accessed on September 12, 2007.
5 For more information, see NORAD’s Web site: http://www.norad.no/default.asp?V_ITEM_
key component of the results-based management approach practiced in CIDA. (CIDA 2007) The CIDA report makes an important distinction, however: “The framework articulates CIDA’s accountabilities in terms of developmental results and operational results at the overall agency level, as well as for its various development initiatives. This distinction is crucial... since the former is defined in terms of actual changes achieved in human development through CIDA’s development initiatives, while the latter represents the administration and management of allocated resources (organisational, human, intellectual, physical/material, etc.) aimed at achieving development results.” In short, there is growing agreement—across think tanks, blue-ribbon panels, donor agencies, and foreign ministries—that current evaluation practices in the area of foreign assistance in general, and of democracy assistance in particular, are inadequate to guide policy and that substantial efforts are needed to improve the knowledge base for policy planning.
Thus, USAID is not alone in struggling with these issues.
CURRENT POLICy AND LEgAL FRAMEWORk FOR
USAID Dg ASSESSMENTS AND EvALUATIONSCurrent DG policies regarding project assessment and evaluation are shaped in large part by broader USAID and U.S. government policies and regulations. Official USAID polices and procedures are set forth in the Automated Directives System (ADS) on its Web site; Series 200 on “Programming Policy” covers monitoring and evaluation in Section 203 on “Assessing and Learning” (USAID ADS 2007). Of particular importance for this report, in 1995 the USAID leadership decided to eliminate the requirement of a formal evaluation for every major project; instead evaluations would be “driven primarily by management need” (Clapp-Wincek and Blue 2001:1). The prior practice of conducting mainly post-hoc evaluations (which were almost entirely process evaluations), often done by teams of consultants brought in specifically for the task, was seen as too expensive and time consuming to be applied to every project.
As a result of the change, the number of evaluations for all types of USAID assistance, not just DG, has declined, and the approach to evaluation has evolved over time (Clapp-Wincek and Blue 2001). ADS 184.108.40.206 (“When Is an Evaluation Appropriate?”) lists a number of situations that
should require an evaluation:
tive or negative) that should be explained (such as gender differential results);
• Customer, partner, or other informed feedback suggests that there are implementation problems, unmet needs, or unintended consequences or impacts;
• Issues of sustainability, cost effectiveness, or relevance arise;
• The validity of Results Framework hypotheses or critical assumptions is questioned (e.g., due to unanticipated changes in the host country environment);
• Periodic Portfolio Reviews have identified key questions that need to be answered or that need consensus; or
• Extracting lessons is important for the benefit of other Operating Units or future programming (USAID ADS 2007:24).
These evaluations generally remain the traditional process evaluations using teams of outside experts undertaken while a project is under way or after it has been completed.6 The second significant policy shaping USAID evaluation practices is the Government Performance and Results Act (GPRA) of 1993. GPRA “establishes three types of ongoing planning, evaluation, and reporting requirements for executive branch agencies: strategic plans..., annual performance plans, and annual reports on program performance. In complying with GPRA, agencies must set goals, devise performance measures, and then assess results achieved” (McMurtry 2005:1). GPRA has led to the development of an elaborate performance monitoring system across the federal government. Performance monitoring is different from
evaluation; as defined by USAID, for example:
Performance monitoring systems track and alert management as to whether actual results are being achieved as planned. They are built around a hierarchy of objectives logically linking USAID activities and resources to intermediate results and strategic objectives through causeand-effect relationships. For each objective, one or more indicators are selected to measure performance against explicit targets (planned results to be achieved by specific dates). Performance monitoring is an ongoing, routine effort requiring data gathering, analysis, and reporting on results at periodic intervals.
Evaluations are systematic analytical efforts that are planned and conducted in response to specific management questions about performance of USAID-funded development assistance programs or activities. Unlike
performance monitoring, which is ongoing, evaluations are occasional— conducted when needed. Evaluations often focus on why results are or are not being achieved. Or they may address issues such as relevance, effectiveness, efficiency, impact, or sustainability. Often, evaluations provide management with lessons and recommendations for adjustments in program strategies or activities. (USAID 1997:1) To implement the system required by GPRA, every USAID operating unit (missions overseas, bureaus or offices in Washington) must develop strategic objectives (SOs). The DG office created a process for strategic assessments that is often used to inform the development of mission strategies (USAID 2000). Typically, a team of experts, which may include a mix of contractors and USAID personnel, spends several weeks evaluating current conditions in a country with respect to key aspects of democracy and governance and analyzing the opportunities for intervention and impact. This assessment is not quite keyed to the four elements in USAID’s DG goals described in Chapter 1, however. Rather, strategic assessments deal with five areas: consensus, rule of law, competition, inclusion, and good governance. After surveying the degree to which the country has these elements, the assessment considers the key actors and institutions whose behavior or condition needs to change to improve democratic development and then suggests policies—with explicit attention to feasibility given the resources of USAID in that country and country conditions—to promote advances in some or all of these areas. Not every country is assessed and some country assessments may be updated if conditions change enough to warrant a reexamination. Since the formal assessment tool was adopted in late 2000, more than 70 assessments have been conducted in 59 countries.7 To achieve their strategic objectives, all USAID operating units develop a Results Framework and a Program Monitoring Plan that include subobjectives that tie more closely to specific projects (see Figure 2-1 for an illustrative results framework). Depending on the size of its budget and other factors, a mission might have anywhere between one and a dozen SOs, of which one or perhaps two will relate to democracy and governance.
Indicators are used to track progress from the project level through intermediate objectives up to the SO. Missions are required to report their performance against these indicators annually, but below the SO level they can choose which indicators to report and can change the indicators they report each year. Generally, each contract or grant must have an
approved performance monitoring plan, which includes both targets and the indicators that will be used to determine whether the project meets its objectives (USAID ADS 2007). Some implementers also develop and track additional indicators, usually to provide further evidence of achieving project goals.