FREE ELECTRONIC LIBRARY - Books, dissertations, abstract

Pages:     | 1 |   ...   | 33 | 34 || 36 | 37 |   ...   | 62 |

«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»

-- [ Page 35 ] --

Senge, P. 2006. The Fifth Discipline: The Art and Practice of the Learning Organization, Rev. Ed.

New York: Doubleday.

Skocpol, T. 2003. Diminished Democracy: From Membership to Management in American Civic Life. Julian T. Rothbaum Lecture Series, Vol. 8. Norman: University of Oklahoma Press.

Shadish, W.R., Cook, T.D., and Campbell, D.T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.

Trochim, W., and Donnelly, J.P. 2007. The Research Methods Knowledge Base, 3rd ed. Cincinnati, OH: Atomic Dog Publishing.

Vermeersch, C., and Kremer, M. 2004. School Meals, Educational Achievement, and School Competition: Evidence from a Randomized Evaluation. World Bank Policy Research Working Paper No. 3523. Washington, DC: The World Bank.

Wholey, J.S., Hatry, H.P., and Newcomer, K.E., eds. 2004. Handbook of Practical Program Evaluation, 2nd ed. San Francisco: Jossey-Bass.

Wilson, E.O. 1998. Consilience: The Unity of Knowledge. New York: Knopf.

World Bank. 2004. Monitoring & Ealuation: Some Tools, Methods & Approaches. Washington, DC: International Bank for Reconstruction, The World Bank.

Implementing Impact Evaluations in the Field


A counterfactual question—how would things have looked in the absence of the U.S. Agency for International Development (USAID) program?—lies at the core of any design for impact evaluations. Chapter 5 made the case that randomized evaluations provide the soundest methodology for generating definitive answers to this question. However, it is one thing to specify what may be optimal theoretically and another thing altogether to implement that methodology on the ground. Practical impediments may make the implementation of randomized evaluation difficult, even impossible, at least in a pure form. For example, factors outside of USAID’s control may render it not feasible to gather baseline data, to identify and monitor outcomes in a control group, or to select by lottery the units in which programs should be implemented. Although Chapter 5 provided examples of several successful randomized evaluations, only a handful of these are in the democracy and governance (DG) area, and none of them are examples of evaluations of USAID’s own programs. Thus, even if willing to accept the desirability in principle of adopting the methodology of randomized evaluation, it is reasonable to wonder how readily it can be applied to the sorts of programs that USAID missions in the field regularly undertake.

To find out, the committee commissioned three expert teams to visit USAID missions overseas in an effort to assess the viability of impact evaluations for past and present DG programming. The key task for each team was to talk with implementers, local partners, and USAID mission   IMPROVING DEMOCRACY ASSISTANCE personnel on the ground to assess the feasibility of actually implementing in practice the evaluation methodologies outlined in the previous chapter.

The first part of this chapter presents the results of those field visits. The second part provides responses to the most commonly raised objections that the committee and its field teams heard expressed about the use of randomized evaluations in DG programs.

Before turning to the details of what the field teams found, it is important to highlight a clear and consistent message that came through from all three field visits. All three teams concluded, first, that the introduction of randomized ealuations into USAID project ealuation was both feasible and cost-effectie in many of the contexts they inestigated. They were unanimous that, where possible, adopting such methods would represent an improvement over current practices. Second, they reported that, for projects where randomized ealuations were not possible, other improements to USAID ealuation—for example, improed measurement, systematic collection of baseline data, and comparisons across treated and untreated units—also hae the potential to yield significant improements in the agency’s ability to attribute project impact.

These issues are discussed in Chapter 7. Finally, the teams returned from the field energized by their interactions with mission staff and confident that a willingness, and even excitement, exists about improving the quality of project evaluations. The teams were also impressed with some of the work already being done as part of current project monitoring, in particular in the broadening of measurement strategies beyond project outputs to include an assessment of outcomes.


As a complement to the deliberations in Washington and extensive engagement with USAID staff and implementers, the committee felt strongly that its recommendations should be informed by a set of extended field visits to USAID missions. The committee therefore identified a set of missions, representing a diversity of regions, that were engaged in substantial programming on DG issues and were in the process of designing large, new projects in one of USAID’s core DG areas (rule of law, elections and political processes, civil society, and governance). From the list of missions provided, USAID explored the willingness of the missions to host the team and consider new approaches to project evaluation. After negotiating issues of timing and access, USAID and the committee agreed to send field teams to Albania, Peru, and Uganda. The field visits were

intended to accomplish three main goals:

–  –  –

2. to explore the feasibility of introducing impact evaluations in the future, including (but not limited to) randomized evaluation; and

3. to obtain the perspectives of mission personnel and USAID implementers regarding the possibilities for, and impediments to, new approaches to evaluation.

The committee encouraged the field teams to explore the range of DG activities currently under way in each mission, assess the adequacy of current evaluation approaches, and provide concrete examples of how existing approaches could be improved. In addition, the field teams were directed to focus particular attention on the development of an impact evaluation design in one specific area in each mission. The teams focused on local government/decentralization in Albania and Peru and support for multiparty democracy in Uganda.1 Each field team was composed of methodological consultants, academic or other experts with relevant experience in research design or program evaluation and DG issues, and country or regional expertise;

a Washington-based USAID staff member who was familiar with the mission, the committee’s work, and USAID policies and practices; and National Research Council professional staff, who assisted the consultants in meeting the team’s objectives and coordinated the logistics of the field visits.

In evaluating the findings of the three field teams, it is important to keep in mind that the field teams visited missions that had expressed an interest in improving their evaluation strategies. The field teams’ conclusions about the applicability of impact evaluations, especially its sense that standard objections to these designs can be addressed, thus reflect the experiences gleaned from this (nonrandom) sample of missions. It is not known if other missions, especially smaller ones with leaner budgets or those in countries experiencing violent conflicts or particularly rapid

political change, would be as amenable to new approaches to evaluation:

The committee has no control group of non-self-selecting missions with which to compare its findings. Yet the committee believes it unlikely that missions that did not invite the committee to send a field team would have offered novel additional objections. Over the 15 months of the study period, the committee talked with numerous USAID staff and implementers from a variety of areas and with backgrounds and experience with DG programming in a great many countries, and the set of objections that are taken up in the second part of this chapter dominated the responses of everyone with whom the committee spoke.

1 Key results of the field visits are discussed in this chapter and the next. Additional infor

–  –  –



Randomized evaluations are widely considered the best method for determining the causal effects of treatment in a broad range of areas, including public health, education, microfinance, and agriculture. As the Olken (2007) and Gugerty and Kremer (2006) studies described in Chapter 5 show, such methodologies are also beginning to be applied to evaluate the effectiveness of projects in the area of democratic governance.

Nonetheless, the committee learned from its consultations with USAID staff and implementers that there is a general feeling that randomized evaluation was not an option for many of the projects that USAID carries out. Even in those cases where randomized evaluations might be possible theoretically, the assumption among USAID staff seemed to be that such approaches would be too difficult to implement in practice, owing to an inability to select treatment groups by lottery, the difficulty of preserving a control group, the difficulty of identifying good indicators for key outcomes, the high cost of the extensive data collection that would be required, or the tension between the flexibility staff believe they need to respond to opportunities and challenges as projects go forward and the need to minimize changes to ensure an effective evaluation.

These are legitimate concerns. To address them, this section discusses how randomized evaluations could be used in current USAID projects, drawing on examples gleaned from the field visits and consultations with practitioners. We begin with a decentralization project in Peru that has already been implemented, outlining how the project monitoring strategy that was employed could have been adjusted to accommodate a randomized component that would have made it an impact evaluation design and showing how such an adjustment would have permitted the mission to generate much stronger inferences about project impact.2 Then a planned multipronged effort to support multiparty democracy in Uganda is described, emphasizing how pieces of the existing project might be amenable to randomized evaluation and showing how adopting such an evaluation method would improve USAID’s ability to assess the project’s effects.3 The committee’s goal is to use these projects as illustrations of the potential payoffs that could accrue from improved evaluation strategies.

2 The discussion here of decentralization in Peru is drawn from the report of a field team led by Thad Dunning, assistant professor of political science, Yale University.

3 These designs were developed by a team led by Devra Moehler, assistant professor of

–  –  –

Decentralization in Peru USAID/Peru launched a project in 2002 to support national decentralization policies initiated by the Peruvian government. Over a five-year

period, the Pro-Decentralization (PRODES) program was intended to:

• support the implementation of mechanisms for citizen participation with subnational governments (such as “participatory budgeting”),

• strengthen the management skills of subnational governments in selected regions of Peru, and

• increase the capacity of nongovernmental organizations in these same regions to interact with their local government (USAID/Peru 2002).

Pages:     | 1 |   ...   | 33 | 34 || 36 | 37 |   ...   | 62 |

Similar works:

«Aus dem Institut für Pathologie der Tierärztlichen Hochschule Hannover und dem Zentrum Anatomie der Medizinischen Hochschule Hannover Zur Pathogenese der akuten Lymphangitis und Elephantiasis beim Pferd -Eine histologische, immunhistochemische und transmissionselektronenmikroskopische StudieINAUGURAL DISSERTATION zur Erlangung des Grades eines DOKTORS DER VETERINÄRMEDIZIN (Dr. med. vet.) durch die Tierärztliche Hochschule Hannover vorgelegt von Marcus Risse aus Hagen am Teutoburger Wald...»

«Aus der Klinik für Innere Medizin, Schwerpunkt Pneumologie Leitung: Prof. Dr. C. Vogelmeier des Fachbereichs Medizin der Philipps-Universität Marburg in Zusammenarbeit mit dem Universitätsklinikum Gießen und Marburg GmbH, Standort Marburg Lungengeräuschanalyse bei herzinsuffizienten Patienten Eine Pilotstudie zur Detektion von Rasselgeräuschen mittels elektronischer Auskultation Inaugural-Dissertation zur Erlangung des Doktorgrades der gesamten Humanmedizin dem Fachbereich Medizin der...»

«Frühes menschliches Leben zwischen Identität und Interessen Jeff McMahan, The Ethics of Killing. Problems at the Margins of Life, OUP 2002 Es sind vor allem drei Bereiche von Handlungskonflikten, die eine „Ethik des Tötens“, also Überlegungen zur möglichen Rechtfertigung des Tötens, erforderlich machen. In die Medizinethik fallen zwei von ihnen: Rechtfertigungen mit Bezug auf den zweifelhaften moralischen Status menschlicher Lebewesen, wie vor allem am Beginn des Lebens, und...»

«Prof. Urs Granacher, PhD February 2015 University of Potsdam Faculty of Human Sciences Research Focus Cognition Sciences Head Division of Training and Movement Sciences Am Neuen Palais 10 (Building 12) 14469 Potsdam Germany Phone: +49-(0)331 977 1543; Fax: +49-(0)331 977 4022 E-mail: urs.granacher@uni-potsdam.de Complete publication record (sorted by reviews, original work, book chapters, and books) Reviews 2015 Granacher, U. (2015). WVL-Projekt Krafttraining im Nachwuchsleistungssport...»

«National Patient Information Reporting System: National Data Warehouse NDW Workload/User Population Data Mart Technical Guide Version 1.2 February 2008 Department of Health and Human Services Indian Health Service Office of Information Technology (OIT) Indian Health Service National Patient Information Reporting System Office of Information Technology National Data Warehouse Contents Version Control Requirements Overview of the User Population/Workload Process Design Parameters Data Parameters...»

«• ( RADIOAKTIVE ISOTOPE IN KLINIK UND FORSCHUNG 15. Band 2. Tell Gasteiner Internationales Symposium Herausgegeben von Rudolf Hofer Helmar Bergmann e Verlag H. Egermann ••'•V •1f*ff'-' • -oT • • * • • • • ISBN 3-900287-14-7 Radioaktive Isotope in Klinik und Forschung veranstaltet von der Nuclearmedizinischen Abteilung der II. Medizinischen Universitatsklinik, Leiter: Prof. Dr. R. Hofer, A-1090 Wien, Garnisongasse 13 Copyright by: Verlag H. Egermann Hernalser HauptstraBe...»

«HEALTH CARE COMMERCIALISATION AND THE EMBEDDING OF INEQUALITY RUIG / UNRISD HEALTH PROJECT SYNTHESIS PAPER Maureen Mackintosh Draft paper prepared for the RUIG/UNRISD project on Globalization, Inequality and Health, a collaborative international project forming part of the RUIG research programme on The Social Challenge of Development September 2003 Acknowledgements Funding for the research programme on the Social Challenge of Development, from the Réseau Universitaire International de...»

«Marijuana and Medicine Assessing the Science Base Janet E. Joy, Stanley J. Watson, Jr., and John A. Benson, Jr., Editors Division of Neuroscience and Behavioral Health INSTITUTE OF MEDICINE NATIONAL ACADEMY PRESS Washington, D.C. Executive Summary Public opinion on the medical value of marijuana has been sharply divided. Some dismiss medical marijuana as a hoax that exploits our natural compassion for the sick; others claim it is a uniquely soothing medicine that has been withheld from patients...»

«Titel: Cannabis als Medikament eine Nutzen/Risiko-Abwägung Cannabis as prescription drug (medicament) – a profit/risk consideration Untertitel: Stellungnahme im Auftrag der Deutschen Gesellschaft für Psychiatrie, Psychotherapie und Nervenheilkunde (DGPPN) auf Anforderung des Bundesministerium für Gesundheit und soziale Sicherung (BGMS) vom 26. September 2003 zu medizinisch-wissenschaftlichen Erkenntnissen in Hinsicht auf die Wirksamkeit und Nutzen/Risiko-Abwägung von Dronabinol und...»

«Internees (Switzerland) By Thomas Bürgisser Swiss governmental and non-governmental organizations were intensely involved in humanitarian activities for the victims of World War I. From 1916 on, the country opened its borders to a select number of sick or injured prisoners of war from Germany, France, Belgium and Britain for healing and recovery. Based on agreements with the belligerent nations – who paid the costs – the prisoners were interned for several months in sanatoria and hotels...»

<<  HOME   |    CONTACTS
2016 www.book.dislib.info - Free e-library - Books, dissertations, abstract

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.