«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
Reich, G. 2002. Categorizing Political Regimes: New Data for Old Problems. Democratization 9(4):1-24.
Schaffer, F.C. 1998. Democracy in Translation: Understanding Politics in an Unfamiliar Culture.
Ithaca, NY: Cornell University Press.
Schumpeter, J.A. 1942. Socialism and Democracy. New York: Harper.
Tetlock, P. 2005. Expert Political Judgment: How Good Is It? How Can We Know? Princeton, NJ:
Princeton University Press.
Thomas, M.A. 2007. What Do the Worldwide Governance Indicators Measure? Unpublished manuscript, School of Advanced International Studies, Johns Hopkins University.
IMPROVING DEMOCRACY ASSISTANCE Transparency International. 2007. Corruption Perceptions Index. Available at: http://www.
transparency.org/policy_research/sureys_indices/cpi. Accessed on September 5, 2007.
Treier, S., and Jackman, S. 2003. Democracy as a Latent Variable. Paper presented at the Political Methodology meetings, University of Minnesota, Minneapolis-St. Paul.
USAID (U.S. Agency for International Development). 1998. Handbook of Democracy and Goernance Program Indicators. Washington, DC: Center for Democracy and Governance.
Available at: http://www.usaid.go/our_work/democracy_and_goenance/publications/pdfs/ pnacc0.pdf. Accessed on August 1, 2007.
Vanhanen, T. 2000. A New Dataset for Measuring Democracy, 1810-1998. Journal of Peace Research 37:251-265.
Vermillion, J. 2006. Problems in the Measurement of Democracy. Democracy at Large 3(1):
Learning from the Past: Using Case Studies of Democratic Transitions to Inform Democracy Assistance
INTRODUCTIONThe U.S. Agency for International Development (USAID) asked the National Research Council (NRC) to recommend methodologies to carry out retrospective analyses of democracy assistance programs. The recommendations were to include “a plan for cross-national case-study research to determine program effectiveness and inform strategic planning.” There is a substantial and growing literature of case studies of democracy assistance programs, many of them commissioned by USAID or other agencies engaged in democracy assistance. The goal of such case studies is to learn what has worked and what has not among the varied democracy and governance (DG) programs in a variety of places.
The vast majority of such studies focus on a particular program in a particular country, such as human rights in Cambodia (Asia Watch 2002), party organization in Uganda (Barya et al 2004), voter education in Ethiopia (McMahon et al 2004), or justice reform in Sierra Leone (Dougherty 2004).
In addition, there have been more ambitious works that looked at multiple countries to try to draw broader lessons about program impacts.
For example, Abbink and Hesseling (2000) bring together several studies of election observation and democratization in Africa; Lippman and Emmert (1997) study legislative assistance in five countries; Blair and Hansen (1994) assess the impact of rule of law programs in six countries;
Kumar (1998) examines the impact of elections in several postconflict conditions; O’Neill (2003) presents lessons from human rights promotion in 00 IMPROVING DEMOCRACY ASSISTANCE varied regions; Carter et al (2003) study the overall impact of USAID DG programs in six countries; and de Zeeuw and Kumar (2006) look at media, human rights, and election programs in nine postconflict states.
While these studies have generated valuable insights into how programs were carried out, how they were received, and how participants and donors perceived their effects, they are not ideal either for “determining program effectiveness” or to “inform strategic planning.” This is because such studies focused almost entirely on specific DG projects, rather than on the broader context of democratization in the countries being studied. They did not systematically compare cases of varying levels of DG assistance or compare the effects of DG projects with comparison groups that did not receive assistance.
CASE STUDy DESIgNS AND METHODSThe basic tool of case study analysis is process tracing (George and Bennett 2005). In this method, researchers track the unfolding of strings of events, testing hypotheses regarding the causal relationships among them by considering multiple hypotheses that could account for the strings of events and searching for confirming and disconfirming evidence. The process is not unlike a detective’s efforts to solve a murder mystery by reconstructing a timeline of events, examining all possible suspects and their alibis, assessing plausible motives and opportunities for the observed actions and events, and building a case in favor of one causal chain as having determined the ultimate outcome rather than others.
Like solving any mystery, process tracing can be painstaking and time-consuming work, and the results often depend on an analyst’s skill in recognizing how specific social conditions, motivations, events, and opportunities link to form a coherent explanatory chain. Also like any criminal case, the persuasiveness of pointing out any one factor or event as causal depends on the analyst’s imagination and skill in identifying and considering alternatie causal pathways and gathering evidence as to how likely or unlikely they were.1 Case studies to demonstrate the effectiveness of aid programs thus face the same challenge as formal statistical evaluations—they must try to determine what would have happened in the absence of the aid program, whether by including studies of both groups receiving aid and those not receiving aid in their case studies (a comparative case study design) or by trying to trace and account for historical trends and confounding fac
tors to estimate the likely causal chains that would have unfolded in the absence of the aid program (a long-term historical case study design).
Yet in most case studies of democracy assistance, researchers have not used such designs. They have instead assumed that the information they needed could be found by studying the unfolding of the aid program itself. For a process-type evaluation, where the main questions asked by researchers are “Did the project achieve the goals expected by the donors?” and “Why or why not?” this is reasonable and most case studies of aid assistance have taken this form.
However, if USAID now wishes to use case studies to study the impact of DG programs on policy goals, they are not the most appropriate tool. This is because retrospective case studies can rarely obtain or reconstruct the comparable baseline and outcome information for appropriate comparison groups that is necessary for sound inference of program effects. The committee’s field studies tried to determine if missions had retained such baseline data if collected before DG projects or if they had collected any comparable baseline data for nonassisted groups. The teams had limited success with finding the former and no success in finding the latter. Thus the committee believes that for most DG programs information on project effects would most credibly be obtained by well-designed impact evaluations, rather than retrospective case studies.
However, case studies can provide information to help inform strategic planning. Comparative and historical case studies that examine varied trajectories of democratic change, and trace the relationship of DG activities to other factors and events that influence long-term democracy outcomes, can help generate hypotheses about opportunities and obstacles for DG assistance to support democratic progress.
In addition, sometimes the greatest insights regarding where and when to intervene with certain programs arise from detailed studies of program failures. One can often learn more from tracing the causes of program failure than from studies of successes, especially if such success rests on chance factors that supported a program but are not observed or reported in the study. Yet case studies of DG assistance rarely seek out failures for sustained examination—there are few rewards in the current incentive structure of donors for seeking out failures and investing in their study.2 This chapter develops guidelines for case studies that better explore the roles that democracy assistance programs may play in varied contexts of social change.
2 One exception is the scholarly work of Carothers (1999, 2004, 2006), who has investigated
INSIgHTS FROM CURRENT RESEARCH: RESULTS OF A
CONFERENCE OF CASE STUDy SPECIALISTS ON DEMOCRACyUnder the “transformational diplomacy” plan of the Bush administration and the closer supervision of USAID by the State Department, it was anticipated that USAID’s DG efforts would often be undertaken as part of broader strategies to help achieve desired outcomes in particular states (Rice 2006). Faced with such demands, USAID would like to be
able to respond to policymakers with information such as the following:
“Based on what we know about transitions to democracy in countries with conditions like that, the chances of achieving a successful transition to democracy in X years is fairly low (or high),” or “Based on what we know about the time and volume of assistance it usually takes to build and stabilize democracy in postconflict societies with these characteristics, we can give you some broad parameters regarding the expected time and financial support required to have a realistic chance of attaining that goal in country Y.” For these objectives a clustered set of case studies, tracing the processes through which advances toward democracy were made from various sets of initial conditions, is an appropriate mode of investigation. A sufficient number of case studies would help build a knowledge base to answer questions such as the following: “For most countries we have observed with initial conditions X, Y, and Z, what have been the observed trajectories of political change, and which factors A, B, C (and others) were most prominent in shaping or constraining those trajectories?” Case studies are particularly valuable in this kind of mapping exercise, where instead of trying to identify the average impact of one or more causal factors across a wide range of conditions, the goal of the investigation is to identify diverse patterns or combinations of relationships that are associated with varying pathways of change over time (Goldstone 1998, 2003).
Rather than starting out to design such a study, the committee first noted that a great deal of case study research is already being done by academics who focus on democracy and democratization. The committee decided that its first step should be to investigate that body of scholarship and see how much value it already provided for meeting USAID’s goals. The committee therefore convened a conference of leading academic experts on case study analyses of democracies and democratic transitions to help it assess the “state of the art” on how such knowledge could guide strategies for democracy assistance (see Appendix D for the details of this conference).
This section presents the main findings that emerged during that conference, followed by the committee’s own conclusions and recommendations for future studies. The committee does not present the folLEARNING FROM THE PAST lowing findings as definitive, nor are they endorsed as the results of the committee’s own research. Rather, what follows is a synopsis of the main points expressed by scholars at the conference, with particular attention to findings relevant to either DG assistance planning or research designs for case studies of DG assistance programs.
I. Democracy research conducted by the academic community generally needs considerable translation to be useful for guiding democracy assistance.
One problem that was immediately evident from discussions between the scholars and practitioners who attended the conference is that much of the academic research on democracy and democratic transitions is not developed or presented in ways that offer much practical guidance to policy professionals. This is much more than a simple matter of pure versus applied research. Rather, policymakers dealing with democracy assistance simply have to act in much more constrained circumstances than the typical academic study implies.