«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
This chapter provides an analysis of existing measures of democracy and points the way toward developing a disaggregated measure of the type requested by USAID. The committee finds that most existing measures of democracy are adequate, and in fair agreement, at the level of crude determination of whether a country is solidly democratic, autocratic, or somewhere in between. However, the committee also finds that all existing measures are severely inadequate at tracking small movements or small differences in levels of democracy between countries or in a single country over time. Moreover, the committee finds that existing measures disaggregate democracy in very different ways and that their measures of various components of democracy do not provide transparent, objective, independent, or reliable indicators of change in those components over time.
While recognizing that it may seem self-serving for an academic committee to recommend “more research,” it is the committee’s belief—after surveying the academic literature and convening a workshop of experts in democracy measures to discuss the issue—that if USAID wishes a measure of democracy that it can use to gauge the impact of its programs and
track the progress of countries in which it is active, it faces a stark choice:
either rely on the current flawed measures of democracy or help support the development of a research project on democracy indicators that—it is hoped—will eventually produce a set of indicators with the broadly accepted integrity of today’s national accounts indicators for economic development.
To provide just a few examples to preview the discussion below,
USAID manages its DG programs with an eye toward four broad areas:
rule of law, elections, civil society, and good governance. Yet consider the two most widely used indicators of democracy: the Polity autocracy/ democracy scale and the Freedom House scales of civil liberties and political rights. The former breaks down its measures of democracy into three components: executive recruitment, executive constraints, and political
MEASURING DEMOCRACYcompetition, measured by six underlying variables. While some of these could be combined to provide indicators of elections, civil society, and aspects of rule of law, Polity does not address “good governance.” Moreover, the validity of the various components and underlying variables in Polity is so greatly debated that there is no reason to believe that a measure of rule of law based on the Polity components would be accepted.
Freedom House rates nations on two scales: civil liberties (which conflates rule of law, civil society, and aspects of good governance) and political rights (which conflates rule of law, elections, and aspects of good governance). Even if these scales were based on objective and transparent measurements (and they are not), there would be no way to extract from them information on components relevant to USAID’s DG policy areas.
Fortunately, while more sensitive and accurate measures to track sectoral movements toward or away from democracy are vital to improving USAID’s policy planning and targeting of DG programs, USAID can still gain knowledge on the impacts of its programs by focusing on changes in outcome indicators at a level relevant to those projects (for which methodologies are examined in Chapters 5 through 7). That is, USAID should seek to determine whether its projects lead to more independent and effective behavior by judges and legislators, broader electoral participation and understanding by citizens, more competitive and fair election practices, fewer corrupt officials, and other concrete changes. The issue of how much those changes contribute to overall trajectories of democracy or democratic consolidation is one that can only be solved by future experience and study and the development of better disaggregated measures for tracking democracy at the sectoral level.
The committee thus agrees that USAID is correct in focusing its interest in measurement on developing a measure of democracy that is disaggregated into discrete and measurable components. This chapter will analyze existing approaches to measuring democracy, identifying why they are flawed, and point the way toward what the committee believes will be a more useful approach to developing disaggregated sectoral or meso-level measures (Table 2-1).
PROBLEMS WITH ExTANT INDICATORSA consensus is growing within the scholarly community that existing indicators of democracy are problematic.2 These problems may be grouped into five categories: (1) problems of definition, (2) sensitivity issues, (3) measurement errors and data coverage, (4) aggregation probSee Bollen (1993), Beetham (1994), Gleditsch and Ward (1997), Bollen and Paxton (2000), Foweraker and Krznaric (2000), McHenry (2000), Munck and Verkuilen (2002), Treier and Jackman (2003), Berg-Schlosser (2004 a, b), Acuna-Alfaro (2005), and Vermillion (2006).
IMPROVING DEMOCRACY ASSISTANCE lems, and (5) lack of convergent validity. What follows is a brief, sometimes rather technical, review of these problems and their repercussions.
Definitions of key terms are provided in the text or in the Glossary at the end of the report.
The focus of the discussion is on several leading democracy indicators: (1) Freedom House; (2) Polity; (3) ACLP (“ACLP” stands for the names of the creators—Alvarez, Cheibub, Limongi, and Przeworski;
Alvarez et al 1996; recently expanded by Boix and Rosato 2001); and (4) the Economist Intelligence Unit (EIU). Freedom House provides two indices: “Political Rights” and “Civil Liberties” (sometimes employed in tandem, sometimes singly). Both are seven-point scales extending back to 1972 and cover most sovereign and semisovereign nations.3 Polity also provides two aggregate indices: “Democracy” and “Autocracy.” Both are 10-point scales and are usually used in tandem (by subtracting one from the other), which provides the 21-point (-10 to 10) Polity2 variable.
Coverage extends back to 1800 for sovereign countries with populations greater than 500,000.4 ACLP codes countries dichotomously (autocracy/ democracy) and includes most sovereign countries from 1950 to 1990.
The expanded dataset provided by Boix and Rosato (2001) stretches back to 1800.5 The EIU has recently developed a highly disaggregated index of democracy with 5 core dimensions and 60 subcomponents, which are combined into a single index of democracy (Kekic 2007). Coverage extends to 167 sovereign or semisovereign nations but only in 2006.
Glancing reference will be made to other indicators in an increasingly crowded field,6 and many of the points made in the following discussion apply quite broadly. However, it is important to bear in mind that each indicator has its own particular strengths and weaknesses. The following brief survey does not purport to provide a comprehensive review.7 3 See www.freedomhouse.org.
4 Both are drawn from the most recent iteration of this project, known as Polity IV. See www.cidcm.umd.edu/inscr/polity.
5 Jose Cheibub and Jennifer Ghandi are currently engaged in updating the ACLP dataset, but results are not yet available.
6See Bollen (1980), Coppedge and Reinicke (1990), Arat (1991), Hadenius (1992), Vanhanen (2000), Altman and Pérez-Liñán (2002), Gasiorowski (1996; updated by Reich 2002 [also known as “Political Regime Change—PRC dataset”]), and Moon et al (2006).
7 The most detailed and comprehensive recent reviews are Hadenius and Teorell (2005) and
Definition There are many ways to define democracy, and each naturally generates a somewhat different approach to measurement (Munck and Verkuilen 2002). Some definitions are extremely “thin,” focusing mainly on the presence of electoral competition for national office. The ACLP index exemplifies this approach: Countries that have changed national leadership through multiparty elections are democracies; other countries are not. Other definitions are rather “thick,” encompassing a wide range of social, cultural, and legal characteristics well beyond elections. For example, the Freedom House Political Rights Index includes the following
questions pertaining to corruption:
Has the government implemented effective anticorruption laws or programs to prevent, detect, and punish corruption among public officials, including conflict of interest? Is the government free from excessive bureaucratic regulations, registration requirements, or other controls that increase opportunities for corruption? Are there independent and effective auditing and investigative bodies that function without impediment or political pressure or influence? Are allegations of corruption by government officials thoroughly investigated and prosecuted without prejudice, particularly against political opponents? Are allegations of corruption given wide and extensive airing in the media? Do whistle-blowers, anticorruption activists, investigators, and journalists enjoy legal protections that make them feel secure about reporting cases of bribery and corruption? What was the latest Transparency International Corruption Perceptions Index score for this country? (Freedom House 2007) It may be questioned whether these aspects of governance, important though they may be, are integral components of democracy.
More generally, many scholars treat good governance as a likely result of democracy; yet many donors (including USAID) treat good governance as an essential component of democracy. Similar complaints might be registered about other concepts and scales of democracy; some are so “thick” as to include diverse elements of accountability, even distributional equity and economic growth.
For example, some definitions treat the United States as a democracy from the passage of its Constitution and first national election in 1789. Yet since George Washington ran uncontested in both 1789 and 1792, even ACLP would not treat the United States as democratic until the appearance of contested multiparty elections in 1796. If slavery is considered a contravention of democracy, the United States could not be considered a democracy until its abolition throughout its territory in 1865. If women’s right to vote is also considered essential to the definition of democracy, the United States does not qualify until 1920. And if the disenfranchisement of African Americans in southern states is considered a block to democIMPROVING DEMOCRACY ASSISTANCE racy, the United States does not become a full democracy until passage of the Civil Rights Act in 1965.
In short, only a “thin” definition of democracy would classify the United States as “fully democratic” from the early nineteenth century. Yet most donor agencies are reluctant to adopt such thin measures as a guide to current democracy assessments, questioning whether “thin” indices of democracy capture all the critical features of this complex concept. The problem of definition is critical but very difficult to resolve.
Sensitivity A related issue is that many of the leading democracy indicators are not sensitive to important gradations in the quality of democracy across countries or through time. At the extreme, dichotomous measures such as ACLP reduce democracy to a dummy variable: A country either is or is not a democracy, with no intermediate stages permitted. While useful for certain purposes, one may wonder whether this captures the complexity of such a variegated concept (Elkins 2000). At best it captures one or two dimensions of democracy (those employed as categorizing principles), while the rest are necessarily ignored.
Most democracy indicators allow for a more elongated scale. As noted above, Freedom House scores democracy on a seven-point index (14 points if the Political Rights and Civil Liberties indices are combined).