«Development, Security, and Cooperation Policy and Global Affairs THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The ...»
Another example is the question of how best to promote a robust and vibrant civil society. USAID regularly makes substantial investments in civil society organizations (CSOs) and local NGOs with the hope of empowering the disadvantaged, building trust, enhancing cooperation, and supporting the flourishing of democratic institutions (Putnam 1993, 2000). Yet some skeptics have warned that outside support for CSOs might be counterproductive: It may produce more professionally run organizations that no longer have strong ties to their grassroots base (Skocpol 2003) and may actually change the leadership of such organizations, disempowering the disadvantaged (Igoe 2003). Knowing whether outside assistance helps or harms CSOs is a question of vital importance, and randomized evaluations have begun to offer some preliminary evidence.
Gugerty and Kremer (2006) conducted a randomized evaluation in which a sample of women’s self-help associations in rural Western Kenya were randomly selected to receive a package of assistance that included organizational and management training as well as a set of valuable agricultural inputs such as tools, seeds, and fertilizer. Forty groups received assistance in the first year, while an additional 40 eligible groups served as the control group (although they were given the same assistance, just two years later). The results are disturbing for advocates of outside funding to community groups. While members of the funded groups reported higher levels of satisfaction with their group leadership, there is little evidence that objective measures of group activity improved. Moreover, Gugerty and Kremer found that outside funding changed the nature of the group and its leadership. Younger, more educated women and women from the formal sector increasingly joined the group, and these new entrants tended to assume leadership positions and to displace older women.
IMPROVING DEMOCRACY ASSISTANCE Compared to their unfunded counterparts, funded groups experienced a two-thirds increase in the exit rate of older women—a troubling finding given the program’s underlying objective of empowering the disempowered. Whereas an analysis of group members’ satisfaction would have led project evaluators to conclude that the project was a success, the careful randomized design led Gugerty and Kremer to the opposite conclusion (and generated significant evidence that the skeptics may be right about the sometimes counterproductive impact of donor funding to CSOs).
These two examples serve to support a broader point: It is both possible and important to conduct randomized impact evaluations of projects designed to support DG. In both cases the randomized evaluations effectively measured a project’s impact, but they also provided new evidence about implicit hypotheses that guide programming more broadly. In the case of corruption, the implicit hypothesis was that community empowerment is an antidote to local-level corruption; in the case of civil society support, the hypothesis was that donors can spur the growth of a vibrant civil society that empowers the disadvantaged through outside support.
The evidence casts some doubt on both hypotheses and should encourage further evaluations to see if these results hold more broadly and perhaps fuel the search for alternative methods to support DG goals.
The larger point, however, is not so much the findings of these studies as the fact that they were successfully conducted on DG projects. The next chapter describes the findings of the committee’s field studies and discusses how these designs could be applied to the evaluation of several of USAID’s own current DG projects. It also explicitly addresses some of the common objections to using randomized evaluations more widely.
REFERENCESAngrist, J.D., and Lavy, V. 1999. Using Maimonides’ Rule to Estimate the Effect of Class Size on Scholastic Achievement. Quarterly Journal of Economics 114:533-575.
Angrist, J.D., Bettinger, E., and Kremer, M. 2006. Long-Term Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia. American Economic Reiew 96:847-862.
Auvert, B., Taljaard, D., Lagarde, E., Sobngwi-Tambekou, J., Sitta, R., and Puren, A. 2005.
Randomized, Controlled Intervention Trial of Male Circumcision for Reduction of HIV Infection Risk: The ANRS 1265 Trial. PLOS Medicine. Available at: http://medicine.
Accessed on February 23, 2008.
Banerjee, A.V., and Duflo, E. 2006. Addressing Absence. Journal of Economic Perspecties 20(1):17-132.
Banerjee, A.V., and Kremer, M. 2002. Teacher-Student Ratios and School Performance in Udaipur, India: A Prospective Evaluation. Washington, DC: Brookings Institution.
Banerjee, A.V., Cole, S., Dutlo, E., and Linden, L. 2007. Remedying Education: Evidence from Two Randomized Experiments in India. Quarterly Journal of Economics 122(3):1235METHODOLOGIES OF IMPACT EVOLUTION Becker, G.S., and Stigler, G.J. 1974. Law Enforcement, Malfeasance, and the Compensation of Enforcers. Journal of Legal Studies 3:1-19.
Bertrand, M., Duflo, E., and Mullainathan, S. 2004. How Much Should We Trust DifferenceIn-Difference Estimates? Quarterly Journal of Economics 119(1):249-275.
Bertrand, M., Djankov, S., Hanna, R., and Mullainathan, S. 2007. Obtaining a Driving License in India: An Experimental Approach to Studying Corruption. Quarterly Journal of Economics 122:1639-1676.
Bloom, H.S., ed. 2005. Learning More from Social Experiments: Eoling Analytic Approaches.
New York: Russell Sage Foundation.
Bollen, K., Paxton, P., and Morishima, R. 2005. Assessing International Evaluations: An Example from USAID’s Democracy and Governance Programs. American Journal of Ealuation 26:189-203.
Bratton, M., Mattes, R., and Gyimah-Boadi, E. 2005. Public Opinion, Democracy, and Market Reform in Africa. New York: Cambridge University Press.
Campbell, D.T. 1968/1988. The Connecticut Crackdown on Speeding: Time-Series Data in Quasi-Experimental Analysis. Pp. 222-238 in Methodology and Epistemology for Social Science, E.S. Overman, ed. Chicago: University of Chicago Press.
Devine, T.J., and Heckman, J.J. 1996. The Economics of Eligibility Rules for a Social Program:
A Study of the Job Training Partnership Act (JTPA)—A Summary Report. Canadian Journal of Economics 29(Special Issue: Part 1):S99-S104.
Duflo, E. 2000. Schooling and Labor Market Consequences for School Construction in Indonesia. Cambridge, MA: MIT Dept. of Economics Working Paper No. 00-06.
Duflo, E., Kremer, M., and Robinson, J. 2006a. Understanding Technology Adoption: Fertilizer in Western Kenya. Evidence from Field Experiments. Available at: http://www.econ.
berkeley.edu/users/webfac/saez/e_s0/esther.pdf. Accessed February 23, 2008.
Duflo, E., Glennerster, R., and Kremer, M. 2006b. Using Randomization in Development Economics Research: A Toolkit. Unpublished paper. Massachusetts Institute of Technology and Abdul Latif Jameel Poverty Action Lab. Cambridge, MA.
Dupas, P. 2007. Relative Risks and the Market for Sex: Teenagers, Sugar Daddies, and HIV in Kenya. Available at: http://www.dartmouth.edu/~pascaline/. Accessed on February 23, 2008.
Gerring, J. 2007. Case Study Research: Principles and Practices. Cambridge: Cambridge University Press.
Gerring, J., and McDermott, R. 2007. An Experimental Template for Case-Study Research.
American Journal of Political Science 51:688-701.
Glewwe, P., Kremer, M., and Moulin, S. 2007. Many Children Left Behind? Textbooks and Test Scores in Kenya. NBER Working Paper No. 13300. Available at: http://papers/nber.
org/papers/w00. Accessed on April 26, 2008.
Gueron, J.M., and Hamilton, G. 2002. The Role of Education and Training in Welfare Reform.
Policy Brief No. 20. Washington, DC: Brookings Institution.
Grootaert, C., Narayan, D., Jones, V.N., and Woolcock, M. 2004. Measuring Social Capital:
An Integrated Questionnaire. World Bank Working Paper No. 18. Washington, DC:
The World Bank.
Gugerty, M.K., and Kremer, M. 2006. Outside Funding and the Dynamics of Participation in Community Associations. Background Paper. Washington, DC: World Bank. Available at http://siteresources.worldbank.org/INTPA/Resources/Training-Materials/OutsideFunding.
pdf. Accessed on April 26, 2008.
Hahn, J., Todd, P., and Van der Klaauw, W. 2001. Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design. Econometrica 69(1):201-209.
Heckman, J. 1997. Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations. Journal of Human Resources 32(3):441-462.
0 IMPROVING DEMOCRACY ASSISTANCE Heckman, J., Ichimura, H., and Todd, P. 1997. Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program. Reiew of Economic Studies 64:605-654.
Hyde, S. 2006. The Observer Effect in International Politics: Evidence from a Natural Experiment. Unpublished paper. Yale University.
Igoe, J. 2003. Scaling Up Civil Society: Donor Money, NGOs and the Pastoralist Land Rights Movement in Tanzania. Deelopment and Change 34(5):863-885.
Kremer, M. 2003. Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons. American Economic Reiew 93(2):102-106.
Kremer, M., Leino, J., Miguel, E., and Zwane, A. 2006. Spring Cleaning: A Randomized Evaluation of Source Water Improvement. Available at http://economics.harard.edu/ faculty/Kremer/files/springclean.pdf. Accessed on April 26, 2008.
Meyer, B.D., Viscusi, W.K., and Durbin, D.L. 1995. Workers’ Compensation and Injury Duration: Evidence from a Natural Experiment. American Economic Reiew 85(3):322-340.
Miguel, E., and Kremer, M. 2004. Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities. Econometrica 72(1):159-217.
Miron, J.A. 1994. Empirical Methodology in Macroeconomics: Explaining the Success of Friedman and Schwartz’s “A Monetary History of the United States, 1867-1960.” Journal of Monetary Economics 34:17-25.
Morley, S., and Coady, D. 2003. Targeted Education Subsidies in Deeloping Countries: A Reiew of Recent Experiences. Washington, DC: Center for Global Development.
Murdoch, J. 2005. The Economics of Microfinance. Cambridge, MA: MIT Press.
Nevill, C.G., Some, E.S., Mung’ala, V.O., Mutemi, W., New, L., Marsh, K., Lengeler, C., and Snow, R.W. 1996. Insecticide-Treated Bednets Reduce Mortality and Severe Morbidity from Malaria Among Children on the Kenyan Coast. Tropical Medicine and International Health 1:139-146.
Newhouse, J.P. 2004. Consumer-Directed Health Plans and the RAND Health Insurance Experiment. Health Affairs 23(6):107-113.
Olken, B.A. 2007. Monitoring Corruption: Evidence from a Field Experiment in Indonesia.
Journal of Political Economy 115:200-249.
Putnam, R. 1993. Making Democracy Work: Ciic Traditions in Modern Italy. Princeton, NJ:
Princeton University Press.
Putnam, R. 2000. Bowling Alone. New York: Simon and Schuster.
Schultz, T.P. 2004. School Subsidies for the Poor: Evaluating the Mexican PROGRESA Poverty Program. Journal of Deelopment Economics 74(1):199-250.