«Abstract. Bayesian model averaging has increasingly witnessed applications across an array of empirical contexts. However, the dearth of available ...»
In this section we examine the ability of these three packages in replicating the results of published work research deploying BMA using handwritten code. Fernandez et al. (2001b) (FLS hereafter) use a cross section of 72 countries along with 41 potential growth determinants for the period 1960 to 1992.16 FLS apply BMA to ﬁnd the key determinants of economic growth given the numerous plausible models that have emerged on the topic. We use the same dataset and deploy all three BMA packages, BMS, BAS, and BMA, to attempt to replicate their results. In order to maximize our opportunity to replicate the FLS results, we set the available options within each of the three packages as close as possible to the speciﬁcations listed in FLS. The BMS package applies the MCMC algorithm to search over the model space, burns the ﬁrst 100,000 models and the number of iteration draws to be sampled by its MCMC sampler is 200,000. It assigns the uniform distribution to the model priors. Similarly, the BAS package employs MCMC method to walk through the model space, discards the ﬁrst 100,000 models, draws samples from the model space 200,000 times, and sets the models priors to the uniform distribution. The BMA package, on the other hand, does not have enough options to directly mimic the setup in FLS (see sections 2.4 and 3.1).
[Table 8 about here.] Table 9 shows the PIPs for the variables of interest. We do not present posterior means or standard deviations since FLS only reported the PIPs in the body of their paper. To eschew making statements regarding results which FLS did not cover, we focus exclusively on the ability of the packages to reproduce the PIPs found in FLS. Column (i) shows the published PIPs that Fernandez et al. (2001b) have reported in their work and the remainder of table presents the PIPs computed via the BMS, BAS, and BMA packages.
As is apparent, only the BMS package is reasonably successful at matching the reported PIPs in FLS while the PIPs produced by the BAS package display signiﬁcant diﬀerences (compare PIPs marked by *). The BMA package also fails achieve the same PIPs of FLS.17 This most likely lies 16This dataset is taken from the larger dataset used by Sala-i-Martin (1997) for his study on robust determinants of growth. The exact FLS dataset is publicly available on the Journal of Applied Econometrics online data archive.
17Both BAS and BMA, however, are computationally much faster than the BMS package.
14 SHAHRAM M. AMINI AND CHRISTOPHER F. PARMETERin the fact that the BMA package was not called using exactly the setup in FLS and the diﬀerence in searching the model space that was described earlier. Interestingly, the PIPs returned from the BMS package almost uniformly match FLS’ PIPs greater than 0.5. A key distinction between the results is that both the BAS and BMA packages suggest a set of variables that belong in the ﬁnal model (PIP 0.5) beyond those found in FLS. Speciﬁcally, the BAS package ﬁnds 13 variables with PIPs 0.5 beyond FLS (and one variable with PIP 0.5 from FLS) while the BMA package ﬁnds 10 variables with PIPs 0.5 (and one variable with PIP 0.5 from FLS).
To further test the limits of these packages to replicate published results on BMA we attempt to reproduce the estimates in Doppelhofer & Weeks’s (2009) research, (hereafter DW), who focused on the use of model averaging when jointness of the covariates is considered. DW’s application is identical to FLS, studying the determinants of economic growth. Appendix B of their paper provides the BMA PIPs which we try to replicate using the ensemble of BMA packages. The data used in DW comprises 88 countries and 67 candidate variables as a cross section for the period 1960 to 1996. The deﬁnition of all 67 variables used can be found in data appendix B in DW and the dataset is publicly available on the Journal of Applied Econometrics data archive. As before we have tried to preserve the setup in DW by setting the packages’ options as identical as possible.
[Table 9 about here.] Table 10 displays our ﬁndings. Column (i) shows the published PIPs 0.50 that DW report in Appendix B of their paper. The rest of table presents the PIPs, posterior means and standard deviations from each of the packages.18 The results indicate that the BMS package is the only one that successfully reproduces the reported PIPs and posterior mean/standard deviation. Both the BAS and BMA packages reasonably reproduce the posterior means/standard deviations but the computed PIPs are signiﬁcantly diﬀerent from the published PIPs. For instance, the probability that “Investment Price” belongs to the ﬁnal model is roughly 77% according to DW but the BAS packages reports this probability at nearly 7% and the BMA package reports it at exactly 100%!
The estimated coeﬃcients are reasonably close for all packages yet there remain some anomalies.
The estimated posterior mean on Fraction of Tropical Area in the BMS package is about a third as small as that reported from the BAS package and half the size of the reported posterior mean from the BMA package. Moreover, both the BAS and BMA packages are suggestive that Fraction of Tropical Area is relevant from a pure t-ratio perspective (see Masanjala & Papageorgiou 2008).
Beyond diﬀerences in several of the posterior means across the packages the standard deviations show noticeable diﬀerences; compare the results for Investment Price where the standard deviation from the BMS package is nearly double that from the BMA package and almost ﬁve times as large from the reported standard deviation in the BAS package.
This paper has outlined the currently available BMA packages (BMS, BAS, and BMA) in the statistical computing environment R. Our goal was to familiarize users with the diﬀerent options that the current versions of the packages have to oﬀer. We highlighted how each of the packages implements a BMA analysis as well as the options available to the user and the outputs that are returned.
To further cement the operation of these packages and to determine how similar the packages are in practice, we presented a simple empirical example that ﬁrst allowed all three packages to fully enumerate the model space. Beyond this we enhanced our empirical example to force all three packages to engage in search mechanisms throughout the model space. When the model space is relatively small, we see that all three packages are successful at matching the PIPs, posterior means, and posterior standard deviations. However, for the larger model space similarity of the PIPs broke down considerably.
To further buttress our investigation and comparison of these packages we also compared runtimes of generic calls to each package for a range of covariate and sample sizes. In most instances the BAS package was the fastest, especially for large problems (both in terms of the number of covariates and the number of observations). Additionally, we also sought to replicate two recent studies that deployed BMA to investigate the determinants of economic growth. Both of these studies used high level programming outside of R and as such represent the perfect opportunity to see how well these freely available packages compare to computer code speciﬁcally tailored to the problem at hand. Our results were striking. The BMS package almost exactly reproduced the results from both studies while the BAS and BMA packages were not able to match the reports PIPS in either study but were reasonably accurate at constructing the posterior means and standard deviations of our second study (compared with the same estimates from the BMS package).
In sum, it appears that while the BMS package is invariably slower than its peers, its numerous options and ﬂexibility suggest that it should makes its way into the toolkit of applied researchers seeking to use BMA in their analysis. The results from the empirical examples from published studies suggest that while both the BMS and BAS packages oﬀer a similar array of options, the BMS package is capable of replicating published studies deploying BMA at the cost of slightly longer run times. Our apparent advocacy of the BMS package does not hinge on its ability to reproduce the results of published studies however, as this presumably just means that the original authors used an implementation similar to that of the BMS package which the other packages were unable to match. This in no way is an indicator of superiority. Lastly, the relative rigidity of the BMA package to that of both BMS and BAS suggests that its use in applied work should be carefully scrutinized.
16 SHAHRAM M. AMINI AND CHRISTOPHER F. PARMETER
ReferencesChib, S. & Greenberg, E. (1995), ‘Understanding the Metropolis-Hastings algorithm’, American Statistician 49(4), 327–335.
Clyde, M. (2010), BAS: Bayesian Adaptive Sampling for Bayesian Model Averaging. R package version 0.92.
URL: http://CRAN.R-project.org/package=BAS Clyde, M., Ghosh, J. & Littman, M. (2010), ‘Bayesian adaptive sampling for variable selection and model averaging’, Journal of Computational and Graphical Statistics, to appear.
Doppelhofer, G. & Weeks, M. (2009), ‘Jointness of growth determinants’, Journal of Applied Econometrics 24(2), 209– 244.
Ehrlich, I. (1973), ‘Participation in illegitimate activities: A theoretical and empirical investigation’, The Journal of Political Economy 81(3), 521–565.
Eicher, T. S., Papageorgiou, C. & Raftery, A. E. (2011), ‘Default priors and predictive performance in Bayesian model averaging with application to growth determinants’, Journal of Applied Econometrics 26, 30–55.
Feldkircher, M. & Zeugner, S. (2009), Benchmark Priors Revisited: On Adaptive Shrinkage and the Supermodel Eﬀect in Bayesian Model Averaging, IMF Working Papers 09/202, International Monetary Fund.
URL: http://ideas.repec.org/p/imf/imfwpa/09-202.html Fernandez, C., Ley, E. & Steel, M. (2001a), ‘Benchmark priors for Bayesian model averaging’, Journal of Econometrics 100(2), 381–427.
Fernandez, C., Ley, E. & Steel, M. (2001b), ‘Model uncertainty in cross-country growth regressions’, Journal of Applied Econometrics 16(5), 563–576.
Furnival, G. & Wilson Jr, R. (1974), ‘Regressions by Leaps and Bounds’, Technometrics 16(4), 499–511.
George, E. & Foster, D. (2000), ‘Calibration and empirical Bayes variable selection’, Biometrika 87(4), 731–747.
Hastings, W. (1970), ‘Monte Carlo sampling methods using Markov chains and their applications’, Biometrika 57(1), 97–109.
Hoeting, J., Madigan, D., Raftery, A. & Volinsky, C. (1999), ‘Bayesian model averaging: A tutorial’, Statistical science 14(4), 382–401.
Leamer, E. (1978), Speciﬁcation searches: Ad hoc inference with nonexperimental data, Wiley New York.
Ley, E. & Steel, M. (2009), ‘On the eﬀect of prior assumptions in Bayesian model averaging with applications to growth regression’, Journal of Applied Econometrics 24(4), 651–674.
Liang, F., Paulo, R., Molina, G., Clyde, M. & Berger, J. (2008), ‘Mixtures of g priors for Bayesian variable selection’, Journal of the American Statistical Association 103(481), 410–423.
Liu, J. (2008), Monte Carlo strategies in scientiﬁc computing, Springer Verlag.
Masanjala, W. & Papageorgiou, C. (2008), ‘Rough and Lonely Road to Prosperity: A reexamination of the sources of growth in Africa using Bayesian Model Averaging’, Journal of Applied Econometrics 23(5), 671–682.
Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A. & Teller, E. (1953), ‘Equation of state calculations by fast computing machines’, The Journal of Chemical Physics 21(6), 1087–1092.
Millar, P. (2011), ‘BIC: Stata module to evaluate the statistical signiﬁcance of variables in a model’, Statistical Software Components, Boston College Department of Economics.
URL: http://econpapers.repec.org/RePEc:boc:bocode:s449507 R Development Core Team (2010), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
URL: http://www.R-project.org Raftery, A. E. (1995), ‘Bayesian model selection in social research’, Sociological Methodology 25, 111–163.