# «Abstract. Bayesian model averaging has increasingly witnessed applications across an array of empirical contexts. However, the dearth of available ...»

4.1. Enumeration of the Model Space. We ﬁrst compare the results of performing basic BMA with the original UScrime dataset, which is small enough to fully enumerate the model space for all three of the separate BMA packages discussed above. Table 4 shows the probabilities that each variable belongs to the ﬁnal model (PIP). All three packages in this setup fully enumerate all 215 = 32, 768 models. Column (i) shows the PIPs calculated by the BMS package. The call to this package uses a uniform distribution for model priors and unit information priors for the distribution of regressions coeﬃcients. Column (ii) presents the PIPs computed by the BAS package. Model priors are set to have uniform distributions and the priors on the regressions coeﬃcients come from ZellnerSiow’s g priors. Column (iii) indicates the PIPs that the BMA package computes. The BMS and BAS packages produce nearly identical PIPs while those of BMA are markedly diﬀerent. The diﬀerent internal algorithm for calculating Bayes factors and other parameters is the main reason for this disparity. Table 5, on the other hand, shows the posterior means and standard errors of regressors calculated from the three packages. Similar to the posterior inclusion probabilities, the estimated coeﬃcients and standard errors computed by the BMS and BAS packages are literally identical but the estimates obtained from the BMA package are relatively diﬀerent from their counterparts.

## 10 SHAHRAM M. AMINI AND CHRISTOPHER F. PARMETER

[Table 3 about here.] [Table 4 about here.]4.2. Model Sampling. In this section we compare the results across the three packages when there is a large model space and model search must be undertaken to ascertain the best models.

We create a large model space by adding 35 independent, ﬁctitious variables obtained from a standard-normal distribution to the original UScrime dataset (for a total of 50 covariates). Tables 6 and 7 present the results. Using the available controls for each package we tried our best to set up the example so that the three packages are as identical as possible.

[Table 5 about here.] Table 6 shows the PIPS. Column (i) presents the probabilities obtained from the BMS package.

This package employs the birth-death sampler using an MCMC search algorithm to ﬁnd the posterior probabilities with a burn-in of 2000 models. Its choices for model priors and the distribution of the model coeﬃcients are uniform distribution and unit information priors, respectively. Column (ii) displays the results from the BAS package. We choose a uniform distribution for the model prior, Zellner-Siow’s g prior for the distributions of the coeﬃcients, and an adaptive MCMC method for model sampling. Column (iii) shows the PIPs that the BMA package assigns to each variable. As discussed earlier, the BMA package takes an entirely diﬀerent model sampling approach than either the BMS or BAS packages. It ﬁrst reduces the initial set of variables using a backward elimination algorithm and then implements the iterated BMA method for variable selection. The package calls repeatedly to a BMA procedure, iterating through the variables in a ﬁxed order. After each call only those variables which have posterior probability greater than a speciﬁed threshold, controlled through thresProbne0 in the call to iBMA.bicreg(), are kept and those variables whose posterior probabilities do not meet the threshold are replaced with the next set of variables. The call to the BMA package uses the BIC approximation as its choice for the distributions of model coeﬃcients which is roughly identical to UIPs.

[Table 6 about here.] Table 7 shows the estimated posterior means and standard deviations of the covariates. Moreover, how long each of the packages takes to run is shown in the last line of the table. The BMS and BAS packages that use similar internal search algorithms for model sampling generate roughly identical posterior means and standard deviations, and fairly identical posterior probability that each variable is non-zero. The BMA package, the fastest of the three, however, calculates signiﬁcantly diﬀerent posterior probabilities, means and standard deviations from what BMS and BAS produce.

This diﬀerence lies in the fact that the BMA package relies on a hierarchical OLS t-statistic to obtain a handful of likely models, directly conﬂicting with the model sampling approaches in the BMS and BAS packages.

at model complexity. The BMS, BMA, and BAS packages all plot the marginal inclusion densities as well as images that show inclusion and exclusion of variables within models using separate colors. However, the BMS and BAS packages provide more graphical visualizations for the users.

The following ﬁgures are some examples of the plots that these packages provide. These ﬁgures all correspond to the above example where we ﬁctitiously incorporated 35 white noise covariates to force the packages to engage in searching over the model space.

[Figure 1 about here.] Figures 1 and 2 are combined plots provided by the BMS package. The upper plot in each ﬁgure shows the prior and posterior distribution of model sizes and helps to illustrate to the user the impact of the model prior assumption on the estimation results. Consistent with the research of Ley & Steel (2009) and Eicher, Papageorgiou & Raftery (2011), who stress the importance of model priors in applied work, the plots produced by the BMS package allow for visual clariﬁcation of choice of model prior on posterior results. For example, the upper plot of Figure 1 assumes a uniform distribution for the model prior whereas the upper plot of Figure 2 assumes a “ﬁxed” common prior inclusion probability for each regressor as an alternative to a uniform prior.13 These plots allow the user to graphically compare across a range of model size priors to determine the impact on the posterior distribution. In the example here it seems that the model prior (either ﬁxed or uniform) has little eﬀect on the posterior distribution of model sizes.

[Figure 2 about here.] The lower plot of both ﬁgures shows the analytical likelihood of the best 200 models and their MCMC frequencies/draws of these models from the MCMC sampler. If the sampler has converged, then the MCMC draws should conform to the analytical/exact likelihoods. This is expressed in the correlation of the two lines by the correlation reported in parentheses beneath the plot’s title.

In other words, these graphs are an indicator of how well the current 200 best models encountered by the search algorithm of the BMS package have converged. These plots are useful in determining the search mechanisms ability to ﬁnd ‘good’ models throughout the model space. If the user deems that the best models are not acceptable then longer runs of the search algorithm or more burn-ins may be speciﬁed to help reﬁne the search over the model space.

Figure 3(a) is a visualization of the mixture marginal posterior density for the coeﬃcient of GDP produced by the BMS package. The bar above the density shows the PIP, and the dotted vertical lines show the corresponding standard deviation bounds in from the MCMC approach. Figure 3(b) shows the same visualization using the BMA package. The posterior probability that the coeﬃcient is zero is represented by a solid line at zero, with height equal to the probability. The nonzero part of the distribution is scaled so that the maximum height is equal to the probability that the coeﬃcient is nonzero. Figure 3(c) shows the same visualization using the BAS package. Similarly, the posterior probability that the coeﬃcient is zero is represented by a solid line at zero, with height equal to the probability. The nonzero part of the distribution is scaled so that the maximum height 13See Sala-i-Martin et al. (2004) for more details.

## 12 SHAHRAM M. AMINI AND CHRISTOPHER F. PARMETER

is equal to the probability that the coeﬃcient is nonzero. We mention here that Figure 3(b) is the only plot produced within the BMA package.[Figure 3 about here.] The plots of Figure 4 are created within the BAS package. The ﬁrst panel, (a), is a plot of the residuals versus the ﬁtted values from the model averaging exercise. This graph allows the user to visualize if the ﬁtted values are heavily dependent on the residuals. If one can see a speciﬁc pattern or trend in this graph, then it could be a sign that an important variable is missing. Panel (b) is a plot of the log marginal likelihood versus the overall model dimension. Using this graph, the user can visually predict the dimensionality of models with the highest marginal likelihoods.

For instance the majority of the best models is concentrated around 10 to 12. Panel (c) shows the posterior marginal inclusion probabilities for each variable.14 This diagnostic plot makes it immediately obvious that the additional white noise variables we included almost uniformly have very low PIPs.

[Figure 4 about here.]

** 5. Time Performance of the Three Packages**

A key component of any statistical software is the amount of time it takes to produce results.

To that end, this section reports the run times for the three packages across various sample sizes and dimensions of the covariate space. The size of the covariate space comes from {15, 30, 60, 120} while the sample size is taken from {50, 500, 5000, 50000}. The dependent variable as well as the covariates are drawn from a standard normal distribution. We run our analysis on a 1.60 GHz Intel(R) Core(TM) 2 Duo processor with 3.00 GB of DDR3 memory. Table 8 presents the run times in seconds for each package. Within each set of parenthesis are the speciﬁc run times for a given covariate-sample size combination from baseline15 calls to the BMS, BAS, and BMA packages, respectively.

The ﬁrst thing to notice is that due to the fact that the BMA package uses a simple model selection algorithm, it is the fastest among the three packages when the number of covariates is less than 30.

The BAS package that provides roughly the same options as the BMS package is slower than the BMA package but noticeably faster than the BMS package. Once the number of covariates is greater than 15 and the sample size is beyond 500, the BAS package is uniformly the fastest of the packages.

One reason that the BAS package realizes faster computing times relative to the BMS package with similar controls is that the BAS package is an R wrapper for a C library. This enables the call to the BAS package to take advantage of the high speed compiled C code used in the package.

Note that all three packages display increases in computation time as the dimensionality of the covariates increases. The number of covariates (intuitively) leads to larger increases in computation time than does increases in the sample size. Note that there are at least two counter-acting eﬀects 14The variables with PIPs0.5 are drawn in red for ease of identiﬁcation for the user.

15Baseline options include uniform model priors, 500,000 iterations of the chosen MCMC algorithm, and discarding the ﬁrst 3000 models (burn-in).

BMA IN R 13 as the sample size, n, increases. A larger n puts more posterior mass on the best models. Therefore we expect that the MCMC sampler will converge quicker. On the other hand, the larger n, the more ‘large’ models will be emphasized, which slows down the sampler. If these two eﬀects oﬀset each other then we expect little change in run time over n. This is apparent in the BMS package. Notice how the BMS package has run times which decrease for a ﬁxed level of covariates as we increase n while both the BAS and BMA packages actually take longer for n = 50, 000 relative to n = 5, 000.

[Table 7 about here.]

** 6. Replicability**