This paper provides an overview of a recently developed class of strategies for model selection, known as the fence methods. It also offers directions of future research as well as challenging problems. 1. Introduction On the morning of March 16, 1971, Hirotugu Akaike, as he was taking a seat on a commuter train, came out with the idea of a connection between the relative Kullback-Liebler discrepancy and the empirical log-likelihood function, a procedure that was later named Akaike’s information criterion, or AIC (Akaike [1, 2]; see Bozdogan [3] for the historical note). The idea has allowed major advances in model selection and related fields. See, for example, de Leeuw [4]. A number of similar criteria have since been proposed, including the Bayesian information criterion (BIC; Schwarz [5]), a criterion due to Hannan and Quinn (HQ; [6]), and the generalized information criterion (GIC; Nishii [7], Shibata [8]). All of the information criteria can be expressed as where is a measure of lack-of-fit by the model, ; is the dimension of , defined as the number of free parameters under ; and is a penalty for complexity of the model, which may depend on the effective sample size, . Although the information criteria are broadly used, difficulties are often encountered, especially in some nonconventional situations. We discuss a number of such cases below. (1) The Effective Sample Size. In many cases, the effective sample size, , is not the same as the number of data points. This often happens when the data are correlated. Take a look at two extreme cases. In the first case, the observations are independent; therefore, the effective sample size should be the same as the number of observations. In the second case, the data are so much correlated that all of the data points are identical. In this case, the effective sample size is 1, regardless of the number of data points. A practical situation may be somewhere between these two extreme cases, such as cases of mixed effects models (e.g., Jiang [9]), which makes the effective sample size difficult to determine. (2) The Dimension of a Model. The dimension of a model, , can also cause difficulties. In some cases, such as the ordinary linear regression, this is simply the number of parameters under , but in other situations, where nonlinear, adaptive models are fitted, this can be substantially different. Ye [10] developed the concept of generalized degrees of freedom (gdf) to track model complexity. For example, in the case of multivariate adaptive regression splines (Friedman [11]), nonlinear terms can have an
References
[1]
H. Akaike, “Information theory as an extension of the maximum likelihood principle,” in Proceedings of the 2nd International Symposium on Information Theory, B. N. Petrov and F. Csaki, Eds., pp. 267–281, Akademiai Kiado, Budapest, Hungary, 1973.
[2]
H. Akaike, “A new look at the statistical model identification,” IEEE Transaction on Automatic Control, vol. 19, no. 6, pp. 716–723, 1974.
[3]
H. Bozdogan, “Editor's general preface,” in Proceedings of the 1st US/Japan Conference on the Frontiers of Statistical Modeling: An Informational Approach, H. Bozdogan, Ed., vol. 3 of Engineering and Scientific Applications, pp. 9–12, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1994.
[4]
J. de Leeuw, “Introduction to Akaike (1973) information theory and an extension of the maximum likelihood principle,” in Breakthroughs in Statistics, S. Kotz and N. L. Johnson, Eds., vol. 1, pp. 599–609, Springer, London, UK, 1992.
[5]
G. Schwarz, “Estimating the dimension of a model,” The Annals of Statistics, vol. 6, no. 2, pp. 461–464, 1978.
[6]
E. J. Hannan and B. G. Quinn, “The determination of the order of an autoregression,” Journal of the Royal Statistical Society B, vol. 41, no. 2, pp. 190–195, 1979.
[7]
R. Nishii, “Asymptotic properties of criteria for selection of variables in multiple regression,” The Annals of Statistics, vol. 12, no. 2, pp. 758–765, 1984.
[8]
R. Shibata, “Approximate efficiency of a selection procedure for the number of regression variables,” Biometrika, vol. 71, no. 1, pp. 43–49, 1984.
[9]
J. Jiang, Linear and Generalized Linear Mixed Models and Their Applications, Springer, New York, NY. USA, 2007.
[10]
J. Ye, “On measuring and correcting the effects of data mining and model selection,” Journal of the American Statistical Association, vol. 93, no. 441, pp. 120–131, 1998.
[11]
J. H. Friedman, “Multivariate adaptive regression splines,” The Annals of Statistics, vol. 19, no. 1, pp. 1–67, 1991.
[12]
K. W. Broman and T. P. Speed, “A model selection approach for the identification of quantitative trait loci in experimental crosses,” Journal of the Royal Statistical Society B, vol. 64, no. 4, pp. 641–656, 2002.
[13]
J. Jiang, J. S. Rao, Z. Gu, and T. Nguyen, “Fence methods for mixed model selection,” The Annals of Statistics, vol. 36, no. 4, pp. 1669–1692, 2008.
[14]
J. Jiang, T. Nguyen, and J. S. Rao, “A simplified adaptive fence procedure,” Statistics & Probability Letters, vol. 79, no. 5, pp. 625–629, 2009.
[15]
J. Shao, “Linear model selection by cross-validation,” Journal of the American Statistical Association, vol. 88, no. 422, pp. 486–494, 1993.
[16]
T. Nguyen and J. Jiang, “Restricted fence method for covariate selection in longitudinal data analysis,” Biostatistics, vol. 13, no. 2, pp. 303–314, 2012.
[17]
T. Nguyen, J. Peng, and J. Jiang, “Fence methods for backcross experiments,” Journal of Statistical Computation and Simulation, vol. 84, no. 3, pp. 644–662, 2014.
[18]
J. Jiang, T. Nguyen, and J. S. Rao, “Invisible fence methods and the identification of differentially expressed gene sets,” Statistics and its Interface, vol. 4, no. 3, pp. 403–415, 2011.
[19]
A. Subramanian, P. Tamayo, V. K. Mootha et al., “Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles,” Proceedings of the National Academy of Sciences of the United States of America, vol. 102, no. 43, pp. 15545–15550, 2005.
[20]
B. Efron and R. Tibshirani, “On testing the significance of sets of genes,” The Annals of Applied Statistics, vol. 1, no. 1, pp. 107–129, 2007.
[21]
J. Mou, Two-stage fence methods in selecting covariates and covariance for longitudinal data [Ph.D. dissertation], Department of Statistics, University of California, Davis, Calif, USA, 2012.
[22]
G. K. Robinson, “That BLUP is a good thing: the estimation of random effects,” Statistical Science, vol. 6, no. 1, pp. 15–51, 1991.
[23]
J. Jiang and P. Lahiri, “Mixed model prediction and small area estimation,” Test, vol. 15, no. 1, pp. 1–96, 2006.
[24]
R. E. Fay and R. A. Herriot, “Estimates of income for small places: an application of James-Stein procedures to census data,” Journal of the American Statistical Association, vol. 74, pp. 269–277, 1979.
[25]
J. Jiang, T. Nguyen, and J. S. Rao, “Best predictive small area estimation,” Journal of the American Statistical Association, vol. 106, no. 494, pp. 732–745, 2011.
[26]
J. N. K. Rao, Small Area Estimation, Wiley, New York, NY, USA, 2003.
[27]
S. Chen, J. Jiang, and T. Nguyen, “Observed best prediction for small area counts,” Journal of Survey Statistics and Methodology. Revised.
[28]
R. J. A. Little and D. B. Rubin, Statistical Analysis with Missing Data, John Wiley & Sons, New York, NY, USA, 2nd edition, 2002.
[29]
J. Jiang, T. Nguyen, and J. S. Rao, “The E-MS algorithm: model selection with incomplete data,” Journal of the American Statistical Association. In Press.
[30]
E. S. Lander and S. Botstein, “Mapping mendelian factors underlying quantitative traits using RFLP linkage maps,” Genetics, vol. 121, no. 1, p. 185, 1989.
[31]
H. Zhan, X. Chen, and S. Xu, “A stochastic expectation and maximization algorithm for detecting quantitative trait-associated genes,” Bioinformatics, vol. 27, no. 1, Article ID btq558, pp. 63–69, 2011.
[32]
G. Verbeke, G. Molenberghs, and C. Beunckens, “Formal and informal model selection with incomplete data,” Statistical Science, vol. 23, no. 2, pp. 201–218, 2008.
[33]
J. G. Ibrahim, H. Zhu, and N. Tang, “Model selection criteria for missing-data problems using the EM algorithm,” Journal of the American Statistical Association, vol. 103, no. 484, pp. 1648–1658, 2008.
[34]
J. Copas and S. Eguchi, “Local model uncertainty and incomplete-data bias,” Journal of the Royal Statistical Society B: Statistical Methodology, vol. 67, no. 4, pp. 459–513, 2005.
[35]
J. Fan and J. Lv, “A selective overview of variable selection in high dimensional feature space,” Statistica Sinica, vol. 20, no. 1, pp. 101–148, 2010.
[36]
R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society B. Methodological, vol. 58, no. 1, pp. 267–288, 1996.
[37]
W. J. Fu, “Penalized regressions: the bridge versus the lasso,” Journal of Computational and Graphical Statistics, vol. 7, no. 3, pp. 397–416, 1998.
[38]
B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” The Annals of Statistics, vol. 32, no. 2, pp. 407–499, 2004.
[39]
J. Fan and R. Li, “Variable selection via nonconcave penalized likelihood and its oracle properties,” Journal of the American Statistical Association, vol. 96, no. 456, pp. 1348–1360, 2001.
[40]
H. Zou, “The adaptive lasso and its oracle properties,” Journal of the American Statistical Association, vol. 101, no. 476, pp. 1418–1429, 2006.
[41]
H. Wang, R. Li, and C. Tsai, “Tuning parameter selectors for the smoothly clipped absolute deviation method,” Biometrika, vol. 94, no. 3, pp. 553–568, 2007.
[42]
Y. Zhang, R. Li, and C. Tsai, “Regularization parameter selections via generalized information criterion,” Journal of the American Statistical Association, vol. 105, no. 489, pp. 312–323, 2010.
[43]
Z. Pang, B. Lin, and J. Jiang, “Regularization parameter selections with divergent and NP-dimensionality via bootstrapping,” Australian & New Zealand Journal of Statistics. In press.
[44]
E. K. Melcon, “On optimized shrinkage variable selection in generalized linear models,” Journal of Statistical Computation and Simulation. Revised.
[45]
J. Fan and J. Lv, “Sure independence screening for ultrahigh dimensional feature space,” Journal of the Royal Statistical Society B, vol. 70, no. 5, pp. 849–911, 2008.
[46]
L. A. Hindorff, P. Sethupathy, H. A. Junkins et al., “Potential etiologic and functional implications of genome-wide association loci for human diseases and traits,” Proceedings of the National Academy of Sciences of the United States of America, vol. 106, no. 23, pp. 9362–9367, 2009.
[47]
S. Müller, J. L. Scealy, and A. H. Welsh, “Model selection in linear mixed models,” Statistical Science, vol. 28, no. 2, pp. 135–167, 2013.
[48]
J. Jiang, T. Nguyen, and J. S. Rao, “The E-MS algorithm: model selection with incomplete data,” Journal of the American Statistical Association, 2013.
[49]
H. D. Bondell, A. Krishna, and S. K. Ghosh, “Joint variable selection for fixed and random effects in linear mixed-effects models,” Biometrics, vol. 66, no. 4, pp. 1069–1077, 2010.
[50]
A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the Royal Statistical Society B: Methodological, vol. 39, no. 1, pp. 1–38, 1977.
[51]
J. G. Ibrahim, H. Zhu, R. I. Garcia, and R. Guo, “Fixed and random effects selection in mixed effects models,” Biometrics, vol. 67, no. 2, pp. 495–503, 2011.
[52]
K. Hu, J. Choi, J. Jiang, and A. Sim, “Best predictive regularized modelling for mixed effects in high-speed network data,” Tech. Rep., Department of Statistics, University of California, Davis, Calif, USA, 2013.
[53]
J. R. Hershey and P. A. Olsen, “Approximating the Kullback Leibler divergence between Gaussian mixture models,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '07), vol. 4, pp. IV317–IV320, Honolulu, Hawaii, USA, April 2007.
[54]
J. Jiang, T. Nguyen, and J. S. Rao, “Fence method for nonparametric small area estimation,” Survey Methodology, vol. 36, no. 1, pp. 3–11, 2010.
