Revisiting Akaike’s Final Prediction Error and the Generalized Cross Validation Criteria in Regression from the Same Perspective: From Least Squares to Ridge Regression and Smoothing Splines
In
regression, despite being both aimed at estimating the Mean Squared Prediction
Error (MSPE), Akaike’s Final Prediction Error (FPE) and the Generalized Cross
Validation (GCV) selection criteria are usually derived from two quite
different perspectives. Here, settling on the most commonly accepted definition
of the MSPE as the expectation of the squared prediction error loss, we provide
theoretical expressions for it, valid for any linear model (LM) fitter, be it
under random or non random designs. Specializing these MSPE expressions for
each of them, we are able to derive closed formulas of the MSPE for some of the
most popular LM fitters: Ordinary Least Squares (OLS), with or without a full
column rank design matrix; Ordinary and Generalized Ridge regression, the
latter embedding smoothing splines fitting. For each of these LM fitters, we
then deduce a computable estimate of the MSPE which turns out to coincide with
Akaike’s FPE. Using a slight variation, we similarly get a class of MSPE
estimates coinciding with the classical GCV formula for those same LM fitters.
References
[1]
Akaike, H. (1970) Statistical Predictor Identification. Annals of the Institute of Statistical Mathematics, 22, 203-217. https://doi.org/10.1007/BF02506337
[2]
Borra, S. and Di Ciaccio, A. (2010) Measuring the Prediction Error. A Comparison of Cross-Validation, Bootstrap and Covariance Penalty Methods. Computational Statistics & Data Analysis, 54, 2976-2989. https://doi.org/10.1016/j.csda.2010.03.004
[3]
Craven, P. and Wahba, G. (1979) Smoothing Noisy Data with Spline Functions: Estimating the Correct Degree of Smoothing by the Method of Generalized Cross-Validation. Numerische Mathematik, 31, 377-403.
https://doi.org/10.1007/BF01404567
[4]
Li, K.-C. (1985) From Stein’s Unbiased Risk Estimates to the Method of Generalized Cross Validation. Journal of the Japan Statistical Society, 38, 119-130.
[5]
Rosset, S. and Tibshirani, R. (2020) From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation. JASA, 115, 138-151. https://doi.org/10.1080/01621459.2018.1424632
[6]
Hastie, T., Tibshirani, R. and Friedman, J. (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd Edition, Springer-Verlag, New York.
[7]
Burman, P. (1989) A Comparative Study of Ordinary Cross-Validation, v-Fold Cross-Validation and Repeated Learning-Testing Methods. Biometrika, 76, 503-514. https://doi.org/10.1093/biomet/76.3.503
[8]
Hastie, T. and Tibshirani, R. (1990) Generalized Additive Models. Monographs on Statistics and Applied Probability. Chapman and Hall, London.
[9]
Eubank, R.L. (1999) Nonparametric Regression and Spline Smoothing. 2nd Edition, Marcel Dekker, New York. https://doi.org/10.1201/9781482273144
[10]
McQuarrie, A.D.R. and Tsai, C.-L. (1998) Regression and Time Series Model Selection. World Scientific Publishing Co. Re. Ltd, Singapore.
https://doi.org/10.1142/3573
[11]
Breiman, L. and Spector, P. (1992) Submodel Selection and Evaluation in Regression. The X-Random Case. International Statistical Review, 60, 291-319.
https://doi.org/10.2307/1403680
[12]
Leeb, H. (2008) Evaluation and Selection of Models for Out-of-Sample Prediction When the Sample Size Is Small Relative to the Complexity of the Data-Generating Process. Bernoulli, 14, 661-690. https://doi.org/10.3150/08-BEJ127
[13]
Dicker, L.H. (2013) Optimal Equivariant Prediction for High-Dimensional Linear Models with Arbitrary Predictor Covariance. The Electronic Journal of Statistics, 7, 1806-1834. https://doi.org/10.1214/13-EJS826
[14]
Dobriban, E. and Wager, S. (2018) High-Dimensional Asymptotics of Prediction. Annals of Statistics, 46, 247-279. https://doi.org/10.1214/17-AOS1549
[15]
Lukas, M.A. (2014) Performance Criteria and Discrimination of Extreme Undersmoothing in Nonparametric Regression. Journal of Statistical Planning and Inference, 153, 56-74. https://doi.org/10.1016/j.jspi.2014.05.006
[16]
Lukas, M.A., de Hoog, F.R. and Anderssen, R.S. (2015) Practical Use of Robust GCV and Modified GCV for Spline Smoothing. Computational Statistics, 31, 269-289. https://doi.org/10.1007/s00180-015-0577-7
[17]
Kitagawa, G. (2008) Contributions of Professor Hirotogu Akaike in Statistical Science. Journal of the Japan Statistical Society, 38, 119-130.
https://doi.org/10.14490/jjss.38.119
[18]
Tibshirani, R. (1996) Regression Shrinkage and Selection via the Lasso. JRSS, Series B, 58, 267-288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
[19]
Hoerl, A.E. and Kennard, R. (1970) Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12, 55-67.
https://doi.org/10.1080/00401706.1970.10488634
[20]
Searle, S.R. (1971) Linear Models. John Wiley & Sons, New York.
[21]
Rao, C.R. (1973) Linear Statistical Inference and Its Applications. 2nd Edition, Wiley, New York. https://doi.org/10.1002/9780470316436
[22]
Rao, C.R., Toutenburg, H., et al. (2008) Linear Models and Generalizations: Least Squares and Alternatives. 3rd Edition, Springer-Verlag, Berlin.
[23]
Olive, D.J. (2017) Linear Regression. Springer International Publishing, Berlin.
https://doi.org/10.1007/978-3-319-55252-1
[24]
Moore, E.H. (1920) On the Reciprocal of the General Algebraic Matrix (Abstract). Bulletin of the AMS, 26, 394-395.
[25]
Penrose, R. (1955) A Generalized Inverse for Matrices. Proceedings—Cambridge Philosophical Society, 51, 406-413. https://doi.org/10.1017/S0305004100030401
[26]
Rao, C.R. (1962) A Note on a Generalized Inverse of a Matrix with Applications to Problems in Mathematical Statistics. JRSS, Series B, 24, 152-158.
https://doi.org/10.1111/j.2517-6161.1962.tb00447.x
[27]
Kendrick, D.A. (2002)) Stochastic Control for Economic Models. 2nd Edition, VTEX Ltd., Vilnius.
[28]
Wahba, G. (1978) Improper Priors, Spline Smoothing and the Problem of Guarding against Model Errors in Regression. JRSS, Series B, 40, 364-372.
https://doi.org/10.1111/j.2517-6161.1978.tb01050.x
[29]
Härdle, W., Hall, P. and Marron, J.S. (1988) How Far Are Automatically Chosen Regression Smoothing Parameters from Their Optimum-(with Discussion). JASA, 83, 86-89. https://doi.org/10.2307/2288922
[30]
Hurvich, C.M., Simonoff, J.S. and Tsai, C.-L. (1998) Smoothing Parameter Selection in Nonparametric Regression Using an Improved Akaike Information Criterion. JRSS, Series B, 60, 271-293. https://doi.org/10.1111/1467-9868.00125
[31]
Cummins, D.J., Filloon, T.G. and Nychka, D. (2001) Confidence Intervals for Nonparametric Curve Estimates: Toward More Uniform Pointwise Coverage. JASA, 96, 233-246. https://doi.org/10.1198/016214501750332811
[32]
Kim, Y.-J. and Gu, C. (2004) Smoothing Spline Gaussian Regression: More Scalable Computation via Efficient Approximation. JRSS, Series B, 66, 337-356.
https://doi.org/10.1046/j.1369-7412.2003.05316.x
[33]
Lukas, M.A. (2006) Robust Generalized Cross-Validation for Choosing the Regularization Parameter. Inverse Probability, 22, 1883-1902.
https://doi.org/10.1088/0266-5611/22/5/021
[34]
Lukas, M.A. (2008) Strong Robust Generalized Cross-Validation for Choosing the Regularization Parameter. Inverse Probability, 24, Article ID: 034006.
https://doi.org/10.1088/0266-5611/24/3/034006
