In this project, we consider obtaining Fourier
features via more efficient sampling schemes to approximate the kernel in LFMs.
A latent force model(LFM) is a Gaussian process whose covariance functions follow an
Exponentiated Quadratic(EQ) form, and the solutions for the cross-covariance are expensive due
to the computational complexity. To reduce the complexity of mathematical
expressions, random Fourier features(RFF) are applied to approximate the EQ kernel. Usually, the
random Fourier features are implemented with Monte Carlo sampling, but this
project proposes replacing the Monte-Carlo method with the Quasi-Monte Carlo(QMC) method. The first-order and second-order
models’ experiment results demonstrate the decrease in NLPD and NMSE, which
revealed that the models with QMC approximation have better performance.
References
[1]
Alvarez, M., Luengo, D. and Lawrence, N.D. (2009) Latent Force Models. Journal of Machine Learning Research, 5, 9-16.
[2]
Gao, P., Honkela, A., Rattray, M. and Lawrence, N.D. (2008) Gaussian Process Modelling of Latent Chemical Species: Applications to Inferring Transcription Factor Activities. Bioinformatics, 24, i70-i75. https://doi.org/10.1093/bioinformatics/btn278
[3]
Alvarez, M.A., Luengo, D. and Lawrence, N.D. (2013) Linear Latent Force Models Using Gaussian Processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 2693-2705. https://doi.org/10.1109/TPAMI.2013.86
[4]
Alvarez, M.A. and Lawrence, N.D. (2011) Computationally Efficient Convolved Multiple Output Gaussian Processes. Journal of Machine Learning Research, 12, 1459-1500.
[5]
Ghosh, S., Reece, S., Rogers, A., Roberts, S., Malibari, A. and Jennings, N.R. (2015) Modeling the Thermal Dynamics of Buildings: A Latent-Force-Model-Based Approach. ACM Transactions on Intelligent Systems and Technology (TIST), 6, 1-27. https://doi.org/10.1145/2629674
[6]
Rahimi, A., Recht, B., et al. (2007) Random Features for Large-Scale Kernel Machines. Conference on Neural Information Processing Systems, 3, 5.
[7]
Williams, C.K. and Rasmussen, C.E. (2006) Gaussian Processes for Machine Learning. Volume 2, MIT Press, Cambridge. https://doi.org/10.7551/mitpress/3206.001.0001
[8]
Wilson, A. and Adams, R. (2013) Gaussian Process Kernels for Pattern Discovery and Extrapolation. International Conference on Machine Learning, 28, 1067-1075.
[9]
Svensen, M. and Bishop, C.M. (2007) Pattern Recognition and Machine Learning. Springer, Berlin.
[10]
Mitchell, T.M., et al. (1997) Machine Learning. McGraw-Hill Education, New York.
[11]
Scholkopf, B., Smola, A. and Muller, K.-R. (1998) Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation, 10, 1299-1319. https://doi.org/10.1162/089976698300017467
[12]
Scholkopf, B., Mika, S., Burges, C.J.C., Knirsch, P., Muller, K.-R., Ratsch, G. and Smola, A.J. (1999) Input Space versus Feature Space in Kernel-Based Methods. IEEE Transactions on Neural Networks, 10, 1000-1017. https://doi.org/10.1109/72.788641
[13]
Muller, K.-R., Mika, S., Ratsch, G., Tsuda, K. and Scholkopf, B. (2001) An Introduction to Kernel-Based Learning Algorithms. IEEE Transactions on Neural Networks, 12, 181-201. https://doi.org/10.1109/72.914517
[14]
Vapnik, V. (2013) The Nature of Statistical Learning Theory. Springer Science & Business Media, Berlin.
[15]
Vapnik, V.N. (1999) An Overview of Statistical Learning Theory. IEEE Transactions on Neural Networks, 10, 988-999. https://doi.org/10.1109/72.788640
[16]
Smola, A.J. and Scholkopf, B. (1998) Learning with Kernels, Volume 4. MIT Press, Cambridge.
[17]
Shawe-Taylor, J., Cristianini, N., et al. (2004) Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511809682
[18]
Aizerman, M.A., Braverman, E.M. and Rozonoer, L.I. (1964) Theoretical Foundations of Potential Function Method in Pattern Recognition. Automation and Remote Control, 25, 917-936.
[19]
Mercer, J. (1909) Xvi. Functions of Positive and Negative Type, and Their Connection the Theory of Integral Equations. Philosophical Transactions of the Royal Society of London. Series A, 209, 415-446. https://doi.org/10.1098/rsta.1909.0016
[20]
Aronszajn, N. (1950) Theory of Reproducing Kernels. Transactions of the American Mathematical Society, 68, 337-404. https://doi.org/10.1090/S0002-9947-1950-0051437-7
[21]
Poggio, T. and Shelton, C.R. (2002) On the Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 39, 1-49. https://doi.org/10.1090/S0273-0979-01-00923-5
[22]
Scholkopf, B., Smola, A.J., Bach, F., et al. (2002) Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge.
[23]
Wahba, G. (1990) Spline Models for Observational Data. SIAM, Philadelphia. https://doi.org/10.1137/1.9781611970128
[24]
Cristianini, N., Shawe-Taylor, J., et al. (2000) An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511801389
Avron, H., Sindhwani, V., Yang, J.Y. and Mahoney, M.W. (2016) Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels. The Journal of Machine Learning Research, 17, 4096-4133.
[27]
Bochner, S. (1933) Monotone funktionen, stieltjessche integrale und harmonische analyse. Mathematische Annalen, 108, 378-410. https://doi.org/10.1007/BF01452844
[28]
Wang, F.S. and Lu, M.Y. (2013) Efficient Visual Tracking via Hamiltonian Monte Carlo Markov Chain. Computer Journal, 56, 1102-1112. https://doi.org/10.1093/comjnl/bxs141
[29]
Mazhdrakov, M., Benov, D. and Valkanov, N. (2018) The Monte Carlo Method: Engineering Applications. ACMO Academic Press, Cambridge.
[30]
Metropolis, N., et al. (1987) The Beginning of the Monte Carlo Method. Los Alamos Science, 15, 125-130.
[31]
Rubinstein, R.Y. and Kroese, D.P. (2016) Simulation and the Monte Carlo Method. Volume 10, John Wiley & Sons, Hoboken. https://doi.org/10.1002/9781118631980
[32]
Feller, W. (1957) An Introduction to Probability Theory and Its Applications. Wiley, Hoboken.
[33]
Press, W.H., Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P. and Vetterling, W.T. (1989) Numerical Recipes in Pascal: The Art of Scientific Computing. Volume 1, Cambridge University Press, Cambridge.
[34]
Asmussen, S. and Glynn, P.W. (2007) Stochastic Simulation: Algorithms and Analysis. Volume 57, Springer Science & Business Media, Berlin. https://doi.org/10.1007/978-0-387-69033-9
Niederreiter, H. (1992) Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia. https://doi.org/10.1137/1.9781611970081
[37]
van der Corput, J.G. (1935) Verteilungsfunktionen. I. Proceedings of the Koninklijke Nederlandse Akademie van Wetenschappen, Amsterdam, Vol. 38, 813-821.
[38]
Gelman, A., Hwang, J. and Vehtari, A. (2014) Understanding Predictive Information Criteria for Bayesian Models. Statistics and Computing, 24, 997-1016. https://doi.org/10.1007/s11222-013-9416-2
[39]
http://www.bramblemet.co.uk
[40]
Nguyen, T.V., Bonilla, E.V., et al. (2014) Collaborative Multi-Output Gaussian Processes. Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI 2014, Quebec City, 23-27 July 2014, 643-652.
[41]
(2021)http://mocap.cs.cmu.edu/
[42]
Kocis, L. and Whiten, W.J. (1997) Computational Investigations of Low-Discrepancy Sequences. ACM Transactions on Mathematical Software (TOMS), 23, 266-294. https://doi.org/10.1145/264029.264064
[43]
Kuo, F.Y. and Nuyens, D. (2016) Application of Quasi-Monte Carlo Methods to Elliptic Pdes with Random Diffusion Coefficients: A Survey of Analysis and Implementation. Foundations of Computational Mathematics, 16, 1631-1696. https://doi.org/10.1007/s10208-016-9329-5
[44]
Joe, S. and Kuo, F.Y. (2008) Constructing Sobol Sequences with Better Two-Dimensional Projections. SIAM Journal on Scientific Computing, 30, 2635-2654. https://doi.org/10.1137/070709359
[45]
Dick, J., Kuo, F.Y. and Sloan, I.H. (2013) High-Dimensional Integration: The Quasi-Monte Carlo Way. Acta Numerica, 22, 133-288. https://doi.org/10.1017/S0962492913000044
