|
|
注意力机制在图神经网络模型中的算法研究
|
Abstract:
图神经网络(Graph Neural Networks，简称GNN)是一种用于处理图结构数据的深度学习模型。随着图数据的广泛应用，如社交网络、推荐系统和生物信息学等领域，研究者们致力于提升GNN的性能和表达能力。注意力机制作为一种强大的工具，已广泛应用于深度学习领域，可以帮助模型更加有效地学习和利用关键信息。本文首先分别对图神经网络和注意力机制进行简单的概念阐述，然后详细介绍了图神经网络中与注意力机制结合的传统图注意力网络，及近几年提出的一些图注意力网络模型相关改进和变体。最后对未来发展方向以及研究趋势进行总结与展望。
Graph Neural Networks (GNN) are a deep learning model for processing graph-structured data. With the wide application of graph data, such as social networks, recommendation systems, and bi-oinformatics, researchers are working to improve the performance and expression of GNN. As a powerful tool, attention mechanisms have been widely used in the field of deep learning to help models learn and utilize critical information more efficiently. In this paper, the concepts of graph neural network and attention mechanism are briefly described, and then the traditional graph at-tention network combined with attention mechanism in graph neural network is introduced in de-tail. Some improvements and variants of graph attention network models have been proposed in recent years. Finally, the future development direction and research trend are summarized and prospected.
| [1] | Gori, M., Monfardini, G. and Scarselli, F. (2005) A New Model for Learning in Graph Domains. Proceedings of the 2005 IEEE International Joint Conference on Neural Networks, Montreal, QC, 31 July-4 August 2005, 729-734.
https://doi.org/10.1109/IJCNN.2005.1555942 |
| [2] | Kipf, T.N. and Welling, M. (2016) Semi-Supervised Classification with Graph Convolutional Networks. Arxiv:1609.02907 |
| [3] | Lv, Q., Ding, M., Liu, Q., et al. (2021) Are We Really Making Much Progress? Revisiting, Benchmarking and Refining Heterogeneous Graph Neural Networks. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore, 14-18 August 2021, 1150-1160. https://doi.org/10.1145/3447548.3467350 |
| [4] | Su, X., Xue, S. and Liu, F. (2022) A Comprehensive Survey on Commu-nity Detection with Deep Learning. IEEE Transactions on Neural Networks and Learning Systems, 1-21. https://doi.org/10.1109/TNNLS.2021.3137396 |
| [5] | Ying, C., Cai, T., Luo, S., et al. (2021) Do Transformers Really Per-form Badly for Graph Representation? Advances in Neural Information Processing Systems, 34, 28877-28888. |
| [6] | Roweis, S.T. and Saul, L.K. (2000) Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science, 290, 2323-2326. https://doi.org/10.1126/science.290.5500.2323 |
| [7] | Perozzi, B., Al-Rfou, R. and Skiena, S. (2014) Deepwalk: Online Learning of Social Representations. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, 24-27 August 2014, 701-710. https://doi.org/10.1145/2623330.2623732 |
| [8] | Tang, J., Qu, M., Wang, M., et al. (2015) LINE: Large-Scale Information Network Embedding. Proceedings of the 24th International Con-ference on World Wide Web, Florence, 18-22 May 2015, 1067-1077.
https://doi.org/10.1145/2736277.2741093 |
| [9] | Grover, A. and Leskovec, J. (2016) node2vec: Scalable Feature Learning for Networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, 13-17 August 2016, 855-864. https://doi.org/10.1145/2939672.2939754 |
| [10] | Yang, R., Shi, J., Yang, Y., et al. (2021) Effective and Scalable Clustering on Massive Attributed Graphs. Proceedings of the Web Conference, Ljubljana, 19-23 April 2021, 3675-3687. https://doi.org/10.1145/3442381.3449875 |
| [11] | Bruna, J., Zaremba, W., Szlam, A., et al. (2013) Spectral Networks and Locally Connected Networks on Graphs. Arxiv:1312.6203 https://doi.org/10.48550/arXiv.1312.6203 |
| [12] | Defferrard, M., Bresson, X. and Vandergheynst, P. (2016) Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. Advances in Neural Information Processing Systems, 29, 3884-3852. |
| [13] | Gilmer, J., Schoenholz, S.S., Riley, P.F., et al. (2017) Neural Message Passing for Quantum Chemistry. In-ternational Conference on Machine Learning, 70, 1263-1272. |
| [14] | Battaglia, P.W., Hamrick, J.B., Bapst, V., et al. (2018) Re-lational Inductive Biases, Deep Learning, and Graph Networks. Arxiv:1806.01261 |
| [15] | Hamilton, W., Ying, Z. and Leskovec, J. (2017) Inductive Representation Learning on Large Graphs. Advances in Neural Information Processing Systems, 30, 1024-1034. |
| [16] | Veli？kovi？, P., Cucurull, G., Casanova, A., et al. (2017) Graph Attention Networks. Arxiv:1710.10903 |
| [17] | Xu, K., Hu, W., Leskovec, J., et al. (2018) How Powerful Are Graph Neural Networks? Arxiv:1810.00826 |
| [18] | 申翔翔, 侯新文, 尹传环. 深度强化学习中状态注意力机制的研究[J]. 智能系统学报, 2020, 15(2): 317-322. |
| [19] | 高芬, 苏依拉, 牛向华, 等. 基于Transformer的蒙汉神经机器翻译研究[J]. 计算机应用与软件, 2020, 37(2): 141-146+225. |
| [20] | Chaudhari, S., Mithal, V., Polatkan, G., et al. (2021) An Attentive Survey of Attention Mod-els. ACM Transactions on Intelligent Systems and Technology (TIST), 12, 1-32. https://doi.org/10.1145/3465055 |
| [21] | 任欢, 王旭光. 注意力机制综述[J]. 计算机应用, 2021, 41(S1): 1-6. |
| [22] | Tsotsos, J.K., Culhane, S.M., Wai, W.Y.K., et al. (1995) Modeling Visual Attention via Selective Tuning. Artificial Intelligence, 78, 507-545. https://doi.org/10.1016/0004-3702(95)00025-9 |
| [23] | Itti, L., Koch, C. and Niebur, E. (1998) A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 1254-1259. https://doi.org/10.1109/34.730558 |
| [24] | Mnih, V., Heess, N. and Graves, A. (2014) Recurrent Models of Visual Attention. Advances in Neural Information Processing Systems, 27, 1106-1114. |
| [25] | Bahdanau, D., Cho, K. and Bengio, Y. (2014) Neu-ral Machine Translation by Jointly Learning to Align and Translate. Arxiv:1409.0473 |
| [26] | Vaswani, A., Shazeer, N., Parmar, N., et al. (2017) Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 3104-3112. |
| [27] | Kitaev, N., Kaiser, ？. and Levskaya, A. (2020) Reformer: The Efficient Transformer. Arxiv:2001.04451 |
| [28] | 王明申, 牛斌, 马利. 一种基于词级权重的Transformer模型改进方法[J]. 小型微型计算机系统, 2019, 40(4): 744-748. |
| [29] | Zhan, Q., Cai, X., Chen, C., et al. (2017) Commented Content Classification with Deep Neural Network Based on Attention Mechanism. Pro-ceedings of the 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference, Chongqing, 25-26 March 2017, 2016-2019. https://doi.org/10.1109/IAEAC.2017.8054369 |
| [30] | Hao, S., Lee, D.-H. and Zhao, D. (2019) Sequence to Sequence Learning with Attention Mechanism for Short-Term Passenger Flow Prediction in Large-Scale Metro System. Transportation Research Part C: Emerging Technologies, 107, 287-300. https://doi.org/10.1016/j.trc.2019.08.005 |
| [31] | 周才东, 曾碧卿, 王盛玉, 等. 结合注意力与卷积神经网络的中文摘要研究[J]. 计算机工程与应用, 2019, 55(8): 132-137. |
| [32] | Salehi, A. and Davulcu, H. (2020) Graph Attention Auto-Encoders. 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), Baltimore, MD, 9-11 November 2020, 989-996.
https://doi.org/10.1109/ICTAI50040.2020.00154 |
| [33] | Zhang, J., Shi, X., Xie, J., et al. (2018) GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs. Arxiv:1803.07294 |
| [34] | Busbridge, D., Sherburn, D., Cavallo, P., et al. (2019) Relational Graph Attention Networks. Arxiv:1904.05811 |
| [35] | Schlichtkrull, M., Kipf, T.N., Bloem, P., et al. (2018) Modeling Relational Data with Graph Convolutional Networks. The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, 3-7 June 2018, 593-607.
https://doi.org/10.1007/978-3-319-93417-4_38 |
| [36] | Li, Y., Tarlow, D., Brockschmidt, M., et al. (2015) Gated Graph Se-quence Neural Networks. Arxiv:1511.05493 |
| [37] | Lee, J., Lee, I. and Kang, J. (2019) Self-Attention Graph Pooling. Interna-tional Conference on Machine Learning, 97, 3734-3743. |
| [38] | Do, K., Tran, T., Nguyen, T., et al. (2019) Attentional Multilabel Learning over Graphs: A Message Passing Approach. Machine Learning, 108, 1757-1781. https://doi.org/10.1007/s10994-019-05782-6 |
| [39] | Pham, T., Tran, T., Phung, D., et al. (2017) Column Networks for Col-lective Classification. Proceedings of the AAAI Conference on Artificial Intelligence, 31, 2485-2491. https://doi.org/10.1609/aaai.v31i1.10851 |
| [40] | Lee, J.B., Rossi, R. and Kong, X. (2018) Graph Classification Using Struc-tural Attention. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, 19-23 August 2018, 1666-1674. https://doi.org/10.1145/3219819.3219980 |
| [41] | Abu-El-Haija, S., Perozzi, B., Al-Rfou, R., et al. (2018) Watch Your Step: Learning Node Embeddings via Graph Attention. Advances in Neural Information Processing Systems, 31, 9180-9190. |
| [42] | Brody, S., Alon, U. and Yahav, E. (2021) How Attentive Are Graph Attention Net-works? Arxiv:2105.14491 |
| [43] | Javaloy, A., Sanchez-Martin, P., et al. (2022) Learnable Graph Convolutional Attention Net-works. Arxiv:2211.11853 |
| [44] | Zhang, W., Sheng, Z., Yin, Z., et al. (2022) Model Degradation Hinders Deep Graph Neural Networks. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington DC, 14-18 August 2022, 2493-2503. https://doi.org/10.1145/3534678.3539374 |
| [45] | Alon, U. and Yahav, E. (2021) On the Bottleneck of Graph Neural Networks and Its Practical Implications. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2006.05205 |
| [46] | Jin, W., Liu, X., Ma, Y., Aggarwal, C. and Tang, J. (2022) Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Washington DC, 14-18 August 2022, 709-719. https://doi.org/10.1145/3534678.3539445 |
| [47] | Zhao, H., Ma, S., Zhang, D., Deng, Z.-H. and Wei, F. (2023) Are More Layers Beneficial to Graph Transformers? International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2303.00579 |
| [48] | Lee, S.Y., Bu, F., Yoo, J., et al. (2023) Towards Deep Attention in Graph Neural Networks: Problems and Remedies. Arxiv:2306.02376 |
| [49] | Kim, D. and Oh, A. (2022) How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision. Arxiv:2204.04879 |
