|
|
基于T5-PEGASUS-PGN模型的中文新闻文本摘要生成方法
|
Abstract:
针对预训练模型训练任务与下游摘要生成任务存在差异、生成文本存在重复内容造成摘要可读性差的问题，基于T5-PEGASUS和指针生成网络，提出了一种自动摘要模型——T5-PEGASUS-PGN。首先利用T5-PEGASUS获取最符合原文语义的词向量表示，然后借助引入覆盖机制的指针生成网络，生成高质量、高可读的最终摘要。在公开的长文本数据集NLPCC2017的实验结果表明，与PGN模型、BERT-PGN等模型相比，结合更贴合下游摘要任务的预训练模型的T5-PEGASUS-PGN模型能够生成更符合原文语义、内容更加丰富的摘要并且能有效的抑制重复内容生成，同时Rouge评价指标Rouge-1提升至44.26%、Rouge-2提升至23.97%以及Rouge-L提至34.81%。
To address the challenges of differences between the training tasks of pretrained models and the downstream summary generation tasks, as well as the poor readability caused by repeated content in the generated texts, an automatic summary model called T5-PEGASUS-PGN is proposed based on T5-PEGASUS and pointer generation networks. This model first utilizes T5-PEGASUS to obtain the most semantically consistent word vector representation. Then, with the help of the pointer gener-ation network that applies the coverage mechanism, high-quality and readable final summaries are generated. Experimental results on the public long-text dataset NLPCC2017 show that compared with models such as PGN and BERT-PGN, the T5-PEGASUS-PGN model, which combines a pretrained model that fits the downstream summary task better, can generate summaries that are more con-sistent with the original text semantics, contain richer content, and effectively suppresses repeated content generation. At the same time, we have raised the Rouge-1 metric to 44.26%, the Rouge-2 metric to 23.97%, and the Rouge-L metric to 34.81%.
| [1] | 李金鹏, 张闯, 陈小军, 等. 自动文本摘要研究综述[J]. 计算机研究与发展, 2021, 58(1): 1-21. |
| [2] | Luhn, H.P. (1958) The Automatic Creation of Literature Abstracts. IBM Journal of Research and Development, 2, 159-165. https://doi.org/10.1147/rd.22.0159 |
| [3] | Sutskever, I., Vinyals. O. and Le. Q.V. (2014) Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems, 27, 3104-3112. |
| [4] | Bahdanau, D., Cho, K. and Bengio, Y. (2014) Neural Machine Translation by Jointly Learning to Align and Translate. Computer Science, 1-16. |
| [5] | See, A., Liu. P.J. and Manning, C.D. (2017) Get to the Point: Summarization with Pointer-Generator Networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, July 2017, 1073-1083. https://doi.org/10.18653/v1/P17-1099 |
| [6] | 谭金源, 刁宇峰, 祁瑞华, 等. 基于BERT-PGN模型的中文新闻文本自动摘要生成[J]. 计算机应用, 2021, 41(1): 127-132. |
| [7] | Vaswani, A., Shazeer, N., Parmar, N., et al. (2017) Attention Is All You Need. arXiv: 1706.03762. |
| [8] | Zhang, J., Zhao, Y., Saleh, M., et al. (2020) PEGASUS: Pre-Training with Extracted Gap-Sentences for Abstractive Summarization. Proceedings of the 37th Interna-tional Conference on Machine Learning, Vol. 119, 13-18 July 2020, 11328-11339. |
| [9] | Yang, T.H., Lu, C.C. and Hsu, W.L. (2021) More than Extracting “Important” Sentences: The Application of PEGASUS. International Conference on Technologies and Applications of Artificial Intelligence, Taichung, 18-20 November 2021, 131-134. https://doi.org/10.1109/TAAI54685.2021.00032 |
| [10] | 张琪, 范永胜. 基于改进T5 PEGASUS模型的新闻文本摘要生成[J]. 电子科技, 2023, 36(12): 72-78. |
| [11] | Radford, A., Wu, J., Child, R., et al. (2019) Language Models Are Unsupervised Multitask Learners. OpenAI Blog, 1, 9. |
| [12] | Devlin, J., Chang, M.W., Lee, K., et al. (2018) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805. |
| [13] | Lin, C.Y. (2004) ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Workshop on Text Summarization Branches Out. Stroudsburg: Association for Computational Linguistics, Barcelona, July 2004, 74-81. |
