Global and Local Context Fusion in Heterogeneous Graph Neural Network for Summarizing Lengthy Scientific Documents

The primary objective of text summarization is to condense a document’s length while preserving its essential content. Extractive summarization methods are commonly used due to their effectiveness and straightforward presentation. However, a significant challenge lies in segmenting documents into distinct concepts and understanding how sentences interact, especially in complex materials such as scientific articles. This process entails identifying relationships between sentences and determining the most significant and informative content within extensive text collections. Traditional techniques often utilize pre-trained models like BERT, known for their ability to capture word context. Nonetheless, these models have limitations, including constrained input lengths and the computational intensity of self-attention mechanisms, which hinder their effectiveness in processing large-scale scientific texts. To address these challenges, we propose a computationally efficient Heterogeneous Graph Neural Network (HGNN) for the extractive summarization of lengthy scientific texts. This framework combines GloVe embeddings with Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) encoders. GloVe offers simple yet effective word embeddings, CNNs focus on capturing local word structures, and BiLSTMs identify long-range dependencies, allowing for flexible encoding of extensive texts. For global context and topic modeling, we utilize an enhanced version of Latent Dirichlet Allocation (LDA) to retain essential document attributes. In this model, words, sentences, and topics are represented as nodes in a heterogeneous graph, with TF-IDF values illustrating the relationships between edges. The graph is processed using a Graph Attention Network (GAT), which refines node representations by integrating both local and global information. This study represents the first instance of combining LDA with CNN and BiLSTM encoders in a Graph Attention-based model for summarizing scientific texts. Experimental results demonstrate that the proposed framework outperforms both BERT-based and non-BERT approaches on publicly available datasets from arXiv and PubMed. Our model achieves a ROUGE-1 score of 46.31, a ROUGE-2 score of 19.98, and a ROUGE-L score of 40.21 on the arXiv dataset. It performs even better on the PubMed dataset, attaining a ROUGE-1 score of 48.85, a ROUGE-2 score of 21.78, and a ROUGE-L score of 42.12. © 2013 IEEE.