MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction (2110.06651v3)

Abstract: Keyphrase extraction (KPE) automatically extracts phrases in a document that provide a concise summary of the core content, which benefits downstream information retrieval and NLP tasks. Previous state-of-the-art (SOTA) methods select candidate keyphrases based on the similarity between learned representations of the candidates and the document. They suffer performance degradation on long documents due to discrepancy between sequence lengths which causes mismatch between representations of keyphrase candidates and the document. In this work, we propose a novel unsupervised embedding-based KPE approach, Masked Document Embedding Rank (MDERank), to address this problem by leveraging a mask strategy and ranking candidates by the similarity between embeddings of the source document and the masked document. We further develop a KPE-oriented BERT (KPEBERT) model by proposing a novel self-supervised contrastive learning method, which is more compatible to MDERank than vanilla BERT. Comprehensive evaluations on six KPE benchmarks demonstrate that the proposed MDERank outperforms state-of-the-art unsupervised KPE approach by average 1.80 $F1@15$ improvement. MDERank further benefits from KPEBERT and overall achieves average 3.53 $F1@15$ improvement over the SOTA SIFRank. Our code is available at \url{https://github.com/LinhanZ/mderank}.

• The paper introduces MDERank, a novel approach that ranks keyphrases based on the semantic shift observed when they are masked in a document.
• It employs contextualized embeddings from models like BERT and integrates a self-supervised contrastive learning model, KPEBERT, to enhance extraction accuracy.
• Evaluations reveal improvements of up to 3.53 in F1@15 on benchmark datasets, highlighting its robust performance, especially in long document processing.

Overview of MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction

The paper "MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction" introduces a novel method aimed at improving the efficacy of unsupervised keyphrase extraction (KPE). KPE is a crucial aspect of natural language processing, serving to automatically summarize documents and benefiting tasks such as information retrieval and summarization. Traditional state-of-the-art KPE methods often face challenges when dealing with long documents due to mismatches in representations of candidate keyphrases and the document itself. The MDERank method, elaborated in this paper, seeks to overcome these limitations through an innovative approach that compares embeddings of the original document with those of a variant in which occurrences of candidate keyphrases are masked.

Methodology

MDERank is predicated on the assumption that the absence of keyphrases significantly alters the semantic representation of a document. The technique involves masking all occurrences of candidate keyphrases within a document and calculating the semantic deviation between the original document and its masked version. This corresponds to the principle that keyphrases significantly impact document semantics. The candidates are then ranked based on the magnitude of semantic change caused by their masking, where a larger change indicates a more crucial keyphrase. Compared to traditional phrase-document methods, the document-document similarity measurement in MDERank is facilitated by contextualized representation models, notably BERT, thus ensuring those semantic similarities are captured reliably.

Further extending the capabilities of MDERank, the authors propose the development of a specialized pre-trained LLM, KPEBERT. This model integrates a novel self-supervised contrastive learning approach, enabling it to improve the ranking efficacy by discerning between documents with and without keyphrases more effectively.

Results

The MDERank method demonstrated definitive improvements over existing unsupervised keyphrase extraction methods. Evaluations across six benchmark datasets show that MDERank presents superior performance, achieving an average improvement of 1.80 in F1@15 over the best current unsupervised approach, SIFRank, and further enhanced when using KPEBERT with an average improvement of 3.53 in F1@15. These results underscore the robustness of MDERank, particularly in handling longer documents compared to previous solutions.

Implications and Future Directions

MDERank suggests a shift in the approach to unsupervised keyphrase extraction, providing a framework to tackle issues associated with long document processing. The paper highlights potential applications that benefit from more accurate and context-aware keyphrase extraction, paving the way for advancements in information retrieval systems and automated summarization tools.

In terms of theoretical implications, the MDERank model encourages further exploration into using masked LLMs for semantic similarity tasks, potentially influencing developments in related areas such as sentiment analysis and topic modeling.

Future work may include refining the self-supervised learning mechanisms used in KPEBERT, enhancing sampling strategies during training sequences to reduce biases from initial unsupervised methods. Additionally, investigation into other contextual embedding models, such as efficient transformers with longer sequence capabilities, can further improve results, especially in handling lengthy texts.

Overall, MDERank and its associated developments reflect a significant step forward in the domain of unsupervised keyphrase extraction, showcasing promising results and encouraging subsequent innovative research in the field.