Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction (2308.08739v2)

Published 17 Aug 2023 in cs.CL

Abstract: Keyphrase extraction (KPE) is an important task in Natural Language Processing for many scenarios, which aims to extract keyphrases that are present in a given document. Many existing supervised methods treat KPE as sequential labeling, span-level classification, or generative tasks. However, these methods lack the ability to utilize keyphrase information, which may result in biased results. In this study, we propose Diff-KPE, which leverages the supervised Variational Information Bottleneck (VIB) to guide the text diffusion process for generating enhanced keyphrase representations. Diff-KPE first generates the desired keyphrase embeddings conditioned on the entire document and then injects the generated keyphrase embeddings into each phrase representation. A ranking network and VIB are then optimized together with rank loss and classification loss, respectively. This design of Diff-KPE allows us to rank each candidate phrase by utilizing both the information of keyphrases and the document. Experiments show that Diff-KPE outperforms existing KPE methods on a large open domain keyphrase extraction benchmark, OpenKP, and a scientific domain dataset, KP20K.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The world wide web conference, pages 2551–2557.
  2. Semeval 2017 task 10: Scienceie-extracting keyphrases and relations from scientific publications. arXiv preprint arXiv:1704.02853.
  3. Florian Boudin. 2018. Unsupervised keyphrase extraction with multipartite graphs. arXiv preprint arXiv:1803.08721.
  4. A text feature based automatic keyword extraction method for single documents. In European conference on information retrieval, pages 684–691. Springer.
  5. Keyphrase generation with correlation constraints. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4057–4066.
  6. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  7. Haoran Ding and Xiao Luo. 2021. Attentionrank: Unsupervised keyphrase extraction using self and cross attentions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1919–1928.
  8. Samhaa R El-Beltagy and Ahmed Rafea. 2009. Kp-miner: A keyphrase extraction system for english and arabic documents. Information systems, 34(1):132–144.
  9. Corina Florescu and Cornelia Caragea. 2017a. A new scheme for scoring phrases in unsupervised keyphrase extraction. In Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Aberdeen, UK, April 8-13, 2017, Proceedings 39, pages 477–483. Springer.
  10. Corina Florescu and Cornelia Caragea. 2017b. Positionrank: An unsupervised approach to keyphrase extraction from scholarly documents. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1105–1115.
  11. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933.
  12. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303.
  13. Neural math word problem solver with reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 213–223, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  14. Learning fine-grained expressions to solve math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 805–814, Copenhagen, Denmark. Association for Computational Linguistics.
  15. Using intermediate representations to solve math word problems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 419–428, Melbourne, Australia. Association for Computational Linguistics.
  16. Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 216–223.
  17. Automatic keyphrase extraction from scientific articles. Language resources and evaluation, 47:723–742.
  18. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
  19. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761.
  20. Large dataset for keyphrases extraction.
  21. Learning rich representation of keyphrases from text. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 891–906.
  22. Towards topic-aware slide generation for academic papers with unsupervised mutual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13243–13251.
  23. Keywords-guided abstractive sentence summarization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8196–8203.
  24. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343.
  25. Xiang Lisa Li and Jason Eisner. 2019. Specializing word embeddings (for parsing) by information bottleneck. arXiv preprint arXiv:1910.00163.
  26. Unsupervised keyphrase extraction by jointly modeling local and global context. arXiv preprint arXiv:2109.07293.
  27. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  28. Scientific information extraction with semi-supervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2641–2651.
  29. Key2vec: Automatic ranked keyphrase extraction from scientific articles using phrase embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 634–639.
  30. Chatgpt vs state-of-the-art models: A benchmarking study in keyphrase generation task. arXiv preprint arXiv:2304.14177.
  31. Human-competitive tagging using automatic keyphrase extraction. Association for Computational Linguistics.
  32. Deep keyphrase generation. arXiv preprint arXiv:1704.06879.
  33. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411.
  34. Keyphrase extraction with span-based feature representations. arXiv preprint arXiv:2002.05407.
  35. Span-based named entity recognition by generating and compressing information. arXiv preprint arXiv:2302.05392.
  36. Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In International conference on Asian digital libraries, pages 317–326. Springer.
  37. OpenAI. 2022. ChatGPT.
  38. Principled paraphrase generation with parallel corpora. arXiv preprint arXiv:2205.12213.
  39. Martin F Porter. 1980. An algorithm for suffix stripping. Program, 14(3):130–137.
  40. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  41. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695.
  42. Keyphrase extraction as sequence labeling using contextualized embeddings. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42, pages 328–335. Springer.
  43. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR.
  44. Hyperbolic relevance matching for neural keyphrase extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5710–5720.
  45. Large language models as zero-shot keyphrase extractor: A preliminary empirical study. arXiv preprint arXiv:2312.15156.
  46. Importance estimation from multiple perspectives for keyphrase extraction. arXiv preprint arXiv:2110.09749.
  47. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.
  48. Capturing global informativeness in open domain keyphrase extraction. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13–17, 2021, Proceedings, Part II 10, pages 275–287. Springer.
  49. Sifrank: a new baseline for unsupervised keyphrase extraction based on pre-trained language model. IEEE Access, 8:10896–10906.
  50. The information bottleneck method. arXiv preprint physics/0004057.
  51. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
  52. Information bottleneck through variational glasses. arXiv preprint arXiv:1912.00830.
  53. Miner: Improving out-of-vocabulary named entity recognition from an information theoretic perspective. arXiv preprint arXiv:2204.04391.
  54. Incorporating multimodal information in open-domain web keyphrase extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1790–1800.
  55. Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle. arXiv preprint arXiv:1909.07405.
  56. Open domain web keyphrase extraction beyond language modeling. arXiv preprint arXiv:1911.02671.
  57. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796.
  58. Seqdiffuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:2212.10325.
  59. One size does not fit all: Generating and evaluating variable number of keyphrases. arXiv preprint arXiv:1810.05241.
  60. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  61. Improving the adversarial robustness of nlp models by information bottleneck. arXiv preprint arXiv:2206.05511.
  62. Diffusum: Generation enhanced extractive summarization with diffusion. arXiv preprint arXiv:2305.01735.
  63. Mderank: A masked document embedding rank approach for unsupervised keyphrase extraction. arXiv preprint arXiv:2110.06651.
  64. Keyphrase extraction using deep recurrent neural networks on twitter. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 836–845.
  65. Qingyu Zhou and Danqing Huang. 2019. Towards generating math word problems from equations and topics. In Proceedings of the 12th International Conference on Natural Language Generation, pages 494–503, Tokyo, Japan. Association for Computational Linguistics.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now