Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task (2304.14177v2)

Published 27 Apr 2023 in cs.CL and cs.AI

Abstract: Transformer-based LLMs, including ChatGPT, have demonstrated exceptional performance in various natural language generation tasks. However, there has been limited research evaluating ChatGPT's keyphrase generation ability, which involves identifying informative phrases that accurately reflect a document's content. This study seeks to address this gap by comparing ChatGPT's keyphrase generation performance with state-of-the-art models, while also testing its potential as a solution for two significant challenges in the field: domain adaptation and keyphrase generation from long documents. We conducted experiments on six publicly available datasets from scientific articles and news domains, analyzing performance on both short and long documents. Our results show that ChatGPT outperforms current state-of-the-art models in all tested datasets and environments, generating high-quality keyphrases that adapt well to diverse domains and document lengths.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. A study on automatically extracted keywords in text categorization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 537–544, Sydney, Australia, July 2006. Association for Computational Linguistics. doi:10.3115/1220175.1220243. URL https://aclanthology.org/P06-1068.
  2. Corephrase: Keyphrase extraction for document clustering. In Petra Perner and Atsushi Imiya, editors, Machine Learning and Data Mining in Pattern Recognition, pages 265–274, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31891-0.
  3. Citation summarization through keyphrase extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 895–903, Beijing, China, August 2010. Coling 2010 Organizing Committee. URL https://aclanthology.org/C10-1101.
  4. Semeval 2017 task 10: Scienceie - extracting keyphrases and relations from scientific publications, 2017.
  5. Enhancing access to scholarly publications with surrogate resources. Scientometrics, 121(2):1129–1164, November 2019. doi:10.1007/s11192-019-03227-. URL https://ideas.repec.org/a/spr/scient/v121y2019i2d10.1007_s11192-019-03227-4.html.
  6. Keyphrase extraction in scientific publications. In Dion Hoe-Lian Goh, Tru Hoang Cao, Ingeborg Torvik Sølvberg, and Edie Rasmussen, editors, Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317–326, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-77094-7.
  7. Incorporating expert knowledge into keyphrase extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi:10.1609/aaai.v31i1.10986. URL https://ojs.aaai.org/index.php/AAAI/article/view/10986.
  8. Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents. In The World Wide Web Conference, WWW ’19, page 2551–2557, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366748. doi:10.1145/3308558.3313642. URL https://doi.org/10.1145/3308558.3313642.
  9. Transkp: Transformer based key-phrase extraction. 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–7, 2020.
  10. Keyphrase extraction as sequence labeling using contextualized embeddings. In European Conference on Information Retrieval, pages 328–335. Springer, 2020.
  11. Learning rich representation of keyphrases from text. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 891–906, Seattle, United States, July 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.findings-naacl.67. URL https://aclanthology.org/2022.findings-naacl.67.
  12. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019.
  13. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  14. Improving language understanding by generative pre-training. 2018.
  15. Training language models to follow instructions with human feedback, 2022.
  16. Language models are few-shot learners, 2020.
  17. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-3252.
  18. TopicRank: Graph-based topic ranking for keyphrase extraction. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 543–551, Nagoya, Japan, October 2013. Asian Federation of Natural Language Processing. URL https://aclanthology.org/I13-1062.
  19. Corpus-independent generic keyphrase extraction using word embedding vectors. In Software engineering research conference, volume 39, pages 1–8, 2014.
  20. Simple unsupervised keyphrase extraction using sentence embeddings. arXiv preprint arXiv:1801.04470, 2018.
  21. Key2vec: Automatic ranked keyphrase extraction from scientific articles using phrase embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 634–639, 2018.
  22. Anette Hulth. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, EMNLP ’03, page 216–223, USA, 2003. Association for Computational Linguistics. doi:10.3115/1119355.1119383. URL https://doi.org/10.3115/1119355.1119383.
  23. Re-examining automatic keyphrase extraction approaches in scientific articles. In Proceedings of the Workshop on Multiword Expressions: Identification, Interpretation, Disambiguation and Applications (MWE 2009), pages 9–16, Singapore, August 2009. Association for Computational Linguistics. URL https://aclanthology.org/W09-2902.
  24. Efficient estimation of word representations in vector space, 2013. URL https://arxiv.org/abs/1301.3781.
  25. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi:10.3115/v1/D14-1162. URL https://aclanthology.org/D14-1162.
  26. Attention is all you need, 2017. URL https://arxiv.org/abs/1706.03762.
  27. TNT-KID: Transformer-based neural tagger for keyword identification. Natural Language Engineering, 28(4):409–448, jun 2021. doi:10.1017/s1351324921000127. URL https://arxiv.org/abs/2003.09166.
  28. Deep keyphrase generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 582–592, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:10.18653/v1/P17-1054. URL https://aclanthology.org/P17-1054.
  29. Hai Ye and Lu Wang. Semi-supervised learning for neural keyphrase generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4142–4153, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:10.18653/v1/D18-1447. URL https://aclanthology.org/D18-1447.
  30. Keyphrase generation with correlation constraints, 2018.
  31. Title-guided encoding for keyphrase generation. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019a. ISBN 978-1-57735-809-1. doi:10.1609/aaai.v33i01.33016268. URL https://doi.org/10.1609/aaai.v33i01.33016268.
  32. Topic-aware neural keyphrase generation for social media language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2516–2526, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10.18653/v1/P19-1240. URL https://aclanthology.org/P19-1240.
  33. Incorporating linguistic constraints into keyphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5224–5233, Florence, Italy, July 2019. Association for Computational Linguistics. doi:10.18653/v1/P19-1515. URL https://aclanthology.org/P19-1515.
  34. Neural keyphrase generation via reinforcement learning with adaptive rewards, 2019.
  35. A preliminary exploration of GANs for keyphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8021–8030, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.645. URL https://aclanthology.org/2020.emnlp-main.645.
  36. Exclusive hierarchical decoding for deep keyphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1095–1105, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.103. URL https://aclanthology.org/2020.acl-main.103.
  37. One size does not fit all: Generating and evaluating variable number of keyphrases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7961–7975, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.710. URL https://aclanthology.org/2020.acl-main.710.
  38. Sgg: Learning to select, guide, and generate for keyphrase generation, 2021.
  39. Adaptive beam search decoding for discrete keyphrase generation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):13082–13089, May 2021. doi:10.1609/aaai.v35i14.17546. URL https://ojs.aaai.org/index.php/AAAI/article/view/17546.
  40. One2Set: Generating diverse keyphrases as a set. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4598–4608, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.354. URL https://aclanthology.org/2021.acl-long.354.
  41. An integrated approach for keyphrase generation via exploring the power of retrieval and extraction. ArXiv, abs/1904.03454, 2019b.
  42. Select, extract and generate: Neural keyphrase generation with layer-wise coverage attention. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1389–1404, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.111. URL https://aclanthology.org/2021.acl-long.111.
  43. Unikeyphrase: A unified extraction and generation framework for keyphrase prediction, 2021a.
  44. Fast and constrained absent keyphrase generation by prompt-based learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11495–11503, Jun. 2022. doi:10.1609/aaai.v36i10.21402. URL https://ojs.aaai.org/index.php/AAAI/article/view/21402.
  45. Ldkp: A dataset for identifying keyphrases from long scientific documents. arXiv preprint arXiv:2203.15349, 2022.
  46. Query-based keyphrase extraction from long documents. The International FLAIRS Conference Proceedings, 35, may 2022. doi:10.32473/flairs.v35i.130737. URL https://doi.org/10.32473%2Fflairs.v35i.130737.
  47. Enhancing keyphrase extraction from long scientific documents using graph embeddings, 2023.
  48. Keyphrase generation beyond the boundaries of title and abstract, 2022.
  49. Is chatgpt a good keyphrase generator? a preliminary study, 2023.
  50. Language models are unsupervised multitask learners. 2019.
  51. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf.
  52. Reward learning from human preferences and demonstrations in atari, 2018.
  53. Fine-tuning language models from human preferences, 2020.
  54. Learning to summarize from human feedback, 2022.
  55. Recursively summarizing books with human feedback, 2021b.
  56. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 2019.
  57. Can neural machine translation be improved with user feedback? In North American Chapter of the Association for Computational Linguistics, 2018.
  58. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1820–1830, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:10.18653/v1/P18-1169. URL https://aclanthology.org/P18-1169.
  59. Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models, 2020.
  60. Towards coherent and cohesive long-form text generation, 2019.
  61. Finding generalizable evidence by learning to convince q&a models. ArXiv, abs/1909.05863, 2019.
  62. Memory-assisted prompt editing to improve gpt-3 after deployment, 2023.
  63. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:10.18653/v1/P18-1031. URL https://aclanthology.org/P18-1031.
  64. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  65. Unified language model pre-training for natural language understanding and generation, 2019.
  66. The natural language decathlon: Multitask learning as question answering, 2018.
  67. Unifying question answering, text classification, and regression via span extraction, 2019.
  68. How good are gpt models at machine translation? a comprehensive evaluation, 2023.
  69. Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, page 21–26, USA, 2010. Association for Computational Linguistics.
  70. Kptimes: A large-scale dataset for keyphrase generation on news documents, 2019.
  71. Single document keyphrase extraction using neighborhood knowledge. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, page 855–860. AAAI Press, 2008. ISBN 9781577353683.
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now