2000 character limit reached
Controllable Citation Sentence Generation with Language Models (2211.07066v2)
Published 14 Nov 2022 in cs.CL
Abstract: Citation generation aims to generate a citation sentence that refers to a chosen paper in the context of a manuscript. However, a rigid citation generation process is at odds with an author's desire to control specific attributes, such as 1) the citation intent, e.g., either introducing background information or comparing results, and 2) keywords that should appear in the citation text. To provide these degrees of controllability during citation generation, we propose to integrate the manuscript context, the context of the referenced paper, and the desired control attributes into a structured template and use it to fine-tune a LLM (LM) via next-token prediction. We then utilize Proximal Policy Optimization to directly optimize the LM in favor of a high score of our proposed controllability metric. The proposed workflow harmoniously combines citation attribute suggestion and conditional citation generation into one LM, allowing for better user control.
- SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
- GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata.
- Jingqiang Chen and Hai Zhuge. 2019. Automatic generation of related work through summarizing citations. Concurrency and Computation: Practice and Experience, 31(3):e4261. Number: 3 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpe.4261.
- Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation. arXiv:1912.02164 [cs]. ArXiv: 1912.02164.
- Qlora: Efficient finetuning of quantized llms.
- BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1466–1478, Online. Association for Computational Linguistics.
- Lora: Low-rank adaptation of large language models.
- Intent-controllable citation text generation. Mathematics, 10(10):1763.
- Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.
- Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 313–323, Brussels, Belgium. Association for Computational Linguistics.
- Ctrl: A conditional transformer language model for controllable generation.
- BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
- Yang Liu. 2019. Fine-tune bert for extractive summarization. ArXiv.
- Automatic generation of reviews of scientific papers. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 314–319.
- Directed beam search: Plug-and-play lexically constrained language generation. CoRR, abs/2012.15416.
- Bart Penders. 2018. Ten simple rules for responsible referencing. PLOS Computational Biology, 14(4):e1006036.
- Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
- Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
- Proximal policy optimization algorithms.
- Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
- Galactica: A large language model for science.
- Llama: Open and efficient foundation language models.
- Trl: Transformer reinforcement learning. https://github.com/lvwerra/trl.
- Disencite: Graph-based disentangled representation learning for context-specific citation generation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11449–11458.
- Bloom: A 176b-parameter open-access multilingual language model.
- Towards generating citation sentences for multiple references with intent control.
- Automatic Generation of Citation Texts in Scholarly Papers: A Pilot Study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6181–6190, Online. Association for Computational Linguistics.
- Tailor: A prompt-based approach to attribute-based controlled text generation.
- Scientific comparative argument generation.