Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
o3 Pro
5 tokens/sec
GPT-4.1 Pro
37 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
Gemini 2.5 Flash Deprecated
12 tokens/sec
2000 character limit reached

Improving Attributed Text Generation of Large Language Models via Preference Learning (2403.18381v1)

Published 27 Mar 2024 in cs.CL and cs.AI

Abstract: LLMs have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Direct preference-based policy optimization without reward modeling. In Neural Information Processing Systems.
  2. Gemini: A family of highly capable multimodal models. ArXiv preprint, abs/2312.11805.
  3. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, ACL 2023, Toronto, Canada, July 9-14, 2023, pages 41–46.
  4. Self-rag: Learning to retrieve, generate, and critique through self-reflection. ArXiv preprint, abs/2310.11511.
  5. A general theoretical paradigm to understand learning from human preferences. ArXiv preprint, abs/2310.12036.
  6. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862.
  7. Attributed question answering: Evaluation and modeling for attributed large language models. ArXiv preprint, abs/2212.08037.
  8. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.
  9. Terrence A Brooks. 1986. Evidence of complex citer motivations. Journal of the American Society for Information Science, 37(1):34–36.
  10. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  11. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299–4307.
  12. A survey for in-context learning. ArXiv preprint, abs/2301.00234.
  13. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567.
  14. RARR: researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 16477–16508.
  15. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6465–6488.
  16. Retrieval-augmented generation for large language models: A survey. ArXiv preprint, abs/2312.10997.
  17. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361.
  18. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 161–175.
  19. LCSTS: A large scale chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1967–1972. The Association for Computational Linguistics.
  20. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
  21. HAGRID: A human-llm collaborative dataset for generative information-seeking with attribution. ArXiv preprint, abs/2307.16883.
  22. Beyond reward: Offline preference-guided policy optimization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 15753–15768.
  23. Human-centered loss functions (halos). Technical report, Contextual AI.
  24. A survey of large language models attribution. ArXiv preprint, abs/2311.03731.
  25. Llatrieval: Llm-verified retrieval for verifiable generation. ArXiv preprint, abs/2311.07838.
  26. Towards verifiable generation: A benchmark for knowledge-aware language model attribution.
  27. Let’s verify step by step. ArXiv preprint, abs/2305.20050.
  28. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 7001–7025.
  29. Expertqa: Expert-curated questions and attributed answers. ArXiv preprint, abs/2309.07852.
  30. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue.
  31. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.
  32. Direct preference optimization: Your language model is secretly a reward model. ArXiv preprint, abs/2305.18290.
  33. Measuring attribution in natural language generation models. ArXiv preprint, abs/2112.12870.
  34. Experience replay for continual learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 348–358.
  35. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347.
  36. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803.
  37. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288.
  38. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  39. Towards verifiable text generation with evolving memory and self-reflection. ArXiv preprint, abs/2312.09075.
  40. How status of research papers affects the way they are read and cited. Research Policy, 51(4):104484.
  41. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288.
  42. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484–13508.
  43. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
  44. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
  45. Search-in-the-chain: Towards the accurate, credible and traceable content generation for complex knowledge-intensive tasks. ArXiv preprint, abs/2304.14732.
  46. Cognitive mirage: A review of hallucinations in large language models. ArXiv preprint, abs/2309.06794.
  47. Effective large language model adaptation for improved grounding. ArXiv preprint, abs/2311.09533.
  48. Automatic evaluation of attribution by large language models. ArXiv preprint, abs/2305.06311.
  49. Siren’s song in the ai ocean: A survey on hallucination in large language models. ArXiv preprint, abs/2309.01219.
  50. Slic-hf: Sequence likelihood calibration with human feedback. ArXiv preprint, abs/2305.10425.
  51. Secrets of RLHF in large language models part I: PPO. ArXiv preprint, abs/2307.04964.
  52. Fine-tuning language models from human preferences. ArXiv preprint, abs/1909.08593.
  53. Chatgpt hallucinates when attributing answers.
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now