Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation (2311.09136v3)

Published 15 Nov 2023 in cs.CL

Abstract: Customizing LLMs for a specific task involves separating high-quality responses from lower-quality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system's improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named Rescue, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Opt-r: Exploring the role of explanations in finetuning and prompting for reasoning skills of large language models. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE). Association for Computational Linguistics.
  2. The falcon series of open language models.
  3. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2204.05862.
  4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  5. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  6. Discovering latent knowledge in language models without supervision.
  7. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 9560–9572.
  8. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics.
  9. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  10. Content planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4319–4338, Online. Association for Computational Linguistics.
  11. Annotation artifacts in natural language inference data. CoRR, abs/1803.02324.
  12. Contrastive preference learning: Learning from human feedback without rl.
  13. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.
  14. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1352–1362, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  15. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8003–8017. Association for Computational Linguistics.
  16. Unsupervised dense information retrieval with contrastive learning.
  17. Mistral 7b.
  18. Sawan Kumar and Partha Talukdar. 2020. NILE : Natural language inference with faithful natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730–8742, Online. Association for Computational Linguistics.
  19. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.
  20. Retrieval-augmented generation for knowledge-intensive nlp tasks.
  21. Explaining competitive-level programming solutions using llms.
  22. Prudent silence or foolish babble? examining large language models’ responses to the unknown.
  23. Chain of hindsight aligns language models with feedback.
  24. Lost in the middle: How language models use long contexts. CoRR, abs/2307.03172.
  25. Brio: Bringing order to abstractive summarization.
  26. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  27. Explanation-based finetuning makes models more robust to spurious cues.
  28. MosaicML. 2023. Introducing mpt-30b: Raising the bar for open-source foundation models. Accessed: 2023-06-22.
  29. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint 2112.09332.
  30. Training language models to follow instructions with human feedback.
  31. A study of generative large language model for medical research and healthcare.
  32. Direct preference optimization: Your language model is secretly a reward model.
  33. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint 2210.01241.
  34. Chatgpt-based investment portfolio selection.
  35. Reflexion: Language agents with verbal reinforcement learning.
  36. The confidence-competence gap in large language models: A cognitive study.
  37. Trustllm: Trustworthiness in large language models.
  38. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  39. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.
  40. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  41. Chain-of-thought prompting elicits reasoning in large language models.
  42. To repeat or not to repeat: Insights from scaling llm under token-crisis. arXiv preprint arXiv:2305.13230.
  43. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
  44. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  45. Complementary explanations for effective in-context learning.
  46. Rrhf: Rank responses to align language models with human feedback without tears.
  47. Principled reinforcement learning with human feedback from pairwise or k𝑘kitalic_k-wise comparisons. arXiv preprint 2301.11270.
  48. Fine-tuning language models from human preferences. arXiv preprint 1909.08593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yikun Wang (25 papers)
  2. Rui Zheng (79 papers)
  3. Haoming Li (19 papers)
  4. Qi Zhang (785 papers)
  5. Tao Gui (127 papers)
  6. Fei Liu (232 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets