LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models (2401.00907v1)
Abstract: Fine-tuning LLMs adapts a trained model to specific downstream tasks, significantly improving task-specific performance. Supervised Fine-Tuning (SFT) is a common approach, where an LLM is trained to produce desired answers. However, LLMs trained with SFT sometimes make simple mistakes and result in hallucinations on reasoning tasks such as question-answering. Without external feedback, it is difficult for SFT to learn a good mapping between the question and the desired answer, especially with a small dataset. This paper introduces an alternative to SFT called Natural Language Feedback for Finetuning LLMs (LaFFi). LaFFi has LLMs directly predict the feedback they will receive from an annotator. We find that requiring such reflection can significantly improve the accuracy in in-domain question-answering tasks, providing a promising direction for the application of natural language feedback in the realm of SFT LLMs. Additional ablation studies show that the portion of human-annotated data in the annotated datasets affects the fine-tuning performance.
- Language Models are Few-Shot Learners. arXiv:2005.14165.
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv:2307.15217.
- Computer, T. 2023. RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training dataset.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805.
- A Survey for In-context Learning. ArXiv, abs/2301.00234.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929.
- OpenLLaMA: An Open Reproduction of LLaMA.
- LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
- Training language models to follow instructions with human feedback. arXiv:2203.02155.
- REFINER: Reasoning Feedback on Intermediate Representations. arXiv:2304.01904.
- Language Models are Unsupervised Multitask Learners.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683.
- Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 784–789. Melbourne, Australia: Association for Computational Linguistics.
- Can Wikipedia Help Offline Reinforcement Learning? arXiv:2201.12122.
- Learning representations by back-propagating errors. nature, 323(6088): 533–536.
- Self-critiquing models for assisting human evaluators. arXiv e-prints, arXiv:2206.05802.
- PEER: A Collaborative Language Model. arXiv:2208.11663.
- BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. arXiv:1904.06690.
- Sequence to Sequence Learning with Neural Networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, 3104–3112. Cambridge, MA, USA: MIT Press.
- Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
- Attention is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, 6000–6010. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781510860964.
- Finetuned Language Models Are Zero-Shot Learners. arXiv:2109.01652.
- Emergent Abilities of Large Language Models. arXiv:2206.07682.
- System-Level Natural Language Feedback. arXiv:2306.13588.
- Qianxi Li (1 paper)
- Yingyue Cao (2 papers)
- Jikun Kang (7 papers)
- Tianpei Yang (25 papers)
- Xi Chen (1035 papers)
- Jun Jin (28 papers)
- Matthew E. Taylor (69 papers)