Papers
Topics
Authors
Recent
Search
2000 character limit reached

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

Published 28 Aug 2024 in cs.CL | (2408.15549v3)

Abstract: As LLMs continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, misalignment with real-world user preferences, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. Given a corpus of multi-turn user-LLM conversation, WildFeedback identifies and classifies user feedback to LLM responses between conversation turns. The user feedback is then used to create examples of preferred and dispreferred responses according to users' preference. Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed checklist-guided evaluation. By incorporating in-situ feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219.
  2. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2023. URL https://api.semanticscholar.org/CorpusID:268232499.
  3. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a. URL https://arxiv.org/abs/2204.05862.
  4. Constitutional ai: Harmlessness from ai feedback, 2022b. URL https://arxiv.org/abs/2212.08073.
  5. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. URL https://arxiv.org/abs/2312.09390.
  6. Scaling instruction-finetuned language models, 2022. URL https://arxiv.org/abs/2210.11416.
  7. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.  9722–9744. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/cui24f.html.
  8. S3-dst: Structured open-domain dialogue segmentation and state tracking in the era of llms, 2023. URL https://arxiv.org/abs/2309.08827.
  9. Learning from naturally occurring feedback, 2024. URL https://arxiv.org/abs/2407.10944.
  10. The faiss library, 2024.
  11. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  12. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. URL https://arxiv.org/abs/2404.04475.
  13. Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2023. URL https://arxiv.org/abs/2307.04657.
  14. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825.
  15. Openassistant conversations - democratizing large language model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=VSJotgbPHF.
  16. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023. URL https://arxiv.org/abs/2309.00267.
  17. CoAnnotating: Uncertainty-guided work allocation between human and large language models for data annotation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.  1487–1505, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.92. URL https://aclanthology.org/2023.emnlp-main.92.
  18. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URL https://arxiv.org/abs/2406.11939.
  19. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023b.
  20. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050.
  21. Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024a. URL https://arxiv.org/abs/2406.04770.
  22. Interpretable user satisfaction estimation for conversational systems with large language models, 2024b. URL https://arxiv.org/abs/2403.12388.
  23. Iterative length-regularized direct preference optimization: A case study on improving 7b language models to gpt-4 level, 2024a. URL https://arxiv.org/abs/2406.11817.
  24. Llms as narcissistic evaluators: When ego inflates evaluation scores, 2024b. URL https://arxiv.org/abs/2311.09766.
  25. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651.
  26. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://arxiv.org/abs/2112.09332.
  27. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774.
  28. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155.
  29. Reuse, don’t retrain: A recipe for continued pretraining of language models, 2024. URL https://arxiv.org/abs/2407.07263.
  30. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022.
  31. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9.
  32. Safer-instruct: Aligning language models with automated preference data. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.  7636–7651, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.422. URL https://aclanthology.org/2024.naacl-long.422.
  33. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels, 2022. URL https://arxiv.org/abs/2210.15893.
  34. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118.
  35. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges, 2024. URL https://arxiv.org/abs/2406.12624.
  36. Zephyr: Direct distillation of lm alignment, 2023. URL https://arxiv.org/abs/2310.16944.
  37. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.  5085–5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.340. URL https://aclanthology.org/2022.emnlp-main.340.
  38. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.754. URL https://aclanthology.org/2023.acl-long.754.
  39. Recursively summarizing books with human feedback, 2021. URL https://arxiv.org/abs/2109.10862.
  40. Wizardlm: Empowering large language models to follow complex instructions, 2023. URL https://arxiv.org/abs/2304.12244.
  41. Wildchat: 1m chatgpt interaction logs in the wild, 2024. URL https://arxiv.org/abs/2405.01470.
  42. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.  46595–46623. Curran Associates, Inc., 2023a. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf.
  43. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b. URL https://arxiv.org/abs/2306.05685.
  44. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces WildFeedback, a three-step framework that leverages real-time user feedback to construct authentic preference datasets for LLM alignment.
  • The methodology integrates feedback signal identification, preference data construction, and user-guided evaluation to overcome subjectivity and scalability challenges.
  • Experimental results indicate that LLMs fine-tuned on WildFeedback data achieve higher win rates on benchmarks like AlpacaEval 2 and MT-Bench compared to standard models.

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

The paper "WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback" introduces a nuanced framework to address one of the critical challenges in the field of machine learning: aligning LLMs with human preferences. Traditional alignment methods, which rely on human or LLM-annotated datasets, face significant limitations. These include the resource-intensive nature of human annotations, inherent subjectivity, and the risk of feedback loops that accentuate existing biases in the models. The authors propose WildFeedback as a solution to these challenges by leveraging real-time, in-situ user interactions to create preference datasets that more accurately reflect authentic human values.

Overview of WildFeedback

WildFeedback is a three-step framework involving feedback signal identification, preference data construction, and user-guided evaluation. The framework was applied to a large corpus of user-LLM conversations, resulting in a rich dataset that captures genuine user preferences. This approach allows the construction of more representative and context-sensitive alignment data, addressing the scalability, subjectivity, and bias issues present in existing alignment methods.

Methodology

Feedback Signal Identification

The first step involves identifying user satisfaction and dissatisfaction (SAT/DSAT) signals within natural conversations. The authors adapted existing user satisfaction estimation techniques to classify these signals in the WildChat dataset, which includes over 148,000 multi-turn conversations between users and ChatGPT. By analyzing these conversations, the framework identifies the parts of the dialogue that contain feedback signals, using criteria such as gratitude, learning, compliance for SAT, and negative feedback, revision, factual error for DSAT.

Preference Data Construction

Upon identifying conversations with feedback signals, the next step is constructing a preference dataset. This includes summarizing user preferences and categorizing responses as either preferred or dispreferred based on user feedback. The authors used both expert (GPT-4) and on-policy (Mistral, Phi 3, and LLaMA 3) models to generate these responses. They ensured that the generated preferred responses aligned with expressed user preferences by integrating summarized user preferences as system instructions.

User-Guided Evaluation

To evaluate model performance, the paper introduces a user-guided evaluation methodology. This involves using actual user feedback as checklists to guide LLM evaluations. By comparing responses with and without these checklists, the evaluation framework aims to provide a more accurate benchmark for assessing how well LLMs align with human values.

Results

The experiments conducted in the paper demonstrate that models fine-tuned on WildFeedback not only show significant improvements in aligning with user preferences but also perform well on traditional benchmarks. For instance, models trained on the GPT-4 version of WildFeedback showed higher win rates across AlpacaEval 2, Arena-Hard, and MT-Bench compared to off-the-shelf instruction models. The results suggest that incorporating real-time feedback from actual users can significantly enhance the alignment of LLMs with the diverse and evolving needs of their users.

Implications and Future Directions

WildFeedback represents a robust and scalable solution for aligning LLMs with true human values, setting a new standard for the development and evaluation of user-centric LLMs. The implications of this work are both practical and theoretical. Practically, it can be applied to enhance the responsiveness and user satisfaction of conversational AI systems. Theoretically, it offers a novel approach to overcoming the biases and limitations inherent in traditional alignment methods.

Given the promising results, future research could focus on refining the feedback signal identification process to capture an even broader range of user preferences. Additionally, exploring methods to filter out spurious or harmful user preferences will be crucial to ensuring that the models learn to prioritize genuine, beneficial human values. Addressing selection bias by incorporating feedback from a more diverse set of users can also further enhance the representativeness of the preference dataset.

Conclusion

WildFeedback offers a comprehensive framework for aligning LLMs with real-time user interactions, addressing key challenges in scalability, subjectivity, and bias. The approach sets a precedent for future developments in creating more user-centric AI systems, ultimately contributing to the advancement of natural language processing and machine learning fields.

File Document Download Save Streamline Icon: https://streamlinehq.com Markdown

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 13 tweets with 194 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free