Papers
Topics
Authors
Recent
Search
2000 character limit reached

Let Me Teach You: Pedagogical Foundations of Feedback for Language Models

Published 1 Jul 2023 in cs.CL | (2307.00279v3)

Abstract: Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning LLMs to human preferences. Despite the diversity of the information it can convey, NLF methods are often hand-designed and arbitrary, with little systematic grounding. At the same time, research in learning sciences has long established several effective feedback models. In this opinion piece, we compile ideas from pedagogy to introduce FELT, a feedback framework for LLMs that outlines various characteristics of the feedback space, and a feedback content taxonomy based on these variables, providing a general mapping of the feedback space. In addition to streamlining NLF designs, FELT also brings out new, unexplored directions for research in NLF. We make our taxonomy available to the community, providing guides and examples for mapping our categorizations to future research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861.
  2. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
  3. Constitutional AI: Harmlessness from AI Feedback. ArXiv:2212.08073 [cs].
  4. The Instructional Effect of Feedback in Test-like Events. Review of Educational Research, 61(2):213–238. Publisher: [Sage Publications, Inc., American Educational Research Association].
  5. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  6. Deborah L. Butler and Philip H. Winne. 1995. Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3):245–281. Publisher: [Sage Publications, Inc., American Educational Research Association].
  7. David Carless and David Boud. 2018. The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8):1315–1325. Publisher: Routledge _eprint: https://doi.org/10.1080/02602938.2018.1463354.
  8. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.
  9. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  10. Carol Evans. 2013. Making sense of assessment feedback in higher education. Review of Educational Research, 83:70–120. Place: US Publisher: Sage Publications.
  11. Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation. ArXiv:2305.00955 [cs].
  12. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459.
  13. Simulating bandit learning from user feedback for extractive question answering. arXiv preprint arXiv:2203.10079.
  14. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations.
  15. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415.
  16. John Hattie and Helen Timperley. 2007. The Power of Feedback. Review of Educational Research, 77(1):81–112.
  17. Avraham N. Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2):254–284.
  18. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08582.
  19. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 92–105, New Orleans - Louisiana. Association for Computational Linguistics.
  20. Raymond W. Kulhavy. 1977. Feedback in Written Instruction. Review of Educational Research, 47(2):211–232. Publisher: [Sage Publications, Inc., American Educational Research Association].
  21. Raymond W. Kulhavy and William A. Stock. 1989. Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1(4):279–308.
  22. Dialogue learning with human-in-the-loop.
  23. Dialogue Learning With Human-In-The-Loop. ArXiv:1611.09823 [cs].
  24. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  25. Toward a Model of Student Response to Feedback. In Handbook of Human and Social Conditions in Assessment. Routledge. Num Pages: 17.
  26. Anastasiya A. Lipnevich and Ernesto Panadero. 2021. A Review of Feedback Models and Theories: Descriptions, Definitions, and Conclusions. Frontiers in Education, 6.
  27. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02676.
  28. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833–2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  29. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
  30. Providing Feedback in Computer-based Instruction: What the Research Tells Us. Center for Instructional Innovation, 15.
  31. Fostering the intelligent novice: Learning from errors with metacognitive tutoring. Educational Psychologist, 40:257 – 265.
  32. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147.
  33. Fixing model bugs with natural language patches. arXiv preprint arXiv:2211.03318.
  34. Susanne Narciss. 2008. Feedback Strategies for Interactive Learning Tasks. pages 125–144.
  35. Susanne Narciss and Katja Huth. 2004. How to design informative tutoring feedback for multi-media learning.
  36. David J. Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2):199–218. Publisher: Routledge _eprint: https://doi.org/10.1080/03075070600572090.
  37. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  38. Training language models to follow instructions with human feedback. ArXiv:2203.02155 [cs].
  39. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  40. Ernesto Panadero and Anastasiya A. Lipnevich. 2022. A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review, 35:100416.
  41. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904.
  42. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  43. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241.
  44. Arkalgud Ramaprasad. 1983. On the Definition of Feedback. Behavioral Science, 28:4–13.
  45. D. Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional Science, 18(2):119–144.
  46. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802.
  47. Training language models with language feedback. arXiv preprint arXiv:2204.14146.
  48. Peer: A collaborative language model. arXiv preprint arXiv:2208.11663.
  49. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210.15893.
  50. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366.
  51. Learning to summarize from human feedback. ArXiv:2009.01325 [cs] version: 3.
  52. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 339–352, Seattle, United States. Association for Computational Linguistics.
  53. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  54. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053.
  55. Jason Weston. 2016. Dialog-based Language Learning. ArXiv:1604.06045 [cs].
  56. Recursively summarizing books with human feedback. arXiv preprint arXiv:2211.00053.
  57. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693.
  58. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint arXiv:2208.03270.
  59. Context-faithful prompting for large language models. arXiv preprint arXiv:2303.11315.
  60. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up