Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Quality-Diversity through AI Feedback (2310.13032v4)

Published 19 Oct 2023 in cs.CL, cs.AI, cs.LG, and cs.NE

Abstract: In many text-generation problems, users may prefer not only a single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim at such outcomes, by continually improving and diversifying a population of candidates. However, the applicability of QD to qualitative domains, like creative writing, has been limited by the difficulty of algorithmically specifying measures of quality and diversity. Interestingly, recent developments in LLMs (LMs) have enabled guiding search through AI feedback, wherein LMs are prompted in natural language to evaluate qualitative aspects of text. Leveraging this development, we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of a specified search space with high-quality samples than do non-QD controls. Further, human evaluation of QDAIF-generated creative texts validates reasonable agreement between AI and human evaluation. Our results thus highlight the potential of AI feedback to guide open-ended search for creative and original solutions, providing a recipe that seemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diversify, evaluate, and improve, which are among the core skills underlying human society's capacity for innovation.

PDF HTML Abstract

An Overview of "Quality-Diversity through AI Feedback" Paper

The paper entitled "Quality-Diversity through AI Feedback" presents an innovative approach to generating diverse and high-quality text outputs through the integration of Quality-Diversity (QD) search algorithms and LLMs (LMs). The authors introduce a novel framework, Quality-Diversity through AI Feedback (QDAIF), which leverages the capabilities of LMs not only in generating candidate solutions but also in evaluating their quality and diversity. This dual-functionality allows QDAIF to automate ideation processes in complex domains such as creative writing, which is traditionally subjective and challenging to define algorithmically.

Methodological Insights

The QDAIF framework is constructed upon the foundation of the MAP-Elites algorithm, a method well-recognized in QD for its efficacy in maintaining diverse and high-quality solution archives. The distinctive feature of QDAIF lies in its use of LMs to determine both the quality and diversity characteristics of text outputs. This method effectively addresses the historical limitation of QD algorithms that necessitated hand-designed diversity measures, which are often inadequate in capturing the nuanced demands of creative domains. By tapping into the evaluation potential of LMs, QDAIF circumvents this challenge, enabling broader applicability across varied domains.

Three primary domains were explored in the paper: opinion writing, short stories, and poetry. In these settings, QDAIF demonstrated its superior ability to discover a wider array of high-quality text outputs compared to other baseline methods, which included Fixed-Few-Shot, Shuffling-Few-Shot, Random-Search, and LMX, Quality-Only. Notably, the framework was also tested with alternative feedback mechanisms such as semantic embedding feedback, showing that AI feedback models offer a more robust evaluation suited to subjective assessments.

Experimental and Theoretical Implications

From an empirical standpoint, the QDAIF framework achieved significant QD scores across domains, corroborated by human evaluations that indicated effective alignment between AI-assessed and human-perceived text quality and diversity. This aspect underscores the potential of AI feedback in automating creative processes and its promising alignment with human intuition, particularly in subjective domains where traditional algorithms falter.

The paper emphasizes that this approach not only enriches the diversity of solutions but also enhances the quality of the best solutions over time, a critical observation for open-ended search tasks that thrive on exploration and exploitation balance. The introduction of human-centric evaluation criteria into automated processes, as demonstrated, could signify a broader implications for future AI systems aimed at independent navigation of creative tasks and innovation.

Challenges and Future Directions

While QDAIF showcases substantial progress, the paper highlights existing limitations, primarily around AI feedback's susceptibility to reward hacking—a known challenge where the model produces seemingly optimal outputs that diverge from human quality standards. This necessitates further research into refining LM-based evaluation mechanisms or potentially employing ensemble approaches for more reliable output assessments.

Moreover, the diversification within new domains may continue to pose challenges, driven by dynamics intrinsic to both model calibration and subjective output evaluation. As the field progresses, an exploration into more sophisticated binning methodologies and diversity measure designs will be vital. Understanding the intersection between AI feedback and evolving capabilities across multi-modal domains broadens the potential applications of the QDAIF strategy.

Conclusion

Overall, "Quality-Diversity through AI Feedback" contributes a pioneering method for creative text generation, emphasizing the integration of advanced LMs within the QD algorithmic framework to address the intricacies of evaluating quality and diversity in qualitative spaces. The research implies a promising trajectory towards AI systems that can coalesce evaluation, generation, and refinement capabilities, advocating for continued exploration into LM-driven open-ended search across varying creative and practical domains. Future works will likely explore overcoming the nuanced challenges of reward hacking and refining diversity measures, steering towards robust AI systems capable of standalone innovation and creative endeavor.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (91)
  1. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  3. Likert scales and data analyses. Quality progress, 40(7):64–65, 2007.
  4. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.
  5. Multifusion: Fusing pre-trained models for multi-lingual, multi-modal image generation. arXiv preprint arXiv:2305.15296, 2023.
  6. Deep surrogate assisted generation of environments. Advances in Neural Information Processing Systems, 35:37762–37777, 2022.
  7. Surrogate assisted generation of human-robot interaction scenarios. arXiv preprint arXiv:2304.13787, 2023.
  8. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  9. OpenELM, January 2023a. URL https://github.com/CarperAI/OpenELM.
  10. Diff models - A new way to edit code. CarperAI Blog, Jan 2023b. URL https://carper.ai/diff-model/.
  11. Minimal criterion coevolution: A new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  67–74, 2017.
  12. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  13. Evoprompting: Language models for code-level neural architecture search. arXiv preprint arXiv:2302.14838, 2023.
  14. Evaluating large language models trained on code. 2021.
  15. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  16. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  17. Jeff Clune. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905.10985, 2019.
  18. Augmenting autotelic agents with large language models. arXiv preprint arXiv:2305.12487, 2023.
  19. Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  81–89, 2019.
  20. Hierarchical behavioral repertoires with unsupervised descriptors. Proceedings of the Genetic and Evolutionary Computation Conference, 2018.
  21. Robots that can adapt like animals. Nature, 521(7553):503–507, 2015.
  22. Quality diversity through human feedback. arXiv preprint arXiv:2310.12103, 2023.
  23. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  24. Open questions in creating safe open-ended ai: tensions between control and creativity. In Artificial Life Conference Proceedings 32, pp.  27–35. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020.
  25. MAGMA - Multimodal augmentation of generative models through adapter-based finetuning. arXiv preprint arXiv:2112.05253, 2021.
  26. Fast and stable map-elites in noisy domains using deep grids. In Artificial Life Conference Proceedings 32, pp.  273–282. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2020.
  27. Uncertain quality-diversity: Evaluation methodology and new methods for quality-diversity in uncertain domains. IEEE Transactions on Evolutionary Computation, 2023.
  28. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34:10040–10052, 2021.
  29. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint arXiv:2302.10893, 2023.
  30. Adam Gaier. Accelerating Evolutionary Design Exploration with Predictive and Generative Models. PhD thesis, Université de Lorraine, 2020.
  31. Data-efficient exploration, optimization, and modeling of diverse designs through surrogate-assisted illumination. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  99–106, 2017.
  32. Are quality diversity algorithms better at generating stepping stones than objective-based search? In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp.  115–116, 2019.
  33. Unsupervised behaviour discovery with quality-diversity optimisation. arXiv preprint arXiv:2106.05648, 2021.
  34. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021.
  35. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp.  2790–2799. PMLR, 2019.
  36. How can we know when language models know? On the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977, 2021.
  37. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  38. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  39. Model-based quality-diversity search for efficient robot learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  9675–9680. IEEE, 2020.
  40. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023.
  41. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. arXiv preprint arXiv:2309.00267, 2023.
  42. Revising the evolutionary computation abstraction: Minimal criteria novelty search. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp.  103–110, 2010.
  43. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189–223, 2011a.
  44. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp.  211–218, 2011b.
  45. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp.  329–336, 2008.
  46. On the critical role of divergent selection in evolvability. Frontiers in Robotics and AI, 3:45, 2016.
  47. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities, 2019.
  48. Evolution through large models. arXiv preprint arXiv:2206.08896, 2022.
  49. Dynamics-aware quality-diversity for efficient learning of skill repertoires. arXiv preprint arXiv:2109.08522, 2021.
  50. Learning to walk autonomously via reset-free quality-diversity. arXiv preprint arXiv:2204.03655, 2022.
  51. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.  74–81, 2004.
  52. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
  53. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021.
  54. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
  55. Christopher D Manning. An introduction to information retrieval. Cambridge university press, 2009.
  56. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170, 2023.
  57. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015.
  58. Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. arXiv:2202.08904, 2022.
  59. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  427–436, 2015a.
  60. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the Genetic and Evolutionary Computation Conference, 2015b.
  61. Understanding innovation engines: Automated creativity and improved stochastic optimization via deep learning. Evolutionary computation, 24(3):545–572, 2016.
  62. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.
  63. Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  866–875, 2021.
  64. OpenAI. GPT-4 technical report, 2023.
  65. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  66. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022.
  67. Aces: generating diverse programming puzzles with autotelic language models and semantic descriptors. arXiv preprint arXiv:2310.10692, 2023.
  68. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
  69. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  70. Sentence-BERT: Sentence embeddings using siamese BERT-networks. arXiv preprint arXiv:1908.10084, 2019.
  71. Can LLMs generate random numbers? evaluating LLM sampling in controlled domains. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023. URL http://people.csail.mit.edu/renda/llm-sampling-paper.
  72. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
  73. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
  74. Picbreeder: A case study in collaborative evolutionary exploration of design space. Evol. Comput., 19(3):373–403, 2011. doi: 10.1162/EVCO_a_00030. URL https://doi.org/10.1162/EVCO_a_00030.
  75. Reflexion: An autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  76. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022.
  77. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp.  1631–1642, 2013.
  78. Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:131–162, 2007.
  79. Why greatness cannot be planned: The myth of the objective. Springer, 2015.
  80. Open-endedness: The last grand challenge you’ve never heard of. O’Reilly Radar, 2017.
  81. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
  82. Mariogpt: Open-ended text2level generation through large language models, 2023.
  83. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games, pp.  1–8, 2023.
  84. Scaling up MAP-Elites using centroidal Voronoi tessellations. arXiv preprint arXiv:1610.05729, 2016.
  85. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a.
  86. Diversity from human feedback. arXiv preprint arXiv:2310.06648, 2023b.
  87. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022a.
  88. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. arXiv preprint arXiv:2204.07705, 2022b.
  89. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  90. OMNI: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711, 2023.
  91. Deep surrogate assisted MAP-Elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  158–167, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Herbie Bradley (10 papers)
  2. Andrew Dai (17 papers)
  3. Hannah Teufel (7 papers)
  4. Jenny Zhang (10 papers)
  5. Koen Oostermeijer (5 papers)
  6. Marco Bellagente (13 papers)
  7. Jeff Clune (65 papers)
  8. Kenneth Stanley (2 papers)
  9. Grégory Schott (3 papers)
  10. Joel Lehman (34 papers)
Citations (20)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Quality-Diversity through AI Feedback
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube