Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Large Language Models as In-context AI Generators for Quality-Diversity (2404.15794v2)

Published 24 Apr 2024 in cs.NE, cs.AI, and cs.LG

Abstract: Quality-Diversity (QD) approaches are a promising direction to develop open-ended processes as they can discover archives of high-quality solutions across diverse niches. While already successful in many applications, QD approaches usually rely on combining only one or two solutions to generate new candidate solutions. As observed in open-ended processes such as technological evolution, wisely combining large diversity of these solutions could lead to more innovative solutions and potentially boost the productivity of QD search. In this work, we propose to exploit the pattern-matching capabilities of generative models to enable such efficient solution combinations. We introduce In-context QD, a framework of techniques that aim to elicit the in-context capabilities of pre-trained LLMs to generate interesting solutions using few-shot and many-shot prompting with quality-diverse examples from the QD archive as context. Applied to a series of common QD domains, In-context QD displays promising results compared to both QD baselines and similar strategies developed for single-objective optimization. Additionally, this result holds across multiple values of parameter sizes and archive population sizes, as well as across domains with distinct characteristics from BBO functions to policy search. Finally, we perform an extensive ablation that highlights the key prompt design considerations that encourage the generation of promising solutions for QD.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Quality-diversity through ai feedback. arXiv preprint arXiv:2310.13032.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Quality-diversity optimization: a novel branch of stochastic optimization. In Black Box Optimization, Machine Learning, and No-Free Lunch Theorems, pages 109–135. Springer.
  4. Reset-free trial-and-error learning for robot damage recovery. Robotics and Autonomous Systems, 100:236–250.
  5. Evoprompting: Language models for code-level neural architecture search. Advances in Neural Information Processing Systems, 36.
  6. Clune, J. (2019). Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905.10985.
  7. Scaling map-elites to deep neuroevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 67–75.
  8. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. Advances in neural information processing systems, 31.
  9. Cully, A. (2021). Multi-emitter map-elites: improving quality, diversity and data efficiency with heterogeneous sets of emitters. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 84–92.
  10. Robots that can adapt like animals. Nature, 521(7553):503–507.
  11. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245–259.
  12. Behavioral repertoire learning in robotics. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pages 175–182.
  13. Quality diversity through human feedback. arXiv preprint arXiv:2310.12103.
  14. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797.
  15. Multiple hands make light work: Enhancing quality and diversity using map-elites with multiple parallel evolution strategies. arXiv preprint arXiv:2303.06137.
  16. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34.
  17. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 genetic and evolutionary computation conference, pages 94–102.
  18. Data-efficient design exploration through surrogate-assisted illumination. Evolutionary computation, 26(3):381–410.
  19. Are quality diversity algorithms better at generating stepping stones than objective-based search? In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 115–116.
  20. Discovering representations for black-box optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 103–111.
  21. Procedural content generation through quality diversity. In 2019 IEEE Conference on Games (CoG), pages 1–8. IEEE.
  22. Real-Parameter Black-Box Optimization Benchmarking 2010: Experimental Setup. Research Report RR-7215, INRIA.
  23. Jang, E. (2021). Just ask for generalization. evjang.com.
  24. Mistral 7b. arXiv preprint arXiv:2310.06825.
  25. Model-based quality-diversity search for efficient robot learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9675–9680. IEEE.
  26. Bop-elites, a bayesian optimisation algorithm for quality-diversity search. arXiv preprint arXiv:2005.04320.
  27. Large language models as evolution strategies. arXiv preprint arXiv:2402.18381.
  28. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations.
  29. Evolution through large models. In Handbook of Evolutionary Machine Learning, pages 331–366. Springer.
  30. Novelty search and the problem with objectives. In Genetic programming theory and practice IX, pages 37–56. Springer.
  31. Dynamics-aware quality-diversity for efficient learning of skill repertoires. In 2022 International Conference on Robotics and Automation (ICRA), pages 5360–5366. IEEE.
  32. Large language models as evolutionary optimizers. arXiv preprint arXiv:2310.19046.
  33. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170.
  34. Millidge, B. (2024). Integer tokenization is insane.
  35. Large language models as general pattern machines. In Proceedings of the 7th Conference on Robot Learning (CoRL).
  36. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909.
  37. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 866–875.
  38. Diversity policy gradient for sample efficient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1075–1083.
  39. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40.
  40. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  41. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475.
  42. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402.16822.
  43. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903.
  44. Stanley, K. O. (2017). Open-endedness: The last grand challenge you’ve never heard of. https://www.oreilly.com/radar/open-endedness-the-last-grand-challenge-youve-never-heard-of/.
  45. Stanley, K. O. (2019). Why Open-Endedness Matters. Artificial Life, 25(3):232–235.
  46. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482.
  47. Discovering the Elite Hypervolume by Leveraging Interspecies Correlation. In GECCO 2018 - Genetic and Evolutionary Computation Conference, Kyoto, Japan.
  48. Large language models as optimizers. arXiv preprint arXiv:2309.03409.
  49. Using large language models for hyperparameter optimization. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Bryan Lim (30 papers)
  2. Manon Flageat (17 papers)
  3. Antoine Cully (68 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets