Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Large Language Models As Evolution Strategies (2402.18381v1)

Published 28 Feb 2024 in cs.AI, cs.LG, and cs.NE
Large Language Models As Evolution Strategies

Abstract: Large Transformer models are capable of implementing a plethora of so-called in-context learning algorithms. These include gradient descent, classification, sequence completion, transformation, and improvement. In this work, we investigate whether LLMs, which never explicitly encountered the task of black-box optimization, are in principle capable of implementing evolutionary optimization algorithms. While previous works have solely focused on language-based task specification, we move forward and focus on the zero-shot application of LLMs to black-box optimization. We introduce a novel prompting strategy, consisting of least-to-most sorting of discretized population members and querying the LLM to propose an improvement to the mean statistic, i.e. perform a type of black-box recombination operation. Empirically, we find that our setup allows the user to obtain an LLM-based evolution strategy, which we call EvoLLM', that robustly outperforms baseline algorithms such as random search and Gaussian Hill Climbing on synthetic BBOB functions as well as small neuroevolution tasks. Hence, LLMs can act asplug-in' in-context recombination operators. We provide several comparative studies of the LLM's model size, prompt strategy, and context construction. Finally, we show that one can flexibly improve EvoLLM's performance by providing teacher algorithm information via instruction fine-tuning on previously collected teacher optimization trajectories.

PDF HTML Abstract

LLMs as Evolvable Operators for Evolution Strategies

Evolution Strategies Leveraged by LLMs

This paper focuses on investigating the capability of LLMs to perform as operators within Evolution Strategies (ES), a subclass of evolutionary optimization algorithms. For this purpose, the authors have put forward a novel approach that utilizes large transformer-based models trained on vast text corpora, proposing that these models can, beyond language understanding and generation, also engage in optimization tasks without being explicitly trained for them. By introducing a tactic of discretized solution representation coupled with a heuristic least-to-most prompt strategy, this work explores the boundary of LLM application beyond traditional language tasks, steering towards optimization via evolutionary strategies.

Methodological Insights

The authors introduce a prompt-based strategy for leveraging LLMs in black-box optimization tasks. The method involves sorting solution candidates based on their fitness and requesting the LLM to propose the next mean statistic to sample from. The innovative aspect lies in representing the solution space in a discretized format conducive to LLM processing capabilities, suggesting a potential paradigm where LLMs act as 'plug-and-play' operators for ES. This paper challenges the conventional understanding by showing smaller LLM models could outperform larger counterparts in tasks of evolutionary optimization, a finding contrary to the prevailing wisdom in LLM applications to natural language processing.

Experimental Demonstrations and Results

The empirical verification involved testing the designed EvoLLM model across synthetic benchmark functions and neuroevolution tasks. The results are impressive, with the EvoLLM outperforming baseline algorithms such as random search and Gaussian Hill Climbing across diverse tasks. These results also highlighted a remarkable aspect of LLM performance, where model size inversely correlated with optimization efficiency within the evolutionary strategies framework. Additionally, fine-tuning the underlying LLM models using trajectories generated by 'teacher' optimization algorithms has shown potential in enhancing EvoLLM's performance, indicating the possibility of tailored LLM applications through appropriate instructional tuning.

Implications and Speculations

The findings imply a broader applicability of LLMs beyond their conventional scope, suggesting their viability as generic pattern recognition and improvement engines, even in domains discreet from natural language. This could potentially pave the way for LLMs to contribute significantly to the field of optimization, particularly in ES and related evolutionary algorithms. One could speculate on the future development of more specialized LLMs or prompt strategies tailored for numerical optimization tasks, potentially extending the utility of LLMs across various scientific and engineering disciplines requiring optimization solutions.

Challenges and Directions for Future Research

Despite the promising findings, the work also outlines limitations related to pretraining, fine-tuning, and contextual understanding of LLMs when applied to numerical optimization tasks. The apparent inversion of the scaling law observed, where smaller models outperformed larger ones in optimization tasks, opens up intriguing questions regarding the underlying mechanics of LLMs in non-linguistic applications. Future research directions could focus on a deeper understanding of these phenomena, development of benchmarks specific to LLM-driven optimization, and exploration into evolving tokenization techniques that are more amenable to numerical representations.

Ethical and Monitoring Considerations

Given the exploration of LLMs in autonomous optimization, ethical considerations and monitoring of their operational boundaries become paramount. Ensuring responsible use and understanding the implications of deploying such powerful models in autonomous or semi-autonomous decision-making processes is critical to mitigate undesired consequences or exploitations.

In conclusion, this paper embarks on an exploratory journey, expanding the horizons of LLM applications into evolutionary strategies and optimization, opening up novel avenues for research and application of generative AI models.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Openai gym. arXiv preprint arXiv:1606.01540 (2016).
  2. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  3. EvoPrompting: Language Models for Code-Level Neural Architecture Search. (2023).
  4. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–15097.
  5. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning. PMLR, 748–756.
  6. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems 35 (2022), 32053–32068.
  7. Multi-step Planning for Automated Hyperparameter Optimization with OptFormer. arXiv preprint arXiv:2210.04971 (2022).
  8. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 (2023).
  9. PaLM 2 Technical Report. (2023). arXiv:cs.CL/2305.10403
  10. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532 (2023).
  11. Nikolaus Hansen. 2006. The CMA evolution strategy: a comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms (2006), 75–102.
  12. Real-parameter black-box optimization benchmarking 2010: Experimental setup. Ph.D. Dissertation. INRIA.
  13. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458 (2022).
  14. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems 34 (2021), 28742–28756.
  15. Generative pretraining for black-box optimization. arXiv preprint arXiv:2206.10786 (2022).
  16. Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimization. In Proceedings of the Genetic and Evolutionary Computation Conference. 929–937.
  17. Discovering evolution strategies via meta-black-box optimization. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation. 29–30.
  18. Robert Tjarko Lange. 2022a. evosax: JAX-based Evolution Strategies. arXiv preprint arXiv:2212.04180 (2022).
  19. Robert Tjarko Lange. 2022b. gymnax: A JAX-based Reinforcement Learning Environment Library. (2022). http://github.com/RobertTLange/gymnax
  20. NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications. arXiv preprint arXiv:2311.02394 (2023).
  21. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215 (2022).
  22. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning. PMLR, 3744–3753.
  23. Evolution through large models. In Handbook of Evolutionary Machine Learning. Springer, 331–366.
  24. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
  25. Yujian Betterest Li and Kai Wu. 2023. SPELL: Semantic Prompt Evolution based on a LLM. arXiv preprint arXiv:2310.01260 (2023).
  26. Large language model for multi-objective evolutionary optimization. arXiv preprint arXiv:2310.12541 (2023).
  27. Algorithm Evolution Using Large Language Model. arXiv preprint arXiv:2311.15249 (2023).
  28. Large Language Models as Evolutionary Optimizers. arXiv preprint arXiv:2310.19046 (2023).
  29. Structured state space models for in-context reinforcement learning. arXiv preprint arXiv:2303.03982 (2023).
  30. Prompt Optimisation with Random Sampling. arXiv preprint arXiv:2311.09569 (2023).
  31. Language Model Crossover: Variation through Few-Shot Prompting. (2023).
  32. Large Language Models as General Pattern Machines. arXiv preprint arXiv:2307.04721 (2023).
  33. Importance of Directional Feedback for LLM-based Optimizers. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  34. OpenAI. 2023. GPT-4 Technical Report. ArXiv abs/2303.08774 (2023). https://arxiv.org/abs/2303.08774
  35. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  36. Ingo Rechenberg. 1978. Evolutionsstrategien. In Simulationsmethoden in der Medizin und Biologie. Springer, 83–114.
  37. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 (2017).
  38. High dimensions and heavy tails for natural evolution strategies. In Proceedings of the 13th annual conference on Genetic and evolutionary computation. 845–852.
  39. Yujin Tang and David Ha. 2021. The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning. Advances in Neural Information Processing Systems 34 (2021), 22574–22587.
  40. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  41. Attention is all you need. Advances in neural information processing systems 30 (2017).
  42. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022).
  43. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  44. Natural evolution strategies. The Journal of Machine Learning Research 15, 1 (2014).
  45. Large language models as optimizers. (2023).
  46. Using Large Language Models for Hyperparameter Optimization. In NeurIPS 2023 Foundation Models for Decision Making Workshop.
  47. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 (2022).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Robert Tjarko Lange (21 papers)
  2. Yingtao Tian (32 papers)
  3. Yujin Tang (31 papers)
Citations (18)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube