The Role of Morphological Variation in Evolutionary Robotics: Maximizing Performance and Robustness (2208.02809v2)
Abstract: Exposing an Evolutionary Algorithm that is used to evolve robot controllers to variable conditions is necessary to obtain solutions which are robust and can cross the reality gap. However, we do not yet have methods for analyzing and understanding the impact of the varying morphological conditions which impact the evolutionary process, and therefore for choosing suitable variation ranges. By morphological conditions, we refer to the starting state of the robot, and to variations in its sensor readings during operation due to noise. In this article, we introduce a method that permits us to measure the impact of these morphological variations and we analyze the relation between the amplitude of variations, the modality with which they are introduced, and the performance and robustness of evolving agents. Our results demonstrate that (i) the evolutionary algorithm can tolerate morphological variations which have a very high impact, (ii) variations affecting the actions of the agent are tolerated much better than variations affecting the initial state of the agent or of the environment, and (iii) improving the accuracy of the fitness measure through multiple evaluations is not always useful. Moreover, our results show that morphological variations permit generating solutions which perform better both in varying and non-varying conditions.
- Scheduling of genetic algorithms in a noisy environment. Evolutionary Computation, 2(2):97–122.
- Noisy optimization with evolution strategies, volume 8. Springer Science & Business Media.
- Branke, J. (2012). Evolutionary optimization in dynamic environments, volume 3. Springer Science & Business Media.
- Selection in the presence of noise. In Genetic and Evolutionary Computation Conference, pages 766–777. Springer.
- Environment-driven distributed evolutionary adaptation in a population of autonomous robotic agents. Mathematical and Computer Modelling of Dynamical Systems, 18(1):101–129.
- Cantú-Paz, E. (2004). Adaptive sampling for noisy problems. In Genetic and Evolutionary Computation Conference, pages 947–958. Springer.
- Pybullet, a python module for physics simulation for games, robotics and machine learning.
- Robots that can adapt like animals. Nature, 521(7553):503–507.
- Evolutionary robots with on-line self-organization and behavioral fitness. Neural Networks, 13(4-5):431–443.
- Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pages 393–400.
- A method for handling uncertainty in evolutionary optimization with an application to feedback control of combustion. IEEE Transactions on Evolutionary Computation, 13(1):180–197.
- The gambler’s ruin problem, genetic algorithms, and the sizing of populations. Evolutionary computation, 7(3):231–253.
- Noise and the reality gap: The use of simulation in evolutionary robotics. In European Conference on Artificial Life, pages 704–720. Springer.
- The transferability approach: Crossing the reality gap in evolutionary robotics. IEEE Transactions on Evolutionary Computation, 17(1):122–145.
- Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556–5566. PMLR.
- Integrating safety constraints into adversarial training for robust deep reinforcement learning. Information Sciences.
- Moderate environmental variation across generations promotes the evolution of robust solutions. Artificial life, 24(4):277–295.
- Ng, A. Y. (2004). Feature selection, l 1 vs. l 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, page 78.
- Evolutionary robotics. In Springer handbook of robotics, pages 2035–2068. Springer.
- Efficacy of modern neuro-evolutionary strategies for continuous control optimization. Frontiers in Robotics and AI, 7:98.
- Robust optimization through neuroevolution. PloS one, 14(3):e0213193.
- Hybrid control for combining model-based and model-free reinforcement learning. The International Journal of Robotics Research, page 02783649221083331.
- A unified approach to evolving plasticity and neural geometry. In The 2012 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
- Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201.
- Improved techniques for training gans. Advances in neural information processing systems, 29.
- Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864.
- Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access.
- High dimensions and heavy tails for natural evolution strategies. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pages 845–852.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
- Parameter-exploring policy gradients. Neural Networks, 23(4):551–559.
- Stagge, P. (1998). Averaging efficiently in the presence of noise. In International Conference on Parallel Problem Solving from Nature, pages 188–197. Springer.
- Scaling covariance matrix adaptation map-annealing to high-dimensional controllers. In Deep Reinforcement Learning Workshop NeurIPS 2022.
- Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949–980.