The Need for a Big World Simulator: A Scientific Challenge for Continual Learning
Abstract: The "small agent, big world" frame offers a conceptual view that motivates the need for continual learning. The idea is that a small agent operating in a much bigger world cannot store all information that the world has to offer. To perform well, the agent must be carefully designed to ingest, retain, and eject the right information. To enable the development of performant continual learning agents, a number of synthetic environments have been proposed. However, these benchmarks suffer from limitations, including unnatural distribution shifts and a lack of fidelity to the "small agent, big world" framing. This paper aims to formalize two desiderata for the design of future simulated environments. These two criteria aim to reflect the objectives and complexity of continual learning in practical settings while enabling rapid prototyping of algorithms on a smaller scale.
- Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, 2023.
- A definition of continual reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- Matthew Cook et al. Universality in elementary cellular automata. Complex systems, 15:1–40, 2004.
- Continual backprop: Stochastic gradient descent with persistent randomness. CoRR, abs/2108.06325v3, 2021.
- Maintaining plasticity in deep continual learning. CoRR, abs/2306.13812, 2023.
- Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(255):1–54, 2022.
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. CoRR, abs/1312.6211v3, 2013.
- Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
- Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Conference on Empirical Methods in Natural Language Processing, 2022.
- Scaling laws for neural language models, 2020.
- Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
- Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021.
- Continual learning as computationally constrained reinforcement learning, 2023a.
- Maintaining plasticity via regenerative regularization. CoRR, abs/2308.11958v1, 2023b.
- Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, 2017.
- Gradient episodic memory for continual learning. Advances in neural information processing systems, 2017.
- Understanding plasticity in neural networks. In International Conference on Machine Learning, 2023.
- Variational continual learning. In International Conference on Learning Representations, 2018.
- Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020.
- The neural testbed: Evaluating joint predictions. Advances in Neural Information Processing Systems, 2022.
- Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, 2020.
- Mark Bishop Ring. Continual learning in reinforcement environments. The University of Texas at Austin, 1994.
- Stream-51: Streaming classification and novelty detection from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
- Compete to compute. Advances in neural information processing systems, 2013.
- Sebastian Thrun. Lifelong learning algorithms. In Learning to Learn, pp. 181–209. Springer, 1998.
- A. Turing. On computable numbers, with an application to the entscheidungsproblem. Proceedings of the London Mathematical Society, 2, 42:230–265, 1936.
- Wild-time: A benchmark of in-the-wild distribution shift over time. In Neural Information Processing Systems (Datasets and Benchmarks Track), 2022.
- Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
