A Role of Environmental Complexity on Representation Learning in Deep Reinforcement Learning Agents (2407.03436v1)
Abstract: The environments where individuals live can present diverse navigation challenges, resulting in varying navigation abilities and strategies. Inspired by differing urban layouts and the Dual Solutions Paradigm test used for human navigators, we developed a simulated navigation environment to train deep reinforcement learning agents in a shortcut usage task. We modulated the frequency of exposure to a shortcut and navigation cue, leading to the development of artificial agents with differing abilities. We examined the encoded representations in artificial neural networks driving these agents, revealing intricate dynamics in representation learning, and correlated them with shortcut use preferences. Furthermore, we demonstrated methods to analyze representations across a population of nodes, which proved effective in finding patterns in what would otherwise be noisy single-node data. These techniques may also have broader applications in studying neural activity. From our observations in representation learning dynamics, we propose insights for human navigation learning, emphasizing the importance of navigation challenges in developing strong landmark knowledge over repeated exposures to landmarks alone.
- Effects of home environment structure on navigation preference and performance: A comparison in veneto, italy and utah, usa. Journal of Environmental Psychology, 74:101580, 2021.
- A geometric perspective on optimal representations for reinforcement learning. Advances in neural information processing systems, 32, 2019.
- Gender differences in relation to wayfinding strategies, navigational support design, and wayfinding task difficulty. Journal of environmental psychology, 29(2):220–226, 2009.
- On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
- Dopamine neurons report an error in the temporal prediction of reward during learning. Nature neuroscience, 1(4):304–309, 1998.
- Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG), pages 1–8. IEEE, 2016.
- Task-dependent representations in rat hippocampal place neurons. Journal of Neurophysiology, 78(2):597–613, 1997.
- Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms, 2018.
- Investigating navigation strategies in the morris water maze through deep reinforcement learning. Neural Networks, 172:106050, 2024.
- On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1–9. PMLR, 2021.
- Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023.
- Cognitive mappers to creatures of habit: differential engagement of place and response learning mechanisms predicts human navigational behavior. Journal of neuroscience, 31(43):15264–15268, 2011.
- Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
- Path integration and the neural basis of the ’cognitive map’. Nature Reviews Neuroscience, 7(8):663–678, 2006.
- Aging and spatial cues influence the updating of navigational memories. Scientific Reports, 9(1):11469, 2019.
- Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016.
- Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016.
- Place cells, grid cells, and memory. Cold Spring Harbor perspectives in biology, 7(2):a021808, 2015.
- A meta-analysis of sex differences in human navigation skills. Psychonomic bulletin & review, 26:1503–1528, 2019.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- The development of spatial representations of large-scale environments. Advances in child development and behavior, 10:9–55, 1975.
- The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902, 2023.
- Reinforcement learning: An introduction. MIT press, 2018.
- Cognitive maps: Some people make them, some people struggle. Current directions in psychological science, 27(4):220–226, 2018.