Mechanism of temporal information compression and integration in recurrent agents

Characterize how the recurrent neural network agents trained on the ViZDoom foraging task compress past sensory information and integrate it with incoming visual observations to support survival, especially in the CIFAR-10 condition where temporal integration was critical.

Background

The study finds that recurrent architectures are crucial to fully exploit complex vision models on visually demanding tasks, notably when object classes are represented by CIFAR‑10 images. This implies a temporal integration mechanism that compresses and accumulates information over time.

The authors explicitly list as an open question how their recurrent agents perform this compression and integration of past and current sensory information, noting its critical role in the most complex task. They suggest this process is analogous to Kalman filtering, motivating a deeper mechanistic analysis.

References

In future work we aim to address at least two open questions from this study. Secondly, we aim to better characterize how our RNN agents compressed past sensory information and integrated it with incoming observations, which was critical for survival in our CIFAR-10 task.

— A computational approach to visual ecology with deep reinforcement learning  (2402.05266 - Sokoloski et al., 2024) in Discussion

The internal role of the LSTM cell remains unresolved. Aggregate recall accuracy does not reveal whether the cell stores compressed snapshots of the trajectory, features from which earlier rows can be reconstructed, a controller-dependent summary, or information that merely improves the residual stream's own recurrent dynamics. Unlike the CoTFormer analysis, the present study does not intervene directly on the LSTM cell, inspect its gates across depth, or test whether a particular cell state is necessary or sufficient for recall.

— LSTM-UT and Recurrent-Depth Transformers on Cellular Automata  (2609.19521 - Kavuncu, 17 Sep 2026) in Section 7.5, “Interpretation and Limitations”