Mechanisms behind long-timescale inductive biases in deep learning and their relation to biological intrinsic timescales

Determine the exact mechanisms by which equipping deep learning architectures with inductive biases appropriate for long-timescale tasks leads to performance improvements, and ascertain whether these mechanisms are related to intrinsic neural timescales observed in biological networks.

Background

The paper surveys advances in deep learning architectures for modeling long-range temporal dependencies, including gated recurrent networks with chrono initialization, and deep state space models such as S4, which show strong performance on long-timescale tasks. It also notes substantial gains from self-supervised pretraining on long sequences for Transformer-based and SSM-based models.

Despite these improvements, the authors point out that the precise mechanisms by which such inductive biases and training strategies enhance long-timescale computation remain to be clarified, and it is not yet determined whether these mechanisms bear a principled relationship to intrinsic neural timescales measured in biological systems.