Neural optimal feedback control with local learning rules (2111.06920v1)

Abstract: A major problem in motor control is understanding how the brain plans and executes proper movements in the face of delayed and noisy stimuli. A prominent framework for addressing such control problems is Optimal Feedback Control (OFC). OFC generates control actions that optimize behaviorally relevant criteria by integrating noisy sensory stimuli and the predictions of an internal model using the Kalman filter or its extensions. However, a satisfactory neural model of Kalman filtering and control is lacking because existing proposals have the following limitations: not considering the delay of sensory feedback, training in alternating phases, and requiring knowledge of the noise covariance matrices, as well as that of systems dynamics. Moreover, the majority of these studies considered Kalman filtering in isolation, and not jointly with control. To address these shortcomings, we introduce a novel online algorithm which combines adaptive Kalman filtering with a model free control approach (i.e., policy gradient algorithm). We implement this algorithm in a biologically plausible neural network with local synaptic plasticity rules. This network performs system identification and Kalman filtering, without the need for multiple phases with distinct update rules or the knowledge of the noise covariances. It can perform state estimation with delayed sensory feedback, with the help of an internal model. It learns the control policy without requiring any knowledge of the dynamics, thus avoiding the need for weight transport. In this way, our implementation of OFC solves the credit assignment problem needed to produce the appropriate sensory-motor control in the presence of stimulus delay.

• The paper introduces Bio-OFC, which integrates local synaptic plasticity with adaptive Kalman filtering to address sensory delays and eliminate the need for predefined noise covariances.
• It employs an online recurrent neural model with policy gradients to bypass explicit dynamic models and solve the credit assignment problem.
• Experimental evaluations in reaching and winged flight tasks demonstrate Bio-OFC's robust adaptability in noisy, delayed sensory environments.

Analysis of "Neural Optimal Feedback Control with Local Learning Rules"

This paper addresses prominent challenges in motor control, specifically investigating how neural systems plan and execute movements amidst delayed and noisy sensory stimuli. The framework of Optimal Feedback Control (OFC) stands pivotal in this context, utilizing Kalman filtering to integrate sensory signals with internal models. Despite its utility, existing neural models of Kalman filtering face limitations, such as inadequately accounting for sensory feedback delays and necessitating prior knowledge of noise covariances.

Key Contributions

The authors introduce Bio-OFC, a novel algorithm that integrates adaptive Kalman filtering with a model-free control approach, employing biologically plausible neural networks with local synaptic plasticity rules. Bio-OFC effectively bypasses the need for distinct learning phases and the prerequisite knowledge of noise covariances. It proficiently tackles state estimation with delayed feedback and learns control policies without explicit dynamic models, addressing the credit assignment problem in sensory-motor control.

Methodology

Typically, OFC employs Kalman filters for state estimation, merging predictions of internal models with sensory inputs. In this research, the authors propose a circuit-based model that implements these processes online within a recurrent neural architecture, leveraging local learning rules aligned with biological principles. Moreover, the approach utilizes policy gradient methods to facilitate control without necessitating transport of weights across the network — a significant advance challenging weight transport issues in other models.

Crucially, Bio-OFC embraces synaptic plasticity rules dependent only on locally available pre- and postsynaptic variables or global neuromodulatory signals, rendering the algorithm more biologically feasible than past proposals. Through these mechanisms, the model achieves adaptive system identification and control in real-time, accommodating feedback delays.

Experimental Evaluation

Bio-OFC was tested in various control tasks, including a discrete-time double integrator and biologically relevant tasks like reaching movements and simplified winged flight. In these experiments, the system demonstrated resilience against changing noise conditions and sensory delays, showcasing the model's adaptability in dynamic environments.

Interestingly, in control tasks like human-resembling reaching movements, Bio-OFC captured human-like learning trajectories, aligning well with empirical performance data. This indicates the potential applicability of the model in simulating biological systems and understanding neural mechanisms underlying motor control.

Implications and Future Directions

The outcomes of this research suggest that biologically plausible frameworks can efficiently solve complex control problems traditionally addressed by less biologically feasible models. The proposed framework contributes to bridging the gap between neuroscience and control theory, with implications extending to neuroprosthetics and brain-machine interfaces.

Looking forward, challenges lie in expanding this linear paradigm to nonlinear dynamics, possibly through locally adapted linearization or hierarchical models. Additionally, exploring the integration of more complex synaptic plasticity mechanisms or continual learning scenarios might extend the model's applicability to broader cognitive tasks.

In summary, this paper presents a significant advancement in neural models of motor control, offering a promising avenue for future research on neural computational models and their applications in artificial intelligence and neuroscience.