Insect-inspired modular architectures as inductive biases for reinforcement learning

Abstract: Most reinforcement-learning (RL) controllers used in continuous control are architecturally centralized: observations are compressed into a single latent state from which both value estimates and actions are produced. Biological control systems are often organized differently. Insects, in particular, coordinate navigation, heading stabilization, memory, and context-dependent action selection through distributed circuits rather than a single monolithic controller. Motivated by this contrast, we study an RL policy architecture that decomposes control into interacting modules for sensory encoding, heading representation, sparse associative memory, recurrent command generation, and local motor control, with a learned arbitration mechanism that allocates motor authority across modules. The model is evaluated on a two-dimensional navigation task that require simultaneous food seeking, obstacle avoidance, and predator escape. In a six-seed predator-navigation experiment trained with Proximal Policy Optimization (PPO) for 75 updates, the modular policy achieves the strongest final mean performance among the tested controllers, with final episodic return $-2798.8\pm964.4$ versus $-3778.0\pm628.1$ for a centralized gated recurrent unit (GRU) and $-4727.5\pm772.5$ for a centralized multilayer perceptron (MLP). The modular policy also attains the lowest final value loss and stable PPO optimization statistics while driving module-assignment entropy to $0.0457\pm0.0244$, indicating highly selective control allocation. These results suggest that distributed control can serve as a useful inductive bias for RL problems involving dynamically competing behavioral objectives.

• The paper introduces an RL architecture that factorizes control into specialized modules inspired by insect neural systems.
• The model, tested on a 2D predator-navigation task, achieved 25.9% and 40.8% performance gains over GRU and MLP baselines respectively.
• The modular design reduces arbitration entropy, highlighting how biologically inspired inductive biases can enhance robustness in RL.

Insect-Inspired Modular Architectures as Inductive Biases in Reinforcement Learning

Motivation and Architectural Rationale

This work investigates the role of biologically motivated architectural decomposition as an inductive bias in continuous control reinforcement learning (RL). Modern RL policies are typically centralized, mapping all perceptual, mnemonic, and action processes through a single monolithic function approximator. Contrary to this, insect nervous systems employ distributed architectures, coordinating navigation, memory, and context switching through specialized modules, notably the structured interplay between the central complex (for heading) and the mushroom body (for memory and associative learning). The paper proposes an RL policy architecture that explicitly factorizes the control pipeline into sensory encoding, structured heading representation, sparse associative memory, recurrent command generation, and a set of local motor controllers, all mediated by a learned arbitration mechanism.

Task and Experimental Setup

The evaluation centers on a 2D continuous predator-navigation benchmark that simultaneously taxes goal-seeking, obstacle avoidance, and predator escape—demanding frequent and robust behavioral switching under partial observability. The observation vector is partitioned into sensory, proprioceptive, and task-specific subspaces, each processed independently via dedicated multilayer perceptrons before fusion. Actions comprise forward thrust and rotational velocity, with environment dynamics and rewards tailored to incentivize efficient survival and task acquisition. Policies are optimized using PPO over 75 updates across six random seeds, with uniform training hyperparameters for all tested models.

Modular Controller Architecture

The insect-inspired architecture maintains a comparable parameter budget with standard baselines (476k trainable parameters vs. 438k for MLP, 464k for GRU), but crucially diverges in structural organization.

• Sensory Encoding: Observations are partitioned and individually encoded, inducing a structural bias for specialized perception.
• Heading State: A recurrent ring-attractor-like module integrates continuous heading with proprioception and command signals, preserving a persistent spatial state analogous to insect central complex models.
• Sparse Associative Memory: A high-dimensional, sparsely activated associative memory (modeled after the mushroom body) augments fused sensory signals to support context-sensitive retrieval.
• Recurrent Command Center: Integrates heading, sensory, and memory representations to generate behavioral mixture weights and low-dimensional control signals.
• Local Controllers and Arbitration: Four specialized controllers (stabilize, avoid, approach, explore) independently output candidate actions and priorities; a soft attention arbiter fuses these proposals based on learned weighting, controlled by entropy regularization that encourages selective behavioral switching.

The architecture motivates a separation of high-level behavioral regime mixtures from decisive low-level arbitration, encouraging distributed yet selective control.

Figure 1: Comparison of conventional centralized RL control (left), the proposed insect-inspired modular architecture (center), and empirical performance in the three-seed predator-navigation experiment, showing superior mean episodic return and decreased module-assignment entropy for the modular controller.

Empirical Results

Quantitative analysis demonstrates consistent performance advantages for the modular policy:

• In the six-seed setting, the insect-inspired model achieves a mean episodic return of $-2798.8 \pm 964.4$, outperforming both the GRU ($-3778.0 \pm 628.1$) and the MLP ($-4727.5 \pm 772.5$). This equates to a substantial 25.9% and 40.8% relative improvement, respectively.
• The modular policy obtains the lowest final value loss ($45842.3 \pm 24958.6$), with improvement margins of 24.8% over GRU and 31.0% over MLP.
• PPO optimization metrics (approximate KL, clip fraction) confirm stable policy updates for both the modular and GRU controllers, with the MLP experiencing pathological clipping and KL divergence—underscoring the brittleness of feedforward architectures under partial observability.
• Critically, the modular controller drives the module-assignment entropy to a low $0.0457 \pm 0.0244$, signifying highly selective arbitration and decisive behavioral allocation, whereas the mode mixture entropy remains relatively high ($1.7433 \pm 0.0431$), indicating simultaneous maintenance of distributed behavioral regime hypotheses.

(Figure 1) provides visual evidence for more efficient learning, sharper control allocation, and markedly reduced return variability for the modular architecture compared to the baselines.

Theoretical and Practical Implications

The results reinforce several theoretical insights:

• Architectural Factorization: Decomposing RL policies into specialized computational modules, each responsible for distinct control aspects, systematically improves behavioral switching in scenarios with dynamically competing objectives—a property that is only weakly supported by centralized recurrent models.
• Dynamical Systems Perspective: The modular architecture changes the learning geometry, reducing task interference by embedding the policy in a higher-dimensional, factorized state space. This enables more robust context switching and mitigates catastrophic forgetting between behavioral drives.
• Biologically-Inspired Design: The approach substantiates that biologically plausible division of labor (intrinsic in insect brains) provides computational leverage—even absent explicit biological realism—by exposing inductive biases not typically accessible in deep RL controllers.

On the practical side, the demonstrated performance gain comes without introducing instability or increased computational cost relative to centralized alternatives. However, the modular policy exhibits increased across-seed variability. This variability highlights an outstanding challenge: improving the robustness and perhaps further regularizing the arbitration mechanism for more consistent outcomes.

Prospects for Future Research

This study opens several future directions:

• Enriching the context-dependent specialization mechanisms, such as introducing explicit reflex arcs for hardwired responses (e.g., predator escape) or auxiliary objectives to enforce module diversity.
• Extending the modular approach to more complex continual-learning and lifelong RL settings, where distributional shifts and task switching are prominent.
• Theoretically characterizing the impact of architectural decomposition on the landscape of RL optimization, especially regarding stability, sample efficiency, and transferability.
• Investigating trade-offs between decisiveness and flexibility in module arbitration, potentially leveraging meta-learning or adaptive regularization strategies.

Conclusion

This paper provides empirical and theoretical evidence that distributed, insect-inspired modular architectures confer systematic benefits as inductive biases in RL settings that require dynamic behavioral arbitration. By concretely separating control pathways and leveraging learned arbitration, the modular controller outperforms centralized baselines in both return and optimization stability on a challenging predator-navigation task. The findings encourage continued exploration of biologically motivated architectural priors as a means to improve sample efficiency, adaptability, and robustness in RL systems.