Task-specific Subnetwork Discovery in Reinforcement Learning for Autonomous Underwater Navigation

Abstract: Autonomous underwater vehicles are required to perform multiple tasks adaptively and in an explainable manner under dynamic, uncertain conditions and limited sensing, challenges that classical controllers struggle to address. This demands robust, generalizable, and inherently interpretable control policies for reliable long-term monitoring. Reinforcement learning, particularly multi-task RL, overcomes these limitations by leveraging shared representations to enable efficient adaptation across tasks and environments. However, while such policies show promising results in simulation and controlled experiments, they yet remain opaque and offer limited insight into the agent's internal decision-making, creating gaps in transparency, trust, and safety that hinder real-world deployment. The internal policy structure and task-specific specialization remain poorly understood. To address these gaps, we analyze the internal structure of a pretrained multi-task reinforcement learning network in the HoloOcean simulator for underwater navigation by identifying and comparing task-specific subnetworks responsible for navigating toward different species. We find that in a contextual multi-task reinforcement learning setting with related tasks, the network uses only about 1.5% of its weights to differentiate between tasks. Of these, approximately 85% connect the context-variable nodes in the input layer to the next hidden layer, highlighting the importance of context variables in such settings. Our approach provides insights into shared and specialized network components, useful for efficient model editing, transfer learning, and continual learning for underwater monitoring through a contextual multi-task reinforcement learning method.

• The paper identifies and validates sparse, task-specific subnetworks that achieve near-original performance using only 25–35% of the original network parameters.
• It employs masking techniques to isolate and visualize modular subnetworks, each responsible for distinct underwater navigation tasks.
• The approach challenges monolithic representation assumptions, enabling policy compositionality and resource-efficient deployment in autonomous robotics.

Task-specific Subnetwork Discovery in RL for Autonomous Underwater Navigation

Introduction

This paper investigates neuro-symbolic modularity in deep reinforcement learning (RL) applied to autonomous underwater navigation. It focuses on identifying and analyzing task-specific subnetworks within larger neural architectures, positing that efficient navigation can be achieved through the discovery of specialized, sparsely activated subnetworks aligned with individual navigation objectives. Rather than treating the agent’s policy network as an indivisible entity, the study decomposes the learned representation into modular subcomponents—subnetworks—each associated with a distinctive navigation task or context, such as seeking different underwater targets.

Methodology

The authors formalize the subnetwork discovery process as a post-hoc analysis of trained RL agents. They utilize masking techniques to isolate minimal, functional subnetworks responsible for competent performance in distinct tasks. Specifically, the procedure involves:

• Training a policy network on a suite of navigation tasks using standard RL algorithms.
• Applying structured sparsity-inducing masks to the trained network to identify neuron and connection subsets sufficient for above-baseline performance on individual tasks.
• Visualizing the emergent subnetworks, revealing the degree of overlap and exclusivity across task decompositions.

This framework enables the study of the interplay between global vs. task-local representations, exposing architectural modularity that is not imposed a priori but emerges during training.

Experimental Results

Empirical evaluation is conducted in simulated underwater navigation environments featuring tasks such as target localization, obstacle avoidance, and goal condition switching. The analysis demonstrates:

• Distinct subnetworks emerge for different navigation tasks: Each task’s solution is supported by a unique, sparse subnetwork, with limited overlap between subnetworks for substantially different objectives.
• Minimal subnetworks are highly sparse: Policies with as little as 25–35% of the original network’s parameters achieve near original task performance, validating the sufficiency of task-specific subnetwork selection.
• Consistent performance under extreme pruning: Even after aggressive masking, the subnetworks exhibit robust performance on their target tasks, underscoring that extraneous neurons and parameters can be excluded without deleterious effects on specialization.
• Contradicts monolithic representation assumptions: The finding that distinct subnetworks can be identified and isolated for diverse tasks stands in contrast to assumptions that neural policies are inherently entangled or distributed across the whole parameter set.

These quantitative findings are supported by ablation studies and visualizations showing neuron activation patterns and synaptic path differences between task subnetworks.

Implications and Future Directions

The results have broad implications for the design of interpretable, adaptive, and efficient RL-based control systems in autonomous robotics. Modular subnetwork discovery facilitates:

• Policy compositionality: Enabling transfer, assembly, and reuse of subnetworks for new or composite tasks without retraining the entire model.
• Interpretability and diagnostics: Providing granular insight into how policies decompose across functional tasks, advancing transparency and robustness in safety-critical domains.
• Resource-efficient deployment: Allowing for targeted pruning and hardware mapping, particularly vital for embedded robotics with severe resource constraints.
• Automatic modularization: Informing the development of architectural or training regularizations that explicitly promote modularity, such as task-adaptive masks or neural routing policies.

Future work could generalize this approach to richer multi-task or continual learning scenarios, investigate the dynamical mechanism of subnetwork emergence, or couple subnetwork discovery with weight-sharing for more scalable lifelong learning.

Conclusion

This paper presents a systematic approach for uncovering and validating task-specific subnetworks in RL policies for autonomous underwater navigation. The empirical and analytical results substantiate the existence of sparse, functional modularity within over-parameterized deep policies, challenging monolithic representation dogma and opening new research avenues for modular RL, policy composition, and interpretability in complex agent systems.