Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2010.02066v3)

Abstract: Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality. In this paper, we present a novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. Using this powerful tool, we contribute an extensive study of emerging modularity in NNs that covers several standard architectures and datasets. We demonstrate how common NNs fail to reuse submodules and offer new insights into the related issue of systematic generalization on language tasks.

• The paper introduces a novel method using differentiable binary weight masks to identify task-specific subnetworks in neural networks.
• Experiments on various architectures show that networks exhibit strong specialization with limited reusability, highlighting module exclusivity.
• Findings indicate that typical neural nets lack intrinsic modularity, prompting future exploration of designs that incentivize systematic compositionality.

Functional Modularity in Neural Networks: An Evaluation of Emerging Modular Structures

The concept of modularity has long been considered both an organizational cornerstone and a beneficial trait in artificial systems, including neural networks (NNs). This paper proposes a novel approach for assessing the potential emergence of modular structures within NNs, emphasizing a need to associate these modules with specific functionalities. The authors introduce a method based on differentiable binary weight masks for identifying subnetworks within an NN that are responsible for particular tasks. Their work explores the extent to which existing NN architectures display functional modularity across diverse datasets and tasks.

Method

The method involves training binary weight masks to discern the weights crucial for the execution of specific functions or subtasks within a pre-trained NN, leaving the network's weights themselves unchanged. This technique uniquely permits the identification of subnetworks that contribute to specific functionalities, thereby offering insights into modular roles within the larger network. The paper applies this methodology to various NN architectures, including RNNs, Transformers, FNNs, and CNNs.

Analysis of Modularity

1. Specialize and Reuse: The primary focus is on two properties: specialize, where modules perform distinct functions, and reuse, where the same module can apply similar functions across multiple input scenarios. The authors evaluate these properties using carefully constructed synthetic tasks that exploit shared or separate input/output interfaces to nudge a network towards these specializations.
2. Synthetic Experiments: Through experiments such as addition/multiplication tasks and double addition experiments, a clear trend emerges: typical NNs showcase more functional independence (specialize) than reusability, with shared weights often being limited to I/O layers rather than indicating genuine functional overlap.
3. Transfer Learning: Transfer learning experiments using permuted MNIST reveal limited transfer and sharing unless network capacity is overwhelmed, which forces shared weight utilization. This bolsters the argument that subtasks in different tasks often require dedicated sets of weights and limits to reuse.

Empirical Demonstrations

1. Algorithmic Tasks: The paper examines how NNs handle tasks like systematic generalization using the SCAN dataset and mathematical reasoning tasks. It finds evidence that NNs tend to create combination-specific weight configurations, highlighting potential shortcomings in systematic compositionality—a trait necessary for robust algorithmic reasoning.
2. CNN Class Exclusivity: Similarly, experiments with CNNs on CIFAR-10 demonstrate significant dependencies on class-specific weights, a behavior that asserts a lack of shared feature representation across similar classes, indicating strong class exclusivity in learned features.

Implications and Future Directions

The research suggests that the traditional design of NNs does not inherently promote modularity defined by functional reuse. This observation raises important questions regarding the interpretability and generalization capabilities of NNs. Identifying ways to introduce modular inductive biases in network architectures could significantly enhance data efficiency and systematic compositionality. Future research could involve integrating new architectural modifications or training protocols that naturally cultivate modularity, echoing biological precedents and enhancing NN integration, performance, and scalability across more complex, real-world environments.