BEAN: Interpretable Representation Learning with Biologically-Enhanced Artificial Neuronal Assembly Regularization

Abstract: Deep neural networks (DNNs) are known for extracting useful information from large amounts of data. However, the representations learned in DNNs are typically hard to interpret, especially in dense layers. One crucial issue of the classical DNN model such as multilayer perceptron (MLP) is that neurons in the same layer of DNNs are conditionally independent of each other, which makes co-training and emergence of higher modularity difficult. In contrast to DNNs, biological neurons in mammalian brains display substantial dependency patterns. Specifically, biological neural networks encode representations by so-called neuronal assemblies: groups of neurons interconnected by strong synaptic interactions and sharing joint semantic content. The resulting population coding is essential for human cognitive and mnemonic processes. Here, we propose a novel Biologically Enhanced Artificial Neuronal assembly (BEAN) regularization to model neuronal correlations and dependencies, inspired by cell assembly theory from neuroscience. Experimental results show that BEAN enables the formation of interpretable neuronal functional clusters and consequently promotes a sparse, memory/computation-efficient network without loss of model performance. Moreover, our few-shot learning experiments demonstrate that BEAN could also enhance the generalizability of the model when training samples are extremely limited.

• The paper introduces BEAN, showing that integrating first- and second-order neuronal correlations improves model interpretability and performance.
• It employs co-activation divergence to encourage modular neuron assemblies, reducing computational overhead and promoting sparsity.
• Empirical evaluations on datasets like MNIST and CIFAR-10 demonstrate that BEAN regularization outperforms conventional methods in few-shot learning scenarios.

Interpretable and Efficient Learning with BEAN Regularization: A Summary

In the paper "BEAN: Interpretable and Efficient Learning with Biologically-Enhanced Artificial Neuronal Assembly Regularization," researchers address core issues of deep neural networks (DNNs) such as limited interpretability, substantial data requirements, and the reliance on complex architectures. The work presents a novel regularization technique named Biologically-Enhanced Artificial Neuronal assembly (BEAN), inspired by the concept of neuronal assemblies in biological neural networks (BNNs). BEAN imposes correlations and dependencies among neurons within dense layers, drawing from the cell assembly theory.

Key Components of BEAN Regularization

1. Layer-wise Neuron Correlation: The BEAN regularization focuses on building correlations among neurons by integrating their connection strengths to subsequent layers. This involves both first-order and second-order neuron correlation methods. First-order correlation uses a single common neighbor for linkage while second-order correlation involves two common neighbors, making it a more stringent criterion.
2. Co-activation Divergence: This component penalizes neurons with strong connectivity but high activation divergence. Hence, neurons with similar activation patterns are encouraged to form stronger connections, fostering modularity and interpretability.
3. Training Objective: The regularization term is combined with the standard DNN loss, and the resulting function is optimized through backpropagation. The objective is to train both conventional parameters and the regularization term to enhance interpretability and efficiency without altering the network architecture significantly.

Empirical Analysis

The paper provides thorough empirical evaluations over standard datasets such as MNIST, Fashion-MNIST, and CIFAR-10, considering tasks ranging from conventional classification to few-shot learning from scratch.

Results on Interpretability

• Neuronal Assemblies Formation: Through K-means clustering and Silhouette analysis, BEAN was shown to form interpretable neuronal groups within dense layers. These assemblies were correlated with high-level semantic content, akin to human cognitive processes.
• Class Selectivity: BEAN-induced networks displayed clear associations between neuron assemblies and concept classes, unlike conventional DNNs where such associations are absent or weak.
• Activation Patterns: Co-activation among neurons within the same assembly parallels population coding in BNNs, leading to enhanced interpretability of the network’s behavior.

Results on Sparse and Efficient Encoding

• BEAN regularization achieves substantial improvements in memory and computational efficiency. Specifically, it reduces the number of non-zero parameters and floating-point operations (FLOPs) by promoting sparsity in dense layers.
• BEAN-2 showed the highest reductions in memory and computational overhead while maintaining or slightly improving performance compared to existing methods like $\ell_{1}$-norm, group sparsity, and exclusive sparsity.

Generalizability in Few-Shot Learning

• BEAN-regularized models outperformed baseline and comparison models in few-shot learning tasks across multiple datasets, with BEAN-1 typically performing better for extremely limited training samples (1-shot, 5-shot) and BEAN-2 excelling in scenarios with slightly more data (10-shot, 20-shot).
• The results indicate that BEAN helps networks to generalize well from minimal data by efficiently using neuronal resources and promoting structured connectivity.

Theoretical and Practical Implications

The use of BEAN regularization introduces several theoretical and practical implications. Theoretically, it bridges a gap between DNNs and BNNs by integrating biological learning principles into artificial systems. Practically, it enables the development of more interpretable, efficient, and generalizable models without necessitating complex architectural changes. This work suggests promising future directions, such as enhancing the applicability of BEAN to various data types (e.g., text or graphs) and exploring additional biological dynamics like excitatory/inhibitory neuron differentiation.

In conclusion, BEAN regularization presents a significant advancement in the effort to make deep learning systems more interpretable and efficient, drawing inspiration from the modular coding observed in biological neural systems. The approach’s ability to reduce computational overheads and improve performance under data-scarce scenarios marks it as a valuable contribution to the field of computer science.