Learned feature representations are biased by complexity, learning order, position, and more (2405.05847v3)

Abstract: Representation learning, and interpreting learned representations, are key areas of focus in machine learning and neuroscience. Both fields generally use representations as a means to understand or improve a system's computations. In this work, however, we explore surprising dissociations between representation and computation that may pose challenges for such efforts. We create datasets in which we attempt to match the computational role that different features play, while manipulating other properties of the features or the data. We train various deep learning architectures to compute these multiple abstract features about their inputs. We find that their learned feature representations are systematically biased towards representing some features more strongly than others, depending upon extraneous properties such as feature complexity, the order in which features are learned, and the distribution of features over the inputs. For example, features that are simpler to compute or learned first tend to be represented more strongly and densely than features that are more complex or learned later, even if all features are learned equally well. We also explore how these biases are affected by architectures, optimizers, and training regimes (e.g., in transformers, features decoded earlier in the output sequence also tend to be represented more strongly). Our results help to characterize the inductive biases of gradient-based representation learning. We then illustrate the downstream effects of these biases on various commonly-used methods for analyzing or intervening on representations. These results highlight a key challenge for interpretability $-$ or for comparing the representations of models and brains $-$ disentangling extraneous biases from the computationally important aspects of a system's internal representations.

• The paper demonstrates that neural networks favor simpler features due to inherent biases from complexity and learning order.
• It reveals that feature learning is path-dependent, where early and frequent features acquire stronger representational weight.
• The study urges rethinking interpretability methods, as standard analyses may misrepresent the computational role of complex features.

Analyzing Biases in Learned Feature Representations

The paper "Learned feature representations are biased by complexity, learning order, position, and more" by Lampinen, Chan, and Hermann, presents an insightful exploration into how various extraneous factors influence the learned feature representations in neural networks. This paper is pivotal for researchers focused on representation learning, interpretability, and cognitive neuroscience, as it uncovers systematic biases that may affect how models represent features, separate from their computational roles.

Key Findings

1. Complexity Bias: The authors demonstrate that simpler features tend to be represented more strongly in neural networks than more complex features. This is observed even when these features serve equivalent computational roles. Easier features, typically linear ones, are learned more rapidly and dominate the feature representations, occupying more representational variance. Importantly, complex features, when decomposed into their constitutive parts, show that the representation gap is attributed to both feature complexity and learning order.
2. Learning Order and Path-Dependency: The paper highlights the path-dependency of representation learning. When features are learned sequentially, the order significantly affects their representational strength. Features learned earlier or in isolation, especially simpler ones, have a stronger representation than those learned later or together with more features.
3. Prevalence and Output Position Impact: The research extends its analysis to show that more frequently occurring features during training, as well as those decoded earlier in an output sequence, tend to have stronger representations. This suggests biases due to data distribution and task configuration, respectively.
4. Architecture-Specific Interactions: Various architectures, including MLPs, transformers, and CNNs/ResNets, exhibit these biases, although the nuances differ. The interactions between architecture, optimizer, and training regime slightly modulate these biases. For example, in transformer models handling sequence data, the number of tokens involved and their position heavily influence feature representation.

Implications for Interpretability

The findings carry significant implications for interpretability in neural networks. Many interpretability methods implicitly assume that representation variance correlates with computational importance—a premise that may be biased due to the extrinsic factors identified. For instance, using top principal components in dimensionality reduction could unjustly overshadow complex features, skewing interpretability outcomes.

Broader Implications

• Neural Model Comparisons: The biases found here caution against straightforward comparisons of representational similarities between different systems (e.g., AI models and brains). It highlights that representational similarities may not reflect true computational similarities due to these representational biases.
• Artificial Intelligence and Cognitive Neuroscience: Within AI, these biases influence downstream applications, affecting how models generalize or transfer to new tasks. In cognitive neuroscience, this research provides considerations for comparing neural representations between models and biological systems.

Speculations on Future Developments

Future research can delve into mitigating these representational biases, perhaps through novel architectures or training paradigms designed to equitably represent feature complexities. Additionally, exploring the impact of these biases in larger, more naturalistic datasets may reveal further intricacies. Given the critical role these biases play, understanding and addressing them will be crucial for developing truly interpretable and comparable intelligent systems.

In summary, this paper provides a detailed characterization of biases in feature representations within neural networks, thereby offering a foundational understanding for the field of representation learning and its ramifications in both machine learning and computational neuroscience.