Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis (2505.11581v1)

Abstract: Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.

• The paper introduces the fractured entangled representation (FER) hypothesis, demonstrating that SGD-trained networks develop disorganized, fragmented internal representations.
• The paper compares conventional SGD training with NEAT-evolved CPPNs, revealing that similar output behaviors can mask significant differences in learned representations.
• The paper suggests that mitigating FER through open-ended search and refined training algorithms could enhance generalization, creativity, and continual learning in AI.

Questioning Representational Optimism

This paper introduces the fractured entangled representation (FER) hypothesis, challenging the common assumption that scaling up deep learning models necessarily leads to improved internal representations. By comparing networks evolved through open-ended search with those trained via conventional SGD, the authors reveal significant differences in their internal representations, despite achieving similar output behaviors. The paper highlights the potential limitations of SGD-trained networks, which tend to exhibit FER, and discusses the implications of FER for generalization, creativity, and continual learning.

Background and Methods

The paper leverages the Picbreeder experiment, where users evolved CPPNs to generate images. CPPNs take pixel coordinates as input and output color values, allowing for the visualization of the entire output behavior of the network. The authors compare CPPNs evolved through NEAT in Picbreeder with those trained using conventional SGD to produce the same images. This comparison allows for a direct examination of the internal representations learned by each method. The code to reproduce the experiments can be found at \href{https://github.com/akarshkumar0101/fer}{https://github.com/akarshkumar0101/fer}.

Figure 1: Overview of the Picbreeder experiment and the differences between open-ended search and conventional SGD.

A key component of this research is the concept of layerization, a process of converting NEAT-evolved CPPNs into dense MLPs. This conversion allows for a fair comparison between the representations learned by NEAT and SGD, as it ensures that both algorithms operate within a comparable function space. The authors demonstrate that SGD struggles to optimize the original NEAT architectures, highlighting the importance of layerization for enabling SGD to reproduce Picbreeder image outputs.

Unified Factored Representations

The paper introduces the concept of unified factored representation (UFR) as an ideal representational strategy. A UFR is characterized by:

• A single, unbroken function for each key capability
• Separation of independent capabilities to prevent interference

The Picbreeder Skull CPPN serves as a canonical example of a UFR. In this representation, symmetry across the y-axis emerges early and persists through the layers. Weight sweeps of the Picbreeder Skull CPPN also yield orderly and meaningful changes that preserve the overall "skull-ness" of the pattern.

Figure 2: An example CPPN, which takes x, y, and $d=\sqrt{x^2+y^2}$ as inputs.

Fractured Entangled Representations

In contrast to UFR, the authors propose the concept of fractured entangled representation (FER) to characterize the disorganized representations often found in SGD-trained networks. FER is characterized by:

• Fractured information: unitary concepts are split into disconnected pieces
• Redundancy: fractured pieces are invoked separately in different contexts
• Entanglement: independent behaviors influence each other inappropriately

The conventional SGD Skull CPPN exhibits FER, with a highly entangled patchwork of features and a failure to encode the underlying symmetry of the skull until the very final layer. Weight sweeps of the SGD CPPN result in incoherent symmetry breaking, demonstrating the lack of understanding of the skull's underlying regularities.

Figure 3: Different trajectories to FER and UFR for outwardly identical solutions.

The paper provides several examples of FER in other models, including the inability of GPT-3 to perform arithmetic consistently across different object types and the challenges faced by text-to-image models in generating images with specific constraints. These examples suggest that FER may be a common phenomenon in deep learning models, potentially limiting their ability to generalize, create, and learn.

Factors Influencing FER and UFR

The authors discuss several factors that may contribute to the emergence of FER and UFR, including:

• Order of learning: Learning concepts in a logical sequence can facilitate the development of UFR, while learning concepts in parallel can lead to FER.
• Network complexity: Starting with small, simple networks and gradually increasing complexity can promote UFR.
• Amount of data: Training on more data may lead to holistic unification and a reduction in FER.
• Open-ended search: Encouraging divergent search processes and serendipitous discoveries can facilitate the evolution of UFR.

The paper highlights the importance of open-ended search in generating UFRs. Open-ended search processes, such as those used in Picbreeder, allow for the discovery of fundamental organizing principles and the evolution of evolvable artifacts. These properties can lead to more robust and adaptable representations.

Implications and Future Directions

The FER hypothesis has significant implications for the future of AI. If FER is indeed a limiting factor in deep learning, then mitigating FER could be critical to improving generalization, creativity, and continual learning. The authors suggest several potential directions for future research, including:

• Developing new training algorithms that encourage UFR
• Exploring architectural changes that mitigate FER
• Investigating the role of open-ended search in generating more robust representations

The paper emphasizes that the goal is not to propose a singular solution, but rather to initiate a broader discussion about the nature of representation in deep learning. By raising awareness of the potential limitations of conventional SGD-trained networks, the authors hope to inspire new approaches to representation learning that produce better internal representations and unlock the full potential of AI.

Conclusion

This paper presents a compelling argument that scaling up deep learning models does not necessarily guarantee improved internal representations. The FER hypothesis challenges the assumption of representational optimism and highlights the potential limitations of SGD-trained networks. By comparing networks evolved through open-ended search with those trained via conventional SGD, the authors demonstrate the importance of UFR and the potential pitfalls of FER. The paper concludes by suggesting several directions for future research aimed at mitigating FER and unlocking the full potential of AI.