Emergent Symbol-like Number Variables in Artificial Neural Networks (2501.06141v2)

Abstract: What types of numeric representations emerge in neural systems? What would a satisfying answer to this question look like? In this work, we interpret Neural Network (NN) solutions to sequence based counting tasks through a variety of lenses. We seek to understand how well we can understand NNs through the lens of interpretable Symbolic Algorithms (SAs), where SAs are defined by precise, abstract, mutable variables used to perform computations. We use GRUs, LSTMs, and Transformers trained using Next Token Prediction (NTP) on numeric tasks where the solutions to the tasks depend on numeric information only latent in the task structure. We show through multiple causal and theoretical methods that we can interpret NN's raw activity through the lens of simplified SAs when we frame the neural activity in terms of interpretable subspaces rather than individual neurons. Depending on the analysis, however, these interpretations can be graded, existing on a continuum, highlighting the philosophical question of what it means to "interpret" neural activity, and motivating us to introduce Alignment Functions to add flexibility to the existing Distributed Alignment Search (DAS) method. Through our specific analyses we show the importance of causal interventions for NN interpretability; we show that recurrent models develop graded, symbol-like number variables within their neural activity; we introduce a generalization of DAS to frame NN activity in terms of linear functions of interpretable variables; and we show that Transformers must use anti-Markovian solutions -- solutions that avoid using cumulative, Markovian hidden states -- in the absence of sufficient attention layers. We use our results to encourage interpreting NNs at the level of neural subspaces through the lens of SAs.

• The paper demonstrates that neural networks can form abstract, mutable number representations purely from training on numeric tasks.
• It reveals that architectures like RNNs and transformers employ distinct strategies, with RNNs using cumulative counting and transformers computing information on-demand.
• The study highlights how task structure and model size influence emergent symbolic alignment, guiding future research in interpretable neural-symbolic processing.

Emergent Symbol-like Number Variables in Artificial Neural Networks

In the paper of numerical cognition in artificial neural networks (ANNs), the paper "Emergent Symbol-like Number Variables in Artificial Neural Networks" explores the emergence of numerical representations within network architectures. Specifically, it investigates whether ANNs can develop abstract, mutable, and slot-like numerical variables akin to those manipulated in symbolic algorithms, and how these representations evolve during training under various conditions.

The authors trained sequence-based neural systems using the Next Token Prediction (NTP) objective on a series of numeric tasks, subsequently analyzing the neural solutions through causal abstractions and symbolic algorithms. The paper utilized causal interventions and visualization techniques to discern that ANNs are indeed capable of constructing changeable, latent number variables purely from NTP objectives. However, these symbol-like representations did not manifest uniformly across all tasks and model architectures, with transformers displaying distinctive solution methods compared to recurrent models.

Key Findings

• Neural Representations: Artificial neural models, when trained on numeric tasks, show the development of representations that resemble interchangeable, mutable number variables. These emerged despite no explicit symbolic teaching, suggesting that neural systems can inherently approximate symbolic number concepts under certain conditions.
• Architecture-Dependent Solutions: The paper found that different network architectures approach numeric problems differently. Recurrent Neural Networks (RNNs), including GRUs and LSTMs, tend to develop a cumulative counting strategy, linking performance closely with a unified internal representation aligning with symbol-like variables. In contrast, transformers leverage their architectural advantage to frequently compute relevant information, avoiding the use of a cumulative state.
• Task Variance Effects: Variations in task structure significantly influenced the models' numeric solutions. Tasks where demonstration and response tokens differed led to stronger symbolic alignment compared to tasks with identical tokens for both demonstrations and responses, suggesting the importance of task structure in shaping neural representations.
• Gradience in Neural Symbols: Despite the emergence of these representations, a degree of gradience persists, highlighting the challenge of fully interpreting neural computations through simplified symbolic stories. This gradience was more pronounced in models with smaller representational capacities and larger numerical manipulations, suggesting that symbolic alignment may improve with increased model complexity.
• Model Size and Training: Larger model architectures showed better alignment with symbolic programs, indicating that model size could play a critical role in facilitating more symbolic-like processing. The paper showed that symbolic alignment begins emerging congruently with task performance, with large models rapidly approaching their peak alignment post the initial performance surge.

Implications and Future Directions

The implications of this research are broad for both the practical applications of neural networks and the theoretical understanding of neural computation. On the practical side, insights into how NNs can approximate symbolic reasoning might guide the development of more interpretable models, utilizing architectures and training regimes that encourage symbolic-like processing. Theoretically, understanding emergent numerical cognition in ANNs provides a basis for exploring analogous phenomena in biological neurons and offers a bridge between neural and symbolic processing paradigms.

Future research could explore the integration of symbolic-like capabilities in larger-scale tasks, pushing the limits of how far emergent numerical reasoning aligns with explicit symbolic logic. Addressing the gradience in symbolic representations might enhance models' reliability and interpretability. Additionally, extending this analysis across varied cognitive tasks could map the boundaries of neural-symbolic processing further, potentially unlocking more sophisticated AI models capable of complex reasoning with transparency akin to symbolic logic systems.