Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors

Abstract: Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small LLM. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.

• The paper decomposes gradient mechanisms into direct learning, pre-caching, and circuit sharing, explaining how seemingly useless features emerge in next-token predictors.
• Experimental analysis on toy tasks and OthelloGPT confirms that disabling pre-caching and circuit sharing significantly hampers the formation of robust internal representations.
• The study introduces SAE-based unsupervised analysis for large models, offering actionable insights to optimize training regimes and improve structural reasoning.

Emergence of Seemingly Useless Features in Next-Token Predictors

Introduction and Motivation

Transformer-based LLMs trained with next-token prediction (NTP) objectives frequently develop abstract internal features that seem irrelevant for predicting the immediate subsequent token. Although prior research has focused primarily on the teleological view — understanding features in terms of their function within model circuits — the developmental dynamics responsible for these features' emergence have remained largely unexplored. This paper decomposes the gradient signal that governs feature formation in causally masked Transformers, classifying learning mechanisms into direct learning, pre-caching, and circuit sharing. These mechanisms explain how NTP-useless features arise, providing a rigorous framework for attributing their origins and quantifying their influence.

Information-Flow Decomposition

The authors formalize gradient decomposition in Transformer architectures, identifying three distinct mechanisms:

• Direct Learning: The canonical route where information in the residual stream at position $i$ and layer $k$ influences the prediction of $x_{i+1}$ via gradients that directly flow through $r_{\theta, i}^k(x)$ and $\hat{x}_{i+1}$.
• Pre-Caching: Information at position $i$ can influence the loss at subsequent positions $j > i$, since future tokens may attend to $i$. Gradients flow from losses at future positions to earlier representations, incentivizing the learning of features that are useful later but not immediately.
• Circuit Sharing: Due to parameter tying across positions, gradient signals can induce cross-position transfer, enabling features optimum for one location to be encoded elsewhere, regardless of their local utility in next-token prediction.

Figure 1: Decomposition of gradient flow for a token in a Transformer: direct, pre-cached, and shared gradient components.

These mechanisms are exhaustively partitioned via a rigorous gradient decomposition theorem, forming the theoretical basis for later attribution experiments.

Experimental Analysis: Toy Tasks

Toy tasks with explicit circuit requirements (Majority, Conditioned Majority) are used to validate that NTP-useless features do not emerge unless pre-caching and circuit sharing mechanisms are active. Models trained with myopic (pre-caching-ablated) and untied parameters (circuit sharing-ablated) consistently fail to develop robust representations of these features, evidenced by decreased probing performance.

Figure 2: Probing performance for NTP-useless features in toy tasks, with performance sharply reduced when pre-caching and circuit sharing are disabled.

Normalized integrated influence analyses demonstrate that the emergence of such features depends on non-direct components, with shared gradients dominating when direct relevance is absent.

Figure 3: Influence component tracking in toy tasks, highlighting the temporal shift between shared and pre-cached contributions at output phases.

OthelloGPT: World Model Fragility and Gradient Attribution

The study applies attribution analysis to the representation of board states in OthelloGPT, distinguishing between NTP-useful (affecting legal moves) and NTP-useless (no effect on next-move legality) square features. A performance gap is documented: NTP-useful features exhibit stronger representational linearity due to direct learning, while NTP-useless features maintain weaker yet nonzero representation via pre-caching and circuit sharing.

Figure 4: Probing gaps and influence attribution for OthelloGPT board features—direct influence is strong for NTP-useful, weak for NTP-useless; pre-cached and shared remain nonzero for both.

Further scrutiny of influence components across board positions reveals nonstationarity with respect to token location, suggesting a dynamic balance between direct and pre-cached contributions as gameplay evolves.

Figure 5: Integrated influence components for central board features, indicating positional and phase-dependent shifts in learning dynamics.

Language Modeling: Syntax, Structure, and Pre-Caching

Experiments on TinyStories-trained small Transformers reveal that pre-caching is dispensable for learning syntactic features (e.g., POS, dependency tags) but necessary for constructing higher-level structural representations (e.g., positional features tracking story progress). Myopic models incur consistently higher losses, underscoring the necessity of pre-caching for tasks requiring coherent text generation.

Figure 6: Probing scores and influence attribution for syntactic (POS, dependency) and high-level (positional) features. Pre-caching influence is significant only for structural features.

Loss curve comparisons further validate the detrimental impact of ablating pre-caching.

Figure 7: Myopic training loss curves for small LMs, showing persistent loss gap relative to non-myopic models.

Feature Attribution in Large-Scale LLMs

Sparse autoencoder (SAE)-based unsupervised analysis is introduced for Gemma 2 2B, circumventing the prohibitive cost of retraining large-scale models. The pre-caching degree $Q(w)$ is estimated via KL-divergence interventions, measuring the relative impact of ablating a feature on immediate versus future token predictions.

A substantial portion of SAE features with extreme $Q(w)$ values are associated with formal reasoning domains (code, math, structural syntax), confirmed via automated prompt-based annotation.

Figure 8: Distribution of $Q(w)$ for Gemma 2 SAE features; tails concentrate formal reasoning-related features.

Steering experiments empirically demonstrate that features with high pre-caching degree induce code-like and punctuation-rich generations when activated, substantiating their functional association.

Figure 9: Steering high $Q(w)$ features in Gemma 2 elevates code and punctuation production.

Look-Ahead vs. Pre-Caching

Contrary to prior hypotheses, the correlation between look-ahead usefulness and pre-caching degree is negative: features with high $Q(w)$ are less aligned with subspaces used for future token prediction. This supports the breadcrumbs hypothesis — look-ahead arises not through explicit planning encoded in pre-cached features, but via recurrent similarities in features useful across positions.

Figure 10: Pearson correlations between look-ahead subspace alignment and pre-caching degree, with negative slopes across look-ahead distances.

Theoretical and Practical Implications

The paper provides a mathematically rigorous, empirically validated framework for understanding how features irrelevant for immediate next-token prediction can still arise and be robustly encoded. The developmental attribution perspective advances mechanistic interpretability, connecting the emergence of features to distinct gradient-induced learning mechanisms rather than their role within circuits or end-task performance alone.

Practically, these findings illuminate the limitations of NTP as a pretraining objective, guiding architectural choices and suggesting alternative training regimes (multi-token objectives, targeted pre-caching modulation) for models tasked with long-horizon reasoning or structural fidelity. The SAE-based attribution framework scales to large models, enabling feature-level interpretability without full retraining.

Conclusion

By decomposing gradient-driven learning mechanisms in Transformers, the paper resolves the apparent paradox of NTP-useless feature emergence. Pre-caching and circuit sharing inject distributed, anticipatory training signals for features with future utility or cross-position relevance. This developmental framework sharpens interpretability, opening new directions for scaling attribution analysis, discovering novel features, and optimizing training procedures for robust structural representations and formal reasoning in LLMs.