Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Abstract: Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer LLMs, yet its role in setting the geometric limits of feature representation remains under-explored. Grounded in the Linear Representation and Superposition Hypotheses - which propose that models encode features as near-orthogonal directions in latent space - we develop a framework for estimating how many such directions a model can support. We first establish the embedding matrix as a measurable proxy for near-orthogonality constraints across the latent space: the boundary between meaningful token relationships and incidental similarity in the pairwise cosine similarity distribution gives a concrete estimate of the model's accepted deviation $\varepsilon$ from perfect orthogonality. Applying this metric across dozens of open-source models reveals two classes: models with high $\varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $\varepsilon$ that maintain it. We then show that the standard Johnson-Lindenstrauss lemma greatly underestimates the packing efficiency of trained representations, and derive an adjusted capacity formula in which the number of near-orthogonal directions depends on the ratio of vectors to dimensions ($k/d$) rather than the raw count - a single modification that cuts prediction error by two orders of magnitude with no extra parameters. Combining these results, we define representational capacity as an upper bound on the number of distinguishable directions available for features and embeddings in a model's latent space. Capacity is exponentially sensitive to $\varepsilon$, and larger models favor tighter orthogonality constraints over maximizing raw capacity - a pattern compatible with several explanations (a stability-capacity trade-off, a ceiling on usable concepts, or confounds with model scale) that we leave to future work.

• The paper presents a geometric framework that quantifies transformer capacity through detailed analysis of the embedding matrix and refined JL bounds.
• It identifies a bimodal behavior in embeddings with distinct ε thresholds, linking near-orthogonality to effective feature superposition.
• Findings suggest that larger models achieve tighter orthogonality, enabling exponential increases in representable features and informing design choices.

Geometric Limits on Feature Representation in Transformer LLMs

Introduction and Theoretical Motivation

This work rigorously quantifies representational capacity in transformers by analyzing the geometric constraints that govern the number of features that can be robustly encoded in the model’s latent space. Central to this investigation are the Linear Representation Hypothesis (LRH), which posits that features correspond to directions in latent space, and the Superposition Hypothesis, which proposes that near-orthogonality enables exponentially many features relative to latent dimensionality. The study grounds its analysis in the embedding matrix, establishing it as a reliable empirical proxy for the geometric structure of the latent space as a whole.

Empirical Characterization of Embedding Geometry

Detailed analysis of the pairwise cosine similarity distributions between token embeddings across dozens of open-source LLMs reveals that most trained models maintain embeddings that are tightly concentrated around zero, denoting a near-orthogonal configuration.

Figure 1: Distribution of pairwise cosine similarity between token embeddings across various models, demonstrating near-orthogonality.

Upon closer inspection, a systematic rightward shift is observed, reflecting a slight aversion to negative similarities—potentially due to optimization for downstream attention mechanism operations—and modestly asymmetric tails indicative of genuine lexical and semantic relationships.

Figure 2: Zoomed view of embedding similarity distributions, revealing the right shift from zero and slightly asymmetric tails.

These relationships are attributed to morphologically related tokens (lexical clusters) and less directly to shared semantic feature directions. For example, the classic "King - Man + Woman ≈ Queen" relationship is observable in vector arithmetic over the embedding space, but only through a distributed semantic structure rather than precise vector arithmetic.

Figure 3: Lexical relationships

Figure 4: The top 40 most similar embeddings to the vector $$x_{\text{king}} - x_{\text{man}} + x_{\text{woman}}$$.

Quantifying and Generalizing the Deviation from Orthogonality

The accepted deviation $\varepsilon$ from perfect orthogonality is estimated via the empirical boundary between incidental and relationship-driven similarity, operationalized as $\mu + 2\sigma$ of the pairwise similarity distribution. Applying this heuristic across a diverse model set reveals two distinct classes:

• Models with high $\varepsilon$ ($>0.2$): embeddings are not close to orthogonal, and superposition is likely not leveraged at the embedding level.
• Models with low $\varepsilon$ ($<0.1$): the embedding space is tightly near-orthogonal, supporting efficient superposition.

Figure 5: Two classes of models

This bimodal split is highly robust, with most high-capacity (large $d_{model}$) models falling into the near-orthogonal regime.

Revisiting the Johnson-Lindenstrauss Framework

The Johnson-Lindenstrauss (JL) lemma is traditionally invoked to bound how many vectors can be mutually near-orthogonal in $d$ dimensions. However, applying the best-proven JL constant still leads to bounds that underestimate real embedding matrices by orders of magnitude. Empirical analysis of random versus optimized vector arrangements demonstrates that training via gradient descent achieves far denser packings than random projections predict.

Figure 6: The standard JL relationship $\varepsilon = \sqrt{C \cdot \ln(k) / d}$.

Figure 7: Standard JL formula (left) vs.\ the adjusted relationship (right) fitted to optimized vector data. Both have one free parameter; the adjusted form fits dramatically better. Red points: empirical data.

By replacing $\varepsilon$0 with $\varepsilon$1, the authors obtain an empirically accurate capacity formula:

$\varepsilon$2

or, equivalently,

$\varepsilon$3

where $\varepsilon$4 is fitted empirically. This single modification reduces prediction error for capacity by two orders of magnitude.

Figure 8: The fully parameterized formula $\varepsilon$5.

Defining Representational Capacity and Model Comparisons

The resulting representational capacity, defined as the upper bound on near-orthogonal directions given $\varepsilon$6 and $\varepsilon$7, sets a strict geometric limit—not merely a function of the raw number of units. This capacity is exponentially sensitive to $\varepsilon$8; small increases in overlap tolerance permit orders-of-magnitude more features.

For "Class 2" models ($\varepsilon$9), empirically observed capacities reach $\mu + 2\sigma$0–$\mu + 2\sigma$1 reachable directions, which suffices to accommodate enormous vocabularies and feature counts. Critically, larger $\mu + 2\sigma$2 values correlate with tighter orthogonality ($\mu + 2\sigma$3 decreases as $\mu + 2\sigma$4 increases), suggesting that scaling models increases not just raw capacity but representational stability.

Embeddings, Unembeddings, and Feature Competition

The analysis further shows that the pool of near-orthogonal directions is shared by embeddings, unembeddings, and feature directions extracted via methods such as sparse autoencoders. Untied models (separate embedding and unembedding matrices) utilize distinct slices of this geometric resource, while tied models' embeddings often do not exhibit near-orthogonality.

Figure 9: Cosine similarity between corresponding token embeddings and unembeddings for several models with untied matrices.

Figure 10: Distribution of cosine similarity between each token unembedding and all others.

Implications and Future Directions

The implications of this geometric framing are multifold:

• Model Design: Representational capacity establishes a principled, quantitative basis for selecting $\mu + 2\sigma$5, in contrast to current heuristics. Explicitly targeting a desired capacity (subject to empirical $\mu + 2\sigma$6) may minimize wasted dimensions or avoid interference-limited instability.
• Feature Scaling Laws: This framework motivates a potential "representational scaling law," paralleling existing compute and data scaling laws, to assess whether capacity bottlenecks align with observed model performance and emergent abilities.
• Architectural Insights: The observed trade-offs between capacity and orthogonality stability (smaller $\mu + 2\sigma$7 in larger models) suggest fundamental scaling constraints not addressed by current depth/width/parameter-count scaling analyses.
• Superposition Utilization: High-capacity models that do not fully exploit maximum possible $\mu + 2\sigma$8 imply either latent stability constraints or an upper bound on usable features.
• Open Questions: The strict bimodal split in observed models raises unaddressed questions about architectural choices (e.g., embedding/unembedding tying, normalization, and training technique) and the extent to which the latent space inherits embedding matrix geometry. Further, measuring layerwise and input-dependent $\mu + 2\sigma$9 would clarify the representational bottlenecks most critical in practice.

Conclusion

This paper establishes a quantitative geometric framework for understanding the representational limits of transformer LLMs. By empirically grounding capacity bounds in embedding matrix structure and showing that learned representations vastly exceed the arrangement efficiency of random projections, the analysis reveals that representational capacity is both a shared, finite resource and a critical axis for model scaling. The exponential sensitivity of capacity to $\varepsilon$0 highlights the importance of structural constraints beyond raw dimension, reframing $\varepsilon$1 as the gatekeeper of geometric possibility rather than feature inevitability. Future work should directly measure latent near-orthogonality, connect representational capacity to downstream abilities, and examine the model-scale processes that select the empirical $\varepsilon$2 observed in the wild.