---
title: Multidimensional Linear Representation Hypothesis
url: https://www.emergentmind.com/topics/multidimensional-linear-representation-hypothesis-mlrh
type: topic
---

# Multidimensional Linear Representation Hypothesis

Searching arXiv for recent papers on the Multidimensional Linear Representation Hypothesis and closely related representation hypotheses.
The **Multidimensional Linear Representation Hypothesis (MLRH)** is a family of hypotheses in mechanistic interpretability that extends the **Linear Representation Hypothesis (LRH)** from single concept directions to higher-dimensional representational objects. In this literature, LRH states that high-level concepts are represented linearly as directions in a representation space, while MLRH relaxes the one-direction assumption by allowing concepts to occupy subspaces, manifolds, frames, simplices, polytopes, or other low-dimensional geometric structures that remain linearly analyzable in ambient model space [2311.03658][2505.18235][2406.01506]. Across recent work, MLRH functions less as a single canonical theorem than as a research program connecting probing, steering, feature learning, concept geometry, and cross-model alignment [2506.00653][2605.28870].

## 1. From LRH to multidimensional concept geometry

The modern formulation of LRH was made explicit for large language models by defining linear representation in both output and input spaces. In the output or unembedding space, a binary concept is represented by a direction shared by counterfactual token differences; in the input or embedding space, a concept is represented by a direction whose addition increases the probability of that concept while preserving causally separable concepts. This framework also introduced a **causal inner product**, typically involving the inverse covariance of unembedding vectors, so that causally separable concepts become orthogonal in the induced geometry [2311.03658].

MLRH generalizes this picture. One explicit statement, given in later work, is that a concept may require “a subspace (possibly of dimension greater than one)” rather than a single direction, while preserving linear structure at the level of representation and intervention [2605.01844]. A more formal multidimensional definition writes a representation as
$$
\Psi(x) = \sum_{f \in F(x)} \rho_f(x) v_f(x),
$$
where \(v_f(x) \in V_f \subset \mathbb{R}^D\) is a unit vector in a feature-specific subspace and \(\rho_f(x) \ge 0\) measures feature strength [2505.18235]. In this sense, MLRH does not discard LRH; it replaces the fixed one-dimensional feature ray with a richer, feature-conditioned linear object.

A theoretical account of why such linear or low-dimensional structure should arise was developed through a latent variable model for next-token prediction. There, contexts and tokens are mapped to shared latent concept variables, and the softmax cross-entropy objective together with the implicit bias of gradient descent promotes aligned steering vectors for the same concept. For dependent concepts in a Markov Random Field, the resulting steering vectors need not remain strictly one-dimensional; instead, they lie in a low-dimensional subspace whose dimension depends on the Markov blanket [2403.03867]. This provides one route from LRH to genuinely multidimensional structure.

## 2. Core formalizations of MLRH

A central formalization treats each feature as a metric space \((\mathcal{Z}_f, d_f)\) together with a continuous, invertible map
$$
\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},
$$
so that feature values map to unit directions and the image \(\mathcal{M}_f = \phi_f(\mathcal{Z}_f)\) becomes a **representation manifold** [2505.18235]. Under the associated continuous correspondence hypothesis, topology is preserved: intervals become curves, circles become loops, and more complex feature spaces yield higher-dimensional manifolds. The same work further hypothesizes a local relation between cosine similarity and intrinsic feature distance,
$$
\left( \phi_f(z), \phi_f(z') \right) = g_f \left( d_f(z, z')^2 \right),
$$
and proves an isometry theorem stating that shortest on-manifold paths in representation space mirror feature-space distances up to scale [2505.18235].

A second formalization arises in cross-model alignment. The **Linear Representation Transferability (LRT) Hypothesis** assumes that representations in different models are projections of a shared universal basis feature space. If source and target hidden states satisfy
$$
\mathbf{h}_{\ell_T}^T(\mathbf{x}) \approx \mathbf{A}\mathbf{h}_{\ell_S}^S(\mathbf{x}) + \mathbf{p},
$$
then steering vectors, sparse features, and hidden-state directions can be transferred by a learned affine map [2506.00653]. In that framework, MLRH is interpreted as the existence of a **shared, multidimensional, linear structure** underlying different models’ learned representations.

A third line of work refines LRH statistically by decomposing representations into **signal, bias, and noise**:
$$
f_\theta(X) = A(\theta, f) \cdot (Z(X) \odot M(X, \theta, f)) + n_{X,\theta,f}.
$$
Here \(Z(X)\) is a sparse Platonic signal over attributes, \(A(\theta,f)\) is a model-dependent dictionary, \(M\) captures model-specific magnitudes, and \(n\) is noise [2605.28870]. This preserves linear combination structure while explaining why aligned local geometry can coexist with imperfect global agreement across architectures.

| Formulation | Representational object | Representative source |
|---|---|---|
| Classical LRH | Single concept direction with causal inner product | [2311.03658] |
| Subspace/manifold MLRH | Feature-specific subspace \(V_f\) or manifold \(\mathcal{M}_f\) | [2505.18235] |
| Cross-model MLRH | Universal basis features with affine inter-model map | [2506.00653] |
| Statistical MLRH | Sparse linear signal plus bias and noise | [2605.28870] |

## 3. Geometric realizations: frames, simplices, lattices, and cylinders

Several papers instantiate MLRH through specific geometric constructions.

One extension addresses the fact that most words are multi-token. The **Frame Representation Hypothesis (FRH)** models a word \(W\) with tokens \(w_1,\dots,w_t\) as the matrix
$$
W = \begin{pmatrix} u(w_1) & u(w_2) & \dots & u(w_t) \end{pmatrix},
$$
that is, an ordered sequence of vectors rather than a single direction [2412.07334]. Concepts are then represented as centroids of word frames, obtained by a Procrustes optimization on the Stiefel manifold. This extends LRH from single-token words to arbitrary textual data and gives a concrete multidimensional object for concept comparison and control.

A second extension concerns categorical and hierarchical concepts. In work on WordNet-derived concepts, binary features are first represented as vectors with constant in-class projection,
$$
\bar\ell_W^\top g(y) =
\begin{cases}
b_w & y \in Y(w)\\
0 & y \notin Y(w),
\end{cases}
$$
after an alignment transform \(g(y) = A(\gamma(y)-\bar{\gamma}_0)\) [2406.01506]. Categorical concepts are then represented as convex hulls of attribute vectors, and under the paper’s assumptions these vertices form simplices. Hierarchical relations induce orthogonality constraints such as
$$
\bar\ell_w \perp (\bar\ell_z - \bar\ell_w)
\quad \text{for } z \prec w,
$$
so semantic refinement corresponds to orthogonal decomposition across levels [2406.01506].

A third geometric formulation is the **Lattice Representation Hypothesis**, which combines linear attribute directions with thresholds. An attribute \(m\) with direction \(d_m\) and threshold \(T_m\) defines a half-space
$$
\{v \in \mathbb{R}^d \mid v \cdot d_m \ge T_m\},
$$
and a concept with attribute set \(Y\) is the intersection
$$
R(Y) = \{v \in \mathbb{R}^d \mid v \cdot d_m \ge T_m\ \forall m \in Y\}.
$$
These regions induce a concept lattice via Formal Concept Analysis, with meet and join corresponding to geometric intersection and union [2603.01227]. This moves MLRH from subspaces to structured regions defined by multiple linear constraints.

Not all later work accepts global linear-subspace control as sufficient. The **Cylindrical Representation Hypothesis (CRH)** preserves linear decomposition but replaces global orthogonal subspaces with a sample-specific geometry: a central axis defined by a difference vector and a normal plane that controls steering sensitivity. Only certain angular regions of that plane, called sensitive sectors, robustly facilitate concept activation; others can suppress or delay it [2605.01844]. CRH therefore retains multidimensional linear structure while disputing the predictability assumptions often attached to LRH and MLRH.

## 4. Empirical evidence

Empirical support for MLRH spans embeddings, hidden activations, lexical hierarchies, and steering behavior.

In the manifold account of feature representations, embeddings from `text-embedding-large-3` and token activations from GPT2-small and Mistral 7B exhibited loops for cyclical features, curves for linear features, and more complex structures for dates and color names. The reported rank correlations between feature orderings and manifold positions were very high, with values above \(0.97\) for years; shortest-path distances along the recovered manifolds showed Pearson correlation \(0.99\) for log-scaled years and \(0.97\) for dates [2505.18235]. The same study also reported that years in GPT2-small are represented logarithmically rather than linearly.

For categorical and hierarchical geometry, a WordNet-based study estimated representations for **900+ hierarchically related concepts** using Gemma and LLaMA-3, with a detailed Gemma-2B setup retaining **593 noun** and **364 verb synsets** having at least 50 vocabulary tokens [2406.01506]. Linear Discriminant Analysis was used to estimate concept directions, and the resulting projections, cosine relations, and simplex visualizations were consistent with the predicted geometry of hierarchies and categories [2406.01506].

The frame-based multi-token extension reported that **over 99% of words are full-rank (frames)** in Llama 3.1, Gemma 2, and Phi 3, matching the underlying Stiefel-manifold assumption [2412.07334]. It also found that real word frames project positively onto associated concept frames, whereas random frames have near-zero projections [2412.07334].

Automatic manifold discovery through **Supervised Multi-Dimensional Scaling (SMDS)** provided another line of evidence. Applied primarily to temporal reasoning, SMDS analyzed **over 60,000 manifold cases** and found stable geometries across Llama, Qwen, and Gemma models, across **3B, 8B, and 70B** scales, and across base and instruction-tuned variants [2510.01025]. The paper reports that perturbing the identified manifold subspace degrades temporal reasoning performance, whereas perturbing random subspaces has minimal effect, indicating that these manifolds are functionally used rather than merely descriptive [2510.01025].

Evidence for cross-model multidimensional alignment also comes from sparse-feature studies. Sparse autoencoders often produced representations with stronger cross-modal alignment than dense features, and centering plus normalization consistently improved cross-model alignment. The same work reported a strong positive correlation between word frequency and alignment, with degradation proportional to \(f^{-1/2}\), interpreting this as a finite-sample noise effect [2605.28870].

## 5. Steering, probing, and transfer

A major motivation for MLRH is operational: if concepts occupy stable low-dimensional structures, they should support measurement and control.

The original LRH formalization already connected output-space directions to linear probing and input-space directions to steering. Under the causal inner product, the same concept representation can be viewed as both an ideal probe and a steering direction, unifying measurement and intervention [2311.03658]. This remains a foundational template for later multidimensional work.

One practical extension is **SAND**—the **Sum of Activation-base Normalized Difference** estimator—which computes concept directions from activation differences rather than unembedding vectors. If normalized activation differences are modeled as i.i.d. samples from a von Mises-Fisher distribution, then the maximum-likelihood estimate of the concept direction is
$$
\hat{\bar{l}} = \frac{\sum_{i=1}^k \widetilde{l}_i}{\left\| \sum_{i=1}^k \widetilde{l}_i \right\|},
$$
with the activation-space form
$$
\hat{\bar{\lambda}} \propto \sum_{i=1}^k \frac{\widetilde{\lambda}_i}{\|\Psi\widetilde{\lambda}_i\|}.
$$
This removes the dependence on single-token counterfactual pairs and extends LRH-style steering to complex, context-dependent concepts [2502.16385].

FRH operationalizes multidimensional control through **Top-\(k\) Concept-Guided Decoding**. At each generation step, among the top-\(k\) logit candidates, the decoder selects
$$
x_{t+1} = \argmax_{i \in 1,\dots,k} \text{corr}(S, H_i(x)),
$$
where \(S\) is the target concept frame and \(H_i\) is the candidate feature frame [2412.07334]. On Llama 3.1, Gemma 2, and Phi 3, this exposed and steered gender and language biases, and the effect size was controllable through \(k\) [2412.07334].

The transferability framework pushes MLRH beyond single-model analysis. By fitting affine maps between hidden states of models trained on the same data and architecture family, steering vectors learned in smaller models were transferred to larger ones while preserving semantic effect [2506.00653]. Evidence included low reconstruction error for mapped sparse feature matrices—reported as **\(114\) versus much larger errors for random matrices**—and high correspondence in behavioral steering metrics, including multiple-choice propensity correlations **up to \(0.98\)**. Example mean correlations included **0.833** for `self-awareness-good-text-model` and **0.968** for `narcissism` [2506.00653]. In this interpretation, MLRH supports the use of small models as steering and interpretability sandboxes for larger systems.

## 6. Limits, critiques, and open questions

The strongest challenge to MLRH is not whether linear structure exists, but how far it can be pushed.

A theoretical analysis of feature capacity separates **linear representation** from **linear accessibility**. If \(m\) features are \(k\)-sparse, classical compressed sensing with nonlinear decoding requires only
$$
d = O\left(k\log\frac{m}{k}\right),
$$
whereas linearly decoding those same features requires substantially more dimensions. The paper proves
$$
d = \Omega_\epsilon\left(\frac{k^2}{\log k}\log\frac{m}{k}\right)
\quad \text{and} \quad
d = O_\epsilon(k^2\log m)
$$
for the linear case [2602.11246]. This establishes a quantitative gap between storing features linearly and recovering them with linear probes, and it implies that linear accessibility is a meaningfully stronger claim than linear representation alone.

CRH sharpens a different limitation: even if concepts are linearly decomposable, steering may remain intrinsically unstable at the sample level because overlapping concept contributions induce a local axis-plus-plane geometry rather than a globally orthogonal subspace. The paper states that the magnitude of the normal-plane component is predictive of steering intensity, but the phase within the plane is fundamentally unpredictable from observables [2605.01844]. In that account, identical axis and plane configurations can still yield opposite steering outcomes depending on latent concept composition.

This suggests that the current literature does not present a single, settled meaning of MLRH. Some papers use it to denote multidimensional **subspaces**; others emphasize **manifolds**, **frames**, **polytopes**, **half-space lattices**, or **universal basis features** [2505.18235][2412.07334][2603.01227][2506.00653]. A plausible implication is that “MLRH” now functions as an umbrella term for a broader hypothesis class: semantic structure in LLMs is substantially linear, but often only after moving from one-dimensional directions to richer low-dimensional geometric objects.

Within that broader program, the main open questions are already explicit in the literature: how to learn feature metric spaces automatically, how to estimate higher-dimensional manifolds robustly in noisy settings, how to reconcile global linear structure with sample-specific steering instability, and how to characterize when cross-model affine transfer should hold [2505.18235][2605.01844][2506.00653].

Source: https://www.emergentmind.com/topics/multidimensional-linear-representation-hypothesis-mlrh