Multidimensional Linear Representation Hypothesis

• MLRH is a hypothesis that extends the Linear Representation Hypothesis by representing high-level concepts as multidimensional geometric objects rather than single vectors.
• It introduces methods such as manifold mapping, affine transfer, and statistical decomposition to analyze and steer latent representations across models.
• Empirical studies using LLM activations and sparse feature analyses validate MLRH by demonstrating robust cross-model alignment and controllable intervention methods.

Searching arXiv for papers on the Multidimensional Linear Representation Hypothesis and closely related representation hypotheses. The Multidimensional Linear Representation Hypothesis (MLRH) is a family of hypotheses in mechanistic interpretability that extends the Linear Representation Hypothesis (LRH) from single concept directions to higher-dimensional representational objects. In this literature, LRH states that high-level concepts are represented linearly as directions in a representation space, while MLRH relaxes the one-direction assumption by allowing concepts to occupy subspaces, manifolds, frames, simplices, polytopes, or other low-dimensional geometric structures that remain linearly analyzable in ambient model space (Park et al., 2023, Modell et al., 23 May 2025, Park et al., 2024). Across recent work, MLRH functions less as a single canonical theorem than as a research program connecting probing, steering, feature learning, concept geometry, and cross-model alignment (Bello et al., 31 May 2025, Bangachev et al., 22 May 2026).

1. From LRH to multidimensional concept geometry

The modern formulation of LRH was made explicit for LLMs by defining linear representation in both output and input spaces. In the output or unembedding space, a binary concept is represented by a direction shared by counterfactual token differences; in the input or embedding space, a concept is represented by a direction whose addition increases the probability of that concept while preserving causally separable concepts. This framework also introduced a causal inner product, typically involving the inverse covariance of unembedding vectors, so that causally separable concepts become orthogonal in the induced geometry (Park et al., 2023).

MLRH generalizes this picture. One explicit statement, given in later work, is that a concept may require “a subspace (possibly of dimension greater than one)” rather than a single direction, while preserving linear structure at the level of representation and intervention (Gao et al., 3 May 2026). A more formal multidimensional definition writes a representation as

$\Psi(x) = \sum_{f \in F(x)} \rho_f(x) v_f(x),$

where $v_f(x) \in V_f \subset \mathbb{R}^D$ is a unit vector in a feature-specific subspace and $\rho_f(x) \ge 0$ measures feature strength (Modell et al., 23 May 2025). In this sense, MLRH does not discard LRH; it replaces the fixed one-dimensional feature ray with a richer, feature-conditioned linear object.

A theoretical account of why such linear or low-dimensional structure should arise was developed through a latent variable model for next-token prediction. There, contexts and tokens are mapped to shared latent concept variables, and the softmax cross-entropy objective together with the implicit bias of gradient descent promotes aligned steering vectors for the same concept. For dependent concepts in a Markov Random Field, the resulting steering vectors need not remain strictly one-dimensional; instead, they lie in a low-dimensional subspace whose dimension depends on the Markov blanket (Jiang et al., 2024). This provides one route from LRH to genuinely multidimensional structure.

2. Core formalizations of MLRH

A central formalization treats each feature as a metric space $(\mathcal{Z}_f, d_f)$ together with a continuous, invertible map

$$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$

so that feature values map to unit directions and the image $\mathcal{M}_f = \phi_f(\mathcal{Z}_f)$ becomes a representation manifold (Modell et al., 23 May 2025). Under the associated continuous correspondence hypothesis, topology is preserved: intervals become curves, circles become loops, and more complex feature spaces yield higher-dimensional manifolds. The same work further hypothesizes a local relation between cosine similarity and intrinsic feature distance,

$$\left( \phi_f(z), \phi_f(z') \right) = g_f \left( d_f(z, z')^2 \right),$$

and proves an isometry theorem stating that shortest on-manifold paths in representation space mirror feature-space distances up to scale (Modell et al., 23 May 2025).

A second formalization arises in cross-model alignment. The Linear Representation Transferability (LRT) Hypothesis assumes that representations in different models are projections of a shared universal basis feature space. If source and target hidden states satisfy

$$\mathbf{h}_{\ell_T}^T(\mathbf{x}) \approx \mathbf{A}\mathbf{h}_{\ell_S}^S(\mathbf{x}) + \mathbf{p},$$

then steering vectors, sparse features, and hidden-state directions can be transferred by a learned affine map (Bello et al., 31 May 2025). In that framework, MLRH is interpreted as the existence of a shared, multidimensional, linear structure underlying different models’ learned representations.

A third line of work refines LRH statistically by decomposing representations into signal, bias, and noise:

$$f_\theta(X) = A(\theta, f) \cdot (Z(X) \odot M(X, \theta, f)) + n_{X,\theta,f}.$$

Here $Z(X)$ is a sparse Platonic signal over attributes, $v_f(x) \in V_f \subset \mathbb{R}^D$0 is a model-dependent dictionary, $v_f(x) \in V_f \subset \mathbb{R}^D$1 captures model-specific magnitudes, and $v_f(x) \in V_f \subset \mathbb{R}^D$2 is noise (Bangachev et al., 22 May 2026). This preserves linear combination structure while explaining why aligned local geometry can coexist with imperfect global agreement across architectures.

Formulation	Representational object	Representative source
Classical LRH	Single concept direction with causal inner product	(Park et al., 2023)
Subspace/manifold MLRH	Feature-specific subspace $v_f(x) \in V_f \subset \mathbb{R}^D$3 or manifold $v_f(x) \in V_f \subset \mathbb{R}^D$4	(Modell et al., 23 May 2025)
Cross-model MLRH	Universal basis features with affine inter-model map	(Bello et al., 31 May 2025)
Statistical MLRH	Sparse linear signal plus bias and noise	(Bangachev et al., 22 May 2026)

3. Geometric realizations: frames, simplices, lattices, and cylinders

Several papers instantiate MLRH through specific geometric constructions.

One extension addresses the fact that most words are multi-token. The Frame Representation Hypothesis (FRH) models a word $v_f(x) \in V_f \subset \mathbb{R}^D$5 with tokens $v_f(x) \in V_f \subset \mathbb{R}^D$6 as the matrix

$v_f(x) \in V_f \subset \mathbb{R}^D$7

that is, an ordered sequence of vectors rather than a single direction (Valois et al., 2024). Concepts are then represented as centroids of word frames, obtained by a Procrustes optimization on the Stiefel manifold. This extends LRH from single-token words to arbitrary textual data and gives a concrete multidimensional object for concept comparison and control.

A second extension concerns categorical and hierarchical concepts. In work on WordNet-derived concepts, binary features are first represented as vectors with constant in-class projection,

$v_f(x) \in V_f \subset \mathbb{R}^D$8

after an alignment transform $v_f(x) \in V_f \subset \mathbb{R}^D$9 (Park et al., 2024). Categorical concepts are then represented as convex hulls of attribute vectors, and under the paper’s assumptions these vertices form simplices. Hierarchical relations induce orthogonality constraints such as

$\rho_f(x) \ge 0$0

so semantic refinement corresponds to orthogonal decomposition across levels (Park et al., 2024).

A third geometric formulation is the Lattice Representation Hypothesis, which combines linear attribute directions with thresholds. An attribute $\rho_f(x) \ge 0$1 with direction $\rho_f(x) \ge 0$2 and threshold $\rho_f(x) \ge 0$3 defines a half-space

$\rho_f(x) \ge 0$4

and a concept with attribute set $\rho_f(x) \ge 0$5 is the intersection

$\rho_f(x) \ge 0$6

These regions induce a concept lattice via Formal Concept Analysis, with meet and join corresponding to geometric intersection and union (Xiong, 1 Mar 2026). This moves MLRH from subspaces to structured regions defined by multiple linear constraints.

Not all later work accepts global linear-subspace control as sufficient. The Cylindrical Representation Hypothesis (CRH) preserves linear decomposition but replaces global orthogonal subspaces with a sample-specific geometry: a central axis defined by a difference vector and a normal plane that controls steering sensitivity. Only certain angular regions of that plane, called sensitive sectors, robustly facilitate concept activation; others can suppress or delay it (Gao et al., 3 May 2026). CRH therefore retains multidimensional linear structure while disputing the predictability assumptions often attached to LRH and MLRH.

4. Empirical evidence

Empirical support for MLRH spans embeddings, hidden activations, lexical hierarchies, and steering behavior.

In the manifold account of feature representations, embeddings from text-embedding-large-3 and token activations from GPT2-small and Mistral 7B exhibited loops for cyclical features, curves for linear features, and more complex structures for dates and color names. The reported rank correlations between feature orderings and manifold positions were very high, with values above $\rho_f(x) \ge 0$7 for years; shortest-path distances along the recovered manifolds showed Pearson correlation $\rho_f(x) \ge 0$8 for log-scaled years and $\rho_f(x) \ge 0$9 for dates (Modell et al., 23 May 2025). The same study also reported that years in GPT2-small are represented logarithmically rather than linearly.

For categorical and hierarchical geometry, a WordNet-based study estimated representations for 900+ hierarchically related concepts using Gemma and LLaMA-3, with a detailed Gemma-2B setup retaining 593 noun and 364 verb synsets having at least 50 vocabulary tokens (Park et al., 2024). Linear Discriminant Analysis was used to estimate concept directions, and the resulting projections, cosine relations, and simplex visualizations were consistent with the predicted geometry of hierarchies and categories (Park et al., 2024).

The frame-based multi-token extension reported that over 99% of words are full-rank (frames) in Llama 3.1, Gemma 2, and Phi 3, matching the underlying Stiefel-manifold assumption (Valois et al., 2024). It also found that real word frames project positively onto associated concept frames, whereas random frames have near-zero projections (Valois et al., 2024).

Automatic manifold discovery through Supervised Multi-Dimensional Scaling (SMDS) provided another line of evidence. Applied primarily to temporal reasoning, SMDS analyzed over 60,000 manifold cases and found stable geometries across Llama, Qwen, and Gemma models, across 3B, 8B, and 70B scales, and across base and instruction-tuned variants (Tiblias et al., 1 Oct 2025). The paper reports that perturbing the identified manifold subspace degrades temporal reasoning performance, whereas perturbing random subspaces has minimal effect, indicating that these manifolds are functionally used rather than merely descriptive (Tiblias et al., 1 Oct 2025).

Evidence for cross-model multidimensional alignment also comes from sparse-feature studies. Sparse autoencoders often produced representations with stronger cross-modal alignment than dense features, and centering plus normalization consistently improved cross-model alignment. The same work reported a strong positive correlation between word frequency and alignment, with degradation proportional to $(\mathcal{Z}_f, d_f)$0, interpreting this as a finite-sample noise effect (Bangachev et al., 22 May 2026).

5. Steering, probing, and transfer

A major motivation for MLRH is operational: if concepts occupy stable low-dimensional structures, they should support measurement and control.

The original LRH formalization already connected output-space directions to linear probing and input-space directions to steering. Under the causal inner product, the same concept representation can be viewed as both an ideal probe and a steering direction, unifying measurement and intervention (Park et al., 2023). This remains a foundational template for later multidimensional work.

One practical extension is SAND—the Sum of Activation-base Normalized Difference estimator—which computes concept directions from activation differences rather than unembedding vectors. If normalized activation differences are modeled as i.i.d. samples from a von Mises-Fisher distribution, then the maximum-likelihood estimate of the concept direction is

$(\mathcal{Z}_f, d_f)$1

with the activation-space form

$(\mathcal{Z}_f, d_f)$2

This removes the dependence on single-token counterfactual pairs and extends LRH-style steering to complex, context-dependent concepts (Nguyen et al., 22 Feb 2025).

FRH operationalizes multidimensional control through Top-$(\mathcal{Z}_f, d_f)$3 Concept-Guided Decoding. At each generation step, among the top-$(\mathcal{Z}_f, d_f)$4 logit candidates, the decoder selects

$(\mathcal{Z}_f, d_f)$5

where $(\mathcal{Z}_f, d_f)$6 is the target concept frame and $(\mathcal{Z}_f, d_f)$7 is the candidate feature frame (Valois et al., 2024). On Llama 3.1, Gemma 2, and Phi 3, this exposed and steered gender and language biases, and the effect size was controllable through $(\mathcal{Z}_f, d_f)$8 (Valois et al., 2024).

The transferability framework pushes MLRH beyond single-model analysis. By fitting affine maps between hidden states of models trained on the same data and architecture family, steering vectors learned in smaller models were transferred to larger ones while preserving semantic effect (Bello et al., 31 May 2025). Evidence included low reconstruction error for mapped sparse feature matrices—reported as $(\mathcal{Z}_f, d_f)$9 versus much larger errors for random matrices—and high correspondence in behavioral steering metrics, including multiple-choice propensity correlations up to $$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$0. Example mean correlations included 0.833 for self-awareness-good-text-model and 0.968 for narcissism (Bello et al., 31 May 2025). In this interpretation, MLRH supports the use of small models as steering and interpretability sandboxes for larger systems.

6. Limits, critiques, and open questions

The strongest challenge to MLRH is not whether linear structure exists, but how far it can be pushed.

A theoretical analysis of feature capacity separates linear representation from linear accessibility. If $$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$1 features are $$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$2-sparse, classical compressed sensing with nonlinear decoding requires only

$$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$3

whereas linearly decoding those same features requires substantially more dimensions. The paper proves

$$\phi_f: \mathcal{Z}_f \rightarrow \mathbb{S}^{D-1},$$4

for the linear case (Garg et al., 11 Feb 2026). This establishes a quantitative gap between storing features linearly and recovering them with linear probes, and it implies that linear accessibility is a meaningfully stronger claim than linear representation alone.

CRH sharpens a different limitation: even if concepts are linearly decomposable, steering may remain intrinsically unstable at the sample level because overlapping concept contributions induce a local axis-plus-plane geometry rather than a globally orthogonal subspace. The paper states that the magnitude of the normal-plane component is predictive of steering intensity, but the phase within the plane is fundamentally unpredictable from observables (Gao et al., 3 May 2026). In that account, identical axis and plane configurations can still yield opposite steering outcomes depending on latent concept composition.

This suggests that the current literature does not present a single, settled meaning of MLRH. Some papers use it to denote multidimensional subspaces; others emphasize manifolds, frames, polytopes, half-space lattices, or universal basis features (Modell et al., 23 May 2025, Valois et al., 2024, Xiong, 1 Mar 2026, Bello et al., 31 May 2025). A plausible implication is that “MLRH” now functions as an umbrella term for a broader hypothesis class: semantic structure in LLMs is substantially linear, but often only after moving from one-dimensional directions to richer low-dimensional geometric objects.

Within that broader program, the main open questions are already explicit in the literature: how to learn feature metric spaces automatically, how to estimate higher-dimensional manifolds robustly in noisy settings, how to reconcile global linear structure with sample-specific steering instability, and how to characterize when cross-model affine transfer should hold (Modell et al., 23 May 2025, Gao et al., 3 May 2026, Bello et al., 31 May 2025).