Representational Phase Structure

• Representational Phase Structure is defined by distinct regimes in internal geometry, controlled by parameters like network depth and weight scaling, influencing overlap and modularity.
• It distinguishes phases such as high-dimensional 'lazy' regimes with diffuse activations and low-dimensional 'rich' regimes with structured, task-specific subspaces.
• The analysis utilizes PCA, eigenvalue spectra, and algebraic filtrations to reveal transitions that impact transfer, interference, and specialization in learned representations.

Representational phase structure refers to the existence of sharply or qualitatively distinct regimes in the geometry and organization of internal representations within a system—typically a neural network, algebraic structure, or syntactic formalism—distinguished by control parameters such as dimensionality, network depth, architectural modularity, or combinatorial filtration degree. Across a range of paradigms, including continual learning, LLMs, deep representations, and algebraic or syntactic systems, these regimes (phases) correspond to distinct geometric, topological, or categorical invariants that sharply condition transfer, interference, specialization, and the efficacy of structural constraints.

1. Quantifying Representational Dimensionality and Phase Boundaries

Central to modern accounts of representational phase structure is the mathematical quantification of internal dimensionality. For a hidden-state response matrix $H\in \mathbb{R}^{N\times T}$ (where $N$ is the number of units and $T$ the number of examples), principal component analysis yields eigenvalues $\{\lambda_i\}$. Key measures include effective dimensionality $D_{\mathrm{eff}}$ and the participation ratio (PR), namely

$$D_\mathrm{eff} = \min \left\{ k: \frac{\sum_{i=1}^k \lambda_i}{\sum_{j=1}^N \lambda_j} \geq \eta \right\},\quad \mathrm{PR} = \frac{\left(\sum_{i=1}^N \lambda_i\right)^2}{\sum_{i=1}^N \lambda_i^2}$$

with $\eta$ typically 0.99 in empirical studies (Korte et al., 30 Apr 2026). These dimensionality proxies demarcate sharp transitions (for example, as a function of rescaling hyperparameter $\gamma$ in RNNs or normalized depth $\gamma$ in transformers) which structurally separate high-dimensional, unconstrained ("liquid" or "lazy") phases from low-dimensional, bottlenecked ("solid" or "rich") phases (Korte et al., 30 Apr 2026, Alpay et al., 16 Jan 2026).

In algebraic and categorical contexts, phases are stratified by canonical filtrations,

$$\mathcal{P}^{(0)} \supseteq \mathcal{P}^{(1)} \supseteq \cdots \supseteq \mathcal{P}^{(d)} \supseteq 0$$

with the defect degree serving as a formal analog of depth or order parameter, and the boundary layers $N$0 sharply mark phase boundaries (Gildea, 26 Jan 2026, Gildea, 26 Jan 2026).

2. Emergence of Distinct Regimes: Lazy/High-Dimensional vs. Rich/Low-Dimensional Phases

Empirical and theoretical work identifies two primary representational regimes:

• Lazy/High-Dimensional Phase: The system's internal representations maintain nearly full ambient dimensionality. In neural networks, this regime is approached for large weight scales—learning occurs almost entirely in output layers, task-specific subspaces overlap freely, and modularity or architectural structure is largely irrelevant (Korte et al., 30 Apr 2026). In stochastic or spectral terms, the covariance matrix spectrum retains a Marchenko–Pastur (MP) "bulk" with high effective rank, and all directions are equally accessible (Alpay et al., 16 Jan 2026).
• Rich/Low-Dimensional Phase: At lower scales of control parameters, the system exhibits compression onto a low-dimensional task submanifold; representations are highly structured, interference and specialization are sharply regulated, and architectural or combinatorial modularity exerts decisive influence (Korte et al., 30 Apr 2026). In deep transformers, this is accompanied by a collapse in PR dimension and the emergence of "semantic" spike eigenvalues ("bulk + outliers"), along with sharply bimodal order parameters for sparsity or object localization (Alpay et al., 16 Jan 2026).

The table below summarizes phase regime characteristics in RNNs and deep transformers:

Regime	RNN Geometry (Korte et al., 30 Apr 2026)	Transformer Geometry (Alpay et al., 16 Jan 2026)
High-dim / "Lazy"	$N$1; overlapping subspaces; modularity irrelevant	Bulk-like spectrum; high $N$2; low $N$3 (spread activations)
Low-dim / "Rich"	$N$4; subspace separation; modularity crucial	Bulk+spikes; $N$5 collapse; high $N$6 (localized, object-like)

These transitions are often abrupt, with critical points (e.g., $N$7 in deep transformers) marking discontinuous changes in both geometric and task-behavioral properties (Alpay et al., 16 Jan 2026).

3. Geometry and Specialization of Task-Specific Subspaces

In sequential and continual learning settings, the internal geometry of representations is determined by the arrangement of task-specific subspaces. This is quantified by principal angles between learned subspaces: for $N$8-dimensional task subspaces $N$9, the largest principal angle $T$0 measures their alignment or orthogonality. In the high-dimensional phase, even dissimilar tasks have $T$1, indicating weak separation. By contrast, the low-dimensional phase exhibits a graded geometry: highly similar tasks yield nearly overlapping subspaces ($T$2), moderately similar tasks have intermediate separation, and dissimilar tasks approach orthogonality ($T$3–$T$4), but only in explicitly modular architectures (Korte et al., 30 Apr 2026).

Adaptive geometry in the rich phase enforces resource competition, and modular or partitioned architectures unlock similarity-dependent allocation of subspaces. In single-module networks, this graded geometry fails to emerge, leading to interference and degraded retention.

4. Phase Structure in Deep LLMs: Curvature, Manifolds, and Objects

In LLMs, representational phase structure is revealed through curvature and manifold statistics.

• Curvature Measures: For a token sequence, representational "straightening" at layer $T$5 is assessed as the reduction in average local curvature,

$T$6

where $T$7 is the mean angle or corresponding Menger curvature along the neural trajectory (Hosseini et al., 29 Jan 2026).

• Multi-phase Dynamics: In continual-prediction tasks, deeper layers exhibit increased straightening and reduced effective dimension, strongly correlated with improved behavioral metrics (e.g., logit difference between valid and invalid next tokens, $T$8–$T$9) (Hosseini et al., 29 Jan 2026). In contrast, in few-shot or structured tasks, straightening is phase-specific: only template or formatting tokens show marked trajectory collinearity, while "core" content often lacks such geometric regularity, indicating heterogeneous, phase-dependent representational strategies within a single feedforward sweep.
• Topological Transitions and "Object Permanence": Layerwise covariance spectra in deep transformers reveal topological phase transitions: at depth fraction $\{\lambda_i\}$0, a sharp collapse in PR and entropy-effective rank is accompanied by the discrete emergence of "Transient Class Objects" (TCOs)—low-dimensional, locally stable basins corresponding to discrete semantic or logical classes (Alpay et al., 16 Jan 2026). The low-entropy "solid" phase exhibits "spiked" covariance spectra and localized activation patterns, interpreted as a dynamical renormalization group flow mapping diffuse input statistics onto structurally stable semantic modes.

5. Algebraic and Combinatorial Perspectives

Structural parallels in algebraic systems arise in the formalism of Algebraic Phase Theory (APT) (Gildea, 26 Jan 2026, Gildea, 26 Jan 2026). Here, representational phase structure consists of the pair $\{\lambda_i\}$1—the functorial filtered representation category and its intrinsic boundary stratification (layers $\{\lambda_i\}$2). APT enforces a strict dichotomy:

• Rigid regime: No defect boundary ($\{\lambda_i\}$3); classic duality recovers the phase uniquely from its representations.
• Non-rigid regime: Nontrivial boundary layer ($\{\lambda_i\}$4) is intrinsic and detectable; reconstruction up to phase equivalence requires explicit boundary data.

Canonical finite generation, universal obstruction objects, rigidity-obstruction equivalence, and finite-depth boundary detectability constitute the minimal structural toolkit for phase theory. In the categorical setting, all equivalence notions (strong, weak, Morita-type) collapse in the rigid regime, and structural boundaries are categorical invariants.

6. Phase Structure in Syntactic and Combinatorial Systems

Approaches based on hypermagmas and colored operads model syntactic "phase" structure as a system of local generation and filtering (bud system) (Marcolli et al., 8 Jul 2025). Binary syntactic trees with head functions are formalized as hypermagmas, and phase boundaries are enforced by colored operad buds. Phase principles such as the Extended Projection Principle (EPP), Phase Impenetrability Condition (PIC), and Empty Category Principle (ECP) are realized as generator/coloring constraints in the operad, determining the permissible loci of movement and composition. This algebraic-combinatorial perspective directly links phase structure to feasible derivations, compositionality, and movement constraints.

A table outlines salient features:

Formalism	Phase Indicator	Boundary Mechanism
APT (Gildea, 26 Jan 2026)	Defect filtration $\{\lambda_i\}$5	Boundary layers $\{\lambda_i\}$6
Hypermagma/operad (Marcolli et al., 8 Jul 2025)	Bud color/operad generator	Generator inclusion/exclusion

7. Broader Principles and Implications

Phase structure serves as an organizing meta-principle for understanding representational geometry, specialization, and transfer in learning systems:

• Dimensionality and Rigidity: Dimensionality acts as a control parameter; only under constraints does modularity, boundary stratification, or explicit structure become functionally or categorically relevant (Korte et al., 30 Apr 2026, Gildea, 26 Jan 2026).
• Adaptive Geometry: Systems in nontrivial phases dynamically trade off overlap and orthogonality, specializing or separating representations according to task statistics or algebraic invariants (Korte et al., 30 Apr 2026).
• Dynamical Analogs: The analogy with phase transitions in statistical physics (liquid/solid, bulk/spikes) or topological transitions extends to learning dynamics and model scaling (Alpay et al., 16 Jan 2026).
• Causality and Intervention: Moving from geometric correlation to mechanistic causality—via phase-sensitive architectural, task, or pretraining interventions—remains an open avenue for robust characterization and control of computational properties (Hosseini et al., 29 Jan 2026).
• Universality: Any functorial, finite-depth reconstruction framework with detectable boundaries admits an intrinsic phase structure; representational phase structure constitutes a universal invariant classifying regimes of transfer, interference, and specialization (Gildea, 26 Jan 2026).

Taken together, these perspectives position representational phase structure as a foundational theoretical construct underpinning the geometry, dynamics, and organization of learned or generated objects across neural, algebraic, and syntactic domains.