Compression Valleys in Complex Systems

• Compression valleys are phenomena characterized by sharp reductions in measurable quantities such as entropy, density, and pore pressure across diverse disciplines.
• They emerge through underlying dynamics that channel information, energy, or matter via bottlenecks and dominant modes, observable in ML, quantum systems, and geomechanics.
• Empirical and theoretical studies validate that these valleys optimize state space transitions and system performance in both engineered and natural settings.

Compression valleys are a class of phenomena observed across disparate disciplines—machine learning, statistical physics, materials science, combinatorics, and geoscience—characterized by regions of strong reduction or “compression” in some measurable quantity (such as entropy, density of states, hydraulic pressure, or representational dimension) within a system's evolution or structure. In many contexts, compression valleys emerge as a consequence of underlying dynamics that channel information, energy, or matter through bottlenecks or dominant modes, resulting in distinctive behavioral, geometric, or computational regimes.

1. Mathematical and Physical Definitions

Compression valleys acquire rigorous developmental meaning depending on the discipline:

• Machine Learning (LLMs): A compression valley is a sharp, layer-localized reduction in the representational entropy or effective rank of hidden-layer activations, typically caused by the emergence of a dominant singular vector in the representation matrix due to massive activations in tokens such as the beginning-of-sequence (bos) token (Queipo-de-Llano et al., 7 Oct 2025). Mathematically, the representation matrix $X$ exhibits a singular value spectrum dominated by $\sigma_1$ such that the entropy $H(X)$ (measured over normalized singular values) attains a minimum, producing a low-dimensional embedding despite high ambient dimension.
• Condensed Matter Physics: In quantum systems with valley degrees of freedom (graphene, bismuth), compression valleys involve the selective occupation, mixing, or even complete depletion (emptying) of electronic valleys due to external fields, disorder, or interactions (Wu et al., 2013, Zhu et al., 2016, Touchais et al., 2023, Wang et al., 2023). For instance, in high magnetic fields, Dirac valleys in bismuth can be completely emptied, leading to a sudden compression of the Fermi sea associated with those valleys (Zhu et al., 2016).
• Stochastic Processes and PDEs: Compression valleys describe spatial intervals between intermittent peaks of solutions (e.g., for the stochastic heat equation) where the process is highly suppressed—quantified as regions where the supremum of the solution decays rapidly, and the effective support of small values (the "valley") expands (Khoshnevisan et al., 2022). Specifically, for solution $u(t,x)$, the region with $u(t,x) \ll 1$ grows exponentially, while the local amplitude decays as $\exp(-\text{const}\cdot t^{1/3})$.
• Combinatorics: Compression valleys are formalized in weighted Dyck path models, imposing uniformity in the level (height) at which all valleys occur within given primitive factors, compressing the range of valley locations and enabling explicit enumerative connections to Motzkin paths, Schröder paths, and tree structures (Sun et al., 2021).
• Geomechanics: In layered sedimentary basins, compression valleys refer to the periodic depressions in hydraulic pore pressure history that develop between episodic fluid venting events, arising via tectonic compression and pressure diffusion; these valleys in overpressure signal major fluid expulsion cycles (Kearney et al., 2022).

2. Emergence and Mechanisms

The origins of compression valleys are rooted in the underlying system dynamics and interactions:

• LLMs: Compression valleys appear in mid-network layers when a token (typically the bos) develops an outsize norm, enforcing a low-rank structure on the representation matrix that also manifests as 'attention sinks', with heads across many layers focusing on the same token (Queipo-de-Llano et al., 7 Oct 2025). The phenomenon can be rigorously attributed to the spectral dominance of a single representation direction, with lower bounds on matrix entropy directly tied to the norm ratio $c$ and the alignment parameter $\alpha$:

$$p_1 \geq \frac{c+\alpha}{c+1}, \qquad H(X) \searrow \text{as } p_1 \nearrow 1$$

The effect is controlled by specific architectural and activation dynamics, notably the MLP-induced contribution to the bos token.

• Quantum Materials: In systems such as bilayer graphene or disordered chiral chains, compression valleys are realized through valley-dependent manipulations:
  • In BLG, compression and manipulation arise from band structure warping and generalized valley–orbit coupling, allowing for qubits and FETs based on distinct valley occupation/compression (Wu et al., 2013).
  • In disordered one-dimensional systems (chiral chains), the interplay between intra- and intervalley disorder, controlled by the correlation length $\xi$ of the disorder profile, leads to crossover from coupled (compressed) to decoupled (expanded) valley behavior. For small $\xi$, valleys are forced into correlated (compressed) states, with a suppressed density of states at zero energy (Touchais et al., 2023).
• Geodynamics: In sedimentary basins, imposed tectonic compression and subsequent hydraulic fracturing, combined with pressure recharge via vertical diffusion, create periodic intervals of low pore pressure—compression valleys in the time-series of overpressure—which regulate episodic venting (Kearney et al., 2022). The period between vents,

$\tau = \frac{\sigma_T/\Gamma_s}{1 + \nu/\gamma},$

is reduced by enhanced recharge due to pressure diffusion from thick mudstone layers (large $\nu/\gamma$).

• Stochastic PDEs: The valleys in solutions to the stochastic heat equation arise from probabilistic "dissipation" between rare high peaks (intermittency islands), leading to large regions of exponentially small amplitude—an intrinsic separation of spatial "action" (Khoshnevisan et al., 2022).
• Combinatorics (Weighted Dyck Paths): Compression valleys are structurally imposed by requiring valleys in each primitive factor to occur at a uniform level, compressing the combinatorial variety and resulting in a generating function amenable to specialization and bijection across models (Sun et al., 2021):

$$V_{a,B,y}(x) = \frac{1}{1 - y(x) - \frac{a(x)^2B(x)}{1-a(x)}}$$

3. Universal Structure and Phase Theories

Compression valleys often organize the state space or computations of complex systems into distinct phases:

• Mix-Compress-Refine Theory in LLMs: LLM computation can be structured into three regimes (Queipo-de-Llano et al., 7 Oct 2025):
  1. Mixing Phase: Early layers perform broad data-mixing (high row-wise entropy in attention matrices).
  2. Compression Valley Phase: Intermediate-to-late layers develop a dominant representation (bos token), driving singular value concentration and entropy troughs, with attention heads converging into sink-like behavior.
  3. Refinement Phase: Final layers redistribute representational power, sharpening attention and restoring per-token expressiveness for fine-grained prediction (especially for generative tasks).

Phase	Characteristic	Dominant Mechanism
Mixing	High entropy, broad	Distributed attention
Compression	Entropy dip/valley	Massive activations, low-rank representation
Refinement	Recovering entropy, localized	Re-normalized token norms, selective attention

This organization is essential for understanding task-dependent representational suitability (e.g., embeddings peak in the compression valley, whereas generation peaks after refinement).

4. Observational and Experimental Evidence

Multiple lines of evidence substantiate the centrality of compression valleys:

• Correlated Metrics: In LLMs, layer-wise measures of entropy, bos token norm, and attention sink rate are strongly correlated. Massive increases in bos token norm (spikes of $10^3$–$10^4$) induce sharp entropy dips and synchronized attention sink surges (Queipo-de-Llano et al., 7 Oct 2025).
• Intervention/Ablation: Targeted ablations to MLP outputs for the bos token eliminate both attention sinks and compression valleys, confirming the causal role of massive activations.
• Phase Diagrams and Density of States: In 1d chiral systems, the density of states near zero energy exhibits a transition from suppressed (‘pseudogapped’ or compressed) to divergent (‘Dyson peak’ or expanded) behavior as disorder-correlation length increases, tracking the degree of valley "compression" (Touchais et al., 2023).
• Fluid Release Cycles: In geological systems, periods between episodic venting events (measured via seafloor pockmarks or overpressure records) match analytical predictions derived from the compression-diffusion cycle, verifying the model that produces compression valleys in pressure histories (Kearney et al., 2022).

5. Broader Significance, Applications, and Theoretical Impact

Compression valleys play a critical role in optimizing and understanding the function of engineered and natural systems:

• Machine Learning: Compression valleys mark regimes of efficient information distillation, explaining why intermediate representations support linearly separable embeddings, while later layers are necessary for full generative capacity (Queipo-de-Llano et al., 7 Oct 2025). Their phase-dependent structure suggests new directions for dynamic architectures and early-exit schemes.
• Quantum/Valleytronics: Controlled compression and manipulation of valleys enable robust quantum and classical devices, including valley-based qubits, field-effect transistors, and valley filters, with potential for improved decoherence properties and logic density (Wu et al., 2013, Wang et al., 2023, Milovanovic et al., 2016).
• Combinatorics: Valley compression enables explicit enumerative generating functions and bijective mappings with other classical structures, advancing combinatorial understanding and algorithmic enumeration (Sun et al., 2021).
• Stochastic PDEs: Precise quantification of compression valleys in solutions advances intermittency theory and could impact simulation and control strategies in noisy, spatially extended systems (Khoshnevisan et al., 2022).
• Earth Sciences: The recognition and analytic description of compression valleys offers predictive power in hydrogeology, resource management, and paleoclimate interpretation via the reading of the periodic signatures in sedimentary records (Kearney et al., 2022).

6. Limitations and Open Problems

Despite extensive progress, the full implications and underlying mechanisms of compression valleys remain an active area of research:

• In LLMs, the precise onset conditions and universality of the mix-compress-refine transition across architectures, scales, and objective functions are active research questions.
• For quantum and disordered systems, the relationship between microscopic disorder features, symmetry, and the degree of valley compression or decoupling continues to motivate detailed analytical and numerical paper, particularly beyond the continuum limit.
• For stochastic PDEs, the optimality and sharpness of current bounds (e.g., the exponent $1/3$ in stretched exponential scaling) as well as their generalization to higher dimensions and different noise structures remain partially unresolved (Khoshnevisan et al., 2022).
• In geoscience, quantification of material properties governing the coupling ratio $\nu/\gamma$ for more complex, heterogeneous basins requires further observational and modeling work.

7. Cross-Disciplinary Connections

Compression valley phenomena share deep structural properties—regardless of system—that reflect a universal dynamic: systems organize information, energy, entropy, or population so that a transient or phase-localized bottleneck or low-dimensional mode dominates the evolution or observable properties. This universality is evident in the similar mathematical structures (entropy reductions, valley-emptying transitions, singular value dominance, generating function compressions) and in their centrality to both mechanistic understanding and engineering control.

Compression valleys, therefore, are not only a descriptive feature but also a powerful analytic lens for unifying phenomena across machine learning, condensed matter, combinatorics, and geoscience.