Middle-Layer Hidden States in Transformer Models

• Middle-layer hidden states are intermediate activations in deep neural networks that capture the shift from low-level token integration to high-level semantic abstractions.
• Empirical studies reveal that these layers exhibit peak information richness, making them optimal for probing, transfer learning, and anomaly detection.
• The transition at middle layers is critical for model diagnostics, enhancing uncertainty quantification and mechanistic interpretability.

Middle-layer hidden states are layer-wise activations in the central strata of deep neural architectures—most prominently large transformer-based models such as LLMs, vision-LLMs, and multimodal LLMs—which serve as intermediates between initial input representations and the final output. Empirical and theoretical analyses now converge on the finding that these middle-layer representations encode a distinctive transition in network function: from low-level context integration and token-wise feature extraction, toward high-level, semantically meaningful, and task-relevant abstractions. Recent work reveals that, across model scales and architectures, middle-layer states exhibit peak information geometric richness and are often the most effective source for downstream probing, transfer learning, anomaly detection, uncertainty quantification, and mechanistic interpretability.

1. Formal Definitions and Dynamics of Middle-Layer Hidden States

A transformer with $L$ layers processes an input sequence into a series of hidden states $\{h_\ell\}_{\ell=0}^{L}$. The intermediate, or "middle," layers refer to indices $\ell$ in $[L/3, 2L/3]$, typically $\ell\approx L/2$. Concretely, for a decoder-only transformer,

$$h_\ell = h_{\ell-1} + \text{MHSA}^\ell(h_{\ell-1}) + \text{FFN}^\ell(h_{\ell-1} + \text{MHSA}^\ell(h_{\ell-1}))$$

with layerwise update

$$\Delta h_\ell = h_\ell - h_{\ell-1} = \text{MHSA}^\ell(h_{\ell-1}) + \text{FFN}^\ell(h_{\ell-1} + \text{MHSA}^\ell(h_{\ell-1}))$$

Middle-layer updates, as quantified by the ICR Score,

$$\mathrm{ICR}_\ell = \frac{\|\Delta h_\ell\|}{\|h_\ell\|}$$

represent the proportion of new information injected at each stage, with $\mathrm{ICR}_\ell\approx 1$ indicating maximal functional contribution. Distinctive rises in this measure are repeatedly observed around $\ell\approx L/2$, signifying a shift from attention-driven redistribution to dominant feed-forward knowledge retrieval (Zhang et al., 22 Jul 2025). Similar definitions and probe structures are used in multi-modal and recurrent architectures, adapting for hidden state matrices and recurrent updates.

2. Empirical Evidence for Functional Specialization at Middle Layers

Across model types and tasks, middle-layer representations consistently display unique functional signatures:

• Compression and Information Bottleneck: Intermediate layers exhibit a "compression valley"—a dip in matrix-based entropy and token-level information complexity, signaling abstraction and denoising prior to the output-specialized upper layers. Peak quality in representation (as assessed by classification, clustering, alignment, and invariance metrics) is seen at 40–70% depth in both transformers and state-space models (Skean et al., 4 Feb 2025).
• Unified Geometric and Information-Theoretic Frameworks: Effective rank, prompt/dataset entropy, curvature, and invariance measures (InfoNCE, LiDAR, DiME) all peak or reach optima in the mid-layers, with downstream accuracy on text embedding tasks improved by up to 16 percentage points relative to final-layer features (Skean et al., 4 Feb 2025).
• Spectral Phase Transition: Large models display a sharp decrease in effective dimension and participation ratio around a critical normalized depth $\{h_\ell\}_{\ell=0}^{L}$0, with localized activation patterns and emergent low-rank "concept basins" (object-like structures) encoded in the hidden-state covariance (Alpay et al., 16 Jan 2026).

3. Middle-Layer Hidden States in Specialized Tasks

Hallucination and Uncertainty Detection: Middle layers are disproportionately informative for hallucination and factuality probes in LLMs, as the ICR Probe achieves peak AUROC at intermediate depths, and ablations removing middle layers degrade detection accuracy more than removing early or late layers. The magnitude and trajectory of $\{h_\ell\}_{\ell=0}^{L}$1 serve as reliable task diagnostics, with middle-layer divergences marking content errors on key tokens (Zhang et al., 22 Jul 2025).

Transfer Learning and Adaptation: Probing for within-class vs. between-class variability reveals a characteristic U-shape across layers; middle layers possess the lowest variability scores and thus the most linearly-separable, task-specialized structure, supporting computationally efficient transfer by tuning or adapting only these layers (Xie et al., 2022).

Uncertainty Quantification (UQ): Hidden-state regression probes for factual correctness generalize best when features are drawn from middle layers—demonstrating robustness under distribution shift not seen in final-layer specialists (Stacey et al., 13 Apr 2026).

Anomaly Detection (VAD): In multimodal models, metrics such as KL-divergence, local discriminant ratio, and feature entropy all reach maxima in the mid-layers, making these features most sensitive for anomaly segmentation and yielding top-frame AUCs (Cai et al., 23 Jul 2025).

Task/Domain	Best Performing Layer(s)	Key Metric (Example)
Hallucination DET	$\{h_\ell\}_{\ell=0}^{L}$2	AUROC=0.77
Embedding/Probe	40–70% network depth	+3–16 pp acc. gain
Transfer Adapt	Middle ($\{h_\ell\}_{\ell=0}^{L}$3 by $\{h_\ell\}_{\ell=0}^{L}$4)	Matches full FT
Anomaly VAD	Middle (e.g. $\{h_\ell\}_{\ell=0}^{L}$5 of 32)	AUC=86.72%

4. Mechanistic Interpretability: Information Flow and Inductive Bias

Middle layers demarcate the boundary between distributed context gathering and the consolidation/retrieval of parametric knowledge:

• Attention-Consolidation Switch: In decoder transformers, manipulations up to the critical middle layer (e.g., noise injection at $\{h_\ell\}_{\ell=0}^{L}$6) disrupt performance, but equivalent top-layer interventions are ineffectual, indicating that aggregation of contextual information completes at the middle stage, and later layers internally process it (Ben-Artzy et al., 2024).
• Phase Transition and Object Formation: Geometric and statistical physics analyses identify a first-order-like phase transition in large transformers at $\{h_\ell\}_{\ell=0}^{L}$7, with activation patterns crystallizing into Transient Class Objects (TCOs)—discrete, reusable basins that encode task-relevant information (Alpay et al., 16 Jan 2026).
• Symbolic Computation and Discrete State Tracking: Middle layers can be probed for implicit discrete state representations (IDSRs) during symbolic tasks such as multi-digit addition, with linear decoding fidelity peaking at early/mid depths, nonlinear transforms emerging near layer 50–65, and fidelity degrading in output-specialized upper layers (Chen et al., 2024).
• Recurrent and Bi-linear Architectures: In RNNs, active bi-linear interactions and sufficient statistics for belief filtering in partially observed environments are most readily encoded in the mid-layers, supporting both theoretical and empirical state-tracking and control (Lambrechts et al., 2022, Ebrahimi et al., 27 May 2025).

5. Probing and Practical Methodologies

Several analysis frameworks and probing strategies are developed specifically to exploit or interpret middle-layer states:

• Layer-sweeping: Always scan activations across all depths; mid-layer selection based on loss curves, entropy minima, or class-separability scores (e.g., $\{h_\ell\}_{\ell=0}^{L}$8) reliably outperforms naive final-layer usage (Skean et al., 4 Feb 2025, Xie et al., 2022).
• Lightweight statistical and geometric probes: Logistic regression, silhouette scores, linear discriminant ratios, and residual norm-ratios provide accessible metrics for layer utility in multiple benchmarks (Cai et al., 23 Jul 2025, Zhang et al., 22 Jul 2025).
• Task vector and attention head analysis: For in-context learning, geometric proxies (separability and alignment) uncover a two-stage mechanism—separability emerging in lower/mid layers, alignment crystallizing in mid/upper layers, with key head types (induction heads) driving phase transitions (Yang et al., 24 May 2025).
• Jump suppression regularization: The JREG regularizer penalizes abrupt final-layer representational “jumps,” leading to redistribution of information evolution throughout the network and improved downstream generalization, especially by strengthening capacity utilization in the middle layers (Shibata et al., 26 Jan 2026).

6. Limitations and Open Questions

Not all phenomena are beneficial: loss of fidelity for symbolic representations in mid to late layers is prominent (e.g., arithmetic tasks, where accuracy drops as layers deepen) (Chen et al., 2024). The generality of results across architectures, domains, and compositional tasks remains under investigation, particularly in the context of data diversity and adversarial distribution shifts (Stacey et al., 13 Apr 2026). Further, while probes offer interpretability, their selectivity and semantic anchoring must be validated against robust baselines and appropriately annotated to discern superficial statistical cues from true semantic or persona-related features (Jackson et al., 11 Apr 2026).

7. Interpretive and Theoretical Synthesis

Middle-layer hidden states are not mere architectural artifacts but loci of fundamental network-phase transitions and representational "bottlenecks." They encode maximal abstraction, class and semantic separation, and parametric knowledge injection. Their analysis offers both practical diagnostic value for task selection and principled insight into learning dynamics and proper model regularization. Probing and leveraging these intermediate activations is now central to robust transfer learning, uncertainty quantification, and mechanistically grounded model auditing across modalities and application domains (Skean et al., 4 Feb 2025, Zhang et al., 22 Jul 2025, Alpay et al., 16 Jan 2026, Shibata et al., 26 Jan 2026).