Layer-wise Semantic Dynamics (LSD)

• Layer-wise Semantic Dynamics (LSD) is the study of how semantic signals evolve and transform across various neural processing stages, including recurrent steps and transformer blocks.
• It utilizes metrics like attention pooling, state-space models, and geometric analyses to quantify semantic accumulation, redistribution, and degradation.
• The framework is applied in language models, multimodal transformers, and systems-level control, providing actionable strategies for model improvement and hallucination detection.

Layer-wise Semantic Dynamics (LSD) denotes the study of how semantic evidence changes across intermediate representational stages of a neural model. In the literature assembled under this broad label, the relevant “layers” are not always literal stacked depth layers: they may be recurrent processing steps, progressively longer prefixes, transformer blocks, denoising stages, dynamic semantic subspaces, or ordered visual compositing layers. This suggests that LSD is best understood as a family of analyses and control strategies concerned with when semantics becomes available, how it is transformed, and how it is preserved or degraded across computation (Gupta et al., 2018, Oh et al., 2022, Jiang et al., 12 May 2026, Li et al., 3 Feb 2026).

1. Conceptual scope and terminological variants

The most important terminological point is that “layer-wise” is not used uniformly. In the recurrent setting of LISA, “layer-wise” refers primarily to the temporal accumulation of meaning as longer and longer prefixes are processed, so the evolving recurrent state functions as the effective representational layer (Gupta et al., 2018). In transformer sentence embedding work, by contrast, layer-wise analysis usually means probing stacked hidden layers directly, often to determine whether semantics is concentrated in shallow, intermediate, or late depth (Oh et al., 2022, Pan, 20 Feb 2025). In dynamic semantic-subspace methods such as DySem, the emphasis shifts again: semantics is treated as distributed across layers but only partially exposed through a sample-specific subset of internal dimensions (Zheng et al., 28 May 2026).

Other strands generalize LSD beyond conventional representation analysis. State-space formulations treat depth itself as a dynamical axis and layer outputs as states of a continuous process, with latent memory summarizing prior layers (Liu et al., 12 Feb 2025). LRD formalizes layer-wise change through three measurement families—Frenet, Neighborhood Retention Score (NRS), and Graph Filtration Mutual Information (GFMI)—so that depth is analyzed through global subspace motion, local nearest-neighbor retention, and alignment with the final layer (Jiang et al., 12 May 2026). Systems work such as LLMCache does not directly study semantic representations, but it uses layer-wise activation reuse across semantically similar inputs as an indirect probe of where representations are stable enough to be memoized (Bansal, 18 Dec 2025).

A further expansion appears in generative and multimodal models. In diffusion transformers, “layer-wise” may refer to the weighting of hidden states from different LLM layers for text conditioning, or to ordered visual layers that determine regional layout and occlusion (Li et al., 3 Feb 2026, Chen et al., 6 Mar 2026). The common thread is not a single architecture or metric, but a shared concern with semantic evolution across representational stages.

2. Recurrent accumulation and prefix-wise semantics

An early and explicit precursor to LSD is LISA, “Layer-wIse Semantic Accumulation,” introduced for relation classification with recurrent neural networks (Gupta et al., 2018). Its central question is process-level rather than endpoint-level: instead of asking only which token mattered, it asks when the representation became semantically sufficient for the target class. Given a sentence $S=[w_1,\dots,w_n]$, LISA constructs all prefixes

$S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$

and tracks the class probability

$$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$

where $M$ is the trained C-BRNN classifier and $h_k^{bi}$ is the combined hidden state after processing prefix $S_{\le k}$. The underlying recurrent architecture is the Connectionist Bi-directional RNN of Vu et al. (2016a), with forward, backward, and combined recurrences.

This procedure turns an opaque final judgment into a temporal score trajectory. The trajectory may remain low while evidence is insufficient, increase gradually as context accumulates, jump sharply when a decisive phrase appears, or even decrease when later words introduce ambiguity. The paper’s canonical example is the sentence “<e1> demolition </e1> was the cause of <e2> terror </e2>”, where the probability for cause-effect(e1, e2) stays modest until the phrase “cause of” appears, then rises sharply from $0.39$ to $0.77$, reaches $0.98$ after “<e2>”, and reaches $1.00$ by the end (Gupta et al., 2018). This is the paper’s clearest demonstration of semantic accumulation as a dynamic process.

LISA is paired with Example2pattern, a threshold-based trigger-pattern extractor. The method computes $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$0 for each prefix, finds the point where the probability exceeds a threshold $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$1, and returns the last $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$2-gram in that prefix. The authors manually inspect several thresholds and report that $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$3 yields the most salient and interpretable patterns. The method is explicitly not gradient-based saliency; it is a trigger-pattern extraction heuristic based on score-trajectory crossing. On SemEval-2010 Task 8 and TAC KBP Slot Filling, the extracted patterns include “cause of <e2>”, “caused by the”, “dropped into the”, “born in <e2>”, and “</e1> married <e2>” (Gupta et al., 2018).

The paper’s major conceptual limitation is also its major terminological peculiarity. Its “layer-wise” analysis is not chiefly about stacked LSTM or GRU depth; it is about recurrent time steps and prefix sensitivity. This suggests that LSD, in one important lineage, began as a study of semantic sufficiency over sequential processing rather than as a pure depth-wise decomposition of internal layers.

3. Transformer depth, sentence embeddings, and dynamic semantic readout

In transformer-based sentence representation, a recurring LSD theme is that semantics is distributed across depth and is often not best recovered from the last layer alone. “Don’t Judge a LLM by Its Last Layer” makes this point directly: on STS-B, using $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$4 from each BERT$S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$5 layer, the 4th layer achieves $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$6, around 20 points higher than the last layer (Oh et al., 2022). The paper proposes Layer-wise Attention Pooling, which collects $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$7 and $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$8 from all layers, computes an attention-based aggregation $S_{\le k}=[w_1,\dots,w_k], \quad k=1,\dots,n,$9, concatenates it with the last-layer $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$0 representation,

$$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$1

and maps the result through an MLP. Across unsupervised, supervised, and hard-negative contrastive learning, the method improves over fixed pooling baselines such as $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$2, $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$3, and $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$4. The strongest ablation on BERT$$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$5 for STS-B is LayerAttPooler($$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$6 attention) + $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$7 concat with $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$8, compared with $$P(R \mid S_{\le k}, M)=\operatorname{softmax}(W_{hy}\cdot h_k^{bi}+b_y),$$9 for $M$0 alone (Oh et al., 2022). The paper also stresses an interpretive caution: attention weights over layers are not guaranteed to equal semantic importance.

RWKV-based zero-shot sentence embeddings provide a contrasting depth profile. In “Exploring RWKV for Sentence Embeddings,” mean pooling over hidden states from layers $M$1 on MRPC yields a monotonic decline in validation Spearman correlation: $M$2 (Pan, 20 Feb 2025). Under this extraction regime, the earliest tested layer is best, and semantic utility declines with depth. The paper interprets this as evidence that earlier layers capture more generalizable semantic features, while deeper layers may be more specialized for the language-modeling objective. It also notes that the result is specific to zero-shot mean pooling: the conclusion is robust only for mean-pooled hidden states, not necessarily for better pooling or task-specific fine-tuning. A simple GloVe averaging baseline reaches $M$3 on validation, outperforming every tested RWKV layer (Pan, 20 Feb 2025).

DySem pushes the argument further by rejecting fixed full-dimensional last-layer comparison as the default for semantic textual similarity (Zheng et al., 28 May 2026). For decoder-only LLMs, it compares three internal representations extracted from the last token under a prompt: final hidden state $M$4, cumulative attention output $M$5, and cumulative FFN output $M$6. The main component-wise result is that $M$7 is best in 61 out of 70 model-task comparisons, $M$8 is best in 9 out of 70, and $M$9 is best in 0 out of 70. DySem then uses multilingual consensus to define a text-specific semantic dimension set $h_k^{bi}$0, constructs a pair-specific union

$h_k^{bi}$1

and computes similarity only in that restricted subspace. Main experiments use $h_k^{bi}$2 selected dimensions. This suggests that LSD is not only a question of which layer to pool, but also which internal components and which sample-specific dimensions should count as semantic for a given comparison.

4. Geometric and dynamical formalisms

A substantial strand of LSD formalizes depth as a trajectory. “From Layers to States” treats outputs of deep layers as states of a continuous process and proposes Selective State Space Model Layer Aggregation (S6LA) (Liu et al., 12 Feb 2025). Starting from the continuous state-space model

$h_k^{bi}$3

the paper derives a discrete recurrence

$h_k^{bi}$4

and interprets $h_k^{bi}$5 as depth rather than physical time. In the S6LA view, a layer representation is not an isolated block output but part of a coupled process in which explicit features $h_k^{bi}$6 and latent memory $h_k^{bi}$7 evolve together. This suggests a version of LSD in which semantics is propagated, filtered, and selectively updated through depth-indexed memory rather than merely passed through residual connections.

“Bridging the Dimensional Chasm” offers a different geometric account by tracking a layer-wise token correlator

$h_k^{bi}$8

and deriving the approximate conservation law

$h_k^{bi}$9

under a projection model of layer transitions (Song et al., 28 Mar 2025). The reported empirical pattern is an expansion–contraction trajectory: token representations first diffuse into a higher-dimensional “working space” and later compress onto lower-dimensional final manifolds. The paper summarizes this as $S_{\le k}$0 and $S_{\le k}$1. The authors interpret the late compression as movement toward a semantic space, though this interpretation remains indirect because the paper uses geometry and correlation rather than explicit semantic supervision.

LRD, or Layer-wise Representation Dynamics, supplies a broader measurement suite for transformer depth (Jiang et al., 12 May 2026). Its three families are: Frenet, based on Grassmann speed $S_{\le k}$2 and Menger curvature $S_{\le k}$3; NRS, based on adjacent-layer Jaccard retention of local $S_{\le k}$4-nearest-neighbor sets; and GFMI, based on mutual information between connected-component partitions in per-layer graph filtrations aligned to the final layer. Across 31 models and 30 MTEB tasks, the strongest model-selection signal is end-to-end subspace displacement $S_{\le k}$5, while GFMI is the only measurement-guided pruning rule that beats Random at the 15% and 20% budgets (Jiang et al., 12 May 2026). Here LSD becomes a quantitative study of global subspace motion, local neighborhood preservation, and multi-scale final-layer alignment.

A distinct geometric formalization appears in “The Geometry of Truth,” which uses the term Layer-wise Semantic Dynamics directly for hallucination detection (Mir, 6 Oct 2025). For each layer, token states are pooled by masked average,

$S_{\le k}$6

projected into a shared semantic space with a truth encoder, and scored by alignment

$S_{\le k}$7

The framework then measures alignment, alignment gain, velocity, acceleration, and convergence layer. This yields an explicitly trajectory-based notion of factual consistency: factual generations preserve stable truth alignment, while hallucinations exhibit semantic drift across depth (Mir, 6 Oct 2025).

5. Generative, multimodal, and systems-level extensions

In diffusion transformers, LSD appears as adaptive conditioning over model depth and denoising time. “Semantic Routing” studies multi-layer LLM hidden states as text conditioning for DiT-based text-to-image models and introduces a normalized convex fusion

$S_{\le k}$8

Three routing variants are evaluated: time-wise, depth-wise, and joint. The central result is that Depth-wise Semantic Routing is the strongest conditioning strategy, while purely time-wise routing can degrade visual generation fidelity under classifier-free guidance because nominal timestep fails to track the effective SNR (Li et al., 3 Feb 2026). On GenAI-Bench Counting, S2 Depth improves from $S_{\le k}$9 for the penultimate-layer baseline to $0.39$0, a $0.39$1 gain. This indicates that, in this setting, semantic hierarchy across LLM layers aligns more robustly with DiT block depth than with nominal diffusion time.

LayerBind extends this logic from text conditioning to regional and occlusion control in DiTs (Chen et al., 6 Mar 2026). Its premise is that spatial layout and occlusion are established at a very early denoising stage, so rearranging the early latent structure is sufficient to modify the final output. The method is split into Layer-wise Instance Initialization and Layer-wise Semantic Nursing. In the initialization phase, each region becomes a branch,

$0.39$2

and is updated by contextual attention against its own regional prompt and shared background context. At selected text-dominant transformer blocks, Hard Binding updates the branch from regional text alone. At the early fusion step $0.39$3, branches are merged into the global latent in explicit layer order, optionally by alpha blending. In the nursing phase, standard global denoising is supplemented by a sequential layered attention path whose updates are composited by

$0.39$4

On BindBench ablation, removing both Hard Binding and Layer-wise Semantic Nursing yields VQAScore $0.39$5, while enabling both yields $0.39$6 (Chen et al., 6 Mar 2026). The paper therefore treats semantics as something formed early, then nursed and composited later.

LLMCache is not framed as representation learning, but it is highly relevant as an indirect systems-level probe (Bansal, 18 Dec 2025). Each transformer layer $0.39$7 has a cache bank $0.39$8, and hidden state reuse is triggered when an input fingerprint $0.39$9 is sufficiently similar to a stored fingerprint: $0.77$0 The strongest LSD-relevant result is that lower and mid transformer layers exhibit higher cache hit rates, up to 92%, while upper layers are more sensitive to semantic variation. Overall the system reports up to $0.77$1 speedup in inference time with $0.77$2 accuracy degradation (Bansal, 18 Dec 2025). This suggests that activation reuse can function as an operational measure of semantic stability across depth.

6. Uncertainty, hallucination, and unresolved methodological questions

Not all LSD results point in the same direction. A major negative result appears in “On the Effect of Uncertainty on Layer-wise Inference Dynamics,” which studies layer-wise probability trajectories of final answer tokens using the Tuned Lens, following the Logit Lens/Tuned Lens distinction introduced by Belrose et al. (Kim et al., 9 Jul 2025). On 11 datasets and 5 models, the paper finds that correct and incorrect multiple-choice predictions show largely aligned trajectories: both display abrupt increases in confidence at similar layers, and the layer where the answer stabilizes is similar across certainty conditions. Pearson correlations between answer incorrectness and prediction depth are generally weak; 80% are below $0.77$3, although 97% are positive (Kim et al., 9 Jul 2025). This challenges the idea that uncertainty necessarily induces a qualitatively different semantic path through the network.

That negative result contrasts sharply with the hallucination-detection framework of “The Geometry of Truth” (Mir, 6 Oct 2025). There, truth-aligned trajectories do separate factual and hallucinated generations strongly: the reported supervised detector reaches F1 $0.77$4, AUROC $0.77$5, and clustering accuracy $0.77$6, while the strongest alignment-based metrics show very large effect sizes. The contrast is methodological. The uncertainty paper analyzes candidate-answer token trajectories in a Tuned Lens output space, whereas the hallucination paper uses learned projection heads, a truth encoder, and trajectory features such as final alignment, mean alignment, alignment gain, and convergence layer. This suggests that whether LSD reveals strong separation depends heavily on the semantic object being tracked and on the projection regime.

Several recurrent caveats follow from the literature. First, “layer-wise” is heterogeneous: it may denote recurrent prefixes, stacked hidden layers, accumulated attention components, graph-filtration alignment, denoising stages, or ordered instance layers (Gupta et al., 2018, Zheng et al., 28 May 2026, Chen et al., 6 Mar 2026). Second, results are often extraction-protocol-specific. RWKV’s shallow-layer advantage is established only for zero-shot mean-pooled hidden states from layers $0.77$7 on MRPC (Pan, 20 Feb 2025). DySem is training-free, but it incurs $0.77$8 inference overhead because each text requires $0.77$9 translations and $0.98$0 forward passes, and it is validated mainly on STS rather than broad downstream transfer (Zheng et al., 28 May 2026). Third, attention-based aggregation does not automatically solve interpretation: “Don’t Judge a LLM by Its Last Layer” explicitly warns that learned attention weights across layers should not be read as a faithful causal decomposition of semantic content (Oh et al., 2022).

A plausible implication is that LSD is most stable as a research program when it is treated as a comparative framework rather than a single doctrine. Across the surveyed work, semantics can accumulate, redistribute, compress, stabilize, drift, or become task-specialized depending on architecture, pooling rule, supervision, and downstream objective. The durable contribution of LSD is therefore not one universal layer law, but a set of methods for asking where semantic signal resides, when it becomes operational, and how its trajectory can be measured or controlled.