Cross-Modal Entropy Collapse

• Cross-modal entropy collapse is defined as the contraction of multimodal state spaces into a stable low‐entropy manifold, reducing representational diversity while systems remain active.
• Mechanisms driving collapse include feedback amplification, modality imbalance in autoregressive processing, and entropy minimization that leads to overconfident yet ungrounded outputs.
• Mitigation strategies such as asymmetric model designs, modality-aware credit assignment, and entropy reservoirs aim to preserve grounding, semantic diversity, and exploratory capacity.

Cross-modal entropy collapse denotes a family of failure modes in multimodal and closely related adaptive systems in which uncertainty, representational diversity, or reliance on a source modality contracts toward a low-entropy regime. Recent work describes allied phenomena under different names: “evidence collapse” in reasoning VLMs, “late-stage modality collapse” in audio reasoning, an “Entropy Collapse Layer” in visual-token processing, “set collapse” in image-text retrieval, and collapse under test-time entropy minimization (Raghu et al., 5 Apr 2026, Xiao et al., 26 May 2026, Wang et al., 19 Feb 2026, Alomari et al., 26 Jun 2025, Chen et al., 27 Sep 2025). Across these settings, the system does not necessarily stop functioning; rather, it often becomes more confident while losing visual grounding, audio dependence, semantic diversity, or exploratory breadth. A broader theoretical literature formalizes collapse as convergence toward a stable low-entropy manifold rather than a literal zero-entropy state, although the extension of that formalism to multimodal systems is, in those works, an inferred application rather than a direct multimodal proof (Khanh et al., 13 Dec 2025, Chen, 16 Dec 2025).

1. Conceptual scope and formal meaning

The term is not yet standardized across the literature. In modality-specific work, the collapse is usually defined operationally: a reasoning VLM progressively stops attending to image evidence while maintaining or increasing answer confidence; an audio reasoning model increasingly abandons audio and falls back on language priors; a multimodal transformer compresses visual token representations into a low-rank redundant regime; or a set-based retrieval model loses diversity because multiple slots converge to nearly identical embeddings (Raghu et al., 5 Apr 2026, Xiao et al., 26 May 2026, Wang et al., 19 Feb 2026, Alomari et al., 26 Jun 2025).

A unifying interpretation is provided by the domain-agnostic theory of entropy collapse. There, the system state is represented as a distribution over states,

$$P_t \in \Delta(\mathcal{S}), \qquad P_{t+1}=\mathcal{F}(P_t;\alpha,\beta),$$

with entropy

$H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$

The central claim is that collapse arises when feedback amplification exceeds bounded novelty regeneration. The resulting regime is not defined by entropy going to zero, system death, or inactivity; it is defined by convergence toward a stable low-entropy manifold and by contraction of effective adaptive dimensionality (Khanh et al., 13 Dec 2025).

Applied to multimodal systems, this suggests that cross-modal entropy collapse is best understood as a contraction of the accessible multimodal state space. The contraction may occur in prediction space, attention dynamics, token representations, or embedding sets. What remains active is a narrower, more rigid, and often more overconfident process. That interpretation is directly stated in the general theory and is compatible with the modality-specific findings, though the fully unified multimodal formalization remains incomplete (Khanh et al., 13 Dec 2025, Chen, 16 Dec 2025).

2. Dynamical mechanisms

Several papers identify a structural asymmetry between a fixed source modality and an expanding autoregressive text process. In reasoning VLMs, the image is encoded once and stays fixed while the text context grows token by token. Attention therefore competes increasingly with the expanding text prefix. The observed consequence is that models can become more accurate while progressively losing visual grounding as reasoning unfolds, producing low-entropy but visually disengaged answers (Raghu et al., 5 Apr 2026).

The audio-reasoning literature identifies a related mechanism during RL post-training. Standard GRPO applies a uniform token average, effectively distributing policy gradient mass across all tokens regardless of whether they depend on audio. The claim is that this structurally favors the language prior, because many tokens are predictable from text alone while only a subset are genuinely modality-critical. As generation lengthens, the model becomes “confidently textual”: fluent, low-entropy, and increasingly ungrounded in the audio stream (Xiao et al., 26 May 2026).

Test-time adaptation work identifies a third route to collapse: entropy minimization itself can reward shortcuts. In ZeroSiam, plain entropy reduction is said to favor inflating logit norm and aligning all predictions toward a dominant class, yielding constant or near one-hot outputs that trivially minimize entropy without meaningful learning. The proposed objective,

$$\mathcal{L} = H(p^o) + \alpha \, D\!\left(p^o \,\|\, \mathrm{sg}[p^r]\right),$$

is designed to remove those trivial minima from the stable dynamics by introducing predictor–stop-gradient asymmetry before the classifier (Chen et al., 27 Sep 2025).

A broader geometric explanation is given by Entropy-Reservoir Bregman Projection. In that framework, self-referential learning repeatedly projects a model onto an empirical distribution sampled from its own current output, with target

$Y=(1-\lambda)\,P+\lambda R.$

Without reservoir coupling, finite-sample self-projection contracts support and drives exponential entropy decay; with a persistent entropy reservoir, the loop can maintain a non-trivial entropy floor (Chen, 16 Dec 2025). A plausible implication is that multimodal pseudo-labeling, self-captioning, or self-retrieval loops are vulnerable whenever one modality is repeatedly regenerated from a progressively narrowing distribution.

3. Empirical manifestations across modalities

In reasoning VLMs, evidence collapse is explicitly documented across three models and three datasets—MathVista, HallusionBench, and MMMU_Pro. Visual attention collapses in all 9 model×dataset cells, evidence-region mass also collapses in all 9 cells, total visual attention decay is positive in 94.3%–100% of samples per cell, evidence decay is positive in 83.5%–100%, and relative evidence loss ranges from 53.3% to 90.8%. The collapse typically occurs during reasoning rather than only at the final answer, and peak discrimination between correct and incorrect generations occurs during reasoning in 7/9 cells, with endpoint-only measurement missing up to 77.5% of the signal (Raghu et al., 5 Apr 2026).

In audio reasoning, late-stage modality collapse is described as decay of audio attention mass $a_t$ toward zero in later parts of chain-of-thought generation. The model may begin grounded in audio and then increasingly autoregress on a textual summary of its own earlier reasoning. MAPO is introduced to counter precisely this failure mode, and on MMAU, MMAR, MMSU, and MMAU-Pro it reports headline scores of 77.80, 70.90, 79.36, and 65.29, respectively, while improving long-horizon reasoning fidelity and multimodal instruction following (Xiao et al., 26 May 2026).

In multimodal token processing, EntropyPrune reports a distinct form of collapse internal to representation flow. Layer-wise matrix entropy of visual representations remains relatively high in early layers and then undergoes a sharp drop at a specific depth, called the Entropy Collapse Layer. In the reported LLaVA settings, the collapse occurs around layer 2. After that point, the visual stream is interpreted as entering a redundancy-dominated regime. This behavior is reported across eight datasets, for both query and key states, and across LLaVA-1.5-7B and LLaVA-NeXT-7B (Wang et al., 19 Feb 2026).

In image-text retrieval, the collapse appears as degeneration of set-based embeddings. Instead of preserving multiple semantic facets, embeddings within a set become nearly identical, destroying the purpose of multi-embedding representations. The paper attributes this either to sparse supervision, as in MIL-style matching, or to semantic homogenization induced by Smooth-Chamfer similarity, which is argued to be minimized when all embeddings in a set are identical (Alomari et al., 26 Jun 2025).

Cross-domain evidence also appears in test-time learning. ZeroSiam reports that the same anti-collapse mechanism stabilizes both vision adaptation and LLM reasoning. On ImageNet-C with label shifts it reaches 52.9% average accuracy, versus 44.4% for DeYO and 38.8% for SAR, and an ablation shows that removing stop-gradient causes severe collapse, with accuracy dropping from 51.6% to 20.5% on ResNet50-GN and from 64.1% to 38.5% on ViT-Base. In Llama3.1-8B-Instruct reasoning experiments, AIME24 improves from 3.33% baseline to 13.33% (Chen et al., 27 Sep 2025).

4. Measurement, diagnostics, and observables

The literature measures collapse through several complementary observables. In reasoning VLMs, text uncertainty is defined from top-$k$ logprobs by

$$p_i=\frac{e^{\ell_i}}{\sum_{j=1}^{k} e^{\ell_j}}, \qquad H_t=-\sum_{i=1}^{k} p_i \log_2 p_i,$$

with token-averaged entropy over a span $S$,

$\bar{H}_S=\frac{1}{|S|}\sum_{t\in S} H_t.$

The key empirical result is that full-response entropy is the most reliable text-only uncertainty signal under cross-dataset transfer, whereas answer-span entropy is unstable and can even invert direction. However, full-response entropy remains blind to whether the model is still visually grounded (Raghu et al., 5 Apr 2026).

To track grounding rather than only confidence, the same work defines total visual attention mass $V_t$, evidence-region attention mass $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$0, and the relevant visual attention ratio $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$1. Grounding layers are selected by evidence localization AUROC on a calibration set. This yields a diagnostic picture in which uncertainty and modality engagement are separable: low entropy can coexist with severe evidence disengagement (Raghu et al., 5 Apr 2026).

The audio-reasoning literature introduces a modality-sensitive entropy signal through cross-modal differential entropy,

$H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$2

When $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$3 is near zero, the token is largely predictable from language alone; when it is large, the token materially depends on audio. This converts entropy from a scalar uncertainty measure into a token-level estimate of modality dependence (Xiao et al., 26 May 2026).

For internal representation collapse, EntropyPrune measures matrix entropy of a trace-normalized covariance matrix,

$H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$4

equivalently the von Neumann entropy of the normalized covariance. Token-level entropy is computed exactly through a dual Gram matrix with the same non-zero spectrum, reducing complexity from $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$5 to $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$6 and yielding a claimed 64× theoretical speedup (Wang et al., 19 Feb 2026).

A broader diagnostic lesson emerges from RL-oriented entropy-collapse studies. HEAL aligns trajectory-level entropy dynamics rather than only scalar entropy magnitude, while RLVR work on SFT overtraining uses pre-RL entropy triage and early entropy monitoring to predict collapse. These studies are not multimodal, but they suggest that endpoint confidence alone is an insufficient statistic for collapse detection (Liu et al., 20 Apr 2026, Aphale et al., 16 Jun 2026).

5. Mitigation strategies

One line of mitigation changes the optimization geometry. ZeroSiam inserts an asymmetric Siamese design into a single forward pass:

$H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$7

with $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$8 and $H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s).$9. The target branch is stop-gradient detached, the predictor is learnable and initialized as identity, and in vision experiments only the normalization affine parameters plus the predictor are updated. The intention is not to weaken entropy minimization but to constrain it so that confidence gains must align with a stable reference branch (Chen et al., 27 Sep 2025).

A second line of work makes post-training explicitly modality-aware. MAPO uses the differential-entropy-derived modality relevance mask to reweight policy gradients toward modality-critical tokens and adds an auxiliary attention loss,

$$\mathcal{L} = H(p^o) + \alpha \, D\!\left(p^o \,\|\, \mathrm{sg}[p^r]\right),$$0

with a temporal factor emphasizing later tokens. The reported interpretation is that the mask improves token credit assignment, while the attention branch directly prevents late-stage collapse by keeping audio accessible in the decision-critical tail of reasoning (Xiao et al., 26 May 2026).

A third strategy exploits collapse for acceleration rather than only treating it as pathology. EntropyPrune identifies the Entropy Collapse Layer as the point after which visual tokens are substantially redundant and prunes low-entropy tokens beyond that depth. On LLaVA-1.5-7B, the method reports a 68.2% reduction in FLOPs while preserving 96.0% of the original performance; on high-resolution and video-based models it is reported to generalize effectively (Wang et al., 19 Feb 2026).

In retrieval, collapse mitigation centers on preserving semantic diversity. Maximal Pair Assignment Similarity uses Hungarian one-to-one matching between image and text embedding sets, while Global Discriminative Loss pushes each embedding away from the global reference embedding and Intra-Set Divergence Loss penalizes similarity among embeddings within the same set. The stated goal is to avoid both MIL-style sparse supervision and Smooth-Chamfer-style homogenization (Alomari et al., 26 Jun 2025).

At the theoretical level, ERBP generalizes many practical heuristics through entropy reservoirs. Real-data mixing, entropy bonuses, knowledge distillation, RLHF, retrieval-augmented generation, and external tool use are all interpreted as reservoir coupling choices that inject high-entropy external structure into a self-referential loop (Chen, 16 Dec 2025). This suggests that multimodal stabilization may often require external grounding, not merely stronger regularization within the closed loop.

6. Misconceptions, limitations, and open directions

A recurrent misconception is that collapse means zero entropy or total inactivity. The general theory explicitly rejects that definition: collapse is convergence toward a stable low-entropy manifold. Activity can remain high while effective adaptive dimensionality contracts (Khanh et al., 13 Dec 2025). This clarification matters for multimodal reasoning, where a model may produce long fluent traces and low-entropy answers while having already ceased to use the image or audio.

A second misconception is that more entropy, by itself, is always beneficial. The RLVR literature shows that simple entropy regularization can lead to entropy explosion, and label smoothing can raise entropy for the wrong reasons rather than restore useful diversity (Liu et al., 20 Apr 2026, Aphale et al., 16 Jun 2026). By analogy, multimodal systems require entropy signals that remain tied to grounding, semantic coverage, or exploratory utility.

A third misconception is that modality disengagement is uniformly harmful. Evidence-collapse results are explicitly task-conditional. In the entropy–vision interaction model,

$$\mathcal{L} = H(p^o) + \alpha \, D\!\left(p^o \,\|\, \mathrm{sg}[p^r]\right),$$1

the low-entropy, low-visual-engagement regime is hazardous on sustained visual-reference tasks such as MMMU_Pro, where $$\mathcal{L} = H(p^o) + \alpha \, D\!\left(p^o \,\|\, \mathrm{sg}[p^r]\right),$$2 is significantly negative, but can be benign on tasks like MathVista, where the interaction is near zero or positive. Consistently, adding vision with a single global linear rule is brittle: across 18 transfer pairs, entropy+vision improves AUC in only 4/18 cases and improves coverage-matched AURC in only 4/18 cases (Raghu et al., 5 Apr 2026).

The main limitations are also clear. The universal entropy-collapse theories are substrate-agnostic rather than modality-specific and do not provide formal multimodal models, cross-attention dynamics, or direct experiments on vision-language or audio-visual systems (Khanh et al., 13 Dec 2025, Chen, 16 Dec 2025). ZeroSiam notes that collapse recovery is possible in some cases but not guaranteed, succeeding in 4 out of 7 domains in the reported recovery experiments (Chen et al., 27 Sep 2025). MAPO is demonstrated only in audio, although its authors argue that the mechanism is modality-agnostic in principle (Xiao et al., 26 May 2026).

The emerging consensus is therefore narrower than a full theory but stronger than a collection of isolated anecdotes. Cross-modal entropy collapse is increasingly characterized as a structural contraction process in which confidence, reinforcement, or self-referential updating outpaces the mechanisms that preserve modality grounding, semantic diversity, or exploratory novelty. The principal design responses—task-aware monitoring, asymmetric objectives, modality-aware credit assignment, diversity-preserving matching, and explicit entropy reservoirs—are all attempts to prevent that contraction from becoming a stable attractor.