--- title: Cross-Modal Entropy Collapse url: https://www.emergentmind.com/topics/cross-modal-entropy-collapse type: topic --- # Cross-Modal Entropy Collapse Cross-modal entropy collapse denotes a family of failure modes in multimodal and closely related adaptive systems in which uncertainty, representational diversity, or reliance on a source modality contracts toward a low-entropy regime. Recent work describes allied phenomena under different names: “evidence collapse” in reasoning VLMs, “late-stage modality collapse” in audio reasoning, an “Entropy Collapse Layer” in visual-token processing, “set collapse” in image-text retrieval, and collapse under test-time entropy minimization [2604.04207], [2605.27741], [2602.17196], [2506.21538], [2509.23183]. Across these settings, the system does not necessarily stop functioning; rather, it often becomes more confident while losing visual grounding, audio dependence, semantic diversity, or exploratory breadth. A broader theoretical literature formalizes collapse as convergence toward a stable low-entropy manifold rather than a literal zero-entropy state, although the extension of that formalism to multimodal systems is, in those works, an inferred application rather than a direct multimodal proof [2512.12381], [2512.14879]. ## 1. Conceptual scope and formal meaning The term is not yet standardized across the literature. In modality-specific work, the collapse is usually defined operationally: a reasoning VLM progressively stops attending to image evidence while maintaining or increasing answer confidence; an audio reasoning model increasingly abandons audio and falls back on language priors; a multimodal transformer compresses visual token representations into a low-rank redundant regime; or a set-based retrieval model loses diversity because multiple slots converge to nearly identical embeddings [2604.04207], [2605.27741], [2602.17196], [2506.21538]. A unifying interpretation is provided by the domain-agnostic theory of entropy collapse. There, the system state is represented as a distribution over states, $$ P_t \in \Delta(\mathcal{S}), \qquad P_{t+1}=\mathcal{F}(P_t;\alpha,\beta), $$ with entropy $$ H(P_t)=-\sum_{s\in\mathcal{S}} P_t(s)\log P_t(s). $$ The central claim is that collapse arises when feedback amplification exceeds bounded novelty regeneration. The resulting regime is not defined by entropy going to zero, system death, or inactivity; it is defined by convergence toward a stable low-entropy manifold and by contraction of effective adaptive dimensionality [2512.12381]. Applied to multimodal systems, this suggests that cross-modal entropy collapse is best understood as a contraction of the accessible multimodal state space. The contraction may occur in prediction space, attention dynamics, token representations, or embedding sets. What remains active is a narrower, more rigid, and often more overconfident process. That interpretation is directly stated in the general theory and is compatible with the modality-specific findings, though the fully unified multimodal formalization remains incomplete [2512.12381], [2512.14879]. ## 2. Dynamical mechanisms Several papers identify a structural asymmetry between a fixed source modality and an expanding autoregressive text process. In reasoning VLMs, the image is encoded once and stays fixed while the text context grows token by token. Attention therefore competes increasingly with the expanding text prefix. The observed consequence is that models can become more accurate while progressively losing visual grounding as reasoning unfolds, producing low-entropy but visually disengaged answers [2604.04207]. The audio-reasoning literature identifies a related mechanism during RL post-training. Standard GRPO applies a uniform token average, effectively distributing policy gradient mass across all tokens regardless of whether they depend on audio. The claim is that this structurally favors the language prior, because many tokens are predictable from text alone while only a subset are genuinely modality-critical. As generation lengthens, the model becomes “confidently textual”: fluent, low-entropy, and increasingly ungrounded in the audio stream [2605.27741]. Test-time adaptation work identifies a third route to collapse: entropy minimization itself can reward shortcuts. In ZeroSiam, plain entropy reduction is said to favor inflating logit norm and aligning all predictions toward a dominant class, yielding constant or near one-hot outputs that trivially minimize entropy without meaningful learning. The proposed objective, $$ \mathcal{L} = H(p^o) + \alpha \, D\!\left(p^o \,\|\, \mathrm{sg}[p^r]\right), $$ is designed to remove those trivial minima from the stable dynamics by introducing predictor–stop-gradient asymmetry before the classifier [2509.23183]. A broader geometric explanation is given by Entropy-Reservoir Bregman Projection. In that framework, self-referential learning repeatedly projects a model onto an empirical distribution sampled from its own current output, with target $$ Y=(1-\lambda)\,P+\lambda R. $$ Without reservoir coupling, finite-sample self-projection contracts support and drives exponential entropy decay; with a persistent entropy reservoir, the loop can maintain a non-trivial entropy floor [2512.14879]. A plausible implication is that multimodal pseudo-labeling, self-captioning, or self-retrieval loops are vulnerable whenever one modality is repeatedly regenerated from a progressively narrowing distribution. ## 3. Empirical manifestations across modalities In reasoning VLMs, evidence collapse is explicitly documented across three models and three datasets—MathVista, HallusionBench, and MMMU_Pro. Visual attention collapses in all 9 model×dataset cells, evidence-region mass also collapses in all 9 cells, total visual attention decay is positive in 94.3%–100% of samples per cell, evidence decay is positive in 83.5%–100%, and relative evidence loss ranges from 53.3% to 90.8%. The collapse typically occurs during reasoning rather than only at the final answer, and peak discrimination between correct and incorrect generations occurs during reasoning in 7/9 cells, with endpoint-only measurement missing up to 77.5% of the signal [2604.04207]. In audio reasoning, late-stage modality collapse is described as decay of audio attention mass \(a_t\) toward zero in later parts of chain-of-thought generation. The model may begin grounded in audio and then increasingly autoregress on a textual summary of its own earlier reasoning. MAPO is introduced to counter precisely this failure mode, and on MMAU, MMAR, MMSU, and MMAU-Pro it reports headline scores of 77.80, 70.90, 79.36, and 65.29, respectively, while improving long-horizon reasoning fidelity and multimodal instruction following [2605.27741]. In multimodal token processing, EntropyPrune reports a distinct form of collapse internal to representation flow. Layer-wise matrix entropy of visual representations remains relatively high in early layers and then undergoes a sharp drop at a specific depth, called the Entropy Collapse Layer. In the reported LLaVA settings, the collapse occurs around layer 2. After that point, the visual stream is interpreted as entering a redundancy-dominated regime. This behavior is reported across eight datasets, for both query and key states, and across LLaVA-1.5-7B and LLaVA-NeXT-7B [2602.17196]. In image-text retrieval, the collapse appears as degeneration of set-based embeddings. Instead of preserving multiple semantic facets, embeddings within a set become nearly identical, destroying the purpose of multi-embedding representations. The paper attributes this either to sparse supervision, as in MIL-style matching, or to semantic homogenization induced by Smooth-Chamfer similarity, which is argued to be minimized when all embeddings in a set are identical [2506.21538]. Cross-domain evidence also appears in test-time learning. ZeroSiam reports that the same anti-collapse mechanism stabilizes both vision adaptation and large language model reasoning. On ImageNet-C with label shifts it reaches 52.9% average accuracy, versus 44.4% for DeYO and 38.8% for SAR, and an ablation shows that removing stop-gradient causes severe collapse, with accuracy dropping from 51.6% to 20.5% on ResNet50-GN and from 64.1% to 38.5% on ViT-Base. In Llama3.1-8B-Instruct reasoning experiments, AIME24 improves from 3.33% baseline to 13.33% [2509.23183]. ## 4. Measurement, diagnostics, and observables The literature measures collapse through several complementary observables. In reasoning VLMs, text uncertainty is defined from top-\(k\) logprobs by $$ p_i=\frac{e^{\ell_i}}{\sum_{j=1}^{k} e^{\ell_j}}, \qquad H_t=-\sum_{i=1}^{k} p_i \log_2 p_i, $$ with token-averaged entropy over a span \(S\), $$ \bar{H}_S=\frac{1}{|S|}\sum_{t\in S} H_t. $$ The key empirical result is that full-response entropy is the most reliable text-only uncertainty signal under cross-dataset transfer, whereas answer-span entropy is unstable and can even invert direction. However, full-response entropy remains blind to whether the model is still visually grounded [2604.04207]. To track grounding rather than only confidence, the same work defines total visual attention mass \(V_t\), evidence-region attention mass \(A_{\text{bbox},t}\), and the relevant visual attention ratio \(\mathrm{RVAR}_t = A_{\text{bbox},t}/V_t\). Grounding layers are selected by evidence localization AUROC on a calibration set. This yields a diagnostic picture in which uncertainty and modality engagement are separable: low entropy can coexist with severe evidence disengagement [2604.04207]. The audio-reasoning literature introduces a modality-sensitive entropy signal through cross-modal differential entropy, $$ \Delta h_t = H(\pi_{\text{text-ref}}(\cdot \mid y_{