Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models

Abstract: Current evaluations for Multimodal LLMs (MLLMs) overwhelmingly focus on utility-driven objectives, leaving model behavior under logic-neutral scenarios largely underexplored. Stochasticity is essential in scenarios where multiple actions are equally valid, such as recommending travel itineraries or daily schedules where multiple options have similar utility. In such settings, deterministic policies may lead to repetitive behaviors and reduced coverage of valid alternatives. To bridge this gap, we propose RandomBench, a benchmark designed to evaluate whether MLLMs can maintain distributionally neutral behavior when selecting among equivalent options. We further introduce three metrics, including RI, BCI, BII, to quantify entropy and distributional bias. Experiments reveal a pervasive phenomenon termed Stochastic Collapse, where MLLMs fail to maintain uniform randomness under explicit random instructions, with top-1 probabilities reaching 97% from the ideal one quarter baseline and RI dropping to 0.068 in Claude Sonnet 4.6. Extensive ablation studies further demonstrate that these deviations persist across languages and representation formats, highlighting the robustness of distributional collapse in logic-neutral decision settings.

• The paper introduces RandomBench, a diagnostic framework that quantifies MLLMs' failure to preserve true randomness in logic-neutral settings.
• Methodology leverages statistical indices like the Randomness Index and Bias Consistency Index to measure entropic degradation across modalities and languages.
• Empirical results reveal severe bias, with some models showing over 90% selection concentration, underscoring deep pretraining-induced heuristics.

Evaluating Stochastic Collapse and Implicit Bias in Multimodal LLMs

Introduction and Problem Formulation

This work introduces RandomBench, a novel diagnostic benchmark designed to systematically evaluate the inability of current Multimodal LLMs (MLLMs) to preserve stochasticity in logic-neutral contexts. The principal claim is that, unlike human cognition, which is able to preserve true randomness when instructed to select among semantically equivalent options, MLLMs instead manifest strong endogenous biases — a phenomenon defined here as "stochastic collapse." This collapse is exhibited across different modalities and persists across multi-language and label permutation ablations, indicating a structural failure mode that is deeply rooted in the models' pretraining-induced parameter landscapes.

The introduced paradigm is motivated by two critical considerations: First, as MLLMs transition from question-answering tools to autonomous planning agents, the capacity for entropy-preserving, utility-neutral choice selection becomes essential to avoid repetitive policy entrapment and ensure equity in generative or recommendation outputs. Second, prior bias evaluation frameworks focus almost exclusively on logic-driven or social stereotypes under utility-maximizing objectives, largely ignoring the logic-neutral regime. RandomBench fills this methodological vacuum, enabling a fine-grained empirical investigation of the system-1 (heuristic) tendencies latent in current MLLMs.

Figure 1: Overview of the RandomBench Framework, highlighting the separation of RB-Text and RB-Vision modalities and the repeated-sampling protocol for unbiased evaluation.

RandomBench Framework Design

RandomBench consists of 200 rigorously curated, logic-neutral test instances evenly divided between RB-Text and RB-Vision modalities. Each instance is engineered to ensure absolute semantic equivalence between candidate options, stripping all cues that could induce utility-based or logical preference. The curation pipeline includes explicit removal of sociocultural, positional, and descriptive attributes and an expert review process for cross-modal structural balance.

Figure 2: Curation pipeline of the RandomBench Framework, illustrating rigorous filtering and expert review procedures for logic-neutrality.

In the RB-Text subset, instances are categorized into abstract symbol/number selection, linguistic/functionally redundant choices, spatial/perceptual context, and affective/social identity scenarios. RB-Vision extends the textual setup to incorporate geometric and saliency-based visual features, spatial layouts, affective/social visual cues, and cross-modal conflicts (Stroop-like tasks) that test modality prioritization under randomness constraints. The overall statistical makeup of RandomBench ensures fine-grained coverage across cognitive bias dimensions, with 50 independent sampling trials per instance yielding robust empirical distributions.

Figure 3: Statistics of the RandomBench Framework, confirming even distribution across modalities and cognitive categories.

Quantitative Evaluation Metrics

RandomBench introduces a set of information-theoretic and statistical indices to measure stochastic degradation:

• Randomness Index (RI): Normalized Shannon entropy across the empirical selection distribution, with $RI=1$ corresponding to ideal uniformity (maximum entropy) and lower values indicating increasing collapse.
• Bias Intensity Index (BII): KL divergence against the uniform distribution, quantifying the total deviation from semantic neutrality.
• Bias Consistency Index (BCI): Difference between the empirical maximum selection probability and the random baseline, measuring the persistence and magnitude of mode collapse.
• Jensen-Shannon Divergence (JSD): Employed for ablation analysis to quantify shifts in decision distributions across languages or label permutations.

Experimental Results and Empirical Observations

Stochastic Collapse as a Structural Failure Mode

Empirical evaluation of seven state-of-the-art MLLMs — GPT 5.1, Gemini 3.1 Flash-Lite, Claude Sonnet 4.6, Kimi K2.5, Qwen 3.6 Plus, Grok 4 Fast, and Doubao Seed 1.6 — demonstrates the ubiquity and severity of stochastic collapse. No model approaches the true uniform baseline under logic-neutral instructions; even with temperature $T=1.0$ and chain-of-thought suppression, probability mass typically concentrates on one or two options, with top-1 probabilities consistently exceeding 90% in representative cases (e.g., Claude Sonnet 4.6 yielding $RI=0.068$ on English RB-Vision tasks).

Figure 4: RB-Text Modality results highlighting model-specific and task-type-specific entropic degradation.

Radar analysis and numeric comparisons show that the phenomenon is more pronounced in the vision modality, where perceptual anchors and spatial saliency cues contribute to farther deviations from uniform stochasticity. Furthermore, the models with stronger instruction-following and higher alignment scores exhibit even more severe collapse, illustrating a capability-consistency paradox: optimization for instruction compliance rigidifies endogenous heuristics rather than mitigating them.

Figure 5: Radar Chart of English RI Results demonstrating fine-grained entropic variance across cognitive tasks and models.

Cross-Lingual and Symbolic Robustness Ablations

Ablation analyses establish that stochastic collapse is neither an artifact of option label choice nor restricted to a particular language. JSD analysis indicates substantial distributional shifts when task instructions are translated across English, Chinese, French, and Spanish. Modality and language interact nontrivially, such that linguistic context conditions the activation of latent biases, even when the visual or logical content is identical. Label permutation to geometric, Greek letter, or randomized hash symbols fails to restore entropy, and, in cases, amplifies collapse due to native token-frequency priors.

Figure 6: JS divergence across languages for photo and text tasks, quantifying language-conditioned distributional perturbations.

Qualitative Probes: Visual Saliency, Social and Emotional Biases

Case studies in the appendix expose the qualitative underpinnings of collapse. Visual saliency (e.g., imperceptible blur, classic perceptual illusions) produces severe anchoring effects, overriding randomness instructions even at the cost of explicit instruction violence or when the "correct" answer is declared in explanatory text. Social identity and affective cues (e.g., popularity, positive vs. negative framing, "Recommended" labels) elicit uniform collapse to culturally-charged options even under maximal entropy protocols. In zero-information and symbol-agnostic tasks, bias remains, indicating the decisive role of structural embedding priors.

Theoretical and Practical Implications

The findings demonstrate that MLLMs, across diverse architectures and alignment philosophies, are fundamentally unable to sustain cognitive neutrality in ambiguous, logic-free environments. This constraint is not remediable by temperature elevation or prompt reformatting, implicating the high-dimensional structure of parameter manifolds shaped by pretraining token-frequency distributions as the root cause. The phenomenon is mathematically framed as embedding manifold regression: the decoding process defaults to native token and modality-specific priors in the absence of discriminative objective gradients.

Practically, this implies a critical barrier for the application of MLLMs in fields requiring stochastic decision-making — autonomous systems, recommendation engines, procedural content generation, and fairness-critical allocation — as current models are not equipped to maintain principled uncertainty or effective exploration in indeterminate regimes. Catastrophic confidence and systematic bias can arise under conditions of ambiguity, leading to brittle and potentially unsafe system behaviors.

From a theoretical perspective, the work sharply distinguishes between the externally-triggered social bias literature and the hitherto unquantified endogenous heuristics defining MLLM system-1 behaviors. Cross-modal and cross-lingual evidence points toward a pervasive architecture-intrinsic vulnerability that cannot be wholly resolved by external calibration or inference-time debiasing.

Future Directions

Follow-up work should target three axes: (1) mechanistic interpretability to trace causal bias propagation through parameter space, (2) inference-time activation steering and regulatory mechanisms beyond conventional instruction alignment or temperature scaling, and (3) extending logic-neutral benchmarks to dynamic, embodied, or interactive multimodal settings such as text-to-video navigation and real-time agent planning. Open models with accessible hidden states will be critical for probing and remediating these biases at the substrate level.

Conclusion

RandomBench establishes a principled platform for evaluating the stochastic limitations and implicit biases of modern MLLMs beyond conventional utility-driven testing. The demonstration of stochastic collapse, visual hijacking, and cross-modal/language-dependent bias exposes a structural deficit that undermines the trustworthiness and generality of current AI decision paradigms. Achieving genuine randomness and cognitive neutrality — foundational requirements for robust AI deployment in ambiguous settings — will require future research into pretraining objectives, embedding geometry, and runtime behavioral modulation.

---

Reference: "Evaluating Stochastic Collapse and Implicit Bias in Multimodal LLMs" (2606.05874)