Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts (2511.04655v1)

Abstract: Robust benchmarks are crucial for evaluating Multimodal LLMs (MLLMs). Yet we find that models can ace many multimodal benchmarks without strong visual understanding, instead exploiting biases, linguistic priors, and superficial patterns. This is especially problematic for vision-centric benchmarks that are meant to require visual inputs. We adopt a diagnostic principle for benchmark design: if a benchmark can be gamed, it will be. Designers should therefore try to game'' their own benchmarks first, using diagnostic and debiasing procedures to systematically identify and mitigate non-visual biases. Effective diagnosis requires directlytraining on the test set'' -- probing the released test set for its intrinsic, exploitable patterns. We operationalize this standard with two components. First, we diagnose benchmark susceptibility using a Test-set Stress-Test'' (TsT) methodology. Our primary diagnostic tool involves fine-tuning a powerful LLM via $k$-fold cross-validation on exclusively the non-visual, textual inputs of the test set to reveal shortcut performance and assign each sample a bias score $s(x)$. We complement this with a lightweight Random Forest-based diagnostic operating on hand-crafted features for fast, interpretable auditing. Second, we debias benchmarks by filtering high-bias samples using anIterative Bias Pruning'' (IBP) procedure. Applying this framework to four benchmarks -- VSI-Bench, CV-Bench, MMMU, and VideoMME -- we uncover pervasive non-visual biases. As a case study, we apply our full framework to create VSI-Bench-Debiased, demonstrating reduced non-visual solvability and a wider vision-blind performance gap than the original.

• The paper introduces a diagnostic pipeline that trains models on test-set non-visual features to expose exploitable shortcuts in multimodal benchmarks.
• The methodology applies both LoRA-finetuned LLMs and Random Forests, demonstrating blind accuracy gains up to 33 percentage points from shortcut exploitation.
• The iterative bias pruning process effectively removes high-bias samples, ensuring that benchmark evaluations more accurately assess genuine visual reasoning.

Exposing and Mitigating Non-Visual Shortcuts in Multimodal Benchmarks via Test-Set Stress-Testing

Multimodal LLMs (MLLMs) have advanced rapidly due to scalable pretraining and increasingly expressive evaluation datasets. However, systematic vulnerabilities in benchmark design permit models to attain strong results by exploiting non-visual shortcuts encoded in questions and answer distributions, rather than demonstrating genuine visual reasoning capabilities. This paper articulates a diagnostic and mitigation pipeline combining adversarial test-set analysis and guided debiasing, demonstrating widespread shortcut susceptibility across prominent benchmarks and presenting concrete methods for robust multimodal evaluation.

---

The Evolving Problem of Benchmark Exploitability

Early benchmarks in vision (e.g., MNIST, ImageNet) focused on controlled, domain-specific tasks, but the proliferation of open-ended tasks like Visual Question Answering (VQA) and the scaling of LLMs have made benchmarks more expressive while simultaneously opening the door to more nuanced vulnerabilities.

Figure 1: The evolution from constrained to open-ended benchmarks increased expressivity but created opportunities for models to exploit linguistic or world knowledge instead of visual content.

This evolution means modern benchmarks often permit models to bypass visual information entirely, leveraging parametric world knowledge or idiosyncratic statistical regularities. Notably, "blind" evaluations, where MLLMs are deprived of visual input, commonly reveal high performance on vision-centric tasks, implicating the presence of exploitable non-visual shortcuts.

---

Defining and Diagnosing Non-Visual Shortcuts

The central claim is that a benchmark fails in its measurement of visual understanding when the visual modality is redundant—when correct answers can be robustly derived from non-visual inputs due to:

• Knowledge-based shortcuts: Models answer correctly using memorized world facts (e.g., the typical height of a refrigerator).
• Statistical shortcuts: Dataset artifacts (e.g., long-tailed or imbalanced answer distributions, object co-occurrence patterns, or sampling artifacts) that are learnable from the question set itself.

Empirical evaluation shows that fine-tuning LLMs on non-visual features of a vision-centric test set yields significant accuracy gains in "blind" settings. For example, blind performance of LLaVA-Video-7B on VSI-Bench jumps from 25.9% to 44.7% after fine-tuning, nearly matching the gain in vision-enabled mode, demonstrating that these statistical shortcuts equally benefit both configurations.

Figure 2: Blind versus vision-enabled performance growth with scaling; several benchmarks, such as MMMU, favor non-visual shortcut exploitation, while VSI-Bench shows robustness to knowledge-based shortcuts.

Additionally, prominent benchmarks show answer distribution skews and persistent statistical artifacts: counting tasks favor low numbers, spatial arrangements have predictable placement biases, and certain multimodal questions are solvable by learning common answer distributions, not by performing the intended visual reasoning.

---

Test-set Stress-Test (TsT): Framework and Implementations

To audit and quantify non-visual shortcut susceptibility, the paper proposes the Test-set Stress-Test (TsT): adversarially training diagnostic models directly on the test set's non-visual features using $k$-fold cross-validation, yielding:

• Global non-visual solvability metrics: The accuracy of diagnostics trained on $k-1$ folds and predicting the held-out fold quantifies the proportion of the test that is answerable non-visually.
• Per-sample bias scores $s(x)$: Model confidence in the ground truth for each sample, enabling targeted debiasing.

The TsT is realized in two primary forms:

• TsT-LLM: LoRA-finetuned LLMs process question and choice text, capturing complex linguistic and statistical shortcuts across benchmark types. Offers high fidelity but limited interpretability.
• TsT-RF: Random Forests operate on engineered non-visual features (e.g., TF-IDF vectors, object frequencies, template properties). Offer computational efficiency and feature importance analysis for root cause diagnosis but require feature engineering.

Applied to four representative benchmarks (VSI-Bench, CV-Bench, MMMU, VideoMME), both diagnostics demonstrate substantial non-visual exploitability with up to +33 percentage point “blind” accuracy gains (TsT-LLM on CV-Bench).

---

Iterative Bias Pruning (IBP): Targeted Benchmark Refinement

Armed with per-sample bias scores, the Iterative Bias Pruning (IBP) removes high $s(x)$ outlier samples in batches, re-computing $s(x)$ post-removal, ensuring that emerging artifacts are iteratively identified and removed. This process is governed by a removal budget, batch size, and an early-stopping threshold on acceptable residual bias.

For VSI-Bench, the IBP induced a 30.7% sample reduction, particularly pruning long-tailed or low-variance categories. The resultant “debiased” dataset (VSI-Bench-Debiased) displays:

• Increased vision-blind performance gap: After in-distribution finetuning, the blind model’s performance drops from 44.7% to 32.0%, while the vision-enabled gain remains larger (+17.4% vs. +11.7%), confirming mitigation of non-visual shortcuts.
• Modified answer distributions: Previously dominant object or answer categories were rendered less predictive and forced reliance on actual visual inputs.

---

Implications, Trade-offs, and Theoretical Significance

Key findings and implications are as follows:

• Benchmark evaluation remains a fundamental bottleneck: The majority of widely-used multimodal benchmarks exhibit nontrivial non-visual exploitability, leading to inflated and misleading progress claims. Blind evaluation alone, while diagnostic, fails to isolate or repair these vulnerabilities.
• Test-set specific bias diagnosis is essential: Generalizing from training set analysis misses idiosyncratic vulnerabilities of specific test splits, especially in datasets with procedural generation or human-in-the-loop design stages.
• Scalability and deployment considerations:
  • TsT-RF affords efficient repeated auditing but requires per-benchmark feature engineering.
  • TsT-LLM is model-agnostic and suitable for rapidly evolving, less-structured benchmarks but is computationally intensive and less interpretable.
• Debiasing via pruning offers practical robustness: While arbitrary sample removal can reduce coverage, stress-test-guided pruning demonstrably increases the vision reliance for the subset retained. However, there is an inherent trade-off between benchmark coverage and exploitable shortcut minimization: aggressive pruning could discard edge-cases, while conservative pruning may leave residual vulnerabilities unaddressed.
• General applicability: The TsT+IBP pipeline can be layered onto extant or future benchmarks (image, video, audio-visual, etc.), providing a general solution for ongoing benchmark curation without requiring complete dataset redesign.

---

Speculation on Future Directions

Future work will need to address several axes:

• Automated question or answer rewriting: While pruning is effective, generative mitigation may recast biased questions to minimize artifact predictability without discarding valuable content.
• Online or continual test-set auditing: As models evolve, so will their shortcut-exploitation abilities, necessitating periodic reauditing of benchmarks.
• Causal and adversarial test construction: Integrating the TsT approach during dataset creation could allow actively adversarial query synthesis, further increasing benchmark robustness.
• Cross-modal extension: While this work focuses on visual and multimodal benchmarks, analogous stress-testing is relevant for modalities such as audio, video, and sensor data, as LLMs integrate more parametric and world knowledge.

---

Conclusion

The work presents a principled, diagnostic paradigm for multimodal benchmark design—explicitly anticipating that “if a benchmark can be gamed, it will be.” By advocating for adversarial, test-set-driven bias analysis and systematic removal of shortcut-prone items, the framework ensures that published benchmarks more accurately measure the desired multimodal competencies. TsT and IBP together should become standard practice for rendering benchmarks immune to superficial pattern matching, thereby supporting both reproducibility and substantive progress in multimodal AI evaluation.