DecoyMNIST: Robust Vision Benchmark

• DecoyMNIST is a synthetic benchmark that introduces label-correlated decoy patches to reveal whether CNNs rely on superficial shortcuts instead of core digit structures.
• It employs a two-phase training regime with language-guided attention alignment and KL-divergence loss to shift model focus toward genuine digit features.
• Quantitative results demonstrate that language-guided methods achieve near state-of-the-art in-distribution accuracy and dramatic generalization improvements under shortcut-shift conditions.

DecoyMNIST is a synthetic vision benchmark specifically designed to probe whether classifiers, especially convolutional neural networks (CNNs), rely on spurious correlations ("^^^^0^^^^") rather than on core semantic content in their predictions. The dataset systematically introduces a tightly correlated visual confound—a small corner patch—with each digit label, making it possible to distinguish models that genuinely attend to digit structure from those that exploit superficial cues. DecoyMNIST has become a standard test case for work on explanation regularization, concept-based supervision, and attention alignment frameworks seeking robust, cognitively plausible model explanations (Yang et al., 25 Sep 2025).

1. Construction and Shortcut Challenge

DecoyMNIST begins with the standard MNIST dataset, comprising grayscale $28 \times 28$ images of handwritten digits. To each image $x \in \mathbb{R}^{28 \times 28}$, a small square "^^^^1^^^^" patch is added in one corner. The critical aspect is that the patch's position and intensity are deterministically mapped to the digit label within the training set (e.g., digit "0" always has a light-gray square in the upper-left, digit "1" a darker square in the upper-right, etc.).

This construction guarantees a nearly bijective mapping between patch and label during training. As a result, a vanilla CNN can attain very high in-distribution test accuracy simply by learning to associate patch characteristics with classes, effectively ignoring the actual digit strokes. However, this "shortcut" reliance is exposed if, at test time, (a) the patch location changes or (b) the intensity-label mapping is randomized—a classic out-of-distribution (OOD) regime known as shortcut-shift. Here, models that memorize patches experience catastrophic performance drops, while models that attend to digits generalize.

2. Model Architectures and Attention Alignment Framework

The canonical architecture in DecoyMNIST studies is a small LeNet-style convolutional backbone $f_\theta$, consistent with prior MNIST work, trained via stochastic gradient descent with momentum and regularization (Yang et al., 25 Sep 2025).

To discourage shortcut exploitation, recent work leverages vision-LLMs to generate language-guided attention maps, $M_{\mathrm{VL}}(x, y)$, for each $(x, y)$ training pair. During training, the model also produces its own class-activation map (CAM), denoted $S_\theta(x, y) \in [0, 1]^{H \times W}$ with $\sum_{h, w} S_\theta(x, y)[h, w] = 1$. The loss minimized for a mini-batch of size $B$ combines standard cross-entropy,

$$\mathcal{L}_{\mathrm{CE}} = -\frac1B \sum_{i=1}^B \log p_\theta(y_i | x_i)$$

with a KL-divergence attention alignment objective,

$$\mathcal{L}_{\mathrm{attn}} = \frac1B \sum_{i=1}^B \mathrm{KL} \left( S_\theta(x_i, y_i) \;\|\; M_{\mathrm{VL}}(x_i, y_i) \right)$$

and the combined training loss is

$x \in \mathbb{R}^{28 \times 28}$0

where $x \in \mathbb{R}^{28 \times 28}$1 tunes attention alignment strength. Training proceeds in two phases: first, optimizing only $x \in \mathbb{R}^{28 \times 28}$2 ("learn to look"), after which the optimizer is reset and full loss is minimized, ramping $x \in \mathbb{R}^{28 \times 28}$3 upward to maintain prioritization of attention alignment.

3. Generation of Language-Guided Attention Maps

DecoyMNIST experiments incorporate WeCLIP$x \in \mathbb{R}^{28 \times 28}$4 (a "Frozen CLIP-DINO" vision-LLM) in a weakly supervised segmentation mode. For each input image-label pair, a minimal natural language prompt set is constructed:

• Foreground: "digit"
• Background/nuisance: "Background", "dark", "black", "corner", "patch", "box", "corner patch"

Passing $x \in \mathbb{R}^{28 \times 28}$5 to WeCLIP$x \in \mathbb{R}^{28 \times 28}$6 produces a raw map $x \in \mathbb{R}^{28 \times 28}$7. Importantly, these maps are sufficiently sharp around digit strokes, obviating the need for post-processing operations such as morphological filtering or edge-based refinement. Minimal prompting is employed to avoid introducing color or style cues from language, and it was empirically observed that more elaborate prompts do not meaningfully improve attention masks.

4. Quantitative and Qualitative Evaluation

Evaluation is conducted in both in-distribution (original patch-label mapping) and OOD (patch mapping randomized) regimes. The table below summarizes test accuracy across five methods, averaged over five random seeds:

Method	In-distribution (%)	OOD with Patch Mapping Randomized (%)
Base	52.8	12.3
CDEP	97.2	14.7
RRR	99.0	16.4
CDBS	98.9	15.8
Ours (Language-guided)	96.19 ± 0.35	82.6 ± 1.1

On in-distribution DecoyMNIST, the language-guided attention alignment strategy achieves 96.19% ± 0.35 accuracy, within 2 percentage points of annotation-heavy baselines (CDBS: 98.9%, RRR: 99.0%) (Yang et al., 25 Sep 2025). In OOD ("shortcut-shifted") evaluation, only models that successfully ignore the patch generalize non-trivially, with the language-guided method achieving 82.6% ± 1.1% versus ≤16.4% for all other baselines. This demonstrates robust digit-centric decision making unsupported by shortcut cues.

Qualitative comparisons (see Figure 1 in (Yang et al., 25 Sep 2025)) highlight that base CNNs focus CAM saliency almost entirely on the ^^^^1^^^^ patch. In contrast, attention-aligned CNNs show saliency sharply concentrated on digit strokes, visually confirming the efficacy of KL-based alignment in shifting model focus away from confounders.

5. Ablation Studies and Hyperparameter Optimization

Ablations involve grid searches over $x \in \mathbb{R}^{28 \times 28}$8 and attention-phase cutoff $x \in \mathbb{R}^{28 \times 28}$9, using composite validation metric $f_\theta$0. For DecoyMNIST, $f_\theta$1 and $f_\theta$2 yield the highest score. Optimization uses a learning rate of $f_\theta$3, decayed by 0.1 every 7 epochs; total training spans 30 epochs. The two-phase regime maximizes attention consistency before classification fine-tuning. The choice of minimalistic prompts ensures that no information beyond the presence of "digit" vs. generic background is available, further validating the framework's ability to provide pseudo-masks without leaking confounding cues (Yang et al., 25 Sep 2025).

6. Significance and Legacy

DecoyMNIST provides a canonical testbed for research into robust model explainability and shortcut avoidance in vision systems. By constructing a scenario where shortcut reliance yields catastrophic OOD generalization collapse, it enables unambiguous discrimination between models that genuinely learn target concepts and those that "cheat." The integration of fully automatic, language-guided region supervision represents a scalable alternative to concept-based explanation methods reliant on expert-labeled saliency or concept regions. Importantly, the attention-alignment techniques validated on DecoyMNIST deliver near state-of-the-art in-distribution results and dramatic improvements in OOD robustness, without any manual region annotation (Yang et al., 25 Sep 2025).

A plausible implication is that similar confounder-injection benchmarks, combined with scalable language-guided explanation constraints, may accelerate progress toward cognitively plausible explanations and robust generalization in deep vision architectures.