Read-or-Ignore Benchmark (RIO-Bench)

• The paper introduces RIO-Bench, a unified evaluation protocol that assesses selective text use in LVLMs by requiring context-dependent decisions on when to read or ignore text.
• It employs same-scene counterfactual construction to systematically vary question types and text conditions, thereby isolating the influence of misleading cues.
• Empirical findings reveal that adaptive training (RIO-RT) improves robustness against typographic attacks while preserving crucial text-reading performance.

Searching arXiv for RIO-Bench and closely related typographic-attack / text-reading benchmark papers. Read-or-Ignore Benchmark (RIO-Bench) is a benchmark and evaluation protocol for selective text use in large vision-LLMs (LVLMs): it measures whether a model can read text when it matters, but ignore misleading text when it should not matter. The benchmark is introduced together with Read-or-Ignore VQA (RIO-VQA), which unifies typographic-attack robustness and text recognition within the same scene rather than treating them as separate capabilities. Its central premise is that robustness evaluations focused on object recognition often reward a text-ignoring paradigm, whereas real deployments frequently require both object understanding and text-aware reasoning in context (Waseda et al., 10 Dec 2025).

1. Conceptual basis and motivation

RIO-Bench was proposed to address a mismatch between prior typographic-robustness evaluation and real multimodal requirements. Earlier evaluations, as characterized in the benchmark description, were largely centered on object recognition or object-centric reasoning. In that setting, a model can appear robust simply by refusing to read text. The benchmark argues that this creates a false notion of robustness, because a model that ignores all text may perform better under typographic attack while simultaneously becoming less useful in settings where written content is semantically indispensable (Waseda et al., 10 Dec 2025).

The benchmark therefore reframes the problem. Rather than asking only whether an LVLM resists misleading overlaid text, it asks whether the model can make a context-dependent decision policy about text relevance. This design explicitly couples two requirements: robustness to typographic distraction in object-centric settings, and accurate text reading when the question depends on in-image text. The motivating examples are operational rather than purely adversarial: an autonomous vehicle may need to recognize a pedestrian while also reading a traffic sign, and an embodied agent may need to follow written instructions in one context while disregarding distracting text in another.

This formulation places RIO-Bench between conventional robustness benchmarks and conventional Text-VQA-style evaluation. A common misconception is that typographic robustness and text-reading competence can be assessed independently without loss of realism. RIO-Bench rejects that separation and treats selective text use as the proper object of evaluation.

2. Read-or-Ignore VQA task formulation

The benchmark is organized around Read-or-Ignore VQA (RIO-VQA). The task is defined by two orthogonal axes: question type and text condition. Question type is either object-centric or text-centric. Text condition is either clean or attacked. Their cross-product yields four sub-tasks (Waseda et al., 10 Dec 2025).

Question type	Text condition	Intended behavior
Object-centric	Clean	answer from depicted object content
Object-centric	Attacked	ignore misleading text and rely on visual/object cues
Text-centric	Clean	read the relevant text
Text-centric	Attacked	read the relevant text despite distracting text

This structure is the operational definition of “read or ignore.” In Object-Attack, the correct behavior is to suppress the influence of misleading text. In Text-Clean and Text-Attack, the correct behavior is the opposite: the model must read the relevant text, including under distraction. In Object-Clean, text is unnecessary, while in Text-Clean it is necessary.

The significance of this formulation is that selective text use is not modeled as a fixed preference for or against reading. It is a conditional policy induced by the question and the visual scene. This suggests that any method optimized only for object-side robustness is incomplete unless it is simultaneously evaluated on cases where text should be read.

3. Same-scene counterfactual construction

RIO-Bench is built around same-scene counterfactuals. The same underlying image scene is reused while varying only the question type and textual content. This is the benchmark’s central control mechanism, because it removes confounds that would arise if object-centric and text-centric evaluation were performed on unrelated images (Waseda et al., 10 Dec 2025).

The construction uses TextVQA and Open Images. TextVQA provides text-centric questions, and Open Images provides object annotations for the same images. This dual-annotation property enables both branches of the benchmark to be instantiated on the same scene. For each base image, the benchmark generates a text-centric question from TextVQA and an object-centric question using Open Images labels. Typographic attacks are then applied to each branch, producing clean and attacked variants.

The result is a controlled four-state setup for each image:

• object question on clean image
• object question with attack text
• text question on clean image
• text question with attack text

This construction isolates the benchmark variable of interest. If a model changes behavior between object-centric and text-centric questions on the same scene, the change is attributable to selective reasoning rather than to dataset shift. Likewise, if a model is stable across clean and attacked variants when it should ignore the attack, that stability is interpretable as robustness rather than as an artifact of scene difficulty or annotation style.

4. Question generation and attack design

The object-centric branch includes two formats. In the multiple-choice (MC) format, the question asks which object is present in the image and provides four options. Negative options are selected using the Open Images class hierarchy, with hard / medium / easy negatives determined by semantic distance in the hierarchy. In the open-ended (OE) format, the model must name the object or objects in the image, and correctness is matched against Open Images labels (Waseda et al., 10 Dec 2025).

For object-centric attacks, the misleading text is an object label rendered into the image. The attack word is semantically related to the scene, does not overlap the actual object, and is placed outside existing text regions using OCR boxes. The benchmark defines three attack levels—easy, medium, and hard—corresponding to semantic proximity between the attack word and the true object label. Harder attacks are more semantically confusable.

For text-centric attacks, the benchmark adopts a different procedure because there is no obvious negative class analogous to object labels. It uses Llama-3 to generate a misleading but plausible contradictory word or short phrase. The benchmark then identifies the key text region supporting the correct answer using OCR matching heuristics in the order: exact match, fuzzy phrase match, and keyword-level fallback. After that, the attack text is placed at controlled distances using a grid-based strategy with near, mid, and far placement. The near condition is discarded because it can make the task ambiguous for humans; the retained mapping is easy = far and hard = mid.

This attack design is important because it does not merely test whether text appears anywhere in the image. It tests whether the model can separate relevant and irrelevant textual evidence under controlled semantic and spatial perturbation.

5. Metrics and formal evaluation

RIO-Bench uses different metrics for the object-centric and text-centric branches. For object-centric multiple choice, accuracy is

$$\text{MC-Acc} = \frac{1}{N} \sum_{n=1}^{N} \mathbf{1}[\hat{\imath}_n = i^\star_n].$$

For object-centric open-ended evaluation, the benchmark adapts CLIP-Match. Given a prediction $t$ and class name $c$, CLIP similarity is defined as

$s(c \mid t)=\langle v_t,\ v_c\rangle$

where $v_t$ and $v_c$ are normalized CLIP embeddings. It is extended to top-5 matching over multiple ground-truth classes:

$$\text{CLIP-M@5}(Y,t) = \mathbf{1}\!\left[ \mathrm{Top}\text{-}5(t)\ \cap\ Y \neq \emptyset \right].$$

To jointly score correctness and resistance to the injected attack word, the benchmark defines Robust CLIP-Match (R-CLIP-M):

$$\text{R-CLIP-M}(Y,a,t) = \text{CLIP-M@5}(Y,t) \;-\; \text{CLIP-M@5}(\{a\},t).$$

This metric explicitly rewards naming the correct object while penalizing attraction to the attack token (Waseda et al., 10 Dec 2025).

For text-centric questions, the benchmark uses the official VQA-style human-agreement score:

$\text{Acc} = \frac{1}{10} \sum_{i=1}^{10} \min\!\left(1,\; \frac{\#\{\,j \ne i : a^{(j)} = \hat{a}\,\}{3} \right).$

The benchmark’s formalism is aligned with its conceptual claim. Object-side evaluation is not limited to plain accuracy under attack; it includes an explicit penalty for prediction drift toward the attack text. Text-side evaluation retains the standard agreement-based VQA metric, preserving comparability with text-centric visual question answering practice.

6. Empirical findings and defense strategies

RIO-Bench evaluates several LVLMs, including LLaVA-1.5-7B / 13B, Qwen-2.5-VL-7B, Qwen3-VL-8B, Llama-3.2-Vision-11B, and SmolVLM-2B. It also evaluates two kinds of defenses: an inference-time defense, namely CoT prompting defense, and training-based defenses, namely IT-RT (Ignore-Text Robust Training) and RIO-RT (Read-or-Ignore Robust Training) (Waseda et al., 10 Dec 2025).

The reported pattern is consistent across model families. Pretrained LVLMs are often strong at reading text, but under object-centric typographic attacks they tend to overtrust textual cues and therefore fail on robustness. CoT defense can improve robustness somewhat, but tends to degrade text-reading performance because it is designed with an object-centric bias. IT-RT improves object robustness strongly, but often by learning to ignore text broadly, which damages text-reading ability and can do so dramatically. The benchmark interprets this as a failure mode of non-adaptive training: robustness is achieved by suppressing a capability that real tasks still require.

The proposed defense, RIO-RT, is trained on a balanced mixture of object-attack and text-attack data. Its distinguishing feature is that it teaches adaptive selective text use rather than a blanket rule to ignore text. The benchmark’s central empirical conclusion is that existing methods fail to balance typographic robustness and text-reading capability, whereas RIO-RT provides the best trade-off by improving robustness on object-attack cases while preserving much more text-reading performance and also improving Text-Attack performance.

The paper states that these training-based defenses use standard supervised fine-tuning. The novel element is not a new loss but the training distribution that supplies both “read” and “ignore” supervision. This suggests that selectivity is being learned as a data-induced behavior rather than being imposed through a hard robustness prior.

7. Position within benchmark design and related directions

RIO-Bench’s broader contribution is conceptual as well as empirical. It recasts typographic robustness as a problem of contextual reasoning rather than pure adversarial resistance. Traditional robustness benchmarks ask whether a model can resist misleading text; Text-VQA-style benchmarks ask whether a model can read text. RIO-Bench asks both questions in the same scene: can the model decide whether to read or ignore text? (Waseda et al., 10 Dec 2025).

This distinguishes it from adjacent multimodal reading benchmarks. For example, MMTR-Bench evaluates the intrinsic ability of multimodal LLMs to reconstruct masked text from visual context without an explicit question prompt. That benchmark is similarly concerned with reducing confounds from instruction-following and with probing more intrinsic reading behavior, but its task is masked-text reconstruction from visual context alone, not a contextual decision over whether text should influence an answer (Guo et al., 23 Apr 2026). The comparison is useful because it clarifies that RIO-Bench is not a generic reading benchmark: its target capability is selective text use under counterfactual scene control.

A further misconception is to treat RIO-Bench as only another dataset of attacked images. The benchmark is more precisely described as a unified evaluation protocol using same-scene counterfactuals to expose whether a model has learned genuine selective multimodal reasoning or has merely become text-blind in the name of robustness. Its significance lies in making the trade-off between robustness and text use measurable rather than anecdotal, and in shifting evaluation from “can the model ignore attack text?” to “can the model infer when text is relevant and when it is harmful?”