RIO-VQA: Selective Text Use in VQA

• The paper introduces RIO-VQA and RIO-Bench to evaluate models' ability to switch between reading and ignoring text based on object-centric or text-centric questions.
• RIO-Bench employs a same-scene counterfactual design with four sub-tasks formed by crossing question type and text condition to expose typographic attack failures.
• The study demonstrates that robust fine-tuning with RIO-RT can balance object and text performance, mitigating over-reading and under-reading in visual question answering.

Read-or-Ignore VQA (RIO-VQA) is a visual question answering task that explicitly evaluates a model’s ability for selective text use: deciding, from context, when to read text and when to ignore it. Introduced together with the Read-or-Ignore Benchmark (RIO-Bench), it addresses the tension exposed by typographic attacks: large vision-LLMs may over-trust misleading in-image text in object-centric questions, yet real applications also require accurate reading of genuinely relevant text in text-centric questions. The benchmark resolves this tension through same-scene counterfactuals that hold the scene fixed while varying only the textual content and the question type (Waseda et al., 10 Dec 2025).

1. Task definition and conceptual boundaries

RIO-VQA is organized along two orthogonal axes: question type and text condition. Question type is either object-centric or text-centric; text condition is either clean or attacked. Their cross-product yields four sub-tasks (Waseda et al., 10 Dec 2025).

Question type	Clean	Attacked
Object-centric	Object-Clean	Object-Attack
Text-centric	Text-Clean	Text-Attack

In object-centric questions, the desired behavior is to answer from visual object evidence and, under attack, to ignore misleading text. In text-centric questions, the desired behavior is to read the relevant scene text, while still ignoring contradictory distractor text when an attack is present. The benchmark therefore rejects both extremes that dominate earlier robustness work: globally reading text and being fooled by typographic overlays, or globally suppressing text sensitivity and losing legitimate text-reading capability (Waseda et al., 10 Dec 2025).

The motivating threat model is the typographic attack, in which misleading text is placed inside an image so that a model’s prediction is steered by the text rather than the underlying scene. The paper argues that prior evaluations are too object-centric: they ask whether a model can still recognize the object when misleading words are added, which makes a crude ignore all text policy look effective even though many real tasks require joint reasoning over objects and text, such as recognizing pedestrians while reading traffic signs (Waseda et al., 10 Dec 2025).

RIO-VQA should be distinguished from neighboring uses of similar language. In particular, R$^3$-VQA denotes “Read-the-Room Reasoning for Video Question Answering”, a benchmark for social reasoning in complex videos rather than selective text use under typographic attacks (Niu et al., 7 May 2025).

2. RIO-Bench construction and same-scene counterfactual design

RIO-Bench is built on TextVQA, whose images come from Open Images. The construction pipeline begins from the TextVQA official split with 34,602 training QA pairs and 5,000 validation QA pairs. Because TextVQA has multiple questions per image, one text-centric QA is retained per image by keeping the QA with the smallest index, yielding 21,953 unique training images and 3,166 unique validation images. For each base image, the benchmark creates one object-centric QA, one text-centric QA, and attacked variants at multiple difficulty levels, producing 11 subsets per split. Each subset contains one QA per image, giving 21,953 train and 3,166 val samples per subset, for totals of 241,483 training and 34,826 validation examples (Waseda et al., 10 Dec 2025).

The object-centric branch is synthesized from Open Images labels and class hierarchy. It includes both multiple-choice and open-ended formats. In the multiple-choice format, the benchmark uses the template:

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$9

Negative options are sampled hierarchically by semantic distance in the Open Images taxonomy: hard negatives share the same parent, medium share the same grandparent, and easy come from a higher-level ancestor region. Open-ended object questions use the template:

$A$0

The text-centric branch uses the original TextVQA questions and preserves OCR bounding boxes, which later constrain attack placement (Waseda et al., 10 Dec 2025).

Attack rendering is standardized. For all attacks, the benchmark uses DejaVuSans, font size 24–32 px, and one of eight high-visibility colors: white, black, red, orange, green, blue, yellow, purple. Candidate placements that overlap OCR boxes are rejected and resampled up to 100 times so that attack text does not overwrite genuine scene text (Waseda et al., 10 Dec 2025).

For object attacks, the attack word is a misleading object label chosen from the negative options prepared during multiple-choice construction. Difficulty levels—Easy, Medium, and Hard—are inherited from the semantic similarity of the distractor label to the ground-truth object. For text attacks, the benchmark generates a misleading word or short phrase with Llama-3, constrained to remain in the same general category or context as the correct answer while contradicting it and not overlapping with it (Waseda et al., 10 Dec 2025).

Text-attack placement depends on identifying the key text region that supports the correct answer. The benchmark matches OCR tokens to the answer or question using exact token/phrase match, fuzzy phrase match, and keyword fallback. The fuzzy similarity is

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$

Difficulty is then defined by distance from the attack box $A$ to the key text box $B$, using an $N \times N$ grid with default $N=7$. If the boxes do not overlap, the normalized distance is

$d = \frac{\min_{a \in \mathrm{corners}(A),\, b \in \mathrm{corners}(B)} \|a - b\|}{\text{image diagonal} \in [0,1].$

Grid cells are sorted by $d$, split into tertiles, near is discarded as often ill-posed, and the remaining cells define

$\text{easy} = far, \qquad \text{hard} = mid.$

The resulting same-scene counterfactual structure is the benchmark’s central methodological contribution: object and text behaviors are tested on the same underlying image, so differences cannot be explained away by scene difficulty (Waseda et al., 10 Dec 2025).

3. Evaluation protocol and metrics

RIO-Bench evaluates both pretrained LVLMs and defenses. The reported model set includes LLaVA-1.5-7B, LLaVA-1.5-13B, Qwen-2.5-VL-7B, Llama-3.2-11B-Vision-Instruct, and SmolVLM-2B, together with CoT-defense, Ignore-Text Robust Training (IT-RT), and Read-or-Ignore Robust Training (RIO-RT) (Waseda et al., 10 Dec 2025).

The benchmark uses task-specific metrics rather than a single composite read-ignore score. For object multiple-choice evaluation, accuracy is

$$\text{MC-Acc} = \frac{1}{N} \sum_{n=1}^{N} \mathbf{1}[\hat{\imath}_n = i^\star_n].$$

For object open-ended evaluation, the paper extends CLIP-Match to a robustness-aware metric. With normalized CLIP similarity

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$0

the top-5 matching criterion is

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$1

and the robustness-aware score is

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$2

This rewards alignment with a correct class name while penalizing alignment with the attack word (Waseda et al., 10 Dec 2025).

For text-centric evaluation, the benchmark uses the official TextVQA scoring rule,

$$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$3

This preserves standard TextVQA compatibility while placing attacked and clean settings into the same protocol (Waseda et al., 10 Dec 2025).

The benchmark’s within-scene evaluation is itself a methodological claim. Same-scene counterfactuals expose whether a model can switch policies—read or ignore—depending on the question, rather than appearing good on separate object and text datasets without ever demonstrating context-sensitive behavior (Waseda et al., 10 Dec 2025).

4. Failure modes revealed by RIO-VQA

The most prominent failure mode in pretrained LVLMs is over-reading: text is treated as privileged evidence even when the question is object-centric. On Qwen-2.5-VL-7B, object multiple-choice accuracy drops from 96.9 on clean images to 65.9 on object attacks, while text-centric performance remains high at 84.6 on clean and 78.1 on attacked text questions. On LLaVA-1.5-7B, object multiple-choice falls from 93.5 to 73.7, while text-centric accuracy moves from 49.5 to 45.1 (Waseda et al., 10 Dec 2025). The benchmark interprets this pattern as evidence that strong text-reading ability and typographic robustness are in tension when selective text use is not learned.

The reciprocal failure mode is under-reading. Defenses that suppress text sensitivity can become robust on object attacks precisely by losing legitimate text-reading capability. This is clearest with IT-RT. For LLaVA-1.5-13B, text-centric clean performance drops from 54.5 to 15.3, and attacked text performance drops from 49.7 to 12.0, even though object robustness improves sharply (Waseda et al., 10 Dec 2025). Under RIO-VQA, such a model is not robust in the intended sense; it has learned a one-sided policy of ignoring text.

A third failure mode is inability to adapt behavior to question context within the same scene. CoT-defense illustrates this instability. For LLaVA-1.5-13B, object multiple-choice attack accuracy improves from 68.3 to 78.7, but text-attack accuracy drops from 49.7 to 46.2. For Qwen3-VL-7B, CoT-defense worsens both object attack accuracy, 72.4 \rightarrow 64.0, and text-attack accuracy, 78.9 \rightarrow 70.3 (Waseda et al., 10 Dec 2025). The paper attributes this to an explicitly object-centric prompt that suppresses text reading without altering underlying model parameters.

The benchmark also includes qualitative attention analysis. In an Obj-Attack example, a pretrained model attends to the attack word “Remote Control” rather than the actual “Computer monitor”. In a Text-Attack example, a pretrained model prioritizes distractor “dell” over the true answer “iMac”. RIO-RT is reported to reduce final attention to the distractor in object-centric cases while still prioritizing the relevant scene text in text-centric cases (Waseda et al., 10 Dec 2025). The paper’s conclusion is explicit: robustness is not achieved by never looking at text, but by reasoning about when text should be trusted.

LoRA placement ablations reinforce this interpretation. For LLaVA-1.5-7B, pretrained performance is 73.7 on object attacks and 45.1 on text attacks. With LoRA on both vision and language, this becomes 97.9 and 53.3. With vision-only LoRA it is 83.8 and 38.4, whereas language-only LoRA gives 97.8 and 50.0. For Qwen2.5-VL-7B, the paper reports that vision-only barely helps and language-only is close to full V+L tuning (Waseda et al., 10 Dec 2025). This indicates that selective text use is learned mainly through the LLM side, not by vision-side text suppression alone.

5. RIO-RT: adaptive selective text use through balanced robust fine-tuning

The benchmark’s proposed defense is Read-or-Ignore Robust Training (RIO-RT). It is a data-driven, context-aware training procedure rather than a new architecture. The default training mixture contains 16K samples: Obj-Attack (Hard) with 4K MC + 4K OE and Text-Attack (Hard) with 8K examples. Training uses LoRA with $$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$4, one epoch, learning rate $$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$5, cosine decay, and a standard VQA-style supervised fine-tuning objective. The method introduces no explicit gate, no external OCR selector, no specialized classifier head, and no specialized routing loss; the adaptive behavior is learned implicitly from balanced SFT data (Waseda et al., 10 Dec 2025).

The reported gains are substantial. On LLaVA-1.5-7B, object multiple-choice attack accuracy improves from 73.7 to 97.9, text-clean accuracy improves from 49.5 to 53.3, and text-attack accuracy improves from 45.1 to 53.3. On Qwen-2.5-VL-7B, object attack accuracy rises from 65.9 to 98.5, text-clean remains effectively preserved at 84.6 \rightarrow 84.3, and text-attack improves from 78.1 to 85.3. On Llama-3.2-11B-Vision, object attack accuracy rises from 59.2 to 98.6, text-clean from 65.5 to 81.3, and text-attack from 56.2 to 81.7 (Waseda et al., 10 Dec 2025). These results are presented as evidence that selective robust training can improve object robustness without paying the under-reading cost incurred by IT-RT.

The paper also reports favorable efficiency properties. Training LLaVA-1.5-7B for one epoch on 16K samples takes 2.5 hours on a single H100, full dataset construction takes roughly 3 hours, and a data-size ablation indicates that performance already converges around the 16K setting (Waseda et al., 10 Dec 2025). The benchmark therefore functions not only as an evaluation suite but also as an enabling training corpus for adaptive selective text use.

At the same time, RIO-RT remains limited in a precise sense. Because it uses standard SFT and lacks an explicit read/ignore decision module, it learns policy implicitly from examples rather than implementing a formal routing mechanism. The paper also states that the current scope is limited to VQA, with no explicit extension yet to broader multimodal tasks, and notes that some transfer results remain domain-dependent (Waseda et al., 10 Dec 2025).

6. Precursor formulations, neighboring problems, and unresolved issues

Although RIO-VQA is explicitly named and formalized only in the typographic-attack benchmark, several earlier lines of work address closely related selective-use questions. In reliable VQA, the task is formulated as selective prediction with abstention, extending a standard VQA model $$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$6 into a selective model $$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$7 using a selection function $$s = \alpha\,\text{LCS\_ratio} + (1-\alpha)\,(1-\text{edit\_distance}), \quad \alpha = 0.5.$$8. That work evaluates coverage and risk, and shows that for CLIP-ViL at 1% risk, a learned multimodal selector improves coverage from 6.83% with maximum softmax probability to 15.66%, a 2.3× increase (Whitehead et al., 2022). This is not a typographic-attacks benchmark, but it supplies a general answer-or-ignore formalism for VQA.

Text-grounding papers in TextVQA expose the shortcut behaviors that RIO-VQA later systematizes. “Making the V in Text-VQA Matter” argues that TextVQA models may answer from OCR or answer priors while ignoring visual grounding, as in predicting “STOP” for signboard questions regardless of the actual image. Its qualitative example “What number is on the middle bike?” shows a standard M4C model answering “598” and a union-trained model answering “30”, illustrating failure and recovery of grounded reading under spatial disambiguation (Hegde et al., 2023). RUArt treats text-based VQA as contextual understanding of OCR tokens and object relations, with question-conditioned attention and OCR-object semantic/positional reasoning functioning as an implicit read-versus-ignore mechanism, even though it does not provide an explicit binary relevance head (Jin et al., 2020).

There is also an older answerability line. “Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions” decomposes VQA rejection into detecting non-visual questions and visual false-premise questions, yielding an early image-grounded answer-vs-refuse formulation (Ray et al., 2016). This is narrower than RIO-VQA’s selective text use, but it shares the core principle that a system should not answer every image-question pair as if grounding were guaranteed.

The unresolved issues follow directly from these comparisons. RIO-Bench does not define a composite read-ignore utility metric analogous to coverage-risk or Effective Reliability; it remains a task-specific battery of clean and attacked accuracies (Waseda et al., 10 Dec 2025). RIO-RT demonstrates that adaptive selective text use can be learned without a new architecture, yet it does so implicitly, not via an explicit gate. A plausible implication is that future work may combine the same-scene counterfactual design of RIO-Bench with the explicit selective-prediction machinery of reliable VQA, and with stronger OCR-grounding mechanisms from TextVQA research, to produce models that not only read and ignore selectively but also know when they should refuse, defer, or request additional context (Whitehead et al., 2022, Hegde et al., 2023, Jin et al., 2020).

In that sense, RIO-VQA marks a change in what typographic robustness is asked to measure. The benchmark’s central claim is not that a model should resist text, but that it should use text selectively. Models that over-read fail Object-Attack; models that under-read fail Text-Clean and Text-Attack; and models that cannot switch behavior within the same scene fail the task that RIO-VQA was designed to expose (Waseda et al., 10 Dec 2025).