Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models

🎉 Accepted to ECCV 2026 🎉

Futa Waseda^{1, 2, 3} Shojiro Yamabe^{1, 4} Daiki Shiono^{1, 5} Kento Sasaki¹ Tsubasa Takahashi¹

¹Turing Inc. ²The University of Tokyo ³National Institute of Informatics ⁴Institute of Science Tokyo ⁵Tohoku University

Desired behavior in our novel task, Read-or-Ignore VQA (RIO-VQA) task.

The scene contains a genuine speed-limit sign reading "20km/h" (blue box) — text that should be read — and an irrelevant "STOP" sign placed nearby as a typographic-attack distractor (red box) — text that should be ignored. A model without defense over-trusts the distractor and wrongly halts the car. A naive ignore-text defense avoids that mistake, but as a side effect also misreads the genuine speed limit. The desired behavior does both at once: ignore the misleading distractor and still correctly read the question-relevant text.

Motivation

LVLMs are vulnerable to typographic attacks, where misleading inserted text can override correct visual understanding. Existing evaluations and defenses are almost exclusively object-centric, achieving robustness by simply suppressing text sensitivity — but real-world use (e.g., reading traffic signs while recognizing pedestrians) needs selective reading, not uniform ignoring. A defense that scores well object-centrically could therefore be quietly destroying real text-reading ability, and current protocols have no way to catch this.

Summary

Task. We formulate RIO-VQA, requiring context-dependent read-or-ignore behavior: read question-relevant scene text when needed, while ignoring inserted distractor text.
Dataset. We introduce RIO-Bench, a same-scene counterfactual benchmark that varies only question intent (object vs. text) and text condition (clean vs. attack), enabling direct, confound-free comparison of model behaviors.
Key finding. Object-centric defenses can gain robustness by suppressing text sensitivity — at the cost of text-reading performance, a trade-off invisible to standard object-centric evaluation.
Baseline defense. We provide RIO-RT, an architecture-agnostic, data-driven defense trained on RIO-Bench that improves robustness while largely preserving text understanding, outperforming text-ignoring baselines across six recent LVLMs.

RIO-VQA Task Taxonomy

RIO-VQA crosses two axes — question intent (object vs. text) and text condition (clean vs. attack) — yielding four settings. Along the object axis, a good model must ignore misleading overlaid text (Obj-Clean → Obj-Attack); along the text axis, it must instead read the same kind of text when the question asks about it (Text-Clean → Text-Attack). A robust model must therefore selectively read or ignore text depending on context, rather than always ignoring it.

RIO-Bench Construction

Starting from real TextVQA images, we build same-scene counterfactuals that vary only the overlaid text. For Obj-Clean/Obj-Attack, an object-level distractor word is inserted, chosen by class hierarchy (harder distractors are semantically closer to the ground truth). For Text-Attack, an LLM generates a plausible-but-incorrect word that is overlaid near the key text, avoiding bounding-box overlap. This yields paired multiple-choice and open-ended questions that isolate a model's selective read-or-ignore behavior.

Main Results

original: the off-the-shelf LVLM, no defense.
CoT: inference-time defense — prompts the model to reason before answering, no training.
IT-RT (Ignore-Text Robust Training): fine-tuned only on Obj-Attack data — the standard object-centric defense recipe, and RIO-RT's baseline.
RIO-RT (ours): same backbone/budget as IT-RT, but fine-tuned on a balanced mixture of Obj-Attack and Text-Attack data.
Finding: original and IT-RT trade off Obj-Attack robustness against Text-Attack accuracy; RIO-RT improves the former while largely preserving the latter.

Qwen3-VL-8B

Qwen2.5-VL-7B

LLaVA-1.5-13B

Llama-3.2-11B-Vision

Why Does RIO-RT Work? An Attention Analysis

Attention rollout across layers shows that the original model and IT-RT often mislocate attention onto the wrong region for the requested behavior, leading to incorrect answers. RIO-RT sharpens attention onto the object when the task calls for ignoring text (left), and onto the key text when the task calls for reading it (right) — correctly answering both cases.

Attention visualization comparing Original, IT-RT, and RIO-RT

BibTeX

                
                    @inproceedings{waseda2026read,
                        title        = {Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models},
                        author       = {Waseda, Futa and Yamabe, Shojiro and Shiono, Daiki and Sasaki, Kento and Takahashi, Tsubasa},
                        booktitle    = {European Conference on Computer Vision (ECCV)},
                        year         = {2026},
                        eprint       = {2512.11899},
                        archivePrefix= {arXiv},
                        primaryClass = {cs.CV},
                        url          = {https://arxiv.org/abs/2512.11899},
                    }