ZoomBench: Fine-Grained Perception Benchmark

• ZoomBench is a hybrid-annotated benchmark of 845 VQA instances across six fine-grained perceptual dimensions, designed to test multimodal large language models.
• It features a dual-view evaluation protocol that compares full-image (global) and micro-cropped (regional) performance to quantify the internal ‘zooming gap’.
• The benchmark employs a Region-to-Image Distillation pipeline with automated generation and human verification to ensure high fidelity in fine-grained evidence retrieval.

Searching arXiv for the specified paper and closely related work on fine-grained multimodal perception benchmarks. arxiv_search(query="Zooming without Zooming Region-to-Image Distillation Fine-Grained Multimodal Perception ZoomBench", max_results=5, sort_by="relevance") ZoomBench is a hybrid-annotated benchmark of 845 VQA data spanning six fine-grained perceptual dimensions, introduced in "Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception" (Wei et al., 12 Feb 2026). It is designed for the evaluation of fine-grained perception in multimodal LLMs (MLLMs), especially in regimes where decisive evidence is small and easily overwhelmed by global context. Its defining feature is a dual-view protocol that compares performance on the full image with performance on the corresponding micro-cropped region, thereby quantifying a global–regional "zooming gap."

1. Motivation and problem setting

ZoomBench is motivated by a specific failure mode of modern MLLMs: strong broad visual understanding coexists with systematic weakness on fine-grained perception tasks in which the decisive evidence is tiny, such as small text, subtle color shades, or minute object parts (Wei et al., 12 Feb 2026). In a single forward pass over the full image, these micro cues are easily overwhelmed by clutter. The benchmark is therefore structured to isolate failures that arise not from the absence of visual evidence, but from the inability to retrieve or use that evidence when it is embedded in the global scene.

This framing is closely tied to the contrast between single-glance inference and agentic "Thinking-with-Images" (TwI). Existing TwI approaches alleviate fine-grained failures by iteratively zooming into regions at test time, but they incur high latency through multiple tool calls and repeated vision encodings. ZoomBench targets the corresponding evaluation problem: it provides a challenge set and protocol that support a direct single-glance setting, with no test-time tool calls, while also measuring how well a model "zooms in" internally by comparing performance with and without explicit region crops.

A central implication of this design is that ZoomBench does not treat fine-grained perception as a unitary capability. Instead, it operationalizes the distinction between what a model can recognize when the relevant evidence is explicitly isolated and what it can recover from the full scene. This distinction underlies both the dataset construction and the evaluation protocol.

2. Dataset composition and perceptual dimensions

The benchmark contains 845 VQA instances. Images are sourced from high-resolution pools, mostly greater than $800 \times 800$ px, drawn from SA-1B, LAION-5B, MetaCLIP, Visual Genome, CC12M, and STPLS3D (Wei et al., 12 Feb 2026). The examples are distributed across six fine-grained perceptual dimensions, with an approximate per-dimension allocation over the 845 samples of about 140 examples for Counting, 145 for OCR, 140 for Color, 135 for Structure, 140 for Material, and 145 for Identification.

The six dimensions are defined as follows.

Dimension	Definition	Example
Fine-Grained Counting	Counting small and densely packed objects	3 fish
OCR (Text Recognition)	Reading tiny text or symbols that are legible only when zoomed in	"8275."
Color Attributes	Discerning subtle color variations in a small part of an object	blue
Structural Attributes	Judging shape or geometric properties of a small component	inverted triangle
Material Attributes	Identifying surface or material composition	plastic
Object Identification	Recognizing a small object or logo	Russian flag

These categories jointly delimit the benchmark’s scope. They are not generic scene-understanding tasks; each category is constructed around evidence that becomes substantially easier to access when the image is cropped to a relevant micro-region. This suggests that ZoomBench is intended less as a broad capability benchmark than as a targeted stress test for fine-grained perceptual retrieval under global visual context.

The answer space is also mixed. The benchmark contains 621 multiple-choice questions and 224 open-ended questions with canonical answers. That mixture allows the benchmark to probe both constrained selection and direct answer generation while retaining a common evaluation framework.

3. Hybrid annotation and Region-to-Image Distillation

ZoomBench is built through a hybrid annotation procedure that combines automated generation with human verification (Wei et al., 12 Feb 2026). Its data-generation core is the Region-to-Image Distillation (R2I) pipeline, presented as Algorithm 1 in the source paper.

For each raw image $I$, the pipeline first proposes candidate micro-crops $\{R\}$ via object detection or segmentation, filtering for regions satisfying $\mathrm{area}(R)/\mathrm{area}(I) < 0.1$. On each crop $R$, a teacher model generates a small set of fine-grained questions $\{Q\}$. Multiple teacher models then vote on answers $A$ to ensure high consensus, with a threshold of at least $6/8$ agreement.

The crucial step is the remapping of regional supervision back to the full image. To do so, the crop’s bounding box $B$ is overlaid onto $I$, forming $I$0; a spatial constraint is appended to the question, forming $I$1; and the agreed answer $I$2 is paired with that transformed input to yield $I$3. Overly easy examples are then filtered out via a smaller MLLM.

This automated stage is followed by human verification. Three PhD-level annotators verify validity, difficulty, and correctness for each auto-generated $I$4, referencing both the full image and the crop. The final retained set comprises 845 high-veracity QA pairs.

Methodologically, the annotation procedure is tightly aligned with the benchmark’s intended use. The questions originate from micro-cropped evidence, but the final benchmark item is expressed against the full image with explicit spatial grounding. A plausible implication is that this construction reduces ambiguity about where the decisive evidence lies while preserving the full-image retrieval challenge that the benchmark is designed to measure.

4. Dual-view protocol and the zooming gap

ZoomBench’s evaluation protocol is organized around two input conditions. In Global-View, the model answers from the full image only. In Regional-View, the model answers from the micro-cropped region $I$5, specifically the same region used during question generation (Wei et al., 12 Feb 2026). These two settings are intended to separate access to fine-grained evidence from the ability to retrieve that evidence inside the global scene.

The benchmark then defines the "zooming gap" by

$I$6

A large $I$7 implies that the model can recognize the detail when explicitly zoomed but fails to retrieve it in the global context. This metric is central to the benchmark’s interpretation: it is not enough for a model to perform well on crops if its full-image performance remains substantially lower.

Answer scoring proceeds in two stages. Each predicted answer is first checked by exact or symbolic matching, for example via mathruler. If unmatched, a secondary LLM judge issues a binary $I$8 decision. Counting tasks in training use a continuous reward for RL,

$I$9

but evaluation remains strict binary.

The recommended reporting protocol therefore consists of three numbers for any model under test: Global-View accuracy, Regional-View accuracy, and the zooming gap $\{R\}$0. This three-way breakdown isolates "recognition" capability from "retrieval" capability and measures fine-grained perception "zooming" ability.

5. Baselines and empirical profile

ZoomBench reports a set of single-glance baseline results expressed as overall accuracy percentages. The listed baselines are Qwen3-VL-4B at 40.2, Qwen2.5-VL-7B at 42.5, Qwen3-VL-8B at 37.9, GLM-4.5V (108B) at 49.2, Qwen3-VL-235B at 49.1, and Kimi-K2.5 (1T) at 56.3 (Wei et al., 12 Feb 2026). These values establish the benchmark as difficult even for large-capacity models.

The same evaluation reports three "Zooming without Zooming" (ZwZ) models: ZwZ-4B at 55.7, ZwZ-7B at 55.6, and ZwZ-8B at 58.1. The source explicitly notes that, despite much smaller backbone sizes of $\{R\}$1–$\{R\}$2B, these models rival or exceed $\{R\}$3B+ models on overall ZoomBench accuracy. Within the evidence provided, this is the main empirical claim linking the benchmark to Region-to-Image Distillation.

A more diagnostic comparison is the dual-view breakdown:

Model	Global	Regional	Gap
Qwen3-VL-8B	37.9	63.1	25.2
Kimi-K2.5	56.3	73.2	16.9
Gemini-3-Flash	59.3	78.3	19.0
ZwZ-8B	58.1	73.4	15.3

The smallest reported gap is achieved by ZwZ-8B at 15.3, compared with 25.2 for its Qwen-VL-8B base and about 19 for larger or tool-using models. The paper interprets this as evidence that Region-to-Image Distillation internalizes "zooming" expertise into a single pass, boosting global-view acuity on micro-details. On the benchmark’s own terms, the important point is that overall accuracy and $\{R\}$4 can move differently: a model may have strong regional recognition but still exhibit a substantial retrieval deficit in Global-View.

6. Relationship to agentic zooming and benchmark significance

ZoomBench is embedded in a broader comparison between single-glance models and agentic "Thinking-with-Images" systems. The reported average over VStar, HR-4K, HR-8K, and MME-RW-en gives Pixel-Reasoner (7 B) at 71.9, DeepEyes (7 B) at 75.5, Thyme (7 B) at 74.0, SenseNova-MARS (8 B) at 80.4, Qwen3-VL-4B + tool at 77.3, ZwZ-4B at 80.6, and ZwZ-8B at 81.9, with ZwZ-8B described as best among all (Wei et al., 12 Feb 2026).

Within the benchmark’s conceptual framework, these results matter because ZoomBench was expressly introduced to evaluate when explicit test-time zooming is necessary and when its gains can be distilled into a single forward pass. The benchmark’s dual-view design makes that comparison possible: if a model’s Regional-View accuracy is high but Global-View accuracy remains much lower, then explicit zooming still exposes a retrieval bottleneck. If the gap narrows, this suggests that zooming behavior has been internalized rather than outsourced to tool use.

The benchmark therefore addresses a common ambiguity in the evaluation of fine-grained multimodal perception. Strong performance with tools does not, by itself, establish that a model possesses strong single-pass fine-grained retrieval. Conversely, a high regional score does not imply strong full-image perception. ZoomBench’s contribution is to turn that ambiguity into a measurable quantity.

In summary, ZoomBench provides a targeted, high-difficulty VQA benchmark for fine-grained perception, a dual-view protocol with a defined zooming gap metric, and an empirical setting in which region-grounded synthetic data can be assessed for its ability to reduce the fine-grained retrieval bottleneck without costly test-time tool use.