GAR-Bench: Evaluating Region-Level MLLMs

• The paper introduces GAR-Bench, a benchmark evaluating region-level multimodal understanding by integrating precise contextual visual cues with RoI-aligned feature replay.
• It employs a multi-region prompting framework to rigorously assess both captioning and visual question answering tasks through advanced relational reasoning protocols.
• Empirical results highlight GAR’s state-of-the-art performance and robust cross-domain transfer from images to video, enabling applications in assistive, security, and robotics domains.

GAR-Bench refers to the suite of evaluation protocols and datasets introduced alongside GAR (Grasp Any Region), a region-level multimodal LLM (MLLM) architecture designed for precise, context-aware comprehension of visual regions in images and videos. GAR-Bench evaluates both single-region understanding and advanced multi-region compositional reasoning, addressing the need for rigorous assessment of models that must leverage both fine-grained local features and global scene context.

1. Motivation and Challenges for Region-Level Visual Understanding

GAR-Bench is motivated by critical limitations in holistic and prior region-level MLLMs. While holistic models capture broad scene content, they frequently fail in dense scenes that require discrimination among closely related objects or detailed region attributes. Existing region-oriented methods typically perceive isolated region crops, omitting contextual dependencies integral to robust understanding—for example, distinguishing a “frog-shaped slipper” from a real frog by leveraging the broader scene. GAR pursues “precise, contextual pixel understanding” by coupling region masks with global context, which GAR-Bench is designed to rigorously evaluate.

2. GAR Architecture: RoI-Aligned Feature Replay and Multi-Prompt Integration

GAR employs a RoI-aligned feature replay mechanism to retain global contextual information. Instead of cropping regions, the model encodes the entire image and replays high-resolution region features using RoI-Align operations, thereby generating context-aware regional representations. A lightweight convolutional encoding of region masks is embedded into ViT patch spaces, allowing simultaneous comparison and interaction of multiple user-defined regions.

The region modeling task in GAR-Bench is formalized as

$R = \text{RegionModel}(I, \{M_i\}_{i=1}^N, T)$

where $I \in \mathbb{R}^{H\times W \times 3}$ is the image, $\{M_i\}$ are $N$ binary region masks, $T$ is an instruction, and $R$ is the model’s text response, which integrates both local region description and relevant global cues.

3. GAR-Bench Composition: Protocols for Comprehensive Evaluation

GAR-Bench is structured to jointly evaluate:

• Captioning: Single-region and multi-region tasks require the model to generate detailed relational descriptions, necessitating inference across both target regions and their surroundings.
• Visual Question Answering (VQA): Divided into basic perception (attributes like color, material, shape, texture) and advanced reasoning (spatial relations, non-entity recognition, ordering) protocols. This dual approach measures both attribute extraction and the ability to model relational reasoning spanning multiple regions.

The benchmark protocols are designed to probe compositional reasoning, systematically increasing difficulty with the number and relational complexity of masked regions.

4. Empirical Results: State-of-the-Art Contextual and Compositional Reasoning

GAR-Bench demonstrates GAR’s efficacy in both captioning and multi-region reasoning. GAR-1B surpasses prior region-level MLLMs (e.g., DAM-3B, InternVL3-78B) on DLC-Bench and GAR-Bench-VQA respectively, with typical overall VQA scores of 50.6 for GAR-1B and 59.9 for zero-shot GAR-8B. In advanced reasoning subtasks, GAR maintains a margin over models with larger parameters but inferior compositional protocols. Zero-shot transfer tests show GAR-8B exceeds the performance of in-domain VideoRefer-7B on VideoRefer-BenchQ, indicating substantial cross-domain generalization from image-trained regional comprehension to video tasks.

5. Technical Principles: RoI-Aligned Contextual Encoding and Multi-Region Prompting

GAR’s fundamental innovation is the RoI-aligned feature replay, which together with the convolutional mask embedding, delivers region features that are explicitly context-aware. The training regimen leverages multi-stage data pipelines, integrating seed datasets and manually fine-grained relational descriptions, enabling robust learning across both basic and complex visual attributes. The prompt encoding architecture is critical to handling multiple region masks, with zero-initialization ensuring stability for newly introduced region-specific channels.

6. Applications and Transfer to Video Domain

GAR’s precise region understanding supports various applications:

• Assistive technology: Detailed descriptions targeted to visually impaired end-users
• Security and surveillance: Multiregion relational reasoning in cluttered or overlapping object scenarios
• Robotics: Scene decomposition and interaction-aware navigation or manipulation
• Video comprehension: Zero-shot transfer to video datasets (VideoRefer-BenchQ) demonstrates practical applicability in temporal domains, though fine-grained motion and temporal sequencing remain future challenges.

The architecture’s compositional representations facilitate transfer to domains beyond static images, with successful application in video despite a lack of explicit temporal training.

7. Future Directions and Identified Limitations

GAR’s current limitations are associated with fine-grained temporal reasoning (in the video extension) and compositional complexity when dealing with very large numbers of interacting prompts (7–9 region assemblies). The paper suggests further architectural refinement, integrating specialized temporal modules, and more diversified real-world region datasets. Cross-domain generalization is promising, but coherent temporal modeling remains an open research avenue.

---

GAR-Bench establishes a new evaluation standard for region-level MLLMs by validating models that integrate both pixel-precise local awareness and global scene context, with demonstrated superiority in both captioning and multi-region reasoning. It is instrumental in assessing practical cross-domain transfer from images to video, and points toward future research in temporally compositional visual reasoning.