---
title: 'RSHR-Bench: Ultra-High-Res RS Evaluation'
url: https://www.emergentmind.com/topics/rshr-bench
type: topic
---

# RSHR-Bench: Ultra-High-Res RS Evaluation

RSHR-Bench is a benchmark for evaluating vision–language models (VLMs) and multimodal large language models (MLLMs) on ultra-high-resolution remote sensing (RS) imagery. Developed to address deficiencies in prior RS benchmarks, which typically rely on low-resolution datasets or contain inadequately designed reasoning tasks, RSHR-Bench targets the operational scale encountered in real-world satellite and UAV remote-sensing workflows. This resource consists of thousands of full-scene images maintaining their native extreme resolution and is structured to enable comprehensive assessment of visual understanding, visual reasoning, and scene-level interpretation in challenging RS scenarios [2512.17319].

## 1. Motivation and Benchmark Design

RSHR-Bench was introduced to counter two main limitations observed in prior RS benchmarks: dependence on downsampled (low-resolution) scenes and reasoning tasks solvable by text-only LLMs. The benchmark is designed to evaluate genuine visual understanding on operational-scale RS data, where image sizes routinely exceed 4,000 pixels on the long side with up to approximately \(3 \times 10^8\) native pixels per image. This supports systematic study of VLM and MLLM capabilities in scenarios that demand extensive spatial reasoning, fine-grained object perception, and robust multi-turn dialog with large-scale scenes.

## 2. Corpus Assembly and Properties

The RSHR-Bench image corpus comprises 5,329 full-scene images drawn from six widely used RS sources: DOTA v1.0/v2.0, XLRS-Bench, MiniFrance, FAIR1M, HRSCD, and an in-house UAV collection (with per-frame resolution near 100 MP). All images retain their original resolution, with an average dimension of approximately 8,700 × 8,065 pixels and a maximum up to 29,200 × 27,620 pixels. This ensures coverage across various domains and sensing platforms. Image selection was governed by criteria \( \max(W,H)\ge4000\) and \(W\times H\le3\times10^{8}\), directly reflecting operational usage.

## 3. Task Families and Annotation Protocol

RSHR-Bench defines four distinct task families designed to probe a wide range of perception and reasoning competencies:

1. **Multiple-choice VQA:** 3,864 closed-set questions targeting fine-grained perception and reasoning.
2. **Open-ended VQA:** Reformulated versions of the multiple-choice items into free-form prompts, resulting in 1,932 vetted answer pairs.
3. **Image Captioning:** 3,913 images annotated with both holistic scene summaries and directional (top, bottom, left, right) region descriptions.
4. **Single-image Evaluation:** 50 images (4K–200 MP) each annotated with ten subtasks (totaling 500 human-written question–answer pairs) spanning perception, reasoning, and captioning.

Within VQA, perception tasks are categorized as Color Detection, Shape/Margin Recognition, Orientation Detection, Object Classification, Object Spatial Relationship, Object Grounding, Regional Grounding, Object Counting, and Regional Counting. Reasoning tasks are partitioned into Anomaly Detection/Interpretation, Future Prediction, Multi-Region Joint Contrast (including multi-image and single-image, multi-box), and Object State Judgment. The design accommodates multi-turn dialogue (e.g., sequential anomaly/future-prediction queries) and multi-image fusion for realistic RS evaluation workflows.

## 4. Adversarial Filtering and Human Verification

To mitigate the risk of language-prior exploitation—where a model answers based on textual patterns rather than genuine visual input—a rigorous two-stage adversarial filtering and validation process was employed:

- **Stage 1 (Adversarial Filtering):** Strong text-only LLMs (Qwen3-8B, Llama3-8B) answered each VQA item without images. Items for which models performed too well were revised or discarded.
- **Stage 2 (Human Verification):** Six trained annotators, spending approximately 300 hours, independently produced and audited question–answer pairs. The criteria included correctness, precise visual grounding, avoidance of textual hints, and unambiguous phrasing. Generation incorporated Qwen2.5-VL-7B and GPT-5 Thinking, with provenance tracking and ~100 GPU hours of model calls.

Iterative rewriting reduced text-only answerability to below 30% accuracy while retaining the visual complexity necessary for meaningful RS evaluation.

## 5. Model Evaluation and Performance Analysis

Fourteen representative models were evaluated on RSHR-Bench, including remote-sensing-specific VLMs (EarthDial, GeoChat, GeoLLaVA-8K, VHM), open-source VLMs (InternVL, MiniCPM2, Phi-Vision, Qwen2.5-VL, DeepSeek-VL, VILA-HD), closed-source MLLMs (GPT-5, GPT-4o, GPT-4o mini, Gemini-2.5-pro), and text-only LLMs (Llama3-8B, Qwen3-8B):

| Task                      | Open-source VLMs | Closed-source VLMs | Text-only LLMs |
|---------------------------|------------------|--------------------|----------------|
| Multiple-choice VQA acc.  | ~25%             | up to ~50%         | --             |
| Open-ended VQA (perception/reasoning) | <50% / 40–60% | <50% / 40–60%   | 30–40% (reasoning) |
| Captioning (BLEU-4)       | ≤5               | ≤5                 | --             |
| Captioning (METEOR/ROUGE-L) | ≤35           | ≤35                | --             |
| Single-image eval. acc.   | ≈30% (4K–8K px)  | degrades further at 100–200 MP | -- |

Observed performance gaps are substantial across perception and reasoning tasks, with open-source models typically clustering at chance-level accuracy and closed-source systems peaking near 50%. Captioning scores confirm limited scene-understanding. Notably, text-only LLMs achieved 30–40% on reasoning tasks in the absence of images, underscoring the effectiveness of the adversarial filtering pipeline and revealing a persistent challenge in visual grounding.

## 6. Significance and Availability

RSHR-Bench establishes a rigorous, high-resolution testbed for remote-sensing visual understanding, supporting a spectrum of tasks from perception to complex reasoning, and both multi-turn and multi-image dialog contexts. The benchmark’s emphasis on operational-scale images and adversarially filtered, human-validated queries addresses critical mismatches identified in prior benchmarks. All code, datasets, prompts, and evaluation scripts are publicly disseminated at https://github.com/Yunkaidang/RSHR, thereby providing an extensible platform for future developments in RS VLM and MLLM research [2512.17319].

## 7. Implications for Future Research

The persistent model shortfalls observed on RSHR-Bench, particularly under ultra-high-resolution and complex reasoning conditions, highlight the unsolved challenge of visually grounded intelligence at industrial RS scales. A plausible implication is the necessity for new architectures or representation strategies capable of handling extreme image resolution, robust visual grounding, and reasoning under severe visual diversity. The inclusion of multi-turn and multi-image dialogue tasks indicates future directions in dialog-centric and situational inference paradigms. The strong baseline established by RSHR-Bench is positioned to catalyze targeted advances in multimodal RS intelligence, improved fusion mechanisms, and adversarially robust evaluation methodologies.

Source: https://www.emergentmind.com/topics/rshr-bench