---
title: 'HRBench: Ambiguities and Evaluation Strategies'
url: https://www.emergentmind.com/topics/hrbench
type: topic
---

# HRBench: Ambiguities and Evaluation Strategies

Searching arXiv for recent papers on HRBench to ground the article.
{"queries":["HRBench benchmark arXiv high-resolution visual question answering Wang 2025", "\"HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs\"", "\"High-Resolution Benchmark\" HRBench vision arXiv", "\"HRBench\" multimodal high-resolution benchmark arXiv"],"max_results":10}
Reviewing the search results for the relevant HRBench papers.
HRBench is an overloaded designation in recent arXiv literature. In multimodal evaluation, HRBench usually denotes a high-resolution visual question-answering benchmark with HRBench-4K and HRBench-8K subsets for fine-grained reasoning over ultra-high-resolution images [2605.07817]. Separately, the paper "HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs" defines a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs [2605.28398]. These usages refer to different evaluation objects: the former is a vision-centric benchmark for 4K and 8K image understanding, whereas the latter is a controlled framework for effectiveness-efficiency trade-offs in explicit reasoning-mode selection.

## 1. Name, scope, and disambiguation

The term HRBench appears in two distinct senses in the cited literature.

| Usage | Definition in the literature | Typical setting |
|---|---|---|
| High-resolution HRBench | "High-Resolution Benchmark" with HRBench-4K and HRBench-8K | VLMs answering questions about ultra-high-resolution images |
| Hybrid-reasoning HRBench | Unified evaluation framework for thinking-mode switching in hybrid-reasoning LLMs | LLMs selecting think/no\_think, effort levels, or token budgets |

In the high-resolution sense, HRBench is described as a benchmark for small, spatially precise details in ultra-high-resolution images, with tasks involving position, color, count, or object relationships [2605.07817]. In the hybrid-reasoning sense, HRBench organizes a design space of switching strategies and training regimes for models that expose explicit controls over reasoning effort [2605.28398].

A plausible implication is that unqualified references to "HRBench" are potentially ambiguous in current research writing. In practice, papers often resolve this by writing HRBench-4K and HRBench-8K for the vision benchmark, while reserving the standalone title HRBench for the hybrid-reasoning framework.

## 2. HRBench as a high-resolution multimodal benchmark

In the multimodal literature, HRBench-4K and HRBench-8K are high-resolution visual question-answering benchmarks designed to stress-test fine-grained visual perception and spatial reasoning. One description gives HRBench-4K as images at approximately \(3840\times2160\) resolution and HRBench-8K as images at approximately \(7680\times4320\) resolution, with each sample consisting of a single high-resolution image and a natural-language query whose answer must be generated as grounded text [2605.07817]. Another description states that HRBench-4K uses images whose shorter side is up to 4096 pixels and HRBench-8K up to 8192 pixels, with the vision encoder splitting each image into \(28\times28\)-pixel patches [2606.30288].

The benchmark descriptions in downstream papers emphasize detail-sensitive settings. Reported domains include real-world photographs and diagrams requiring fine-grained spatial reasoning, as well as scientific charts, technical diagrams, remote-sensing tiles, and other detail-sensitive visuals [2605.07817]. Questions target local visual evidence such as data points on plots, small object identity or localization, chart axis values, attributes, and spatial relations [2604.21409].

Some implementations expose additional supervision structure. VisReflect states that each QA example is accompanied by a single bounding-box annotation marking the small region of interest, and that no segmentation masks or keypoint labels are used [2606.30288]. S1-VL describes inference as single-turn by default, but permits optional multi-turn Thinking-with-Images tool calls for up to eight rounds when evaluating its own model on HRBench [2604.21409].

## 3. Evaluation protocol for HRBench-4K and HRBench-8K

Across the cited multimodal papers, the primary metric is answer accuracy. One formulation is

$$
\mathrm{Acc} \;=\; \frac{\#\{\text{correct model answers}\}}{\#\{\text{total test samples}\}} \times 100\%.
$$

This is the scoring rule used in S1-VL's HRBench reporting [2604.21409]. Chain-of-Visual-Thought likewise reports plain accuracy for HR\(_{4K}\) and HR\(_{8K}\), with no separate segmentation or depth-error sub-scores broken out [2511.19418]. VisReflect further states that HRBench is evaluated as a VQA-style task with standard multi-choice accuracy, and that object-detection style mAP or IoU metrics are not applicable [2606.30288].

GazeVLM adds benchmark-time efficiency measurements for high-resolution reasoning. In that paper, the primary metric remains accuracy, but two additional quantities are reported: tokens per trace, defined as the average number of output tokens generated during the model’s entire reasoning trace, and calls per trace, defined as the average number of active-vision interventions such as \(\texttt{<LOOK>}\) or ZOOM [2605.07817]. The same paper notes that an external LLM, gpt-oss-120B, is used to compare generated answers against gold labels, and that no explicit grounding score is reported at benchmark time.

The benchmark is therefore used in two related but not identical evaluation styles. Some papers treat HRBench strictly as an answer-accuracy benchmark; others retain answer accuracy as the core metric but supplement it with trace-level cost indicators.

## 4. Model performance and methodological patterns on HRBench-4K and HRBench-8K

Recent multimodal work uses HRBench to evaluate different mechanisms for preserving or recovering fine-grained visual evidence in long or high-resolution contexts. These mechanisms include continuous visual tokens, internal attention control, multi-turn image manipulation, and latent visual reflection.

| Method | Reported HRBench result | Core mechanism |
|---|---|---|
| CoVT | 72.9 / 69.4 or 72.5 / 69.9 on HR\(_{4K}\) / HR\(_{8K}\) | Continuous visual tokens for segmentation, depth, edge, and DINO cues |
| GazeVLM | 83.4 / 78.0 | \(\texttt{<LOOK>}\)-driven suppression bias and SFT+GRPO |
| S1-VL-32B-RL | 91.38 / 93.50 | Thinking-with-Images via Python tool calls plus RL |
| VisReflect | 73.8 / 70.1 | Continuous latent visual reflection in a single forward pass |

Chain-of-Visual-Thought augments a VLM with a small set of continuous visual tokens. In the main setups, segmentation, depth, edge, and DINO tokens total \(N=20\) in the 4-token variant and \(N=16\) in the 3-token variant; the model is trained to autoregressively predict these visual tokens and reconstruct dense supervision signals such as depth, segmentation, edges, and DINO features [2511.19418]. On HRBench, the baseline Qwen2.5-VL is reported at 68.6% on HR\(_{4K}\) and 64.9% on HR\(_{8K}\); CoVT with Seg+Depth+DINO reaches 72.9% and 69.4%, while CoVT with Edge added reaches 72.5% and 69.9%. The same paper reports single-token ablations in which Seg-only and Depth-only both achieve 71.9% on HR\(_{4K}\), while Depth-only reaches 69.4% on HR\(_{8K}\).

GazeVLM proposes an internal active-vision primitive in which the model emits \(\texttt{<LOOK bbox=[x_1,y_1,x_2,y_2]>}\) and applies a continuous suppression bias over pre-encoded visual tokens. Its reported HRBench accuracy is 83.4% on HRBench-4k and 78.0% on HRBench-8k, versus 79.5% and 73.6% for the base Qwen3-VL-4B [2605.07817]. The same paper reports computational efficiency relative to DeepEyes-ZOOM: 265.98 / 292.91 tokens per trace and 1.19 / 1.19 calls per trace for GazeVLM-\(\texttt{<LOOK>}\), compared with 1325.33 / 1268.10 tokens per trace and 1.81 / 1.71 calls per trace for DeepEyes-ZOOM.

S1-VL frames HRBench as one of the "Thinking-with-Images" benchmarks. Its model interleaves textual reasoning with Python code execution in a stateful Jupyter sandbox, generating `<tool_call>` code to crop, zoom, or annotate the current image and then continuing reasoning over the returned intermediate image [2604.21409]. The paper reports 91.38% on HRBench-4K and 93.50% on HRBench-8K for S1-VL-32B-RL, compared with 85.00% and 85.10% for S1-VL-32B-SFT, 83.00% and 80.40% for Qwen3-VL-235B-A22B-Thinking, and 83.90% and 81.50% for Gemini 2.5 Pro.

VisReflect attributes HRBench difficulty to the visual attention sink phenomenon in long visual contexts and addresses it with continuous "visual reflection" embeddings that emphasize question-relevant visual features in latent space [2606.30288]. It reports 73.8% on HRBench-4K and 70.1% on HRBench-8K, compared with 70.9% and 66.9% for the Qwen2.5-VL backbone. The same paper reports an ablation in which cosine alignment yields 73.8 / 70.1, while MSE yields 72.8 / 69.3.

Across these papers, the recurring technical pattern is that high HRBench performance is associated with explicit mechanisms for retaining dense local evidence without relying exclusively on a single global visual encoding. This suggests that HRBench-4K and HRBench-8K are functioning as stress tests for the interaction between visual tokenization, attention allocation, and reasoning policy.

## 5. HRBench as a framework for thinking-mode switch strategies

The paper titled "HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs" uses the same name for a different benchmarking object [2605.28398]. Here, HRBench is a unified evaluation framework for hybrid-reasoning LLMs that expose an explicit thinking-mode control \(m\in\mathcal{M}\), where \(\mathcal{M}\) may be binary \(\{\text{think}, \text{no\_think}\}\), discrete effort levels such as \(\{\text{low}, \text{mid}, \text{high}\}\), or a continuous token budget.

The framework formalizes the cost-accuracy trade-off through average token cost \(C(m)\) and accuracy \(A(m)\), and studies switching policies \(\pi_{\text{switch}}\) that map a query \(q\) to a mode \(m\). It organizes the design space along three strategy families: Prompt-Tuning, in which the LLM self-decides the mode in one pass; Routing, in which a separate router \(\pi_\psi\) selects the mode before generation; and Speculative execution, in which a fast mode is attempted first and deeper reasoning is triggered only when uncertainty is detected [2605.28398].

HRBench further crosses these strategy families with four training regimes: Training-Free, Supervised Fine-Tuning, Offline RL through Direct Preference Optimization, and Online RL through GRPO. Its evaluation spans 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, for a total of 1,261 problems. The primary metrics are Pass@1 accuracy and average output tokens including chain-of-thought. Token reduction relative to Full-Think is defined as

$$
{\rm Red}\% = \frac{\text{Tok}_{\rm FT}-\text{Tok}_{\rm method}}{\text{Tok}_{\rm FT}} \times 100\%.
$$

This HRBench is therefore not a visual benchmark. It is a controlled platform for comparing adaptive reasoning-effort policies under matched models, datasets, and implementation assumptions.

## 6. Findings, limitations, and research significance

For the hybrid-reasoning framework, the central empirical claim is that different switching strategies occupy distinct effectiveness-efficiency trade-off regions. On Qwen3.5-9B in the training-free setting averaged over five benchmarks, Prompt-Tuning is reported at \(+6.7\%\) accuracy change and \(+24\%\) token reduction, Routing at \(+3.2\%\) and \(+13\%\), and Speculative execution at \(+4.9\%\) and \(-11\%\) [2605.28398]. The same study reports that the preferred strategy varies with model scale and task domain, that DPO is best for accuracy, that GRPO is best for efficiency, and that no single external method dominates across all tasks.

For the high-resolution multimodal benchmark, the cited papers converge on a different set of limitations. Chain-of-Visual-Thought states that HRBench is not broken into sub-tasks in its reporting and that no p-values or significance tests are reported [2511.19418]. GazeVLM notes that no explicit grounding score is reported at benchmark time, even though grounding quality is enforced during GRPO training [2605.07817]. VisReflect states that exact dataset counts are provided in the original HRBench release rather than in its own paper [2606.30288]. S1-VL likewise delegates exact data provenance and sample counts to the original HRBench reference [2604.21409].

These limitations matter because the benchmark is repeatedly used to support claims about dense spatial reasoning, geometric awareness, active attention control, and image-manipulation policies. The absence of a uniformly reproduced public description in downstream papers means that HRBench-4K and HRBench-8K are often most precisely understood through the way they are operationalized in model-specific evaluation sections. A plausible implication is that future citations should disambiguate the benchmark name, specify whether HRBench refers to the 4K/8K multimodal benchmark or the hybrid-reasoning switching framework, and report the exact evaluation protocol used in each case.

Source: https://www.emergentmind.com/topics/hrbench