IMEB: Multi-Entity Benchmark for Search

• IMEB is a human-curated benchmark that defines multi-entity multimodal search with concurrent grounding and external retrieval.
• It evaluates agents on both answer accuracy and interaction efficiency using metrics like tool-call turns and cost-aware scores.
• Its construction involves expert authoring, double-blind peer review, and automated tool-necessity filtering to ensure robust, multi-object retrieval challenges.

Searching arXiv for the cited papers to ground the article in the primary sources. IMEB denotes the Image Multi-Entity Benchmark introduced in “HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents” (Li et al., 8 May 2026). It is a human-curated benchmark for multimodal search agents that jointly evaluates answer correctness and interaction efficiency in image-based queries requiring external retrieval over multiple grounded entities. Its defining premise is that conventional multimodal search evaluation is incomplete when it measures only final-answer accuracy, because real-world visual questions often decompose into several independent sub-retrievals whose sequential execution induces redundant rounds, inflated latency, and compounding retrieval noise. IMEB is therefore constructed so that each instance requires concurrent localization and retrieval across multiple entities rather than single-hop or chain-style search (Li et al., 8 May 2026).

1. Definition, motivation, and scope

IMEB was introduced to address three omissions in prior multimodal search evaluation: the neglect of interaction cost, the omission of inference latency, and the lack of tests for parallel, multi-entity retrieval (Li et al., 8 May 2026). The benchmark targets settings in which a question references several distinct objects, such as people, animals, landmarks, or products, and cannot be resolved from the image pixels alone. In such settings, a serial “crop-then-search” workflow issues one tool call per entity and typically accumulates redundant rounds.

The benchmark contains 300 human-curated instances, each pairing a single image with a question that references on average 4.6 distinct entities. Its images are drawn from a wide variety of real-world photographs, including group shots, products on shelves, and animals in herds, so that no single object class dominates (Li et al., 8 May 2026). The benchmark’s core requirement is stringent: every question in IMEB cannot be answered correctly via single-hop or chain-style search; it strictly requires the agent to ground multiple bounding boxes in parallel and then retrieve or verify attributes for each.

This design makes efficiency a first-class evaluation axis alongside correctness. A plausible implication is that IMEB is not merely a harder accuracy benchmark; it is a benchmark for a different operational regime, namely parallel grounded retrieval under tool-use constraints.

2. Construction protocol

IMEB was assembled in three successive phases, each intended to enforce correctness, ambiguity-free formulation, and “tool-necessity” (Li et al., 8 May 2026).

First, in the image and question authoring phase, five PhD-level annotators scoured the web for images containing multiple plausible target entities. They then authored questions whose answers depended on external knowledge, such as biographical facts or object metadata, and could not be inferred from the pixels alone.

Second, in the double-blind peer review phase, each question-image pair was solved independently by other annotators using search tools. Only pairs satisfying three conditions were retained: they had to be unambiguously solvable given correct retrievals, free from conflicting interpretations, and impossible to resolve by sequential single-entity searches. Queries admitting a sequential shortcut were rewritten or dropped.

Third, in the automated tool-necessity filtering phase, every QA pair was passed to Qwen3-VL-235B under a tool-free pass@1 setting. Any question that the model solved correctly without invoking search was eliminated. The surviving 300 instances were therefore designed to uniformly require external, parallel retrieval (Li et al., 8 May 2026).

The resulting protocol combines human curation with automatic filtering. This suggests that IMEB operationalizes “tool necessity” not only as a semantic property of the question, but also as a practical failure mode for a strong parametric model in a tool-free setting.

3. Evaluation protocol and metrics

IMEB evaluates agents in a zero-shot tool-calling setup using three measures: Accuracy (Acc), Average Tool-Call Turns ($N_{\text{tool}}$), and Cost-Aware Score (CAS) (Li et al., 8 May 2026).

Accuracy is judged by an LLM-as-judge using a JSON-structured prompt that compares the agent’s final answer to the ground truth.

Average Tool-Call Turns is the mean number of decoding passes in which the agent emits a <tool_call> block. If multiple tools, such as regions or text queries, are called in a single turn, that still counts as one turn. Formally,

$R = \frac{1}{N}\sum_{i=1}^{N} t_{c,i},$

where $t_{c,i}$ is the number of turns for instance $i$ (Li et al., 8 May 2026).

Cost-Aware Score is a unifying metric defined as

$$\mathrm{CAS} = \frac{\mathrm{Acc}^2 \times 100}{N_{\text{tok}} + 2N_{\text{tool}} + 1}.$$

Here $N_{\text{tok}}$ is the total token consumption in thousands and $N_{\text{tool}}$ is the number of tool-call turns. The squared accuracy term is intended to keep correctness paramount, while the denominator approximates real-world latency using 1 sec per thousand tokens and 1 sec per tool turn (Li et al., 8 May 2026).

The metric design is consequential. Because multiple grounded queries can be dispatched within a single turn, IMEB’s accounting distinguishes between the breadth of search within a round and the number of interaction rounds across time. This makes it particularly suitable for evaluating agents designed around concurrent tool use.

4. Reported results on IMEB

The paper reports Accuracy / Turns under the standard Agentic Workflow for a range of open-source and commercial agents (Li et al., 8 May 2026).

System	IMEB result
Qwen3-VL-30B w/o RL	17.7% / 4.3 turns
Qwen3-VL-235B w/o RL	30.0% / 4.8 turns
DeepEyes-V2 (serial baseline)	18.0% / 4.7 turns
VDR (large serial agent)	21.2% / 12.3 turns
Kimi-K2.5 (commercial)	55.3% / 8.8 turns
Claude-Opus-4.6	41.7% / 3.4 turns
Gemini-3.1-Pro	51.3% / 2.1 turns
HE-30B (SFT only)	42.0% / 3.8 turns; CAS ≈ 0.54
HE-30B (RL fine-tuned)	46.7% / 3.1 turns; CAS = 0.910
HE-235B (SFT)	50.0% / 3.3 turns
HE-235B (RL)	52.7% / 3.0 turns

Several comparisons are explicitly highlighted. HE-30B(RL) achieves +25.5 pp accuracy with −9.2 tool-call rounds relative to the strongest open-source baseline, VDR, on IMEB. Against Gemini-3.1-Pro, HE-235B(RL) narrows the gap to within 1.4 pp accuracy while requiring only ~3 turns versus 2.1 turns for Gemini-3.1-Pro (Li et al., 8 May 2026).

The reported CAS for HE-30B(RL) is 0.910, and the authors state that under CAS it outperforms the next best open-source agent by over 7× on IMEB. This framing emphasizes that the benchmark is designed to expose differences in efficiency-adjusted utility rather than raw correctness alone.

5. What IMEB reveals about multimodal search agents

IMEB is used to contrast Parallel-Grounded Search with Sequential Crop-Then-Search (Li et al., 8 May 2026). The benchmark is described as showing that serial agents inflate turn counts and accumulate distracting retrieval noise, whereas HyperEyes’s Unified Grounded Search action can dispatch all entity bounding boxes and independent text queries in a single turn. On IMEB, this reduces average rounds from ~12 (serial Code-Crop) or ~7 (LLM-Crop) to ~3, while increasing accuracy by 20–30 pp (Li et al., 8 May 2026).

The benchmark also serves to quantify the effect of efficiency-aware RL. The authors report that the Dual-Grained RL framework (TRACE + OPD) tightens turn budgets during training, penalizes redundant rounds and spurious broad parallel calls, and injects token-level corrective signals on failed rollouts. On IMEB, this stage lifts HE-30B from 42.0% to 46.7% accuracy while reducing turns from 3.8 to 3.1 (Li et al., 8 May 2026).

A central analytical claim is that more tool calls do not necessarily yield higher accuracy. IMEB is presented as exposing this phenomenon clearly enough to motivate efficiency-aware objectives in future multimodal search benchmarks. In that sense, the benchmark functions both as an evaluation dataset and as an argument about the inadequacy of accuracy-only assessment.

6. Position within benchmark design and limitations

IMEB is described as the first publicly released 300-instance benchmark that jointly measures both the multimodal search capability and tool-calling efficiency of agents in true multi-entity settings (Li et al., 8 May 2026). Its significance derives from the conjunction of three properties: multi-entity visual grounding, mandatory external retrieval, and explicit efficiency measurement.

The benchmark is nevertheless bounded in scope. The authors note that IMEB is a strong test of static image scenarios, and they identify extending to dynamic or video modalities and further broadening domain diversity as open challenges (Li et al., 8 May 2026). These stated limitations matter because the benchmark’s notion of parallel retrieval is defined over a single image; temporal continuity, streaming evidence, and dynamic scene changes are outside its current formulation.

A plausible implication is that IMEB should be read as a benchmark for a specific class of agentic multimodal search problems rather than as a universal evaluation suite. Its rigor lies in the narrowness of its target behavior: efficient external retrieval over multiple grounded entities within static imagery.

7. Terminological note and disambiguation

IMEB refers here to the Image Multi-Entity Benchmark introduced in HyperEyes (Li et al., 8 May 2026). A plausible source of confusion is the similarly named IME-B, which denotes a different system in a separate domain: a deep multimodal denoising framework for EEG motion artifact removal using IMU signals and a fine-tuned LaBraM backbone (Zhang et al., 1 Sep 2025). The two terms are unrelated in task definition, modality, and evaluation target.

Within multimodal search research, IMEB specifically designates a benchmark whose principal contribution is to make search capability and tool-calling efficiency jointly measurable in true multi-entity settings. Its construction protocol, evaluation metrics, and reported findings are all organized around that objective (Li et al., 8 May 2026).