SearchEyes: Multimodal Deep Search Framework

• SearchEyes is a multimodal search-agent framework that unifies data generation, environment design, and reward construction using a typed knowledge graph.
• It introduces Perception-Knowledge Chains (PKC) to generate constrained multi-hop search trajectories with anti-shortcut measures for reliable evidence synthesis.
• The framework employs Hop-Anchored Policy Optimization (HaPO) for precise step-level RL credit assignment, achieving state-of-the-art results on multimodal benchmarks.

SearchEyes is a multimodal search-agent framework introduced to address a specific systems problem in multimodal deep search: existing pipelines often construct training data, search environments, and reward signals independently, so synthesized structural metadata is discarded, environments depend on irreproducible external engines, and reinforcement learning rewards remain sparse at the trajectory level. In its 2026 formulation, SearchEyes uses a typed knowledge graph as the backbone of a simulated search world, proposes Perception-Knowledge Chains (PKC) for constrained multi-hop sampling, and proposes Hop-Anchored Policy Optimization (HaPO) for step-level credit assignment without a separately trained process reward model. Reported experiments on six multimodal knowledge-intensive benchmarks place SearchEyes at the state of the art among open-source multimodal search agents, with SearchEyes-27B improving over the strongest open-source baseline by 6.2 points on average (Jiao et al., 7 Jul 2026).

1. Search-world simulation paradigm

The core claim of SearchEyes is that multimodal search training benefits from a unified substrate in which data generation, environment design, and reward construction are all derived from the same underlying structure. The framework formalizes that structure as a typed knowledge graph

$G=(V,E,\tau_V,\tau_E),$

where $V$ is the set of entities, $E \subseteq V \times P \times V$ is the set of directed triples, $\tau_V$ assigns each entity a modality type in $\{\text{visual}, \text{knowledge}\}$, and $\tau_E$ assigns each predicate to a semantic domain $$T=\{\text{Person},\text{Work},\text{Org},\text{Geo}\}$$. The graph is partitioned as $V=V_v \cup V_k$, with $V_v$ equipped with high-quality Wikipedia images. Hub nodes with $\deg > d_{\max}$ and blacklist predicates such as instance_of are removed to avoid trivial shortcuts (Jiao et al., 7 Jul 2026).

This design makes the search environment self-contained. SearchEyes constructs its world from Wikidata5M triples, Wiki6M textual descriptions, and Wikipedia images. It also defines a fixed tool layer over that world: text_search(q), lookup(v), visual_search(image, bbox), summarize(o,q), and python_interpreter(code). Retrieval is implemented as hybrid BM25 plus dense embedding fused by reciprocal-rank fusion, and the paper explicitly states that determinism and reproducibility are guaranteed by the self-contained knowledge base with no external APIs (Jiao et al., 7 Jul 2026).

The significance of this formulation is methodological rather than purely architectural. SearchEyes does not treat retrieval, environment interaction, and RL supervision as separate engineering stages. Instead, the same graph structure generates the search space, the intermediate latent targets, and the reward anchors. This suggests a tighter coupling between benchmark construction and policy optimization than in pipelines where evidence retrieval is delegated to an uncontrolled external engine.

2. Simulated environment and tool semantics

Within SearchEyes, the simulated world is not merely a database; it is a controlled search environment with typed transitions and multimodal observables. The document corpus is defined as

$V$0

where $V$1 is the Wikipedia summary from Wiki6M. Search proceeds by invoking tools that expose text, image, or programmatic evidence over this corpus and graph (Jiao et al., 7 Jul 2026).

The operational consequences are important. text_search(q) returns top-$V$2 entities and snippets by hybrid retrieval score. lookup(v) returns the full document for a QID. visual_search(image,bbox) returns top-$V$3 entities and snippets through image-embedding cosine similarity. summarize(o,q) provides a query-focused extract, and python_interpreter(code) provides sandboxed execution. Because all tool calls resolve against the same internal world, the environment is reproducible and does not drift with web freshness, search-engine ranking changes, or API instability (Jiao et al., 7 Jul 2026).

This environment differs structurally from open-web search systems. A plausible implication is that SearchEyes sacrifices some open-world coverage in exchange for strict experimental control, stable reward attribution, and exact replayability. That trade-off is central to its identity: SearchEyes is a simulated-search framework, not a wrapper around a live external search engine.

3. Perception–Knowledge Chains

Perception–Knowledge Chains are the data-construction mechanism that links the graph to search trajectories. A PKC is defined as

$V$4

where each $V$5 is a multimodal entity and each $V$6 is a tool action. The chain alternates between perception hops and knowledge hops: perception hops use visual_search and end in visual entities, whereas knowledge hops use text_search or lookup and end in knowledge entities (Jiao et al., 7 Jul 2026).

The sampling procedure is constrained rather than free-form. SearchEyes samples $V$7 uniformly from visual entities with out-degree at least 2, alternates hop types, enforces treewidth $V$8, requires domain diversity with $V$9 and $E \subseteq V \times P \times V$0, requires at least one one-to-few predicate with an attached disambiguating constraint, and applies anti-shortcut filters including hub removal, blacklist filtering, and answer-anchor deduplication. The chain retains metadata

$E \subseteq V \times P \times V$1

which later becomes the basis for reward anchoring (Jiao et al., 7 Jul 2026).

The paper’s qualitative example is a 4-hop P–K alternating trajectory: visual_search to Guggenheim Museum Bilbao, lookup to architect Frank Gehry, visual_search on the background to city Bilbao as a constraint, and text_search to Pritzker Architecture Prize. That example clarifies the intended behavior: genuine alternation between visually grounded search and symbolic relational search, rather than shortcutting directly to the answer (Jiao et al., 7 Jul 2026).

PKC therefore serves three roles simultaneously. It is a synthetic-data generator, a latent task graph, and a supervision scaffold. The paper argues that free-form synthetic generation with GPT-4o is materially weaker than constrained PKC construction, and the ablation results support that claim.

4. Hop-Anchored Policy Optimization

HaPO is the RL component that turns PKC metadata into step-level credit assignment. The policy is $E \subseteq V \times P \times V$2, parameterized by an MLLM using a ReAct-style decomposition into thought $E \subseteq V \times P \times V$3 and action $E \subseteq V \times P \times V$4, and the value function is $E \subseteq V \times P \times V$5. SearchEyes adapts GRPO by introducing both episode-level and hop-level advantages (Jiao et al., 7 Jul 2026).

The episode reward is binary:

$E \subseteq V \times P \times V$6

The episode-level normalized advantage is

$E \subseteq V \times P \times V$7

For each hop $E \subseteq V \times P \times V$8, the rollout stores the first step $E \subseteq V \times P \times V$9 at which the retrieval observation contains the gold entity $\tau_V$0; those detections define hop anchors. The final token-level advantage is then

$\tau_V$1

with fallback to $\tau_V$2 if the hop is uncaptured (Jiao et al., 7 Jul 2026).

Two additional mechanisms are explicit. First, fatal-aware masking nullifies $\tau_V$3 after $\tau_V$4 consecutive tool errors while one-sided clamping retains positive prefix signals. Second, optimization includes a trust-region constraint through a KL penalty $\tau_V$5. The paper emphasizes that HaPO does not require a separate reward model; it reuses PKC’s gold entity sequence as the source of intermediate supervision (Jiao et al., 7 Jul 2026).

The training protocol reported for the open-source instantiation is concrete. Supervised fine-tuning starts from Qwen3.5-9B for 3 epochs with learning rate $\tau_V$6, batch size 32, a cosine schedule, and masked observations. RL with HaPO uses $\tau_V$7, $\tau_V$8 rollouts, $\tau_V$9, $\{\text{visual}, \text{knowledge}\}$0, $\{\text{visual}, \text{knowledge}\}$1, fatal $\{\text{visual}, \text{knowledge}\}$2, 200 update steps, and learning rate $\{\text{visual}, \text{knowledge}\}$3 (Jiao et al., 7 Jul 2026).

5. Empirical performance and ablations

SearchEyes is evaluated on six multimodal knowledge-intensive benchmarks: SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, and FVQA, plus VisSearch Bench, a 1000-question held-out set of multi-hop perception–knowledge queries. The primary metric is accuracy as judged by GPT-4o, using Exact Match and a secondary substring-match for VisSearch Bench (Jiao et al., 7 Jul 2026).

Setting	SearchEyes result	Reported comparison
Six-benchmark average, SearchEyes-9B	59.3	OpenSearch-VL-30B: 59.8
Six-benchmark average, SearchEyes-27B	68.1	OpenSearch-VL-32B: 61.9
VisSearch Bench, SearchEyes-9B	18.6% EM	OpenSearch-VL-8B: 6.8%
VisSearch Bench, SearchEyes-27B	24.3%	OpenSearch-VL-32B: 9.4%

The ablation studies identify which parts of the framework carry the performance gains. On the PKC side, removing P–K alternation drops average accuracy by 4.2 points; removing anti-shortcut filtering drops it by 7.7; removing the treewidth $\{\text{visual}, \text{knowledge}\}$4 constraint drops it by 2.5; removing the cross-modal filter drops it by 3.3; removing information concealment drops it by 2.5; removing retrieval boost by setting $\{\text{visual}, \text{knowledge}\}$5 drops it by 4.7; and replacing PKC with GPT-4o free-form synthesis drops it by 12.6. On the RL side, full HaPO improves over standard GRPO by 4.0 points on average; removing hop advantage yields only +1.3, removing gating yields +2.7, removing fatal masking yields +2.1, setting $\{\text{visual}, \text{knowledge}\}$6 gives +1.2, and setting $\{\text{visual}, \text{knowledge}\}$7 gives +3.3 (Jiao et al., 7 Jul 2026).

These results support the paper’s central thesis that search performance depends not only on model scale but on structural alignment between training data, environment, and reward. The unusually large penalty from replacing PKC with unconstrained free-form synthesis suggests that controlled chain construction is not an implementation detail but a major source of the observed gains.

6. Relationship to adjacent search literatures

SearchEyes belongs to a broader family of search-oriented systems, but it occupies a distinct niche. Earlier work on human visual search asked whether a person’s target can be inferred from gaze behavior. InferNet, for example, uses “error fixations” on non-target objects, extracts CNN features from those fixations, computes similarity maps over the search image, and infers the target identity in a zero-shot manner without object-specific training on the inference task (Zhang et al., 2018). Other work modeled search scanpaths directly: a unified Bayesian searcher guided by saliency priors reproduced human scanpaths in natural-scene search and argued that saliency maps alone degrade toward chance after the first few fixations (Sclar et al., 2020), while eccNET modeled classical visual-search asymmetry through eccentricity-dependent recognition and target-dependent top-down cues (Gupta et al., 2021).

A separate line of research studies task-constrained search in medical imaging. GazeSearch refines REFLACX and EGD into a finding-aware chest X-ray search benchmark with 2,081 samples, 13 target-present categories, fixation sequences of length up to 7, and a ChestSearch baseline tailored to scan-path prediction in radiology (Pham et al., 2024). In parallel, EEG-plus-eye-tracking systems have been used to infer whether an observer is in a navigational or informational search mode; one such study reports 84.5% leave-one-user-out cross-user accuracy and 85.5% within-user accuracy for intent recognition (Sharma et al., 3 Aug 2025).

SearchEyes also sits near recent work on agentic perception. WebEyes formalizes “Perception Deep Research” in an open-world setting with 120 images, 473 annotated object instances, 645 unique QA pairs, and 1,927 task samples, while Pixel-Searcher combines multi-round web search with grounding, segmentation, and VQA over that benchmark (Yang et al., 12 May 2026). The contrast is instructive: WebEyes studies evidence acquisition from the live web, whereas SearchEyes deliberately replaces external web dependence with a self-contained simulated search world (Yang et al., 12 May 2026, Jiao et al., 7 Jul 2026).

Taken together, these literatures show that “search” spans at least three technical regimes: human gaze and intention decoding, scanpath generation under perceptual constraints, and agentic multimodal reasoning over symbolic and visual evidence. SearchEyes is specifically a framework in the third regime. Its contribution is not eye tracking, saliency prediction, or radiological fixation modeling, but the unification of data synthesis, environment construction, and RL supervision for multimodal deep search intelligence (Jiao et al., 7 Jul 2026).