---
title: 'SearchEyes: Multimodal Deep Search Framework'
url: https://www.emergentmind.com/topics/searcheyes
type: topic
---

# SearchEyes: Multimodal Deep Search Framework

SearchEyes is a multimodal search-agent framework introduced to address a specific systems problem in multimodal deep search: existing pipelines often construct training data, search environments, and reward signals independently, so synthesized structural metadata is discarded, environments depend on irreproducible external engines, and reinforcement learning rewards remain sparse at the trajectory level. In its 2026 formulation, SearchEyes uses a typed knowledge graph as the backbone of a simulated search world, proposes Perception-Knowledge Chains (PKC) for constrained multi-hop sampling, and proposes Hop-Anchored Policy Optimization (HaPO) for step-level credit assignment without a separately trained process reward model. Reported experiments on six multimodal knowledge-intensive benchmarks place SearchEyes at the state of the art among open-source multimodal search agents, with SearchEyes-27B improving over the strongest open-source baseline by 6.2 points on average [2607.05943].

## 1. Search-world simulation paradigm

The core claim of SearchEyes is that multimodal search training benefits from a unified substrate in which data generation, environment design, and reward construction are all derived from the same underlying structure. The framework formalizes that structure as a typed knowledge graph
$$
G=(V,E,\tau_V,\tau_E),
$$
where $V$ is the set of entities, $E \subseteq V \times P \times V$ is the set of directed triples, $\tau_V$ assigns each entity a modality type in $\{\text{visual}, \text{knowledge}\}$, and $\tau_E$ assigns each predicate to a semantic domain $T=\{\text{Person},\text{Work},\text{Org},\text{Geo}\}$. The graph is partitioned as $V=V_v \cup V_k$, with $V_v$ equipped with high-quality Wikipedia images. Hub nodes with $\deg > d_{\max}$ and blacklist predicates such as `instance_of` are removed to avoid trivial shortcuts [2607.05943].

This design makes the search environment self-contained. SearchEyes constructs its world from Wikidata5M triples, Wiki6M textual descriptions, and Wikipedia images. It also defines a fixed tool layer over that world: `text_search(q)`, `lookup(v)`, `visual_search(image, bbox)`, `summarize(o,q)`, and `python_interpreter(code)`. Retrieval is implemented as hybrid BM25 plus dense embedding fused by reciprocal-rank fusion, and the paper explicitly states that determinism and reproducibility are guaranteed by the self-contained knowledge base with no external APIs [2607.05943].

The significance of this formulation is methodological rather than purely architectural. SearchEyes does not treat retrieval, environment interaction, and RL supervision as separate engineering stages. Instead, the same graph structure generates the search space, the intermediate latent targets, and the reward anchors. This suggests a tighter coupling between benchmark construction and policy optimization than in pipelines where evidence retrieval is delegated to an uncontrolled external engine.

## 2. Simulated environment and tool semantics

Within SearchEyes, the simulated world is not merely a database; it is a controlled search environment with typed transitions and multimodal observables. The document corpus is defined as
$$
D=\{\mathrm{doc}(v)\mid v\in V\},
$$
where $\mathrm{doc}(v)$ is the Wikipedia summary from Wiki6M. Search proceeds by invoking tools that expose text, image, or programmatic evidence over this corpus and graph [2607.05943].

The operational consequences are important. `text_search(q)` returns top-$k$ entities and snippets by hybrid retrieval score. `lookup(v)` returns the full document for a QID. `visual_search(image,bbox)` returns top-$k$ entities and snippets through image-embedding cosine similarity. `summarize(o,q)` provides a query-focused extract, and `python_interpreter(code)` provides sandboxed execution. Because all tool calls resolve against the same internal world, the environment is reproducible and does not drift with web freshness, search-engine ranking changes, or API instability [2607.05943].

This environment differs structurally from open-web search systems. A plausible implication is that SearchEyes sacrifices some open-world coverage in exchange for strict experimental control, stable reward attribution, and exact replayability. That trade-off is central to its identity: SearchEyes is a simulated-search framework, not a wrapper around a live external search engine.

## 3. Perception–Knowledge Chains

Perception–Knowledge Chains are the data-construction mechanism that links the graph to search trajectories. A PKC is defined as
$$
P=[(v_0,a_1,v_1),(v_1,a_2,v_2),\ldots,(v_{H-1},a_H,v_H)],
$$
where each $v_i \in V$ is a multimodal entity and each $a_i$ is a tool action. The chain alternates between perception hops and knowledge hops: perception hops use `visual_search` and end in visual entities, whereas knowledge hops use `text_search` or `lookup` and end in knowledge entities [2607.05943].

The sampling procedure is constrained rather than free-form. SearchEyes samples $v_0$ uniformly from visual entities with out-degree at least 2, alternates hop types, enforces treewidth $\leq 2$, requires domain diversity with $|\{\phi(p_i)\}| \geq 3$ and $\phi(p_i)\neq\phi(p_{i+1})$, requires at least one one-to-few predicate with an attached disambiguating constraint, and applies anti-shortcut filters including hub removal, blacklist filtering, and answer-anchor deduplication. The chain retains metadata
$$
e^*=[v_0,v_1,\ldots,v_H],
$$
which later becomes the basis for reward anchoring [2607.05943].

The paper’s qualitative example is a 4-hop P–K alternating trajectory: `visual_search` to Guggenheim Museum Bilbao, `lookup` to architect Frank Gehry, `visual_search` on the background to city Bilbao as a constraint, and `text_search` to Pritzker Architecture Prize. That example clarifies the intended behavior: genuine alternation between visually grounded search and symbolic relational search, rather than shortcutting directly to the answer [2607.05943].

PKC therefore serves three roles simultaneously. It is a synthetic-data generator, a latent task graph, and a supervision scaffold. The paper argues that free-form synthetic generation with GPT-4o is materially weaker than constrained PKC construction, and the ablation results support that claim.

## 4. Hop-Anchored Policy Optimization

HaPO is the RL component that turns PKC metadata into step-level credit assignment. The policy is $\pi_\theta(a_t\mid s_t)$, parameterized by an MLLM using a ReAct-style decomposition into thought $z_t$ and action $a_t$, and the value function is $V_\phi(s_t)$. SearchEyes adapts GRPO by introducing both episode-level and hop-level advantages [2607.05943].

The episode reward is binary:
$$
R^{(i)}=\mathbf{1}_{\text{correct final answer}}.
$$
The episode-level normalized advantage is
$$
\hat A_{\mathrm{ep}}^{(i)}=\frac{R^{(i)}-\mu_R}{\sigma_R+\epsilon}.
$$
For each hop $k$, the rollout stores the first step $t_k^{(i)}$ at which the retrieval observation contains the gold entity $v_k$; those detections define hop anchors. The final token-level advantage is then
$$
A_{\mathrm{final}}^{(i,\ell)}=
\alpha\,\hat A_{\mathrm{ep}}^{(i)}+(1-\alpha)\,\hat A_{\mathrm{hop}}^{(i,\ell)},
$$
with fallback to $\hat A_{\mathrm{ep}}$ if the hop is uncaptured [2607.05943].

Two additional mechanisms are explicit. First, fatal-aware masking nullifies $A_{\mathrm{final}}$ after $M$ consecutive tool errors while one-sided clamping retains positive prefix signals. Second, optimization includes a trust-region constraint through a KL penalty $\lambda D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}})$. The paper emphasizes that HaPO does not require a separate reward model; it reuses PKC’s gold entity sequence as the source of intermediate supervision [2607.05943].

The training protocol reported for the open-source instantiation is concrete. Supervised fine-tuning starts from Qwen3.5-9B for 3 epochs with learning rate $2\times10^{-5}$, batch size 32, a cosine schedule, and masked observations. RL with HaPO uses $\alpha=0.3$, $G=8$ rollouts, $\lambda=0.02$, $\gamma^+=1.0$, $\gamma^-=1.05$, fatal $M=3$, 200 update steps, and learning rate $5\times10^{-7}$ [2607.05943].

## 5. Empirical performance and ablations

SearchEyes is evaluated on six multimodal knowledge-intensive benchmarks: SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, and FVQA, plus VisSearch Bench, a 1000-question held-out set of multi-hop perception–knowledge queries. The primary metric is accuracy as judged by GPT-4o, using Exact Match and a secondary substring-match for VisSearch Bench [2607.05943].

| Setting | SearchEyes result | Reported comparison |
|---|---:|---:|
| Six-benchmark average, SearchEyes-9B | 59.3 | OpenSearch-VL-30B: 59.8 |
| Six-benchmark average, SearchEyes-27B | 68.1 | OpenSearch-VL-32B: 61.9 |
| VisSearch Bench, SearchEyes-9B | 18.6% EM | OpenSearch-VL-8B: 6.8% |
| VisSearch Bench, SearchEyes-27B | 24.3% | OpenSearch-VL-32B: 9.4% |

The ablation studies identify which parts of the framework carry the performance gains. On the PKC side, removing P–K alternation drops average accuracy by 4.2 points; removing anti-shortcut filtering drops it by 7.7; removing the treewidth $\leq 2$ constraint drops it by 2.5; removing the cross-modal filter drops it by 3.3; removing information concealment drops it by 2.5; removing retrieval boost by setting $\beta=1$ drops it by 4.7; and replacing PKC with GPT-4o free-form synthesis drops it by 12.6. On the RL side, full HaPO improves over standard GRPO by 4.0 points on average; removing hop advantage yields only +1.3, removing gating yields +2.7, removing fatal masking yields +2.1, setting $\alpha=1$ gives +1.2, and setting $\alpha=0$ gives +3.3 [2607.05943].

These results support the paper’s central thesis that search performance depends not only on model scale but on structural alignment between training data, environment, and reward. The unusually large penalty from replacing PKC with unconstrained free-form synthesis suggests that controlled chain construction is not an implementation detail but a major source of the observed gains.

## 6. Relationship to adjacent search literatures

SearchEyes belongs to a broader family of search-oriented systems, but it occupies a distinct niche. Earlier work on human visual search asked whether a person’s target can be inferred from gaze behavior. InferNet, for example, uses “error fixations” on non-target objects, extracts CNN features from those fixations, computes similarity maps over the search image, and infers the target identity in a zero-shot manner without object-specific training on the inference task [1807.11926]. Other work modeled search scanpaths directly: a unified Bayesian searcher guided by saliency priors reproduced human scanpaths in natural-scene search and argued that saliency maps alone degrade toward chance after the first few fixations [2009.08373], while eccNET modeled classical visual-search asymmetry through eccentricity-dependent recognition and target-dependent top-down cues [2106.02953].

A separate line of research studies task-constrained search in medical imaging. GazeSearch refines REFLACX and EGD into a finding-aware chest X-ray search benchmark with 2,081 samples, 13 target-present categories, fixation sequences of length up to 7, and a ChestSearch baseline tailored to scan-path prediction in radiology [2411.05780]. In parallel, EEG-plus-eye-tracking systems have been used to infer whether an observer is in a navigational or informational search mode; one such study reports 84.5% leave-one-user-out cross-user accuracy and 85.5% within-user accuracy for intent recognition [2508.01860].

SearchEyes also sits near recent work on agentic perception. WebEyes formalizes “Perception Deep Research” in an open-world setting with 120 images, 473 annotated object instances, 645 unique QA pairs, and 1,927 task samples, while Pixel-Searcher combines multi-round web search with grounding, segmentation, and VQA over that benchmark [2605.12497]. The contrast is instructive: WebEyes studies evidence acquisition from the live web, whereas SearchEyes deliberately replaces external web dependence with a self-contained simulated search world [2605.12497; 2607.05943].

Taken together, these literatures show that “search” spans at least three technical regimes: human gaze and intention decoding, scanpath generation under perceptual constraints, and agentic multimodal reasoning over symbolic and visual evidence. SearchEyes is specifically a framework in the third regime. Its contribution is not eye tracking, saliency prediction, or radiological fixation modeling, but the unification of data synthesis, environment construction, and RL supervision for multimodal deep search intelligence [2607.05943].

Source: https://www.emergentmind.com/topics/searcheyes