CVSearch: Adaptive Visual Search Framework

• CVSearch is a training-free adaptive visual search framework that leverages an Assess-then-Search workflow to guide high-resolution image perception in multimodal LLMs.
• It integrates visual expert-assisted search and Semantic Guided Adaptive Patching to preserve semantic coherence and detailed object information in large images.
• Dynamic bottom-up search in CVSearch balances accuracy and throughput, yielding significant benchmark improvements on high-resolution datasets.

CVSearch is a training-free adaptive visual search framework for high-resolution image perception in multimodal LLMs (MLLMs). It is designed for settings in which fixed-resolution visual encoders lose critical small objects, text, and fine-grained spatial relations when large images are downsampled. Rather than relying on uniform cropping or exhaustive scanning, CVSearch introduces an Assess-then-Search workflow that first tests whether a global view is sufficient, then invokes visual expert-assisted localization, and finally falls back to semantic-aware scanning only when necessary. In this formulation, high-resolution perception is treated as a failure-aware search problem over semantically coherent regions rather than as a purely architectural scaling problem (Li et al., 22 May 2026).

1. Problem setting and design rationale

CVSearch is motivated by a specific bottleneck in MLLMs: standard visual encoders are built around fixed input sizes such as $336 \times 336$, so large images must either be resized or decomposed. The paper argues that each common remedy introduces a distinct failure mode. Cropping-based methods preserve local detail but often cause semantic fragmentation, because an object may be cut across patch boundaries. High-resolution visual encoders add architectural complexity yet still process globally and may not allocate computation selectively. Existing visual search methods split into two classes: visual expert-assisted search, which is efficient but fragile because it depends on proposal quality, and scan-based search, which guarantees coverage at the cost of computational redundancy and semantic naivety (Li et al., 22 May 2026).

Within that landscape, CVSearch is presented as a hybrid system that aims to retain the efficiency of expert-guided search while recovering the robustness of scanning. The target use cases are those in which tiny or crowded details matter, including attribute recognition, spatial reasoning, text-rich scenes, cluttered natural images, and UAV imagery. A central claim of the method is that high-resolution perception requires not only access to local detail but also region formation that respects semantic coherence, so that object identity and relations are not destroyed by rigid partitioning (Li et al., 22 May 2026).

2. Assess-then-Search as an adaptive controller

The top-level control policy in CVSearch is an Assess-then-Search workflow inspired by human perception. The system first performs a global cognitive assessment to decide whether the current image already contains enough information to answer the query. This is formalized through an information sufficiency score

$c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$

where $M$ is the MLLM, $p_q(Q)$ is a prompt asking whether the current image is sufficient for query $Q$, and $I$ is the image. If $c_q(I)>\tau_q$, the system answers directly; otherwise it activates search (Li et al., 22 May 2026).

The first search stage is visual expert-assisted search. The query is parsed into object concepts

$O=\{o_1,o_2,\cdots,o_m\},$

and a visual expert $E$, specifically SAM 3, is used to obtain proposals and dense visual features,

$E(I,O)\to B_e,H_e,$

where $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$0 denotes candidate bounding boxes and $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$1 denotes expert features. If these proposals sufficiently cover the target, CVSearch crops the region and answers with the MLLM. If the expert fails, the system does not terminate; instead, it uses the extracted features $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$2 to initialize a deeper scan-based search stage. This makes the controller explicitly failure-aware: global insufficiency triggers search, expert failure triggers scanning, and unresolved cases can enter another round using the best candidate region from the previous step (Li et al., 22 May 2026).

A common misconception is that training-free inference policies must be one-shot heuristics. CVSearch is not described that way. Its controller is iterative, and its adaptation occurs through inference-time scheduling rather than parameter updates. The paper therefore positions CVSearch as a search policy layered on top of existing MLLMs rather than as a new trained backbone (Li et al., 22 May 2026).

3. Semantic Guided Adaptive Patching

The core innovation of the scan-based stage is Semantic Guided Adaptive Patching (SGAP), which replaces rigid grid partitioning with semantically coherent region formation. Starting from the expert feature map $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$3, CVSearch over-segments the image into atomic superpixels

$c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$4

using SLIC in feature space. It then builds a region adjacency graph $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$5 over these atoms and applies Agglomerative Clustering constrained by $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$6 to form $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$7 semantic clusters. Each cluster is converted into a bounding-box patch (Li et al., 22 May 2026).

The number of clusters is selected adaptively by minimizing

$c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$8

where $c_q(I)=M(\text{``Yes''}\mid p_q(Q), I),$9 is an overlap penalty on the cluster boxes $M$0, and $M$1 is a silhouette score measuring clustering quality for atomic features $M$2 under labels $M$3. This objective trades off semantic quality against excessive spatial overlap. The intended effect is to mitigate the fragmentation induced by regular grids, especially when objects occupy irregular regions or cross conventional tile boundaries (Li et al., 22 May 2026).

SGAP also computes a Visual Complexity score for each patch,

$M$4

where $M$5 indexes the atoms inside patch $M$6, $M$7 is the atomic feature, and $M$8 is the patch centroid. Patches with $M$9 are pruned. In the paper’s interpretation, low-complexity patches tend to correspond to simple background regions, whereas high-complexity patches are more likely to contain semantically rich detail worth sending to the MLLM (Li et al., 22 May 2026).

4. Dynamic Bottom-Up Search

CVSearch complements SGAP with Dynamic Bottom-Up Search, which changes the traversal order used in hierarchical scanning. Instead of searching top-down from coarse nodes, the method starts from the deepest, most detailed nodes and moves upward if necessary. The paper argues that top-down search is brittle on high-resolution images because the root is too coarse and early branch errors propagate. Bottom-up search is intended to be more robust for tiny targets, where decisive evidence may first appear only at fine spatial scales (Li et al., 22 May 2026).

Each node $p_q(Q)$0 is assigned a priority

$p_q(Q)$1

where $p_q(Q)$2 is visual complexity, $p_q(Q)$3 is existence confidence of the target object, and $p_q(Q)$4 is the best priority among child nodes. If a node has no children, then $p_q(Q)$5. The weights balance semantically rich regions, regions with direct object evidence, and regions supported by strong children. This is a bottom-up evidence accumulation rule rather than a uniform scan schedule (Li et al., 22 May 2026).

The stopping rule is also dynamic. The current threshold begins at $p_q(Q)$6, is relaxed by $p_q(Q)$7 as search proceeds, and is never allowed to drop below $p_q(Q)$8. The paper describes this as a way to remain strict on easier samples while tolerating weaker but still useful evidence on harder ones. Reported hyperparameters include $p_q(Q)$9, $Q$0, $Q$1, $Q$2, $Q$3, $Q$4 for single-object cases, $Q$5 for multi-object cases, and $Q$6, $Q$7, $Q$8 (Li et al., 22 May 2026).

5. Benchmarks, ablations, and efficiency

CVSearch is evaluated on a mix of high-resolution and general multimodal benchmarks. The high-resolution suite includes V* Bench, with average resolution $Q$9, HR-Bench 4K, and HR-Bench 8K. The general suite includes MME-RealWorld-Lite, TreeBench, and FineRS-4K, a UAV-captured small-object dataset with MVQA and OVQA evaluation. The framework is plugged into LLaVA-OV-7B, Qwen2.5-VL-7B, and InternVL2.5-8B, and the paper also studies scalability across smaller models, larger models, and newer generations such as Qwen3-VL (Li et al., 22 May 2026).

The reported improvements are large. For Qwen2.5-VL-7B, V* overall rises from 71.2 to 90.1, HR-Bench 4K overall from 68.8 to 76.6, and HR-Bench 8K overall from 65.3 to 75.6. For LLaVA-OV-7B, V* overall rises from 75.4 to 91.6, HR-Bench 4K overall from 63.0 to 75.6, and HR-Bench 8K overall from 59.8 to 74.8. For InternVL2.5-8B, HR-Bench 4K rises from 66.0 to 77.0, and HR-Bench 8K from 57.4 to 77.6. On general benchmarks, Qwen2.5-VL-7B improves on MME-RealWorld-Lite from 42.3 to 46.7 and on TreeBench from 37.0 to 40.7, while LLaVA-OV-7B improves on FineRS-4K OVQA from 49.7 to 61.2 (Li et al., 22 May 2026).

Efficiency is a central part of the argument. On V* with Qwen2.5-VL-7B, throughput in samples per minute is 8.30 for the vanilla model, 3.60 with the SAM 3 expert, 0.68 with ZoomEye, 0.66 with RAP, and 1.02 with CVSearch. On HR-Bench 4K, CVSearch reaches 3.77, compared with 1.29 for ZoomEye and 1.22 for RAP. The paper frames this as a better accuracy-throughput balance than scan-based methods, while still being slower than a vanilla single-pass MLLM (Li et al., 22 May 2026).

Ablations attribute the gains to both major scanning components. The comparison between rigid grid plus top-down search, SGAP plus top-down search, SGAP plus bottom-up search, and the full system shows that semantic patching and bottom-up traversal each contribute, and that the full system is best overall. Additional analyses show that increasing the information sufficiency threshold $I$0 routes more cases into search and generally improves accuracy, while the optimal search depth depends on whether the task involves single or multiple objects (Li et al., 22 May 2026).

6. Relation to adjacent search paradigms and stated limitations

CVSearch belongs to a broader research program that treats perception and retrieval as sequential search rather than one-shot encoding, but it occupies a specific point in that landscape. Visual Active Search formulates geospatial exploration as an MDP over region queries under a fixed budget and learns the policy with reinforcement learning (Sarkar et al., 2022). VSearcher trains a long-horizon multimodal web-search agent that uses text search, image search, and webpage visiting through an SFT-then-RL pipeline (Zhang et al., 3 Mar 2026). Earlier system-oriented search frameworks include content-based video retrieval driven by color descriptors (Bhute et al., 2013) and progressive vehicle search that combines detection, indexing, appearance, plate features, and spatiotemporal context in surveillance networks (Liu et al., 2019). CVSearch differs from all of these in that it is a training-free inference-time controller for high-resolution image perception in MLLMs, not an RL-trained search policy, web agent, or corpus indexer. This suggests a broader convergence around adaptive search, while preserving very different action spaces, supervision regimes, and notions of evidence.

The limitations stated for CVSearch are narrow but important. First, it remains slower than vanilla single-pass MLLMs even though it is faster than scan-based baselines. Second, it is evaluated only with SAM 3 as the visual expert, so the effect of alternative experts is not established. The paper identifies parallelizing independent node inference, using memory-efficient attention, and testing different visual foundation models as future directions (Li et al., 22 May 2026).

Taken together, CVSearch presents high-resolution visual perception as a controlled search problem with three explicit commitments: direct answering should be attempted only when global evidence is sufficient, expert proposals should be exploited when available but not trusted unconditionally, and fallback scanning should preserve semantic coherence while pruning low-value regions. Its significance lies less in a new trained model than in a search policy that reorganizes inference for existing MLLMs under high-resolution constraints (Li et al., 22 May 2026).