Humanoid Visual Search

• Humanoid visual search is a process where anthropomorphic robots employ active perception to locate targets in complex 3D environments.
• It integrates human visual cognition, deep neural architectures, and spatial mapping to create robust and efficient search strategies.
• Systems leverage hierarchical planners, multimodal large language models, and embodied sensorimotor integration for adaptive, real-time navigation.

Humanoid visual search is the process by which an embodied, anthropomorphic agent—typically a bipedal robot with active perception—locates target objects or persons in complex, real-world 3D environments without exhaustive, pixel-wise enumeration. The field merges principles from human visual cognition, deep neural architectures, and robotic perception/planning to enable robust, efficient, and generalizable search strategies. This article synthesizes mechanistic models, algorithmic pipelines, and contemporary multimodal LLM (MLLM) approaches underpinning state-of-the-art humanoid visual search systems, with a focus on technical components, mathematical formulations, and empirical performance in diverse environments.

1. Core Principles and Historical Models

Human visual search is characterized by a blend of parallel and serial processing, feature-based attention, and strong spatial priors. Classical paradigms distinguish “pop-out” detection (parallel, capacity-unlimited) from conjunctive, capacity-limited serial search, measurable via the dependence of response time or accuracy on distractor set size:

• Disjunctive (single-feature) search: $RT(N)\approx b_\text{disj}$, $a_{\text{disj}}\approx 0$.
• Conjunctive (multi-feature) search: $RT(N)=a_\text{conj} N+b_\text{conj}$, $a_{\text{conj}}>0$.

Modern computational models reproduce these behaviors by combining feed-forward feature detectors, top-down target modulation, and spatial attention mechanisms. The Invariant Visual Search Network (IVSN) (Zhang et al., 2018) implements bottom–up deep feature extraction via VGG-16, caches target-exemplar activations, and forms an attention map via convolutional matching. Sequential fixations are selected by a winner-take-all and inhibition-of-return (IOR) scheme, yielding efficient (sub-exhaustive) search and zero-shot generalization to novel objects and poses.

eccNET (Gupta et al., 2021) adds biologically realistic foveation, using eccentricity-dependent pooling to replicate the high-resolution fovea and low-resolution periphery, and applies target-driven attention at multiple hierarchical feature layers. The model explains classical asymmetry phenomena (e.g., “T among L” easier than “L among T”) and demonstrates that both architectural biases and the “developmental diet” (training set statistics) shape emergent search behavior.

2. Algorithmic Components of Humanoid Visual Search

Humanoid visual search systems implement several interacting subsystems:

• Perceptual front end: Processes RGB/Depth or stereo images, foveates the visual input (optionally through a Gaussian or Laplacian pyramid to mimic primate retina (Luzio et al., 16 Apr 2024)), and detects objects or semantic regions through a CNN or transformer-based backbone.
• Top-down attention and saliency: Forms goal-driven attention maps (e.g., by multiplicative alignment of target and scene feature vectors (Yuan et al., 2020), or convolution of target template over search feature maps (Zhang et al., 2018)). Saliency alone is insufficient for class-based or semantic search; hybrid attention is necessary.
• Spatial context and movement generation: 3D semantics and spatial topology are essential for navigation in multi-room environments. Systems employ occupancy-grid SLAM, back-projection of segmented masks into voxel maps, and cluster safe waypoints to define navigable graphs $G = (V, E, R)$ annotated with region-category labels (Fung et al., 27 Nov 2024). For humanoid locomotion, footstep planners replace differential drive, and stability/kinematic constraints are imposed.
• Planner hierarchy and policy learning: Search is decomposed into region-level prioritization (scoring likelihood, proximity, and recency) and waypoint-level path generation (possibly with chain-of-thought-prompts for context-aware traversal). Reinforcement, supervised, or contrastive learning may be used—combined with domain randomization and egocentricity to induce human-like behaviors (Sorokin et al., 2020, Yu et al., 25 Nov 2025).

3. MLLMs and Multimodal Zero-Shot Reasoning

Recent advances leverage multimodal LLMs to fuse language, vision, and environmental context. MLLM-Search (Fung et al., 27 Nov 2024) illustrates a cutting-edge zero-shot framework in which:

• Semantic and waypoint maps are constructed from RGB-D and SLAM.
• Region planners utilize a triad of scores: semantic-likelihood (language- and scenario-informed), proximity (spatial distance), and recency (visit frequency, exponentially discounted), all derived via chain-of-thought MLLM prompting.
• Spatial chain-of-thought (SCoT) prompts guide path selection at the waypoint level, reasoning about local object arrangements and search goals.
• Retrieval-augmented generation injects external schedule or event databases during planning.

This architecture supports real-time, retraining-free adaptation to new environments and robustly generalizes to both simulated and real-world deployments. Experimental ablations highlight the indispensability of each region-planner score and the SCoT mechanism for optimal search-time and path efficiency.

MLLMs also demonstrate emergent human-like visual search phenomena under controlled paradigm assessments: GPT-4o and Claude Sonnet 3.5 manifest pop-out in disjunctive (color/size) search and capacity limits for conjunction, precisely matching psychophysical predictions (Burden et al., 22 Oct 2025). Mechanistic interpretability reveals that disjunctive feature detection is instantiated in early model layers, with binding and natural-scene priors (e.g., light-from-above) arising later.

4. Visual Representation Learning for Person and Object Search

Text-based person search in large-scale unstructured or surveillance scenarios is an archetypal application. The VFE-TPS model (Shen et al., 30 Dec 2024) demonstrates that enhancing the visual encoder with text-guided masked image modeling (TG-MIM) and identity-supervised global feature calibration (IS-GVFC) is critical for overcoming pure CLIP baseline limitations (which suffer from poor local detail encoding and “identity confusion”).

• TG-MIM requires the image encoder to reconstruct masked patches conditioned on the query, aligning local visual details with semantic tokens.
• IS-GVFC sharpens feature clustering for same-identity instances and maximizes margin from others via KL divergence over cosine similarity-based probability assignments.

On public benchmarks, these mechanisms yield state-of-the-art Rank-1 improvements (+1–9 pp vs. prior methods), and ablation studies confirm an additive benefit of each auxiliary task.

5. Integration with Physical Embodiment and 3D Sensing

Deploying visual search algorithms on humanoid robots requires alignment between perception and embodied action:

• Stereo/depth-driven attention: On iCub, the ELAS algorithm provides real-time, robust disparity maps for segmenting and fixating on the closest salient object, with a modular pipeline integrating calibration, disparity, segmentation, 3D triangulation, and gaze control (Pasquale et al., 2015).
• Egocentric control policies: In the approach of Xia et al. (Sorokin et al., 2020), policies are trained in a POMDP setting via soft actor-critic with contrastive unsupervised representation learning (CURL). Realistic head-body coordination and look-and-walk patterns arise, and “online replanning” synthesizes full-body, collision-free humanoid motion from abstract policies. Ablative results demonstrate the necessity of independent head-cam motion and hierarchical planning for high success and efficient paths.
• Active perception with semantic memory: Bayesian semantic maps updated through foveated object detections guide predictive next-fixation selection. Predictive semantic selection decisively outperforms bottom-up saliency, with cumulative performance advantage in early fixations and near-perfect scene coverage with 30 fixations (COCO2017 image set) (Luzio et al., 16 Apr 2024).

6. Embodied Search in 360° and Spatial Commonsense

Humanoid visual search in open, real-world 360° environments presents unique spatial reasoning challenges. The H* Bench and HVS-3B (Yu et al., 25 Nov 2025) characterize performance in urban, retail, and public-institution scenes, where search actions are discrete head rotations within the panoramic sphere. Success rates for top proprietary and open models are subhuman, especially in path search—where interpreting visual cues (signs, arrows) and integrating physical and socio-spatial commonsense (e.g., illegal paths, inaccessible directions) are critical. Multi-turn RL post-training (GRPO) improves performance $3\times$, but plateauing on “hard” or “extreme” path tasks highlights the persistent gap between LLM-guided search and human spatial planning.

7. Open Challenges and Future Directions

Across the literature, several limitations and opportunities recur:

• Capacity limits: Serial attention and feature-binding mechanisms bottleneck conjunctive search in both humans and advanced MLLMs; targeted fine-tuning attenuates but does not remove these constraints (Burden et al., 22 Oct 2025).
• Scene priors: Light-from-above, object co-occurrence, and social/physical norms robustly shape performance; embedding explicit priors or enriching developmental diets remains an open strategy (Gupta et al., 2021).
• Integration of multi-modal cues: Fusing semantic, spatial, and event-based contexts via standard interfaces to the planner is emerging as best practice (Fung et al., 27 Nov 2024).
• Scaling to unconstrained environments: Search in dynamic, populated public spaces (e.g., H* Bench) reveals critical deficits in current models’ spatial commonsense understanding (Yu et al., 25 Nov 2025).

Plausible implications include increasing focus on hierarchical, memory-augmented architectures employing explicit top-down task representations, chain-of-thought or spatial reasoning prompts, hybrid symbolic–subsymbolic fusion, and continuous adaptation via self-supervised or reinforcement signals in the embodied regime. The field is converging on cognitive architectures that unify rapid, low-level pop-out attention, serial binding, strong semantic and scene priors, and whole-body action integration for robust, human-equivalent humanoid visual search.