Thinking in 360°: Humanoid Visual Search in the Wild (2511.20351v2)

Abstract: Humans rely on the synergistic control of head (cephalomotor) and eye (oculomotor) to efficiently search for visual information in 360°. However, prior approaches to visual search are limited to a static image, neglecting the physical embodiment and its interaction with the 3D world. How can we develop embodied visual search agents as efficient as humans while bypassing the constraints imposed by real-world hardware? To this end, we propose humanoid visual search where a humanoid agent actively rotates its head to search for objects or paths in an immersive world represented by a 360° panoramic image. To study visual search in visually-crowded real-world scenarios, we build H* Bench, a new benchmark that moves beyond household scenes to challenging in-the-wild scenes that necessitate advanced visual-spatial reasoning capabilities, such as transportation hubs, large-scale retail spaces, urban streets, and public institutions. Our experiments first reveal that even top-tier proprietary models falter, achieving only ~30% success in object and path search. We then use post-training techniques to enhance the open-source Qwen2.5-VL, increasing its success rate by over threefold for both object search (14.83% to 47.38%) and path search (6.44% to 24.94%). Notably, the lower ceiling of path search reveals its inherent difficulty, which we attribute to the demand for sophisticated spatial commonsense. Our results not only show a promising path forward but also quantify the immense challenge that remains in building MLLM agents that can be seamlessly integrated into everyday human life.

• The paper introduces a novel computational framework that uses a 360° panoramic simulator to enable humanoid agents to perform discrete head rotations as motor actions.
• It demonstrates that supervised fine-tuning significantly improves object and path search success, while reinforcement learning refines long-horizon exploratory strategies.
• The study’s H*Bench benchmark, comprising ~3,000 diverse task instances, highlights remaining challenges in spatial, physical, and socio-spatial commonsense reasoning.

Humanoid Visual Search in the Wild: A Technical Evaluation of "Thinking in 360°" (2511.20351)

Motivation and Problem Statement

This paper establishes a rigorous computational framework for embodied visual search in 360° panoramic environments, motivated by the comparative limitations of state-of-the-art Multimodal LLMs (MLLMs) relative to human spatial search. Existing methods confine reasoning to static, disembodied images, lacking the critical perception–action loop needed for “active” search in visually complex, real-world scenes such as transportation hubs and retail spaces. The research addresses both the technical gap—efficiency, scale, and annotation in realistic 3D environments—and the methodological leap: empowering MLLMs to reason, act, and plan head rotations as discrete motor actions that seek information for goal-oriented tasks.

Framework and Benchmark Design

Scalable Hardware-Free Paradigm

A central technical contribution is the paradigm shift enabled by treating a single high-resolution 360° panorama as a lightweight simulator for head rotation. At each timestep, the agent receives a perspective image defined by its current azimuth ($\phi$) and polar angle ($\gamma$), samples the next view by rotating the head, and ultimately submits a direction as a solution. This closes the perception–action loop, abstracting away full-body locomotion and hardware constraints while maintaining critical decision points of embodied search.

Figure 1: Two-stage pipeline: SFT trains mapping of perspective images to actions; RL refines this into exploratory and strategic visual search policies.

H$^*$Bench: Systematic Data Acquisition

The authors introduce H$^*$Bench, a benchmark of ~3,000 task instances annotated across 6 major scene categories and 18 fine-grained types spanning global metropolitan locations. Each panoramic scene hosts tasks for:

• Humanoid Object Search (HOS): Locate a specified object and foveate it in perspective.
• Humanoid Path Search (HPS): Identify a navigable path to a destination, requiring comprehension of spatial and social constraints.

Task annotation leverages human-in-the-loop Chain-of-Thought (CoT) rationales, boosting dataset quality and interpretability.

Figure 2: Data diversity: scene/type distribution, object instance histograms, and task difficulty taxonomies for HOS/HPS.

Model Training, Post-Training and Evaluation Protocols

Model Architecture and Action Space

A tool-augmented MLLM (Qwen2.5-VL) is finetuned for multi-image input and sequence prediction. The agent policy $\pi_\theta (y_t, a_t | o_t, x, \mathcal{H}_t)$ generates textual thoughts ($y_t$) and actions ($a_t$): discrete head rotations $(\Delta \phi, \Delta \gamma)$, or submission. Each search episode unfolds as a multi-step sequence, allowing for exploration and chain reasoning.

Two-Stage Post-Training

• SFT (Supervised Fine-Tuning): Inculcates behavioral priors for task-oriented planning and action generation from multimodal input. Quantitative jumps in success rates are achieved, especially in establishing basic search capability.
• RL (Reinforcement Learning) via GRPO: Refines exploration strategies, incentivizes long-horizon reasoning, and optimizes policies for reward-aligned search actions. However, the RL impact is task-dependent, with gains for simpler object search and mixed or negative trends for complex path search due to reward-hacking and imperfect reward shaping.

Experimental Analysis

Main Results and Model Comparisons

Extensive benchmarking shows a marked gap between proprietary models (e.g., Gemini2.5-Pro) and open-source baselines, confirmed across both HOS and HPS tasks.

Figure 3: Success rates for in-task vs. cross-task training/test regimes, revealing transfer synergy between object and path search skillsets.

Strong claims are substantiated:

• Numerical Gains: HVS-3B (SFT+RL) outperforms base model on object search (47.38% vs 14.83%) and markedly improves path search (24.94% vs 6.44%). Notably, SFT is responsible for the dominant portion of these gains.
• Model Scaling: Smaller model variants (e.g., Qwen2.5-VL-3B, Gemma-3-4B) yield superior results compared to larger ones, undermining a presumed correlation between scale and embodied reasoning capability.
• Task Difficulty: Success sharply declines for extreme HPS cases, underscoring bottlenecks in spatial and social commonsense.

Ablation and Error Analysis

Figure 4: Typical failure modes—including vision-action mismatch, lack of physical/socio-spatial commonsense—and Gemma3-4B-it results breakdown.

• Error Types: Agents misinterpret spatial cues, fail to obey physical constraints (e.g., walls, vertical transitions), and exhibit ignorance of implicit social conventions (e.g., pedestrian lanes, restricted zones).
• Reward Shaping: RL reward design boosts easy tasks but degrades performance on harder scenarios, implying the need for robust, context-aware objective functions.
• Active vs. Passive Paradigms: Active embodied search decisively outperforms passive analysis of full panoramas; low distortion and task alignment are key to modeling real human-like strategies.

  Figure 5: Active search cumulative success rates and visual paradigm comparisons.

Practical and Theoretical Implications

This study reveals that while SFT and RL improve low-level perceptual-motor coupling in MLLM agents, cognitive bottlenecks remain in spatial, physical, and socio-spatial commonsense reasoning. Capabilities acquired from passive Internet data are fundamentally non-transferable to embodied 3D search without explicit post-training. The paradigm of using panoramic images as lightweight simulators scales annotation and benchmarking, stimulating future advancements in training generalist agents capable of seamless digital and physical world operation.

Practically, these insights inform robotics, assistive technologies, autonomous vehicles, and AR/VR systems. Theoretically, the work calls for deeper integration of embodied cognition into neural architectures, more nuanced reward functions, and next-generation datasets targeting critical points of spatial decision-making.

Conclusion

This paper advances embodied AI by architecting and benchmarking humanoid visual search agents in the wild, demonstrating that post-training substantially improves task-oriented exploration and grounding but falls short on higher-order commonsense. The scalability and diversity of the H$^*$Bench framework position it as a foundation for future work. Substantial open challenges remain—especially in commonsense, reward design, and cross-modal generalization—indicating fertile ground for ongoing research in humanoid agent reasoning and real-world deployment.