RobotPan: A 360$^\circ$ Surround-View Robotic Vision System for Embodied Perception

Abstract: Surround-view perception is increasingly important for robotic navigation and loco-manipulation, especially in human-in-the-loop settings such as teleoperation, data collection, and emergency takeover. However, current robotic visual interfaces are often limited to narrow forward-facing views, or, when multiple on-board cameras are available, require cumbersome manual switching that interrupts the operator's workflow. Both configurations suffer from motion-induced jitter that causes simulator sickness in head-mounted displays. We introduce a surround-view robotic vision system that combines six cameras with LiDAR to provide full 360$^\circ$ visual coverage, while meeting the geometric and real-time constraints of embodied deployment. We further present \textsc{RobotPan}, a feed-forward framework that predicts \emph{metric-scaled} and \emph{compact} 3D Gaussians from calibrated sparse-view inputs for real-time rendering, reconstruction, and streaming. \textsc{RobotPan} lifts multi-view features into a unified spherical coordinate representation and decodes Gaussians using hierarchical spherical voxel priors, allocating fine resolution near the robot and coarser resolution at larger radii to reduce computational redundancy without sacrificing fidelity. To support long sequences, our online fusion updates dynamic content while preventing unbounded growth in static regions by selectively updating appearance. Finally, we release a multi-sensor dataset tailored to 360$^\circ$ novel view synthesis and metric 3D reconstruction for robotics, covering navigation, manipulation, and locomotion on real platforms. Experiments show that \textsc{RobotPan} achieves competitive quality against prior feed-forward reconstruction and view-synthesis methods while producing substantially fewer Gaussians, enabling practical real-time embodied deployment. Project website: https://robotpan.github.io/

• The paper presents a novel 360° surround-view perception system that integrates six RGB cameras with a 40-beam LiDAR to deliver full spatial coverage for navigation and manipulation.
• It employs a hierarchical spherical Gaussian splatting pipeline with feed-forward transformer architecture and adaptive voxelization to achieve efficient, high-fidelity 3D reconstruction.
• Its streaming fusion mechanism distinguishes dynamic and static regions, ensuring temporal coherence and real-time updates while minimizing computational redundancy.

RobotPan: A 360$^\circ$ Surround-View Robotic Vision System for Embodied Perception

System Architecture and Hardware Design

RobotPan introduces an integrated surround-view robotic vision system, leveraging six outward-facing RGB cameras and a centrally mounted 40-beam LiDAR sensor to achieve full 360$^\circ$ spatial coverage, essential for navigation, manipulation, and teleoperation in complex, dynamic environments. The sensor array is distributed in a precise circular ring at a consistent elevation, offering multi-view visual overlap and omnidirectional sensing with minimal occlusion. The LiDAR, positioned at the crown, complements visual sensors by providing dense metric ground truth for downstream tasks.

Figure 1: RobotPan vision system enables real-time omnidirectional perception with six cameras and a LiDAR for teleoperation and autonomous navigation.

Figure 2: Tiangong 3.0 humanoid robot head layout illustrating sensor placement and fields of view.

Figure 3: Sensor rig and data acquisition setup, matching robot sensor geometry for collection under real-world settings.

Hierarchical Spherical Gaussian Splatting Pipeline

RobotPan employs a feed-forward transformer architecture with alternating view-wise and global attention blocks, built atop DINOv2 feature extraction, to encode time-synchronized multi-view images. The pipeline produces per-view metric-scaled point maps, confidence maps, and dense feature maps, subsequently lifted to a robot-centric spherical coordinate system. Hierarchical voxelization partitions space, allocating fine resolution near the robot and coarser resolution in the far field, supporting adaptive spatial representation.

Voxel anchors aggregate local appearance, geometry, and viewing direction via inverse-distance weighted pooling, followed by per-voxel sparse 3D convolutions for cross-voxel context propagation. Gaussian parameters—center, opacity, covariance, and spherical harmonics—are predicted per anchor, yielding a compact, radius-adaptive set of 3D Gaussians optimized for rendering fidelity and computational efficiency.

Figure 4: RobotPan pipeline: multi-view encoding, depth prediction, spherical voxelization, anchor aggregation, and Gaussian decoding enable real-time rendering and reconstruction.

Streaming Fusion and Dynamic Region Identification

RobotPan introduces a streaming update mechanism for online scene integration. Rather than concatenating frame-wise predictions, which induces severe redundancy and storage inefficiency, the method maintains a persistent shared Gaussian set for static content and adds or updates dynamic-region Gaussians only as needed. Dynamic regions are segmented per view, fused in a panoramic range image, and projected into spherical coordinates to create robust masks for dynamic/static splitting.

To reinforce temporal coherence, previous-frame Gaussians undergo refinement via per-frame tiny-MLP residual correction. Holes in coverage are filled by online instantiation from new frame inputs, and multi-view consistency is enforced through range-image fusion masks. This approach prevents unbounded primitive growth, reduces artifacts at frame boundaries, and supports efficient reconstruction of long unbounded streams.

Figure 5: Multi-view consistent dynamic region identification via range-image fusion for robust streaming updates and scene segmentation.

Experimental Evaluation and Benchmarking

Point Map Estimation

RobotPan demonstrates high geometrical robustness on its robotic benchmark—even under sparse-view, limited-overlap conditions—and generalizes competitively to DTU and ETH3D datasets. Comparisons with Dust3R, Fast3R, FLARE, VGGT, Pi3, MASt3R, and Spann3R show RobotPan achieves strong accuracy, completeness, and overall Chamfer metrics. Qualitatively, RobotPan recovers more complete structures with significant detail preservation in sparse-view reconstructions.

Figure 6: Comparative feed-forward 3D reconstruction across methods, showing RobotPan's improved completeness and edge precision.

Figure 7: RobotPan depth prediction showcases superior boundary definition and geometry consistency versus other approaches.

Generalized Novel View Synthesis

On robotic and public benchmarks (DL3DV-Benchmarks, RealEstate10K), RobotPan achieves the highest PSNR, SSIM, and lowest LPIPS, while using 3–12$\times$ fewer Gaussians than pixel-wise baselines. Its spherical voxel-wise representation is markedly more compact and efficient, facilitating real-time rendering. Qualitative synthesis reveals sharper geometries and fewer artifacts than competing solutions.

Figure 8: Qualitative view synthesis comparison, highlighting RobotPan's sharper volumes and artifact suppression across diverse scenes.

Figure 9: Novel-view renderings on DL3DV and RealEstate10K, with clean boundaries and refined spatial detail using RobotPan.

Streaming and Ablative Analysis

Streaming evaluation over dynamic scenes confirms RobotPan's superiority in rendering quality (28.59 dB PSNR), training efficiency (0.47 s/update), real-time throughput (230 FPS), and storage (7.2 MB), outperforming both offline (Kplanes, 4DGS, Spacetime-GS) and online baselines (StreamRF, 3DGStream, IGS). Ablative experiments demonstrate that spherical voxel-wise prediction achieves optimal trade-offs between compactness and quality compared to pixel-wise and Cartesian voxel paradigms. Removal of multi-view fusion or tiny-MLP refinement degrades results, reaffirming the necessity of these architectural components.

Implications, Applications, and Future Directions

RobotPan advances embodied robotic vision by shifting from forward-centric to omnidirectional, hierarchical perception, directly supporting teleoperation, navigation, and manipulation. Its compact, scalable representation mitigates bandwidth and storage constraints inherent in real-time systems, while streaming fusion ensures adaptability in dynamic environments. The system's reliance on metric-scaled reconstruction via LiDAR supervision, range-image fusion, and spherical voxel hierarchies strengthens its suitability for SLAM, closed-loop control, and high-fidelity mapping tasks. The released dataset further positions RobotPan as a benchmark for future embodied perception research.

Practical applications extend to robotics operating in cluttered and dynamic spaces, emergency response scenarios, and human-robot interaction, where enhanced situational awareness and real-time 3D streaming are critical. Theoretically, the approach provides compelling evidence that hierarchical, anchor-wise 3D scene representations can fundamentally improve the balance among geometric fidelity, efficiency, and adaptability.

Future work may explore further integration with semantic segmentation, upstream task-driven learning (navigation, manipulation), distributed sensor fusion across heterogeneous platforms, and scaling to longer streams and larger environments without loss of compactness or quality. Extension to unposed input streams, self-supervised learning, and integration with large-scale pretraining (VGGT, $\pi^3$) may further improve robustness and generalization.

Conclusion

RobotPan presents a unified solution for surround-view robotic perception, combining a hardware-integrated sensor array with a hierarchical, feed-forward spherical Gaussian splatting architecture and streaming fusion strategy. Its compactness, efficiency, and high rendering quality directly address the demands of real-time embodied deployment and offer a new paradigm for omnidirectional robotic vision (2604.13476).