Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation

Abstract: Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal navigation, successfully doubling the navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.

• The paper introduces Fast-SegSim, achieving 3D-consistent, open-vocabulary segmentation for robotics by fusing multi-view RGB-D data with a 2D Gaussian Splatting scene representation.
• The method employs precise tile intersection and Top-K Hard Selection, markedly enhancing rendering speeds (exceeding 45 FPS) while maintaining high segmentation quality.
• Integration in robotic simulators like Gazebo boosts navigation success rates, underlining the framework's efficacy in sim-to-real transfer and embodied AI applications.

Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation

Introduction and Motivation

Open-vocabulary panoptic reconstruction is fundamental for embodied AI and robotics, enabling the capture of dense geometric and semantic information within 3D environments. However, real-time execution remains a critical bottleneck due to inefficiencies in rendering high-dimensional features per-pixel, which is exacerbated by the computational complexity of conventional neural representations such as NeRF and high-channel feature accumulation inherent to Gaussian Splatting variants. Fast-SegSim directly targets this problem by presenting a highly optimized, end-to-end framework for 3D-consistent, real-time open-vocabulary segmentation, with profound implications for robotic simulation and sim-to-real transfer.

Figure 1: Real-time rendering speed enables robots to acquire realistic sensor observations and segmentation ground-truth, providing a crucial simulation-to-reality bridge for embodied AI.

Fast-SegSim Framework and Pipeline

Fast-SegSim leverages a 2D Gaussian Splatting (2DGS) scene representation, which models environments as explicit surfel-based collections and naturally preserves geometric details. Multi-view RGB-D frames and supervision from high-capacity Vision-LLMs (VLMs) generate dense 2D masks, which are then lifted and fused into a unified Gaussian Feature Field. The semantic and instance features ($\mathbf{f}_{\text{sem}}$, $\mathbf{f}_{\text{ins}}$) are processed through a semantic decoder and further refined using query-guided cross-attention during training, establishing robust instance associations across perspectives while preserving panoptic consistency.

Figure 2: End-to-end training of the Gaussian Feature Field, leveraging multi-view RGB-D streams and VLM-generated 2D supervision, with highly optimized inference via Top-$K$ selection.

Key to the Fast-SegSim pipeline are two technical innovations for rendering acceleration under high-channel settings:

1. Precise Tile Intersection (Snugbox + AccuTile): This module computes the exact screen-space bounds for each surfel projection using a perspective-corrected conic matrix and efficiently reduces redundant rasterization assignments, addressing memory and compute bottlenecks at the raster-setup stage.
2. Top-$K$ Hard Selection: By exploiting geometric sparsity, only the $K$ most contributory Gaussians per pixel are selected (as measured by a composite depth and feature criterion), resulting in orders-of-magnitude reduction in channel-wise feature accumulation cost with negligible segmentation accuracy loss.

3D-Consistent Instance Segmentation

Fast-SegSim introduces a cross-attention-based mechanism for query-guided instance association. Each learnable query token encapsulates both semantic feature priors and 3D spatial distributions, facilitating explicit scene instance separation even in the presence of ambiguous visual cues or occlusions. Assignment between 2D mask-derived pseudo ground truth and model predictions is performed via the Hungarian algorithm, ensuring global label consistency without error propagation associated with multi-stage pipelines.

The system utilizes a compound loss combining geometric, photometric, semantic, and instance-level objectives, enabling efficient end-to-end optimization of all components with respect to panoptic reconstruction fidelity.

Quantitative Performance and Ablation Insights

Empirical analysis across standard benchmarks (ScanNet-V2, ScanNet++) demonstrates superior segmentation quality compared to both NeRF-based and explicit Gaussian baselines. Fast-SegSim achieves panoptic quality (PQ) scores of 74.12 on ScanNet and 77.09 on ScanNet++ at render rates exceeding 45 FPS. Notably, instance recognition scores (RQ) reach 100.0 and 93.6, respectively—indicative of robust instance-level separation under real-time constraints.

Ablation studies clarify the impact of the proposed optimizations. Activation of the AccuTile module alone halves redundant tile assignments and improves render throughput; the Top-$K$ Hard Selection further prunes the number of processed Gaussians per tile with only minor PQ fluctuation (under 1%). The full method achieves segmentation with a mean inference time of 22 ms per frame.

Figure 3: Segmentation outputs and performance metrics before and after Fast-SegSim acceleration highlight substantial speedup with preserved quality.

Robotic Application and Simulation Integration

Fast-SegSim is directly integrated into robotic simulation platforms—specifically Gazebo—to provide photo-realistic, multi-modal sensor streams (RGB, depth, panoptic segmentation) at low-latency. This capability is leveraged as both a simulation sensor and as a generator of accurate, 3D-consistent supervisory labels.

In object-goal navigation tasks, fine-tuning the downstream perception module with Fast-SegSim-provided labels doubles navigation task success rates compared to unimproved modules, underscoring the sim-to-real transfer efficacy enabled by 3D-consistent ground-truth generation. The real-time capacity of the system reliably satisfies the perception-action loops necessary for closed-loop robot operation in dynamic contexts.

Figure 4: Real-time multi-modal view synthesis in Gazebo, utilizing Fast-SegSim outputs for high-fidelity, latency-sensitive robot sensor emulation.

Discussion and Implications

Fast-SegSim demonstrates that precise pipeline optimization for high-channel 2DGS rendering is both tractable and pragmatically valuable. The explicit decoupling of rasterization and feature accumulation bottlenecks, alongside rigorous instance association, enables simultaneous improvements in both speed and reconstruction consistency. This framework closes a critical gap for real-time 3D perception in robotics, where previous methods either compromised on throughput or quality.

The architectural pattern—i.e., leveraging Top-$K$ geometric selection with rigorous panoptic supervisory signals—generalizes to other high-dimensional dense prediction settings. It invites future work on adaptive, content-aware selection algorithms, cross-modal integration (e.g., point cloud and event-based sensor fusion), and inference-time optimizations for edge-robotics deployments.

Conclusion

Fast-SegSim establishes a scalable, real-time solution for open-vocabulary 3D panoptic reconstruction, introducing technical contributions that realize explicit geometric and feature efficiency without sacrificing panoptic segmentation quality. The framework advances both theoretical understanding of high-dimensional explicit rendering and provides tangible gains in simulation-based robotics, catalyzing robust sim-to-real transfer and high-frequency embodied AI research (2604.10951).