Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

Abstract: Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.

• The paper introduces Stream3Dv2, a method that combines geometric and semantic information for accurate, real-time 3D scene understanding with streaming constraints and noisy 2D masks.
• Stream3Dv2 achieves state-of-the-art streaming zero-shot 3D instance segmentation, with a 27.1% class-agnostic AP and 20.6% semantic AP on ScanNet200, out-performing offline and streaming methods.
• The framework's unique architecture implements geometric-semantic fusion with exact semantic partitioning and efficient temporal updates, reducing historical mask management complexity and improving overall speed

Problem setting and motivation

Stream3Dv2 addresses open-vocabulary 3D scene understanding under two simultaneous constraints that most existing vision-foundation-model (VFM) pipelines fail to satisfy: streaming perception and robustness to noisy 2D masks. The paper formalizes streaming zero-shot 3D instance segmentation (SZ3DS) as a recursion in which, at each frame $t$, the method observes only the current RGB-D pair and incrementally updates segmentation over the partially reconstructed scene, in contrast to full-sequence zero-shot methods (OVIR-3D, MaskClustering, SAI3D, MV3DIS, etc.) that require the globally reconstructed point cloud. Two failure modes motivate the design. First, naively rerunning a full-sequence method per frame incurs cost growing with the accumulated point cloud, precluding timely responses. Second, masks produced by 2D VFMs are inherently noisy—dynamically inconsistent across frames, over-segmented on textured surfaces, or semantically erroneous under lighting artifacts—and in a streaming setting these errors cannot be corrected retroactively from a global view, so they accumulate.

The work extends the authors' prior conference system Stream3D [(2608.21136)'s companion, Xu & Zhao, CVPR Findings], which was purely geometric and Euclidean-distance based. Three specific limitations are identified in the predecessor: absence of semantic guidance during local mask merging, reliance on Euclidean metrics for boundary refinement, and dependence on manually tuned soft-feature similarity thresholds.

Framework architecture

Stream3Dv2 decomposes SZ3DS into two nested sub-tasks over a sliding window of $k=20$ frames: a local multi-view stage $\mathcal{F}_{\text{Loc}}$ producing robust local segmentation, and a local-to-historical stage $\mathcal{F}_{\text{Loc2his}}$ merging local results into the historical mask pool. A key input design choice is dual prompting of 2D segmentation: SAM2 with dense grid prompts yields class-agnostic masks with full coverage but coarse boundaries and noise; SAM3 with semantic prompts yields precise boundaries and class labels but low scene coverage. Rather than intersecting these complementary mask sets—which the authors show fails—the framework fuses them through explicit optimization formulations.

Local multi-view coarse-grained segmentation

Three steps produce coarse class-agnostic 3D masks. Geometry-based single-view denoising enforces intra-instance continuity and inter-instance separation via a distance threshold $\delta$, implemented with DBSCAN and Open3D range queries. SCP-based multi-view filtering selects key points by farthest point sampling ($\gamma = 0.05$) and solves a set covering problem (greedy approximation, since SCP is NP-hard) to retain the minimal set of masks covering all key points, discarding redundant or spurious cross-view masks. IoU-based key mask merging builds a connectivity graph over key masks at overlap threshold $\alpha = 0.2$ and merges each connected component into one coarse mask via its reachability matrix.

Semantic-driven fine-grained segmentation

This is the principal semantic contribution. Per-point label voting over semantic masks resolves ambiguity, after which an integer program—formulated as a set partitioning problem (SPP)—decides, for each coarse mask, whether to decompose it into semantic sub-parts ($z_g=1$), and which disjoint semantic masks to promote as novel instances ($w_s=1$). Because semantic masks are made mutually disjoint and assignments $g^*(s)$ are deterministic, the objective separates and admits a closed-form globally optimal solution, avoiding any iterative solver. This is a notable property: the semantic fusion is exact rather than heuristic, and requires no similarity-threshold tuning on high-dimensional CLIP features, directly addressing limitation (iii) of Stream3D.

Point cloud manifold refinement

To assign points not covered by any mask, the method constructs a super-point graph over voxelized nodes whose edges carry a symmetric Mahalanobis weight using per-node inverse covariance (Riemannian metric). This metric penalizes off-surface displacement while keeping tangential motion cheap, so shortest paths follow intrinsic surface geometry—an explicit remedy for the boundary delineation failures of Euclidean metrics in Stream3D. Assignment is cast as competing geodesic distance fields satisfying discrete Eikonal equations, solved by Bellman relaxation with $k=20$0 iterations. The appendix provides a convergence proof: the relaxation operator is monotone and deflationary, bounded below by the true geodesic solution, and converges exactly within hop-diameter iterations (empirically $k=20$1), so $k=20$2 suffices. Complexity analysis shows total refinement cost is dominated by $k=20$3 nearest-neighbor queries already incurred elsewhere, i.e., near-linear overhead.

Local-to-historical update

Historical masks are indexed by axis-aligned bounding boxes; only AABB-intersecting candidates undergo point-level intersection tests. Historical candidates sharing points with a local mask are "dynamic" and updated by union with the maximally overlapping local mask while being subtracted from all others; non-overlapping local masks become new instances. Detection-based localization reduces per-frame localization time by up to 48.8% on MatterPort3D relative to exhaustive comparison, with asymptotic savings of roughly a factor $k=20$4.

Experimental results

Evaluation covers ScanNet200, ScanNet++, and MatterPort3D using AP, AP$k=20$5, AP$k=20$6 for both class-agnostic and semantic instance segmentation, plus detection on ScanNet200. The headline claims are strong:

• On ScanNet200, Stream3Dv2 reaches 27.1% class-agnostic AP and 20.6% semantic AP in a streaming, training-free setting, exceeding the offline SOTA zero-shot method MV3DIS (35.5/15.5 class-agnostic vs. 20.6 semantic AP) on semantic AP despite MV3DIS seeing the full sequence.
• Naive streaming baselines built by applying the Loc2his module to OVIR-3D and MaskClustering achieve only 8.5% and 7.5% semantic AP, underscoring the difficulty of the combined constraint set.
• Against dedicated streaming methods (EmbodiedSAM, OnlineAnySeg, MoonSeg3R, SAM3D), Stream3Dv2 attains the best accuracy at comparable throughput (11 FPS).
• Gains over Stream3D are substantial: +4.7/+2.7/+3.8 class-agnostic AP and +7.6/+6.9/+5.9 semantic AP across the three benchmarks. Per-class results show large jumps (office chair 22.6→56.6, bed 26.8→60.0, picture 1.4→35.4 AP$k=20$7), with average semantic AP$k=20$8 rising from 20.2% to 38.3% (+18.1%) on ScanNet200.
• For detection, Stream3Dv2 achieves 14.2 AP$k=20$9 / 24.8 AP$\mathcal{F}_{\text{Loc}}$0, surpassing even the non-zero-shot full-sequence Zoo3D$\mathcal{F}_{\text{Loc}}$1 on AP$\mathcal{F}_{\text{Loc}}$2 (24.8 vs. 23.5).

Ablations confirm orthogonality of the three modules (MVF, SDS, PMR); SDS alone lifts semantic AP from 1.7% to 12.8% on ScanNet++, identifying semantic-driven partitioning as the dominant factor. Prompt ablation shows grid-only prompting favors class-agnostic coverage while semantic-only prompting favors semantics, and their combination dominates both. Latency measurements on a long MatterPort3D sequence show local segmentation stable below ~100 ms, though both Loc2his time (91→377 ms from frame 400 to 2800) and memory (10.4→21.3 MB) grow linearly with sequence length—a consequence of the unbounded historical mask pool. Finally, integration with an LLM agent (Codex v0.142.0 / gpt-5.5) demonstrates grounding, reasoning, navigation, and indirect-query capabilities built on the segmented output.

Limitations and open questions

Several constraints deserve plain statement. All benchmarks are indoor; the authors explicitly concede that scaling to unconstrained outdoor environments with dynamic occlusions remains open. The historical mask pool grows linearly in memory and update time, so very long streams may eventually strain the constant-response guarantee implied by the fixed local window. Several hyperparameters ($\mathcal{F}_{\text{Loc}}$3, $\mathcal{F}_{\text{Loc}}$4) are density-dependent and would require adaptation for point clouds with different sampling characteristics, as the parameter analysis indicates stability only within bounded ranges. The evaluation relies on ground-truth camera poses and depth-derived reconstruction; robustness to pose drift or monocular settings (as MoonSeg3R targets) is not examined. The LLM-agent demonstration is qualitative, without quantitative benchmarking of downstream task accuracy.

Conclusion

Stream3Dv2 delivers a training-free, open-vocabulary streaming 3D perception pipeline whose central contributions are the closed-form semantic-driven mask partitioning, manifold-metric-based point refinement with proven finite-step convergence, and bounding-box-accelerated temporal mask updating. It establishes state-of-the-art results among streaming zero-shot methods and competes favorably with offline counterparts, while extending naturally to detection and LLM-driven interaction. Its principal unresolved questions concern outdoor scalability, long-sequence memory growth, and quantitative validation of embodied downstream tasks.