Intrinsic 4D Gaussian Segmentation from Scene Cues

Abstract: Dynamic 4D Gaussian Splatting reconstructs deforming scenes with high fidelity and is increasingly adopted as a representation for dynamic 3D scenes. Putting such a scene to use, for editing, manipulation or motion analysis, first requires segmenting it: grouping the Gaussian primitives into coherent objects. Current pipelines obtain this grouping by importing 2D masks from foundation models such as SAM and lifting or distilling them into the Gaussian representation. In dynamic scenes these masks must be generated across many frames and views, which is costly, and the resulting segmentation can depend strongly on the quality and consistency of those external masks. We ask how much object-level structure can instead be recovered from the Gaussians themselves, and propose Intrinsic-GS, a training-free, mask-free method that builds a sparse affinity graph over Gaussian primitives from appearance, orientation, scale, deformation-trajectory and non-learned rendered-boundary cues. The graph is partitioned with Leiden community detection, requiring no foundation model and no learned feature field. On the standard 4D Gaussian segmentation benchmarks, Neu3D and HyperNeRF, Intrinsic-GS recovers substantial object structure without mask supervision, reaching 0.746 mIoU on Neu3D and 0.575 on HyperNeRF; on Neu3D, a geometry-only variant reaches 0.902 mIoU, matching SAM-supervised TRASE. On HyperNeRF, Intrinsic-GS runs 12.5x faster than the mask-generation and feature-rendering stages used by mask-supervised pipelines. These results suggest that much of the segmentation signal is already encoded in the Gaussians themselves, offering a fast, mask-free direction for 3D and 4D Gaussian segmentation that may also point toward more generalizable, robust segmentation in settings where external masks are unreliable or expensive.

• The paper introduces Intrinsic-GS, a mask-free segmentation method that partitions 4D Gaussian primitives using intrinsic geometric, appearance, and motion cues without external mask supervision.
• The method builds a k-NN affinity graph by fusing multi-modal signals and employs Leiden community detection to efficiently segment dynamic scenes at scale.
• Empirical evaluations on Neu3D and HyperNeRF demonstrate competitive mIoU and a 12.5× speedup over mask-supervised pipelines, highlighting both practical advantages and limitations.

Intrinsic 4D Gaussian Segmentation from Scene Cues: Technical Analysis

Motivation and Problem Statement

Scene segmentation in dynamic 4D Gaussian Splatting (4DGS) is critical for downstream tasks such as editing, manipulation, and dynamic analysis. Conventional pipelines rely heavily on 2D mask supervision from foundation models (e.g., SAM), which introduces both computational overhead and dependency on the external mask quality. This work addresses the fundamental question: how much object-level structure can be recovered from the Gaussians themselves, independent of external mask supervision or feature field training? The proposed method—Intrinsic-GS—establishes a mask-free, training-free pipeline that directly partitions the Gaussian primitives using intrinsic geometric, appearance, and motion cues.

Methodology and Algorithmic Contributions

The segmentation pipeline builds a $k$-NN affinity graph over the Gaussian primitives obtained from a frozen 4DGS scene (Figure 1). Edge weights in the graph fuse multi-modal signals: color, scale, orientation, deformation-trajectory motion, and boundary cues. The geometric term aggregates appearance, orientation, and scale using a geometric mean, enhancing discrimination when cues disagree. The motion term draws from deformation trajectories, explicitly capturing common fate. The boundary term is computed via non-learned edge detectors (e.g., Sobel) applied to rendered depth and RGB, denoting likely object boundaries.

Figure 1: Pipeline overview – a mask-free, training-free system forms a $k$-NN affinity graph over 4D Gaussians and segments via Leiden community detection.

Partitioning is performed using the Leiden community detection algorithm under the RB-configuration modularity objective, requiring no supervision and scaling efficiently to millions of graph nodes. To prevent spatially separated parts of a single object from splitting, a post-processing long-range merge step based on trajectory covisibility is included.

Empirical Evaluation

The segmentation performance is benchmarked on HyperNeRF and Neu3D, evaluating mean Intersection-over-Union (mIoU) against mask annotations initialized from SAM2 and manually refined. Intrinsic-GS achieves $0.746$ mIoU on Neu3D and $0.575$ on HyperNeRF using a single global configuration. Notably, on Neu3D, a geometry-only variant (no motion affinity, no boundary cue) attains $0.902$ mIoU, matching the SAM-supervised TRASE pipeline. On HyperNeRF, the gap localizes to cue-degenerate, near-rigid monocular scenes where intrinsic cues fail and mask priors become essential.

Runtime experiments demonstrate a $12.5\times$ speedup over mask-generation and feature-rendering stages in mask-supervised pipelines. All segmentation steps are performed without learned components or foundation model dependency.

Figure 2: Comparison of object masks from Intrinsic-GS and SAM-supervised TRASE; Intrinsic-GS recovers substantial object extent in several examples.

Detailed ablation studies show that in multi-view Neu3D, uniform cue fusion is suboptimal; geometry-only segmentation outperforms full fusion, indicating that deformation fields in these scenes are not reliable indicators of true object motion. Conversely, full multi-modal fusion is preferred for monocular HyperNeRF. Boundary suppression using classical, non-learned edge detectors outperforms learned ones (e.g., PiDiNet), confirming the mask-free nature of the pipeline.

Figure 3: Neu3D qualitative ablation – geometry-only segmentation retains object coherence comparable to full multi-modal fusion.

Theoretical Analysis

A sufficient condition for object separation within the RB-modularity framework is formulated. If any intrinsic modality (geometry, motion, boundary) yields a strong separation, the product structure of the affinity weight ensures a low cross-object affinity, favoring split communities under modularity maximization. Non-vanishing edge weights occur only when all cues are degenerate—i.e., textureless, co-moving, rigidly contacting objects lacking boundaries—defining the theoretical limits of mask-free segmentation.

Practical and Theoretical Implications

Intrinsic-GS demonstrates that dynamic 4D Gaussian scenes encode substantial object-level information in their geometric and motion attributes. This mask-free, training-free approach offers a computationally efficient alternative and a robust complementary operating point, enabling segmentation in scenarios where external mask generation is unreliable or expensive. For multi-view capture regimes, geometric affinity alone is often sufficient, underscoring the importance of reliability-aware cue fusion.

The results advocate for hybrid segmentation systems: intrinsic affinities should initialize or regularize mask-supervised pipelines, and external priors invoked only in cue-degenerate regimes. The findings also suggest a shift in representation-centric segmentation benchmarks, where mask-supervised training aligns closely with evaluation targets.

Limitations and Future Directions

Intrinsic-GS inherits limitations stemming from scene representation: errors in the initial Gaussian reconstruction, sensitivity to boundary cues in transparent regions, and granularity tuning via a fixed Leiden resolution. Intrinsic cues alone cannot resolve segmentation in degenerate cases; here, mask or semantic priors remain necessary. Further research should focus on reading richer, hierarchical, and open-vocabulary structure directly from Gaussian attributes and on learning lightweight, mask-free segmentation policies adaptive to cue reliability across diverse capture regimes.

Conclusion

This study establishes that dynamic Gaussian primitives carry substantial intrinsic object-level structure. Intrinsic-GS segments scenes efficiently without external mask or feature supervision, matching state-of-the-art mask-supervised methods on multi-view datasets and sharply delineating the limits of intrinsic cues. The approach provides a scalable, generalizable, and foundation-model-free direction for 3D/4D scene segmentation, laying groundwork for both practical deployment and theoretical advances in representation-based scene understanding.