DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction

Abstract: Egocentric video is crucial for next-generation 4D scene reconstruction, with applications in AR/VR and embodied AI. However, reconstructing dynamic first-person scenes is challenging due to complex ego-motion, occlusions, and hand-object interactions. Existing decomposition methods are ill-suited, assuming fixed viewpoints or merging dynamics into a single foreground. To address these limitations, we introduce DP-DeGauss, a dynamic probabilistic Gaussian decomposition framework for egocentric 4D reconstruction. Our method initializes a unified 3D Gaussian set from COLMAP priors, augments each with a learnable category probability, and dynamically routes them into specialized deformation branches for background, hands, or object modeling. We employ category-specific masks for better disentanglement and introduce brightness and motion-flow control to improve static rendering and dynamic reconstruction. Extensive experiments show that DP-DeGauss outperforms baselines by +1.70dB in PSNR on average with SSIM and LPIPS gains. More importantly, our framework achieves the first and state-of-the-art disentanglement of background, hand, and object components, enabling explicit, fine-grained separation, paving the way for more intuitive ego scene understanding and editing.

• The paper introduces a unified probabilistic Gaussian decomposition framework that achieves explicit separation of hands, objects, and backgrounds in egocentric 4D scenes.
• It employs category-specific deformation branches and robust controls like brightness, motion-flow, and mask regulation to enhance reconstruction fidelity.
• Experimental results demonstrate significant improvements in PSNR, SSIM, and LPIPS metrics along with efficient training compared to prior methods like EgoGaussian.

DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction

Introduction

DP-DeGauss addresses the demands of egocentric 4D scene reconstruction, particularly the challenges imposed by dynamic, first-person interactions that include significant ego-motion, frequent occlusions, and complex hand-object manipulation. Prior solutions typically rely either on static-dynamic binary decomposition, fixed viewpoints, or manual labeling, failing to achieve fine-grained separation of hand, object, and background components essential in egocentric sequences. DP-DeGauss introduces a unified and dynamic probabilistic Gaussian decomposition framework, overcoming limitations of previous approaches with explicit category assignment, robust initialization, and category-specific deformation and control strategies.

Figure 1: Fine-grained decomposition of egocentric scenes with DP-DeGauss, separating background, hands, and objects, surpassing prior methods in quality and detail.

Methodology

Unified Gaussian Representation and Probabilistic Decomposition

DP-DeGauss leverages COLMAP-based SFM priors to instantiate a unified Gaussian cloud covering all scene elements—background, hands, and objects. Each Gaussian is augmented with a learnable probability vector $\mathbf{p}$ denoting its likelihood of belonging to each category, and a brightness attribute $b$. This probabilistic assignment enables robust initialization, unlike random or manually segmented approaches in previous works.

The model employs category-specific deformation branches for background (identity mapping), hand, and object components, driven by HexPlane encoders and MLP decoders. Gaussian probability vectors dynamically route each point to the respective branch via a two-stage gating: a soft stage enables continuous refinement of category assignments, while a hard stage commits each Gaussian to its most probable category for exclusive deformation, mitigating cross-branch contamination and enhancing category separability.

Figure 2: Overview of DP-DeGauss’s pipeline, showing COLMAP-initialized unified Gaussians, probabilistic routing, category-specific branches, and two-stage gating for robust decomposition and reconstruction.

Category-Level Controls

To optimize category separation and rendering fidelity, DP-DeGauss introduces three targeted controls:

• Brightness Control: Employs a mask rasterized from Gaussian brightness attributes in the background branch to stabilize illumination and prevent artifacts from lighting fluctuations.
• Motion-Flow Control: Applies optical flow guidance to dynamic branches, correcting for camera and scene-induced motion, and supervising predicted flow against combined ground truth and pose-based flows.
• Mask Control: Enforces spatially-aware loss computation using segmentation masks and zero-gradient enforcement in occluded or overlapping regions, preventing inaccurate cross-branch updates.

Rendering and Loss Functions

Rendering is performed either via composite images (all categories) or per-category rasters. In the soft stage, opacities are modulated by probabilistic assignments, while in the hard stage, Gaussians contribute exclusively to their assigned categories. The overall loss integrates RGB, opacity, SSIM, entropy, flow, and mask-driven terms to enforce photometric fidelity and structural disentanglement.

Experimental Evaluation

Reconstruction and Decomposition Performance

DP-DeGauss demonstrates substantial quantitative gains over baselines—including 4DGaussians, MotionGS, NeuralDiff, and DeGauss—with average PSNR improvements of +1.70dB and consistent SSIM and LPIPS superiority across HOI4D, EPIC-Field, and Hot3D datasets. Qualitatively, reconstructions feature sharper geometry, reduced motion blur, and fewer scene holes, especially within dynamic hand and object regions.

Figure 3: Qualitative comparison: DP-DeGauss yields sharper geometry and minimal artifacts in full-scene reconstruction, outperforming dynamic and static baselines.

Fine-Grained Decomposition

DP-DeGauss markedly improves foreground-background and hand-object separation, eliminating prevalent artifacts such as boundary leakage and misclassification observed in previous methods. Explicit instance-level separation enables clean disaggregation without sacrificing detail or introducing false positives.

Figure 4: DP-DeGauss achieves accurate, artifact-free decomposition of background, object, and hand components, with precise boundaries and minimal cross-category contamination.

Comparisons with EgoGaussian reveal that DP-DeGauss not only achieves comparable or finer object detail but also includes hands in reconstruction, delivering superior alignment and efficiency (2-hour training vs. 24+ hours for EgoGaussian).

Figure 5: Comparison with EgoGaussian showing DP-DeGauss's superior object detail and inclusion of hand components in both full-scene and separated reconstructions.

Ablation Studies

Rigorous ablation validates the importance of category-level controls. Brightness control removes ghosting from dynamic elements, motion flow sharpens dynamic reconstructions, and mask-based zero gradients rectify occlusions.

Figure 6: Ablation studies highlighting the efficacy of brightness, motion flow, and zero-gradient masking in recovering occluded and dynamic regions.

Implications and Future Directions

DP-DeGauss establishes a new paradigm for egocentric 4D scene reconstruction, enabling explicit and fine-grained separation of interacting entities without reliance on object or hand priors. The model's probabilistic routing and category-level controls lay foundational ground for compositional scene understanding, intuitive editing, and embodied AI reasoning. Practically, DP-DeGauss facilitates adaptive AR/VR content creation and personalized interaction modeling in unconstrained settings. Theoretically, its decomposition architecture could be extended to more granular semantic splits, multi-modal integration, or self-supervised refinement, advancing generalization across diverse egocentric domains.

Conclusion

DP-DeGauss implements dynamic probabilistic Gaussian decomposition with unified initialization, learnable category assignment, and bespoke control strategies, achieving state-of-the-art egocentric 4D reconstruction and explicit background-hand-object separation. This framework offers robust performance and opens prospects for scalable, compositional scene modeling in AR/VR and embodied AI.