CubifyGS: Object-Centric 3D Gaussian Splatting for Lifelong Dynamic Scene Maintenance

Abstract: Lifelong scene mapping under rigid object rearrangement remains a fundamental challenge in robotics. While 3D Gaussian Splatting (3DGS) enables high-fidelity modeling, primitive-level updates often cause persistent ghosting and slow recovery. We propose CubifyGS, an object-level mapping framework that shifts dynamic maintenance from passive re-optimization to active asset management. CubifyGS models movable instances as reusable Gaussian assets, detects object appearance and disappearance, and updates maps through asset retrieval, rigid transformation, and explicit pruning rather than reconstruction from scratch. To address geometric voids and local photometric mismatch after such edits, we further propose an event-triggered adaptive optimization strategy that focuses computation on affected regions. We validate our approach on a newly constructed high-fidelity dynamic benchmark, demonstrating that CubifyGS improves artifact suppression and maintenance efficiency over representative reproducible baselines in the evaluated object-rearrangement setting.

• The paper introduces an active asset management framework using object-centric 3D Gaussian splatting to rapidly update maps after object rearrangement.
• It leverages spatio-temporal perception and explicit object tracking to achieve up to 7.58 dB PSNR improvement and real-time performance.
• The method decouples dynamic object updates from static backgrounds through reusable Gaussian assets, DINOv3-based feature matching, and event-triggered adaptive optimization.

CubifyGS: Object-Centric 3D Gaussian Splatting for Lifelong Dynamic Scene Maintenance

Introduction

CubifyGS addresses the persistent limitations of state-of-the-art 3D Gaussian Splatting (3DGS) methods in lifelong scene mapping, focusing on dynamic environments with rigid object rearrangement. Prior 3DGS-based robotic mapping systems predominantly rely on passive, primitive-level gradient optimization, leading to persistent ghosting artifacts and slow recovery when objects move or disappear. These systems are not explicitly designed to maintain instance-level scene structure or efficiently handle discrete rearrangement events, which are common in practical indoor robotics and embodied perception. CubifyGS introduces a paradigm shift, transitioning from passive to active object-level scene maintenance through explicit asset management, leveraging reusable Gaussian assets, spatio-temporal dynamic perception, and event-triggered adaptive optimization for rapid, robust map updates.

Figure 1: Lifelong mapping under rigid object rearrangement. In dynamic indoor scenes, objects frequently move or vanish, and CubifyGS rapidly and explicitly updates the scene by managing object assets, achieving artifact-free reconstructions.

Unified Object-Centric Scene Representation

CubifyGS formalizes the environment as a unified object-centric map, representing each object as an encapsulated set of Gaussian primitives, an object-specific multi-view DINOv3 feature bank, and a globally parameterized bounding box. This explicit instance abstraction enables modular asset manipulation and decouples dynamic object updates from the static background, which remains as a separate Gaussian set.

A persistent global asset library stores canonical object templates. When an object is detected as moved or newly added, CubifyGS directly retrieves and aligns the corresponding asset, transforming its Gaussians to world space. Asset registration requires coverage and gradient stability checks, ensuring only well-observed and stable objects populate the library.

Spatio-Temporal Perception and Object Lifecycle Management

CubifyGS integrates a robust object tracking module that employs multi-view association (built on BoxFusion) and hierarchical data association for identity persistence. For each image, RGB-D cues anchor instance proposals, linked temporally via 3D/2D IoU-driven association schemes.

The system's Ray-Casted Occupancy Analysis actively detects object disappearances by comparing real-time depth measurements to expected instance volumes via raycasting, enabling explicit removal ("pruning") from the scene. Objects are flagged as vanished based on a decay-weighted existence probability, supporting rapid and unambiguous asset removal while preventing false deletion under occlusions.

Scene Maintenance: Instant Asset Pruning, Retrieval, and Local Optimization

The core maintenance protocol of CubifyGS replaces slow, primitive-level gradient "erase-and-reconstruct" cycles with deterministic asset-level editing. Upon detecting a vanished object, all associated Gaussians are instantaneously pruned, and ghosting artifacts are eliminated. When a new object appears, the system retrieves a matching asset from the library using DINOv3 feature similarity, applies coarse alignment based on canonical viewpoint retrieval, then performs differentiable fine alignment by optimizing only the object's rigid pose, greatly reducing computational overhead compared to full-attribute optimization.

Figure 2: Overview of the CubifyGS framework, illustrating dynamic detection, asset-based scene editing, and event-triggered adaptive optimization centered on changing regions.

This approach bypasses the fundamental inefficiency and instability observed in previous systems during rearrangement events.

Event-Triggered Adaptive Optimization

To rapidly resolve background voids or photometric inconsistencies resulting from dynamic edits, CubifyGS employs an event-triggered adaptive optimization routine. The framework generates a region-of-interest mask projected from the 3D bounding box of the changed object and skews the loss weighting to focus gradient descent within the ROI. This ensures rapid convergence in dynamic regions while avoiding catastrophic forgetting elsewhere in the map. Theoretical convergence is accelerated in active regions, supporting interactive or real-time applications.

Empirical Evaluation

CubifyGS is validated on a custom high-fidelity dynamic indoor benchmark constructed in Blender and a real-world dataset, featuring multiple rearrangement events per sequence. All mapping methods are evaluated with ground truth camera poses to isolate maintenance effectiveness.

System-Level Results

CubifyGS achieves the highest post-event reconstruction quality across all tested dynamic settings, outperforming MonoGS, GS-ICP SLAM, SplaTAM, and WildGS-SLAM by significant PSNR margins (up to 7.58 dB). Recovery after object change is consistently faster and with fewer artifacts, both visually and in terms of objective metrics. The framework delivers an average 35.83% improvement in PSNR over the strongest baseline.

Figure 3: Qualitative comparison of post-change recovery; CubifyGS immediately restores sharp geometry and eliminates lingering artifacts seen in all baselines.

Notably, CubifyGS executes active dynamic detection and maintenance at 20 FPS, exceeding WildGS-SLAM by a factor of 40 on identical hardware.

Frontend Perception and Registration

Object-level perception yields a 3D Average Precision (AP$_{25}$) of 65.2 versus 26.7 for the single-frame CA baseline, underscoring the importance of temporal association and multi-view aggregation in robust instance tracking.

Figure 4: 3D object localization results; CubifyGS bounding boxes tightly align with ground-truth, establishing high-confidence spatial priors for asset retrieval and map manipulation.

Retrieval and Alignment Analysis

DINOv3-based instance features display strong viewpoint-invariant clustering, supporting discriminative asset matching. Coarse viewpoint retrieval always contains the ground-truth view within $\pm 15^\circ$; the fine alignment procedure recovers the correct pose whenever initialized within $25^\circ$. This guarantees robust, autonomous pose recovery after asset insertion.

Figure 5: (a) t-SNE visualization of instance features; (b) similarity localization to ground truth; (c) fine alignment succeeds throughout the coarse retrieval error basin.

Ablation Study

Disabling focus-driven loss results in substantial degradation, particularly in high-density scenes (PSNR drop of 2.92 dB on Livingroom-1), confirming its necessity for responsive map repair. The explicit asset retrieval mechanism is indispensable: removing it and falling back to gradient-based reconstruction causes a 4.52 dB loss in the same scenario.

Theoretical and Practical Implications

CubifyGS fundamentally alters dynamic map maintenance for 3DGS. The explicit asset-level scene formulation, coupled with semantic-driven retrieval and event-localized optimization, refines the autonomy and operational efficiency of SLAM and scene representation in dynamic environments. By decoupling object identities, CubifyGS directly enables downstream tasks that require tight instance tracking and editable scene memory, including long-term manipulation, robot planning, and environment change detection.

While the current system focuses on rigid, reusable assets, its formalism invites generalization to nonrigid or evolving assets, asset-versioning, and multi-session cross-instance association. Integration with larger-scale spatio-temporal memory systems will be essential for full autonomy in lifelong mapping.

Conclusion

CubifyGS redefines dynamic 3DGS-based mapping via object-centric, active asset management, achieving rapid, artifact-free scene maintenance under rigid object rearrangement. The framework's advances in instance-level representation, explicit editing, and event-driven optimization translate to notable practical and theoretical gains, with immediate applicability to robotic autonomy and long-term scene understanding. Future directions include extending to generalized dynamics, robust perception under operational noise, and leveraging rapid map maintenance for real-time embodied reasoning.