Mono4DEditor: Text-Driven 4D Scene Editing

• Mono4DEditor is a framework that enables text-driven, localized editing of 4D scenes using a dynamic Gaussian representation with embedded language features.
• It employs a two-stage point-level localization process to accurately select and refine target regions based on CLIP-derived semantics.
• The system integrates diffusion-based video optimization with optical flow cues to ensure spatial detail and temporal coherence in scene modifications.

Mono4DEditor is a framework for text-driven editing of 4D scenes reconstructed from monocular video, leveraging point-level localization of language-embedded 3D Gaussians. This system achieves semantically precise modification of arbitrary regions within dynamic, complex scenes while preserving the geometry and appearance of unedited content. Mono4DEditor integrates quantized CLIP features into a dynamic Gaussian representation and applies a two-stage localization strategy—enabling high-fidelity, spatio-temporally coherent editing via diffusion-based video optimization.

1. Motivation and Scope

Mono4DEditor addresses the problem of localized 4D scene editing based on natural language instructions, where 4D refers to the three spatial dimensions plus time. Traditional neural rendering methods—such as NeRFs and vanilla Gaussian Splatting—support high-quality reconstruction from monocular video but lack mechanisms for region-specific text-driven edits. Existing text-conditioned approaches suffer from imprecise localization, resulting in over-editing, unintended modifications outside the target region, and the loss of temporal coherence. The framework overcomes these limitations by embedding language semantics directly in the dynamic 3D representation and introducing point-level localization to restrict edits accurately to the specified context.

2. Language-Embedded Dynamic Gaussian Representation

At the core of Mono4DEditor is a dynamic 3D Gaussian field, where each Gaussian is characterized by:

• Center $\mu \in \mathbb{R}^3$,
• Rotation $R \in SO(3)$,
• Scale $s$,
• Opacity $o$,
• Color $c \in \mathbb{R}^3$, and most critically, a learnable semantic feature $f \in \mathbb{R}^{d}$.

Semantic embedding is constructed by:

1. Extracting dense CLIP features from video frames.
2. Quantizing these features using a learnable codebook $\mathcal{B}$.

Rendering the semantic channel replaces standard color rasterization:

$$I_f^{(t)} = \mathcal{R}(\mu_t, R_t, s, o, f;\mathcal{C}_t)$$

where temporal dynamics are encoded by $\mu_t$, $R_t$ for each time step $t$.

During rendering, a lightweight MLP decoder maps the per-Gaussian semantic vector $f$ to a codebook index distribution (softmax over $\mathcal{B}$), thus creating a “language-embedded” Gaussian field where semantic querying is possible in native 3D.

3. Two-Stage Point-Level Localization Process

Localization of the region specified by a text prompt $q$ is achieved via a two-stage strategy:

• Stage I: Initial Selection
  • In each frame $t$, the dynamic Gaussians render a 2D semantic feature map $I_f^{(t)}$.
  • The MLP decoder produces a soft discrete distribution $\hat{M}^{(t)}$ over codebook entries.
  • Cosine similarity between the decoded features and the query feature $F_q$ (CLIP embedding of $q$) yields:

  $$R^{(t)}(p) = \cos(\hat{M}^{(t)}(p) \cdot \mathcal{B}, F_q)$$ - Thresholding $R^{(t)}(p)$ by $\tau$ yields a binary mask $M_\text{2D}^{(t)}$. - In parallel, each 3D Gaussian $i$ is decoded to CLIP feature space and compared to $F_q$; Gaussians with $r_i > \tau$ form an initial 3D localization set.

• Stage II: Refinement

  • Recall-Oriented Refinement: Gaussians not initially selected are nudged toward the target semantic region via cross-entropy loss on pixels within $M_\text{2D}^{(t)}$.
  • Precision-Oriented Refinement: Gaussians inside the mask are frozen, and those outside are optimized to suppress erroneous activations. This ensures that only Gaussians corresponding semantically and spatially to the query are included.

This dual refinement process counters both false negatives (missed target regions) and false positives (spurious activation outside the desired area), leading to high recall and precision in localization.

4. Targeted Text-Driven Editing via Diffusion Models

Editing proceeds by applying a diffusion-based video editing model (e.g., VACE) restricted to the set of localized Gaussians:

• Only parameters of selected Gaussians are updated; others are frozen for content preservation.
• Optical flow extracted from the original video provides motion cues, enforcing temporal consistency.
• Scribble guidance and mask controls ensure fine spatial detail and adherence to the editing region.

The optimization objective is:

$$\mathcal{L}_\text{edit} = \left\| V_\text{render} - V_\text{edit} \right\|_2^2$$

where $V_\text{render}$ is the video rendered from the edited Gaussian set and $V_\text{edit}$ is the output guide from the video diffusion editing model. Gradients affect only the localized, text-matched region, preserving untouched areas both semantically and geometrically.

5. Empirical Evaluation

Mono4DEditor achieves high-quality, flexible, and localized editing capability using several benchmarks:

• Experiments on DAVIS, DyCheck iPhone, DyNeRF, and in-the-wild datasets demonstrate accurate region selection and modification (e.g., “turn the cat into a tiger” or “change the balloon’s color”) without perturbing the context.
• Renderings preserve temporal coherence and novel-view consistency.
• Quantitative evaluation uses CLIP Text-Image directional similarity, confirming superior alignment with text prompts compared to methods such as IN4D and CTRL-D.
• Ablation studies show that both recall-oriented and precision-oriented refinements enhance localization accuracy; omitting either results in over-selection (background noise) or under-selection (missed object fragments).
• User preference studies further highlight perceived improvement in visual quality and flexibility.

6. Applications and Broader Implications

Mono4DEditor facilitates new capabilities for content creation and interactive virtual environments:

• Artists and animators can execute high-fidelity, localized edits of dynamic scenes using only textual descriptions, eliminating the need for manual segmentation.
• Virtual reality and game developers gain tools for dynamic, real-time adaptation of environments and objects based on user input.
• Advertising and creative industries benefit from rapid prototyping and mass customization, modifying scenes from single input videos without retraining or multi-view capture.

Embedding language semantics within dynamic 3D representations provides a foundation for advanced, controllable scene manipulation. This capability suggests future development toward multi-modal interfaces—potentially incorporating text, sketch, and gesture inputs—and extension to multi-view and topologically variable reconstructions for wider applicability.

7. Key Formulations and Technical Significance

Crucial mathematical components include:

• Semantic feature rendering: $$I_f^{(t)} = \mathcal{R}(\mu_t, R_t, s, o, f;\mathcal{C}_t)$$
• Semantic index decoding: $$\hat{M}^{(t)} = \text{softmax}(\mathcal{D}(I_f^{(t)}))$$
• Cosine similarity relevance: $$R^{(t)}(p) = \cos(\hat{M}^{(t)}(p) \cdot \mathcal{B}, F_q)$$
• Localized editing loss: $$\mathcal{L}_\text{edit} = \left\| V_\text{render} - V_\text{edit} \right\|_2^2$$

Mono4DEditor thus establishes a rigorous pipeline for integrating natural language with spatiotemporal Gaussian fields, achieving both theoretical and practical advances in localized, temporally stable 4D scene editing from monocular video. This framework outperforms previous approaches in precision, flexibility, and visual fidelity, establishing a technical foundation for subsequent research and application in dynamic scene manipulation.