DisCo3D: 3D Scene Editing Framework

• The paper proposes a novel three-stage framework that distills multi-view consistency from a scene-adapted 3D model into a 2D editor for Gaussian Splatting scene editing.
• It utilizes a 3D teacher (ViewCrafter) and a 2D student (InstructPix2Pix) with KL divergence-based score distillation to enforce both semantic fidelity and geometric coherence.
• Empirical evaluations on benchmarks like Tanks-and-Temples and IN2N show improved CLIP metrics and higher user preference compared to alternative 3D editing methods.

Searching arXiv for the specified DisCo3D paper and closely related naming to ground the article in the current literature. DisCo3D is a framework for text-driven 3D scene editing that distills multi-view consistency priors from a scene-adapted 3D novel-view-synthesis diffusion model into a 2D instruction-guided editor, then optimizes the edited outputs into a 3D Gaussian Splatting representation (Chi et al., 3 Aug 2025). It is designed for the setting in which a reconstructed 3D scene, represented as a Gaussian Splatting model, must be edited so that all views remain semantically aligned with a text instruction and mutually consistent under camera motion. The name should be distinguished from Disc3D, a separate work on automated curation of 3D dialog data; the two differ in both purpose and official naming (Wei et al., 24 Nov 2025).

1. Problem formulation and conceptual scope

DisCo3D addresses a central difficulty in diffusion-based 3D editing: maintaining multi-view consistency when edits are generated from 2D image editors. In the formulation used by the method, the input scene is a Gaussian Splatting model $\mathcal{G}$, together with rendered source views $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$, camera parameters, and a text editing instruction such as “Turn him into a clown” or “Add fruits to the cargo bed of the truck.” The outputs are a set of edited views $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$ and an updated Gaussian Splatting scene $\mathcal{G}_e$ (Chi et al., 3 Aug 2025).

Within this setting, multi-view consistency means that the same physical point, when visible in multiple views, should retain matching color and texture and should remain geometrically coherent across viewpoints. The paper positions this requirement as a failure mode of conventional 2D diffusion editing: most 2D editors operate independently per view, while existing multi-view editing methods that propagate 2D attention or depth-based control remain fragile under large camera baselines, occlusions, and complex geometry. DisCo3D therefore enforces consistency not through an explicit geometric loss in diffusion space, but by aligning a 2D editor’s multi-view output distribution with a consistency distribution induced by a 3D-aware teacher model.

A frequent misunderstanding is to treat DisCo3D as a direct 3D diffusion editor. The method does not primarily edit by generating novel views from the 3D teacher itself. Instead, the scene-adapted 3D model functions as a teacher of consistency, while semantic editing remains centered in the 2D instruction-based editor. This division of labor is one of the framework’s defining choices.

2. Three-stage architecture

The pipeline is organized into three stages: 3D NVS diffusion model adaptation, consistency distillation to a 2D editor, and 3D Gaussian Splatting update (Chi et al., 3 Aug 2025).

Stage	Core mechanism	Output
1	ViewCrafter fine-tuning with LoRA in temporal attention	Scene-adapted 3D consistency teacher
2	KL-based consistency distillation into InstructPix2Pix with self-attention LoRA	Multi-view-consistent 2D editor
3	$\ell_1$ and LPIPS supervision on edited views	Edited Gaussian Splatting scene

In Stage 1, the teacher model is ViewCrafter, described as a 3D U-Net-based NVS diffusion model based on DynamiCrafter and DUSt3R. For a given scene, DisCo3D renders multi-view source images from Gaussian Splatting, selects a reference view, encodes all views into latents, and fine-tunes ViewCrafter with LoRA modules inserted into temporal attention layers. The temporal fine-tuning objective is

$$\mathcal{L}_{\text{temporal}} = \mathbb{E}_{t \sim U(0, 1), \epsilon \sim \mathcal{N}(0, \mathbf{I})} \left[ \| \epsilon_\theta(\mathbf{z_t}, t, \mathbf{c}, I_s^{\text{ref}}) - \epsilon \|_2^2 \right].$$

This stage adapts the NVS model to the specific scene’s geometry and textures along the chosen camera trajectory.

In Stage 2, the student editor is InstructPix2Pix. The method initializes from pretrained InstructPix2Pix weights and inserts LoRA only into self-attention layers, leaving cross-attention intact so that prompt conditioning remains preserved. The result is a 2D editor that is intended to retain instruction-following behavior while learning consistency-constrained editing.

In Stage 3, the edited views supervise direct optimization of the Gaussian Splatting scene. Because the edited views are already intended to be mutually consistent, the 3D optimization no longer depends on iterative editing-updating loops of the kind used in IDU-style methods.

3. Consistency distillation objective

The technical core of DisCo3D is the distillation of a consistency distribution $p_{\text{cons}}$ into the editor’s output distribution $p_{\text{edit}}$. The paper defines the KL divergence

$$D_{\text{KL}}(p_{\text{edit}} \| p_{\text{cons}}) = \mathbb{E}_{\mathbf{z} \sim \mathcal{N}(0, \mathbf{I}),\ \mathbf{I_e} = E_\theta(\mathbf{z}, \mathbf{I_s})} \left[ - \left( \log p_{\text{cons}}(\mathbf{I_e}) - \log p_{\text{edit}}(\mathbf{I_e}) \right) \right],$$

with gradient

$$\nabla_{\theta} D_{\text{KL}} = \mathbb{E}_{\mathbf{z}, \mathbf{I_e}} \left[ - (s_{\text{cons}}(\mathbf{I_e}) - s_{\text{edit}}(\mathbf{I_e})) \nabla_{\theta}E_{\theta} (\mathbf{z},\mathbf{I_s}) \right].$$

The scores $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$0 and $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$1 are approximated by diffusion denoisers. The frozen score for the consistency distribution is supplied by the fine-tuned NVS teacher from Stage 1. The score for the edited distribution is supplied by a trainable copy of the NVS model, parameterized by $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$2, which is updated to track the editor’s output distribution through the objective

$$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$3

The editor is then updated by a score-difference gradient that depends on the difference between the frozen teacher denoiser and the trainable score network. The paper characterizes this as a two-network score-distillation formulation.

A second component is the regularization term that keeps the editor’s behavior on a reference view close to the original edited output produced before distillation. If $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$4 is the fixed reference edit and $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$5 is the evolving output during training, then

$$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$6

and the final editor loss is

$$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$7

The reported empirical setting uses $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$8, and the ablations state that too small a value leads to semantic drift or blur. This suggests that the regularizer is not merely stabilizing optimization, but is structurally necessary to prevent the teacher’s consistency prior from overwhelming the pretrained editor’s instruction-following semantics.

4. Training procedure, rendering, and 3D optimization

For efficient optimization, DisCo3D uses ReFL-style sampling during Stage 2 rather than backpropagating through the full denoising chain. A late denoising step is selected, the earlier steps are executed without gradients, and the final step is differentiated. The paper states that InstructPix2Pix sampling uses the EDM noise schedule, with text guidance scale $$\mathbf{I_s} = \{ I_s^0, I_s^1, \dots, I_s^{N-1} \}$$9 and image guidance scale $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$0, adjustable per scene (Chi et al., 3 Aug 2025).

The implementation is scene-specific. ViewCrafter uses a lower-resolution $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$1 variant; 49 total views are covered as one shared reference view plus two interleaved 24-view clips; camera poses within each clip are estimated by DUSt3R; and render maps are concatenated to images as inputs. The LoRA rank for ViewCrafter is 64, while the LoRA rank for the editor’s self-attention is 32. Stage 1 is trained for 200 iterations with learning rate $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$2, and Stage 2 uses editor learning rate $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$3 and about 100 iterations to obtain consistent multi-view edits.

The final 3D update fits the Gaussian Splatting scene to the edited views with $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$4 and LPIPS reconstruction losses:

$$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$5

and

$$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$6

The paper further notes that Segment Any 3D Gaussians can be used to obtain a semantic 3D mask so that only selected Gaussians are updated. This permits localized edits such as modifying one object while leaving nearby structure unchanged.

The runtime summary reports approximately 5 minutes for Stage 1, approximately 7 minutes for Stage 2, and less than 1 minute for Stage 3. For a new instruction on an already fine-tuned scene, the cost is therefore approximately 8 minutes. All experiments are reported on a single NVIDIA H800 GPU.

5. Evaluation and empirical behavior

The experimental evaluation uses 3 scenes from IN2N and 3 scenes from Tanks-and-Temples, with 10 editing instructions per scene group, 20 in total. Baselines are Instruct-GS2GS, GaussianEditor, and DGE. Metrics are CLIP Similarity, CLIP Directional Similarity, CLIP Directional Consistency, and a user study with 25 participants over 20 comparison tasks involving four methods (Chi et al., 3 Aug 2025).

On Tanks-and-Temples, DisCo3D reports the best CLIP metrics: CLIP Similarity 0.277, CLIP Directional Similarity 0.118, and CLIP Directional Consistency 0.807. The user study preference on the same benchmark is 52%, compared with 17–18% for the alternatives. On IN2N, the method is described as comparable to DGE and IGS2GS in CLIP metrics, but stronger in human preference, at 33% versus DGE 23%, IGS2GS 19%, and GaussianEditor 25%.

The qualitative findings in the paper are consistent with these aggregate results. IDU-based methods are described as prone to blurred textures or partial instruction failures because inconsistent per-view edits are averaged during 3D reconciliation. DGE preserves higher-frequency texture better than the IDU methods, but still exhibits view-to-view inconsistencies, such as changes in facial details across views. DisCo3D is reported to maintain detailed textures while preserving appearance consistency across viewpoints, particularly in complex 360° scenes with large baselines.

The ablations identify three components as especially consequential. Removing Stage 1 and using a non-finetuned ViewCrafter weakens the teacher’s consistency prior and yields color and texture inconsistencies. Replacing Stage 2 with direct “edit reference → generate novel views from NVS” produces semantic distortions, including deformed faces. On the “face” dataset, the reported comparison is CLIP Similarity 0.209, Directional Similarity 0.140, and Directional Consistency 0.868 without Stage 2, versus 0.237, 0.190, and 0.895 for the full pipeline. Varying $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$7 shows that $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$8 gives the best balance; smaller values lead to semantic misalignment, unrealistic outputs, or excessive smoothing.

6. Position within 3D editing research and limitations

DisCo3D is situated between two families of prior methods. Relative to SDS- and DreamFusion-style optimization, the framework avoids the slow optimization and texture blur associated with SDS/VSD-based 3D editing. Relative to Iterative Dataset Update methods such as Instruct-NeRF2NeRF and GaussianEditor, it avoids repeated edit-and-reconstruct loops by generating already consistent multi-view targets. Relative to attention-propagation methods such as DGE, ViCA-NeRF, GaussCtrl, and VCEdit, it does not depend on heuristic key-view propagation or feature warping. Instead, it treats consistency as a global property of the multi-view output distribution and enforces it through score-based distillation (Chi et al., 3 Aug 2025).

This positioning clarifies a second common misconception: DisCo3D is not primarily a hand-engineered cross-view attention mechanism. Its central claim is that multi-view consistency can be transferred as a learned prior from a scene-adapted 3D model into a 2D editor. A plausible implication is that the method’s gains depend as much on the quality of the teacher’s scene adaptation as on the downstream editor architecture.

The limitations identified in the paper are correspondingly specific. The method requires a reconstructed Gaussian Splatting scene, camera poses, DUSt3R, and a reasonably effective NVS model. There is scene-specific training overhead, although the Stage 1 teacher can be reused across multiple instructions. Performance depends on the quality of ViewCrafter adaptation and on DUSt3R camera trajectories and render maps. The regularization hyperparameter $$\mathbf{I_e} = \{ I_e^0, I_e^1, \dots, I_e^{N-1} \}$$9 is sensitive. The paper also notes that extremely large or structural edits may remain difficult. Future directions suggested in the text include stronger scene adaptation, improved distillation strategies, and extensions to dynamic scenes or temporally coherent edits.

In the literature on 3D scene editing, DisCo3D is therefore best understood as a teacher–student framework for transferring 3D consistency priors into instruction-based image editing, with Gaussian Splatting serving as the final geometric substrate. Its contribution lies less in replacing 2D editors or 3D generators than in coupling them so that instruction fidelity and cross-view coherence can be optimized jointly within a scene-specific workflow.