CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-centric 3D Scene Generation

Abstract: Challenges remain in ego-centric 3D scene generation due to limited view overlap and the dominant influence of individual perspectives on scene interpretation. These factors hinder the creation of viewpoint-consistent and semantically aligned visual content, as well as the construction of accurate geometric structures. In this paper, we propose CGGS, a text-to-3D framework aiming to enhance 3D-content-awareness and address geometric distortions in ego-centric scene generation. Firstly, the Ego-centric Generator is proposed by fine-tuning a Multi-View Latent Diffusion Model with consistency-augmented loss to generate consistent, high-fidelity 2D content aligned with textual descriptions. Then, Layout Decorator leverages optical flow and point-track correspondence to estimate depth, therefore producing dense point clouds as coarse layouts from the ego-centric 2D priors. Building on this initialization, Geometric Refiner is proposed to enhance 3D Gaussian reconstruction via an entropy-based Mutual Information Depth Loss (MID) combined with a hierarchical optimization scheme for improving visual quality and geometric structure. Comprehensive experiments demonstrate that \textcolor{softred}{CGGS} outperforms previous methods in generating coherent and accurate text-driven 3D scenes. Project page: https://cggs-26.github.io/cggs26/.

• The paper introduces a three-stage pipeline combining consistency-augmented latent diffusion, flow-guided depth estimation, and MID-regularized 3D Gaussian splatting for robust ego-centric scene generation.
• It achieves high semantic and geometric fidelity with superior PSNR, SSIM, and LPIPS scores compared to existing methods.
• The approach demonstrates strong cross-domain generalization, making it applicable for AR/VR, robotics, and immersive content creation.

Consistency-Augmented Geometric Gaussian Splatting (CGGS) for Egocentric 3D Scene Generation

Introduction and Motivation

Ego-centric 3D scene generation poses specific challenges distinct from general text-to-3D tasks, notably the limited overlap between sequential views, strong viewpoint-induced structural ambiguities, and difficulties ensuring semantic and geometric consistency. Existing pipelines based on panoramic priors yield strong global continuity but suffer substantial geometric distortion, particularly near the poles of equirectangular projections, leading to errors during 3D construction. Conversely, multi-view generation can offer distortion-free perspective images but is hampered by weak cross-view consistency (Figure 1).

Figure 1: Visualization of geometric distortions arising in panoramic generation (using DreamScene360 as example), highlighting omitted prompt details, severe projection-induced stretching at image poles, and structural artifacts.

To address these limitations, the CGGS framework introduces a novel workflow that tightly integrates (i) a consistency-augmented multi-view latent diffusion generator, (ii) an optical flow and point-track-based depth estimator for robust 3D layout initialization, and (iii) an information-theoretic geometric refinement module leveraging 3D Gaussian Splatting (3DGS). This system is tailored to synthesize semantically aligned, geometrically accurate, and domain-agnostic 3D scenes from text prompts in an ego-centric (first-person) perspective.

Framework Overview

CGGS operates via three primary components, orchestrated to maximize 3D consistency and semantic fidelity (Figure 2 & Figure 3):

• Ego-centric Generator: Fine-tuned Multi-View Latent Diffusion Model with a novel consistency-augmented loss for perspective-consistent, text-aligned 2D priors.
• Layout Decorator: Optical flow-guided, point-track-regularized depth estimator constructing dense point cloud layouts from multi-view priors.
• Geometric Refiner: Hierarchical 3D Gaussian optimization, supervised via a Mutual Information Depth (MID) loss, to improve reconstruction accuracy and visual coherence.

  Figure 2: CGGS pipeline schematic—Ego-centric Generator produces multi-view 2D priors, Layout Decorator estimates depth and point-based layouts, and Geometric Refiner optimizes for 3D geometric and photometric fidelity.

  

  Figure 3: Detailed CGGS architecture—highlighting the integration of consistency-augmented MV-LDM, Flow-Depth Estimator, and MID-guided hierarchical 3D Gaussian optimization.

Technical Contributions

1. Consistency-Augmented Latent Diffusion

Standard multi-view diffusion (e.g., MVDiffusion/Corr-Aware Attention) yields semantically inconsistent, artifact-prone outputs due to independent gradient flow for each perspective. CGGS enforces a consistency-augmented loss term, projecting the denoising loss through a stationary random CNN (He-initialized, VGG-16 structured and frozen), harmonizing feature-level updates across all views. This novel regularization:

• Reduces cross-view optimization conflicts by aligning gradient directions.
• Maintains multi-scale, spatially localized features without semantic bias (via untrained random projection).
• Enhances both local and global semantic alignment, confirmed by ablation showing improved CLIP-Score, Q-Align, and perceptual quality metrics.

Ablation demonstrates significant degradation without this loss, with chaotic layouts and texture inconsistencies emerging (Figure 4).

Figure 4: Removing the consistency-augmented loss causes pronounced cross-view texture discrepancies and semantic artifacts.

2. Optical Flow and Point-Track Decorated 3D Layouts

Off-the-shelf monocular depth predictions suffer from projective scale ambiguity and inter-view misalignment, particularly in ego-centric settings. The Layout Decorator module:

• Utilizes dense optical flow (RAFT) across interpolated multi-view sequences to establish pixel-level correspondences.
• Integrates long-term point tracking (CoTracker) to correct flow drift and enforce global structural coherence.
• Supervises depth estimation with correspondence loss, harmonizing scale and geometric consistency across frames.

This approach substantially outperforms standard SfM/COLMAP under sparse, low-overlap trajectories, both quantitatively and qualitatively (see ablation Table, and Figure 5).

Figure 5: Ablation of Geometric Refiner modules (MID+HO, MID, PD+HO, PD, and baseline), showing superior texture recovery and fewer artifacts with MID+HO.

3. Hierarchical 3DGS Optimization with Mutual Information Depth Loss

Conventional 3DGS pipelines with Pearson correlation depth loss tend to oversmooth, neglecting nonlinear geometric features. CGGS’s Mutual Information Depth (MID) Loss directly maximizes the mutual information between rendered and reference depth distributions, promoting:

• Nonlinear, high-frequency geometric alignment, even under complex scene structures.
• Robustness against global scale ambiguity and outliers inherent to ego-centric motion.
• Synergy with a hierarchical optimization scheme (progressive introduction of supplementary camera viewpoints), which further enforces scene-wide geometric integrity.

Ablations confirm that hierarchical optimization (HO) and MID together enable the sharpest, most faithful reconstructions, as reflected in SSIM, PSNR, and LPIPS.

Experimental Results

Quantitative evaluation across 24 challenging scenes (Table in manuscript) demonstrates that CGGS achieves:

• Highest CLIP-Score (26.253) and Q-Align (0.839): Demonstrating optimal semantic-text alignment and perceptual coherence.
• Superior reconstruction metrics: PSNR (37.345), SSIM (0.977), and lowest LPIPS (0.0193), surpassing baselines including DreamScene360, LucidDreamer, and Director3D.
• Generalization: Notably, CGGS’s architecture, despite fine-tuning only the multi-view generator on indoor datasets, retains strong out-of-domain performance, accurately reconstructing semantically diverse scenes such as deserts, urban environments, and aquatic biomes.

Qualitative examples confirm realistic geometric structures, high-fidelity photometric rendering, and strong viewpoint consistency in both in-domain and cross-domain (OOD) datasets (Figure 6, Figure 7, Figure 8, Figure 9).

Figure 6: Sample generation results—Ego-centric priors, Gaussian point clouds, novel view synthesis, and estimated depth maps.

Figure 7: Qualitative baseline comparison—CGGS produces multi-view images with richer detail and superior semantic coherence than prior art.

Figure 8: Additional diverse generation results, demonstrating structural and semantic consistency across varied scenes.

Figure 9: Further generation examples highlighting fidelity and diversity.

Implications and Future Directions

CGGS’s integration of semantic-aligned diffusion, robust geometric layout estimation, and information-theoretic 3DGS optimization advances the field of generative scene synthesis, particularly for ego-centric and AR/VR applications with challenging viewpoints. The demonstrated generalization and superiority in both perceptual and structural metrics suggest its applicability to synthetic data generation for robotics, autonomous driving, and immersive content creation.

Key open avenues include:

• Reducing per-scene optimization and improving scalability for dynamic scenes.
• Integrating downstream embodied reasoning or visual language agents into the generated environments.
• Exploring privacy and security considerations for synthetic scene content in practical deployments.

Conclusion

CGGS introduces a three-stage pipeline—consistency-augmented multi-view latent diffusion, flow/track-supervised depth estimation, and MID-regularized 3D Gaussian Splatting—that establishes state-of-the-art semantic and geometric performance for ego-centric text-to-3D scene generation (2607.03819). It robustly addresses the core challenges unique to first-person 3D capture, with demonstrated superiority over existing baselines and promising generalization to unseen domains. The proposed methodologies set new technical benchmarks for semantic and structural fidelity in 3D generative vision.