M2StyleGS: Multi-Modality 3D Style Transfer with Gaussian Splatting

Abstract: Conventional 3D style transfer methods rely on a fixed reference image to apply artistic patterns to 3D scenes. However, in practical applications such as virtual or augmented reality, users often prefer more flexible inputs, including textual descriptions and diverse imagery. In this work, we introduce a novel real-time styling technique M2StyleGS to generate a sequence of precisely color-mapped views. It utilizes 3D Gaussian Splatting (3DGS) as a 3D presentation and multi-modality knowledge refined by CLIP as a reference style. M2StyleGS resolves the abnormal transformation issue by employing a precise feature alignment, namely subdivisive flow, it strengthens the projection of the mapped CLIP text-visual combination feature to the VGG style feature. In addition, we introduce observation loss, which assists in the stylized scene better matching the reference style during the generation, and suppression loss, which suppresses the offset of reference color information throughout the decoding process. By integrating these approaches, M2StyleGS can employ text or images as references to generate a set of style-enhanced novel views. Our experiments show that M2StyleGS achieves better visual quality and surpasses the previous work by up to 32.92% in terms of consistency.

• The paper introduces a robust multi-modal 3D style transfer framework using Gaussian splatting and subdivisive flow to achieve precise feature alignment.
• It utilizes CLIP-based conditioning and auxiliary loss functions to maintain real-time, high-quality stylization across diverse inputs.
• Experimental results show enhanced inter-view consistency and reduced artifacts, outperforming previous state-of-the-art methods.

Multi-Modality 3D Style Transfer with M$^2$StyleGS

Overview

M$^2$StyleGS introduces a flexible and effective framework for 3D style transfer grounded in Gaussian Splatting (3DGS) and multi-modal CLIP-based conditioning (2604.03773). Unlike previous approaches restricted to fixed reference images, M$^2$StyleGS allows stylistic transformations using arbitrary image or textual references. The method achieves real-time, high-quality color-mapped 3D views and addresses longstanding issues in feature alignment and stylization regularization inherent to multi-modal transfer regimes.

Figure 1: Multi-modality 3D style transfer with M$^2$StyleGS using both image and text references across novel scene perspectives.

Methodological Contributions

Framework and Pipeline

M$^2$StyleGS leverages 3D Gaussian primitives for explicit scene representation, maintaining fixed geometry while transforming color attributes according to target styles. Input modalities are processed via CLIP encoders, producing image or text-conditioned features for alignment with VGG-based style representations.

The pipeline comprises a mapping module for style feature alignment, an AdaIN-based compositional transfer module, and a shared-weight decoder for color transformation. Notably, it introduces a subdivisive flow, an ODE-based feature trajectory learning scheme, facilitating finer, step-wise feature alignment between the CLIP and VGG spaces, mitigating the inaccuracies produced by direct mapping routines.

Figure 2: The pipeline of M$^2$StyleGS enabling multi-modal stylization through unified feature processing for both images and texts.

Subdivisive Flow for Feature Alignment

Previous multi-modal mapping approaches exhibited coarse alignment between CLIP features and VGG style descriptors, resulting in stylistic mismatches and artifacts. M$^2$StyleGS deploys a subdivisive flow, articulating the alignment process as a trajectory guided by an ODE with multiple subdivision rounds and intermediate feature states. This procedure utilizes iterative linear interpolation enriched with least-squares optimization on drift vectors, ensuring that the end feature distribution closely matches the target VGG space.

Figure 3: Comparison of direct mapping vs. subdivisive flow for CLIP-to-VGG alignment, with subdivisive flow yielding precise feature trajectories.

Auxiliary Loss Functions

Two auxiliary losses are introduced to reinforce stylization fidelity and multi-view consistency:

• Observation Loss ($\mathcal{L}_{obs}$): Utilizes outputs from a pre-trained 2D stylization generator as priors, correlating 3D stylized views with reference artworks in VGG feature space.
• Suppression Loss ($\mathcal{L}_{sup}$): Employs a multi-scale discriminator to globally suppress misleading scene color artifacts, regularizing the decoding process.

These losses are integrated with conventional content and style losses to form a comprehensive training objective, controlling the balance between stylization and structural content preservation.

Experimental Results

Qualitative Evaluation

M$^2$StyleGS demonstrates pronounced qualitative improvement in transferring fine stylistic details with both image and text prompts while maintaining structural consistency and color integrity across multiple views. Compared to ConRF, StyleGaussian, and StyleRF, the outcomes show enhanced clarity and reduced artifacts, especially in challenging scenes. For text-based stylization, M$^2$0StyleGS efficiently captures both chromatic richness and semantic depth, outperforming prior approaches such as CLIPNeRF and CLIPStyler.

Figure 4: Qualitative comparison of M$^2$1StyleGS with multiple SOTA 3D style transfer methods on image and text-based prompts.

Quantitative Metrics

Evaluations on LLFF and Tanks & Temples datasets involved RMSE and LPIPS consistency metrics calculated via optical flow-wrapped views. M$^2$2StyleGS achieves substantial reductions in short-range LPIPS (32.92% lower than ConRF; 25.68% lower than StyleRF) and maintains superior RMSE values, indicating robust inter-view consistency.

Ablation Studies

To validate the efficacy of subdivisive flow, ablation experiments measured cosine similarity (SIM) and Fréchet Inception Distance (FID) between intermediate CLIP and VGG feature states across subdivisive rounds. SIM improved significantly for image transfer, while FID decreased for text transfer, evidencing tighter feature matching. Loss function analysis showed that removal of $^2$3 or $^2$4 led to increased inconsistency, underscoring their necessity for high-quality stylization.

Figure 5: Ablation studies on subdivisive flow demonstrating incremental alignment improvement across subdivisive rounds for image and text modalities.

Figure 6: Impact of different loss functions on stylization fidelity and inter-view consistency.

Implications and Future Directions

The methodological advances in feature trajectory learning and multi-modality conditioning suggest significant practical utility for interactive VR/AR, creative content generation, and real-time scene stylization. The subdivisive flow paradigm offers theoretical insights into domain alignment for heterogeneous feature spaces, potentially informing future neural style transfer models in non-Euclidean and non-Gaussian regimes.

Future developments may involve:

• Extending subdivisive flow to deeper architectures or alternative backbone encoders.
• Incorporating differentiable rendering for finer spatial and temporal stylization control.
• Exploring multi-modal conditioning beyond CLIP, including audio or motion cues for richer scene transformations.
• Integration with dynamic scenes and temporal consistency modules for video or simulation-based applications.

Conclusion

M$^2$5StyleGS delivers a robust, real-time framework for multi-modality 3D style transfer utilizing subdivisive feature alignment and auxiliary loss regularization. The model surpasses prior SOTA methods by achieving enhanced stylization fidelity and inter-view consistency, facilitating flexible input modalities and addressing key alignment and regularization challenges in contemporary neural 3D stylization.