ClipGS: Multimodal Gaussian Stylization

• ClipGS is a plug‑in style transfer framework that optimizes anisotropic Gaussian primitives for multimodal 2D, 3D, and 4D stylization.
• It employs a two‑stage pipeline combining VGG‑19 and CLIP losses to jointly refine appearance and geometry while ensuring temporal coherence.
• The method integrates seamlessly into existing Gaussian Splatting pipelines without changing the Gaussian count, enabling efficient stylization.

ClipGS, in the usage associated with “CLIPGaussians / CLIPGaussian,” denotes a plug‑in style transfer framework for Gaussian Splatting that operates directly on Gaussian primitives and uses CLIP to drive text‑guided or image‑guided stylization across 2D images, videos, 3D objects, and 4D scenes. It is presented as the first unified style transfer framework supporting text‑ and image‑guided stylization across these modalities, integrates into existing Gaussian Splatting pipelines without requiring large generative models or retraining from scratch, and keeps the number of Gaussians unchanged (Howil et al., 28 May 2025).

1. Definition, representation, and problem setting

ClipGS is built on the observation that Gaussian Splatting represents scenes as explicit sets of anisotropic Gaussians rather than as implicit radiance fields. In the formulation used by the method, a scene can be written as

$$G = \{ (N(m_i,\Sigma_i), \sigma_i, c_i) \}_{i=1}^{n},$$

where $m_i$ is the 3D mean position, $\Sigma_i$ is the 3D covariance, $\sigma_i$ is opacity, and $c_i$ is color represented as Spherical Harmonics coefficients. A more general form used for multimodal transfer is

$$G = G_{m_i,\Sigma_i, \sigma_i, c_i, \theta_i } = \{ (N(m_i,\Sigma_i), \sigma_i, c_i, \theta_i) \}_{i=1}^{n},$$

with $\theta_i$ denoting modality‑specific parameters such as deformation parameters for 4D scenes or folding functions for videos (Howil et al., 28 May 2025).

The method is motivated by the fact that Gaussian Splatting has been generalized beyond static 3D scenes to 2D images, videos, and 4D dynamic scenes, yet stylization for these representations is nontrivial. The difficulty is not limited to recoloring pixels. The representation is a set of Gaussians in 3D or 4D, appearance is view dependent, temporal coherence must be preserved in video and 4D settings, and nontrivial styles such as “Fire” or “Starry Night” may require geometry changes rather than color changes alone. The paper therefore frames style transfer for Gaussian Splatting as direct optimization of Gaussian parameters under multimodal style guidance, rather than as an external image‑space postprocess (Howil et al., 28 May 2025).

The targeted setting is explicitly unified. The same style transfer algorithm and loss structure are applied to MiRaGe for 2D images, VeGaS for videos, vanilla 3DGS for 3D scenes and objects, and D‑MiSo for 4D dynamic scenes. This suggests that the central abstraction is not a particular renderer, but the shared Gaussian primitive parameterization that these models expose (Howil et al., 28 May 2025).

2. Method architecture and objective

ClipGS uses a two‑stage pipeline. First, a base Gaussian Splatting model is trained for reconstruction in the native modality. Second, a style transfer stage optimizes the already trained Gaussian representation without modifying the underlying architecture. In this second stage, a training image $I_l \in \mathcal{I}$ is sampled, rendered as $R_G(I_l)$, randomly cropped into fixed‑size patches, and augmented with random perspective transformations before loss computation. Gradients are then backpropagated directly into Gaussian parameters, and in 4D also into deformation networks (Howil et al., 28 May 2025).

The framework combines VGG‑19 and CLIP. Content preservation is imposed with

$$L_c(R_G(I_l), I_l) = MSE\big( \Phi_{VGG}(R_G(I_l)), \Phi_{VGG}(I_l) \big),$$

computed on conv4_2 and conv5_2 features of VGG‑19. Global style is controlled by a directional CLIP loss: $m_i$0 Here $m_i$1 is either a text prompt or a style image, and “Photo” is used as the neutral reference prompt. Local style detail is imposed through a patch loss,

$m_i$2

with randomly sampled patches $m_i$3 and perspective augmentation. When backgrounds should remain fixed, a background loss is added; in one formulation it is the mean $m_i$4 distance between rendered background pixels and a target background color, and in another 4D setting it is defined through an alpha‑derived mask (Howil et al., 28 May 2025).

The total training objective is

$m_i$5

Typical default weights are reported as $m_i$6, $m_i$7, and $m_i$8 for 3D, 4D, and 2D, while video uses $m_i$9. The CLIP encoder used in losses is ViT‑B/32, whereas evaluation metrics use ViT‑L/14. The method typically runs for about 5000 optimization steps, and because it neither adds nor removes Gaussians, the model size remains roughly constant (Howil et al., 28 May 2025).

A key methodological point is that ClipGS does not add a separate stylization network around the renderer. It updates the parameters of the existing Gaussians—positions, covariances, opacities, colors, and modality‑specific parameters—making it a plug‑in stage rather than a replacement for the base Gaussian Splatting pipeline (Howil et al., 28 May 2025).

3. Modality-specific optimization and geometry control

ClipGS distinguishes sharply between modalities in terms of which attributes are trainable. In 3D, 2D, and 4D settings it performs joint optimization of position $\Sigma_i$0, covariance, opacity $\Sigma_i$1, and SH color coefficients $\Sigma_i$2. For 4D, it additionally updates deformation network parameters and multi‑Gaussian parameters, including trainable offsets $\Sigma_i$3. For VeGaS video, by contrast, it optimizes only SH color coefficients $\Sigma_i$4, keeping geometry fixed to preserve temporal coherency and avoid breaking motion (Howil et al., 28 May 2025).

The paper argues that geometry optimization matters because many styles imply structural changes as well as appearance changes. Allowing $\Sigma_i$5 and $\Sigma_i$6 to move or deform Gaussians lets the renderer approximate style‑dependent structural patterns such as swirling waves or flames. This is explicitly contrasted with several earlier Gaussian Splatting stylization methods that are described as color‑only or opacity‑only. ClipGS is designed to support “joint appearance and geometry optimization” in 3D and 4D while restricting video to appearance‑only adaptation (Howil et al., 28 May 2025).

For 2D MiRaGe, optimization uses position $\Sigma_i$7, rotation $\Sigma_i$8, scale $\Sigma_i$9 with one dimension fixed to $\sigma_i$0 for flatness, opacity $\sigma_i$1, and SH colors $\sigma_i$2. For 4D D‑MiSo, the paper distinguishes core and sub‑Gaussians,

$\sigma_i$3

and

$\sigma_i$4

together with learned deformation fields. For VeGaS, the dynamic primitive is a 3D Folded Gaussian,

$\sigma_i$5

and ClipGS modifies only the SH colors $\sigma_i$6 (Howil et al., 28 May 2025).

The method controls over‑distortion through several constraints already built into the formulation: a strong VGG content loss, a fixed number of Gaussians, and balanced loss weights. The reported ablation analysis states that too high a feature learning rate or too many patches can overshoot content preservation. This suggests that ClipGS relies on bounded deformation capacity rather than on an explicit geometric regularizer of the kind used in some deformation‑field models (Howil et al., 28 May 2025).

4. Temporal coherence, multimodal scope, and pipeline integration

ClipGS does not introduce an explicit handcrafted temporal loss. Temporal coherence instead follows from the time structure of the underlying Gaussian representation. In D‑MiSo, the same Gaussian identities persist through time and are deformed by shared deformation networks, so changes to their parameters affect all timesteps consistently. In VeGaS, each Gaussian spans multiple frames through temporal folding functions, and ClipGS changes only colors of these time‑spanning Gaussians. The paper therefore attributes temporal stability to parameter sharing in the underlying Gaussian representation rather than to per‑frame regularization (Howil et al., 28 May 2025).

Temporal metrics are defined using CLIP embeddings. The paper reports

$\sigma_i$7

and

$\sigma_i$8

The paper states that it reports significantly better temporal consistency than diffusion‑based video stylization methods (Howil et al., 28 May 2025).

Operationally, the framework takes as input a pretrained GS model, a style prompt or reference image, and optionally background masks. The output is the same GS model with modified parameters and unchanged Gaussian count. The procedure is: train the base GS model, keep Gaussian parameters trainable, initialize the ClipGS optimization with modality‑specific weights and parameters such as feature_lr, patch_size, and num_patch, run about 5000 optimization steps, and then use the original renderer for inference. Reported training times on an RTX 4090 for about 5000 steps are 11:29 for hotdog with 0.14M Gaussians, 11:36 for lego with 0.31M, 11:37 for bonsai with 1.35M, and 21:03 for garden with 4.48M. Video clips from DAVIS with fewer than 100 frames take about 11 minutes, while 4D scenes are described as taking on the order of tens of minutes depending on batch size and the number of time steps (Howil et al., 28 May 2025).

A recurrent design claim is that ClipGS leaves model size unchanged. The paper reports a memory size change of +0% relative to the base models and contrasts this with G‑Style, which is described as adding Gaussians and incurring +126% memory (Howil et al., 28 May 2025).

5. Empirical results across 3D, 4D, 2D, and video

The most detailed quantitative evaluation is on 3D style transfer for NeRF‑Synthetic and Mip‑NeRF 360 objects and scenes. In text‑guided 3D style transfer, the reported CLIP‑based metrics are: I‑GS2GS with CLIP‑S 16.80, CLIP‑SIM 12.03, CLIP‑F 99.19, CLIP‑CONS 13.53, and memory size $\sigma_i$9; DGE with 17.59, 12.27, 99.31, 12.46, and $c_i$0; and CLIPGaussian with 26.86, 26.31, 98.80, 2.34, and $c_i$1. In image‑guided 3D style transfer, StyleGaussian reports CLIP‑S 63.69 and CLIP‑SIM 13.07 at $c_i$2 memory, G‑Style reports 76.94 and 24.94 at $c_i$3 memory, and CLIPGaussian reports 72.65 and 20.72 at $c_i$4 memory. The paper characterizes this as dramatically outperforming text‑editing baselines on style metrics while preserving model size, and as being close to G‑Style in image‑guided style metrics without doubling the Gaussian count (Howil et al., 28 May 2025).

A 3D user study is reported through four surveys with 30 participants each via Clickworker. The findings are that for image‑guided object stylization, CLIPGaussian is most often rated most similar to the reference; for image‑guided scenes, G‑Style is slightly preferred for similarity though CLIPGaussian is rated better than StyleGaussian and uses far fewer Gaussians; for text‑guided objects, CLIPGaussian is most preferred for stylization quality; and for text‑guided scenes, the results are mixed due to prompt subjectivity. Friedman tests and Conover‑Friedman post‑hoc tests are said to show significant differences between methods (Howil et al., 28 May 2025).

For 4D style transfer, the base model is D‑MiSo and the datasets include D‑NeRF, DyNeRF, and PanopticSports. The paper reports that in image‑guided settings ClipGS captures more detailed textures and patterns than 4DStyleGaussian, and in text‑guided settings it produces strong geometric and appearance changes on both object‑like sequences and scenes. The reported interpretation is that it achieves temporally consistent stylization over time and viewing angle without an explicit temporal loss (Howil et al., 28 May 2025).

For 2D image style transfer, the base model is MiRaGe and the dataset is a COCO subset. The reported finding is that ClipGS may not reach the ultra‑sharp quality of large diffusion models, but preserves content identity better than diffusion edits and captures detailed local style patterns because of the patch‑based objective. The paper presents this as useful when explicit GS‑based controllability and compatibility with 3D and 4D workflows are desired (Howil et al., 28 May 2025).

For video style transfer on DAVIS using VeGaS, the appendix table reports that in image‑guided stylization CCPL scores 18.89 CLIP‑S, 8.20 CLIP‑SIM, 97.92 CLIP‑F, and $c_i$5 CLIP‑CONS; UniST scores 15.93, 3.85, 99.36, and 5.16; and ClipGS scores 74.31, 17.60, 99.18, and 1.27. In text‑guided stylization, RerenderAVideo scores 19.40, 9.83, 98.23, and $c_i$6; Text2Video scores 26.05, 24.99, 93.63, and 0.03; and ClipGS scores 26.25, 24.53, 99.00, and 1.92. The qualitative summary given is that ClipGS shows better temporal consistency than RerenderAVideo and Text2Video, less flickering, more faithful global style than CCPL and UniST, and good preservation of motion (Howil et al., 28 May 2025).

6. Relation to earlier methods and the naming ambiguity of “ClipGS”

Within Gaussian Splatting stylization, ClipGS is positioned against several earlier families of methods. The paper describes StyleGaussian, ReGS, InstantStyleGaussian, Style3D, and StyleSplat as largely modifying colors and opacities while leaving geometry fixed. It also groups InstantStyleGaussian, Style3D, Morpheus, Instruct‑4D‑to‑4D, and diffusion‑based 3D or 4D edits as relying on large generative models, and identifies G‑Style as a method that explicitly adds Gaussians for geometry refinement, roughly doubling the Gaussian count with $c_i$7 memory. ClipGS is proposed as a lightweight alternative that works directly on Gaussians, does not change the model size, and supports both text and image guidance across multiple modalities (Howil et al., 28 May 2025).

The term “ClipGS” is also ambiguous in the broader literature. It has been used for “ClipGS: Clippable Gaussian Splatting for Interactive Cinematic Visualization of Volumetric Medical Data” (Li et al., 9 Jul 2025), where it refers to a medical volume visualization framework with clipping‑plane support rather than a CLIP‑guided style transfer system. Closely related medical work appears in “ClipGS‑VR: Immersive and Interactive Cinematic Visualization of Volumetric Medical Data in Mobile Virtual Reality” (Tong et al., 27 Jan 2026). The spelling “CLIP‑GS” is also used for distinct multimodal and semantic frameworks, including “CLIP‑GS: Unifying Vision‑Language Representation with 3D Gaussian Splatting” (Jiao et al., 2024) and “CLIP‑GS: CLIP‑Informed Gaussian Splatting for View‑Consistent 3D Indoor Semantic Understanding” (Liao et al., 2024). A related but differently named multimodal style transfer system is “M2StyleGS: Multi‑Modality 3D Style Transfer with Gaussian Splatting” (Miao et al., 4 Apr 2026).

Accordingly, in the 2025 style‑transfer literature, “ClipGS” most precisely denotes the CLIPGaussian or CLIPGaussians framework of multimodal Gaussian Splatting stylization, whereas in adjacent literatures the same or closely similar label denotes unrelated Gaussian Splatting systems for medical visualization, semantic understanding, or multimodal representation learning. This naming overlap has become a small but genuine source of bibliographic ambiguity.