Multimodal Gaussian Splatting

• Multimodal Gaussian splatting is an extension of 3D Gaussian Splatting that incorporates additional modalities such as thermal, LiDAR, and semantic cues to enrich scene representation.
• It augments Gaussian primitives with extra parameters and modality-specific channels, enabling improved initialization, cross-modal fusion, and robust depth and semantic estimation.
• The framework achieves enhanced rendering quality and performance, with demonstrated improvements in PSNR, SSIM, and computational efficiency across applications from autonomous driving to medical imaging.

Searching arXiv for papers on multimodal Gaussian splatting and related variants. Multimodal Gaussian splatting denotes a set of extensions to 3D Gaussian Splatting (3DGS) in which Gaussian primitives, rendering, optimization, or downstream inference are coupled to more than one modality. In the surveyed literature, those modalities include RGB, thermal infrared, LiDAR, radar or RF depth, monocular depth priors, semantic labels, language, style prompts, and medical imaging sequences. The common substrate remains an explicit Gaussian scene representation rendered by projecting anisotropic or isotropic splats into image space and compositing them by alpha blending, but the function of the additional modality varies substantially: it may initialize geometry, add modality-specific channels, regularize depth and semantics, drive hierarchical sampling, or provide a prior for editing, retrieval, SLAM, or manipulation (Xiong et al., 3 Mar 2026, Lu et al., 2024, Xie et al., 14 Oct 2025, Gau et al., 19 Feb 2026).

1. Formal basis and Gaussian parameterization

Most multimodal variants preserve the canonical 3DGS primitive while extending its attribute set. In ThermalGaussian, each 3D Gaussian $G_k$ is parameterized by a mean $\mu_k\in\mathbb{R}^3$, an anisotropic covariance $\Sigma_k\in\mathbb{R}^{3\times3}$, an opacity $\alpha_k\in[0,1]$, spherical-harmonic coefficients $c_k$ for RGB color, and spherical-harmonic coefficients $t_k$ for thermal intensity, with

$$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$

Rendering uses modality-specific alpha blending,

$$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$

so the same geometric carrier can emit both visible and thermal outputs (Lu et al., 2024).

Other systems enlarge the primitive to encode geometry, semantics, or pruning state directly. UniGS uses

$G=\{\mu,q,s,\alpha,h,o,k\},$

where $q$ is a unit quaternion, $\mu_k\in\mathbb{R}^3$0 is diagonal scale, $\mu_k\in\mathbb{R}^3$1 stores spherical-harmonic RGB coefficients, $\mu_k\in\mathbb{R}^3$2 stores semantic logits, and $\mu_k\in\mathbb{R}^3$3 is a learnable “gradient factor” for pruning; its ellipsoid covariance is $\mu_k\in\mathbb{R}^3$4 (Xie et al., 14 Oct 2025). GS3LAM similarly represents the scene as a Semantic Gaussian Field,

$\mu_k\in\mathbb{R}^3$5

where $\mu_k\in\mathbb{R}^3$6 is a low-dimensional semantic feature later decoded into semantic probabilities through a CNN (Li et al., 29 Mar 2026).

A more radical extension appears in MedGS, which lifts the primitive into space plus slice-time. A Folded-Gaussian has mean $\mu_k\in\mathbb{R}^3$7, covariance $\mu_k\in\mathbb{R}^3$8, opacity coefficient $\mu_k\in\mathbb{R}^3$9, grayscale coefficient $\Sigma_k\in\mathbb{R}^{3\times3}$0, and learnable time-conditioning polynomials $\Sigma_k\in\mathbb{R}^{3\times3}$1 and $\Sigma_k\in\mathbb{R}^{3\times3}$2. This permits direct interpolation between medical slices and later iso-surface extraction from densely sampled renders (Marzol et al., 20 Sep 2025).

These variants show that “multimodal” does not imply a single standardized primitive. Rather, the literature uses Gaussian splats as an explicit carrier whose attribute space can be extended toward thermal radiance, semantic structure, temporal interpolation, or pruning diagnostics.

2. Cross-modal fusion mechanisms and optimization objectives

A central design axis is how modalities are fused during training. In sparse-view novel-view synthesis, "Multimodal-Prior-Guided Importance Sampling for Hierarchical Gaussian Splatting in Sparse-View Novel View Synthesis" defines a local recoverability score

$\Sigma_k\in\mathbb{R}^{3\times3}$3

where $\Sigma_k\in\mathbb{R}^{3\times3}$4 is an $\Sigma_k\in\mathbb{R}^{3\times3}$5 photometric rendering residual, $\Sigma_k\in\mathbb{R}^{3\times3}$6 uses a pretrained ResNet-18 segmentation head with boundary and foreground boosts, and $\Sigma_k\in\mathbb{R}^{3\times3}$7 combines monocular depth gradients from DPT with second-order depth curvature. The paper sets $\Sigma_k\in\mathbb{R}^{3\times3}$8, $\Sigma_k\in\mathbb{R}^{3\times3}$9, and $\alpha_k\in[0,1]$0, masks unreliable regions through $\alpha_k\in[0,1]$1, and samples new fine Gaussians from the resulting probability distribution. A protection mechanism clamps the opacity of newly injected fine Gaussians for $\alpha_k\in[0,1]$2 iterations, preventing premature pruning in underconstrained regions (Xiong et al., 3 Mar 2026).

ThermalGaussian instead formulates joint RGB–thermal learning through a dynamic modality-balancing loss,

$\alpha_k\in[0,1]$3

Its RGB branch uses an $\alpha_k\in[0,1]$4 plus $\alpha_k\in[0,1]$5-SSIM loss, while the thermal branch adds a thermal-specific smoothness term

$\alpha_k\in[0,1]$6

reflecting the physical continuity of thermal fields. The same framework introduces multimodal regularization constraints to avoid overfitting either modality and reports that these constraints reduced the model's storage cost by $\alpha_k\in[0,1]$7 (Lu et al., 2024).

ThermoSplat argues that shared representations alone fail to adaptively handle structural correlations and physical discrepancies between spectrums. It therefore conditions RGB synthesis on thermal priors through Cross-Modal FiLM Modulation,

$\alpha_k\in[0,1]$8

with $\alpha_k\in[0,1]$9, and decouples visible and thermal geometry through a learned thermal opacity offset,

$c_k$0

Its RGB output is hybrid: an explicit spherical-harmonic rendering $c_k$1 is summed with an implicit residual $c_k$2 (Su et al., 22 Jan 2026).

In autonomous-driving reconstruction, LT-Gaussian fuses LiDAR submaps, multi-view images, relative depth from Depth Anything V2, and sky masks from Mask2Former. It optimizes

$c_k$3

where $c_k$4 is a Pearson-correlation loss between rendered depth and non-metric relative depth priors, and $c_k$5 ignores sky regions to suppress spurious far-field Gaussians (Cheng et al., 3 Aug 2025).

3. Geometry-aware rendering and multimodal scene fields

A second major line of work makes multiple modalities part of the rendered state itself rather than merely auxiliary supervision. UniGS renders color, depth, normals, semantic logits, and a pruning map simultaneously. Its key geometric departure is differentiable depth rendering via ray–ellipsoid intersection rather than Gaussian centers. For a pixel ray $c_k$6, the method solves the ellipsoid intersection in scaled local coordinates, takes the midpoint

$c_k$7

and reprojects it into camera-space depth. This yields analytic gradients with respect to $c_k$8, rotation, and scale. UniGS also derives surface normals from the implicit ellipsoid gradient

$c_k$9

and defines a total objective

$t_k$0

with $t_k$1 and $t_k$2 in its experiments (Xie et al., 14 Oct 2025).

GS3LAM transfers the same principle into online mapping. Its Semantic Gaussian Field composes color, depth, and semantic features through front-to-back alpha blending,

$t_k$3

and alternates between pose tracking and field mapping under photometric, geometric, and semantic constraints. Two auxiliary mechanisms are distinctive. Depth-adaptive Scale Regularization penalizes Gaussians that are too large or too small relative to the global scale distribution, while Random Sampling-based Keyframe Mapping mitigates catastrophic forgetting by pairing the current frame with randomly sampled keyframes instead of relying on local covisibility alone (Li et al., 29 Mar 2026).

GSPR uses explicit multimodal Gaussian scenes as input to a descriptor network for place recognition. LiDAR returns determine Gaussian positions, multi-view RGB images initialize the spherical-harmonic appearance, pseudo-points are added beyond LiDAR range to suppress distant floaters, and Mask2Former-based mixed masking removes or detaches unstable static or dynamic image regions during 3DGS training. The optimized Gaussian scene is then voxelized, passed through a 3D graph convolution backbone, processed by a transformer with learnable positional embeddings, and aggregated by NetVLAD into a 256-dimensional place descriptor (Qi et al., 2024).

These systems indicate that multimodal Gaussian splatting increasingly treats geometry, semantics, and modality-specific outputs as co-rendered observables, not just hidden regularizers.

4. Sensor-driven initialization, calibration, and mapping

Many multimodal pipelines use non-visual sensors to establish metric geometry before photometric optimization. LiDAR-3DGS exemplifies the simplest form: color-mapped LiDAR point clouds are aligned with image-based structure-from-motion points and then fed into an otherwise unmodified 3DGS optimizer. The paper uses an Ouster OS0-32 LiDAR, FLIR Blackfly S RGB camera, ROS-based calibration, ChromaFilter subsampling, CloudCompare for coarse alignment, and two-stage ICP for fine registration. At $t_k$4 iterations, the best trade-off appears at LiDAR density $t_k$5, with PSNR increasing from $t_k$6 to $t_k$7 and SSIM from $t_k$8 to $t_k$9, corresponding to $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$0 PSNR and $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$1 SSIM (Lim et al., 2024).

3DGS-Calib turns Gaussian splatting into a calibration engine. Gaussian centers are initialized from an accumulated LiDAR point cloud and kept fixed, while per-Gaussian opacity, scale, rotation, and color are predicted by a shared MLP. Camera extrinsics $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$2 and time offsets $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$3 are optimized jointly with the Gaussian parameters under a photometric loss and a scale regularizer. The method uses a coarse-to-fine voxel schedule of $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$4 cm, $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$5 cm, and $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$6 cm, warms up the hash grid for $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$7 steps, crops images to the bottom half, and reports, on KITTI-360, a final median error of $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$8 rotation, $$G_k(x)=\exp\!\Bigl(-\tfrac12\,(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k)\Bigr),\qquad \Sigma_k = R_k\,S_k\,S_k^T\,R_k^T.$$9 cm translation, and $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$0 ms time offset in approximately $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$1 s, versus $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$2 slower NeRF-based alternatives (Herau et al., 2024).

RF-informed initialization appears in "3D Scene Rendering with Multimodal Gaussian Splatting". There, a single automotive-radar sweep is interpolated by localized Gaussian processes over partitioned angular regions, producing a dense 3D point cloud from sparse RF depth. The localized GP improves single-sweep depth MAE from $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$3 m to $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$4 m and reduces runtime from $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$5 s to $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$6 s on CPU. When used to initialize Gaussian splats for the View-of-Delft scene with $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$7 training views, multimodal GS improves LPIPS from $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$8 to $$C(x')=\sum_{k\in N(x')} c_k\,\alpha_k\prod_{j<k}(1-\alpha_j),\qquad T(x')=\sum_{k\in N(x')} t_k\,\alpha_k\prod_{j<k}(1-\alpha_j),$$9, SSIM from $G=\{\mu,q,s,\alpha,h,o,k\},$0 to $G=\{\mu,q,s,\alpha,h,o,k\},$1, and PSNR from $G=\{\mu,q,s,\alpha,h,o,k\},$2 dB to $G=\{\mu,q,s,\alpha,h,o,k\},$3 dB, while cutting initialization time from approximately $G=\{\mu,q,s,\alpha,h,o,k\},$4 min for COLMAP SfM to approximately $G=\{\mu,q,s,\alpha,h,o,k\},$5 s (Gau et al., 19 Feb 2026).

ReefMapGS embeds 3DGS in a multimodal underwater SLAM loop. Its base pose graph combines IMU, DVL velocities, pressure, RGB images, and AprilTag landmark detections; reconstruction then proceeds ring by ring from a high-certainty seed region. New camera poses are locally refined by minimizing

$G=\{\mu,q,s,\alpha,h,o,k\},$6

against the current 3DGS model, new Gaussians are initialized from DepthAnythingV2 pseudo-metric depth, and refined poses are reinserted into the factor graph as external priors $G=\{\mu,q,s,\alpha,h,o,k\},$7. On the Tektite and Yawzi reef surveys, ReefMapGS reports ATE RMSE of $G=\{\mu,q,s,\alpha,h,o,k\},$8 m and $G=\{\mu,q,s,\alpha,h,o,k\},$9 m, respectively, and reconstruction quality of $q$0 and $q$1 in PSNR/SSIM/LPIPS/depth-RMSE, while running in $q$2 and $q$3, compared with $q$4 and $q$5 for COLMAP SfM (Yang et al., 13 Apr 2026).

5. Semantic, generative, and downstream uses

Not all multimodal Gaussian splatting is reconstruction-centric. CLIP-GS treats 3DGS itself as a multimodal representation to be aligned with CLIP’s image–text space. Each Gaussian is serialized as a 14-dimensional token, grouped into local patches by FPS plus $q$6-NN, ordered by xyz-sort, Hilbert-curve sort, and Z-order, and then processed by a ViT-Base initialized from Uni3D. The model is trained with a 3D–text contrastive loss and an image-voting loss over $q$7 rendered views. On Objaverse-GS, Text$q$83D retrieval improves from $q$9 to $\mu_k\in\mathbb{R}^3$00 in $\mu_k\in\mathbb{R}^3$01, and zero-shot classification on ModelNet-GS improves from $\mu_k\in\mathbb{R}^3$02 to $\mu_k\in\mathbb{R}^3$03 (Jiao et al., 2024).

Stylization frameworks use Gaussian splats as explicit appearance carriers under text or image conditioning. AnyStyle augments a frozen AnySplat feed-forward backbone with a style branch driven by Long-CLIP embeddings and zero-initialized style injectors. At inference, unposed images are reconstructed in one forward pass, the style input is encoded into $\mu_k\in\mathbb{R}^3$04, and only appearance tokens are modulated before rendering stylized novel views. On four scenes and $\mu_k\in\mathbb{R}^3$05 held-out WikiArt styles, AnyStyle$\mu_k\in\mathbb{R}^3$06 reports ArtFID approximately $\mu_k\in\mathbb{R}^3$07 at approximately $\mu_k\in\mathbb{R}^3$08 s per scene, while AnyStyle$\mu_k\in\mathbb{R}^3$09 reports ArtScore approximately $\mu_k\in\mathbb{R}^3$10 and ArtFID approximately $\mu_k\in\mathbb{R}^3$11 (Kaleta et al., 3 Feb 2026). CLIPGaussian generalizes the same idea across 2D images, videos, 3D objects, and 4D scenes by fine-tuning Gaussian parameters with directional CLIP, patch CLIP, VGG content, and background-consistency losses. In 3D text-guided style transfer, it reports CLIP-SIM $\mu_k\in\mathbb{R}^3$12 and CLIP-S $\mu_k\in\mathbb{R}^3$13 with no increase in primitives; in video image-guided stylization, it reports CLIP-SIM $\mu_k\in\mathbb{R}^3$14 and CLIP-S $\mu_k\in\mathbb{R}^3$15 (Howil et al., 28 May 2025).

Diffusion and manipulation systems also use 3DGS as a multimodal prior rather than a final renderer. MultiEditor introduces a dual-branch latent diffusion framework for jointly editing masked camera images and LiDAR range-view data. Its 3DGS priors provide pixel-level pasted RGB and depth conditions, global semantic codes via CLIP, and a depth-guided deformable cross-modality condition module. The reported gains include FID $\mu_k\in\mathbb{R}^3$16 versus $\mu_k\in\mathbb{R}^3$17, LPIPS $\mu_k\in\mathbb{R}^3$18 versus $\mu_k\in\mathbb{R}^3$19, Chamfer Distance $\mu_k\in\mathbb{R}^3$20 versus $\mu_k\in\mathbb{R}^3$21, FPD $\mu_k\in\mathbb{R}^3$22 versus $\mu_k\in\mathbb{R}^3$23, and Depth Alignment Score $\mu_k\in\mathbb{R}^3$24 versus $\mu_k\in\mathbb{R}^3$25 (Lu et al., 29 Jul 2025). RobMRAG uses 3DGS-enhanced Multimodal Retrieval-Augmented Generation for robotic manipulation: after hierarchical text, CLIP, and Instance Matching Distance retrieval, a 3D-aware pose refinement stage aligns a reference Gaussian object to the target and reprojects it for multimodal large-language-model reasoning. On unseen household-object categories, the full system improves average success rate to $\mu_k\in\mathbb{R}^3$26, compared to $\mu_k\in\mathbb{R}^3$27 for the RAM baseline and $\mu_k\in\mathbb{R}^3$28 for the variant without 3D align (Xie et al., 28 Feb 2026).

A compact semantic-structural formulation appears in CUS-GS. It organizes Gaussians in a voxelized anchor scaffold, allocates multimodal features from foundation models including CLIP, DINOv2, and SEEM via memory-bank attention, and uses a feature-aware significance score to drive anchor growing and pruning. The framework reports competitive performance with $\mu_k\in\mathbb{R}^3$29 M parameters, compared with approximately $\mu_k\in\mathbb{R}^3$30 M for the nearest competitor M3 (Ming et al., 22 Nov 2025). In medical imaging, MedGS applies Gaussian splatting per modality rather than jointly across modalities, but it extends the design space to interpolation and mesh reconstruction from ultrasound and MRI sequences; on MRI leave-frame-out interpolation every $\mu_k\in\mathbb{R}^3$31nd frame, it reports PSNR $\mu_k\in\mathbb{R}^3$32 dB, compared with $\mu_k\in\mathbb{R}^3$33 dB for linear interpolation and $\mu_k\in\mathbb{R}^3$34 dB for optical flow (Marzol et al., 20 Sep 2025).

6. Empirical profile, misconceptions, and unresolved issues

A common misconception is that multimodal Gaussian splatting is confined to RGB–thermal fusion. The surveyed literature spans sparse-view hierarchical densification, RGB–thermal rendering, RGB–depth–normal–semantic reconstruction, LiDAR- and RF-initialized scene reconstruction, SLAM, calibration, place recognition, stylization, diffusion-based editing, robotic manipulation, compact semantic scene representations, and medical interpolation (Xiong et al., 3 Mar 2026, Su et al., 22 Jan 2026, Li et al., 29 Mar 2026, Qi et al., 2024).

Another misconception is that multimodality always implies a shared geometry for all branches. Several papers explicitly challenge that assumption. ThermoSplat introduces modality-adaptive geometric decoupling because visible and thermal sensors can exhibit different occlusions and transparencies, while UniGS shows that rendering depth from Gaussian centers is inferior to ray–ellipsoid midpoint depth and reports that replacing center-depth with ray–ellipsoid midpoint reduces depth-error by more than $\mu_k\in\mathbb{R}^3$35 (Su et al., 22 Jan 2026, Xie et al., 14 Oct 2025).

The empirical profile is heterogeneous but consistently favors multimodal priors when the auxiliary modality contributes metric structure or robust semantics. The sparse-view hierarchical sampler reports an average PSNR of $\mu_k\in\mathbb{R}^3$36 dB on DTU with $\mu_k\in\mathbb{R}^3$37 views, surpassing NexusGS at $\mu_k\in\mathbb{R}^3$38 dB by $\mu_k\in\mathbb{R}^3$39 dB (Xiong et al., 3 Mar 2026). ThermalGaussian reports $\mu_k\in\mathbb{R}^3$40 dB thermal PSNR and $\mu_k\in\mathbb{R}^3$41 dB RGB PSNR on RGBT-Scenes, while ThermoSplat reports $\mu_k\in\mathbb{R}^3$42 dB RGB and $\mu_k\in\mathbb{R}^3$43 dB thermal PSNR on the same dataset (Lu et al., 2024, Su et al., 22 Jan 2026). UniGS reports RGB novel-view PSNR of approximately $\mu_k\in\mathbb{R}^3$44 dB on Replica, semantic mIoU of approximately $\mu_k\in\mathbb{R}^3$45, and rendering speed above $\mu_k\in\mathbb{R}^3$46 FPS on an RTX 4090, while GS3LAM reports $\mu_k\in\mathbb{R}^3$47 FPS RGB/depth/semantic rendering at $\mu_k\in\mathbb{R}^3$48 on a single RTX 3090 (Xie et al., 14 Oct 2025, Li et al., 29 Mar 2026).

Limitations remain domain-specific. ThermalGaussian assumes static scenes and notes that thermal resolution $\mu_k\in\mathbb{R}^3$49 limits fine-detail recovery (Lu et al., 2024). 3DGS-Calib assumes a static scene and requires an initial LiDAR trajectory (Herau et al., 2024). GSPR notes that dynamic objects are only masked rather than explicitly modeled, and that the Multimodal Gaussian Splatting stage remains costly at $\mu_k\in\mathbb{R}^3$50 iterations of full 3D-GS (Qi et al., 2024). LT-Gaussian is motivated precisely by the observation that generating Gaussian scenes incurs substantial time and computational cost, making long-term map updating difficult; its update formulation addresses this by reusing old Gaussians, which the paper states can slash training time by approximately $\mu_k\in\mathbb{R}^3$51 while improving quality over reconstruction from scratch (Cheng et al., 3 Aug 2025).

Taken together, the literature suggests that multimodal Gaussian splatting is best understood not as a single algorithm but as an architectural regime. Its central commitment is the use of explicit Gaussian primitives as a shared computational object across sensing, rendering, semantics, and control. The main open questions concern when modalities should share geometry, when they should be decoupled, how compact a unified scene representation can remain without sacrificing semantics, and how to extend current systems from mostly static scenes toward long-term, dynamic, and continuously updated world models (Ming et al., 22 Nov 2025, Yang et al., 13 Apr 2026).