MMOne: 3D Multimodal Gaussian Scene Representation

• The paper introduces a 3D Gaussian-based framework that unifies multimodal scene rendering with modality-specific opacities.
• It addresses property and granularity disparities by decoupling geometry from modality-specific visibility and adapting Gaussian decomposition.
• Empirical results demonstrate improved PSNR, SSIM, and mIoU across RGB–Thermal–Language settings while reducing the overall Gaussian count.

Searching arXiv for the exact MMOne paper and closely related multimodal 3DGS work to support the article. MMOne is a 3D Gaussian–based framework for learning one scene representation capable of rendering multiple modalities from arbitrary viewpoints, including RGB images, thermal infrared images, language features for open-vocabulary queries, and, in a scalability experiment, monocular depth (Gu et al., 15 Jul 2025). It is designed for multimodal scene representation under two explicit sources of cross-modal conflict—property disparity and granularity disparity—and addresses them by combining a shared geometric backbone with modality-specific opacities, modality-specific features, and a multimodal decomposition mechanism that separates multi-modal Gaussians into single-modal Gaussians when optimization signals diverge across modalities (Gu et al., 15 Jul 2025).

1. Conceptual scope and problem formulation

MMOne studies multimodal scene representation in a single static 3D scene learned from multi-view multimodal observations with known camera poses (Gu et al., 15 Jul 2025). The central objective is to construct a single 3D representation that can be rendered into several modalities simultaneously while preserving modality-specific physical and semantic behavior. In the reported experiments, the core modality combinations are RGB–Thermal, RGB–Language, and RGB–Thermal–Language, with an additional RGB–Thermal–Language–Depth setting used to test extensibility (Gu et al., 15 Jul 2025).

The framework identifies two distinct challenges. The first is property disparity, meaning that different modalities have different feature dimensionalities and different physical behavior. The examples given are that RGB is high-frequency, thermal is smooth, and language is piecewise constant at object or part level; further, a sheet of paper may occlude a hot cup in RGB and language but not in thermal (Gu et al., 15 Jul 2025). The second is granularity disparity, meaning that different modalities require different effective resolutions in the underlying primitive set: RGB benefits from many small Gaussians, thermal from fewer and coarser Gaussians, and language from still coarser, object-level support (Gu et al., 15 Jul 2025).

Prior multimodal 3D Gaussian methods are described as typically using one set of Gaussians with a shared opacity while attaching modality-specific feature vectors to each Gaussian (Gu et al., 15 Jul 2025). MMOne is formulated precisely against that design pattern. Its core claim is that forcing all modalities to share exactly the same opacity and geometric primitives creates redundant scene representations, induces cross-modal conflicts during optimization, and makes scaling to additional modalities difficult (Gu et al., 15 Jul 2025). The method therefore preserves shared geometry only where modalities agree, and otherwise permits modality-specific visibility and modality-specific refinement (Gu et al., 15 Jul 2025).

2. Underlying representation and multimodal rendering

MMOne is built on 3D Gaussian Splatting and represents a scene as a set of 3D Gaussians

$\mathcal{G} = \{ g_i \mid i = 1,\dots,N \},$

where each Gaussian $g_i$ has a mean position $\mu_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, modality-dependent opacity $\alpha_i^m$, and modality-specific features $m_i^m \in \mathbb{R}^{d_m}$ (Gu et al., 15 Jul 2025). The 3D Gaussian density is

$$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$

For a camera view, Gaussians are projected to the image plane, sorted by depth, and rasterized with alpha compositing (Gu et al., 15 Jul 2025).

The decisive modification relative to standard single-modality 3DGS is the use of modality-specific opacities, called modality indicators, together with modality-specific features. For modality $M$, MMOne renders

$$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$

This yields a separate transmittance $T_i^M$ for each modality because the compositing weights themselves depend on the modality-specific opacities $g_i$0 (Gu et al., 15 Jul 2025).

In the reported implementation, modality modeling is parameter-level rather than network-heavy. RGB uses SH color coefficients as in 3DGS, thermal uses an SH-like parameterization following ThermalGaussian, and language uses per-Gaussian low-dimensional features following LangSplat’s language rasterization; modality indicators are learned scalar parameters $g_i$1 for each Gaussian and modality (Gu et al., 15 Jul 2025). This design keeps a shared geometric backbone—positions and covariances—but decouples visibility and signal content across modalities (Gu et al., 15 Jul 2025).

3. Modality indicators and the handling of property disparity

The modality indicator for modality $g_i$2 on Gaussian $g_i$3 is the scalar $g_i$4 (Gu et al., 15 Jul 2025). It has two explicitly stated roles. First, it acts as the per-modality alpha weight in the rendering equation, controlling how much a Gaussian contributes to modality $g_i$5 at each pixel (Gu et al., 15 Jul 2025). Second, it functions as a modality switch during pruning and decomposition: if $g_i$6 is effectively off, that Gaussian does not participate in rendering or gradient updates for modality $g_i$7 (Gu et al., 15 Jul 2025).

This mechanism is the primary response to property disparity. Because RGB, thermal, and language need not share the same visibility structure, a Gaussian can be highly opaque in one modality and nearly transparent in another (Gu et al., 15 Jul 2025). The paper gives the explicit motivation that forcing language and RGB to share both geometry and opacity causes language supervision to distort geometry optimized for photorealistic RGB rendering, and that thermal may need different visibility behavior because of modality-dependent occlusion (Gu et al., 15 Jul 2025).

MMOne uses these modality indicators to permit both shared components and modality-specific behavior. A Gaussian with high $g_i$8, $g_i$9, and $\mu_i \in \mathbb{R}^3$0 acts as a shared multimodal primitive, whereas a Gaussian with only one large indicator is effectively single-modal (Gu et al., 15 Jul 2025). The paper further reports that the distribution of indicator differences, such as $\mu_i \in \mathbb{R}^3$1 and $\mu_i \in \mathbb{R}^3$2, is non-trivial, which is presented as empirical evidence that each modality carves out its own effective subset of Gaussians (Gu et al., 15 Jul 2025).

A plausible implication is that MMOne performs a primitive-level separation between modality agreement and modality disagreement. The paper itself phrases this as disentangling multimodal information into shared and modality-specific components, but the mechanism is implemented through indicators, decomposition, and pruning rather than through an explicit latent-variable factorization (Gu et al., 15 Jul 2025).

4. Multimodal decomposition and granularity-aware optimization

The multimodal decomposition mechanism addresses granularity disparity and optimization conflict directly (Gu et al., 15 Jul 2025). A Gaussian that is currently shared across modalities can be split into several single-modal Gaussians when gradients indicate that different modalities want the Gaussian to change in conflicting ways (Gu et al., 15 Jul 2025).

Let $\mu_i \in \mathbb{R}^3$3 denote the accumulated gradient of Gaussian $\mu_i \in \mathbb{R}^3$4 with respect to modality $\mu_i \in \mathbb{R}^3$5. For two modalities $\mu_i \in \mathbb{R}^3$6, the paper defines the gradient difference

$\mu_i \in \mathbb{R}^3$7

using the L2 norm in practice (Gu et al., 15 Jul 2025). If $\mu_i \in \mathbb{R}^3$8 exceeds a threshold $\mu_i \in \mathbb{R}^3$9, set to $\Sigma_i \in \mathbb{R}^{3 \times 3}$0 in the experiments, the Gaussian is decomposed into several single-modal Gaussians (Gu et al., 15 Jul 2025). Geometry parameters are copied as initialization, indicators are reassigned so that each resulting Gaussian is active for only one modality or a subset, and subsequent optimization lets them diverge (Gu et al., 15 Jul 2025).

This mechanism is integrated into the standard 3DGS densification process. Standard 3DGS clones or splits Gaussians based on gradient magnitude, whereas MMOne adds a cross-modal gradient criterion to decide whether the offspring should remain shared or become modality-specific (Gu et al., 15 Jul 2025). The stated purpose is to avoid forcing one Gaussian to simultaneously satisfy conflicting RGB, thermal, and language updates (Gu et al., 15 Jul 2025).

Pruning is likewise modified. Standard hard pruning is unsuitable in the multimodal setting because a Gaussian may be unimportant for one modality but essential for another (Gu et al., 15 Jul 2025). MMOne therefore introduces soft prune: rather than removing the Gaussian entirely, the corresponding modality indicator is switched off for the modality that no longer needs it (Gu et al., 15 Jul 2025). The Gaussian remains available to the remaining modalities. To control redundancy, the pruning threshold for single-modal Gaussians is raised, encouraging removal of weak modality-specific primitives and consolidation into shared components when possible (Gu et al., 15 Jul 2025).

The paper frames the resulting representation as an emergent separation between shared and modality-specific components. Shared components are Gaussians with significant indicators for multiple modalities and shared geometry $\Sigma_i \in \mathbb{R}^{3 \times 3}$1, while modality-specific components are single-modal Gaussians produced by decomposition or soft pruning (Gu et al., 15 Jul 2025). This factorization is not imposed by an explicit regularizer; it emerges from rendering losses, modality indicators, and gradient-driven decomposition (Gu et al., 15 Jul 2025).

5. Training objectives, implementation, and extensibility

The global objective is a weighted sum of modality-specific losses,

$\Sigma_i \in \mathbb{R}^{3 \times 3}$2

where $\Sigma_i \in \mathbb{R}^{3 \times 3}$3 is the set of active modalities (Gu et al., 15 Jul 2025).

For RGB, MMOne uses a standard 3DGS photometric reconstruction loss and sets $\Sigma_i \in \mathbb{R}^{3 \times 3}$4 in all experiments (Gu et al., 15 Jul 2025). For thermal, it follows ThermalGaussian and combines a reconstruction term with a smoothness regularization term weighted by $\Sigma_i \in \mathbb{R}^{3 \times 3}$5, with $\Sigma_i \in \mathbb{R}^{3 \times 3}$6 in two- and three-modality settings (Gu et al., 15 Jul 2025). For language, it follows LangSplat: ground-truth feature maps are built from SAM segments and CLIP embeddings, and the rendered language feature image is matched to those maps using an L1 loss; the reported weights are $\Sigma_i \in \mathbb{R}^{3 \times 3}$7 in RGB–Language and $\Sigma_i \in \mathbb{R}^{3 \times 3}$8 in RGB–Thermal–Language (Gu et al., 15 Jul 2025). In the scalability experiment, depth supervision is added through an L1 loss to a monocular depth estimate (Gu et al., 15 Jul 2025).

Implementation remains close to the 3DGS codebase. Initial Gaussians come from an SfM sparse point cloud using COLMAP; optimization uses PyTorch with the same optimizer family as 3DGS, described as Adam with default learning rates; training runs for 30K iterations on a single NVIDIA RTX 4090 GPU (Gu et al., 15 Jul 2025). The decomposition threshold is $\Sigma_i \in \mathbb{R}^{3 \times 3}$9, and the single-modal pruning threshold is $\alpha_i^m$0 (Gu et al., 15 Jul 2025). The main architectural changes are in the CUDA rasterizer so that it can render multiple modalities using modality-specific opacities and respect the modality switch used for soft pruning (Gu et al., 15 Jul 2025).

The framework is explicitly intended to be extensible. Adding a new modality requires adding a new feature vector $\alpha_i^m$1, a new indicator $\alpha_i^m$2, a rasterization head for the modality, and a corresponding supervision term in the global loss (Gu et al., 15 Jul 2025). The rest of the system—shared geometry, decomposition, and pruning—remains unchanged (Gu et al., 15 Jul 2025). The reported RGB–Thermal–Language–Depth experiment is presented as evidence that this extension mechanism does not degrade the existing modalities (Gu et al., 15 Jul 2025).

6. Empirical results, ablations, and limitations

MMOne is evaluated on RGBT-Scenes and the LERF dataset extension used by LangSplat (Gu et al., 15 Jul 2025). RGBT-Scenes contains real-world indoor and outdoor scenes with aligned RGB and thermal images, with 10 scenes and approximately 1000 images each; LERF is used for RGB–Language evaluation with ground-truth masks for text queries and additional challenging localization samples (Gu et al., 15 Jul 2025). The baselines include 3DGS, ThermalGaussian, LangSplat, a joint LangSplat baseline denoted LS-J, Feature-3DGS, and a joint RGB–Thermal–Language baseline denoted MM-J (Gu et al., 15 Jul 2025).

In the RGB–Thermal setting, MMOne improves over ThermalGaussian on average from RGB PSNR $\alpha_i^m$3 to $\alpha_i^m$4 and thermal PSNR $\alpha_i^m$5 to $\alpha_i^m$6, while reducing the average number of Gaussians from $\alpha_i^m$7 to $\alpha_i^m$8 (Gu et al., 15 Jul 2025). In the RGB–Language setting on LERF, MMOne improves over LangSplat from RGB PSNR $\alpha_i^m$9 to $m_i^m \in \mathbb{R}^{d_m}$0, mIoU $m_i^m \in \mathbb{R}^{d_m}$1 to $m_i^m \in \mathbb{R}^{d_m}$2, and localization accuracy $m_i^m \in \mathbb{R}^{d_m}$3 to $m_i^m \in \mathbb{R}^{d_m}$4 (Gu et al., 15 Jul 2025). Against the joint shared-opacity baseline LS-J, MMOne improves both RGB and language simultaneously, from PSNR $m_i^m \in \mathbb{R}^{d_m}$5 to $m_i^m \in \mathbb{R}^{d_m}$6, mIoU $m_i^m \in \mathbb{R}^{d_m}$7 to $m_i^m \in \mathbb{R}^{d_m}$8, and accuracy $m_i^m \in \mathbb{R}^{d_m}$9 to $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$0 (Gu et al., 15 Jul 2025).

In the RGB–Thermal–Language setting, MMOne improves over MM-J on the four reported scenes from RGB PSNR $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$1 to $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$2, RGB SSIM $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$3 to $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$4, thermal PSNR $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$5 to $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$6, thermal SSIM $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$7 to $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$8, and language mIoU $$g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).$$9 to $M$0 (Gu et al., 15 Jul 2025). The paper emphasizes the modality-conflict comparison: adding language to ThermalGaussian-style shared-opacity joint training reduces RGB PSNR by $M$1 dB and thermal PSNR by $M$2 dB, whereas MMOne slightly improves both when moving from RGB–Thermal to RGB–Thermal–Language, from RGB PSNR $M$3 to $M$4 and thermal PSNR $M$5 to $M$6 (Gu et al., 15 Jul 2025).

The ablation study decomposes the contribution of the major components. Starting from the shared-opacity joint baseline MM-J, adding the modality modeling module alone improves RGB PSNR from $M$7 to $M$8, thermal PSNR from $M$9 to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$0, and language mIoU from $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$1 to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$2, while reducing the number of Gaussians from $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$3 to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$4 in units of $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$5 (Gu et al., 15 Jul 2025). Hard prune further improves mIoU to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$6 and reduces Gaussians to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$7; soft prune improves RGB PSNR to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$8, thermal PSNR to $$\begin{aligned} M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \ T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)). \end{aligned} \tag{1}$$9, mIoU to $T_i^M$0, and reduces Gaussians to $T_i^M$1 (Gu et al., 15 Jul 2025). Full decomposition yields the best overall results: RGB PSNR $T_i^M$2, thermal PSNR $T_i^M$3, language mIoU $T_i^M$4, and the lowest Gaussian count, $T_i^M$5 (Gu et al., 15 Jul 2025). Threshold ablations over $T_i^M$6 are reported as stable, with $T_i^M$7 the best setting (Gu et al., 15 Jul 2025).

The limitations are explicitly bounded. Current experiments assume static scenes and given camera poses from SfM; future directions named in the paper include COLMAP-free Gaussian learning and dynamic-scene extensions (Gu et al., 15 Jul 2025). The representation does not introduce an explicit high-level semantic latent disentangling beyond Gaussian-level decomposition, and the method relies on heuristic thresholds and loss weights even though ablations indicate robustness (Gu et al., 15 Jul 2025). Evaluation is limited to RGB, thermal, language, and depth, with broader sensory modalities left for future work (Gu et al., 15 Jul 2025).

Within the broader literature on multimodal 3D representations, MMOne is positioned as retaining the efficiency of the 3DGS regime while changing where sharing occurs: geometry is shared unless gradients diverge, whereas visibility and effective primitive density are modality-specific (Gu et al., 15 Jul 2025). This suggests a primitive-level factorization of multimodal structure that is physically grounded by rendering and gradient behavior rather than imposed through a separate latent-variable model (Gu et al., 15 Jul 2025).