---
title: 'MMOne: 3D Multimodal Gaussian Scene Representation'
url: https://www.emergentmind.com/topics/mmone
type: topic
---

# MMOne: 3D Multimodal Gaussian Scene Representation

Searching arXiv for the exact MMOne paper and closely related multimodal 3DGS work to support the article.
MMOne is a 3D Gaussian–based framework for learning *one* scene representation capable of rendering *multiple* modalities from arbitrary viewpoints, including RGB images, thermal infrared images, language features for open-vocabulary queries, and, in a scalability experiment, monocular depth [2507.11129]. It is designed for multimodal scene representation under two explicit sources of cross-modal conflict—*property disparity* and *granularity disparity*—and addresses them by combining a shared geometric backbone with modality-specific opacities, modality-specific features, and a multimodal decomposition mechanism that separates multi-modal Gaussians into single-modal Gaussians when optimization signals diverge across modalities [2507.11129].

## 1. Conceptual scope and problem formulation

MMOne studies multimodal scene representation in a single static 3D scene learned from multi-view multimodal observations with known camera poses [2507.11129]. The central objective is to construct a single 3D representation that can be rendered into several modalities simultaneously while preserving modality-specific physical and semantic behavior. In the reported experiments, the core modality combinations are RGB–Thermal, RGB–Language, and RGB–Thermal–Language, with an additional RGB–Thermal–Language–Depth setting used to test extensibility [2507.11129].

The framework identifies two distinct challenges. The first is **property disparity**, meaning that different modalities have different feature dimensionalities and different physical behavior. The examples given are that RGB is high-frequency, thermal is smooth, and language is piecewise constant at object or part level; further, a sheet of paper may occlude a hot cup in RGB and language but not in thermal [2507.11129]. The second is **granularity disparity**, meaning that different modalities require different effective resolutions in the underlying primitive set: RGB benefits from many small Gaussians, thermal from fewer and coarser Gaussians, and language from still coarser, object-level support [2507.11129].

Prior multimodal 3D Gaussian methods are described as typically using one set of Gaussians with a shared opacity while attaching modality-specific feature vectors to each Gaussian [2507.11129]. MMOne is formulated precisely against that design pattern. Its core claim is that forcing all modalities to share exactly the same opacity and geometric primitives creates redundant scene representations, induces cross-modal conflicts during optimization, and makes scaling to additional modalities difficult [2507.11129]. The method therefore preserves shared geometry only where modalities agree, and otherwise permits modality-specific visibility and modality-specific refinement [2507.11129].

## 2. Underlying representation and multimodal rendering

MMOne is built on 3D Gaussian Splatting and represents a scene as a set of 3D Gaussians
\[
\mathcal{G} = \{ g_i \mid i = 1,\dots,N \},
\]
where each Gaussian \(g_i\) has a mean position \(\mu_i \in \mathbb{R}^3\), covariance \(\Sigma_i \in \mathbb{R}^{3 \times 3}\), modality-dependent opacity \(\alpha_i^m\), and modality-specific features \(m_i^m \in \mathbb{R}^{d_m}\) [2507.11129]. The 3D Gaussian density is
\[
g_i(x) = \exp\left(-\frac{1}{2}(x - \mu_i)^T \Sigma_i^{-1} (x - \mu_i)\right).
\]
For a camera view, Gaussians are projected to the image plane, sorted by depth, and rasterized with alpha compositing [2507.11129].

The decisive modification relative to standard single-modality 3DGS is the use of **modality-specific opacities**, called *modality indicators*, together with modality-specific features. For modality \(M\), MMOne renders
\[
\begin{aligned}
M(x) &= \sum_{i=1}^N T_i^M \cdot \alpha_i^M \cdot g_i^{2D}(x) \cdot m_i^M, \\
T_i^M &= \prod_{j=1}^{i-1} (1 - \alpha_j^M \cdot g_j^{2D}(x)).
\end{aligned}
\tag{1}
\]
This yields a separate transmittance \(T_i^M\) for each modality because the compositing weights themselves depend on the modality-specific opacities \(\alpha_i^M\) [2507.11129].

In the reported implementation, modality modeling is parameter-level rather than network-heavy. RGB uses SH color coefficients as in 3DGS, thermal uses an SH-like parameterization following ThermalGaussian, and language uses per-Gaussian low-dimensional features following LangSplat’s language rasterization; modality indicators are learned scalar parameters \(\alpha_i^M\) for each Gaussian and modality [2507.11129]. This design keeps a shared geometric backbone—positions and covariances—but decouples visibility and signal content across modalities [2507.11129].

## 3. Modality indicators and the handling of property disparity

The modality indicator for modality \(M\) on Gaussian \(g_i\) is the scalar \(\alpha_i^M \in [0,1]\) [2507.11129]. It has two explicitly stated roles. First, it acts as the per-modality alpha weight in the rendering equation, controlling how much a Gaussian contributes to modality \(M\) at each pixel [2507.11129]. Second, it functions as a modality switch during pruning and decomposition: if \(\alpha_i^M\) is effectively off, that Gaussian does not participate in rendering or gradient updates for modality \(M\) [2507.11129].

This mechanism is the primary response to property disparity. Because RGB, thermal, and language need not share the same visibility structure, a Gaussian can be highly opaque in one modality and nearly transparent in another [2507.11129]. The paper gives the explicit motivation that forcing language and RGB to share both geometry and opacity causes language supervision to distort geometry optimized for photorealistic RGB rendering, and that thermal may need different visibility behavior because of modality-dependent occlusion [2507.11129].

MMOne uses these modality indicators to permit both **shared components** and **modality-specific behavior**. A Gaussian with high \(\alpha_i^R\), \(\alpha_i^T\), and \(\alpha_i^L\) acts as a shared multimodal primitive, whereas a Gaussian with only one large indicator is effectively single-modal [2507.11129]. The paper further reports that the distribution of indicator differences, such as \(\alpha^R - \alpha^T\) and \(\alpha^R - \alpha^L\), is non-trivial, which is presented as empirical evidence that each modality carves out its own effective subset of Gaussians [2507.11129].

A plausible implication is that MMOne performs a primitive-level separation between modality agreement and modality disagreement. The paper itself phrases this as disentangling multimodal information into shared and modality-specific components, but the mechanism is implemented through indicators, decomposition, and pruning rather than through an explicit latent-variable factorization [2507.11129].

## 4. Multimodal decomposition and granularity-aware optimization

The multimodal decomposition mechanism addresses **granularity disparity** and optimization conflict directly [2507.11129]. A Gaussian that is currently shared across modalities can be split into several single-modal Gaussians when gradients indicate that different modalities want the Gaussian to change in conflicting ways [2507.11129].

Let \(g_{m_i}\) denote the accumulated gradient of Gaussian \(g_i\) with respect to modality \(m\). For two modalities \(m_i, m_j\), the paper defines the gradient difference
\[
gd_{ij} = \text{norm}(g_{m_i} - g_{m_j}),
\tag{2}
\]
using the L2 norm in practice [2507.11129]. If \(gd_{ij}\) exceeds a threshold \(\tau\), set to \(0.0002\) in the experiments, the Gaussian is decomposed into several single-modal Gaussians [2507.11129]. Geometry parameters are copied as initialization, indicators are reassigned so that each resulting Gaussian is active for only one modality or a subset, and subsequent optimization lets them diverge [2507.11129].

This mechanism is integrated into the standard 3DGS densification process. Standard 3DGS clones or splits Gaussians based on gradient magnitude, whereas MMOne adds a cross-modal gradient criterion to decide whether the offspring should remain shared or become modality-specific [2507.11129]. The stated purpose is to avoid forcing one Gaussian to simultaneously satisfy conflicting RGB, thermal, and language updates [2507.11129].

Pruning is likewise modified. Standard hard pruning is unsuitable in the multimodal setting because a Gaussian may be unimportant for one modality but essential for another [2507.11129]. MMOne therefore introduces **soft prune**: rather than removing the Gaussian entirely, the corresponding modality indicator is switched off for the modality that no longer needs it [2507.11129]. The Gaussian remains available to the remaining modalities. To control redundancy, the pruning threshold for single-modal Gaussians is raised, encouraging removal of weak modality-specific primitives and consolidation into shared components when possible [2507.11129].

The paper frames the resulting representation as an emergent separation between shared and modality-specific components. Shared components are Gaussians with significant indicators for multiple modalities and shared geometry \((\mu_i,\Sigma_i)\), while modality-specific components are single-modal Gaussians produced by decomposition or soft pruning [2507.11129]. This factorization is not imposed by an explicit regularizer; it emerges from rendering losses, modality indicators, and gradient-driven decomposition [2507.11129].

## 5. Training objectives, implementation, and extensibility

The global objective is a weighted sum of modality-specific losses,
\[
\mathcal{L} = \sum_{M \in \mathcal{M}} \lambda_M \, \mathcal{L}_M,
\tag{3}
\]
where \(\mathcal{M}\) is the set of active modalities [2507.11129].

For RGB, MMOne uses a standard 3DGS photometric reconstruction loss and sets \(\lambda_R = 0.5\) in all experiments [2507.11129]. For thermal, it follows ThermalGaussian and combines a reconstruction term with a smoothness regularization term weighted by \(\lambda_{\text{smooth}} = 0.6\), with \(\lambda_T = 0.5\) in two- and three-modality settings [2507.11129]. For language, it follows LangSplat: ground-truth feature maps are built from SAM segments and CLIP embeddings, and the rendered language feature image is matched to those maps using an L1 loss; the reported weights are \(\lambda_L = 0.5\) in RGB–Language and \(\lambda_L = 0.2\) in RGB–Thermal–Language [2507.11129]. In the scalability experiment, depth supervision is added through an L1 loss to a monocular depth estimate [2507.11129].

Implementation remains close to the 3DGS codebase. Initial Gaussians come from an SfM sparse point cloud using COLMAP; optimization uses PyTorch with the same optimizer family as 3DGS, described as Adam with default learning rates; training runs for 30K iterations on a single NVIDIA RTX 4090 GPU [2507.11129]. The decomposition threshold is \(\tau = 0.0002\), and the single-modal pruning threshold is \(0.5\) [2507.11129]. The main architectural changes are in the CUDA rasterizer so that it can render multiple modalities using modality-specific opacities and respect the modality switch used for soft pruning [2507.11129].

The framework is explicitly intended to be extensible. Adding a new modality requires adding a new feature vector \(m_i^{M'}\), a new indicator \(\alpha_i^{M'}\), a rasterization head for the modality, and a corresponding supervision term in the global loss [2507.11129]. The rest of the system—shared geometry, decomposition, and pruning—remains unchanged [2507.11129]. The reported RGB–Thermal–Language–Depth experiment is presented as evidence that this extension mechanism does not degrade the existing modalities [2507.11129].

## 6. Empirical results, ablations, and limitations

MMOne is evaluated on RGBT-Scenes and the LERF dataset extension used by LangSplat [2507.11129]. RGBT-Scenes contains real-world indoor and outdoor scenes with aligned RGB and thermal images, with 10 scenes and approximately 1000 images each; LERF is used for RGB–Language evaluation with ground-truth masks for text queries and additional challenging localization samples [2507.11129]. The baselines include 3DGS, ThermalGaussian, LangSplat, a joint LangSplat baseline denoted LS-J, Feature-3DGS, and a joint RGB–Thermal–Language baseline denoted MM-J [2507.11129].

In the **RGB–Thermal** setting, MMOne improves over ThermalGaussian on average from RGB PSNR \(24.38\) to \(24.89\) and thermal PSNR \(25.51\) to \(25.89\), while reducing the average number of Gaussians from \(32.7 \times 10^4\) to \(12.2 \times 10^4\) [2507.11129]. In the **RGB–Language** setting on LERF, MMOne improves over LangSplat from RGB PSNR \(24.02\) to \(24.35\), mIoU \(47.6\) to \(56.6\), and localization accuracy \(72.4\%\) to \(76.5\%\) [2507.11129]. Against the joint shared-opacity baseline LS-J, MMOne improves both RGB and language simultaneously, from PSNR \(23.23\) to \(24.35\), mIoU \(55.3\) to \(56.6\), and accuracy \(73.5\) to \(76.5\%\) [2507.11129].

In the **RGB–Thermal–Language** setting, MMOne improves over MM-J on the four reported scenes from RGB PSNR \(22.32\) to \(23.19\), RGB SSIM \(0.796\) to \(0.812\), thermal PSNR \(23.38\) to \(24.24\), thermal SSIM \(0.849\) to \(0.867\), and language mIoU \(45.1\) to \(48.1\) [2507.11129]. The paper emphasizes the modality-conflict comparison: adding language to ThermalGaussian-style shared-opacity joint training reduces RGB PSNR by \(0.56\) dB and thermal PSNR by \(0.52\) dB, whereas MMOne slightly improves both when moving from RGB–Thermal to RGB–Thermal–Language, from RGB PSNR \(23.12\) to \(23.19\) and thermal PSNR \(24.17\) to \(24.24\) [2507.11129].

The ablation study decomposes the contribution of the major components. Starting from the shared-opacity joint baseline MM-J, adding the modality modeling module alone improves RGB PSNR from \(22.32\) to \(22.38\), thermal PSNR from \(23.38\) to \(23.73\), and language mIoU from \(45.1\) to \(45.3\), while reducing the number of Gaussians from \(32.9\) to \(29.0\) in units of \( \times 10^4 \) [2507.11129]. Hard prune further improves mIoU to \(46.9\) and reduces Gaussians to \(13.4 \times 10^4\); soft prune improves RGB PSNR to \(22.98\), thermal PSNR to \(23.99\), mIoU to \(47.0\), and reduces Gaussians to \(10.6 \times 10^4\) [2507.11129]. Full decomposition yields the best overall results: RGB PSNR \(23.19\), thermal PSNR \(24.24\), language mIoU \(48.1\), and the lowest Gaussian count, \(9.9 \times 10^4\) [2507.11129]. Threshold ablations over \(\tau \in [0.0001, 0.0004]\) are reported as stable, with \(0.0002\) the best setting [2507.11129].

The limitations are explicitly bounded. Current experiments assume static scenes and given camera poses from SfM; future directions named in the paper include COLMAP-free Gaussian learning and dynamic-scene extensions [2507.11129]. The representation does not introduce an explicit high-level semantic latent disentangling beyond Gaussian-level decomposition, and the method relies on heuristic thresholds and loss weights even though ablations indicate robustness [2507.11129]. Evaluation is limited to RGB, thermal, language, and depth, with broader sensory modalities left for future work [2507.11129].

Within the broader literature on multimodal 3D representations, MMOne is positioned as retaining the efficiency of the 3DGS regime while changing *where* sharing occurs: geometry is shared unless gradients diverge, whereas visibility and effective primitive density are modality-specific [2507.11129]. This suggests a primitive-level factorization of multimodal structure that is physically grounded by rendering and gradient behavior rather than imposed through a separate latent-variable model [2507.11129].

Source: https://www.emergentmind.com/topics/mmone