ColorGS: High-Fidelity Surgical Reconstruction

• ColorGS is a framework for reconstructing deformable surgical scenes by addressing fixed per-Gaussian color and linear deformation limitations.
• It introduces Colored Gaussian Primitives for spatially adaptive color encoding and an Enhanced Deformation Model that captures both local and global tissue motions.
• Evaluations on EndoNeRF and StereoMIS benchmarks show state-of-the-art PSNR, SSIM, and LPIPS results while maintaining real-time rendering efficiency.

Searching arXiv for ColorGS and closely related 3D Gaussian Splatting color/deformation papers. ColorGS is a framework for high-fidelity surgical scene reconstruction with Colored Gaussian Splatting that targets deformable tissues in endoscopic video. It is introduced to address two specific limitations of prior dynamic 3D Gaussian Splatting approaches: fixed per-Gaussian color assignment, which struggles with subtle color variations and intricate textures, and linear deformation modeling, which struggles to model consistent global deformation. The method combines Colored Gaussian Primitives for spatially adaptive color encoding with an Enhanced Deformation Model (EDM) for joint local and global motion modeling, and is evaluated on DaVinci robotic surgery videos and the EndoNeRF and StereoMIS benchmarks (Ji et al., 26 Aug 2025).

1. Problem formulation and motivation

ColorGS is situated in the problem of reconstructing deformable surgical scenes from endoscopic videos, with the stated application areas of intraoperative navigation, surgical guidance, and AR/VR-based training. The motivating observation is that endoscopic scenes contain subtle, spatially varying color differences that distinguish tissues with similar appearances, often under challenging, dynamic lighting. In parallel, surgical motion contains both localized deformation and global, coherent motion induced by tool interaction or patient motion. The paper treats these as coupled deficiencies in earlier methods rather than independent nuisances (Ji et al., 26 Aug 2025).

Within this formulation, conventional 3DGS color parameterization is described as insufficient because each Gaussian carries a fixed color, whereas deformable surgical imagery requires color variation that depends on location on the rendering plane as well as on view direction. Likewise, deformation models built primarily from localized basis functions are characterized as inadequate for capturing consistent, long-range/global deformations. ColorGS therefore modifies both the photometric and kinematic parameterization of Gaussian primitives rather than only refining rendering or only refining motion.

2. Colored Gaussian Primitives

The first core component is the introduction of Colored Gaussian Primitives, described as a spatially adaptive color model for each 3D Gaussian. For every Gaussian, the method places a set of $k$ color anchors $A_i=(A_i^x, A_i^y)$ on the 2D rendering plane in image coordinates, and associates each anchor with a learnable color vector $c_i$. The local contribution of an anchor to a ray–Gaussian intersection point $p=(u,v)$ is weighted by an exponential spatial decay:

$$F_{A_i}(p) = \exp\left(-\lambda_e \|p - A_i\|^2\right)$$

where $\lambda_e$ controls locality, stated as typically $0.1$, and $k=4$ anchors per Gaussian are used by default (Ji et al., 26 Aug 2025).

The aggregated spatial color term is

$F_c(p) = \sum_{i=0}^{k-1} F_{A_i}(p)\, c_i$

and the rendered color at point $p$ in direction $A_i=(A_i^x, A_i^y)$0 is

$A_i=(A_i^x, A_i^y)$1

Here, $A_i=(A_i^x, A_i^y)$2 is the standard spherical-harmonics directional color term from 3DGS, while $A_i=(A_i^x, A_i^y)$3 injects spatial variation on the rendering plane. In the paper’s interpretation, this allows each Gaussian to encode color that is simultaneously view-dependent and spatially varying, improving expressiveness for tissues with visually similar appearance under complex illumination.

This design is significant because it preserves compatibility with the standard directional color machinery of 3DGS while extending it with a location-sensitive residual. A plausible implication is that ColorGS treats texture fidelity as a primitive-level representational issue rather than as a purely rendering-stage correction.

3. Enhanced Deformation Model

The second core component is the Enhanced Deformation Model (EDM), which decomposes deformation into a local time-aware component and a global, time-independent component. For position along the $A_i=(A_i^x, A_i^y)$4-dimension, the local component is written as a weighted sum of Gaussian basis functions:

$A_i=(A_i^x, A_i^y)$5

$A_i=(A_i^x, A_i^y)$6

with $A_i=(A_i^x, A_i^y)$7 basis functions and $A_i=(A_i^x, A_i^y)$8 as learnable weights. The full deformation adds a learnable time-independent offset:

$A_i=(A_i^x, A_i^y)$9

The same design is extended to Gaussian rotation and scaling parameters (Ji et al., 26 Aug 2025).

The explicit role of $c_i$0 is to capture global translation or consistent drift shared across time, while the Gaussian basis expansion captures temporally localized changes. In the paper’s framing, this separation permits simultaneous modeling of local tool–tissue interaction detail and global organ-level motion consistency. This is presented as reducing the need for either a very large number of basis functions or a more heavily over-parameterized deformation model.

Conceptually, EDM shifts dynamic surgical reconstruction away from a purely local temporal basis representation. This matters in surgical settings because coherent displacement of a tissue mass can be clinically relevant even when fine-grained local deformation is also present.

4. Training pipeline and loss formulation

ColorGS follows an optimization pipeline that begins by using camera parameters to reconstruct a point cloud for the first frame and initialize Gaussian primitives. During optimization, the framework updates both the color anchors and color parameters of the Colored Gaussian Primitives and the parameters of EDM. For each training viewpoint, it renders RGB and depth images using the current Gaussian primitives, anchor parameters, and deformation parameters (Ji et al., 26 Aug 2025).

The training objective is a tissue-masked loss that focuses supervision on relevant anatomical regions:

$c_i$1

where $c_i$2 is the tissue mask, $c_i$3 and $c_i$4 are rendered color and depth, and $c_i$5 and $c_i$6 are the ground-truth color and depth. The use of a tissue mask is integral to the formulation as presented, because the target domain includes occlusions and non-relevant image regions in endoscopic video.

The resulting system is explicitly designed to balance high fidelity and computational practicality. The abstract states that the method maintains real-time rendering efficiency, and the broader argument is that Gaussian Splatting provides the computational substrate needed to avoid the slow training and inference behavior often associated with NeRF-style methods (Ji et al., 26 Aug 2025).

5. Quantitative evaluation and ablations

ColorGS is evaluated on EndoNeRF and StereoMIS, with metrics reported as PSNR, SSIM, and LPIPS. The paper reports the following results (Ji et al., 26 Aug 2025):

Dataset	Method	PSNR ↑	SSIM (%) ↑	LPIPS ↓
EndoNeRF	EndoNeRF	35.92	94.18	0.06
EndoNeRF	EndoGaussian	37.86	96.09	0.04
EndoNeRF	Deform3DGS	38.35	96.39	0.05
EndoNeRF	ColorGS	39.85	97.25	0.03
StereoMIS	EndoNeRF	28.86	74.15	0.27
StereoMIS	EndoGaussian	30.39	83.75	0.21
StereoMIS	Deform3DGS	30.68	84.74	0.23
StereoMIS	ColorGS	32.64	89.64	0.14

The abstract summarizes these results as state-of-the-art performance, including a PSNR of 39.85 (1.5 higher than prior 3DGS-based methods) and SSIM of 97.25\% while maintaining real-time rendering efficiency. On EndoNeRF, the reported gain over Deform3DGS is +1.5 dB PSNR, with an SSIM increase of nearly +1\%, alongside a lower LPIPS value. On StereoMIS, the gains are also substantial across all three metrics (Ji et al., 26 Aug 2025).

The ablation studies distinguish the contribution of each major component. The paper states that Colored Gaussians improve performance over 3DGS/2DGS/SuperGaussian, and that EDM outperforms Fourier/Polynomial series (FPS) and local-basis-only Gaussians (GS). These ablations are used to support the claim that the observed improvements arise specifically from spatial color adaptivity and explicit local-plus-global deformation modeling, rather than from training variance or benchmark idiosyncrasy.

6. Position within color-aware Gaussian splatting

A recurrent source of confusion is that the name ColorGS can be read as referring to general color editing in Gaussian Splatting. In the literature represented here, ColorGS is instead a reconstruction framework for deformable surgical scenes. This differs from several color-aware 3DGS systems whose primary objective is editing rather than reconstruction.

ReCoGS introduces a pipeline for real-time recoloring of pre-trained Gaussian Splatting scenes, using brush-based 2D selection, 2D-to-3D unprojection, and color-only optimization of spherical harmonics coefficients; it explicitly performs pure color manipulation—no geometry fitting, pruning, or densification (Rutayisire et al., 23 Nov 2025). VIRGi targets view-dependent instant recoloring of 3D Gaussian Splats, separating color into diffuse and specular/view-dependent components and propagating edits from one manually edited image while preserving specular effects (Mazzucchelli et al., 3 Mar 2026). ColorGradedGaussians addresses palette-based color grading through view-space sparse decomposition, enabling palette recoloring, per-palette tone curves, and pixel-level constraints (Chao et al., 2 Apr 2026).

Against this background, ColorGS occupies a distinct niche: it is not formulated as a post hoc recoloring or grading system for an already reconstructed scene, but as a model that augments 3DGS itself to improve fidelity of surgical reconstruction under complex lighting, tissue similarity, and deformation. This suggests that its main contribution lies in representation design for medical dynamic reconstruction, even though it belongs to the broader class of color-aware Gaussian methods (Ji et al., 26 Aug 2025).

7. Relevance to surgical imaging and visualization

The paper frames ColorGS as relevant to surgical guidance and navigation, AR/VR, and potentially broader real-time surgical imaging workflows. The underlying rationale is direct: higher-fidelity reconstruction preserves subtle color and shape cues that surgeons rely on, while more accurate local-plus-global deformation modeling supports scene understanding when tissues undergo both fine interaction and coherent bulk motion (Ji et al., 26 Aug 2025).

The method’s reported combination of high reconstruction fidelity and real-time rendering efficiency is particularly important in this domain, where latency and visual plausibility both matter. A plausible implication is that ColorGS is intended not merely as a benchmark improvement over EndoNeRF-, EndoGaussian-, or Deform3DGS-style baselines, but as a step toward systems that can remain computationally tractable while representing the photometric and kinematic complexity of endoscopic surgery.

In that sense, ColorGS can be understood as a specialized extension of dynamic Gaussian Splatting for medical vision: it couples spatially adaptive color encoding and enhanced deformation modeling to address the specific failure modes of reconstructing deformable tissue from surgical video, and reports state-of-the-art results on the benchmarks used to evaluate that objective (Ji et al., 26 Aug 2025).