Gradient-Weighted Feature Back-Projection

• Gradient-Weighted Feature Back-Projection is a training-free method that assigns high-dimensional features to 3D Gaussians via gradient-weighted accumulation.
• It leverages 2D features from frozen backbones to deliver fast and accurate segmentation in both 2D images and 3D scenes.
• The approach efficiently scales with view coverage and feature dimensions, achieving segmentation in milliseconds with minimal computational overhead.

Gradient-Weighted Feature Back-Projection (GW-FBP) is a training-free methodology for assigning high-dimensional feature representations to 3D Gaussians in pre-trained 3D Gaussian Splatting (3DGS) scenes. The core innovation is the back-projection of 2D features—extracted by powerful frozen backbone networks—via gradient-weighted accumulation, producing feature fields suited for fast and accurate 2D and 3D segmentation while circumventing the time and computational demands of neural feature field training (Joseph et al., 2024).

1. Mathematical Framework

A 3DGS scene is represented by $N$ Gaussians ("splats"), each with mean position $\boldsymbol{\mu}_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3\times 3}$ (often diagonal), scalar opacity $\alpha_i$, color $\boldsymbol{c}_i \in \mathbb{R}^3$, and, in GW-FBP, a feature vector $\boldsymbol{f}_i \in \mathbb{R}^D$. Rendering colors proceeds by front-to-back sorting and alpha compositing. The transmittance $T_n(x,y)$ at pixel $(x,y)$ through splats $1,\dotsc,n-1$ is

$T_n(x,y) = \prod_{m<n} (1-\alpha_m(x,y))$

and the pixel color is

$\boldsymbol{\mu}_i \in \mathbb{R}^3$0

The partial derivative $\boldsymbol{\mu}_i \in \mathbb{R}^3$1 quantifies the influence of splat $\boldsymbol{\mu}_i \in \mathbb{R}^3$2 on the rendered color. Let $\boldsymbol{\mu}_i \in \mathbb{R}^3$3 denote a D-dimensional per-pixel feature from a 2D backbone at pixel $\boldsymbol{\mu}_i \in \mathbb{R}^3$4 in view $\boldsymbol{\mu}_i \in \mathbb{R}^3$5. The gradient-weighted feature back-projection assigns to each Gaussian $\boldsymbol{\mu}_i \in \mathbb{R}^3$6:

$\boldsymbol{\mu}_i \in \mathbb{R}^3$7

where $\boldsymbol{\mu}_i \in \mathbb{R}^3$8. In practice, accumulation without explicit normalization is used due to subsequent $\boldsymbol{\mu}_i \in \mathbb{R}^3$9-normalization, rendering both forms equivalent up to scale. This ensures feature vectors that, when projected, recover the input 2D features in expectation.

2. Computational Pipeline and Complexity

Inputs and Outputs

• Inputs: Pre-trained 3DGS scene (with $\Sigma_i \in \mathbb{R}^{3\times 3}$0, $\Sigma_i \in \mathbb{R}^{3\times 3}$1, $\Sigma_i \in \mathbb{R}^{3\times 3}$2), $\Sigma_i \in \mathbb{R}^{3\times 3}$3 training views, and per-pixel feature maps $\Sigma_i \in \mathbb{R}^{3\times 3}$4 from a selected backbone.
• Outputs: Per-Gaussian feature vectors $\Sigma_i \in \mathbb{R}^{3\times 3}$5.

Workflow

1. For every training view $\Sigma_i \in \mathbb{R}^{3\times 3}$6: a. Render the 3DGS scene to compute, for each pixel $\Sigma_i \in \mathbb{R}^{3\times 3}$7, which Gaussians contribute and their $\Sigma_i \in \mathbb{R}^{3\times 3}$8 values. b. Extract $\Sigma_i \in \mathbb{R}^{3\times 3}$9 for all pixels using the frozen backbone. c. For each contributing Gaussian $\alpha_i$0 at a pixel, accumulate:

   $\alpha_i$1

   $\alpha_i$2

2. After all images, compute normalized features:

$\alpha_i$3

Pseudocode

$\boldsymbol{f}_i \in \mathbb{R}^D$6

Complexity

The process requires $\alpha_i$4 time for rendering and accumulation, where $\alpha_i$5 is the average number of contributing Gaussians per pixel. Normalization requires $\alpha_i$6. GW-FBP completes feature assignment in 2–3 minutes, compared to 20–30 minutes for training-based distillation (7,000 SGD iterations). Inference segmentation is performed in $\alpha_i$7 ms, being 10× faster in feature assignment and $\alpha_i$8900× faster in end-to-end segmentation compared to optimization-based methods.

3. Segmentation and Downstream Applications

3D Segmentation

Given the derived $\alpha_i$9, a query embedding $\boldsymbol{c}_i \in \mathbb{R}^3$0 (e.g., from LSeg’s text encoder) is compared to each Gaussian via cosine similarity $\boldsymbol{c}_i \in \mathbb{R}^3$1. Gaussians with $\boldsymbol{c}_i \in \mathbb{R}^3$2 or among the top-$\boldsymbol{c}_i \in \mathbb{R}^3$3 are labeled as belonging to the target object, yielding a 3D mask.

2D Segmentation

Rendered scene features $\boldsymbol{c}_i \in \mathbb{R}^3$4 are compared against $\boldsymbol{c}_i \in \mathbb{R}^3$5:

$\boldsymbol{c}_i \in \mathbb{R}^3$6

No segmentation loss is used during back-projection, as the method is entirely training-free in feature attribution. For auxiliary Identity Encoder tasks, losses include classification and orthogonality regularization:

• $\boldsymbol{c}_i \in \mathbb{R}^3$7
• $\boldsymbol{c}_i \in \mathbb{R}^3$8

Evaluation is performed using IoU, mIoU, and mAP.

4. Empirical Evaluation

Segmentation Performance

• Datasets: Varied indoor/outdoor scenes from 3DGS.
• Baselines: Feature-3DGS, which requires training for 7,000 steps ($\boldsymbol{c}_i \in \mathbb{R}^3$920–30 min).
• GW-FBP achieves visually comparable results in 2D mask prediction and superior 3D mask quality with fewer outlier Gaussians.
• Processing time: 2–3 min for back-projection and feature extraction versus 20–30 min for feature field distillation.
• Segmentation queries: 30 ms.

Affordance Transfer

Affordance transfer from annotated images to 3D scenes is performed considerably faster—$\boldsymbol{f}_i \in \mathbb{R}^D$040×—with competitive mIoU, as summarized:

Scene	mIoU ↑ (2D-2D-3D)	mIoU ↑ (2D-3D)	Recall ↑	Time ↓
1	47.87	42.80	67.77/67.11	293.9s / 5.22s
2	55.63	53.28	81.07/82.55	317.1s / 8.03s
3	60.50	57.82	86.95/86.68	142.8s / 7.58s
Mean	54.67	51.30	78.60/78.78	251.3s / 6.94s

Identity Encoding

Comparison of contrastive vs. Gaussian Grouping on 3D-OVS/LERF-Mask shows grouped mIoU performance: 73.4 vs. 72.8 mean, with identity-classifier training and back-projection taking ≈20 s total.

Ablation Studies

• Weight normalization vs. pure accumulation: negligible difference post $\boldsymbol{f}_i \in \mathbb{R}^D$1-normalization.
• Feature dimension $\boldsymbol{f}_i \in \mathbb{R}^D$2 and number of Gaussians $\boldsymbol{f}_i \in \mathbb{R}^D$3: stable performance; GW-FBP scales with $\boldsymbol{f}_i \in \mathbb{R}^D$4 typical values such as 384 for DINOv2.
• Using only top-$\boldsymbol{f}_i \in \mathbb{R}^D$5 contributors per Gaussian maintains accuracy with reduced computational load.

5. Characteristics, Advantages, and Limitations

Advantages

• Training-free: Direct feature projection without SGD-based refinement.
• Speed: Feature assignment in minutes, segmentation in milliseconds.
• High-quality segmentation: Features are aligned with actual rendering influence per Gaussian, yielding high 3D mask fidelity.
• Scalability: Suitable for affordance transfer, identity encoding, and related downstream 3D vision tasks with minimal overhead.

Limitations

• No parameter refinement: Geometry or opacities of Gaussians are not updated; in contrast to trained feature distillation, this restricts adaptive scene refinement.
• Dependency on view coverage: Sparse training views or low-quality backbone features can reduce mask fidelity and introduce noise.

6. Extensions and Open Directions

Proposed directions for further investigation include:

• Joint refinement: Combining GW-FBP with neural parameter tuning (gradient steps) on Gaussian attributes for enhanced regularization.
• Dynamic/non-rigid scene extension: Adapting GW-FBP to time-dependent or non-rigid settings by leveraging temporally varying 2D features.
• Integration with task-specific objectives: Embedding GW-FBP within semantic-aware or task-refined Gaussian operations (e.g., splitting, merging).
• Alternative weighting schemes: Introducing learned or color-gradient-based weights for more nuanced back-projection.

GW-FBP emerges as a principled, efficient, and domain-agnostic method for feature assignment in 3DGS, providing a practical alternative to feature field training while supporting a broad array of scene understanding tasks (Joseph et al., 2024).