PVSO in MV-3DRES

• PVSO is an optimization technique that mitigates foreground gradient dilution by shifting supervision from sparse 3D data to denser 2D views.
• It employs per-view Dice losses with a suppression mechanism for no-target views to ensure effective gradient propagation during training.
• Integrating PVSO into the MVGGT framework significantly boosts mIoU metrics, demonstrating its value in multi-view 3D referring expression segmentation.

Per-view No-target Suppression Optimization (PVSO) is an optimization technique introduced within the MVGGT framework for Multiview 3D Referring Expression Segmentation (MV-3DRES), designed to address the gradient vanishing problem that arises during training with sparse multi-view RGB images. In settings where the target object occupies a minute fraction of the reconstructed point cloud, conventional 3D segmentation losses yield negligible foreground gradients, impeding effective learning. PVSO mitigates this by shifting supervision into the 2D image space, where foreground pixels are more prevalent, using per-view Dice losses augmented by a suppression mechanism for views not containing the target.

1. Foreground Gradient Dilution in Sparse 3DRES

Foreground Gradient Dilution (FGD) describes the attenuation of segmentation gradients when the referred object is sparsely represented in a reconstructed 3D point cloud. In the MV-3DRES scenario, the network infers a binary mask $M \in \{0, 1\}^K$ over $K$ points from $N$ RGB views. Standard 3D segmentation is penalized via a Dice loss: $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$ where $p_j \in [0, 1]$ are predicted probabilities and $g_j \in \{0, 1\}$ ground-truth labels. In low-occupancy regimes ($<$2% of points as foreground), $U$ is dominated by background, shrinking the gradient for positives to $10^{-9}$–$10^{-11}$ and stagnating learning.

2. Mathematical Formulation of PVSO

PVSO circumvents FGD by supervising in the 2D domain, where the foreground pixel fraction is higher (∼10–15%). Given views $K$0, the set of target-visible views is $K$1, with the complement $K$2 consisting of no-target views. Let $K$3 denote the sampled subset per iteration, enforcing a minimal positive-view ratio: $K$4 For each sampled view $K$5, the predicted mask $K$6 and ground-truth $K$7 are used with the standard per-view Dice loss: $K$8 No-target views are weighted by $K$9, yielding the PVSO loss: $N$0 The total training objective is: $N$1 where $N$2 controls the balance between 3D and 2D losses ($N$3 in reported experiments) (Wu et al., 11 Jan 2026).

3. Integration into MVGGT Training Workflow

PVSO integrates into MVGGT’s pipeline as follows:

• The frozen Pi3 geometric branch processes $N$4 RGB images to yield depth maps and camera poses, reconstructing $N$5.
• Image tokens and their geometric injections are encoded by the multimodal branch, predicting per-view 2D masks $N$6 and constructing the aggregated 3D mask $N$7.
• Using ground-truth masks, views are partitioned into $N$8 and $N$9, selecting $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$0 to maintain $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$1.
• Compute $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$2 over 3D predictions and $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$3 using per-view Dice losses, with suppressed contribution from $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$4.
• Gradients from both terms update only the multimodal branch (geometry branch remains static), and AdamW is used for parameter updates.

4. PVSO Algorithmic Structure

The training loop for PVSO within MVGGT is summarized as:

Step	Description	Output
1	Encode images through Pi3 (frozen)	Depth $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$5, poses $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$6, $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$7
2	Encode text using RoBERTa	Text embedding $$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$8
3	Multimodal branch predicts 2D masks	$$\mathcal{L}_{\mathrm{Dice}^{\mathrm{3D}}} = 1 - \frac{2I}{U}, \quad I = \sum_{j=1}^{K} p_j g_j, \quad U = \sum_{j=1}^{K} p_j + \sum_{j=1}^{K} g_j$$9
4	Back-project $p_j \in [0, 1]$0 using $p_j \in [0, 1]$1 to 3D mask $p_j \in [0, 1]$2	$p_j \in [0, 1]$3
5	Partition and sample views, enforce $p_j \in [0, 1]$4	$p_j \in [0, 1]$5, $p_j \in [0, 1]$6, $p_j \in [0, 1]$7
6	Compute $p_j \in [0, 1]$8, $p_j \in [0, 1]$9	Loss values
7	Backpropagate and update multimodal parameters	Updated weights

This structure preserves gradient magnitude across both positive and suppressed no-target views, ensuring stable training even under severe spatial sparsity.

5. Empirical Ablations and Performance

Ablation studies on MVRefer benchmark demonstrate PVSO’s effect in isolation and combined with MVGGT:

Configuration	mIoU$g_j \in \{0, 1\}$0	mIoU$g_j \in \{0, 1\}$1
2D-Lift (baseline)	17.8	20.4
MVGGT only (no PVSO)	26.9	41.1
PVSO only (no MVGGT)	32.0	47.5
MVGGT + PVSO (full model)	39.9	69.3

PVSO independently increases global mIoU by ∼5 points and view-level mIoU by ∼6 points over MVGGT without PVSO. The combined approach realizes additive benefits, yielding state-of-the-art segmentation accuracy. Further ablations reveal:

• Disabling $g_j \in \{0, 1\}$2 reduces global mIoU to ≈32.4.
• Employing $g_j \in \{0, 1\}$3 with random sampling improves mIoU to ≈36.7.
• Enforcing $g_j \in \{0, 1\}$4 yields the best mIoU (39.9). These results support PVSO’s crucial role in boosting positive gradient strength, avoiding trivial solutions in no-target views, and stabilizing convergence through controlled sampling ratios (Wu et al., 11 Jan 2026).

6. Hyper-parameters, Computational Overhead, and Limitations

PVSO’s practical deployment involves:

• $g_j \in \{0, 1\}$5: set to 1; higher/lower values shift supervision balance.
• $g_j \in \{0, 1\}$6: best at 0.5; low values under-sample target, high values omit negatives.
• $g_j \in \{0, 1\}$7: normalizes loss impact of negative views.

Computational overhead is minimal—PVSO only requires projecting the 3D mask back into 2D and computing simple Dice losses per image. This cost is insignificant relative to backbone processing.

Limitations include:

• Dependency on ground-truth visibility masks per-view at training; PVSO is not directly applicable under purely weakly supervised regimes.
• Hyper-parameters $g_j \in \{0, 1\}$8 and $g_j \in \{0, 1\}$9 are sensitive to view count and sparsity.
• PVSO ameliorates gradient weakness but does not address geometric misalignments or depth noise, contingent on Pi3 geometry branch efficacy.

7. Summary and Significance

Per-view No-target Suppression Optimization (PVSO) constitutes a robust, lightweight module for MV-3DRES. By relocating part of the supervision signal to the dense 2D domain and balancing loss contributions between target-present and target-absent views, PVSO addresses the acute gradient dilution affecting sparse-view 3D segmentation. When incorporated into the MVGGT architecture, PVSO delivers stable, efficient, and high-fidelity learning, substantiating its critical role in state-of-the-art multi-view 3D referring expression segmentation (Wu et al., 11 Jan 2026).