Region-Based Blending Estimation Strategy

• The paper introduces a region-based approach that decomposes synthesized views into overlapping, single-view, and mutual disocclusion regions for precise view synthesis distortion estimation.
• It employs geometric partitioning using depth discontinuity detection and disparity projections to compute region proportions and tailor distortion models for each distinct area.
• Empirical validation shows up to 49% RMSE improvement in challenging configurations, highlighting its real-time applicability in versatile multi-view 3D video systems.

A region-based blending estimation strategy refers to a principled methodology that decomposes a synthesized (virtual) view into geometrically distinct regions—overlapping regions, single-view regions, and mutual disocclusion regions—and then predicts the overall view synthesis distortion (VSD) by integrating the expected distortion in each region, weighted by their geometric proportions. This approach is particularly designed to address the heterogeneity in distortion characteristics seen in multi-view 3D video systems, especially under challenging large baseline configurations, where naive linear blend models can fail due to pronounced regions of mutual disocclusion and strong geometric disparities.

1. Geometric Partitioning of Synthesized Views

The first principle of the region-based blending estimation strategy is the explicit geometric partitioning of the synthesized virtual view into three main types of regions, each associated with different data sources and distortion behaviors:

• Overlapping regions (𝛺₍overlap₎): Pixels where both left and right reference views contribute data. These regions are typically blended with a linear weighting determined by baseline distances.
• Single-view regions (𝛺_L, 𝛺_R): Pixels exposed in only one reference view, often due to disocclusion arising from depth discontinuities. The synthesis uses only the available view for these regions.
• Mutual disocclusion regions (𝛺₍none₎): Pixels for which neither reference view provides valid data, requiring the use of inpainting or interpolation algorithms to fill in missing pixels.

This partitioning is computed via geometric operations including depth discontinuity (edge) detection (e.g., using a Sobel operator) and disparity-based projection, determining the disocclusion width and matching opposing depth edges. The normalized area of each region type (P₍overlap₎, P_L, P_R, P₍none₎) is calculated with:

$$O_{\mathrm{disocc},\mathrm{norm}} = \frac{\sum W_{\mathrm{disocc}}(x,y)}{WH}$$

where $W_{\mathrm{disocc}}(x,y)$ is the disocclusion width at edge pixel $(x,y)$, and $W,H$ are the frame dimensions. Mutual disocclusion is computed via projected edge matching with an area threshold.

2. Distortion Modeling per Region

Each region type exhibits distinct synthesis distortion characteristics:

• Overlapping region distortion: Modeled as a blend of the left and right reference view distortions, with blending weight $\alpha$ based on baseline geometry,

$$E[e_{\text{overlap}}^2] \approx \alpha^2 E_{\text{dep}}[Z_L^2] + (1-\alpha)^2 E_{\text{dep}}[Z_R^2]$$

• Single-view region distortion: Directly taken as the distortion estimated from the contributing reference view,

$$E[e_L^2] = E_{\text{dep}}[Z_L^2], \qquad E[e_R^2] = E_{\text{dep}}[Z_R^2]$$

• Mutual disocclusion distortion: Estimated via a variance-based model,

$$E[e_{\text{none}}^2] = \frac{\nu^2_{\text{local},L} + \nu^2_{\text{local},R}}{2}$$

where $\nu^2_{\text{local},i}$ is the local variance of texture pixels near depth discontinuities in view $i$.

These judiciously selected formulas reflect region-specific synthesis mechanisms and variance propagation due to missing data and edge structure.

3. Weighted Summation of Regional Contributions

The overall depth-coding-induced distortion ($E_{\text{dep}}[Z^2]$) is calculated as the sum over regional contributions, weighted by the respective geometric area proportions:

$$E_{\text{dep}}[Z^2] = P_{\text{overlap}} \cdot E[e_{\text{overlap}}^2] + P_L \cdot E[e_L^2] + P_R \cdot E[e_R^2] + P_{\text{none}} \cdot E[e_{\text{none}}^2]$$

This weighted blending, strongly grounded in geometric partitioning, provides an accurate prediction of overall VSD, adaptable to both symmetric and asymmetric synthesis configurations and arbitrary baseline layouts.

Table: Region Types and Their Distortion Models

Region Type	Source Views Used	Distortion Model
Overlapping	Left + Right (blended)	$$\alpha^2 E_{\text{dep}}[Z_L^2] + (1-\alpha)^2 E_{\text{dep}}[Z_R^2]$$
Single-view L	Left only	$E_{\text{dep}}[Z_L^2]$
Single-view R	Right only	$E_{\text{dep}}[Z_R^2]$
Mutual disoccl.	None (hole-filled)	$$\frac{1}{2}(\nu^2_{\text{local},L}+\nu^2_{\text{local},R})$$

4. Region Proportion Computation and Algorithmic Efficiency

The region proportions ($P_{\text{overlap}}$, $P_L$, $P_R$, $P_{\text{none}}$) are calculated via:

• Detection of depth edges;
• Computation of disocclusion width using geometric projection formulas;
• Quantification of mutual disocclusion by matching projected depth edges with opposing gradients (allowing for thresholded spatial overlap).

These operations require only basic arithmetic and edge operations, with the computational burden limited to pixels near depth edges (typically 1–5% of the frame). This enables real-time applicability and training-free deployment, essential in practical 3D video systems.

5. Large Baseline Compensation and Synthesis Flexibility

The methodology explicitly compensates for large baseline scenarios by:

• Using a baseline indicator to adapt the blending weights;
• Ensuring the region partitioning accurately reflects the geometric disparities that arise with increased camera spacing;
• Modeling the amplified impact of disocclusions and single-view regions occurring under large baselines, where naive linear blending would otherwise incur significant prediction errors.

Empirical validation demonstrates reduced root mean squared error (RMSE) of up to 49% in asymmetric configurations and over 23% in symmetric ones compared to uniform linear blend estimators. The method maintains prediction error below 5% in baseline scenarios as wide as 8.0 camera units, even when previous methods fail (e.g., introducing errors >30%).

6. Practical Implications and Integration in 3D Content Acquisition

The region-based blending estimation strategy enables:

• Accurate online estimation of synthesis distortion for real-time optimization, encoding, and resource allocation in 3D video, interactive free-viewpoint, and streaming workflows;
• Support for flexible, non-uniform camera arrangements with large baselines, facilitating cost-effective and robust multi-camera capture;
• Adaptability across encoding and rate–distortion optimization workflows without training overhead; the geometric quantities are computed directly from depth and texture maps.

A plausible implication is the extension of this strategy to more general multi-view synthesis pipelines, where highly variable region coverage and occlusion patterns are present, and where distortion prediction must be both region-aware and computationally tractable.

7. Significance and Validation

Experimental results conducted on standard MPEG 3D video sequences confirm the superiority of the region-based blending strategy. The method's accuracy, computational efficiency, and robustness to challenging configurations position it as foundational for next-generation flexible 3D reconstruction and streaming technologies.

This region-aware decomposition and summation paradigm represents a rigorous and practical advance for view synthesis distortion estimation, especially in heterogeneous, large-baseline environments, by integrating geometric insights with statistical error modeling.