Virtual Correspondence Error (VCE)

• Virtual Correspondence Error (VCE) is a quantitative measure in camera geometry that uses virtual correspondences to assess reprojection residuals, even with low view overlap.
• The method integrates coplanarity constraints and soft penalties in bundle adjustment to jointly optimize camera poses and 3D scene structure.
• Empirical results demonstrate that VCE-based approaches yield improved pose estimates with lower reprojection errors in challenging wide-baseline scenarios.

Virtual Correspondence Error (VCE) is a quantitative measure used in camera geometry estimation and bundle adjustment involving virtual correspondences—pixel pairs across images whose corresponding camera rays intersect in 3D but may not be co-visible or observe the same 3D point. The VCE is defined as the sum of squared reprojection residuals associated with these pairs, optionally augmented by a coplanarity penalty enforcing that the two rays lie in a common plane. This formulation generalizes the standard epipolar geometry framework to scenarios with little or no view overlap, providing robust constraints for multi-view pose estimation and scene reconstruction in extremely wide baseline setups (Ma et al., 2022).

1. Formal Definition of Virtual Correspondence Error

Given two images $I_1$ and $I_2$ with intrinsic matrices $K_1$, $K_2$ and poses $(R_1, t_1)$, $(R_2, t_2)$, a virtual correspondence is a pixel pair $(u_1, u_2)$ such that there exist depths $d_1 > 0$, $d_2 > 0$ satisfying:

$f_1(d_1) = f_2(d_2)$

where

$I_2$0

$I_2$1. No requirement is made for $I_2$2 and $I_2$3 to observe the same visible 3D scene point; their defining property is the 3D intersection of the two rays.

Once 3D intersection points $I_2$4 are found for each pair, the VCE for a virtual correspondence indexed by $I_2$5 is:

$I_2$6

where $I_2$7 denotes the camera projection function for image $I_2$8.

2. Coplanarity Constraints and Bundle Adjustment

To maintain geometric consistency, a coplanarity constraint is usually imposed for each virtual correspondence. For camera centers $I_2$9 and $K_1$0,

$K_1$1

(Eq. 4, (Ma et al., 2022)) enforcing that the two rays and both camera centers are coplanar, as required by epipolar geometry.

The full hard-constrained bundle adjustment objective for $K_1$2 such correspondences is:

$K_1$3

subject to the above coplanarity constraint for all $K_1$4.

3. Re-parameterization and Soft Constraints

To avoid hard constraints, the coplanarity condition is often relaxed via re-parameterization:

$K_1$5

(Eq. 5, (Ma et al., 2022)) where scalars $K_1$6, $K_1$7 are optimized alongside $K_1$8. If $K_1$9, the correspondence reduces to a standard one.

In practical optimization, the coplanarity constraint is enforced softly using a penalty term:

$K_2$0

with $K_2$1 balancing data fit and coplanarity.

4. Minimization and Solvers

The minimization procedure mirrors standard bundle adjustment. Key steps:

• Camera poses $K_2$2 are initialized via RANSAC and five-point algorithms applied to VCs.
• Each virtual pair's depths are set by intersecting image rays with a 3D mesh (e.g., a hallucinated human model).
• Optimization (typically via L-BFGS) jointly refines camera parameters and all $K_2$3.
• The coplanarity term is treated as a soft penalty; standard inlier filtering is used during initialization. No custom robustification is reported beyond this (Ma et al., 2022).

5. Quantitative Behavior and Empirical Performance

Specific per-correspondence VCE values in pixels are not tabulated, but epipolar errors are visualized in several figures (e.g., Figure 1 in (Ma et al., 2022)). After bundle adjustment, typical reprojection residuals fall below a few pixels per correspondence, supporting accurate pose estimates.

Results on the CMU Panoptic dataset for two-view pose estimation demonstrate the impact of VCE-based methods (Table 1, (Ma et al., 2022)):

Method	AUC@15°
Five-point + BA (SIFT/SuperGlue)	~10%
VCs only (w/ coplanarity BA)	~16%
Combined (classic + VC)	~18%

These indicate VCE minimization yields pose estimates within $K_2$4 to $K_2$5 even under extreme baselines.

6. Relation to Standard Correspondence and Broader Implications

Minimizing Virtual Correspondence Error generalizes bundle adjustment to “virtual” rather than purely co-visible correspondences. Whereas standard feature matching relies on direct pixel-to-3D-point associations visible across multiple views, the VC paradigm operates solely on the intersection geometry of rays, independent of direct visibility.

A plausible implication is that the VCE formulation unlocks camera pose and scene geometry estimation in scenarios with little or no visual overlap, where traditional feature-based methods fail. This framework also allows integration of prior knowledge (e.g., human shape estimation) into geometric reasoning via ray-mesh intersection.

7. Applications and Future Directions

The virtual correspondence and VCE approach enables robust estimation of camera layout and scene structure across wide baselines, supporting downstream tasks such as:

• Multi-view scene reconstruction in low-overlap scenarios
• Novel view synthesis from sparse images
• Improved pose estimation in human-centric environments

Ongoing work could explore alternative geometric cues for virtual correspondence detection and further generalization of the VCE formulation beyond current human-centric priors (Ma et al., 2022).