Epipolar Loss Weighting Strategies

• Epipolar loss weighting is a strategy that integrates epipolar geometric constraints into loss functions to enhance multiview learning accuracy.
• It employs explicit or implicit weighting schemes to balance geometric consistency with photometric, keypoint, or correspondence losses.
• Empirical evaluations show that incorporating epipolar terms improves outcomes in self-supervised depth, pose estimation, and neural reconstruction tasks.

Epipolar loss weighting refers to methodological strategies that incorporate epipolar geometric constraints into loss functions for optimization in multiview learning, correspondence estimation, and inverse problems. These strategies modulate the relative influence of epipolar consistency versus other objectives (typically photometric, keypoint, or correspondence losses) via explicit or implicit weighting schemes. Epipolar loss weighting is central in applications ranging from self-supervised depth and pose estimation, semi-supervised keypoint detection, correspondence refinement, to neural field reconstruction in tomography. Key developments have focused on the construction of effective epipolar loss terms, the integration of these losses with other objectives, selection and scheduling of the associated weights, and empirical evaluation of their quantitative impact.

1. Mathematical Formulations of Epipolar Losses

Epipolar loss terms operationalize multiview geometry by penalizing violations of the fundamental (or essential) matrix constraints between corresponding points or distributions in different images:

• Point-to-line constraint: For correspondences $(p_i, q_i)$ in homogeneous coordinates, the algebraic distance to the epipolar line is

$$\mathrm{dist}(l, q) = \frac{|a x_0 + b y_0 + c|}{\sqrt{a^2 + b^2}},$$

with $l = (a, b, c)^T = F p_i$, and forms the basis of the geometric loss in self-supervised VO pipelines (Shen et al., 2019).

• Per-pixel epipolar violation: For depth and ego-motion learning, the epipolar loss at pixel $p_1$ is $|\tilde{p}_2^T E \tilde{p}_1|$, with $\tilde{p}$ denoting normalized coordinates and $E$ the essential matrix. This distance directly enters either as an explicit penalty or as a weighting factor on photometric residuals (Prasad et al., 2018).
• Distributional extensions: MONET generalizes the classical epipolar constraint by defining an epipolar-divergence loss between view-specific keypoint heatmaps, using a KL divergence between distributions $Q_i(\theta)$ (pooled along epipolar planes) and $Q_{j \rightarrow i}(\theta)$ (transferred from the other view) (Yao et al., 2018).
• Correspondence-level regression: SCENES penalizes the signed distance of predicted correspondences to their epipolar lines, with both coarse (classification) and fine (regression) epipolar losses, weighted by a hyperparameter $\lambda$ (Kloepfer et al., 2024).
• CT consistency conditions: In Epi-NAF, epipolar consistency is enforced between corresponding epipolar lines in cone-beam projections by matching their Radon-derivative along paired lines, again weighted by a scalar $\lambda$ (Gilo et al., 2024).

2. Integration and Weighting within Loss Objectives

Epipolar terms are incorporated into overall training objectives via explicit scalar weights or through their functional role in modulating other losses:

• Additive weighting: In joint depth and pose estimation, the total loss is a sum of photometric, smoothness, and epipolar losses, weighed as

$$L_{\text{total}} = M(P_M) \odot L_{\mathrm{img}} + w_s L_{\mathrm{smooth}} + w_g L_{\mathrm{geo}} + \cdots$$

with $w_g$ (e.g., $w_g=0.001$) specifying the contribution of the epipolar term (Shen et al., 2019). No scheduled annealing is typically employed.

• Implicit weighting via the loss structure: In some methods, the epipolar error does not appear as a separate additive term. Instead, it functions as a pixel-wise weight. For instance, (Prasad et al., 2018) applies an exponential factor to the photometric error:

$$L_{\mathrm{warp}} = \frac{1}{N} \sum_{s=1}^S \sum_p |I_t(p) - \hat{I}_s(p)| \; \exp(|\tilde{p}_2^T E \tilde{p}_1|).$$

This automatically down-weights high-violation regions without adjusting a scalar loss balance.

• Semi-supervised and multi-term balancing: In MONET, the total loss is $L_L + \lambda_e L_E + \lambda_p L_B$; $\lambda_e$ (typically $5$) is tuned so that the geometric loss is comparable in scale to the supervised loss (Yao et al., 2018).
• Coarse-fine tradeoff: SCENES introduces a two-term epipolar loss, blending classification and regression penalties with a hyperparameter $\lambda$ controlling their relative weight, retained from the pre-trained supervised model (Kloepfer et al., 2024).
• Consistency regularization: In Epi-NAF for CT, the total loss is

$$\mathcal{L} = \mathcal{L}_{\text{Recon}} + \lambda \mathcal{L}_{\text{ECC}},$$

with $\lambda$ tuned via validation; a staged schedule (warm-up then fixed $\lambda$) is effective (Gilo et al., 2024).

3. Hyperparameter Selection and Scheduling

Careful selection and scheduling of loss weights is critical for stability and performance:

• Fixed weights: Many works, such as (Shen et al., 2019) and (Prasad et al., 2018), use fixed scalar weights ($w_g=0.001$ for epipolar loss, or an exponential map with no explicit $\lambda_\text{epi}$), avoiding dynamic annealing or epoch-based schedules.
• Empirical tuning using validation: MONET selects $\lambda_e$ by grid-search on a validation set, targeting values where $\mathcal{L}_E$ matches the scale of $\mathcal{L}_L$, with recommended sweeps in the range $[1,10]$ (Yao et al., 2018). If the epipolar geometry is noisy, a lower initial value is preferred, possibly ramped up once predictions stabilize.
• Blended fixed ratios: SCENES maintains the same coarse/fine epipolar loss split as inherited from the original supervised model. Ablations indicate both terms are necessary, as each alone is insufficient for pose accuracy or match reliability (Kloepfer et al., 2024).
• Scheduled activation: Epi-NAF defers the epipolar consistency term ($\lambda=0$) for an initial “warm-up” period (200 epochs), after which a fixed $\lambda$ is enabled ($10^{-3}$ for ill-posed regimes, $10^{-4}$ for less ill-posed), chosen by validation (Gilo et al., 2024). Overly large $\lambda$ values degenerate quality; too small gives negligible effect.
• Hard vs. soft constraints: In optimization-based methods (e.g., JET (Bradler et al., 2017)), the epipolar penalty multiplier $\lambda$ is set very large (imposing a hard constraint per-feature in the inner loop), bypassing the need for delicate outer-loop weighting.

4. Empirical Evaluation and Quantitative Impact

Ablation studies consistently demonstrate the benefit of epipolar loss weighting over baselines without geometric supervision:

• Depth and pose estimation: Adding the epipolar matching loss ($w_g=0.001$) to self-supervised depth/pose recovery improved Abs Rel error from $0.163$ to $0.156$ and RMSE from $6.397$ to $6.139$, with relative pose error on KITTI Seq. 09 dropping from $0.014$ m (photometric only) to $0.0089$ m (Shen et al., 2019).
• Two-view depth with epipolar weights: Using the exponential epipolar weighting, Abs Rel error improved from $0.199$ to $0.175$, and RMSE from $6.314$ to $6.378$ compared to the non-epipolar version (Prasad et al., 2018).
• Keypoint detection: MONET’s best PCK was achieved at $\lambda_e=5$, with values too low ($0.1$) under-utilizing geometry and too high ($20$) amplifying noise (Yao et al., 2018).
• Correspondence and pose AUC: SCENES observed that only the mixed epipolar loss (neither coarse nor fine alone) yields both high match precision and pose AUC; the ablation table confirms the drop in either metric if either term is dropped (Kloepfer et al., 2024).
• CT reconstruction: Epi-NAF’s inclusion of the epipolar-derivative consistency term ($\lambda=10^{-3}$) yielded $0.3-0.8$ dB improvement in PSNR and reduced structural artifacts versus $\lambda=0$ (Gilo et al., 2024).

5. Task-Specific Implementation Details

Implementation and sampling strategies are tailored to the data modality and optimization pipeline:

• Feature selection: In learning-based VO, $8000$ SIFT features per image are detected, with $\approx2000$ inliers after RANSAC; $100$ are randomly sampled per minibatch for epipolar loss (Shen et al., 2019).
• Matrix estimation: Classical (normalized eight-point or five-point) solvers are used to estimate $F$ or $E$ from sparse matches. Epipolar supervision is not backpropagated into this estimation (Prasad et al., 2018).
• Distributional pooling: MONET computes maximum-probability pooling along rectified scan-lines using a homography to transform epipolar lines into horizontal lines, reducing cost and aliasing (Yao et al., 2018).
• Correspondence granularity: SCENES defines epipolar neighborhoods via a thresholded distance $(\theta \cdot w/2)$, setting $\theta \approx \sqrt{2}$ (Kloepfer et al., 2024).
• Epipolar line sampling for inverse problems: In Epi-NAF, epipolar planes are sampled by fixing two source angles and a random internal point; discrete central differences approximate the necessary derivatives along paired lines (Gilo et al., 2024).

6. Practical Guidelines for Tuning and Deployment

Several concrete strategies and recommendations are reported:

• Loss scaling: Target initial scales where epipolar loss terms are comparable to the main (supervised or photometric) terms (Yao et al., 2018).
• No dynamic annealing unless necessary: Fixed weights are preferred unless instability or overfitting is observed; simple ramp-up schedules are effective if initial geometry or predictions are noisy.
• Validation-guided selection: Use a representative validation subset and grid search to optimize weighting hyperparameters, particularly in scenarios with variable geometric accuracy across datasets.
• Hard constraint regime for optimization methods: For iterative solvers, setting the epipolar penalty weight $\lambda$ very high substitutes the epipolar constraint exactly, which often leads to best geometric fidelity (Bradler et al., 2017).
• Down-weight outliers/occlusions: Epipolar weighting (especially exponential) naturally suppresses the influence of outlier pixels and regions that violate geometric assumptions due to dynamics, occlusions, or poor matches (Prasad et al., 2018).

7. Limitations and Domain-Specific Considerations

Reported limitations include:

• Dependence on reliable geometric estimates: Epipolar losses rely on correct estimation of $F$ or $E$, which is sensitive to calibration and feature-matching accuracy. Epipolar weights can become noisy if the underlying geometry is inaccurate (Prasad et al., 2018).
• Risk of over-penalization: Aggressive weighting (e.g., large $\lambda$ or exponential maps) can "oversmooth" or over-regularize, suppressing useful signal and leading to underfitting, especially in ill-posed or limited-angle tomographic settings (Gilo et al., 2024).
• No backpropagation through classical solvers: In most pipelines, $F$ or $E$ is externally computed; the network cannot learn to correct errors in the geometric estimate.
• Transfer to new domains: Epipolar losses function as a drop-in replacement for more strongly supervised correspondence or 3D supervision, provided that the epipolar geometry can be estimated or bootstrapped from pose or odometry, even in the absence of 3D ground truth (Kloepfer et al., 2024).

Epipolar loss weighting provides a geometrically grounded, task-agnostic regularization mechanism that enforces multiview consistency. Carefully tuned, it delivers measurable benefits in depth, correspondence, and reconstruction accuracy, with transparent empirical guidance available for weighting selection and integration in a range of learning and optimization pipelines.