Learnable Cost Volumes

• Learnable cost volumes are differentiable modules that adapt classical matching techniques by using trainable similarity metrics to model complex high-dimensional correlations.
• They employ techniques such as learnable Mahalanobis metrics via Cayley reparameterization and adaptive convolutional shifting to optimize performance in stereo and optical flow tasks.
• Integration into deep architectures improves accuracy and robustness while maintaining computational efficiency, as evidenced by better performance on benchmarks like KITTI and Sintel.

A learnable cost volume is a parameterized, differentiable module designed to enhance the representational power and flexibility of classical cost volumes in dense correspondence estimation problems such as optical flow and stereo matching. Conventional cost volumes typically rely on hand-crafted or fixed similarity metrics, limiting their capacity to model complex correlations in high-dimensional feature spaces, particularly in the context of deep learning architectures. Learnable cost volumes address these constraints by allowing end-to-end adaptation of the matching kernel, shifting mechanism, or aggregation strategy, thereby optimizing both accuracy and robustness across a wide spectrum of geometric and photometric scenarios.

1. Mathematical Foundations and Evolution

Classical cost volumes aggregate matching evidence between two feature maps (for example, from stereo pairs or sequential frames) at each spatial location $p$ and offset or disparity $d$. The vanilla volume construction is typically:

$C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$

where $f_1, f_2 \in \mathbb{R}^{C \times H \times W}$ are local feature descriptors. This approach is limited by its use of the standard Euclidean inner product, which ignores inter-channel correlations and imposes uniform weighting among channels. Such restrictions become detrimental as networks and input scenes grow in complexity (Xiao et al., 2020).

Learnable cost volumes generalize this mechanism by employing either (a) a learnable matching metric (elliptical inner product with a positive definite kernel), (b) data-adaptive shifting via convolutional filters, or (c) hybrid multi-scale or multi-modal aggregation.

2. Learnable Mahalanobis Metrics via Spectral-Cayley Reparametrization

One major paradigm for learnable cost volumes replaces the Euclidean product with an elliptical inner product (Mahalanobis metric):

$$C_{\text{LCV}}(p, d) = f_1(p)^\top M f_2(p + d),\qquad M \succ 0$$

Here, $M \in \mathbb{R}^{C \times C}$ is a learned, positive definite kernel. $M$ admits spectral decomposition:

$$M = Q^\top \Lambda Q,\quad Q \in O(C),\, \Lambda = \mathrm{diag}(\lambda_1, \ldots, \lambda_C),\, \lambda_i > 0$$

To ensure constraint preservation during gradient-based optimization, $Q$ is parameterized via the Cayley transform acting on a skew-symmetric matrix $K$:

$d$0

The eigenvalues $d$1 are parameterized by a monotonic mapping (e.g., arctangent-based) to guarantee positivity:

$d$2

Training proceeds by back-propagating through unconstrained parameters $d$3 and $d$4, yielding a fully differentiable and constraint-satisfying module (Xiao et al., 2020).

3. Alternative Formulations: Learnable Shifting and Cost Volume Construction

In settings where the geometric relationship between the two images is complex or non-uniform (e.g., 360° stereo under equirectangular projection), spatial shifts required to build the cost volume are not constant across the image. 360SD-Net incorporates a learnable shifting filter, implemented as a grouped convolutional layer:

Let $d$5 denote features from the top/bottom camera views. For each disparity $d$6 and spatial location $d$7, the shifted bottom features are:

$d$8

where $d$9 is a learnable filter and $C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$0 denotes convolution. This mechanism is initialized to a fixed 1-pixel shift and, after a warm-up phase, becomes trainable to learn distortion-aware correspondence shifts (Wang et al., 2019).

The resulting cost volume concatenates these shifted features with the unshifted reference features, feeding downstream aggregation and regression modules.

4. Integration in Deep Architectures

Learnable cost volumes are modular and compatible with established deep architectures for optical flow and stereo matching such as PWC-Net, VCN, RAFT, and variants based on hourglass aggregation or U-Net feature extraction. Typical integration strategies involve:

• Replacing all vanilla inner-product-based cost volume operators with learnable versions at each pyramid level.
• Adding level-specific kernel parameters ($C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$1, $C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$2, $C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$3) or filter weights, resulting in minimal parameter overhead ($C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$457,000 weights for a five-level VCN hierarchy, compared to $C_{\text{std}}(p, d) = f_1(p)^\top f_2(p + d)$56M total parameters).
• For hybrid multi-scale/hybrid volumes (e.g., MSCVNet), compositing several 3D cost volumes constructed via classical (Census, absolute difference) and learned (correlation) modules, then aggregating via lightweight 2D cascade hourglass networks (Jia et al., 2021).

These design choices yield negligible architectural disruptions and minimal computational overhead relative to baseline architectures.

5. Training, Loss Formulations, and Differentiability

Learnable cost volumes are amenable to end-to-end supervision under both regression and classification loss paradigms. Common approaches include:

• Standard L1 or Huber (smooth-L1) regression between soft-argmin disparity predictions and ground truth.
• Multi-scale supervision inside hourglass modules.
• Specialized loss formulations incorporating discontinuity detection in disparity maps, with down-weighted penalties near jumps or occlusions (Jia et al., 2021).

Backpropagation passes through all cost volume parameters or shifting filters, with positive definiteness constraints or convolutional weight freezing handled as prescribed by each technique.

6. Empirical Performance and Robustness

Learnable cost volumes demonstrate improvements in both in-domain accuracy and out-of-domain robustness:

• Optical flow: Incorporation of the Cayley-parameterized learnable cost volume in PWC-Net, VCN, and RAFT reduces average endpoint error (AEPE) and outlier ratios across the Sintel and KITTI benchmarks (e.g., RAFT+LCV achieves 1.31 AEPE on Sintel Final train, improving over the vanilla baseline) (Xiao et al., 2020).
• Stereo matching: In 360° stereo, learnable cost volumes yield up to 21.1% improvement (RMSE) over fixed-shift cost volume baselines (Wang et al., 2019).
• Further, learnable cost volumes enhance robustness to photometric perturbations (illumination, noise, adversarial patch attacks) and offer improved edge sharpness without additional run-time cost (Xiao et al., 2020, Jia et al., 2021).

Ablation studies confirm the necessity of correct positive definiteness enforcement and the superiority of fully-learned (rotation+scaling) kernels over axis-aligned or naive 1×1 conv parameterizations.

7. Generalization, Limitations, and Extensions

Learnable cost volumes provide a direct path to closing the gap between hand-crafted similarity functions and fully end-to-end 4D cost volume architectures. Approaches such as MSCVNet combine multi-scale and multi-modal evidence, dramatically reducing memory and computational cost while preserving competitive accuracy and enabling real-time inference (41 ms per stereo pair with accuracy rivaling slower 4D-convolutional methods on KITTI) (Jia et al., 2021).

Constraints remain in scenarios demanding very high accuracy in textured or high-disparity regimes, where top-performing full 4D cost volume architectures still surpass lighter-weight approaches. There is promising evidence to suggest further generalizations: incorporation into multi-view stereo, fusion with depth completion/LiDAR modalities, and extension to other 3D correspondence tasks (e.g., non-rigid registration) via analogous learnable volume constructions.

---

Key References:

• "Learnable Cost Volume Using the Cayley Representation" (Xiao et al., 2020)
• "Multi-Scale Cost Volumes Cascade Network for Stereo Matching" (Jia et al., 2021)
• "360SD-Net: 360° Stereo Depth Estimation with Learnable Cost Volume" (Wang et al., 2019)