Adaptive Geodesic Convolutions (AGC)

• Adaptive Geodesic Convolutions are metric-based operators that replace fixed Euclidean kernels with unit geodesic balls defined by learnable Finsler metrics over image manifolds.
• They employ a heat-flow approximation and geodesic flow to sample local, signal-dependent neighborhoods, ensuring geometric regularization and efficient parameter use.
• AGC enhances performance in image denoising and classification tasks by reducing parameter overhead while aligning convolution filters with critical image structures.

Adaptive Geodesic Convolutions (AGC) are a specialized form of metric-based convolutional operators introduced within the framework of "Metric Convolutions: A Unifying Theory to Adaptive Image Convolutions" (Dagès et al., 2024). AGC generalize standard convolution by replacing the fixed Euclidean kernel support with a unit geodesic ball derived from a (potentially signal-dependent) Finsler metric over the image manifold. The construction results in a locally adaptive mechanism where the spatial neighborhood sampled for convolution is shaped by the geometry of the underlying data, yielding interpretable, parameter-efficient, and geometrically regularized operators applicable as drop-in replacements for conventional CNN layers.

1. Mathematical Foundations

Images are modeled as parameterized two-dimensional manifolds $X$, interpreted as height-maps over a planar domain $\Omega \subset \mathbb{R}^2$, each equipped with tangent planes $T_x X$. A Finsler metric $F$ is assigned to each $T_x X$, formally $F_x: T_x X \to \mathbb{R}_+$, satisfying positive 1-homogeneity and the triangle inequality. The Riemannian case has $F_x(u) = \sqrt{u^\top M(x) u}$ for positive-definite $M(x)$; the Randers case introduces a drift $\omega(x)$ so $F_x(u) = \sqrt{u^\top M(x) u} + \omega(x)^\top u$ with $\|\omega(x)\|_{M(x)^{-1}} < 1$.

For a curve $\gamma: [0,1] \to X$, its length under $F$ is $$L_F(\gamma) = \int_0^1 F_{\gamma(t)}(\gamma'(t)) dt$$. Geodesic distance is defined by $$d_F(x,y) = \inf_{\gamma(0)=x, \gamma(1)=y} L_F(\gamma)$$. The unit geodesic ball at $x$ is $B^g_1(x) = \{ y \in X \mid d_F(x,y) \leq 1 \}$; for Riemannian metrics, this set is symmetric, while for Randers metrics it may be skewed. Sampling from the unit geodesic ball produces neighborhoods that adapt both spatially and to the signal, in contrast to the rigid grids of ordinary convolution.

2. Construction of Adaptive Geodesic Convolutions

The AGC operator locally filters a function $f$ by

$$C_f(x) = \int_{B^g_1(x)} k(w) \cdot f(x \oplus_g w) \, dw,$$

where $k(w)$ is a kernel on the unit geodesic ball, and $x \oplus_g w$ denotes advancing from $x$ along a geodesic in direction $w \in T_x X$ (with $F_x(w) \leq 1$) to arrive on the manifold at $x \oplus_g w$.

2.1 Signal-Dependent Metric Design

A parametric Finsler (Randers) metric $F_x^\gamma$ is constructed using a small number of parameters $\gamma(x) = (M(x), \omega(x))$, typically 5–7 per pixel. These are produced by a lightweight intermediate convolutional layer and decoded to yield valid metric parameters, with positive-definiteness and norm constraints enforced via offsets and sigmoid-based clamping.

2.2 Approximate Geodesic Ball Sampling

Exact computation of $B^g_1(x)$ requires solving a geodesic PDE, which is computationally prohibitive. The practical AGC approach employs a three-stage heat-flow-inspired approximation:

1. Heat-Flow Approximation: The Dirac $\delta_x$ at $x$ is convolved with a small-time Finsler heat kernel $h_x(y) \propto t^{-1}\exp(-F_x^*(y)^2/(4t))$, yielding $\delta_{x,t}$.
2. Geodesic Flow Field: The normalized gradient field $$v_x(y) = -\nabla \delta_{x,t}(y)/\|\nabla \delta_{x,t}(y)\|$$ forms approximate geodesics under suitable conditions.
3. Stencil Flow: Radial stencil points in the tangent plane (e.g., polar grid samples) at $x$ are advected along $v_x$ to produce a discrete sample set $\Delta^g_x$ approximating $B^g_1(x)$.

2.3 Geodesic Offset and Kernel Warping

Each sampled $w_{i,j} = s_i \cdot u_{\theta_j}$ is mapped to the image manifold by following an approximate geodesic, leading to the warped position $x \oplus_g w_{i,j}$. The feature map $f$ is interpolated (typically bilinearly) at these positions, each weighted by $k(w_{i,j})$, and the sum approximates the convolution integral.

2.4 Resampling and Interpolation

Since $x \oplus_g w_{i,j}$ need not coincide with a grid point, bilinear or higher-order interpolation is employed. Non-uniform sampling weights $m_x(w)$, such as those from barycentric kernels or regression, can further refine the local integral.

3. Implementation Considerations

AGC can be seamlessly implemented as a convolutional layer in existing deep networks. All components—metric parameter decoding, heat-diffusion, gradient operations, warping, and interpolation—are differentiable, facilitating end-to-end training.

3.1 Forward Pass Outline

γ = Conv_intermediate(f) # 5–7 channels (M, ω) = decode_metric(γ) # M ≻ 0, ‖ω‖_{M⁻¹} < 1 for each pixel x: δ = one-hot(x) δ_t = heat_diffuse(δ, t, M, ω) v = -normalize(∇δ_t) for each polar sample (s_i, θ_j): p = s_i * unit_vec(θ_j) y = advect(p, v, x, Δt) sample_positions[x, i, j] = x + y C_f(x) = sum_ij k(s_i, θ_j) * f_interpolated(sample_positions[x, i, j])

3.2 Backpropagation

The key computational graph path is through the warping positions, which depend on the differentiable computation of $\gamma(x)$. The gradients flow from the interpolated feature values to the metric parameters through both position and value.

3.3 Parameter Efficiency

AGC layers require only 5–7 intermediate channels per pixel (encoding $M, \omega$), independent of the kernel size $k$. In contrast, deformable convolutions demand $2k^2$ channels for offsets alone. Kernel weights in AGC match $k^2$ (or may be fixed uniform). The resulting additional channel count is $\mathcal{O}(1)$ for AGC versus $\mathcal{O}(k^2)$ for deformable convolutions.

4. Empirical Performance and Comparative Properties

AGC exhibits competitive or superior benchmark results for key vision tasks.

4.1 Denoising

• On Cameraman (σ = 0.3), AGC achieves PSNR ≈ 30.2 dB versus ≈ 28.9 dB for deformable convolutions, with fewer parameters and a markedly reduced generalization gap.
• On larger datasets (BSDS300, PascalVOC), AGC matches or slightly outperforms deformable convolution as $k$ increases (e.g., $k=5,\,11,\,31$), in both MSE and the generalization gap $\Delta_{MSE}$.

4.2 Image Classification

• Replacement of 3×3 convolutions with AGC in ResNet-18 (layers 2–4) yields CIFAR-10 test accuracy ≈ 93.1% (transferred weights) compared to 92.6% (deformable) and 92.64% (standard).
• On CIFAR-100, top-1 accuracy for AGC is ≈ 70.4% versus deformable’s ≈ 70.0%. AGC retains stability across fixed kernel weights and training schemes, unlike deformable variants which are prone to overfitting and collapse.

4.3 Interpretability and Geometric Regularization

• The learned metric parameters $(M(x), \omega(x))$ are directly visualizable; eigenvectors of $M(x)$ align with image edges, and $\omega(x)$ quantifies directional drift.
• AGC’s geometric bias ensures that neighborhoods avoid crossing high-gradient edges, implementing edge-preserving filtering analogous to geodesic avoidance.

5. Generalization Capacity and Extensions

Due to the minimal (5–7 per pixel) parametric overhead, AGC demonstrates superior generalization, particularly on small datasets and high-resolution imagery. The metric-convolution framework supporting AGC generalizes beyond Randers to arbitrary parametric Finsler metrics, accommodates multiscale and nonlocal geodesic balls, and extends readily to surface (mesh) data. This adaptability enables application to a broad class of geometrically motivated tasks.

In summary, Adaptive Geodesic Convolutions realize the principle of supporting convolutional kernels on unit geodesic balls defined by explicit, learnable metrics over an image manifold. Through a heat-flow-based geodesic sampling procedure coupled with efficient parameterization, AGC delivers geometric regularization and parameter efficiency alongside state-of-the-art performance on canonical denoising and classification challenges (Dagès et al., 2024).