Forward Curvature-Matching (FCM) Update Method

• The paper introduces a curvature-informed adaptive update mechanism that eliminates heuristic tuning by dynamically selecting optimal step sizes.
• FCM leverages a finite-difference approach to estimate local gradient curvature, enabling robust 3D reconstruction from single- and multi-view inputs.
• Empirical results on ShapeNet demonstrate improved F-scores and Chamfer Distances with reduced computational costs compared to fixed-step methods.

Forward Curvature-Matching (FCM) is an adaptive update method introduced for likelihood refinement within diffusion-based 3D reconstruction pipelines, with particular emphasis on reconstructing point clouds from images. Unlike prior generative-model-based 3D reconstruction approaches, FCM dynamically determines curvature-informed step sizes for each likelihood update, removing the need for heuristic tuning and task-specific retraining. FCM is designed for integration into the reverse sampling process of diffusion models, and supports high-fidelity, flexible reconstructions from both single- and multi-view inputs, as well as across multiple input modalities through operator substitution.

1. Motivation and Context in Diffusion Sampling

In 3D reconstruction tasks employing diffusion models, generating a sample that is consistent with given observations (e.g., images, depth maps) necessitates an update step incorporating data likelihood information. Traditional methods apply a fixed step size $\gamma$ to each likelihood-gradient-based refinement: $$\mathbf x_t \leftarrow \mathbf x_t + \gamma \nabla_{\mathbf x} \log p(\mathbf y \mid \mathbf x)$$ Analytic step-size rules are rendered ineffective because the rendering operator $\mathcal R$ is nonlinear and lacks a tractable adjoint. Empirically, fixed-$\gamma$ heuristics yield suboptimal convergence and generalize poorly across input types or modalities, often requiring hand-tuning and retraining per task configuration. FCM replaces this procedure with a forward-mode, curvature-informed rule that is robust to the nonlinearity of $\mathcal R$ and requires no retraining when changing input conditions.

2. Mathematical Derivation of Curvature-Based Step Size

The core principle underlying FCM is the computation of an optimal step size $\alpha$ for likelihood refinement, based on second-order local information of the measurement loss

$$\mathcal L(\mathbf x) = \| \mathbf y - \mathcal R(\mathbf x) \|_2^2$$

Given the gradient $g = \nabla \mathcal L(\mathbf x)$ and Hessian $H = \nabla^2 \mathcal L(\mathbf x)$, a second-order Taylor expansion along the direction $-g$ yields: $$\mathcal L(\mathbf x-\alpha g) \approx \mathcal L(\mathbf x) - \alpha g^T g + \tfrac12 \alpha^2 g^T H g + \mathcal O(\alpha^3)$$ Optimizing with respect to $\alpha$, the Newton-optimal step is

$\alpha^* = -\frac{g^Tg}{g^T H g}$

The local directional curvature is given by

$\kappa = \frac{g^T H g}{\|g\|^3}$

so the optimal step size may equivalently be written as $\alpha^* = -\|g\|/\kappa$. This formulation ensures that the update size adapts to the local geometry of the likelihood landscape.

3. Finite-Difference Curvature Estimation

Direct computation of $Hg$ is infeasible due to the lack of an adjoint or tractable Hessian through $\mathcal R$. FCM circumvents this by employing a finite-difference approach, evaluating the change in gradient under a small, scaled step along the (negative) gradient direction. Specifically, for each update: $$\delta = \delta_0 \frac{\|\mathbf x\|}{\|g\|} \ \mathbf x' = \mathbf x - \delta g \ g' = \nabla \mathcal L(\mathbf x')$$ The curvature is then approximated by

$H g \approx \frac{g - g'}{\delta}$

with $\delta_0$ a fixed small constant (e.g., $2 \times 10^{-2}$), and the rescaling ensuring consistency across a wide range of parameter magnitudes.

4. Algorithmic Integration with Diffusion Models

The FCM method is implemented as a refinement module within each reverse (denoising) step of a DDIM-based diffusion sampler. The iteration at each timestep $t$ proceeds as follows:

1. DDIM Prior Step:

$$\hat{\mathbf X}_{0|t} \gets \frac{\mathbf X_t - \sqrt{1-\bar\alpha_t} \epsilon_\theta(\mathbf X_t, t)}{\sqrt{\bar\alpha_t}}$$

1. FCM Likelihood Refinement:
   • Initialize $\mathbf x_0 \gets \hat{\mathbf X}_{0|t}$
   • For $k = 0, ..., K-1$ (with $K \approx 4$ typically):
     • Compute $$g_k = \nabla \|\mathbf y - \mathcal R(\mathbf x_k)\|_2^2$$
     • Set $\delta_k = \delta_0 \|\mathbf x_k\| / \|g_k\|$
     • Compute perturbed gradient $$g'_k = \nabla \|\mathbf y - \mathcal R(\mathbf x'_k)\|_2^2$$, where $\mathbf x'_k = \mathbf x_k - \delta_k g_k$
     • Directional curvature: $h_k = (g_k - g'_k) / \delta_k$
     • Barzilai–Borwein raw step:

   $$\alpha_k^{\text{raw}} = \frac{\|g_k\|^2}{\langle g_k, h_k \rangle + \varepsilon}$$

- Cap $\alpha_k = \min(\alpha_k^{\text{raw}}, 1/L)$, with $L$ a user-defined Lipschitz bound and $\varepsilon = 10^{-12}$ for stabilization. - Armijo check: if

$$\mathcal L(\mathbf x_k - \alpha_k g_k) > \mathcal L(\mathbf x_k) - \eta_{\mathrm{FCM}}\alpha_k \|g_k\|^2$$

halve $\alpha_k$. ($\eta_{\mathrm{FCM}}$ typically $10^{-4}$.) - Set $\mathbf x_{k+1} = \mathbf x_k - \alpha_k g_k$ - Output $\tilde{\mathbf X}_{0|t} = \mathbf x_K$

1. DDIM Update:
   • Combine with stochastic prior and noise to form next diffusion state.

This inner refinement structure enables FCM to supply a curvature-informed adaptive update at each denoising iteration, interfacing only through the rendering operator via first-order gradients and forward differences.

5. Empirical Performance and Flexibility

FCM demonstrates significant improvement over heuristic, fixed-step-size likelihood updates according to established 3D reconstruction metrics (F-score, Chamfer Distance (CD), Earth Mover’s Distance (EMD)). On ShapeNet single-view, FCM yields F-score 0.382 (vs. 0.337 for DDPM+DPS, 0.312 for DDIM+DPS) and Chamfer Distance 0.561 (vs. 0.688/0.728). FCM achieves similar or higher quality at reduced computational cost, converging within approximately 256 neural function evaluations (NFEs), in contrast to 1000+ NFEs required for fixed-step DPS, reducing runtime by about 30%.

FCM’s step size rule is independent of measurement modality; switching from single-view to multi-view or to depth map input requires only a substitution of the rendering operator $\mathcal R$, without retraining the diffusion prior. This direct operator modularity extends applicability across tasks.

6. Hyperparameters, Assumptions, and Limitations

The assumptions underpinning FCM are local $L$-smoothness and lower boundedness of the loss $\mathcal L(\mathbf x)$. Local convexity is not strictly required for empirical robustness. Key hyperparameters include

• $\delta_0$: Initial finite-difference scale (e.g., $2\times10^{-2}$ for ShapeNet, $6\times10^{-3}$ for CO3D)
• $L$: Upper bound for capping (typical range $2/3$ to 10)
• $\eta_{\mathrm{FCM}}$: Armijo factor (typically $10^{-4}$)
• $\varepsilon$: Numerical stabilizer ($10^{-12}$)
• $K$: Number of inner refinement steps (typically 4)

Overestimating $L$ simply produces more conservative updates; underestimates are mitigated by the Armijo halving check. The method is robust to $\delta_0$ variations within an order of magnitude and to broad $L$ settings. Main limitations arise when reconstructing very thin geometric structures, which can be lost due to rasterizer radius constraints, and with complex or out-of-distribution shapes, which may degrade curvature estimation.

7. Computational Complexity and Implementation Considerations

Each FCM likelihood refinement involves two backward passes for gradient computation (at $\mathbf x_k$ and $\mathbf x'_k$) and three forward passes for rendering ($\mathbf x_k$, $\mathbf x'_k$, and the Armijo check). The overall number of diffusion timesteps is typically 256 with FCM, as opposed to over 1000 with fixed-step DPS. The Barzilai–Borwein step and capping with $1/L$ combined with a single Armijo back-off efficiently regularize noisy curvature estimates, with the Armijo condition serving as a lightweight safeguard in place of a full Wolfe line search.

The modularity and forward-differentiable design of FCM enable rapid adaptation across data modalities and experimental conditions, rendering it applicable for practical high-fidelity 3D reconstruction without retraining or extensive parameter tuning.

---

FCM realigns the likelihood update process within diffusion posterior sampling by employing forward automatic differentiation and adaptive, curvature-informed step sizing. This architectural advance delivers accelerated and more accurate 3D reconstructions in a unified, operator-flexible framework, as quantitatively established on canonical datasets.