---
title: Derivative-Free LM Optimization
url: https://www.emergentmind.com/topics/derivative-free-lm-dflm
type: topic
---

# Derivative-Free LM Optimization

A derivative-free Levenberg–Marquardt (DFLM) algorithm is any method for nonlinear least-squares optimization that incorporates a Levenberg–Marquardt-style (LM) trust-region or regularization framework, but does not require analytic derivatives. DFLM methods are designed for problems where the Jacobian of the residuals is unavailable, expensive, or unreliable. They construct gradient or Jacobian models from function evaluations using techniques such as orthogonal spherical smoothing, interpolation, $\ell_1$-minimization, or stochastic sampling, and are especially relevant in high-dimensional, black-box, or sparse-least-squares contexts. Modern DFLM algorithms can provide provable convergence to stationary points, with global complexity bounds matching those of gradient-based methods, and demonstrate strong empirical performance on large-scale or structured least-squares problems.

## 1. Problem Formulation and Core Principles

The canonical target of DFLM algorithms is the nonlinear least-squares problem:
\[
\min_{x\in\mathbb R^n} f(x) = \frac{1}{2}\|r(x)\|^2 = \frac{1}{2}\sum_{i=1}^m [r_i(x)]^2,
\]
where $r: \mathbb R^n \to \mathbb R^m$ are smooth residuals and the analytic Jacobian $J(x) = [\nabla r_1(x), ..., \nabla r_m(x)]$ is unavailable or impractical to compute. DFLM techniques replace $J(x)$ with probabilistically accurate models $\widetilde{J}(x)$ derived from function evaluations, estimating the gradient $\nabla f(x) = J(x)^{\top} r(x)$ by $\widetilde{J}(x)^{\top} r(x)$.

The DFLM paradigm also extends to:
- **Equality-constrained nonlinear least-squares**: $\min f(x)$ subject to $c(x)=0$ where neither $J(x)$ nor the constraint Jacobian $J^c(x)$ is available [2507.05623].
- **Regularized least-squares**: $\min_x \Phi(x) := f(x) + r(x)$ where $r(x)$ is a convex, possibly nonsmooth regularizer with a cheap proximal operator [2407.14915].
- **Sparse Jacobian scenarios**: When $J(x)$ is known to be sparse but the sparsity pattern is unknown and function evaluation is expensive [2507.06772].

## 2. Construction of Derivative-Free Gradient and Jacobian Models

### Orthogonal Spherical Smoothing

DFLM algorithms approximate the Jacobian by averaging directional differences over random or orthogonal frames. For each residual $r_i$,
\[
\nabla \widetilde{r}_i(x) = \frac{n}{b} \sum_{j=1}^b \frac{r_i(x+\gamma u_j) - r_i(x)}{\gamma} u_j,
\]
where $\{u_1,\dots,u_b\}$ are $b$ orthonormal directions sampled uniformly from the sphere $\mathbb S^{n-1}$ and $\gamma$ is a smoothing radius [2407.12542, 2507.05623]. This construction satisfies:
\[
\mathbb E[\nabla \widetilde{r}_i(x)] = \nabla r_{s,i}(x), \quad \text{with} \quad \nabla r_{s,i}(x) = \frac{n}{\gamma} \mathbb E_{u \sim U(\mathbb S^{n-1})} [ r_i(x+\gamma u)u].
\]
Provided $b \geq n/2$, these models have controlled variance and bias as a function of $\gamma$, $b$, and the Lipschitz constants of $r_i$.

### Interpolation and $\ell_1$-Minimization for Sparse Jacobians

When Jacobians are sparse, DFLM can use compressed-sensing style interpolation. For each $r_i$, given $p \ll n$ random directions $v^j$ and $\sigma>0$, construct interpolation points $x_k + \sigma v^j$ and solve:
\[
\min_{g \in \mathbb R^n} \|g\|_1 \quad \text{s.t.} \quad \|A g - \sigma^{-1} y\|_2 \leq \xi,
\]
where $A$ comprises the $v^j$ as rows and $y$ are finite difference target values [2507.06772]. If $A$ satisfies a suitable restricted isometry property, the recovered $g$ is close (in probability) to the true sparse gradient.

### Quadratic and Fully Linear Models

For regularized or trust-region variants, DFLM forms quadratic models $p_k(s) = \frac{1}{2} \| r(x_k) + J_k s \|^2$ by interpolating $r$ at a poised set of points, where $J_k$ satisfies $r(y^t) = r(x_k) + J_k(y^t - x_k)$ for $t=1,\ldots,n$ [2407.14915]. This ensures that the model is "fully linear" on a trust region.

## 3. Algorithmic Structure and Parameter Updates

A typical DFLM iteration includes the following components [2407.12542, 2507.06772, 2407.14915]:

| Step | Operation | Purpose                                   |
|------|-----------|--------------------------------------------|
| 1    | Model construction  | Approximate $\nabla r_i(x_k)$ or $J(x_k)$ |
| 2    | Compute predicted step | Solve LM-type system $(\widetilde{J}^{\top}\widetilde{J} + \lambda_k I)d_k = -\widetilde{J}^{\top} r(x_k)$ |
| 3    | Step acceptance | Compare actual and predicted reduction: $ \rho_k = \frac{ \| r(x_k) \|^2 - \| r(x_k + d_k) \|^2 }{ \| r(x_k) \|^2 - \| r(x_k) + \widetilde{J}d_k \|^2 }$ |
| 4    | Update regularization | If $\rho_k > p_0$ (accept), decrease $\theta_k$ or $\lambda_k$; otherwise, increase them |
| 5    | Adapt smoothing, sample size, or interpolation radius | Reduce $\gamma$ as steps get smaller; increase number of directions/interpolation points as needed |

Parameter recommendations from the literature:
- Smoothing radius: $\gamma_k = \| d_{k-1} \|$
- Sample size: $b \geq n/2$ for orthogonal sampling
- Thresholds: $p_0 \in (0,1)$, $a_1>1$, $a_2\in(0,1)$, $a_1 a_2 < 1$, with specific $p_1, p_2$ for model gradient control
- Sparse Jacobian models: $p \sim s \log(n/s)$ interpolation points if $s$-sparsity is known [2507.06772].

## 4. Probabilistic Accuracy, Complexity, and Convergence Guarantees

DFLM methods offer formal guarantees on the probabilistic accuracy of their stochastic models, and on the global rate of convergence to first-order stationary points.

### Probabilistic Model Accuracy

DFLM algorithms attain $\alpha$-probabilistically $(\xi_1, \xi_2)$ first-order accurate gradient models if
\[
\Pr\left\{ \| \nabla \widetilde{f}(x_k) - \nabla f(x_k) \| \leq \xi_1 \| \nabla f(x_k) \| + \xi_2 \| d_{k-1} \| \right\} \geq \alpha,
\]
where $\xi_1, \xi_2$ scale with $\gamma_k$, $b$, and the Lipschitz constants. Taking $\gamma_k = \| d_{k-1} \|$ yields tighter bounds as stationarity is approached [2407.12542].

### Complexity Bounds

Under Lipschitz continuity and boundedness of residuals and Jacobians, DFLM with spherical smoothing achieves the following [2407.12542]:
- With probability at least $1-\exp(-c\,k)$,
\[
\min_{0 \leq \ell < k} \| \nabla f(x_\ell) \| \leq \left( \frac{2\beta}{c\,\kappa_\theta^2\,k} \right)^{1/2},
\]
implying that $O(\epsilon^{-2})$ iterations suffice (up to logarithmic terms) to drive the gradient below $\epsilon$ with high probability.

A similar $O(\epsilon^{-2})$ complexity holds for the regularized, nonsmooth, or constrained DFLM variants [2407.14915, 2507.05623].

### Almost Sure Global Convergence

For sparse DFLM, if the interpolation matrix $A(v_k)$ is RIP with probability $\geq \beta$ (beta large enough), the stationary condition
\[
\lim_{k \to \infty} \| J(x_k)^\top F(x_k) \| = 0
\]
holds almost surely [2507.06772]. For equality constrained DFLM, the algorithm either achieves approximate KKT contraction with arbitrary high probability or the constraint violation converges to zero almost surely [2507.05623].

## 5. Variants and Extensions: Constrained, Regularized, and Sparse DFLM

### Regularized/Composite DFLM

For objectives of the form $f(x) + r(x)$ where $r$ is convex nonsmooth, DFLM uses either model-based trust region steps or a Moreau envelope smoothing approach. Stationarity is measured by $S(x) = \text{dist}(0, \nabla f(x) + \partial r(x))$ and trust-region subproblems include the regularizer [2407.14915].

### DFLM for Sparse Jacobians

Exploiting sparsity, $\ell_1$-based interpolation reconstructs a sparse Jacobian using $O(s \log(n/s))$ function evaluations, yielding strong empirical and theoretical performance on large-scale problems with structured sparsity [2507.06772].

### Equality-Constrained DFLM

For $\min f(x)$ s.t. $c(x)=0$, DFLM is integrated in a regularized augmented Lagrangian framework. Smoothing-based Jacobian models are used for both $r$ and $c$, with the DFLM step applied when Newton-like steps do not yield sufficient KKT residual decrease [2507.05623].

## 6. Practical Implementation and Empirical Performance

DFLM algorithms are implemented efficiently by:
- Sampling orthonormal direction sets via QR factorization of random Gaussian matrices
- Reusing direction sets for several iterations when function evaluations are expensive
- Automatic radius and sample/adaptation strategies as stationarity is approached
- Ensuring stability by safeguarding regularization parameters and maintaining bounded curvature conditions

Empirical profiles show that DFLM methods, including sparsity-exploiting and regularized variants, outperform BFGS, classical LM, and finite-difference-based solvers in the number of function evaluations and robustness, especially on large-scale and noisy or structured least-squares test sets [2407.12542, 2407.14915, 2507.06772].

## 7. Theoretical and Practical Considerations

Key theoretical principles established for DFLM methods include:
- Probabilistically first-order accurate stochastic models, controllable via smoothing radius and sampling/embedding dimension
- Global high-probability complexity bounds matching gradient-based methods under mild smoothness and boundedness assumptions
- Robustness to model inaccuracy via adaptive regularization and step-rejection/acceptance policies
- Almost sure global convergence in the compressed sensing/sparse setting when interpolation matrices satisfy appropriate RIP with sufficient probability

The overall DFLM approach unifies several strands of derivative-free optimization, offering scalable, theoretically justified, and practically effective methodologies for high-dimensional, black-box, regularized, constrained, and structured nonlinear least-squares problems [2407.12542, 2407.14915, 2507.05623, 2507.06772].

Source: https://www.emergentmind.com/topics/derivative-free-lm-dflm