Weighted Levenberg–Marquardt Method

• Weighted Levenberg–Marquardt is a nonlinear least-squares method that explicitly incorporates weighting of residuals and parameters to address measurement uncertainties and data heterogeneity.
• It extends the classical LM algorithm by embedding weights into the data misfit norm, Fisher information metric, and subset updates to balance contributions from diverse data groups.
• Empirical applications, such as nuclear cross-section fitting, demonstrate that weighted updates and geometric scaling significantly enhance convergence speed and physical consistency.

Searching arXiv for recent and foundational papers on weighted Levenberg–Marquardt methods and related variants. I’m going to look up relevant arXiv papers so the article is grounded in the current literature. Weighted Levenberg–Marquardt (wLM) denotes a class of Levenberg–Marquardt algorithms for nonlinear least-squares problems in which the residual geometry, the data misfit, or the update aggregation is explicitly weighted. In the most classical usage, the method minimizes a weighted least-squares functional such as $\tfrac12\|\Gamma^{-1/2}(y-G(u))\|_Y^2$, where $\Gamma$ is a measurement-error covariance. A recent extension for multichannel nuclear cross-section fitting constructs a weighted Fisher Information Metric by integrating prior distributions over dataset weights, thereby producing a parameter-dependent metric and a weighted LM step tailored to heterogeneous reaction channels. Related large-scale implementations also use weighted subset updates when full Jacobians are impractical, as in GPU-based Gaussian-splatting optimization (Imbrišak et al., 26 Aug 2025, Iglesias et al., 2013, Höllein et al., 2024).

1. Classical Levenberg–Marquardt structure

In its standard nonlinear least-squares form, the method considers a model $f(\theta)$ with parameters $\theta\in\mathbb R^{N_p}$ and residuals

$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$

with objective

$C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$

If $J_{i\mu}=\partial_\mu r_i(\theta)$ denotes the Jacobian, the Gauss–Newton approximation replaces the Hessian of $C$ by $J^T J$, and the classical Levenberg–Marquardt step solves

$$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$

or, in index notation,

$\Gamma$0

where

$\Gamma$1

and $\Gamma$2 is the diagonal matrix defined by the diagonal of $\Gamma$3. The damping parameter $\Gamma$4 is adjusted each iteration to ensure descent (Imbrišak et al., 26 Aug 2025).

A weighted formulation arises naturally when the residuals are measured in a covariance-weighted norm. For a parameter-to-observation map $\Gamma$5, noisy data $\Gamma$6, and positive-definite measurement-error covariance operator $\Gamma$7, the weighted data misfit is

$\Gamma$8

Here $\Gamma$9 “whitens” the data by scaling each component according to its measurement variance, so that directions in data space along which the noise is large are down-weighted in the misfit (Iglesias et al., 2013).

2. Distinct meanings of “weighted” in the literature

The literature uses weighting in several technically distinct ways. A common misconception is that weighted LM refers only to inverse-variance scaling of residuals. The cited work shows that weighting can enter through the misfit norm, through a parameter-dependent Fisher metric over grouped datasets, or through the combination of partial LM updates computed on data subsets.

Weighting mechanism	Mathematical object	Role
Measurement-error weighting	$f(\theta)$0 or $f(\theta)$1	Scales the data misfit by measurement covariance
Group-dependent weighting	$f(\theta)$2	Reweights dataset groups in the Fisher metric and cost
Subset-update weighting	$f(\theta)$3	Merges per-subset LM updates by a weighted mean

In the inverse-problem setting, weighting is tied to a statistically motivated least-squares norm. In the multichannel nuclear-data setting, the weighting is hierarchical: each dataset group receives a hyperparameter $f(\theta)$4, a prior is placed on each $f(\theta)$5, and marginalization induces effective group weights $f(\theta)$6. In the GPU-based Gaussian-splatting setting, the weighted step is not defined through a hierarchical likelihood; instead, one solves one damped normal equation per image subset and combines the resulting directions by an element-wise weighted mean built from diagonal curvature surrogates (Imbrišak et al., 26 Aug 2025, Iglesias et al., 2013, Höllein et al., 2024).

This suggests that “weighted Levenberg–Marquardt” is better understood as a structural pattern—LM with nonuniform geometry or nonuniform data influence—than as a single fixed algorithmic formula.

3. Weighted Fisher Information Metric for heterogeneous multichannel data

For multichannel nuclear cross-section fitting, the data are partitioned into $f(\theta)$7 groups, such as different reaction channels or experiments, with group $f(\theta)$8 containing $f(\theta)$9 points and group misfit

$\theta\in\mathbb R^{N_p}$0

A hyperparameter $\theta\in\mathbb R^{N_p}$1 is introduced for each group so that the conditional likelihood becomes

$\theta\in\mathbb R^{N_p}$2

Placing a prior $\theta\in\mathbb R^{N_p}$3 on each $\theta\in\mathbb R^{N_p}$4 and marginalizing over the admissible interval $\theta\in\mathbb R^{N_p}$5 yields a marginalized likelihood $\theta\in\mathbb R^{N_p}$6. The resulting Fisher information metric is

$\theta\in\mathbb R^{N_p}$7

or equivalently

$\theta\in\mathbb R^{N_p}$8

For an exponential prior $\theta\in\mathbb R^{N_p}$9, the group-dependent weight has the closed form

$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$0

A corresponding weighted cost can be written as

$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$1

with gradient

$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$2

and approximate Hessian $r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$3 (Imbrišak et al., 26 Aug 2025).

The motivation for this construction is the uneven distribution of experimental data across reaction channels and the presence of “sloppy” directions in parameter space in the CoH$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$4 code, which is based on the Hauser-Feshbach statistical model and involves a large number of interdependent parameters. The weighted Fisher metric is introduced specifically to obtain a more balanced treatment of heterogeneous data and to improve parameter estimation and convergence robustness. In the $r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$5Sm example, the method is reported to yield more physically consistent fits for both raw and smoothed datasets (Imbrišak et al., 26 Aug 2025).

4. Regularization, priors, and inverse-problem formulations

In nonlinear inverse problems, weighted LM is frequently embedded in a regularization framework. A Gaussian prior with mean $r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$6 and covariance operator $r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$7 leads to the Tikhonov-regularized objective

$r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$8

The associated natural norm in parameter space is $r_i(\theta)=\frac{y_i-f_i(\theta)}{\sigma_i},$9, and the inversion may be viewed as taking place in the Hilbert space $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$0, the completion of $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$1 under $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$2 (Iglesias et al., 2013).

Linearizing at iterate $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$3 with Jacobian $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$4 gives the Gauss–Newton normal equations

$C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$5

Levenberg–Marquardt damping modifies this to

$C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$6

which is equivalent to solving the Tikhonov-regularized linear problem

$C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$7

When $C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$8, one may instead use the data-space form

$C(\theta)=\frac12\sum_{i=1}^N r_i(\theta)^2.$9

A practical implementation employs an adaptive choice of $J_{i\mu}=\partial_\mu r_i(\theta)$0 via a discrepancy principle and stops when the weighted residual is commensurate with the noise level. Under assumptions that $J_{i\mu}=\partial_\mu r_i(\theta)$1 is locally bounded, that a tangential cone condition holds locally, and that the initial guess lies sufficiently close to the true solution, the damped LM scheme with discrepancy-principle choice of $J_{i\mu}=\partial_\mu r_i(\theta)$2 terminates in finitely many steps and converges, as the noise level tends to zero, to a stable minimizer of the noise-free misfit (Iglesias et al., 2013).

5. Weighted updates and geometry-aware step control

For the multichannel weighted Fisher formulation, the weighted LM update is

$J_{i\mu}=\partial_\mu r_i(\theta)$3

where

$J_{i\mu}=\partial_\mu r_i(\theta)$4

A high-level implementation initializes parameters $J_{i\mu}=\partial_\mu r_i(\theta)$5, damping $J_{i\mu}=\partial_\mu r_i(\theta)$6, and previous step $J_{i\mu}=\partial_\mu r_i(\theta)$7; then, at each iteration, computes $J_{i\mu}=\partial_\mu r_i(\theta)$8 and $J_{i\mu}=\partial_\mu r_i(\theta)$9 for each group, evaluates residuals and Jacobians, forms $C$0 and $C$1, computes the gradient $C$2, solves

$C$3

and accepts or rejects the proposed step according to whether $C$4 decreases. Typical choices are $C$5, $C$6, and $C$7 (Imbrišak et al., 26 Aug 2025).

A further refinement is geometric scaling based on the local geometry of the manifold. If $C$8 and $C$9 are successive updates, their alignment under the weighted metric is measured by

$J^T J$0

If the updates are well aligned, the valley is smooth and the step can be stretched; if they de-align, the step is shrunk. Concretely,

$J^T J$1

before the usual accept/reject-and-damping update for $J^T J$2. An equivalent alternative is to diagonalize the metric $J^T J$3 and rescale directions by $J^T J$4, although the angular-alignment rule is described as simpler to implement (Imbrišak et al., 26 Aug 2025).

The explicit use of local manifold geometry is closely tied to the “sloppy” parameter-space structure emphasized in the nuclear-data application. A plausible implication is that the weighted metric and the geometric scaling are complementary: the former redistributes influence across heterogeneous datasets, while the latter modulates step length along narrow valleys of the likelihood manifold.

6. Applications, comparative behavior, and computational realizations

In multichannel nuclear cross-section fitting, the weighted method is presented as a practical and robust alternative to conventional trust-region methods for analyzing experimental data. The stated advantages include balanced use of heterogeneous data, implicit trust regions, robustness in sloppy models, improved physical consistency, and faster convergence. More specifically, by marginalizing over dataset-scale parameters $J^T J$5, the method automatically down-weights over-represented or highly discrepant groups, avoiding domination by the largest-$J^T J$6 channel. In the $J^T J$7Sm cross-section example, classical LM is described as over-fitting the total or capture channel individually and often pushing optical-model parameters to unphysical extremes, whereas the weighted fits lie well within physically motivated ranges, produce balanced z-scores across neutron energies, and yield more reliable covariance estimates via $J^T J$8. The combination of weighting and geometric scaling is reported empirically to cut the number of iterations by $J^T J$950–80 % in high-dimensional CoH$$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$0 fits (Imbrišak et al., 26 Aug 2025).

A distinct computational realization appears in 3D Gaussian Splatting. There, the nonlinear least-squares energy

$$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$1

is formed from residuals derived from an $$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$2 term and an SSIM term, with the weights $$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$3 absorbed directly into the residual definition, so that $$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$4 in the weighted LM normal equations. Because both the number of residuals and the number of parameters are large, the Jacobian is never built explicitly. Instead, the images are split into disjoint batches, one damped system is solved per subset with matrix-free PCG using Jacobian–vector products and transposed Jacobian–vector products, and the per-subset updates are merged by

$$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$5

where $$\bigl(J^T J + \lambda\,D\bigr)\,\delta\theta=-\,J^T r,$$6. The implementation uses a caching data structure for intermediate gradients, custom CUDA kernels, 8 PCG iterations, a line search on a small random subset of images, and an adaptive damping schedule based on a gain ratio. The reported outcome is that the method is 30% faster than the original 3DGS while obtaining the same reconstruction quality (Höllein et al., 2024).

Taken together, these results show that weighted LM methods span a broad range of problem classes: ill-posed inverse problems with measurement covariance, multichannel heterogeneous datasets with hierarchical group weights, and large-scale graphics optimization with weighted aggregation of subsetwise second-order information. The unifying feature is not a single canonical formula, but the insertion of weighting into the residual norm, the metric, the damping, or the update composition so that the local quadratic model better reflects the statistical or computational structure of the problem.