---
title: Residual-Weighted Optimization Strategy
url: https://www.emergentmind.com/topics/residual-weighted-optimization-strategy
type: topic
---

# Residual-Weighted Optimization Strategy

Searching arXiv for the cited papers and closely related residual-weighted methods.
Searching for "Residual Expansion Algorithm" and related residual-weighted methods on arXiv.
I’m going to look up the core papers by title to ground the article in current arXiv records.
Searching arXiv by title.
Residual-weighted optimization strategy denotes, in a broad cross-disciplinary sense, a class of methods that use residual information—the discrepancy between observations and model predictions, between consecutive feedback values, between Bellman backups and current value estimates, or between observed outcomes and treatment-free baselines—to modify an objective, a sampling law, a search direction, or an update magnitude. Taken together, the literature suggests a common purpose: residuals are not treated merely as diagnostics, but as active signals that reshape optimization geometry, suppress unreliable constraints, reduce estimator variance, or enlarge the attraction region of desirable solutions. Representative instances include residual expansion for nonconvex least squares, weighted residual branches in very deep networks, one-point online optimization with residual feedback, residual-weighted learning for individualized treatment rules, residual-based adaptivity in neural PDE solvers and operator learning, residual-weighted randomized linear solvers, and weighted Bellman-residual minimization [1705.09549] [1605.08831] [2010.07378] [1508.03179] [2509.14198] [2606.01232] [2604.06837].

## 1. Domain and conceptual scope

The literature does not present a single canonical algorithm under this label. Rather, it presents several mathematically distinct constructions in which a residual determines either how strongly a term contributes to the loss, which coordinates or samples are emphasized, or how far an iterate is allowed to move. This suggests that “residual-weighted optimization strategy” is best understood as an organizing principle rather than a unique method.

| Setting | Residual-weighted object | Representative paper |
|---|---|---|
| Nonconvex least squares | Expanded data \( \hat y^{(t)} = y + \alpha^{(t)} r^{(t)} \) | [1705.09549] |
| Very deep residual networks | Scalar residual-branch weight \( \lambda_i \in (-1,1) \) | [1605.08831] |
| One-point online ZO | Consecutive feedback residual in gradient estimator | [2010.07378] |
| LiDAR scan registration | Diagonal matrix \(W=\mathrm{diag}(w_i)\) on ICP residuals | [2606.03905] |
| Individualized treatment rules | Residual weight \( w=(Y-g(X))/\pi(A|X) \) | [1508.03179], [2310.00864] |
| Randomized linear solvers | Sampling probabilities proportional to residual magnitudes | [2606.01232], [2106.13606] |
| Neural PDE / Bellman residual minimization | Sampling distribution or norm induced by residual transforms | [2509.14198], [2604.06837] |

Across these settings, the residual may be Euclidean, probabilistic, geometric, causal, or operator-theoretic. In nonconvex least squares it is the observation-model discrepancy \(y-f(\theta)\); in scan registration it is the point-to-plane error \(n_i^\top(Tp_i^t-q_i)\); in treatment learning it is the outcome residual after removing the treatment-free effect; in randomized stationary methods it is the coordinate residual \(r_j=b_j-(Ax)_j\); and in reinforcement learning it is the Bellman residual or a state-action residual policy perturbation [1705.09549] [2606.03905] [1508.03179] [2606.01232] [2002.05522].

## 2. Mathematical formulations

A first major formulation uses residuals to deform the objective directly. For a generic nonconvex least-squares problem,
\[
\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,
\]
the residual expansion method defines \(r^{(t)}=y-f(\theta^{(t)})\), expands the data to \(\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}\), and solves the surrogate subproblem
\[
E_t(\theta)=\frac12\|\hat y^{(t)}-f(\theta)\|_2^2.
\]
The intended effect is to “push” the data along the current residual direction so that shallow local minima disappear or become less attractive [1705.09549].

A second formulation assigns explicit per-residual weights inside an otherwise standard quadratic or \(L_p\)-type objective. In semantic-weighted ICP, the unweighted point-to-plane cost \( \frac12 r^\top I r \) is replaced by
\[
E_{\rm sem}(T)=\frac12\,r^\top W r
=\frac12\sum_{i\in\mathcal C_t} w_i \bigl[n_i^\top(Tp_i^t-q_i)\bigr]^2,
\]
with \(w_i=\hat w_i \times \lambda(\mathcal S_{P(i)})\), where \(\hat w_i\) combines surfel stability, a Huber robustifier, and semantic compatibility, and \(\lambda(\cdot)\) is a fixed class-aware scalar weight [2606.03905]. In soft Bellman residual minimization, the residual \(\delta_\theta=F_\lambda(\Phi\theta)-\Phi\theta\) is measured in a weighted \(L_p\)-norm through
\[
f_p(\theta)=\frac1p\|\delta_\theta\|_{p,w}^p
=\frac1p\sum_{i=1}^n w_i\,|e_i^\top\delta_\theta|^p,
\]
so the weight vector \(w\) and exponent \(p\) jointly determine the optimization geometry [2604.06837].

A third formulation uses residuals to gate a branch or perturbation term. In weighted residual networks,
\[
x_{i+1}=x_i+\lambda_i\,\Delta L_i(x_i;\theta_i),\qquad \lambda_i\in(-1,1),
\]
where \(\lambda_i\) is a learnable scalar controlling how much of the \(i\)-th block’s residual to add, and the post-addition ReLU on the highway path is removed [1605.08831]. In batch RL, BRPO writes the learned policy as a behavior policy plus a residual,
\[
\pi(a\mid s)=\pi_b(a\mid s)+\delta(a\mid s),
\qquad
\delta(a\mid s)=\lambda(s,a)\bigl(\rho(a\mid s)-\pi_b(a\mid s)\bigr),
\]
so the state-action-dependent factor \(\lambda(s,a)\in[0,1]\) acts as a learned residual weight on policy deviation [2002.05522].

A fourth formulation uses residuals to define a sampling law or a low-variance estimator. In one-point derivative-free online optimization, the residual-feedback oracle is
\[
\tilde g_t(x_t):=\frac{u_t}{\delta}\Bigl[f_t(x_t+\delta u_t)-f_{t-1}(x_{t-1}+\delta u_{t-1})\Bigr],
\]
which reuses the previous function evaluation as a baseline [2010.07378]. In neural PDE solvers and operator learning, the continuous objective
\[
J[w](\theta)=\int_\Omega w(R(u(\cdot;\theta);x))\,dx
\]
induces the adaptive sampling distribution
\[
\pi(x)=\frac{w(R(u;x))}{\int_\Omega w(R(u;y))\,dy},
\]
so residual magnitude directly determines where the optimizer samples or concentrates gradient effort [2509.14198].

## 3. Algorithmic patterns

Despite the heterogeneity of applications, the main algorithmic templates recur. Residual expansion is an outer-inner method: solve or partially solve the original least-squares problem under expanded data, update a residual momentum term
\[
r^{(t+1)}=p\bigl(y-f(\theta^{(t+1)})\bigr)+(1-p)r^{(t)},
\]
form the new expanded data \( \hat y^{(t+1)} = y + \alpha^{(t)} r^{(t+1)} \), and continue while \(\alpha^{(t)}\) decreases to \(0\). The method requires only a routine that solves the original least-squares problem; the extra storage for \(\hat y\) and \(r\) is \(O(m)\), and the extra cost per iteration is \(O(m)\) [1705.09549].

In weighted residual networks, training uses projected stochastic gradient descent because the residual weights are constrained to \((-1,1)\). The filter parameters \(\theta_i\) are updated by momentum-SGD, while each \(\lambda_i\) is updated and then clipped by \(P_{(-1,1)}\). The design choice \(\lambda_i=0\) at initialization makes every block initially the identity map, so very deep networks behave like a shallow one at the start of training [1605.08831].

In residual-feedback online optimization, the algorithm performs projected gradient updates on the smoothed objective,
\[
x_{t+1}=\Pi_X[x_t-\eta\,\tilde g_t(x_t)],
\]
with only one function query per round. The same framework extends to stochastic observations \(F_t(x;\xi_t)\) by replacing the noiseless evaluations with noisy ones [2010.07378].

In semantic-weighted ICP, the residual weights enter the normal equations,
\[
J^\top W J\,\delta x=-J^\top W r,
\]
after correspondences, normals, semantic labels, and stability flags have been assembled. The weights combine a baseline term \(\hat w_i\) with a class-dependent factor \(\lambda(\mathcal S_{P(i)})\), and the pose is updated by \(T\leftarrow \exp(\delta x)\cdot T\) until convergence [2606.03905].

In treatment-rule estimation, both RWL and MLRWL lead to nonconvex empirical risks but admit difference-of-convex decompositions. RWL uses the smoothed ramp loss and a d.c. / CCCP procedure, while MLRWL uses the generalized \(\psi\)-loss,
\[
\psi(Z)=\max_k\{\max(1-Z^{(k)},0)\}-\max_k\{\max(0-Z^{(k)},0)\},
\]
and solves successive convex subproblems, including quadratic programs for linear rules [1508.03179] [2310.00864].

In randomized linear solvers, residuals govern index selection. Residual-weighted randomized Jacobi chooses component \(j\) with probability
\[
p_j^{(i)}=\frac{|r_j^{(i)}|^\ell}{\sum_{k=1}^n |r_k^{(i)}|^\ell},
\]
interpolating between uniform randomized Jacobi as \(\ell\to0\) and Gauss–Southwell as \(\ell\to\infty\). GRKO and MWRKO use related residual-based row-selection strategies for oblique-projection Kaczmarz updates, with GRKO sampling from an active set and MWRKO selecting the largest normalized residual deterministically [2606.01232] [2106.13606].

## 4. Theoretical properties

The theoretical role of residual weighting differs by domain, but a recurrent theme is that residual-weighting changes either the local geometry or the estimator variance in a quantifiable way. In residual expansion, a point \(\theta^*\) is an “\(\alpha\)-RE stationary point” if it remains a local minimizer of the expanded objective with that \(\alpha\), and the supremal \(\alpha\) for which this persists is its RE constant. The one-dimensional quartic example shows that among two local minima, the one with the larger RE constant is guaranteed to be the global minimum; the paper also notes that in general nonconvex problems this ranking may fail, although the RE constant often correlates with basin depth. In the differentiable unconstrained case,
\[
\nabla^2 E_\alpha(\theta^*)=J^\top J+(1+\alpha)S,
\]
so a large \(\alpha\) may destroy positive-semidefiniteness around poor minima and cause them to vanish [1705.09549].

Residual feedback in online ZO optimization is unbiased for the gradient of the Gaussian-smoothed loss:
\[
\mathbb E[\tilde g_t(x_t)] = \nabla f_{\delta,t}(x_t).
\]
Its main advantage is variance reduction under bounded residual variation. The convex regret bounds are
\[
R_T = O\bigl((L_0 + L_0^{-1}V_f^2)\sqrt d\,R\,T^{3/4}\bigr)
\]
for convex Lipschitz losses and
\[
R_T = O\bigl((L_0+L_1+L_0^{-1}V_f^2)(dRT)^{2/3}\bigr)
\]
for convex smooth and Lipschitz losses. The analysis explicitly replaces the global bound \(B\) on \(f_t\) with the residual variation term \(V_f\), and states that the bounded-variation assumption is strictly weaker than a uniform bound on \(f_t\) over \(X\) [2010.07378].

In treatment learning, residual-weighting is tied to decision-theoretic consistency. RWL proves Fisher consistency, universal consistency in a universal-kernel RKHS when \(\lambda_n\to0\) and \(n\lambda_n\to\infty\), and a convergence rate
\[
\mathcal R(\mathrm{sign}(f_n))-\mathcal R^*=O_p\bigl(n^{-\beta/(2\beta+1)}\bigr)
\]
under an approximation-error condition. MLRWL proves Fisher consistency for both the outcome-weighted and residual-weighted forms of the generalized \(\psi\)-risk, an excess-risk bound
\[
\mathcal R(\mathrm{sign}\,f)-\mathcal R^*
\le
\mathcal R_{\psi,g}(f)-\mathcal R^*_{\psi,g},
\]
and estimation consistency in RKHS settings [1508.03179] [2310.00864].

For residual-weighted randomized Jacobi, the central theoretical quantity is the inverse participation ratio
\[
\nu^2(r)=\frac{n\|r\|_4^4}{\|r\|_2^4},
\]
which lies in \([1,n]\) and measures residual concentration. In the \(\ell=2\) case, the expected one-step contraction in the \(A\)-norm is amplified by exactly \(\nu_i^2\) over the uniform-sampling baseline. The same paper extends the analysis to asynchronous power-weighted Jacobi and derives an epoch-based convergence theorem in which the IPR controls both progress and the allowed-delay window [2606.01232].

Weighted Bellman-residual minimization formalizes a different type of geometric alignment. With
\[
\gamma_{p,w}:=\gamma\,n^{1/p}(w_{\max}/w_{\min})^{1/p},
\]
the soft Bellman operator satisfies
\[
\|F_\lambda Q-F_\lambda Q'\|_{p,w}\le \gamma_{p,w}\|Q-Q'\|_{p,w},
\]
and for sufficiently large \(p\), \(\gamma_{p,w}<1\). The quasi-optimality constant
\[
C(p)=\frac{1+\gamma_{p,w}}{1-\gamma_{p,w}}
\]
is strictly decreasing in \(p\) and converges to \(\frac{1+\gamma}{1-\gamma}\) as \(p\to\infty\). The paper interprets this as alignment of the optimization objective with the \(L_\infty\)-contraction geometry of the Bellman operator [2604.06837].

## 5. Empirical behavior across application areas

In nonconvex least squares, residual expansion exhibits strong empirical performance on several tasks. In k-means clustering, with a modest \(T\approx30\), it already outperforms k-means++ in objective error at similar cost, and on real COIL20 data it reduced relative error by \(5\%\)–\(10\%\) over k-means++. In 3D point-set registration, under 50 random trials at \(\phi=5\pi/12\), ICP alone succeeded in \(\sim 4/50\), whereas RE+ICP succeeded in \(\sim 25/50\) with only marginally more iterations; when combined with Go–ICP, it reduced the required bound-tightening time by over \(2\times\). In optimized product quantization on SIFT-1M with \(128\)-dimensional descriptors, \(M=8\), \(k=256\), RE reached the same objective value in \(30\) outer iterations that the baseline needed \(100\)–\(200\). In blind image deblurring, RE lifted PSNR from \(\sim 16\) dB to \(\sim 20\) dB on severely blurred cases, and the RE-ADMM variant often gave an additional \(0.5\)–\(1\) dB gain [1705.09549].

In very deep CNNs, the weighted-residual design improves both convergence and final accuracy on CIFAR-10. The reported no-dropout test accuracies are \(\sim 93.6\%\) for ResNet-110, \(\sim 93.8\%\) for WResNet-292, \(\sim 94.2\%\) for WResNet-604, and \(\sim 94.9\%\) for WResNet-1192. With dropout on residuals, WResNet-1192 reaches \(95.3\%\) in \(64\)k iterations. The learned \(\lambda_i\) values are approximately symmetric in \([-0.5,0.5]\), indicating that residuals can reinforce or attenuate the highway signal [1605.08831].

In non-stationary one-point online optimization, residual feedback was tested on Linear-Quadratic Regulator tasks with slowly drifting dynamics and on distributed resource allocation with time-varying penalty parameters. Across 10 trials, the residual method cut the empirical variance almost to two-point levels, while the conventional one-point method had \(5\)–\(10\times\) higher noise. Its regret was \(30\)–\(50\%\) smaller than the conventional one-point method and tracked the two-point oracle closely [2010.07378].

In LiDAR odometry, semantic residual reweighting improved pose estimation on both SemanticKITTI and RELLIS-3D. On SemanticKITTI, baseline SuMa++ with uniform \(\lambda\equiv1\) achieved average rotation \(0.310^\circ/100\) m and translation \(0.759\%\), whereas the balanced semantic-weighted configuration achieved \(0.286^\circ/100\) m and \(0.695\%\), corresponding to an \(\approx 8.4\%\) reduction in translational error. On RELLIS-3D, baseline SuMa++ yielded rotation \(41.46^\circ/100\) m and translation \(41.04\%\), while the best semantic-weighted configuration yielded \(24.67^\circ/100\) m and \(25.77\%\), an \(\approx 37\%\) reduction in translational drift [2606.03905].

In treatment-rule learning, MLRWL consistently achieved the highest empirical value and accuracy across simulated settings with \(K=2\) and \(K=3\) treatments and sample sizes \(n=400,800,2000\), often improving accuracy by \(10\)–\(40\) percentage points. In a type-2 diabetes EHR application with \(K=4\) single treatments, 16 combinations, and \(n\approx1{,}100\), MLRWL-linear and MLRWL-kernel attained test-set value \(\approx 2.61\) and \(2.65\), versus \(2.53\) for OWL-DL and \(2.35\)–\(2.55\) for other multicategory methods, corresponding to a \(4\)–\(10\%\) improvement in value and smaller standard errors [2310.00864]. Earlier RWL simulations and the EPIC cystic fibrosis trial similarly reported lower variance than OWL, treatment-matching factors \(\approx1\), and better estimated value under residual centering [1508.03179].

In residual-weighted iterative linear solvers, asynchronous \(\ell=2\) residual-weighted Jacobi converged \(2\)–\(4\times\) faster per sweep than uniform or cyclic baselines on the tested SPD systems, and the IPR trajectory changed by less than \(3\%\) as the thread count varied from 1 to 128 [2606.01232]. For Kaczmarz-type oblique-projection methods, GRKO and MWRKO used \(\approx 20\)–\(40\%\) fewer iterations and \(3\)–\(5\times\) less time than their orthogonal counterparts on random matrices, and MWRKO was fastest overall on all six sparse test matrices reported [2106.13606].

## 6. Limitations, distinctions, and recurrent misconceptions

A frequent misconception is that residual-weighting is equivalent to hard rejection of difficult or unreliable data. The scan-registration literature shows the opposite in a concrete form: hard rejection of dynamic classes by setting \(\lambda=0\) under-performed soft suppression, because it may remove too many potentially valid correspondences such as parked vehicles in a static frame [2606.03905]. More generally, residual-weighting often acts as a soft prior rather than a binary filter.

Another misconception is that larger residual emphasis is always beneficial. The nonconvex least-squares theory of residual expansion explicitly notes that the RE-constant ranking that succeeds in the one-dimensional quartic example may fail in general nonconvex problems [1705.09549]. In semantic-weighted ICP, very low class weights on large semantic regions can under-constrain the system and degrade convergence, especially in nearly planar highway or rural settings; the paper states that effectiveness is highly environment-dependent [2606.03905]. In weighted Bellman residual minimization, increasing \(p\) improves contraction alignment, but the implementation section warns that very large \(p\) may produce numeric overflow and motivates normalized gradients [2604.06837].

A separate source of confusion concerns the role of initialization and temporal smoothing. In weighted residual networks, the benefit does not come from large initial residual contributions; it comes from setting all \(\lambda_i\) to zero so that the network is initially an exact identity, then allowing projected-SGD with small \(\eta_\lambda\) to bring residual branches online gradually [1605.08831]. In neural PDE training, residual-based adaptivity had been widely used but remained largely heuristic; the variational framework interprets different residual transforms as selecting different primal objectives and sampling distributions, so linear and exponential transforms target different norms rather than merely different heuristics [2509.14198].

The asynchronous linear-solver literature also reports a notable controversy. The same IPR quantity that amplifies progress in power-weighted Jacobi also amplifies a thread-collision rate. Unexpectedly, consistent-reads execution destabilized power-weighted sampling at high concurrency, while inconsistent reads remained stable in nearly all trials. The paper proposes a feedback-damping mechanism that scales atomic commits to avoid overshoot on heavily contended coordinates [2606.01232].

Taken together, these distinctions suggest that residual-weighted optimization is neither a single recipe nor a uniform bias toward high-error regions. It is a structured way of coupling residual information to optimization so that the search process becomes class-aware, variance-aware, geometry-aware, or confidence-aware, depending on the problem class and the particular mathematical role assigned to the residual.

Source: https://www.emergentmind.com/topics/residual-weighted-optimization-strategy