Residual-Weighted Optimization Strategy

• Residual-Weighted Optimization Strategy is a broad organizing principle using residual signals to modify loss formulations, sampling distributions, and update magnitudes.
• It employs techniques such as residual expansion, weighted branches in deep networks, and adaptive feedback to refine optimization geometry and reduce variance.
• Empirical studies across tasks like nonconvex least squares, reinforcement learning, and LiDAR registration demonstrate improved convergence and robust performance.

Searching arXiv for the cited papers and closely related residual-weighted methods. Searching for "Residual Expansion Algorithm" and related residual-weighted methods on arXiv. I’m going to look up the core papers by title to ground the article in current arXiv records. Searching arXiv by title. Residual-weighted optimization strategy denotes, in a broad cross-disciplinary sense, a class of methods that use residual information—the discrepancy between observations and model predictions, between consecutive feedback values, between Bellman backups and current value estimates, or between observed outcomes and treatment-free baselines—to modify an objective, a sampling law, a search direction, or an update magnitude. Taken together, the literature suggests a common purpose: residuals are not treated merely as diagnostics, but as active signals that reshape optimization geometry, suppress unreliable constraints, reduce estimator variance, or enlarge the attraction region of desirable solutions. Representative instances include residual expansion for nonconvex least squares, weighted residual branches in very deep networks, one-point online optimization with residual feedback, residual-weighted learning for individualized treatment rules, residual-based adaptivity in neural PDE solvers and operator learning, residual-weighted randomized linear solvers, and weighted Bellman-residual minimization (Ikami et al., 2017, Shen et al., 2016, Zhang et al., 2020, Zhou et al., 2015, Toscano et al., 17 Sep 2025, Coleman, 31 May 2026, Yang et al., 8 Apr 2026).

1. Domain and conceptual scope

The literature does not present a single canonical algorithm under this label. Rather, it presents several mathematically distinct constructions in which a residual determines either how strongly a term contributes to the loss, which coordinates or samples are emphasized, or how far an iterate is allowed to move. This suggests that “residual-weighted optimization strategy” is best understood as an organizing principle rather than a unique method.

Setting	Residual-weighted object	Representative paper
Nonconvex least squares	Expanded data $\hat y^{(t)} = y + \alpha^{(t)} r^{(t)}$	(Ikami et al., 2017)
Very deep residual networks	Scalar residual-branch weight $\lambda_i \in (-1,1)$	(Shen et al., 2016)
One-point online ZO	Consecutive feedback residual in gradient estimator	(Zhang et al., 2020)
LiDAR scan registration	Diagonal matrix $W=\mathrm{diag}(w_i)$ on ICP residuals	(Carvalho et al., 2 Jun 2026)
Individualized treatment rules	Residual weight $w=(Y-g(X))/\pi(A|X)$	(Zhou et al., 2015, Xu et al., 2023)
Randomized linear solvers	Sampling probabilities proportional to residual magnitudes	(Coleman, 31 May 2026, Wang et al., 2021)
Neural PDE / Bellman residual minimization	Sampling distribution or norm induced by residual transforms	(Toscano et al., 17 Sep 2025, Yang et al., 8 Apr 2026)

Across these settings, the residual may be Euclidean, probabilistic, geometric, causal, or operator-theoretic. In nonconvex least squares it is the observation-model discrepancy $y-f(\theta)$; in scan registration it is the point-to-plane error $n_i^\top(Tp_i^t-q_i)$; in treatment learning it is the outcome residual after removing the treatment-free effect; in randomized stationary methods it is the coordinate residual $r_j=b_j-(Ax)_j$; and in reinforcement learning it is the Bellman residual or a state-action residual policy perturbation (Ikami et al., 2017, Carvalho et al., 2 Jun 2026, Zhou et al., 2015, Coleman, 31 May 2026, Sohn et al., 2020).

2. Mathematical formulations

A first major formulation uses residuals to deform the objective directly. For a generic nonconvex least-squares problem,

$$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$

the residual expansion method defines $r^{(t)}=y-f(\theta^{(t)})$, expands the data to $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$, and solves the surrogate subproblem

$\lambda_i \in (-1,1)$0

The intended effect is to “push” the data along the current residual direction so that shallow local minima disappear or become less attractive (Ikami et al., 2017).

A second formulation assigns explicit per-residual weights inside an otherwise standard quadratic or $\lambda_i \in (-1,1)$1-type objective. In semantic-weighted ICP, the unweighted point-to-plane cost $\lambda_i \in (-1,1)$2 is replaced by

$\lambda_i \in (-1,1)$3

with $\lambda_i \in (-1,1)$4, where $\lambda_i \in (-1,1)$5 combines surfel stability, a Huber robustifier, and semantic compatibility, and $\lambda_i \in (-1,1)$6 is a fixed class-aware scalar weight (Carvalho et al., 2 Jun 2026). In soft Bellman residual minimization, the residual $\lambda_i \in (-1,1)$7 is measured in a weighted $\lambda_i \in (-1,1)$8-norm through

$\lambda_i \in (-1,1)$9

so the weight vector $W=\mathrm{diag}(w_i)$0 and exponent $W=\mathrm{diag}(w_i)$1 jointly determine the optimization geometry (Yang et al., 8 Apr 2026).

A third formulation uses residuals to gate a branch or perturbation term. In weighted residual networks,

$W=\mathrm{diag}(w_i)$2

where $W=\mathrm{diag}(w_i)$3 is a learnable scalar controlling how much of the $W=\mathrm{diag}(w_i)$4-th block’s residual to add, and the post-addition ReLU on the highway path is removed (Shen et al., 2016). In batch RL, BRPO writes the learned policy as a behavior policy plus a residual,

$W=\mathrm{diag}(w_i)$5

so the state-action-dependent factor $W=\mathrm{diag}(w_i)$6 acts as a learned residual weight on policy deviation (Sohn et al., 2020).

A fourth formulation uses residuals to define a sampling law or a low-variance estimator. In one-point derivative-free online optimization, the residual-feedback oracle is

$W=\mathrm{diag}(w_i)$7

which reuses the previous function evaluation as a baseline (Zhang et al., 2020). In neural PDE solvers and operator learning, the continuous objective

$W=\mathrm{diag}(w_i)$8

induces the adaptive sampling distribution

$W=\mathrm{diag}(w_i)$9

so residual magnitude directly determines where the optimizer samples or concentrates gradient effort (Toscano et al., 17 Sep 2025).

3. Algorithmic patterns

Despite the heterogeneity of applications, the main algorithmic templates recur. Residual expansion is an outer-inner method: solve or partially solve the original least-squares problem under expanded data, update a residual momentum term

$w=(Y-g(X))/\pi(A|X)$0

form the new expanded data $w=(Y-g(X))/\pi(A|X)$1, and continue while $w=(Y-g(X))/\pi(A|X)$2 decreases to $w=(Y-g(X))/\pi(A|X)$3. The method requires only a routine that solves the original least-squares problem; the extra storage for $w=(Y-g(X))/\pi(A|X)$4 and $w=(Y-g(X))/\pi(A|X)$5 is $w=(Y-g(X))/\pi(A|X)$6, and the extra cost per iteration is $w=(Y-g(X))/\pi(A|X)$7 (Ikami et al., 2017).

In weighted residual networks, training uses projected stochastic gradient descent because the residual weights are constrained to $w=(Y-g(X))/\pi(A|X)$8. The filter parameters $w=(Y-g(X))/\pi(A|X)$9 are updated by momentum-SGD, while each $y-f(\theta)$0 is updated and then clipped by $y-f(\theta)$1. The design choice $y-f(\theta)$2 at initialization makes every block initially the identity map, so very deep networks behave like a shallow one at the start of training (Shen et al., 2016).

In residual-feedback online optimization, the algorithm performs projected gradient updates on the smoothed objective,

$y-f(\theta)$3

with only one function query per round. The same framework extends to stochastic observations $y-f(\theta)$4 by replacing the noiseless evaluations with noisy ones (Zhang et al., 2020).

In semantic-weighted ICP, the residual weights enter the normal equations,

$y-f(\theta)$5

after correspondences, normals, semantic labels, and stability flags have been assembled. The weights combine a baseline term $y-f(\theta)$6 with a class-dependent factor $y-f(\theta)$7, and the pose is updated by $y-f(\theta)$8 until convergence (Carvalho et al., 2 Jun 2026).

In treatment-rule estimation, both RWL and MLRWL lead to nonconvex empirical risks but admit difference-of-convex decompositions. RWL uses the smoothed ramp loss and a d.c. / CCCP procedure, while MLRWL uses the generalized $y-f(\theta)$9-loss,

$n_i^\top(Tp_i^t-q_i)$0

and solves successive convex subproblems, including quadratic programs for linear rules (Zhou et al., 2015, Xu et al., 2023).

In randomized linear solvers, residuals govern index selection. Residual-weighted randomized Jacobi chooses component $n_i^\top(Tp_i^t-q_i)$1 with probability

$n_i^\top(Tp_i^t-q_i)$2

interpolating between uniform randomized Jacobi as $n_i^\top(Tp_i^t-q_i)$3 and Gauss–Southwell as $n_i^\top(Tp_i^t-q_i)$4. GRKO and MWRKO use related residual-based row-selection strategies for oblique-projection Kaczmarz updates, with GRKO sampling from an active set and MWRKO selecting the largest normalized residual deterministically (Coleman, 31 May 2026, Wang et al., 2021).

4. Theoretical properties

The theoretical role of residual weighting differs by domain, but a recurrent theme is that residual-weighting changes either the local geometry or the estimator variance in a quantifiable way. In residual expansion, a point $n_i^\top(Tp_i^t-q_i)$5 is an “$n_i^\top(Tp_i^t-q_i)$6-RE stationary point” if it remains a local minimizer of the expanded objective with that $n_i^\top(Tp_i^t-q_i)$7, and the supremal $n_i^\top(Tp_i^t-q_i)$8 for which this persists is its RE constant. The one-dimensional quartic example shows that among two local minima, the one with the larger RE constant is guaranteed to be the global minimum; the paper also notes that in general nonconvex problems this ranking may fail, although the RE constant often correlates with basin depth. In the differentiable unconstrained case,

$n_i^\top(Tp_i^t-q_i)$9

so a large $r_j=b_j-(Ax)_j$0 may destroy positive-semidefiniteness around poor minima and cause them to vanish (Ikami et al., 2017).

Residual feedback in online ZO optimization is unbiased for the gradient of the Gaussian-smoothed loss: $r_j=b_j-(Ax)_j$1 Its main advantage is variance reduction under bounded residual variation. The convex regret bounds are

$r_j=b_j-(Ax)_j$2

for convex Lipschitz losses and

$r_j=b_j-(Ax)_j$3

for convex smooth and Lipschitz losses. The analysis explicitly replaces the global bound $r_j=b_j-(Ax)_j$4 on $r_j=b_j-(Ax)_j$5 with the residual variation term $r_j=b_j-(Ax)_j$6, and states that the bounded-variation assumption is strictly weaker than a uniform bound on $r_j=b_j-(Ax)_j$7 over $r_j=b_j-(Ax)_j$8 (Zhang et al., 2020).

In treatment learning, residual-weighting is tied to decision-theoretic consistency. RWL proves Fisher consistency, universal consistency in a universal-kernel RKHS when $r_j=b_j-(Ax)_j$9 and $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$0, and a convergence rate

$$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$1

under an approximation-error condition. MLRWL proves Fisher consistency for both the outcome-weighted and residual-weighted forms of the generalized $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$2-risk, an excess-risk bound

$$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$3

and estimation consistency in RKHS settings (Zhou et al., 2015, Xu et al., 2023).

For residual-weighted randomized Jacobi, the central theoretical quantity is the inverse participation ratio

$$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$4

which lies in $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$5 and measures residual concentration. In the $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$6 case, the expected one-step contraction in the $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$7-norm is amplified by exactly $$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$8 over the uniform-sampling baseline. The same paper extends the analysis to asynchronous power-weighted Jacobi and derives an epoch-based convergence theorem in which the IPR controls both progress and the allowed-delay window (Coleman, 31 May 2026).

Weighted Bellman-residual minimization formalizes a different type of geometric alignment. With

$$\min_{\theta\in\Theta}\;E(\theta)=\frac12\|y-f(\theta)\|_2^2,$$9

the soft Bellman operator satisfies

$r^{(t)}=y-f(\theta^{(t)})$0

and for sufficiently large $r^{(t)}=y-f(\theta^{(t)})$1, $r^{(t)}=y-f(\theta^{(t)})$2. The quasi-optimality constant

$r^{(t)}=y-f(\theta^{(t)})$3

is strictly decreasing in $r^{(t)}=y-f(\theta^{(t)})$4 and converges to $r^{(t)}=y-f(\theta^{(t)})$5 as $r^{(t)}=y-f(\theta^{(t)})$6. The paper interprets this as alignment of the optimization objective with the $r^{(t)}=y-f(\theta^{(t)})$7-contraction geometry of the Bellman operator (Yang et al., 8 Apr 2026).

5. Empirical behavior across application areas

In nonconvex least squares, residual expansion exhibits strong empirical performance on several tasks. In k-means clustering, with a modest $r^{(t)}=y-f(\theta^{(t)})$8, it already outperforms k-means++ in objective error at similar cost, and on real COIL20 data it reduced relative error by $r^{(t)}=y-f(\theta^{(t)})$9–$\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$0 over k-means++. In 3D point-set registration, under 50 random trials at $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$1, ICP alone succeeded in $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$2, whereas RE+ICP succeeded in $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$3 with only marginally more iterations; when combined with Go–ICP, it reduced the required bound-tightening time by over $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$4. In optimized product quantization on SIFT-1M with $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$5-dimensional descriptors, $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$6, $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$7, RE reached the same objective value in $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$8 outer iterations that the baseline needed $\hat y^{(t)}=y+\alpha^{(t)}r^{(t)}$9–$\lambda_i \in (-1,1)$00. In blind image deblurring, RE lifted PSNR from $\lambda_i \in (-1,1)$01 dB to $\lambda_i \in (-1,1)$02 dB on severely blurred cases, and the RE-ADMM variant often gave an additional $\lambda_i \in (-1,1)$03–$\lambda_i \in (-1,1)$04 dB gain (Ikami et al., 2017).

In very deep CNNs, the weighted-residual design improves both convergence and final accuracy on CIFAR-10. The reported no-dropout test accuracies are $\lambda_i \in (-1,1)$05 for ResNet-110, $\lambda_i \in (-1,1)$06 for WResNet-292, $\lambda_i \in (-1,1)$07 for WResNet-604, and $\lambda_i \in (-1,1)$08 for WResNet-1192. With dropout on residuals, WResNet-1192 reaches $\lambda_i \in (-1,1)$09 in $\lambda_i \in (-1,1)$10k iterations. The learned $\lambda_i \in (-1,1)$11 values are approximately symmetric in $\lambda_i \in (-1,1)$12, indicating that residuals can reinforce or attenuate the highway signal (Shen et al., 2016).

In non-stationary one-point online optimization, residual feedback was tested on Linear-Quadratic Regulator tasks with slowly drifting dynamics and on distributed resource allocation with time-varying penalty parameters. Across 10 trials, the residual method cut the empirical variance almost to two-point levels, while the conventional one-point method had $\lambda_i \in (-1,1)$13–$\lambda_i \in (-1,1)$14 higher noise. Its regret was $\lambda_i \in (-1,1)$15–$\lambda_i \in (-1,1)$16 smaller than the conventional one-point method and tracked the two-point oracle closely (Zhang et al., 2020).

In LiDAR odometry, semantic residual reweighting improved pose estimation on both SemanticKITTI and RELLIS-3D. On SemanticKITTI, baseline SuMa++ with uniform $\lambda_i \in (-1,1)$17 achieved average rotation $\lambda_i \in (-1,1)$18 m and translation $\lambda_i \in (-1,1)$19, whereas the balanced semantic-weighted configuration achieved $\lambda_i \in (-1,1)$20 m and $\lambda_i \in (-1,1)$21, corresponding to an $\lambda_i \in (-1,1)$22 reduction in translational error. On RELLIS-3D, baseline SuMa++ yielded rotation $\lambda_i \in (-1,1)$23 m and translation $\lambda_i \in (-1,1)$24, while the best semantic-weighted configuration yielded $\lambda_i \in (-1,1)$25 m and $\lambda_i \in (-1,1)$26, an $\lambda_i \in (-1,1)$27 reduction in translational drift (Carvalho et al., 2 Jun 2026).

In treatment-rule learning, MLRWL consistently achieved the highest empirical value and accuracy across simulated settings with $\lambda_i \in (-1,1)$28 and $\lambda_i \in (-1,1)$29 treatments and sample sizes $\lambda_i \in (-1,1)$30, often improving accuracy by $\lambda_i \in (-1,1)$31–$\lambda_i \in (-1,1)$32 percentage points. In a type-2 diabetes EHR application with $\lambda_i \in (-1,1)$33 single treatments, 16 combinations, and $\lambda_i \in (-1,1)$34, MLRWL-linear and MLRWL-kernel attained test-set value $\lambda_i \in (-1,1)$35 and $\lambda_i \in (-1,1)$36, versus $\lambda_i \in (-1,1)$37 for OWL-DL and $\lambda_i \in (-1,1)$38–$\lambda_i \in (-1,1)$39 for other multicategory methods, corresponding to a $\lambda_i \in (-1,1)$40–$\lambda_i \in (-1,1)$41 improvement in value and smaller standard errors (Xu et al., 2023). Earlier RWL simulations and the EPIC cystic fibrosis trial similarly reported lower variance than OWL, treatment-matching factors $\lambda_i \in (-1,1)$42, and better estimated value under residual centering (Zhou et al., 2015).

In residual-weighted iterative linear solvers, asynchronous $\lambda_i \in (-1,1)$43 residual-weighted Jacobi converged $\lambda_i \in (-1,1)$44–$\lambda_i \in (-1,1)$45 faster per sweep than uniform or cyclic baselines on the tested SPD systems, and the IPR trajectory changed by less than $\lambda_i \in (-1,1)$46 as the thread count varied from 1 to 128 (Coleman, 31 May 2026). For Kaczmarz-type oblique-projection methods, GRKO and MWRKO used $\lambda_i \in (-1,1)$47–$\lambda_i \in (-1,1)$48 fewer iterations and $\lambda_i \in (-1,1)$49–$\lambda_i \in (-1,1)$50 less time than their orthogonal counterparts on random matrices, and MWRKO was fastest overall on all six sparse test matrices reported (Wang et al., 2021).

6. Limitations, distinctions, and recurrent misconceptions

A frequent misconception is that residual-weighting is equivalent to hard rejection of difficult or unreliable data. The scan-registration literature shows the opposite in a concrete form: hard rejection of dynamic classes by setting $\lambda_i \in (-1,1)$51 under-performed soft suppression, because it may remove too many potentially valid correspondences such as parked vehicles in a static frame (Carvalho et al., 2 Jun 2026). More generally, residual-weighting often acts as a soft prior rather than a binary filter.

Another misconception is that larger residual emphasis is always beneficial. The nonconvex least-squares theory of residual expansion explicitly notes that the RE-constant ranking that succeeds in the one-dimensional quartic example may fail in general nonconvex problems (Ikami et al., 2017). In semantic-weighted ICP, very low class weights on large semantic regions can under-constrain the system and degrade convergence, especially in nearly planar highway or rural settings; the paper states that effectiveness is highly environment-dependent (Carvalho et al., 2 Jun 2026). In weighted Bellman residual minimization, increasing $\lambda_i \in (-1,1)$52 improves contraction alignment, but the implementation section warns that very large $\lambda_i \in (-1,1)$53 may produce numeric overflow and motivates normalized gradients (Yang et al., 8 Apr 2026).

A separate source of confusion concerns the role of initialization and temporal smoothing. In weighted residual networks, the benefit does not come from large initial residual contributions; it comes from setting all $\lambda_i \in (-1,1)$54 to zero so that the network is initially an exact identity, then allowing projected-SGD with small $\lambda_i \in (-1,1)$55 to bring residual branches online gradually (Shen et al., 2016). In neural PDE training, residual-based adaptivity had been widely used but remained largely heuristic; the variational framework interprets different residual transforms as selecting different primal objectives and sampling distributions, so linear and exponential transforms target different norms rather than merely different heuristics (Toscano et al., 17 Sep 2025).

The asynchronous linear-solver literature also reports a notable controversy. The same IPR quantity that amplifies progress in power-weighted Jacobi also amplifies a thread-collision rate. Unexpectedly, consistent-reads execution destabilized power-weighted sampling at high concurrency, while inconsistent reads remained stable in nearly all trials. The paper proposes a feedback-damping mechanism that scales atomic commits to avoid overshoot on heavily contended coordinates (Coleman, 31 May 2026).

Taken together, these distinctions suggest that residual-weighted optimization is neither a single recipe nor a uniform bias toward high-error regions. It is a structured way of coupling residual information to optimization so that the search process becomes class-aware, variance-aware, geometry-aware, or confidence-aware, depending on the problem class and the particular mathematical role assigned to the residual.