Truncated Spectral Descent (TSD)

• Truncated Spectral Descent (TSD) is a family of optimization methods that trim spectral details—via Chebyshev, eigen-directions, or singular directions—to reduce computational cost and maintain convergence.
• The approaches include unbiased stochastic gradient estimation in spectral-sum setups, rate interpolation in SPD quadratic problems, and low-rank updates in non-smooth convex matrices.
• TSD techniques strategically replace full spectral computations with truncated approximations, offering a balance between efficiency and rigorous convergence guarantees.

Searching arXiv for papers on Truncated Spectral Descent and closely related formulations. Truncated Spectral Descent (TSD) is a label used in arXiv literature for several optimization procedures that truncate spectral information in order to reduce iteration cost while retaining explicit convergence guarantees. In Han et al.’s treatment of spectral-sum optimization, TSD arises from randomized truncation of Chebyshev expansions combined with Monte-Carlo trace estimation, yielding unbiased stochastic gradients for objectives of the form $\nabla_\theta \operatorname{tr}(f(A(\theta)))$ (Han et al., 2018). In a quadratic optimization setting, a truncated spectral-coordinate variant augments randomized coordinate descent by a finite set of eigen-directions associated with the smallest eigenvalues of an SPD matrix (Kovalev et al., 2018). In a non-smooth convex matrix setting, TSD denotes descent along only the top-$s$ singular directions of a subgradient matrix, with global linear convergence under convexity, Lipschitz continuity, and sharpness (Yang et al., 26 May 2026). Taken together, these works suggest a family of methods built around the same structural idea: replace a full spectral object by a truncated one that is cheaper to compute.

1. Terminological scope and canonical variants

Across the cited works, “Truncated Spectral Descent” does not denote a single standardized algorithm. The term is attached to distinct update mechanisms, each defined by what is being truncated and by the ambient optimization model.

Variant	Optimization setting	Truncated object
Chebyshev TSD	Spectral-sum objectives $\Sigma_f(A)=\operatorname{tr}(f(A))$	Polynomial degree $D$ of a Chebyshev expansion
Spectral-coordinate TSD	SPD quadratic minimization	$k$ spectral directions $\{u_1,\dots,u_k\}$
Matrix-sign TSD	Non-smooth convex matrix optimization	Top-$s$ singular directions of a subgradient

The three formulations are linked by spectral compression rather than by identical algebra. In (Han et al., 2018), truncation occurs in polynomial approximation, but stochastic reweighting removes the bias of a fixed truncation. In (Kovalev et al., 2018), truncation means using only a subset of eigenvectors, so the method interpolates between randomized coordinate descent and a fully spectral method. In (Yang et al., 26 May 2026), truncation means replacing the full matrix-sign $msgn(G)$ by the rank-$s$ operator $Tmsgn_s(G)$, which changes both the descent geometry and the required SVD computation.

This distinction matters because convergence statements, computational costs, and admissible problem classes differ substantially across the three settings. A recurrent source of confusion is to treat TSD as if it were always either a stochastic spectral-gradient method or always a low-rank singular-vector method; the literature here shows that both interpretations occur, but in different models.

2. Chebyshev-based TSD for spectral-sum optimization

Han et al. consider spectral-sum objectives for symmetric $s$0 with spectrum in $s$1, where

$s$2

After the affine rescaling

$s$3

the spectrum of $s$4 lies in $s$5, and an analytic $s$6 admits a uniformly convergent Chebyshev expansion

$s$7

The trace terms are estimated with Hutchinson’s method,

$s$8

where $s$9 has independent Rademacher entries. A fixed degree-$\Sigma_f(A)=\operatorname{tr}(f(A))$0 truncation combined with $\Sigma_f(A)=\operatorname{tr}(f(A))$1 probe vectors gives a natural estimator, but that estimator is biased unless $\Sigma_f(A)=\operatorname{tr}(f(A))$2 is a polynomial of degree $\Sigma_f(A)=\operatorname{tr}(f(A))$3 (Han et al., 2018).

The defining step of this TSD construction is stochastic truncation. Instead of truncating at a deterministic degree, the method samples $\Sigma_f(A)=\operatorname{tr}(f(A))$4 from a distribution $\Sigma_f(A)=\operatorname{tr}(f(A))$5 and uses the reweighted random polynomial

$\Sigma_f(A)=\operatorname{tr}(f(A))$6

This is designed so that $\Sigma_f(A)=\operatorname{tr}(f(A))$7, provided $\Sigma_f(A)=\operatorname{tr}(f(A))$8 for all $\Sigma_f(A)=\operatorname{tr}(f(A))$9. Consequently,

$D$0

satisfies $D$1. If $D$2 depends smoothly on parameters, then each coordinate estimator

$D$3

is unbiased for $D$4. The vectors $D$5 and their derivatives can be built in $D$6 extra matrix-vector multiplies per degree by using the chain rule together with the Chebyshev three-term recurrence (Han et al., 2018).

The method’s remaining design freedom is the degree law $D$7. Under analytic decay $D$8, TSD derives a variance-optimal truncation distribution under a fixed expected degree $D$9, with a closed-form solution $k$0 and exponentially small variance for large $k$1. The resulting estimator plugs into projected SGD and into an SVRG-style semi-stochastic loop. Under $k$2-strong convexity and smoothness assumptions on $k$3, SGD with $k$4 yields

$k$5

with estimator variance

$k$6

while the semi-stochastic construction attains linear convergence (Han et al., 2018).

3. Truncated spectral-coordinate descent for SPD quadratics

A different use of the term appears in the quadratic setting

$k$7

with $k$8 symmetric positive-definite and eigen-decomposition $k$9, $\{u_1,\dots,u_k\}$0. Here TSD is the variant called SSCD in the source paper: it augments the standard coordinate directions $\{u_1,\dots,u_k\}$1 by the $\{u_1,\dots,u_k\}$2 eigenvectors corresponding to the $\{u_1,\dots,u_k\}$3 smallest eigenvalues, $\{u_1,\dots,u_k\}$4 (Kovalev et al., 2018).

At each iteration, the method samples a direction from

$\{u_1,\dots,u_k\}$5

according to a distribution $\{u_1,\dots,u_k\}$6, and then performs exact one-dimensional minimization,

$\{u_1,\dots,u_k\}$7

The sampling law is constructed from

$\{u_1,\dots,u_k\}$8

and

$\{u_1,\dots,u_k\}$9

with probabilities

$s$0

The principal convergence theorem states that

$s$1

and equivalently

$s$2

Because $s$3 is monotonically decreasing in $s$4, the rate improves as more spectral directions are added. The endpoints are explicit: $s$5 recovers randomized coordinate descent with diagonal sampling, for which $s$6, while $s$7 matches the pure spectral method SSD with $s$8, independent of $s$9 (Kovalev et al., 2018).

This formulation makes truncation a mechanism for rate interpolation. The method retains the cheap directional structure of coordinate descent while inserting a small number of “hard” spectral directions associated with the smallest eigenvalues. For large sparse $msgn(G)$0, the paper notes that the $msgn(G)$1 eigenvectors may be obtained with Lanczos or randomized SVD at cost roughly $msgn(G)$2, and it also analyzes inexact SSCD, in which approximate eigenvectors $msgn(G)$3 are permitted, as well as a mini-batch extension mSSCD. Reported experiments on synthetic problems of size $msgn(G)$4 and up to $msgn(G)$5 show a sharp phase-transition once $msgn(G)$6 exceeds the number of “bad” small eigenvalues, factor $msgn(G)$7 improvements under exponentially decaying spectra, and order-of-magnitude iteration-count reductions versus RCD in sparse tests with $msgn(G)$8 and $msgn(G)$9 (Kovalev et al., 2018).

4. Top-$s$0 singular-direction TSD in non-smooth convex optimization

In the non-smooth convex matrix setting, TSD is defined for

$s$1

where $s$2 is proper, lower-semicontinuous, convex, and possibly non-smooth. If $s$3 has compact SVD $s$4, the full matrix-sign operator is

$s$5

TSD truncates this object to the first $s$6 singular directions. Writing

$s$7

and letting $s$8 and $s$9 collect the first $Tmsgn_s(G)$0 singular vectors, the truncated operator is

$Tmsgn_s(G)$1

which is an element of the subdifferential of the Ky-Fan $Tmsgn_s(G)$2-norm $Tmsgn_s(G)$3. The update rule is

$Tmsgn_s(G)$4

This is the TSD algorithm analyzed as a truncated counterpart of Spectral Descent (SD) in the context of Muon-type methods (Yang et al., 26 May 2026).

The analysis assumes convexity, Lipschitz continuity, and sharpness. Specifically, $Tmsgn_s(G)$5 is convex; there exists $Tmsgn_s(G)$6 such that $Tmsgn_s(G)$7; and $Tmsgn_s(G)$8 is $Tmsgn_s(G)$9-sharp in the sense that

$s$00

Defining $s$01, the subgradient ranks $s$02, $s$03, and

$s$04

the paper assumes $s$05, which implies $s$06. With a geometric step-size,

$s$07

the iterates satisfy the rank-$s$08 descent recursion

$s$09

and hence the global linear rate

$s$10

The corresponding iteration complexity to reach $s$11 is $s$12 (Yang et al., 26 May 2026).

The proof centers on a uniform lower bound for the truncated descent term $s$13, where $s$14 is the residual to the solution set. The source identifies the core new technical lemma as a geometric bound that combines sharpness, the Lipschitz subgradient bound $s$15, and the fact that only the top-$s$16 directions are retained. Relative to full SD, whose worst-case contraction constant depends on the instantaneous rank $s$17, TSD replaces that dependence by the fixed truncation level $s$18. The paper states that choosing $s$19 relaxes the condition the most, at the cost of computing a truncated SVD of rank $s$20 instead of a full SVD. The corresponding computational trade-off is explicit: full SVD $s$21 versus top-$s$22 SVD $s$23 or power-method $s$24 (Yang et al., 26 May 2026).

5. Regularization, weight decay, and robust low-rank recovery

The non-smooth framework also introduces regularized TSD with decoupled weight decay, denoted RTSD-WD. With $s$25, the momentum-free MuonW update becomes

$s$26

where $s$27 is a “spatially smoothed” subgradient from the $s$28-neighborhood of $s$29. The paper shows that this is exactly a Frank-Wolfe (Conditional Subgradient) update over the spectrally constrained set

$s$30

Under the same convexity, Lipschitz, and sharpness assumptions, but no longer requiring $s$31 large, the distance to the solution set decays at rate $s$32 (Yang et al., 26 May 2026).

The same paper applies RTSD-WD to robust low-rank matrix recovery. The measurement model is

$s$33

where $s$34 is low-rank, $s$35 is sparse, and $s$36 is dense but small. The optimization problem is the LAD formulation

$s$37

Under standard $s$38-RIP on $s$39 and its restriction to outlier indices, the paper proves that $s$40 is Lipschitz with respect to the nuclear norm with constant $s$41 and establishes a restricted sharpness inequality

$s$42

Applying RTSD-WD with $s$43 and $s$44 then yields, with high probability,

$s$45

while requiring only the top-1 singular vector per iterate, for an $s$46 per-step cost (Yang et al., 26 May 2026).

This application highlights a specific advantage of truncation in the non-smooth setting: the update is aligned with the dominant singular direction rather than with the full subgradient spectrum. In regimes where a top-1 or low-rank SVD is much cheaper than a full factorization, the resulting reduction in per-iteration cost is central to the method’s usefulness.

6. Comparative interpretation and recurring misconceptions

The three TSD constructions differ first in the mathematical object that is truncated. In (Han et al., 2018), truncation is over Chebyshev degree, and the estimator is made unbiased by randomization and reweighting. In (Kovalev et al., 2018), truncation is over eigen-directions, and the method samples from a mixed set of coordinate and spectral directions. In (Yang et al., 26 May 2026), truncation is over singular directions of a subgradient, producing a low-rank matrix-sign step. These are not interchangeable algorithmic templates.

A second misconception is that truncation necessarily introduces uncontrolled approximation error. That is not true in the Chebyshev formulation, where the point of the random degree $s$47 and the weights $s$48 is precisely to remove truncation bias and obtain an unbiased estimator of both $s$49 and its gradient (Han et al., 2018). In the spectral-coordinate and non-smooth matrix formulations, truncation does alter the update rule, but it does so within a convergence theory that makes the trade-off explicit: $s$50 in the SPD quadratic case, and $s$51, $s$52, and $s$53 in the non-smooth sharp setting (Kovalev et al., 2018).

A third misconception is to equate “spectral” with “full eigendecomposition” or “full SVD.” All three papers move in the opposite direction. The stochastic Chebyshev method relies on matrix-vector products and randomized trace probing instead of full spectral differentiation. SSCD uses only the $s$54 smallest eigenvectors rather than the full eigenbasis. The non-smooth TSD method replaces full $s$55 by $s$56, with stated complexity reductions from full SVD $s$57 to top-$s$58 SVD $s$59 or power-method $s$60 (Yang et al., 26 May 2026).

The broader significance of the term therefore lies less in a single algorithm than in a common design pattern. These works collectively show that truncating spectral structure can support unbiased stochastic gradients, interpolation between coordinate and spectral descent rates, or low-rank singular-direction updates for non-smooth sharp problems. For arXiv readers, the essential interpretive step is to identify which spectral object is being truncated—Chebyshev degree, eigen-directions, or singular directions—before transferring convergence intuitions from one TSD formulation to another.