--- title: Truncated Spectral Descent (TSD) url: https://www.emergentmind.com/topics/truncated-spectral-descent-tsd type: topic --- # Truncated Spectral Descent (TSD) Searching arXiv for papers on Truncated Spectral Descent and closely related formulations. Truncated Spectral Descent (TSD) is a label used in arXiv literature for several optimization procedures that truncate spectral information in order to reduce iteration cost while retaining explicit convergence guarantees. In Han et al.’s treatment of spectral-sum optimization, TSD arises from randomized truncation of Chebyshev expansions combined with Monte-Carlo trace estimation, yielding unbiased stochastic gradients for objectives of the form \(\nabla_\theta \operatorname{tr}(f(A(\theta)))\) [1802.06355]. In a quadratic optimization setting, a truncated spectral-coordinate variant augments randomized coordinate descent by a finite set of eigen-directions associated with the smallest eigenvalues of an SPD matrix [1802.03703]. In a non-smooth convex matrix setting, TSD denotes descent along only the top-\(s\) singular directions of a subgradient matrix, with global linear convergence under convexity, Lipschitz continuity, and sharpness [2605.26977]. Taken together, these works suggest a family of methods built around the same structural idea: replace a full spectral object by a truncated one that is cheaper to compute. ## 1. Terminological scope and canonical variants Across the cited works, “Truncated Spectral Descent” does not denote a single standardized algorithm. The term is attached to distinct update mechanisms, each defined by what is being truncated and by the ambient optimization model. | Variant | Optimization setting | Truncated object | |---|---|---| | Chebyshev TSD | Spectral-sum objectives \(\Sigma_f(A)=\operatorname{tr}(f(A))\) | Polynomial degree \(D\) of a Chebyshev expansion | | Spectral-coordinate TSD | SPD quadratic minimization | \(k\) spectral directions \(\{u_1,\dots,u_k\}\) | | Matrix-sign TSD | Non-smooth convex matrix optimization | Top-\(s\) singular directions of a subgradient | The three formulations are linked by spectral compression rather than by identical algebra. In [1802.06355], truncation occurs in polynomial approximation, but stochastic reweighting removes the bias of a fixed truncation. In [1802.03703], truncation means using only a subset of eigenvectors, so the method interpolates between randomized coordinate descent and a fully spectral method. In [2605.26977], truncation means replacing the full matrix-sign \(msgn(G)\) by the rank-\(s\) operator \(Tmsgn_s(G)\), which changes both the descent geometry and the required SVD computation. This distinction matters because convergence statements, computational costs, and admissible problem classes differ substantially across the three settings. A recurrent source of confusion is to treat TSD as if it were always either a stochastic spectral-gradient method or always a low-rank singular-vector method; the literature here shows that both interpretations occur, but in different models. ## 2. Chebyshev-based TSD for spectral-sum optimization Han et al. consider spectral-sum objectives for symmetric \(A\in\mathbb{R}^{d\times d}\) with spectrum in \([a,b]\), where \[ \Sigma_f(A)=\operatorname{tr}(f(A))=\sum_{i=1}^d f(\lambda_i). \] After the affine rescaling \[ \widetilde A=\frac{2}{b-a}A-\frac{b+a}{b-a}I, \] the spectrum of \(\widetilde A\) lies in \([-1,1]\), and an analytic \(f\) admits a uniformly convergent Chebyshev expansion \[ f(x)=\sum_{k=0}^\infty b_k T_k(\widetilde x), \qquad \Sigma_f(A)=\sum_{k=0}^\infty b_k\,\operatorname{tr}(T_k(\widetilde A)). \] The trace terms are estimated with Hutchinson’s method, \[ \operatorname{tr}(X)=\mathbb{E}_v[v^\top Xv], \] where \(v\) has independent Rademacher entries. A fixed degree-\(n\) truncation combined with \(M\) probe vectors gives a natural estimator, but that estimator is biased unless \(f\) is a polynomial of degree \(\le n\) [1802.06355]. The defining step of this TSD construction is stochastic truncation. Instead of truncating at a deterministic degree, the method samples \(D\) from a distribution \(\{q_d\}_{d\ge 0}\) and uses the reweighted random polynomial \[ \widehat p_D(x)=\sum_{k=0}^D \frac{b_k}{1-\sum_{i0\) for all \(k\). Consequently, \[ \widehat\Sigma=\operatorname{tr}(\widehat p_D(A))=\mathbb{E}_v[v^\top \widehat p_D(A)v] \] satisfies \(\mathbb{E}_{D,v}[\widehat\Sigma]=\Sigma_f(A)\). If \(A=A(\theta)\) depends smoothly on parameters, then each coordinate estimator \[ \widehat g_i=\frac{\partial}{\partial \theta_i}\bigl[v^\top \widehat p_D(A(\theta))v\bigr] \] is unbiased for \(\frac{\partial}{\partial\theta_i}\Sigma_f(A(\theta))\). The vectors \(w_k=T_k(\widetilde A)v\) and their derivatives can be built in \(O(1)\) extra matrix-vector multiplies per degree by using the chain rule together with the Chebyshev three-term recurrence [1802.06355]. The method’s remaining design freedom is the degree law \(\{q_d\}\). Under analytic decay \(|b_k|=O(\rho^{-k})\), TSD derives a variance-optimal truncation distribution under a fixed expected degree \(N\), with a closed-form solution \(q_d^*\) and exponentially small variance for large \(N\). The resulting estimator plugs into projected SGD and into an SVRG-style semi-stochastic loop. Under \(\alpha\)-strong convexity and smoothness assumptions on \(\Sigma_f(A(\theta))+g(\theta)\), SGD with \(\eta_t=1/(\alpha t)\) yields \[ \mathbb{E}\bigl[\|\theta^{(T)}-\theta^*\|^2\bigr]=O\!\left(\frac{1}{T}\right), \] with estimator variance \[ O\bigl((L_A^2/M+d' L_{\rm nuc}^2)(C_1+C_2N^4\rho^{-2N})\bigr), \] while the semi-stochastic construction attains linear convergence [1802.06355]. ## 3. Truncated spectral-coordinate descent for SPD quadratics A different use of the term appears in the quadratic setting \[ \min_x f(x)=\frac12 x^\top A x-b^\top x, \qquad x_*=A^{-1}b, \] with \(A\in\mathbb{R}^{n\times n}\) symmetric positive-definite and eigen-decomposition \(A=\sum_{i=1}^n \lambda_i u_i u_i^\top\), \(0<\lambda_1\le \cdots \le \lambda_n\). Here TSD is the variant called SSCD in the source paper: it augments the standard coordinate directions \(\{e_1,\dots,e_n\}\) by the \(k\) eigenvectors corresponding to the \(k\) smallest eigenvalues, \(\{u_1,\dots,u_k\}\) [1802.03703]. At each iteration, the method samples a direction from \[ D_k=\{e_1,\dots,e_n\}\cup\{u_1,\dots,u_k\} \] according to a distribution \(\mathcal D_k\), and then performs exact one-dimensional minimization, \[ x_{t+1}=x_t+h_t s_t, \qquad h_t=-\frac{s_t^\top(Ax_t-b)}{s_t^\top A s_t}. \] The sampling law is constructed from \[ \beta_i=\lambda_{k+1}-\lambda_i,\quad i=1,\dots,k,\qquad \alpha=1, \] and \[ C_k=\alpha\operatorname{Tr}(A)+\sum_{i=1}^k \beta_i=(k+1)\lambda_{k+1}+\sum_{i=k+2}^n \lambda_i, \] with probabilities \[ p(e_i)=\alpha A_{ii}/C_k,\qquad p(u_i)=\beta_i/C_k. \] The principal convergence theorem states that \[ \mathbb{E}\bigl[\|x_t-x_*\|^2\bigr]\le (1-\rho(k))^t\|x_0-x_*\|^2, \qquad \rho(k)=\lambda_{k+1}/C_k, \] and equivalently \[ \mathbb{E}[f(x_{t+1})-f(x_*)]\le (1-\rho(k))(f(x_t)-f(x_*)). \] Because \(C_k/\lambda_{k+1}\) is monotonically decreasing in \(k\), the rate improves as more spectral directions are added. The endpoints are explicit: \(k=0\) recovers randomized coordinate descent with diagonal sampling, for which \(\rho(0)=\lambda_1/\operatorname{Tr}(A)\), while \(k=n-1\) matches the pure spectral method SSD with \(\rho(n-1)=1/n\), independent of \(\kappa(A)\) [1802.03703]. This formulation makes truncation a mechanism for rate interpolation. The method retains the cheap directional structure of coordinate descent while inserting a small number of “hard” spectral directions associated with the smallest eigenvalues. For large sparse \(A\), the paper notes that the \(k\) eigenvectors may be obtained with Lanczos or randomized SVD at cost roughly \(O(\operatorname{nnz}(A)\cdot k+k^3)\), and it also analyzes inexact SSCD, in which approximate eigenvectors \(\bar u_i\) are permitted, as well as a mini-batch extension mSSCD. Reported experiments on synthetic problems of size \(n=30\) and up to \(n=10^5\) show a sharp phase-transition once \(k\) exceeds the number of “bad” small eigenvalues, factor \(\alpha^{-k}\) improvements under exponentially decaying spectra, and order-of-magnitude iteration-count reductions versus RCD in sparse tests with \(n=10^5\) and \(k=10^4\) [1802.03703]. ## 4. Top-\(s\) singular-direction TSD in non-smooth convex optimization In the non-smooth convex matrix setting, TSD is defined for \[ \min_{X\in\mathbb{R}^{n_1\times n_2}} f(X), \] where \(f\) is proper, lower-semicontinuous, convex, and possibly non-smooth. If \(G\in\partial f(X)\) has compact SVD \(G=U_r\Sigma_r V_r^\top\), the full matrix-sign operator is \[ msgn(G)=U_rV_r^\top. \] TSD truncates this object to the first \(s\) singular directions. Writing \[ G=U_r\operatorname{diag}(\sigma_1,\dots,\sigma_r)V_r^\top, \] and letting \(U_s\) and \(V_s\) collect the first \(s\) singular vectors, the truncated operator is \[ Tmsgn_s(G)=U_sV_s^\top, \] which is an element of the subdifferential of the Ky-Fan \(s\)-norm \(\|G\|_{(s)}=\sum_{i=1}^s \sigma_i(G)\). The update rule is \[ X^{(t+1)}=X^{(t)}-\eta_t\,Tmsgn_s(G^{(t)}), \qquad G^{(t)}\in\partial f(X^{(t)}). \] This is the TSD algorithm analyzed as a truncated counterpart of Spectral Descent (SD) in the context of Muon-type methods [2605.26977]. The analysis assumes convexity, Lipschitz continuity, and sharpness. Specifically, \(f\) is convex; there exists \(L>0\) such that \(|f(X)-f(Y)|\le L\|X-Y\|_F\); and \(f\) is \(\mu\)-sharp in the sense that \[ f(X)-f^*\ge \mu\,\operatorname{dist}(X,\mathcal X^*). \] Defining \(\kappa=\mu/L\in(0,1)\), the subgradient ranks \(r_t=\operatorname{rank}(G^{(t)})\), \(\bar r=\max_t r_t\), and \[ \alpha_s=\min\{1,s/\sqrt{\bar r}\}, \qquad \widetilde C=\kappa\alpha_s-\sqrt{s-\alpha_s^2}\sqrt{1-\kappa^2}, \] the paper assumes \(\kappa>\sqrt{1-\alpha_s^2/s}\), which implies \(\widetilde C>0\). With a geometric step-size, \[ \eta_t=\frac{\widetilde C}{s}\,\gamma^t\,\operatorname{dist}(X^{(0)},\mathcal X^*), \] the iterates satisfy the rank-\(s\) descent recursion \[ \operatorname{dist}^2(X^{(t+1)},\mathcal X^*)\le \operatorname{dist}^2(X^{(t)},\mathcal X^*)+\eta_t^2 s -2\eta_t \widetilde C\,\operatorname{dist}(X^{(t)},\mathcal X^*), \] and hence the global linear rate \[ \operatorname{dist}(X^{(t)},\mathcal X^*)\le \gamma^t\,\operatorname{dist}(X^{(0)},\mathcal X^*). \] The corresponding iteration complexity to reach \(\operatorname{dist}\le \varepsilon\) is \(\mathcal O(\ln(1/\varepsilon))\) [2605.26977]. The proof centers on a uniform lower bound for the truncated descent term \(\langle R,Tmsgn_s(G)\rangle\), where \(R\) is the residual to the solution set. The source identifies the core new technical lemma as a geometric bound that combines sharpness, the Lipschitz subgradient bound \(\|G\|_F\le L\), and the fact that only the top-\(s\) directions are retained. Relative to full SD, whose worst-case contraction constant depends on the instantaneous rank \(r_t\), TSD replaces that dependence by the fixed truncation level \(s\). The paper states that choosing \(s\approx \sqrt{\bar r}\) relaxes the condition the most, at the cost of computing a truncated SVD of rank \(s\) instead of a full SVD. The corresponding computational trade-off is explicit: full SVD \(O(n^3)\) versus top-\(s\) SVD \(O(ns^2)\) or power-method \(O(ns)\) [2605.26977]. ## 5. Regularization, weight decay, and robust low-rank recovery The non-smooth framework also introduces regularized TSD with decoupled weight decay, denoted RTSD-WD. With \(\lambda>0\), the momentum-free MuonW update becomes \[ X^{(t+1)}=X^{(t)}-\eta_t\Bigl(Tmsgn_s(\tilde G^{(t)})+\lambda X^{(t)}\Bigr), \] where \(\tilde G^{(t)}\) is a “spatially smoothed” subgradient from the \(\epsilon\)-neighborhood of \(X^{(t)}\). The paper shows that this is exactly a Frank-Wolfe (Conditional Subgradient) update over the spectrally constrained set \[ \{\|X\|_2\le 1/\lambda,\ \|X\|_*\le s/\lambda\}. \] Under the same convexity, Lipschitz, and sharpness assumptions, but no longer requiring \(\kappa\) large, the distance to the solution set decays at rate \(O(1/\sqrt T)\) [2605.26977]. The same paper applies RTSD-WD to robust low-rank matrix recovery. The measurement model is \[ b=\mathcal A(X^*)+e_1+e_2, \] where \(X^*\in\mathbb{R}^{n\times n}\) is low-rank, \(e_1\) is sparse, and \(e_2\) is dense but small. The optimization problem is the LAD formulation \[ \min_{X:\|X\|_*\le R}\; f(X)=\frac1m\|\mathcal A(X)-b\|_1, \qquad R=\|X^*\|_*. \] Under standard \(\ell_1/\ell_2\)-RIP on \(\mathcal A\) and its restriction to outlier indices, the paper proves that \(f\) is Lipschitz with respect to the nuclear norm with constant \(L_{\mathcal A}=\sqrt{2/\pi+\delta_1}\) and establishes a restricted sharpness inequality \[ f(X)-f(X^*)\ge \mu\|X-X^*\|_F-\frac{2}{m}\|e_2\|_1. \] Applying RTSD-WD with \(\lambda=1/R\) and \(s=1\) then yields, with high probability, \[ \|X^{(T)}-X^*\|_F \le C_3\,\frac{L_{\mathcal A}\sqrt m\,R}{\mu\sqrt T} +\frac{\|e_2\|_1/m}{\mu} =O\!\left(\frac1{\sqrt T}\right)+O(\text{dense-noise}), \] while requiring only the top-1 singular vector per iterate, for an \(O(n^2)\) per-step cost [2605.26977]. This application highlights a specific advantage of truncation in the non-smooth setting: the update is aligned with the dominant singular direction rather than with the full subgradient spectrum. In regimes where a top-1 or low-rank SVD is much cheaper than a full factorization, the resulting reduction in per-iteration cost is central to the method’s usefulness. ## 6. Comparative interpretation and recurring misconceptions The three TSD constructions differ first in the mathematical object that is truncated. In [1802.06355], truncation is over Chebyshev degree, and the estimator is made unbiased by randomization and reweighting. In [1802.03703], truncation is over eigen-directions, and the method samples from a mixed set of coordinate and spectral directions. In [2605.26977], truncation is over singular directions of a subgradient, producing a low-rank matrix-sign step. These are not interchangeable algorithmic templates. A second misconception is that truncation necessarily introduces uncontrolled approximation error. That is not true in the Chebyshev formulation, where the point of the random degree \(D\) and the weights \((1-\sum_{i