Sketched Trust-Region (STR)

• Sketched Trust-Region (STR) is a multifidelity trust-region method that augments the standard high-fidelity step with a corrective low-dimensional direction computed via a random sketch.
• It constructs a reduced objective by projecting the data onto a lower-dimensional space using a Gaussian sketch, preserving geometric properties essential for convergence.
• The method seamlessly falls back to classical trust-region steps when the sketched correction fails to improve the objective, ensuring robust and efficient convergence.

Sketched Trust-Region (STR) is a multifidelity trust-region method for large-scale, unconstrained nonlinear optimization in which a classical full-space trust-region step is augmented by a sketch-based corrective direction computed from a low-fidelity objective in a reduced feature space. In the formulation introduced in “Trust-Region Methods with Low-Fidelity Objective Models,” STR is instantiated within the Magical Trust Region (MTR) framework: a primary step is obtained from the standard high-fidelity trust-region model of the original objective, while a secondary “magical” direction is produced by solving a reduced trust-region subproblem built from a random sketch of the data matrix (Angino et al., 1 Nov 2025). The method is motivated in particular by supervised binary classification problems of the form

$$\min_{w\in\mathbb{R}^n} f(w) = \frac{1}{q}\sum_{i=1}^q \ell(w;x_i,y_i),$$

with both $n$ and $q$ potentially large.

1. Problem class and conceptual role

STR is proposed for large-scale, unconstrained nonlinear optimization under the assumptions that $f$ is bounded below and twice continuously differentiable. The motivating setting is supervised binary classification with classifier weights $w\in\mathbb{R}^n$, data pairs $\{(x_i,y_i)\}_{i=1}^q$, data matrix $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$, and a smooth loss $\ell$, including logistic loss and squared loss with softmax (Angino et al., 1 Nov 2025).

The method is best understood relative to the classical trust-region (TR) framework. Standard TR methods build a quadratic model

$$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$

and compute a step by solving a trust-region subproblem in the full parameter space. STR does not replace this mechanism. Instead, it preserves the full-space TR step and supplements it with a second direction derived from a cheaper low-fidelity model. This design places STR within multifidelity optimization, where an accurate but expensive model $f$ is combined with a cheaper but inexact model $n$0 to reduce cost while retaining the convergence logic of the high-fidelity method.

Within the MTR interpretation, the primary step is the standard TR step from the full model, and the secondary step is an auxiliary correction coming from additional information. In STR, that additional information is obtained by dimension reduction on the feature space: rather than optimizing directly over $n$1, the method constructs a reduced parameter $n$2 with $n$3, linked to $n$4 through a linear projector $n$5. This suggests that STR should be viewed as a corrective, data-driven augmentation of trust-region iterations rather than as a replacement for the underlying TR method.

2. Construction of the sketched low-fidelity model

At iteration $n$6, STR first computes the high-fidelity trust-region step $n$7 and forms the intermediate point

$n$8

Around this intermediate point, it constructs a low-dimensional objective by sketching the feature space (Angino et al., 1 Nov 2025).

The sketch is defined through a random matrix $n$9 with $q$0. The features are projected as

$q$1

producing the compressed data matrix $q$2. The reduced objective is then

$q$3

where $q$4 is a low-fidelity loss derived from the original loss, often with the same functional form in the reduced feature space. The model is evaluated at

$q$5

and a second-order model $q$6 is built around this reduced point.

A central feature of STR is that the sketching matrix is random and recomputed every iteration. Its entries are i.i.d. Gaussian,

$q$7

so the sketch is a dense Gaussian Johnson–Lindenstrauss embedding. In the experiments, $q$8 is chosen much smaller than $q$9, for example $f$0–$f$1 of $f$2. The stated effect of sketching is to map the feature space from $f$3 to $f$4 while approximately preserving distances and inner products with high probability for appropriate $f$5. In the STR context, this means that the geometry of the loss landscape along data-induced directions is approximately preserved in the sketched space.

The low-fidelity model is therefore cheap for two reasons. First, its variables live in $f$6 rather than $f$7. Second, the associated trust-region subproblem can be handled with low-dimensional vectors and at most $f$8 inner conjugate-gradient iterations. A plausible implication is that STR is especially attractive when curvature information is informative but full-space subproblem solves are expensive.

3. Mathematical formulation and algorithmic structure

The full high-fidelity model at iteration $f$9 is the quadratic Taylor approximation

$w\in\mathbb{R}^n$0

and the high-fidelity step is

$w\in\mathbb{R}^n$1

usually computed approximately, for example by Steihaug–Toint CG or the Cauchy point (Angino et al., 1 Nov 2025).

The low-fidelity quadratic model is

$w\in\mathbb{R}^n$2

with $w\in\mathbb{R}^n$3, and the reduced subproblem is

$w\in\mathbb{R}^n$4

Because $w\in\mathbb{R}^n$5 lives in the reduced parameter space, STR lifts it back to the original space using the adjoint of the sketch. The lifted “magical” direction is

$w\in\mathbb{R}^n$6

and the composite step is

$w\in\mathbb{R}^n$7

where $w\in\mathbb{R}^n$8 is either fixed or found by a small line search. The correction is used only if it improves the original objective: $w\in\mathbb{R}^n$9 If this inequality fails, the method sets $\{(x_i,y_i)\}_{i=1}^q$0, so the iteration reduces to standard TR.

The trust-region ratio is modified to account for the fact that the high-fidelity model predicts only the effect of $\{(x_i,y_i)\}_{i=1}^q$1, not of the added magical direction. STR defines

$\{(x_i,y_i)\}_{i=1}^q$2

The denominator therefore combines the model reduction for $\{(x_i,y_i)\}_{i=1}^q$3 with the actual full-objective reduction produced by the correction. The iterate is accepted if $\{(x_i,y_i)\}_{i=1}^q$4, and the trust-region radius is updated using thresholds $\{(x_i,y_i)\}_{i=1}^q$5 and shrink/expand factors $\{(x_i,y_i)\}_{i=1}^q$6 in the standard set-valued TR form. Convergence is detected, for example, when $\{(x_i,y_i)\}_{i=1}^q$7.

This algorithmic structure makes explicit that STR is additive rather than substitutive: the primary step remains the full trust-region step, and the sketched solve supplies an auxiliary correction only when it is demonstrably useful for the original objective.

4. Sketching mechanism and theoretical interpretation

The sketching mechanism in STR is Gaussian random sketching, with a new sketch $\{(x_i,y_i)\}_{i=1}^q$8 drawn at each outer iteration and applied to the data matrix $\{(x_i,y_i)\}_{i=1}^q$9 to form $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$0 (Angino et al., 1 Nov 2025). The stated cost impact is $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$1 to construct $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$2, $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$3 to multiply $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$4, and a reduced trust-region solve in vectors of dimension $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$5 with at most $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$6 CG iterations.

The theoretical interpretation relies on classical Johnson–Lindenstrauss geometry. For any vector $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$7, with high probability,

$X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$8

provided $X=[x_1,\dots,x_q]\in\mathbb{R}^{n\times q}$9 for an $\ell$0-point subset or relevant subspace. The paper also states that inner products and angles are approximately preserved. In the STR setting, this suggests that the geometry of the loss landscape along directions induced by the data $\ell$1 is approximately preserved in the reduced space, and that gradient and Hessian information extracted from $\ell$2 remains meaningful for the full objective.

A useful antecedent is “Random projections for trust region subproblems,” which studies trust-region subproblems approximated by random projections and shows that solving the projected problem in low dimension and lifting via $\ell$3 yields feasibility and objective-value approximation guarantees with high probability (Vu et al., 2017). That work does not formulate a full multifidelity TR algorithm, but it provides a closely related subproblem-level perspective: a sketched or projected trust-region problem can serve as a high-probability surrogate for the original subproblem. This suggests a broader interpretation of STR as part of a line of work in which trust-region computations are accelerated by low-dimensional surrogate spaces.

The choice of sketch size $\ell$4 governs the central trade-off. Larger $\ell$5 captures more directions and more accurately approximates the full problem, typically strengthening the magical direction and reducing outer iterations. Smaller $\ell$6 lowers the cost of the reduced problem but risks weaker corrections. The experimental discussion reports monotone improvement with $\ell$7: as $\ell$8 increases, outer iteration counts decrease, while per-iteration work increases.

5. Convergence properties, implementation, and numerical behavior

The theoretical discussion in the STR paper is primarily algorithmic and experimental rather than theorem-heavy, but its design is explicitly chosen so that if the low-fidelity correction is rejected, the iteration becomes a standard trust-region step (Angino et al., 1 Nov 2025). Under standard trust-region assumptions—$\ell$9 bounded below, $$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$0 Lipschitz continuous, and approximate TR subproblem solves satisfying typical sufficient-reduction conditions—the method inherits the classical guarantee

$$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$1

The paper states that the global convergence properties match those of classical trust-region methods, while the low-fidelity model is used solely to accelerate convergence in practice. It does not present explicit iteration-complexity bounds such as $$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$2 evaluations, but it argues that worst-case complexity can be no worse than that of standard TR because the method falls back to TR whenever the magical direction is unhelpful.

In the reported implementation, the training losses are the regularized logistic loss

$$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$3

and the regularized least-squares-type loss

$$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$4

with $$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$5. The high-fidelity TR solver is either Steihaug–Toint CG with a small number of inner iterations or the Cauchy point; the low-fidelity solver is Steihaug–Toint CG in the reduced space with at most $$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$6 iterations. The implementation is in Python / PyTorch 2.8.0 and the experiments are CPU-only.

The numerical study compares TR, STR, and SVDTR on three LIBSVM datasets: Australian ($$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$7, $$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$8), Mushroom ($$m_k^H(s) := f(w_k) + \nabla f(w_k)^T s + \tfrac{1}{2}s^T B_k s,$$9, $f$0), and Gisette ($f$1, $f$2). The primary metric is $f$3 versus outer iterations, and wall-clock time is also reported for larger problems. On Australian and Mushroom, all methods converge quickly, but STR and SVDTR consistently require fewer outer iterations than TR, with iteration-count reduction monotone in $f$4. On Gisette, the gains from dimension reduction are more pronounced: for both $f$5 and $f$6, STR and SVDTR significantly reduce iteration counts relative to TR, and wall-clock plots show that the reduced iterations more than compensate for the extra cost of the low-fidelity subproblems. The main stated conclusion is that STR reliably accelerates classical TR, particularly in high-dimensional problems where accurate full-space TR solves are expensive.

6. Relation to SVDTR, earlier sketched TR ideas, and acronym ambiguity

The STR paper introduces two methods within the same MTR template: STR and SVDTR. Their distinction lies in how the reduced space is constructed.

Method	Reduced space construction	Stated characteristics
STR	Random sketch $f$7, recomputed every iteration	No preprocessing; on-the-fly; robust; attractive when preprocessing is not possible
SVDTR	Top $f$8 left singular vectors of $f$9, computed once	Captures dominant directions of variability; requires truncated SVD preprocessing
Classical TR	No reduced space	Uses only the full-space model

STR uses a random sketch recomputed at each iteration, requires no preprocessing, and is described as robust and easy to implement, especially when computing an SVD of $n$00 is too expensive. SVDTR constructs $n$01 once from the top $n$02 left singular vectors of $n$03, capturing dominant directions of variability and often performing best when singular values decay rapidly and the preprocessing cost is acceptable (Angino et al., 1 Nov 2025). Empirically, both outperform TR; on high-dimensional logistic regression for Gisette, SVDTR with sufficiently large $n$04 tends to give the best iteration/time performance, while STR remains attractive when preprocessing is impossible or data are changing.

The broader literature contains closely related but not identical uses of sketching in trust-region methods. “Random projections for trust region subproblems” studies projected trust-region subproblems in derivative-free optimization and analyzes their feasibility and approximation properties, but it does not introduce the STR acronym (Vu et al., 2017). By contrast, “A Stochastic Trust Region Method for Non-convex Minimization” uses STR to denote a stochastic trust-region method with variance-reduced gradient and Hessian estimators, not a sketched multifidelity method (Shen et al., 2019). This terminological overlap is a common source of confusion. A precise usage therefore distinguishes Sketched Trust-Region in the multifidelity MTR sense from stochastic trust-region methods that use the same acronym for a different algorithmic family.

A second misconception is to treat STR as a method that solves only a reduced problem. The defining feature of the 2025 STR construction is the opposite: sketching is used only to generate a secondary correction direction, while the primary step, objective evaluation, and acceptance logic remain anchored in the full model $n$05. This suggests that its robustness stems less from the accuracy of the sketch alone than from the fallback structure that preserves the behavior of classical trust-region iterations whenever the sketched correction is not beneficial.