Sub-Sampled Trust-Region Methods

• The paper introduces sub-sampled trust-region methods that approximate objective values, gradients, and Hessians using subsets of data to reduce computation while retaining global convergence.
• These methods integrate probabilistic and deterministic sampling rules to adjust derivative accuracy dynamically, ensuring optimal worst-case complexity in finite-sum optimization.
• Practical experiments show significant runtime savings and efficiency improvements over full trust-region approaches in non-convex optimization settings with noisy evaluations.

Sub-sampled trust-region methods are trust-region algorithms for large-scale smooth optimization in which the objective value, gradient, Hessian, or trust-region subproblem itself is approximated using only a subset of the available information. In the finite-sum setting,

$$\min_{x\in\mathbb{R}^n} f(x)=\frac1d\sum_{i=1}^d f_i(x),$$

the central objective is to reduce per-iteration cost without losing the globalization, stationarity, and worst-case complexity properties associated with classical trust-region methods. Across recent formulations, the common pattern is a local quadratic or higher-order model, a radius-constrained step computation, and an acceptance mechanism based on actual-versus-predicted reduction, but with derivative or function information replaced by noisy, inexact, or subsampled estimates (Yao et al., 2018, Bellavia et al., 2021, Goncalves et al., 23 Jul 2025).

1. Core trust-region structure

At the algorithmic level, the finite-sum trust-region model typically has the form

$$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$

where $g_k$ and $B_k$ are exact or approximate derivatives. In first-order sub-sampled variants, only the gradient is subsampled, while $B_k$ is any symmetric matrix satisfying a uniform bound; in second-order variants, both the gradient and Hessian are subsampled (Goncalves et al., 23 Jul 2025). In large-scale non-convex optimization, a closely related inexact trust-region formulation uses approximate gradient $g_t$, approximate Hessian $H_t$, and an approximate subproblem solution $s_t$, terminating when

$$\|g_t\|\le \epsilon_g,\qquad \lambda_{\min}(H_t)\ge -\epsilon_H,$$

so that the returned point is approximately second-order stationary (Yao et al., 2018).

The globalization mechanism remains recognizably classical. The standard trust-region ratio is

$\rho_k=\frac{f(x_k)-f(x_k+d_k)}{m_k(0)-m_k(d_k)},$

and successful iterations accept the trial step and enlarge the radius, while unsuccessful iterations reject the step and shrink it (Goncalves et al., 23 Jul 2025). In the inexact Newton-type formulation, the same ratio is written as

$$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$0

with acceptance threshold $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$1 and multiplicative radius updates by a factor $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$2 (Yao et al., 2018).

Sub-sampled trust-region methods differ from purely first-order stochastic schemes because they preserve this model-based structure. The step is not an arbitrary stochastic descent direction; it is the outcome of an approximate constrained minimization, typically required to satisfy a Cauchy-type decrease condition and, in second-order settings, also to exploit negative curvature when present (Yao et al., 2018, Goncalves et al., 23 Jul 2025).

2. Inexactness, noise, and complexity theory

A major theoretical development is the extension of trust-region complexity analysis to noisy and probabilistic models. The TR$$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$3NE framework studies smooth unconstrained optimization with noisy objective values and noisy derivatives up to arbitrary order $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$4. It defines the approximate optimality measure

$$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$5

and calls $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$6 a $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$7-th order $$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$8-approximate minimizer when

$$m_k(d)=f(x_k)+\langle g_k,d\rangle+\frac12\langle B_k d,d\rangle,\qquad \|d\|\le \Delta_k,$$9

Under suitable probabilistic assumptions on model accuracy and function-value accuracy, the expected number of inexact evaluations required to obtain such a point satisfies

$g_k$0

which matches the sharp order known for deterministic trust-region methods with exact derivatives (Bellavia et al., 2021).

For second-order finite-sum optimization, inexact trust-region methods with approximate gradients, approximate Hessians, and approximate subproblem solves retain the optimal worst-case iteration complexity

$g_k$1

provided the approximation errors satisfy

$g_k$2

with mild conditions linking $g_k$3 and $g_k$4 to $g_k$5 and $g_k$6 (Yao et al., 2018). A deterministic adaptive subsampling framework later established the same asymptotic second-order trust-region complexity for finite-sum problems,

$g_k$7

while also giving the first-order bound $g_k$8 (Goncalves et al., 23 Jul 2025).

A distinctive feature of the probabilistic analysis is its treatment of intrinsic noise. In the first-order specialization TR$g_k$9NE, if the probabilistic assumptions fail for a particular realization, the failure itself implies a degraded stationarity statement: the realized iterate must satisfy a bound of the form

$B_k$0

This does not recover the full theory, but it quantifies what can still be concluded when stochastic errors dominate near the minimizer (Bellavia et al., 2021).

3. Sampling rules and accuracy control

The literature separates into probabilistic batch-size control and deterministic sample-size control. In finite-sum inexact Newton-type trust-region methods, the gradient and Hessian are estimated by uniform random minibatches,

$B_k$1

and a sampling lemma states that if

$B_k$2

then the desired derivative-accuracy bounds hold with probability at least $B_k$3 (Yao et al., 2018).

The probabilistic framework of TR$B_k$4NE translates similar ideas to function values and derivatives through accurate-model and accurate-function events. In the finite-sum case, the function decrease estimate is formed from a batch difference,

$B_k$5

and concentration inequalities imply exponentially decaying tail bounds for the failure probabilities. The practical message stated in the paper is that larger batches are needed as one approaches a solution, because the relative noise must shrink enough for the trust-region acceptance and complexity theory to apply (Bellavia et al., 2021).

Deterministic adaptive subsampling replaces probability bounds by explicit radius-dependent rules. If a subsample omits only a fraction $B_k$6 of indices, then

$B_k$7

The adaptive procedure chooses sample sizes through quantities such as $B_k$8, $B_k$9, and $B_k$0, all tied to the current trust-region radius $B_k$1, so that smaller trust regions force more accurate derivative estimates (Goncalves et al., 23 Jul 2025).

A different deterministic mechanism appears in inexact restoration trust-region methods. There the sample size is treated as a feasibility variable, with infeasibility measure

$B_k$2

and a merit function

$B_k$3

This schedules sample-size growth by balancing progress in the subsampled objective against progress toward full precision (Bellavia et al., 2019).

4. Principal algorithmic variants

The family now includes probabilistic, deterministic, non-monotone, and variance-reduced formulations.

Variant	Distinctive mechanism	Stated guarantee
TR$B_k$4NE (Bellavia et al., 2021)	Noisy derivatives, probabilistic model/function events, adaptive model order $B_k$5	$B_k$6
Inexact TR (Yao et al., 2018)	Sub-sampled gradient and Hessian, approximate subproblem solve	$B_k$7
Deterministic STR (Goncalves et al., 23 Jul 2025)	Radius-dependent deterministic sample-size adjustment	$B_k$8 and $B_k$9
ASNTR (Krejic et al., 2023)	Non-monotone acceptance and independent additional sampling	$g_t$0 a.s.
TR-SVR (Zheng, 2024)	SVRG-style control variate inside a trust-region/SQP model	$g_t$1-type average squared-gradient bound

ASNTR is notable because it explicitly addresses the dependence problem created when the same minibatch is used both to build the step and to evaluate the trial point. It introduces an independent additional sample $g_t$2 and, in the stochastic phase, accepts a step only if both

$g_t$3

This permits a non-monotone trust-region method with noisy oracles and adaptive sample growth driven by either failure of the independent test or the relation $g_t$4 (Krejic et al., 2023).

TR-SVR occupies a different position. It uses a two-loop structure with a full gradient at an outer reference point $g_t$5 and a variance-reduced estimator

$g_t$6

whose variance satisfies

$g_t$7

The paper explicitly notes that it does not use a classical ratio-based acceptance test and that the radius update is less formal than in standard trust-region algorithms (Zheng, 2024).

5. Trust-region subproblems and low-dimensional reductions

Sub-sampling concerns not only derivatives but also the inner trust-region subproblem. Inexact second-order trust-region methods rely on steps that are at least as good as the Cauchy and eigen points,

$g_t$8

so approximate subproblem solves are acceptable provided they preserve sufficient decrease along gradient and negative-curvature directions (Yao et al., 2018). Deterministic adaptive STR uses a Cauchy-type decrease condition in its first-order form and a stronger condition exploiting either gradient reduction or negative curvature reduction in its second-order form; numerically, it uses Dogleg (Goncalves et al., 23 Jul 2025). Inexact restoration variants use CG-Steihaug for quadratic models and the normalized gradient step for linear models (Bellavia et al., 2019).

Several related works reduce the dimensionality of the subproblem itself. Random projection methods solve a projected lower-dimensional surrogate obtained from a Gaussian matrix $g_t$9, replacing the original constrained trust-region subproblem by one in $H_t$0 and then lifting the solution back with $H_t$1. The approximation guarantees depend on Johnson–Lindenstrauss-type norm preservation, feasibility transfer, and the fullness of the feasible set (Vu et al., 2017). A different randomized reduction, TLTR, computes a full-space trust-region step $H_t$2, then adds a correction from a randomly sketched subspace,

$H_t$3

accepting the composite step through a trust-region ratio that combines the full-space model decrease with the actual decrease produced by the subspace correction (Angino et al., 2024).

For repeated trust-region subproblems with slowly varying or reused Hessians, the TREK method supplies a reusable inner solver. It builds an extended Krylov subspace from one factorization of $H_t$4, reduces the trust-region problem to a tiny projected problem, and then solves the projected problem cheaply by root-finding. The paper explicitly positions this as useful when a sampled or approximate Hessian is reused and the trust-region radius changes across outer iterations (Daas et al., 14 Nov 2025).

6. Reported performance, limitations, and recurring issues

The empirical results consistently frame sub-sampled trust-region methods as mechanisms for reducing derivative cost rather than changing the outer trust-region logic. In the inexact Newton-type study, inexact TR was reported as roughly 3–5× faster than SubH TR and 5–10× faster than Full TR in terms of propagation count on non-convex binary classification problems (Yao et al., 2018). In deterministic adaptive STR, reported cost reductions relative to full TR were 4% for $H_t$5, 39% for $H_t$6, 33% for $H_t$7, and 50% for $H_t$8; for $H_t$9, the sample size sometimes dropped as low as 273 components (Goncalves et al., 23 Jul 2025).

For deep-learning-oriented stochastic trust-region methods, ASNTR was reported to achieve higher training and testing accuracy and lower loss than STORM for the same gradient-evaluation budget, while often requiring significantly fewer gradient evaluations (Krejic et al., 2023). TLTR reported that larger random-subspace dimensions reduce iteration count but increase cost per iteration, and that sketch sizes around 20–30% of $s_t$0 were often a good compromise in the experiments (Angino et al., 2024).

Several limitations recur across the literature. First, derivative accuracy requirements typically tighten near stationarity; subsampling is therefore not a license for permanently small batches. The probabilistic analysis of noisy trust-region methods makes this explicit by showing that larger batches are needed near a solution (Bellavia et al., 2021). Second, not every stochastic trust-region method preserves the classical trust-region acceptance framework: TR-SVR lacks an explicit actual-versus-predicted reduction ratio and provides an average-based first-order guarantee rather than a detailed oracle complexity statement (Zheng, 2024). Third, intrinsic noise may prevent probabilistic assumptions from holding forever; in that regime the strongest complexity results may fail, although degraded stationarity conclusions remain available in the first-order setting (Bellavia et al., 2021).

A plausible implication is that “sub-sampled trust-region methods” should be understood less as a single algorithm and more as a design space. The unifying principle is the preservation of trust-region globalization under cheaper information acquisition, while the main technical differences lie in how derivative accuracy is certified, how sample sizes are adapted, and how the trust-region subproblem is solved or reduced.