Cost-Aware Stochastic Gradient Descent

• Cost-Aware SGD is a stochastic optimization strategy that explicitly models the trade-off between gradient quality and costs such as data sampling, communication, and computational effort.
• It adapts parameters like batch size, surrogate gradient computation, and estimator bias to optimize performance across diverse cost models from PDE solves to federated learning contexts.
• This design principle underpins methods in execution cost control, variance reduction, and distributed learning while ensuring convergence properties and practical efficiency gains.

Cost-aware stochastic gradient descent denotes, in the literature summarized here, a class of stochastic optimization methods in which the gradient estimator, the sampling rule, or the update schedule is chosen by explicitly trading optimization progress against an associated cost. The relevant cost may be the number of sampled data points, full-gradient passes, communication rounds, memory, PDE solves, trajectory rollouts, agent participation costs, or the execution cost of trading itself. Under this view, cost-awareness is not a single algorithmic device but a design principle: batch size can be optimized for improvement per sample, variance reduction can be made cheaper by replacing exact anchors with surrogates, biased but consistent gradients can replace expensive unbiased ones, and distributed incentives can be used to control the quality of stochastic gradients supplied by strategic agents (Pirotta et al., 2017, Shah et al., 2016, Kolev, 2024, Akbay et al., 2022, Baumgarten et al., 3 Jun 2025, Nobile et al., 28 Jul 2025).

1. Conceptual foundation

Classical SGD fixes an update rule of the form

$w_{k+1}=w_k-\gamma_k g_k,$

or, with constraints,

$w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$

and typically treats the stochastic gradient oracle $g_k$ as given. Cost-aware variants instead optimize some aspect of how $g_k$ is obtained or used. In the papers considered here, this includes choosing minibatch size online, replacing full-gradient anchors by subset surrogates, constructing reduced-support empirical measures, using biased but consistent estimators when unbiased ones are too expensive, allocating work across fidelity levels, or rewarding agents according to gradient quality rather than unverifiable effort (Chen et al., 2018, Pirotta et al., 2017, Shah et al., 2016, Cosentino et al., 2020, Akbay et al., 2022, Baumgarten et al., 3 Jun 2025).

Two recurring themes organize the area. First, gradient quality and computational burden are jointly modeled rather than separated. In batch-size adaptation, a larger batch improves the reliability of the gradient estimate but consumes more samples; in federated learning, a larger minibatch lowers gradient variance but raises each agent’s cost; in graph learning, larger neighborhood samples reduce estimator error but increase computation; in multilevel PDE optimization, fine-level samples reduce bias but are far more expensive than coarse ones (Pirotta et al., 2017, Akbay et al., 2022, Chen et al., 2018, Baumgarten et al., 3 Jun 2025). Second, many methods replace exactness by structured approximation. The approximation may be statistical, as with biased but consistent graph gradients; algebraic, as with least-squares control variates; geometric, as with Carathéodory recombination; or economic, as in execution-cost control where SGD approximates an optimal control law that is known in principle but inconvenient to deploy (Chen et al., 2018, Nobile et al., 28 Jul 2025, Cosentino et al., 2020, Kolev, 2024).

A distinctive feature of the literature is that “cost” does not always mean wall-clock time alone. In trade execution, the optimized quantity is the implementation shortfall itself; in strategic federated learning, the relevant cost is borne by the agents and includes data collection, computation, communication, and privacy or operational burden; in decentralized SGD, communication structure and privacy constraints determine which stochastic input sequences are feasible (Kolev, 2024, Akbay et al., 2022, Hu et al., 2022).

2. Cost models and optimization criteria

The most explicit cost-sensitive criterion in the surveyed work appears in batch-size adaptation. The paper "Cost-Sensitive Approach to Batch Size Adaptation for Gradient Descent" selects the batch size by maximizing a lower bound on expected improvement per sample,

$$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$

or, in the quadratic variant, by replacing the linear lower bound with a curvature-aware lower bound derived from a Taylor approximation (Pirotta et al., 2017). This formulation makes the trade-off operational: additional samples are acquired only when the expected improvement justifies their cost.

A different criterion appears in variance-reduced finite-sum methods. CheapSVRG replaces the exact SVRG anchor

$$\widetilde{\mu}=\frac{1}{n}\sum_{i=1}^n \nabla f_i(\widetilde w)$$

with a subset surrogate

$$\widetilde{\mu}_S=\frac{1}{s}\sum_{i\in S}\nabla f_i(\widetilde w).$$

Here the parameter $s$ directly controls the quality of the surrogate and the per-epoch complexity: $s=0$ degenerates to vanilla SGD, $s=n$ recovers SVRG, and intermediate $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$0 spans a continuum between the two (Shah et al., 2016). SCSG makes a similar move at the outer level, replacing full-gradient epochs by mini-batch gradients of size $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$1 and pairing them with a geometrically distributed number of inner updates whose expectation is also $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$2, thereby targeting regimes where a full pass over the data is unnecessary (Lei et al., 2016).

In constrained uncertainty quantification and scientific computing, the cost model becomes multilevel. The multilevel stochastic gradient descent method for PDE-constrained control decomposes the gradient estimator into coarse and fine components and chooses level-dependent sample counts $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$3 so that coarse levels absorb most samples while fine levels are sampled sparsely. The corresponding mean-square error is split into sampling error and numerical bias, and the method chooses mesh level $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$4, sample counts $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$5, and step size $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$6 according to the current error target and computational resources (Baumgarten et al., 3 Jun 2025). The least-squares control-variate method for continuous-distribution objectives keeps exactly one new gradient evaluation per iteration, but adds memory and least-squares overhead to reduce variance, making cost-awareness a trade-off between gradient evaluations, memory size, and approximation-space dimension (Nobile et al., 28 Jul 2025).

The following summary captures the main cost notions that recur across the literature.

Cost notion	Mechanism	Representative papers
Samples per update	Adaptive batch size, minibatch incentives	(Pirotta et al., 2017, Akbay et al., 2022)
Full-gradient passes	Subset surrogates, mini-batch outer gradients	(Shah et al., 2016, Lei et al., 2016)
Gradient-estimator complexity	Biased consistent estimators, recombination	(Chen et al., 2018, Cosentino et al., 2020)
Fidelity-dependent solve cost	MLMC level allocation, budgeted batching	(Baumgarten et al., 3 Jun 2025)
Per-iteration PDE/expectation cost	Least-squares control variates	(Nobile et al., 28 Jul 2025)
Market execution cost	SGD approximation to optimal control policy	(Kolev, 2024)

This diversity of criteria suggests that cost-aware SGD is best understood as a family of optimization designs in which the stochastic oracle is co-optimized with the objective, rather than as a single algorithm.

3. Principal algorithmic families

One major family adapts batch size directly. L-PAST and Q-PAST estimate a probabilistic lower bound on expected improvement and divide it by batch size, yielding an online rule that increases the batch when gradient uncertainty is large relative to gradient magnitude and decreases it when cheaper, noisier updates are preferable. The quadratic variant additionally uses Hessian information, making it more conservative but also more expensive (Pirotta et al., 2017).

A second family reduces the cost of variance reduction. CheapSVRG keeps the SVRG inner-loop structure but replaces the exact full gradient with a subset average, thereby obtaining linear convergence up to a residual neighborhood whose size depends on the surrogate quality (Shah et al., 2016). SCSG also belongs to the SVRG family, but its distinctive features are the mini-batch outer gradient and a geometrically distributed inner-loop length. Its stated advantage is that both computation cost and communication cost need not scale linearly with $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$7, and can be independent of $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$8 when target accuracy is low (Lei et al., 2016).

A third family replaces random minibatching by structured reduction of the empirical measure. Carathéodory Sampling constructs a reduced-support probability measure $w_{k+1}=\Pi_S(w_k-\gamma_k g_k),$9 that preserves chosen expectations exactly at a recombination point, with the gradient coordinates serving as the preserved statistics. After recombination, multiple descent steps are taken on the reduced measure until a control statistic indicates that the approximation has degraded. Combined with Block Coordinate Descent, this becomes CaBCD, which exploits small block sizes such as $g_k$0 to make recombination practical in high dimensions (Cosentino et al., 2020).

A fourth family accepts bias to lower oracle cost. In graph learning, exact unbiased gradients may cost $g_k$1 because each training example depends on multihop neighborhoods. The consistent-gradient framework therefore samples a fixed number of neighbors or nodes per layer, producing biased but consistent gradient estimators. The paper’s central claim is that such estimators preserve the standard asymptotic convergence behavior of SGD under appropriate assumptions, while substantially reducing the cost of gradient computation in structured settings (Chen et al., 2018). A related but distinct construction appears in data-driven policy gradients for LQR, where both indirect identification-based and direct zeroth-order policy-gradient estimates are modeled as biased stochastic gradient oracles, and the design problem becomes one of selecting estimator parameters so that the bias decays sufficiently fast for SGD convergence (Song et al., 21 Feb 2026).

A fifth family uses control variates and multilevel structure for expensive expectation-based objectives. In multilevel SGD for PDE-constrained optimal control, the stochastic gradient is estimated by a parallel multilevel Monte Carlo procedure that controls both discretization bias and sampling error while allocating work optimally across levels (Baumgarten et al., 3 Jun 2025). In stochastic gradient with least-squares control variates, a weighted discrete least-squares surrogate of the gradient map is fitted from past samples and then used as a control variate in an SGD update that still requires exactly one new gradient evaluation per iteration (Nobile et al., 28 Jul 2025).

Finally, cost-awareness can be embedded in the organization of distributed learning itself. In the federated mechanism-design formulation, agents strategically choose minibatch sizes $g_k$2, incur a convex increasing cost $g_k$3, and may opt out. Because the server cannot verify minibatch sizes directly, it rewards each gradient according to its distance from a peer-based reference $g_k$4. The paper proves that this output-agreement mechanism has a cooperative Nash equilibrium and is budget balanced at the requested minibatch size sequence $g_k$5 (Akbay et al., 2022). In this setting, cost-aware SGD is simultaneously an optimization method and an incentive mechanism.

4. Theoretical guarantees

The theoretical literature does not offer one universal guarantee; instead, it shows that many different cost-aware relaxations preserve useful convergence properties. For biased but consistent gradients, the strongest general statement is that strongly convex, convex, and nonconvex objectives retain the same asymptotic rates as standard unbiased SGD, although the results are given as high-probability bounds rather than expectation bounds. The reported rates are $g_k$6 in the strongly convex case, $g_k$7 in the convex case, and $g_k$8 for the nonconvex stationarity measure (Chen et al., 2018).

Variance-reduced cost-aware methods generally obtain stronger per-iteration progress but under more structure. CheapSVRG converges linearly in expectation up to a residual neighborhood, with the contraction factor deteriorating as the subset size $g_k$9 decreases and the residual term containing a $g_k$0 component due to the surrogate anchor (Shah et al., 2016). SCSG achieves non-asymptotic complexity bounds in which the leading dependence can be governed by the heterogeneity quantity

$g_k$1

rather than by the dataset size $g_k$2, and the paper emphasizes that less than one pass over the data may suffice in the low-accuracy regime (Lei et al., 2016). Carathéodory-based methods preserve the full gradient exactly at recombination points and inherit convergence from an oracle-driven gradient scheme under convexity, twice differentiability, and Lipschitz continuity of the gradient (Cosentino et al., 2020).

In dynamic and online problems, the theory shifts from convergence to tracking. For drifting least-squares regression, the fOLS-GD method updates with one $g_k$3 stochastic-gradient step per new observation and tracks the moving ordinary least squares estimator at the $g_k$4 rate in both expectation and high probability under bounded features, bounded noise, and eventual strong convexity of the empirical covariance (Korda et al., 2013). For time-varying stochastic optimization with decision-dependent distributions, online projected gradient descent and its stochastic counterpart track performatively stable points under strong convexity, joint smoothness, Wasserstein sensitivity, and a contraction condition $g_k$5; with sub-Weibull gradient noise, the tracking error is ultimately bounded in expectation and in high probability (Wood et al., 2021).

Two recent lines extend the theory to particularly expensive control problems. Multilevel SGD for PDE-constrained optimal control proves linear convergence in the number of optimization steps,

$g_k$6

while giving MLMC complexity bounds that can improve on single-level Monte Carlo by avoiding full high-fidelity gradients at every step (Baumgarten et al., 3 Jun 2025). The least-squares control-variate method proves geometric decay to a plateau for a fixed approximation space and full convergence to zero with variable approximation spaces, with the precise rate tied to how well the parametric gradient map can be approximated in the chosen spaces (Nobile et al., 28 Jul 2025). For stochastic LQR, SGD with biased gradient oracles converges asymptotically to the optimal policy provided the bias decays at least as $g_k$7, $g_k$8, and the step sizes satisfy a Robbins–Monro-type summability regime (Song et al., 21 Feb 2026).

A separate theoretical axis concerns the source of stochasticity itself. The efficiency-ordering framework shows that when two stochastic input sequences have the same stationary distribution, the one with smaller asymptotic covariance in the MCMC sense yields smaller asymptotic covariance of scaled SGD iterate errors, in the Loewner order, for both raw and Polyak–Ruppert averaged iterates (Hu et al., 2022). This result is especially relevant when communication cost or privacy constraints force SGD to use random walks or shuffling rather than i.i.d. access.

5. Empirical domains and representative findings

In optimal trade execution, cost-aware SGD is used literally to optimize execution cost. The paper "Stochastic Gradient Descent in the Optimal Control of Execution Costs" studies the implementation shortfall problem under the Bertsimas–Lo permanent-impact model with information,

$g_k$9

and compares AdaGrad, RMSprop, Adam, and a custom adaptive SGD rule against the closed-form optimal benchmark $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$0. All SGD methods produced total execution costs very close to the optimum, but the period-by-period decisions had higher variance; under the reported settings, AdaGrad and the custom SGD rule performed best, while RMSprop and Adam produced identical strategies and performed worse (Kolev, 2024).

In federated learning with strategic participants, the cost of a gradient update is borne by agents rather than by a centralized optimizer. The federated mechanism-design paper models this cost as a strictly increasing, twice differentiable, convex function $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$1 of minibatch size, and proves that rewarding agents according to agreement with a peer-based reference induces a cooperative Nash equilibrium at the server’s requested minibatch size sequence while remaining budget balanced (Akbay et al., 2022). This formulation is aimed at settings in which organizations can undersample strategically to save cost and thereby silently increase gradient noise.

In scientific computing and PDE-constrained optimization, the dominant cost is often the PDE solve itself. Carathéodory Sampling reports runtime improvement up to about $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$2 over GD in logistic-regression synthetic experiments and stronger performance than ADAM and SAG in the reported large-scale LASSO experiments, especially when combined with block coordinate updates (Cosentino et al., 2020). The multilevel PDE-control method reports a speedup of more than $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$3 over standard batch estimation in gradient-norm reduction, and in the budget-aware variant BMLSGD obtains comparable results $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$4 faster with errors about $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$5 smaller for the same computational cost on the stated 2D elliptic diffusion problem (Baumgarten et al., 3 Jun 2025). The least-squares control-variate method reports faster convergence than SGD and Adam and better convergence rate and final error than SAGA in the reported 5D advection–diffusion experiments, while preserving the one-new-gradient-per-iteration structure (Nobile et al., 28 Jul 2025).

In graph learning and related structured-data settings, the main empirical benefit is reduced per-epoch cost. The consistent-gradient paper reports that smaller neighborhood sample sizes lead to faster epochs and that consistent-gradient SGD is faster than standard SGD and Adam on the cited graph datasets, while larger sample sizes move the optimization behavior closer to unbiased SGD (Chen et al., 2018). In online bandits and recommendation systems, replacing repeated exact least-squares solves by SGD yields an $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$6 per-update improvement and dramatic runtime reductions on the Yahoo! news recommendation data, while keeping click-through-rate performance about $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$7 close to that of regular LinUCB in the reported experiments (Korda et al., 2013).

In posterior sampling, the SGLD analysis offers a cautionary empirical message rather than a simple performance win. For a target posterior accuracy on the order of $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$8, the paper argues that naive subsampled SGLD does not obtain a real computational gain across batch sizes because the step size must be reduced accordingly; control variates can reduce cost substantially, but only if a good reference point such as the posterior mode is available (Nagapetyan et al., 2017). In decentralized optimization and swarm learning, asymptotic covariance experiments support the efficiency-ordering principle: non-backtracking random walks can outperform simple random walks, and shuffling can outperform i.i.d. sampling in the sense of smaller long-run SGD error covariance (Hu et al., 2022).

6. Limitations, misconceptions, and open directions

A central misconception is that cheaper stochastic gradients automatically lower total cost. The SGLD study shows that this is false when the target is statistically meaningful posterior accuracy: smaller batches force smaller step sizes, and the product of number of steps and cost per step remains roughly unchanged in the credible-interval regime (Nagapetyan et al., 2017). A related misconception is that greater algorithmic sophistication necessarily dominates simpler schemes. In execution-cost control, AdaGrad and the custom adaptive SGD rule outperformed RMSprop and Adam under the reported hyperparameters, even though all four methods optimized the same cost functional (Kolev, 2024). In the same spirit, the batch-size adaptation literature treats batch size as a dynamic optimization variable rather than a monotone knob to be increased whenever possible (Pirotta et al., 2017).

Another limitation is dependence on structural assumptions. CheapSVRG requires smooth strongly convex finite-sum structure for its clean linear theory and still converges only up to a residual neighborhood unless the surrogate error vanishes (Shah et al., 2016). Carathéodory Sampling depends on recombination, control-statistic design, and often block-coordinate structure to remain scalable in high dimensions; the paper explicitly points to exact second-derivative matching for quasi-Newton methods as future work (Cosentino et al., 2020). The federated reward mechanism assumes knowledge of the single-sample gradient variance $$n^*=\arg\max_{n\in\mathbb N^+}\frac{\Delta_L^n}{n},$$9; although an estimator based on peer gradients is proposed, the full game-theoretic guarantees for the estimated-variance version are deferred (Akbay et al., 2022).

Bias management is a persistent theme. The consistent-gradient framework demonstrates that unbiasedness is not indispensable, but its guarantees are probabilistic and depend on exponential-tail control of the relative gradient error as a function of sample size (Chen et al., 2018). The direct zeroth-order LQR policy-gradient method is conceptually simpler than the indirect identification-based method, yet asymptotically much more sample hungry because convergence to the exact optimum requires shrinking perturbations and growing rollout length and sample counts over time (Song et al., 21 Feb 2026). The least-squares control-variate method likewise trades lower variance for extra memory, QR updates, and approximation-space design (Nobile et al., 28 Jul 2025).

Open directions in the surveyed literature are therefore highly problem-dependent. They include improved hyperparameter tuning in execution-cost optimization, full strategic guarantees under estimated gradient-variance models in federated learning, second-derivative-preserving recombination in Carathéodory methods, and extension of efficiency-ordering theory from vanilla SGD to accelerated SGD and Adam, where only empirical evidence is currently reported (Kolev, 2024, Akbay et al., 2022, Cosentino et al., 2020, Hu et al., 2022). This suggests that cost-aware SGD is less a closed theory than a unifying viewpoint on how stochastic optimization should be redesigned when gradient information is expensive, strategic, structured, or physically constrained.