---
title: Stochastic Average Gradient (SAG)
url: https://www.emergentmind.com/topics/stochastic-average-gradient-sag
type: topic
---

# Stochastic Average Gradient (SAG)

The Stochastic Average Gradient (SAG) method is an incremental optimization algorithm designed to solve large-scale finite-sum problems involving smooth convex and strongly convex objective functions. SAG combines the low per-iteration cost of stochastic gradient (SG) approaches with the rapid convergence properties of full gradient (FG) methods by maintaining a memory of past gradient evaluations for each function component. This memory-based mechanism and the associated variance reduction enable SAG to achieve significantly improved convergence rates relative to vanilla stochastic methods, making it a foundational algorithm in modern large-scale optimization, particularly in machine learning and statistical inference contexts [1309.2388][1506.03662][2310.12771][1407.0202][1710.07783].

## 1. Problem Setting and Algorithmic Structure

SAG is designed for optimization problems represented as a finite sum of smooth convex (or strongly convex) functions:
$$
f(x) = \frac{1}{n} \sum_{i=1}^{n} f_i(x)
$$
where each $f_i : \mathbb{R}^p \rightarrow \mathbb{R}$ is convex, differentiable, and $L$-smooth (i.e., its gradient is $L$-Lipschitz continuous). SAG is particularly effective in the regime where $n$ is large and a full gradient computation is prohibitively expensive [1309.2388][1407.0202].

The core innovation of SAG is its use of a table of stored gradients $y_i^k \approx \nabla f_i(\cdot)$ for each component, which are incrementally updated as the algorithm progresses.

### Pseudocode for the Basic SAG Algorithm

- **Initialization**: Set $x^0$, $y_i^0 = 0$ for all $i = 1,\dots, n$, $d^0 = 0$.
- **Iteration** ($k = 1, 2, \dots$):
  1. Sample index $i_k \in \{1, \dots, n\}$ uniformly at random.
  2. Compute the new gradient $g_{i_k}^k = \nabla f_{i_k}(x^{k-1})$.
  3. Update running sum: $d^k = d^{k-1} - y_{i_k}^{k-1} + g_{i_k}^k$.
  4. Set $y_{i_k}^k = g_{i_k}^k$; for $i \neq i_k$, $y_i^k = y_i^{k-1}$.
  5. Update $x^k = x^{k-1} - \frac{\alpha}{n} d^k$ [1309.2388][2310.12771][1506.03662].

This update rule allows SAG to maintain an averaged, memory-improved estimate of the full gradient while incurring only $O(1)$ gradient evaluations and $O(p)$ vector operations per iteration.

## 2. Memory Mechanism and Variance Reduction

The principal distinction between SAG and classical SG methods lies in its memory array of historical gradients. Each $y_i^k$ stores the most recent evaluation of $\nabla f_i$ at some previous iterate, ensuring that over time, the stored values converge toward the gradients at the global optimum. The step at each iteration can be interpreted as an incremental aggregated gradient step, yielding a gradient estimate whose variance diminishes as more components are revisited and their stored gradients approach the optimum values [1309.2388][1506.03662][1710.07783].

For comparison:
- **SGD** computes the search direction using only the current sample gradient, discarding all past information. The resulting estimator has persistent variance, which necessitates a decaying stepsize and yields a sublinear convergence rate.
- **SAG** utilizes the running average of stored gradients as a control variate, achieving variance reduction and rapid convergence without requiring shrinking stepsizes [1506.03662][2310.12771].

## 3. Convergence Guarantees and Complexity

SAG provides sharp convergence rates for both convex and strongly convex objectives.

- **Convex Case ($\mu=0$):**
  $$
  \mathbb{E}\left[ f(\bar{x}^{k}) \right] - f(x^*) = O\left( \frac{n}{k} \right)
  $$
  where $\bar{x}^{k}$ is the average (or "best") iterate. This rate is $O(1/k)$ in terms of effective data passes, improving upon the typical $O(1/\sqrt{k})$ of SGD [1309.2388][2310.12771][1506.03662][1903.09009].

- **Strongly Convex Case ($\mu > 0$):**
  $$
  \mathbb{E}[f(x^k)] - f(x^*) \leq (1-\delta)^k C_0, \quad \delta = \min\{\mu/(16L), 1/(8n)\}
  $$
  yielding a linear (geometric) convergence rate [1309.2388][2602.05304][1407.0202].

The per-iteration cost is comparable to SGD ($O(d)$ for $d$-dimensional parameters), with memory requirements of $O(n d)$ to store the gradient table, and no need for decaying stepsizes [2310.12771][2602.05304][1506.03662].

### Comparison Table: Work Complexity and Convergence Rates

| Method      | Per-iteration cost | Storage         | Convergence Rate        |
|-------------|--------------------|-----------------|------------------------|
| SGD         | $O(d)$             | $O(d)$          | $O(1/\sqrt{k})$, $O(1/k)$ (strongly convex) |
| FG          | $O(nd)$            | $O(d)$          | $O(1/k)$, geometrical (strongly convex)     |
| SAG         | $O(d)$             | $O(nd)$         | $O(1/k)$, geometric (strongly convex)      |
| SAGA        | $O(d)$             | $O(nd)$         | Improved linear; unbiased                  |
| SVRG        | $O(d)$*            | $O(d)$          | Improved linear; requires periodic FG pass  |

*\*: Average per-iteration cost; periodic full-gradient passes incur $O(nd)$ cost [1506.03662][1407.0202].

## 4. Extensions, Variants, and Enhancements

Multiple enhancements and extensions of SAG have been proposed, addressing limitations and expanding applicability:

- **Non-uniform Sampling:** Sampling indices with probabilities proportional to local smoothness parameters ($L_i$) accelerates convergence by aligning sampling with the worst-case smoothness [1309.2388][1504.04406].
- **Mini-batching:** Grouping examples for parallelism and reducing storage, with convergence rates preserved under suitable stepsize adjustments [1309.2388].
- **SAG with Momentum / SAG+Adam:** Incorporating momentum or adaptive coordinate-wise scaling improves empirical performance on ill-conditioned or nonconvex landscapes; both hybridizations preserve low variance while enhancing optimization dynamics [2310.12771].
- **Proximal Acceleration:** The SAG update extends to composite minimization by including a proximal map for nonsmooth regularizers; convergence rates in this composite regime are established [1504.04406][1906.01133].
- **Compositional SAG (C-SAG):** For compositional finite-sum objectives, C-SAG maintains memory at both inner and outer function layers, preserving the oracle efficiency of SAG in more complex optimization topologies [1809.01225].
- **Stratified and Structured Variants:** SSAG pools gradients across stratified cohorts/classes, reducing the dimension of the memory and accelerating convergence when $C\ll N$ [1710.07783].

**Sufficient-Decrease Variants:** Recent work incorporates sufficient-decrease line search into SAG updates (SAG-SD), guaranteeing descent and adapting steps on the fly, with the same linear rates [1703.06807].

## 5. Bias, Unbiasedness, and Theoretical Developments

SAG's gradient estimator is *biased* in general, due to the averaging over a mixture of current and stale memory entries. Despite this bias, SAG attains provable linear rates; rigorous analysis shows bias vanishes as the stored gradients converge to the true gradients at the optimum [1906.01133][1903.09009].

Unbiased alternatives, such as SAGA, leverage a modified update that ensures the expectation coincides exactly with the true gradient at each iteration, resulting in marginally improved theoretical constants and facilitating easier extension to composite and non-strongly convex settings [1407.0202][1906.01133].

High-probability convergence results and unified proof frameworks have recently bridged the gap between biased (SAG) and unbiased (SAGA) estimators, supplying modular Lyapunov-based analyses and extending guarantees to regimes involving Markov sampling and non-convexity [2602.05304].

## 6. Applications and Empirical Observations

SAG and its variants have been widely applied in machine learning for large-scale empirical risk minimization, conditional random field training, neural network optimization, and structured prediction problems. Empirical studies show:

- Superior convergence speed relative to classic SGD, requiring 2–5× fewer data passes to reach high-accuracy regimes after a short warm-up [2310.12771][1506.03662].
- Robustness on ill-conditioned and nonconvex problems, especially with hybrid momentum or adaptive schemes [2310.12771][2310.12771].
- Storage cost $O(n d)$ can be limiting for deep learning with massive sample sizes, motivating structured memory reductions and stratified designs [1202.13212][1504.04406][1710.07783].

## 7. Limitations, Open Directions, and Comparative Analysis

The primary limitation of SAG is its $O(n d)$ memory footprint, a constraint for massive datasets or high-dimensional settings. The bias in its estimator, while vanishing asymptotically, may induce oscillations or slower transient convergence in early epochs. The step-size restriction, originally conservative ($\alpha < 1/(16L)$), has been relaxed in recent analyses to optimal order ($<1/(2L)$), aligning SAG with SAGA and improving practical stability [1903.09009][2602.05304].

Comparative studies show that SAGA generally achieves better constants and applies natively to composite problems, while SVRG (without memory) remains preferable in contexts with tight memory budgets and can be tailored for non-Euclidean geometries or infrequent storage update scenarios [1407.0202][1506.03662][1906.01133].

Advances in sufficient decrease techniques, stratified or compositional adaptions, and plug-in variance reduction mechanisms (e.g., SARAH, SARGE) continue to refine the theoretical and empirical landscape. Incorporating adaptive sampling, mini-batching, and structured memory layouts are active research areas aimed at extending the reach of SAG-type algorithms to ever larger and more structured machine learning regimes [1504.04406][1809.01225][1202.13212][1710.07783].

---

**References**: [1309.2388], [1506.03662], [2310.12771], [1407.0202], [1504.04406], [1710.07783], [1906.01133], [1809.01225], [2202.13212], [1903.09009], [2602.05304]

Source: https://www.emergentmind.com/topics/stochastic-average-gradient-sag