Online Annealing Pruning Techniques

Updated 14 July 2026

Online annealing pruning is a method that gradually transforms dense models into sparse ones by adjusting a control variable (e.g., temperature) during optimization.
It encompasses diverse techniques such as differentiable NAS, direct L0 sparsity control, and stochastic subnetwork hardening, each using specific annealing schedules.
Empirical studies in image classification, recommendation systems, and reinforcement learning demonstrate that annealing-based pruning improves convergence and model efficiency.

Online annealing pruning denotes a family of pruning and discrete-structure selection procedures in which sparsification is performed during optimization, or during online operation, under an annealed control variable rather than by a single post hoc hard prune. In the literature, the control variable may be a temperature, a time-varying sparsity budget, Bernoulli inclusion probabilities, a free-energy parameter, or an implicit penalty that progressively hardens masks. The term is therefore broader than a single algorithmic template: it spans differentiable neural architecture search, direct $L_0$ sparsity control, stochastic subnetwork hardening, temperature-annealed channel indicators, adaptive dense-to-sparse recommendation pipelines, and related online pruning formulations in reinforcement learning and system identification (Noy et al., 2019, Guo et al., 2020, Guan et al., 2020).

1. Scope and terminology

Across the cited work, two properties recur. First, pruning is online in the sense that it is interleaved with the main optimization or inference loop rather than delegated to a separate prune-and-retrain phase. Second, pruning is annealed in the sense that the induced sparsity pattern evolves gradually: a soft or probabilistic structure is made increasingly selective, or a keep-budget is tightened over time, until a discrete sparse model emerges.

The phrase is nevertheless used heterogeneously. In differentiable NAS, annealing sharpens a mixed operation toward a single surviving operator and prunes weak alternatives during the search itself (Noy et al., 2019). In direct sparsity control, annealing is an explicit schedule over the number of kept weights or channels, followed by hard projection onto an $L_0$ set (Guo et al., 2020). In stochastic subnetwork annealing, the sparse topology is not fixed during fine-tuning; binary masks are resampled every forward pass and their inclusion probabilities are annealed to deterministic $0/1$ values (Whitaker et al., 2024). In channel pruning, differentiable gates are annealed toward binarized indicators under bi-level optimization (Guan et al., 2020).

Related work also marks the boundary of the term. Some online pruning methods are gradual but do not use an explicit annealing schedule. A comprehensive gating framework prunes neurons, filters, layers, branches, and blocks during training via learnable scaling factors and thresholding, yet explicitly reports no temperature parameter or time-varying regularization schedule (Haider et al., 2020). Likewise, a recent large-foundation-model method performs per-token online structural pruning through a retained/candidate/pruned partition, but states that it uses no simulated annealing or annealing schedule (Chen et al., 6 Feb 2026). An arXiv record on multi-agent actor-critic for dynamic spectrum access reports a “harmonic annealing sparsity scheduler” integrated into an independent actor global critic sparse recurrent MARL framework, but the supplied material does not provide the exact schedule or experimental tables, so only that high-level characterization is available here (Stamatelis et al., 19 Mar 2025).

2. Canonical mechanisms and mathematical forms

The main technical variants can be organized by the object being annealed and by the criterion used to convert a soft structure into a sparse one.

Representative formulation	Annealed quantity	Pruned object
ASAP (Noy et al., 2019)	tempered softmax over candidate operations	mixed operations on cell edges
DSC (Guo et al., 2020)	keep-budget $M_e$ / sparsity schedule	weights or channels
Stochastic Subnetwork Annealing (Whitaker et al., 2024)	Bernoulli inclusion probabilities $p_{ij}(t)$ or temperature $\tau(t)$	individual weights
DAIS (Guan et al., 2020)	sigmoid gate temperature $T$	channels
D2S/AUX (Ye et al., 2020)	mask latents driven by penalty $\lambda$	fully connected weights

In ASAP, each edge in a cell is a mixed operation

$\tilde{o}^{(i,j)}(x; T)=\sum_{o\in\mathcal{O}} P_o(\alpha^{(i,j)};T)\,o(x),$

with

$P_o(\alpha^{(i,j)};T)=\frac{\exp(\alpha_o^{(i,j)}/T)}{\sum_{o'\in\mathcal{O}}\exp(\alpha_{o'}^{(i,j)}/T)}.$

A high temperature yields an almost uniform mixture, while annealing sharpens the distribution toward a nearly one-hot discrete choice; operations are pruned online when their annealed probability drops below a threshold, fixed as $L_0$ 0 in the practical schedule (Noy et al., 2019).

In direct sparsity control, the optimization is posed with an explicit cardinality constraint,

$L_0$ 1

or, for channels, with $L_0$ 2. Annealing is realized by a two-phase schedule over $L_0$ 3: a fast initial phase prunes to $L_0$ 4, and a second phase applies smaller periodic decrements until the final pruned fraction $L_0$ 5 is reached, after which the network is fine-tuned (Guo et al., 2020).

In stochastic subnetwork annealing, the sparse model is represented by stochastic binary masks,

$L_0$ 6

Two variants are used. Random Annealing drives each $L_0$ 7 monotonically to $L_0$ 8 or $L_0$ 9; Temperature Annealing keeps a target deterministic mask $0/1$0 but lets excluded weights participate transiently through

$0/1$1

with $0/1$2 annealed to $0/1$3 by a linear or cosine schedule (Whitaker et al., 2024).

In DAIS, binary channel indicators are relaxed to continuous gates

$0/1$4

and annealing is imposed by decreasing $0/1$5. The default schedule is

$0/1$6

which progressively closes the train–deploy gap between the continuous model used during search and the binarized model used after pruning (Guan et al., 2020).

The recommendation-system AUX method is distinctive in that it does not define an explicit sparsity schedule $0/1$7. Instead, it introduces a binary mask $0/1$8 for each weight and updates the latent variable $0/1$9 by a penalty-driven rule combining Taylor and magnitude criteria. The supplied description characterizes this as an “implicit annealing” effect: weights whose importance is too small relative to $M_e$ 0 are gradually pushed below zero and pruned (Ye et al., 2020).

3. Optimization structure, convergence, and pruning safety

A large part of the online annealing pruning literature is defined by its optimization structure. ASAP and DAIS are both bi-level procedures. ASAP uses alternating first-order updates, splitting data so that network weights are optimized on training loss and architecture weights on validation loss; no explicit architecture regularizer is required because annealing acts as an implicit sparsity driver (Noy et al., 2019). DAIS likewise places network weights in the lower-level problem and annealed indicator parameters in the upper-level problem, where the validation loss is augmented by FLOPs-aware, symmetry, and optional lasso regularization (Guan et al., 2020).

The strongest formal pruning-safety statement among the neural papers appears in ASAP. Under independence and bounded-loss assumptions, the theoretical schedule yields an $M_e$ 1-PAC guarantee that only suboptimal operations are pruned with high probability. The practical method does not use the conservative PAC schedule; it uses an exponential decay $M_e$ 2, with $M_e$ 3 and $M_e$ 4 on CIFAR-10, together with five grace epochs during which network weights are trained without pruning (Noy et al., 2019).

A different theoretical route appears in online deterministic annealing for switched-system identification. There, annealing is defined by a free-energy objective

$M_e$ 5

whose Gibbs-form soft assignments collapse to sharper partitions as $M_e$ 6 is lowered. New “effective codevectors” appear through bifurcation, and redundant codevectors are removed by the merging condition

$M_e$ 7

The method is explicitly two-timescale: slow online deterministic annealing estimates the mode-switching signal, and a faster recursion updates local model parameters (Mavridis et al., 2024).

For federated learning, annealing is treated as a special case of arbitrary adaptive online pruning. Each client trains a masked local model $M_e$ 8, where the mask can vary by client and by round. Under smoothness, bounded pruning-induced noise, and sufficient parameter coverage, the framework proves convergence to a stationary point of standard FL with rate $M_e$ 9. The analysis identifies two key quantities: the pruning-induced noise $p_{ij}(t)$ 0 and the minimum coverage index $p_{ij}(t)$ 1, directly linking convergence quality to how aggressively masks are annealed and how jointly they cover the parameter space (Zhou et al., 2022).

4. Neural architecture search and model compression results

In differentiable NAS, online annealing pruning was introduced to address the noncontinuous end-of-search hard prune used in DARTS. ASAP reports that online annealing and gradual pruning cut compute during search and stabilize convergence: with batch size 96, one search epoch averages 5.8 minutes on a single NVIDIA GTX 1080Ti, the full search takes 4.8 hours (about 0.2 GPU days), and the discovered architectures achieve CIFAR-10 test errors of 1.99% for ASAP-Small, 1.75% for ASAP-Medium, and 1.68% for ASAP-Large. The same search cell transfers to ImageNet at 24.4% Top-1 error with 5.7M parameters in the 600M FLOPs regime, and it is reported as best or tied best on CINIC-10, Freiburg, CIFAR-100, SVHN, and Fashion-MNIST among the compared NAS baselines (Noy et al., 2019).

For direct weight and channel pruning, DSC frames online annealing pruning as an $p_{ij}(t)$ 2-constrained optimization with direct sparsity control. On MNIST, DSC-1 reduces LeNet-300-100 from 267K parameters at 1.64% error to 17.4K parameters at 1.57% error, corresponding to 93.5% pruning; on LeNet-5 it reaches 15.8K parameters at 0.77% error, corresponding to 96.4% pruning. For channel pruning, the same framework reports CIFAR-10 VGG-16 at 6.14% error with 60% channels pruned and 4.40M parameters, and DenseNet-40 at 5.48% error with 60% channels pruned and 0.45M parameters; comparable or slightly better trade-offs are also reported on CIFAR-100 and SVHN (Guo et al., 2020).

DAIS extends the annealed paradigm to structured channel pruning under explicit computational budgets. On CIFAR-10, it reports 95.02% Top-1 for ResNet-110 with 60.0% FLOPs reduction and 93.53% for ResNet-56 with 70.9% FLOPs reduction. On ImageNet, it reports 74.45% Top-1 and 92.21% Top-5 for ResNet-50 at $p_{ij}(t)$ 3 FLOPs, corresponding to 55.3% reduction, and 72.77% Top-1 for ResNet-34 at $p_{ij}(t)$ 4 FLOPs, corresponding to 41.9% reduction. Mobile CPU measurements are also reported: the pruned ResNet-18 attains 0.19 s latency on a Galaxy S9, a 1.68× speedup relative to the dense baseline (Guan et al., 2020).

Stochastic subnetwork annealing addresses a different failure mode: the steep accuracy drop induced by pruning too many parameters at once. On ResNet-18 for CIFAR-100 with 20 fine-tuning epochs, the reported random-pruning accuracy at 98% sparsity improves from $p_{ij}(t)$ 5 for one-shot pruning to $p_{ij}(t)$ 6 for Random Annealing and $p_{ij}(t)$ 7 for Temperature Annealing under constant learning rate. With one-cycle learning rate, the one-shot baseline at 98% rises from $p_{ij}(t)$ 8 to $p_{ij}(t)$ 9 under Temperature Annealing. Under magnitude pruning and one-cycle learning rate, the same 98% setting improves from $\tau(t)$ 0 to $\tau(t)$ 1. The paper also reports smaller but consistent gains on ViTs, including 36.9 to 37.6 at 90% sparsity on Tiny ImageNet with Adam and one-cycle (Whitaker et al., 2024).

5. Online adaptation, nonstationarity, and decision-time pruning

The most explicit treatment of online pruning under distribution shift appears in large-scale recommendation systems. The adaptive dense-to-sparse paradigm maintains a dense model that tracks non-stationary data and periodically regenerates a sparse model from the latest dense snapshot. The proposed AUX mask update combines normalized Taylor and magnitude information, avoids a hand-crafted annealing schedule, and automatically learns different sparsity levels across layers. On the reported production-style setup, the last-window look-ahead relative CE and eval relative CE are 0.149 and 0.191 for iterative magnitude pruning, 0.157 and 0.184 for Taylor pruning, and 0.097 and 0.144 for AUX; AUX also reaches steady state after about $\tau(t)$ 2 samples, compared with about $\tau(t)$ 3 for magnitude pruning (Ye et al., 2020).

In deep reinforcement learning, XiNet inserts Bernoulli gates into OFENet feature extractors and SAC networks, trains weights and gate parameters jointly, and hardens the remaining gates during the last 20% of training. The method is explicitly cost-aware: regularization coefficients are matched to the expected parameter complexity of the DenseNet-style OFENet and the SAC submodules. On MuJoCo, the reported final returns show that large-and-prune configurations remain close to or above dense OFE baselines while drastically shrinking deployment models. On Walker2D, for example, OFE-Big reaches 5490, while Prune-A reaches 5679 with deployment/training parameter counts of 361K/2187K; on Hopper, OFE-Big reaches 3699, while Prune-B reaches 3515 with 15K/246K parameters; on HalfCheetah, OFE-Big reaches 17299, while Prune-B reaches 16033 with 28K/390K parameters (Guenter et al., 16 Jul 2025).

Online annealing pruning has also been extended from model parameters to training data. OPERA’s dynamic pruning strategy modulates sampling probabilities at both query and document levels through cosine-annealed strengths and cutoffs. Static Pruning improves ranking at the cost of retrieval coverage, whereas Dynamic Pruning maintains nonzero access to the full dataset and resolves that trade-off. Across eight datasets, the abstract reports that Static Pruning improves ranking over standard finetuning by NDCG@10 +0.5%, while Dynamic Pruning achieves NDCG@10 +1.9% and Recall@20 +0.7%, with comparable performance in less than 50% of the training time required by standard finetuning (Fang et al., 17 Mar 2026).

Decision-time pruning in large foundation models provides a contrasting form of online sparsification. POP partitions FFN channels into retained, candidate, and pruned regions during prefilling, then recomputes only the candidate set at each decoding step. The paper explicitly states that no annealing schedule is used, but the per-token mask update serves a function analogous to annealed adaptation. On Llama2-7B, POP reports 1.14× end-to-end speedup at 20% target pruning and 1.38× at 40%, with about 2.85% model-level FLOPs overhead and less than 4% FFN FLOPs overhead at the default candidate fraction $\tau(t)$ 4 (Chen et al., 6 Feb 2026).

A further extension replaces model compression by fairness repair. Attention Pruning uses surrogate simulated annealing to search over binary attention-head masks for pre-trained LLMs. Although the paper describes an “online phase,” the supplied description makes clear that this is an offline, pre-deployment optimization stage over surrogate models rather than streaming inference-time pruning. The reported search evaluates about 2940 states per second on CPU, yields a speed-up of about 2,260,000× over direct LLM evaluation, and reaches the largest relative gender-bias reduction on GPT-J-6B, from 0.446 to 0.264, or about 40.8% below baseline (Dasu et al., 20 Mar 2025).

Outside deep learning, online deterministic annealing in hybrid system identification exhibits pruning-like mode selection. In the reported PWARX experiment, progressive splitting yields $\tau(t)$ 5 effective codevectors but only $\tau(t)$ 6 consolidated modes; in the state-space piecewise-affine experiment, the same mechanism ends with $\tau(t)$ 7 effective codevectors and $\tau(t)$ 8 modes. Here pruning is not weight removal but the online merging and elimination of redundant components under an annealed free-energy objective (Mavridis et al., 2024).

6. Limitations, boundary conditions, and common misconceptions

A first limitation is hardware realism. Direct sparsity control explicitly notes that irregular non-structured sparsity may not translate into runtime gains unless specialized sparse kernels are available, whereas structured channel pruning yields immediate memory and inference-speed gains on standard hardware (Guo et al., 2020). This distinction helps explain why structurally constrained methods such as DAIS and POP emphasize channels and FFN widths rather than arbitrary unstructured masks (Guan et al., 2020, Chen et al., 6 Feb 2026).

A second limitation is schedule sensitivity. DSC warns that aggressive schedules can destabilize training and recommends a fast coarse phase followed by small $\tau(t)$ 9 increments; stochastic subnetwork annealing reports that too aggressive stochasticity or too few annealing epochs can harm convergence; DAIS shows that fixed-relaxation ablations underperform its annealed schedule, and its deep residual results depend strongly on symmetry regularization (Guo et al., 2020, Whitaker et al., 2024, Guan et al., 2020). These results collectively indicate that annealing is not a cosmetic addition: the rate at which pruning pressure hardens materially affects optimization.

A third issue is reversibility. ASAP and DSC use irreversible pruning within a run once an operation, weight, or channel has been removed (Noy et al., 2019, Guo et al., 2020). By contrast, DAIS explicitly reports recoverability during search, with channels pruned early later reactivated as optimization progresses, while stochastic subnetwork annealing delays any final hard commitment until the end of the annealing window (Guan et al., 2020, Whitaker et al., 2024). This distinction separates hard online pruning from probabilistic or relaxed variants that preserve a path for error correction.

The literature also supports several terminological corrections. “Online pruning” is not automatically “annealing pruning”: the learnable-scaling-factor framework is online and gradual but explicitly reports no annealing schedule (Haider et al., 2020). Conversely, an annealing-based method need not be online at inference time: the attention-head fairness repair method is driven by surrogate simulated annealing but produces a static final mask for deployment (Dasu et al., 20 Mar 2025). The federated-learning theory paper makes the broadest point: annealing schedules are only one subclass within arbitrary adaptive online pruning, and their behavior is constrained by pruning-induced noise and parameter coverage rather than by annealing alone (Zhou et al., 2022).

Taken together, these works present online annealing pruning not as a single recipe but as a general design principle: replace abrupt discrete sparsification by a controlled trajectory from dense or stochastic structures toward sparse deterministic ones, and couple that trajectory to the actual optimization or serving loop. The empirical record is strongest where the annealed variable is aligned with the deployment constraint—operations in NAS, channels in structured CNN pruning, probabilities in subnetwork tuning, per-round masks in federated learning, or per-step importance masks in autoregressive decoding—even though the precise meaning of “annealing” remains domain-dependent.