Test-Time Sparsity Overview

• Test-Time Sparsity is a set of techniques that reduce inference computation by imposing structured or unstructured sparsity on weights, activations, or attention mechanisms.
• It encompasses both static methods (like post-training pruning and fixed subnetwork selection) and dynamic approaches (such as activation masking and per-head routing) to optimize latency, memory, and robustness.
• Researchers balance efficiency and accuracy through adaptive routing and selective measurement allocation, while open challenges remain in aligning sparsity granularity with hardware support and continual adaptation.

Test-time sparsity is a family of inference-time and adaptation-time techniques that reduces the amount of computation, memory traffic, active parameters, activations, tokens, attention heads, or measurements used on a given input, device budget, or target stream. In recent work, the mechanism ranges from static post-training subnet selection and post-training sparsification to dynamic activation masking, per-head attention routing, token-level key–value selection, residual-block reuse in diffusion, and adaptive measurement allocation. The unifying objective is not singular: some methods target latency or memory, some target robustness-plasticity during continual adaptation, and some target exact interpretability or more powerful statistical inference under sparsity (Yang et al., 2021, Gong et al., 2024, Tang et al., 24 Jan 2026, Luo et al., 8 Oct 2025, Ji et al., 13 May 2026).

1. Operational regimes and objects of sparsification

The literature suggests that test-time sparsity is best understood by the object being sparsified and by whether the sparsity pattern is fixed or input-adaptive. Weight sparsity fixes a sparse parameterization before deployment or selects a subnetwork for a budget. Activation sparsity zeros portions of intermediate representations during adaptation or inference. Attention sparsity reduces the number of heads, tokens, or key–value entries participating in attention. In statistical settings, sparsity may instead refer to adaptively concentrating measurements on a sparse support, or to testing whether compressed observations admit sparse structure. In interpretable models, sparsity can be architectural, with each node touching only a small fixed subset of inputs at inference (Zhang et al., 18 May 2026, Ma et al., 26 Mar 2025, Soegeng et al., 8 Mar 2026, Malloy et al., 2012).

Paradigm	Sparse object	Reported role
Omni-sparsity DNN (Yang et al., 2021)	Structured 8×1 block-sparse encoder weights	Per-device streaming ASR subnetwork selection from one supernet
FCPTS (Gong et al., 2024)	Unstructured weight masks	Minutes-scale post-training sparsity with a predetermined global sparsity rate
MoASE / MoASE++ (Zhang et al., 2024, Zhang et al., 18 May 2026)	High- and low-activation token components	Continual test-time adaptation and robustness-plasticity control
SURGEON (Ma et al., 26 Mar 2025)	Layer-wise pruned activations	Memory reduction during fully test-time adaptation
Elastic Attention (Tang et al., 24 Jan 2026)	Head-wise full/sparse attention modes	Per-input adaptive sparsity ratios in long-context transformers
AsyncSpade (Luo et al., 8 Oct 2025)	Token-level KV selection	Asynchronous sparse decoding for test-time scaling
Action diffusion sparsity (Ji et al., 13 May 2026)	Residual block computations and cache reuse	Extreme fast action generation under rollout dynamics
TT-Sparse (Soegeng et al., 8 Mar 2026)	Hard TopK feature routing per node	Exact sparse forward passes and symbolic rule extraction

A central distinction is between static specialization and dynamic routing. Static methods produce masks that remain fixed at deployment; dynamic methods recompute sparsity decisions from the current input, query state, rollout state, or target-domain stream. A second distinction is between efficiency-oriented and robustness-oriented sparsity. Some methods explicitly state that sparsity is “not just for efficiency” but for disentangling domain-agnostic and domain-specific signals, whereas others treat sparsity primarily as a latency, memory, or FLOPs control mechanism (Zhang et al., 2024, Zhang et al., 18 May 2026).

2. Static specialization: sparse subnetworks, post-training pruning, and test-time scaling with pruned models

A prominent static regime trains or derives a sparse model once and then specializes it to a deployment budget without per-budget retraining. Omni-sparsity DNN does this with a weight-sharing supernet for streaming Emformer RNN-T automatic speech recognition. A subnetwork is defined by a layer-wise sparsity vector $s = [s_1,\ldots,s_L]$ with $s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$, and the encoder is pruned with structured 8×1 block-wise masks. The mask criterion uses Adam’s second-moment estimate,

$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$

and training minimizes

$$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$

After training once, an evolutionary search chooses a per-layer sparsity profile under a model-size budget, then the corresponding block-sparse encoder is materialized, quantized to 8-bit, and deployed with no retraining or fine-tuning. Reported LibriSpeech results span 50–80% encoder sparsity, with 2%–6.6% better WER on test-other than individually pruned sparse models at matched sparsity levels, while total training cost falls from 6912 or 8448 GPU-hours for individual sparse models to 2304 GPU-hours for the supernet; post-training evolutionary search costs about 50 GPU-hours for about 4000 subnetworks (Yang et al., 2021).

FCPTS addresses a different static regime: post-training sparsity from a trained dense model using a small unlabeled calibration set. It optimizes layer-wise thresholds rather than mask logits, combining a reconstruction term and a control term,

$$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$

and links thresholds to sparsity rates through a differentiable KDE-based bridge

$r_\ell = \int_{-t_\ell}^{t_\ell} p(w)\, d(w).$

The stated purpose is minutes-scale optimization with convergence to a predetermined global sparsity rate. Reported runtimes are 9 minutes for ResNet-32 on CIFAR-100 and 29 minutes for ResNet-18 on ImageNet. On ImageNet with ResNet-50 at 80% sparsity, FCPTS reports 71.26/91.23 top-1/top-5, versus 17.03/30.31 for POT (L2Norm), 22.92/42.13 for POT (ERK), and 39.47/70.97 for ProbMask. On Ambarella CV22, ResNet-18 latency falls from 18.194 ms dense to 7.599 ms at 70% sparsity, with memory dropping from 11.632 MB to 5.245 MB (Gong et al., 2024).

A third static regime appears in reasoning LLM pruning for test-time scaling. “Doing More With Less” studies unstructured pruning before inference and then applies standard thinking-token budgets $C \in \{512,1024,2048,4096,8192\}$. Global sparsity is

$s_{\mathrm{global}} = 1 - \|W\|_0/|W|,$

with masks chosen by one-shot Magnitude or Wanda criteria under Uniform, OWL, or LayerIF layer-wise allocations. Across s1.1-7B and Qwen3-8B on AIME24, AMC23, GPQA-Diamond, and MATH500, unstructured 10–20% sparsity matches or improves the full model, whereas structured layer removal with ShortGPT substantially degrades accuracy under long chain-of-thought. At 8192 tokens on Qwen3-8B with Uniform 20% sparsity, Magnitude reports 0.5889 on AIME24, 0.5690 on GPQA, 0.9167 on AMC23, and 0.9467 on MATH500; Wanda reports 0.5667, 0.5926, 0.9167, and 0.9533, respectively (Monjur et al., 28 Apr 2026).

3. Dynamic activation sparsity in continual and fully test-time adaptation

Activation sparsity becomes dynamic when masks are recomputed from the current batch or stream. In continual test-time adaptation, MoASE treats explicit sparsification as a way to separate domain-agnostic structure from domain-specific texture. Spatial Differentiate Dropout operates on feature tensors $\mathcal{F}\in \mathbb{R}^{b\times n\times d}$ by retaining either top-$K$ or bottom-$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$0 activations across the token-channel plane, with threshold

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$1

High-activation components are routed to domain-agnostic experts, low-activation components to domain-specific experts. Two gates modulate the mixture: the Domain-Aware Gate produces expert weights from low-activation features, and the Activation Sparsity Gate outputs per-expert thresholds

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$2

The method operates in an EMA teacher–student framework with a Homeostatic-Proximal regularizer,

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$3

Reported results show mean ImageNet-C error dropping from 55.8% for the source model to 40.5%, CIFAR10-C from 28.2% to 16.8%, CIFAR100-C from 35.4% to 25.8%, and Cityscapes→ACDC mean mIoU rising from 56.7% to 62.2%. The paper explicitly notes that practical runtime savings depend on exploiting the sparse mask in implementation, and that the reported PyTorch implementation prioritizes representational disentanglement rather than speed (Zhang et al., 2024).

MoASE++ extends this line by combining activation sparsity with Domain-Adaptive On-Policy Distillation. It retains the high-/low-activation decomposition, but adds input-adaptive keep-ratios

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$4

a Domain-Aware Router over low-activation tokens, and an EMA-anchored reverse-KL objective on the same on-policy augmented view,

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$5

Its stated goal is robustness-plasticity balance rather than acceleration. Reported means are 13.7 error on CIFAR10-C, 24.7 on CIFAR100-C, 39.7 on ImageNet-C, and 62.5 mIoU on Cityscapes→ACDC. The same paper states that MoASE FLOPs are about 677.31 GMac and MoASE++ about 850 GMac, so sparsity primarily filters noisy tokens and increases robustness while compute can increase due to mixture routing and distillation (Zhang et al., 18 May 2026).

SURGEON targets a different adaptation bottleneck: memory. It observes that cached activations dominate FTTA memory and defines per-layer Gradient Importance and Layer Activation Memory,

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$6

which are combined as

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$7

The retained sparse activations are cached as nonzero values plus an index map. On CIFAR10-C, SURGEON reports 18.1% mean online error with 325 MB cache, versus 18.9% and 3697 MB for Full-Tuning and 18.6% and 1762 MB for TENT; SURGEON with CSS+CR reaches 16.9% at 581 MB. On ImageNet-C with ResNet-50 AugMix, SURGEON reports 55.5% mean error and 1125 MB cache, and SURGEON (BN)+CSS+CR reports 53.9% at 1834 MB. On DeeplabV3+ for Cityscapes→ACDC, SURGEON improves over the source model while staying far below TENT and CoTTA in cache usage (Ma et al., 26 Mar 2025).

4. Adaptive attention and cache sparsity in long-context transformers

In long-context transformers, test-time sparsity often means that attention computation is reduced conditionally on the input instance rather than fixed a priori. Elastic Attention formalizes this with a per-head routing variable $s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$8 and two model-level sparsity metrics:

$s_\ell \in \{0.0, 0.5, 0.6, 0.7, 0.8\}$9

$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$0

A lightweight router consumes pooled key states, uses a Task MLP and Router MLP, and makes head-wise full-versus-sparse decisions with Gumbel-Softmax and a straight-through estimator. Training augments the language-model objective with a task-dependent sparsity penalty. Reported overhead is about 0.27M router parameters per layer for $I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$1 and about 0.196 ms router latency, stable from 512 to 1M tokens. Trained in 12 hours on 8×A800 GPUs, Elastic Attention is evaluated on Qwen3-4B, Qwen3-8B, and Llama-3.1-8B(-Instruct) across LongBench-E, RULER, and LongBench-V2. The paper reports, for example, FA–XA average 73.87 on RULER for Qwen3-8B with $I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$2, and up to 1.51× acceleration for FA–XA at long lengths, with XA–SSA reaching speedups up to 3.28× at extreme $I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$3 (Tang et al., 24 Jan 2026).

AsyncSpade addresses a different transformer bottleneck: query-aware sparse decoding under long chain-of-thought. Instead of waiting for the current query inside the autoregressive loop, it predicts the next-step query from a sliding window of recent queries using a training-free temporal-regressive module based on ridge regression and single-token shifting. The predicted query $I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$4 is then used to score all cached keys and select the top-$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$5 tokens per head. Crucially, cache filtering is moved off the critical path by a two-rank architecture: an Inference Rank runs the forward pass, while a Cache Rank predicts the next query, reorganizes token-level KV tensors, and communicates them back asynchronously. The resulting time-per-output-token is

$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$6

which attains the theoretical optimum when selection plus communication fit under forward compute. On A100 nodes, AsyncSpade reports over 20% TPOT reduction versus Quest and at least 50% TPOT reduction versus full attention on Qwen3-8B and Qwen3-32B, while matching or surpassing accuracy on AIME-24/25, GPQA-Diamond, and MATH-500. For Qwen3-8B, batch 8, context 32k, select 2k, pack 6 layers, the paper reports 7.13 ms inference and 6.42 ms cache latency on A100 (Luo et al., 8 Oct 2025).

5. Test-time sparsity in iterative generative inference

Diffusion policies expose a different failure mode for naive sparsification: the policy state changes across denoising steps and across rollout iterations. “Test-time Sparsity for Extreme Fast Action Diffusion” therefore defines test-time sparsity as a prune-then-reuse paradigm in which a lightweight pruner predicts, for each residual block and denoising step, whether to compute the block or reuse a cached feature. The pruner emits a four-way decision over compute, forward-direction reuse, timestep-direction reuse, and rollout-direction reuse:

$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$7

The sparse residual update is

$I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$8

To prevent the pruner itself from becoming the bottleneck, the method shares the encoder with the diffusion transformer, batches all denoising timesteps for pruning, and overlaps pruning with decoder inference asynchronously.

The reported motivation is that cached features from previous timesteps alone do not sufficiently constrain pruning errors under aggressive sparsity, while naive in-loop pruning offsets most speed gains. Omnidirectional reuse is therefore presented as the mechanism that stabilizes 95% sparsity. The paper reports 92% FLOPs reduction, 5× acceleration, lossless performance, and 47.5 Hz inference frequency. On Diffusion Policy with Proficient Human data at 95% sparsity, average success is reported as 84% with 5.18× speedup and 0.42 GFLOPS; on Kitchen multi-stage tasks, 93–95% sparsity yields 5.90×–6.33× speedup with about 99–100% success rate (Ji et al., 13 May 2026).

6. Statistical and interpretable variants of test-time sparsity

Outside neural efficiency, test-time sparsity also denotes selective use of measurements or inputs at inference. TT-Sparse makes this architectural. Each Learnable Truth Table node selects exactly $I_{\mathrm{Adam}}(w) = |w|\cdot \sqrt{\hat v_w},$9 inputs with a hard TopK mask and never touches the remaining $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$0 features at test time. Training uses a soft TopK relaxation with an exact cardinality constraint and straight-through estimation, but the forward pass remains sparse:

$$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$1

Inference cost is therefore $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$2 for $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$3 nodes, and the selected fan-in makes exact truth-table enumeration and Quine–McCluskey minimization practical. Across 28 datasets, the paper reports the best average rank for TT-Sparse on multiclass tasks and strong performance-complexity trade-offs on binary and regression problems (Soegeng et al., 8 Mar 2026).

Sequential support recovery gives a statistical version of test-time sparsity. “Sequential Testing for Sparse Recovery” studies coordinate-wise sequential sampling and shows an information-theoretic separation between adaptive and non-adaptive procedures. In the sparse regime, any coordinate-wise sequential method fails if the average number of measurements per coordinate grows more slowly than $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$4, while non-sequential procedures fail below $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$5. The SPRT attains the sequential bound when $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$6 and $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$7 are known, and Sequential Thresholding attains it under a monotone likelihood ratio assumption. The operative sparsity is therefore not a sparse network but a test-time measurement policy that abandons likely-null coordinates early and concentrates samples on the surviving candidates (Malloy et al., 2012).

A related property-testing line asks whether compressed observations admit sparse representations. For a known design matrix, “Testing Sparsity over Known and Unknown Bases” gives a tester that projects to $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$8 dimensions and accepts if $$L(\theta)=\mathbb{E}_{s\sim Q}\Big[\mathbb{E}_{(x,y)\sim D_{\mathrm{trn}}}\big[-\log p(y\mid x;\theta_s)\big]\Big].$$9 for some $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$0-sparse unit vector, while rejecting if $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$1 for any $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$2-sparse vector. For unknown $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$3, the projection dimension becomes $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$4 under a $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$5-restricted isometry assumption, and Gaussian width becomes the key test statistic (Barman et al., 2016).

The $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$6-test illustrates yet another use of sparsity at test time: exact finite-sample inference in the Gaussian linear model. It uses the absolute LASSO coefficient $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$7 as the statistic, calibrates it exactly under $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$8 by conditioning on a sufficient statistic, and exploits sparsity to obtain higher power and shorter confidence intervals without losing exact validity. The paper states that the resulting confidence intervals are more than 10% shorter than standard $$L_c = \left|\frac{\sum_i r_i N_i}{\sum_i N_i} - r_0\right|,\qquad L = L_{\mathrm{rec}} + L_c,$$9-test intervals, and that the two-sided $r_\ell = \int_{-t_\ell}^{t_\ell} p(w)\, d(w).$0-test can behave like a one-sided $r_\ell = \int_{-t_\ell}^{t_\ell} p(w)\, d(w).$1-test under sparsity even though no sign information is supplied (Sengupta et al., 2024).

7. Trade-offs, misconceptions, and open technical issues

A common misconception is that test-time sparsity is uniformly an efficiency method. Several papers explicitly contradict this. MoASE and MoASE++ use sparsity primarily to disentangle stable structure from malleable texture, and MoASE++ reports higher FLOPs than MoASE. SURGEON reduces memory rather than necessarily latency, and its dynamic importance-estimation pre-pass adds overhead. By contrast, Elastic Attention, AsyncSpade, and action-diffusion sparsity make efficiency claims contingent on routing quality, overlap conditions, and implementation details such as fused kernels, asynchronous communication, or reuse buffers (Zhang et al., 2024, Zhang et al., 18 May 2026, Ma et al., 26 Mar 2025, Tang et al., 24 Jan 2026, Luo et al., 8 Oct 2025, Ji et al., 13 May 2026).

Another misconception is that all sparsity patterns are interchangeable. The evidence in the supplied literature is strongly against this. Omni-sparsity emphasizes 8×1 block-wise pruning rather than unstructured single-weight pruning for fast edge inference. FCPTS is explicitly unstructured and notes that latency gains depend on hardware such as Ambarella CV22. In reasoning LLMs, modest unstructured pruning can preserve or improve test-time scaling, while structured removal of whole blocks harms long chain-of-thought. Elastic Attention similarly argues that static hybrid ratios are suboptimal because sparsity sensitivity differs across tasks, and AsyncSpade argues that page-level query-aware sparsity is limited by sequential dependence and coarse token selection (Yang et al., 2021, Gong et al., 2024, Monjur et al., 28 Apr 2026, Tang et al., 24 Jan 2026, Luo et al., 8 Oct 2025).

The main open technical tension is therefore not simply “more sparsity versus less sparsity,” but where, when, and at what granularity sparsity should be imposed. Static masks can be simple and deployable but may miss task- or input-specific sensitivity. Dynamic routing can adapt to the current regime but introduces risks of misrouting, unstable gradients, or negated speedups if selection overhead exceeds the saved compute. The literature also leaves several broader questions open: how to exploit sparse kernels consistently in activation-sparsity CTTA, how to align routing with KV compression or rollout reuse policies, how far unstructured pruning for reasoning LLMs can be pushed beyond 10–20% sparsity, and how to move from soft or straight-through relaxations to hardware-realized sparse execution without losing the reported accuracy or stability (Tang et al., 24 Jan 2026, Gong et al., 2024, Monjur et al., 28 Apr 2026, Luo et al., 8 Oct 2025).