CAST: Continuous Adaptive Sparse Trainer

• CAST is a sparsity-aware retraining framework that dynamically updates binary masks to achieve 2:4 semi-structured sparsity while keeping forward propagation dense.
• It integrates three core components—AdamS, Weight Scaling, and Knowledge Distillation—to apply selective L1 decay and scale weights for robust sparse model recovery.
• Empirical results demonstrate that CAST recovers near-dense model quality with minimal perplexity loss and improved zero-shot accuracy, enabling significant speed and memory benefits on deployment hardware.

Continuous Adaptive Sparse Trainer (CAST) is a sparsity-aware retraining framework for transforming pretrained LLMs into hardware-friendly semi-structured sparse models, especially the $2\!:\!4$ format, while preserving language modeling quality and zero-shot performance. CAST is presented as a fully continuous and differentiable sparsity-aware training framework for semi-structured sparse models: it keeps the model dense during forward propagation throughout training, dynamically updates a binary $N\!:\!M$ mask from current weight magnitudes, selectively decays would-be-pruned weights toward zero, and applies the final hard mask only after training. Its three core components are AdamS, Weight Scaling, and Knowledge Distillation. In the reported LLaMA2-7B $2\!:\!4$ setting, an extended CAST run yields a sparse model with a perplexity increase of $0.09$ and a $0.36\%$ gain in zero-shot accuracy relative to the dense model, using $40$B retraining tokens, i.e. $2\%$ of the original $2$T-token pretraining budget (Huang et al., 30 Sep 2025).

1. Definition and problem setting

CAST targets semi-structured or $N\!:\!M$ sparsity. If a layer weight matrix is $W^k \in \mathbb{R}^{R_k \times C_k}$, each row is partitioned into contiguous groups

$N\!:\!M$0

and the mask $N\!:\!M$1 must satisfy

$N\!:\!M$2

The sparse weights are

$N\!:\!M$3

The main target is $N\!:\!M$4 sparsity, where exactly $N\!:\!M$5 of every $N\!:\!M$6 weights remain nonzero. The paper motivates this regime by its deployment relevance: $N\!:\!M$7 sparsity is natively supported on NVIDIA GPUs and can enable up to $N\!:\!M$8 speedup in both prefill and decoding under suitable kernels (Huang et al., 30 Sep 2025).

CAST is positioned against two limitations in prior LLM sparsification. First, many methods optimize masks and weights separately, as in one-shot pruning followed by optional retraining. Second, STE-based semi-structured retraining methods use a discrete sparse forward pass and surrogate backpropagation, which CAST argues introduces discontinuity and biased gradients. CAST instead keeps the forward model dense during optimization and uses the evolving semi-structured mask only to determine which parameters receive sparsity-inducing decay. This suggests a different notion of “continuous”: continuity is preserved at the level of model optimization, not through soft masks or hard-concrete gate parameterizations.

2. Semi-structured sparsification as dense training with dynamic masking

The defining mechanism of CAST is a repeated projection of each $N\!:\!M$9-weight group onto the top-$2\!:\!4$0-by-magnitude pattern, without using that hard mask in the forward graph. Every $2\!:\!4$1 iterations, CAST recomputes the mask. For each group, the second-largest absolute value $2\!:\!4$2 defines the binary decision rule

$2\!:\!4$3

This enforces an exact $2\!:\!4$4 pattern at every mask refresh. The final sparse model is then

$2\!:\!4$5

A common misconception is that CAST uses continuous masks, soft top-$2\!:\!4$6 relaxations, or STE-style sparse forward propagation. The paper states the opposite. CAST does not introduce mask logits, sigmoid gates, Gumbel relaxations, or straight-through estimators for the mask itself. Its “fully continuous and differentiable” characterization derives from the fact that the model remains dense during forward propagation throughout training, so the task loss is optimized over dense weights and scaling factors. The discrete $2\!:\!4$7 mask acts only as a selector for sparsity pressure.

This leads to a distinct training trajectory. Important weights keep receiving task gradients and remain among the top two magnitudes in their group. Unimportant weights recurrently lose the within-group competition and are selectively driven toward zero. By the end of training, the final hard projection is intended to be almost lossless because the suppressed coordinates are already negligible in magnitude.

3. Core components: AdamS, Weight Scaling, and Knowledge Distillation

CAST introduces three coupled components.

AdamS is a sparsity-inducing Adam-style optimizer. The paper begins from the standard lasso-style objective

$2\!:\!4$8

but argues that ordinary $2\!:\!4$9 decay is insufficient for semi-structured $0.09$0 optimization. AdamS applies selective $0.09$1-style decay only to currently masked parameters and linearly increases the relative weight of decay over training: $0.09$2 Its full update is

$0.09$3

$0.09$4

$0.09$5

$0.09$6

The distinguishing claim is that AdamS decouples the first-order moment from stale $0.09$7 sign accumulation while still using Adam-style adaptive scaling.

Weight Scaling compensates for magnitude shrinkage induced by repeated decay. For a matrix $0.09$8, CAST introduces trainable scaling factors $0.09$9 and applies row-wise or group-wise multiplicative rescaling. In the row-wise form,

$0.36\%$0

The more general implementation reshapes $0.36\%$1 into groups and uses $0.36\%$2, with $0.36\%$3 in experiments. Because scaling is multiplicative, it preserves the learned sparsity pattern. The factors are initialized to $0.36\%$4, trained jointly, and folded into weights before deployment, so the paper reports no inference-time overhead (Huang et al., 30 Sep 2025).

Knowledge Distillation uses the dense model as a self-teacher. CAST employs KL divergence on output distributions,

$0.36\%$5

and combines it with the language modeling objective as

$0.36\%$6

The paper reports that simple output-distribution KL performed better than more elaborate feature-level distillation, and that KD substantially improves retraining efficiency under constrained token budgets.

4. Training algorithm, optimization behavior, and common misconceptions

CAST begins from a pretrained dense checkpoint, computes an initial $0.36\%$7 mask, initializes the scaling parameters to ones, and then iterates a dense-forward sparse-aware retraining loop. At each step, if $0.36\%$8, the $0.36\%$9 mask is recomputed from current magnitudes. Forward propagation uses the scaled dense weights. Backpropagation computes gradients through the dense network. AdamS then applies selective decay to currently masked coordinates, with $40$0 gradually increasing sparsity pressure toward the end of training.

This schedule clarifies a second misconception: CAST is not a sparse-training method in the sense of sparse forward and backward execution throughout retraining. The paper explicitly states that training is still dense. The gain is a deployable sparse endpoint rather than a sparse compute path during retraining. Knowledge distillation also adds overhead; the paper estimates roughly $40$1 more FLOPs. CAST therefore trades retraining efficiency for final sparse-model quality and hardware compatibility (Huang et al., 30 Sep 2025).

The constrained optimization target is introduced as

$40$2

where $40$3 denotes the set of weights satisfying the $40$4 constraint. CAST operationalizes this objective indirectly through dynamic mask recomputation plus selective decay, rather than by solving a discrete constrained problem or by introducing soft mask variables.

5. Empirical results, scaling law, and deployment behavior

CAST is evaluated on GPT-2, OPT, LLaMA2, and LLaMA3, spanning $40$5M to $40$6B parameters, using strict $40$7 sparsity throughout the main experiments. The main retraining budget is often $40$8B tokens, with extended runs denoted $40$9. On LLaMA2-7B, the dense baseline reports WikiText perplexity $2\%$0 and average zero-shot accuracy $2\%$1. Standard CAST at $2\%$2B tokens yields $2\%$3 and $2\%$4, while $2\%$5 at $2\%$6B tokens yields $2\%$7 and $2\%$8, corresponding to a perplexity increase of $2\%$9 and a $2$0 zero-shot gain relative to dense (Huang et al., 30 Sep 2025).

Relative to prior sparse baselines on LLaMA2-7B under similar budgets, CAST improves over Wanda, MaskLLM, Naive Retraining, and SR-STE. The ablation results attribute nontrivial contributions to all three components: removing KD, replacing AdamS mask learning with a fixed mask, or removing Weight Scaling all reduce zero-shot accuracy. The paper further proposes a token-only empirical scaling law

$2$1

with $2$2, and reports $2$3 for LLaMA2-7B and 13B and $2$4 for LLaMA3-8B. The fitted law predicts that dense-level sparse recovery requires $2$5B retraining tokens for LLaMA2-7B, $2$6B for LLaMA2-13B, and $2$7B for LLaMA3-8B.

Deployment results are central to CAST’s rationale. Under TensorRT-LLM on NVIDIA hardware, the reported throughput speedups range from $2$8 to $2$9, with memory use reduced to about $N\!:\!M$0–$N\!:\!M$1 of dense. CAST also remains robust under quantization and downstream adaptation. On LLaMA2-7B, the sparse model at 3-bit AWQ reports perplexity $N\!:\!M$2, versus $N\!:\!M$3 for the dense 3-bit model. In LoRA fine-tuning on GSM8K, sparse LLaMA2-7B improves from $N\!:\!M$4 to $N\!:\!M$5, and sparse LLaMA2-13B improves from $N\!:\!M$6 to $N\!:\!M$7.

6. Relation to earlier sparse-training research and limitations

CAST occupies a distinct position within sparse training. “Growing Efficient Deep Networks by Structured Continuous Sparsification” optimized continuous structural gates over filters, layers, and recurrent units, but it targeted structured architecture adaptation rather than semi-structured $N\!:\!M$8 weight sparsity (Yuan et al., 2020). “Top-KAST: Top-K Always Sparse Training” maintained constant sparsity in both forward and backward passes via top-$N\!:\!M$9 reselection of a latent dense parameterization, whereas CAST keeps dense forward propagation and uses dynamic $W^k \in \mathbb{R}^{R_k \times C_k}$0 mask updates only to determine selective decay (Jayakumar et al., 2021). The most direct precursor is AST, which used SR-STE-based dynamic mask learning, knowledge distillation, and optional SLoRB for semi-structured LLM retraining; CAST replaces sparse-forward STE training with dense forward optimization, AdamS, and Weight Scaling (Huang et al., 2024).

This suggests a useful clarification of what CAST is and is not. It is not a weight-sparse training system that preserves sparse execution throughout retraining, and it is not a continuous-mask method based on differentiable gates. It is a dense retraining framework that progressively aligns a pretrained LLM with a deployable $W^k \in \mathbb{R}^{R_k \times C_k}$1 sparse endpoint.

Its limitations follow directly from that design. Training remains dense; KD increases retraining cost; the experimental focus is overwhelmingly on $W^k \in \mathbb{R}^{R_k \times C_k}$2; and the scaling law is explicitly empirical rather than fundamental. The paper also notes that models pretrained on stronger corpora, such as LLaMA3-8B, require substantially more sparse retraining tokens to fully recover perplexity. A plausible implication is that CAST is most attractive when the deployment objective is hardware-friendly semi-structured sparsity under limited but nontrivial retraining budgets, rather than maximal training-time sparsity or sparse-from-scratch optimization.