Sparse Memory Finetuning

• Sparse memory finetuning is an adaptive technique that leverages parameter sparsity via selective update and pruning to reduce resource usage.
• It uses methods like sparse memory layers, evolutionary connectivity, and gradient compression to achieve significant speed and memory gains.
• The approach also mitigates catastrophic forgetting in continual learning, ensuring robust adaptation in resource-constrained environments.

Sparse memory finetuning is an umbrella term for optimization techniques that exploit, induce, or leverage parameter sparsity to reduce the memory and computational requirements during the adaptation of large neural networks. These techniques span architectural strategies, optimizer-level interventions, and explicit update selection mechanisms, all focused on ensuring that only a carefully chosen, small subset of parameters—or memory—contributes to parameter updates during task adaptation. This approach is increasingly crucial for deploying and adapting large-scale models in environments with limited resources and for enabling continual learning without catastrophic interference.

1. Fundamental Principles of Sparse Memory Finetuning

Sparse memory finetuning is grounded in the observation that neural networks—particularly large transformers, LSTMs, and vision transformers—exhibit significant redundancy in both their parameter space and intermediate activations. Only a fraction of parameters is necessary to achieve high performance on a given task. This motivates fine-tuning strategies that:

• Update only a subset of parameters based on importance or usage statistics,
• Activate (and thus store or backpropagate through) only a sparse subset of memory or model components per input token or sample,
• Leverage architectural modifications, such as sparse memory layers or adapters, to localize task-specific adaptation,
• Employ optimizer or gradient projection techniques to restrict updates to compact subspaces or to a set of “winning ticket” subnetworks.

Sparsity can be either intrinsic (from the structure of the architecture, such as memory layers or MoE) or induced (via pruning, update masking, or gradient compression techniques).

2. Architectural Strategies for Inducing Memory Sparsity

2.1 Sparse Memory Layers and Hierarchies

Memory-based layers—such as those in SPARTAN and UltraMem—replace classical fully connected components with a large, sparse memory module. At each forward pass, only a small number of memory slots are activated by an input. For SPARTAN (Deshpande et al., 2022), memory consists of a two-level hierarchy:

• Parental cells are selected via input-dependent attention (top-K);
• Only associated child memory cells of these selected parents participate in computation;
• Backpropagation and adaptation are further limited to the active subset.

UltraMem (Huang et al., 19 Nov 2024) implements a 2D product-key memory, leveraging top-m sparsity in both row and column address spaces. Tucker-Decomposed Query-Key Retrieval (TDQKR) and Implicit Value Expansion (IVE) are used to control scaling and efficiently expand capacity, while only a handful of memory slots are touched per token, minimizing memory bandwidth and update propagation.

2.2 Evolutionary Sparse Connectivity

SET-LSTM (Liu et al., 2019) replaces dense LSTM gates and embeddings with sparse, Erdős–Rényi–initialized bipartite graphs. An evolutionary rewiring mechanism prunes low-importance connections epoch-wise and regrows randomly, adaptively sculpting a sparse subnetwork per task and yielding dramatic parameter savings with maintained (or improved) downstream performance.

2.3 Routed and Selectively Activated Subnets

SPT (Gui et al., 2023) introduces sparse multi-head attention, where only top-L attention scores are stored and used (selected via product quantization and SDDMM/SpMM), and routed FFNs, which dynamically select a subset of FFN parameters for activation based on a routing network, leading to both computational and memory savings.

Sparse-Tuning (Liu et al., 23 May 2024) applies token sparsification in vision transformers by preserving only the most semantically relevant tokens (as measured by [CLS] attention) and merges the rest into a fused representation. Dense Adapters aggregate information from multiple layers to counteract information loss due to aggressive token pruning.

3. Algorithmic and Optimizer-Level Approaches

3.1 Dynamic Sparse Finetuning and Update Selection

SpIEL (Ansell et al., 29 Jan 2024) adaptively maintains a sparse delta vector over a changing set of parameter indices. An iterative update–prune–regrow cycle ensures that only parameters with significant deltas (as measured by magnitude or momentum) are kept active. Regrowth can be driven by accumulated gradients (SpIEL-AG) or SM3-approximated momenta (SpIEL-MA), further reducing active memory and optimizer state.

Sparse Fine-tuning with Pruning (Li et al., 17 Feb 2025) (SPruFT) and SparseGrad (Chekalina et al., 9 Oct 2024) perform principled neuron or gradient selection. SPruFT leverages classical pruning metrics (magnitude, Taylor expansion, Quantile–Mean Taylor) to select “important” neurons for finetuning and restricts gradient flow accordingly, reducing both parameter and activation memory footprints. SparseGrad uses HOSVD to find bases where MLP layer gradients are highly sparse, updating only top-k elements per layer.

3.2 Gradient and Optimizer State Compression

Sparse Gradient Compression (SGC) (Yang et al., 1 Feb 2025) projects sparsified gradients (retaining s largest entries) into a low-dimensional space via a fixed projection matrix. Optimizer states are maintained entirely in this compressed space (dimension k ≪ d), with the original update vector recovered using compressed sensing (OMP). This decouples optimizer state memory from the full parameter dimension, allowing memory-efficient scaling to very large models.

Grass (Muhamed et al., 25 Jun 2024) projects gradients into a sparse, structured subspace—using “top-r” deterministic or stochastic row sampling—thereby shrinking not only optimizer state but also communication and computational cost. Projection matrices are designed to have exactly one nonzero per column, making the projection and recovery highly efficient.

3.3 Row-Based and Structured Sparsity

Block-diagonal and row-based methods (Lima et al., 13 Oct 2025, Li et al., 17 Feb 2025) employ explicit and structured matrix pruning (using Monarch factorization or neuron selection, respectively), which facilitates mapping and scheduling onto hardware accelerators such as compute-in-memory (CIM) arrays and allows for aggressive parallelization and memory reuse. Dense permutation folding and scheduling ensure that efficiency gains in memory storage are realized during both training and inference.

4. Sparsity-Driven Continual Learning and Catastrophic Forgetting Mitigation

Sparse parameter updates offer unique benefits for continual learning scenarios by localizing knowledge acquisition and reducing interference. Continual learning via sparse memory finetuning (Lin et al., 16 Oct 2025) demonstrates that selectively updating only those memory slots uniquely activated by new data—quantified via TF-IDF scores relative to pretraining usage—achieves task adaptation with substantially less forgetting compared to dense or LoRA-based finetuning. Pareto analyses show that this approach unlocks both high knowledge acquisition and knowledge retention, suggesting an important mechanism for practical continual adaptation.

5. Modality-Specific and System-Level Designs

5.1 Vision and Multimodal Applications

Sparse-Tuning (Liu et al., 23 May 2024) achieves compute and memory savings in vision transformers by sparsifying at the token (patch) level rather than only at the parameter level, enabling quadratic reductions in compute and memory during both training and inference on image and video tasks (e.g., VTAB-1K, Kinetics-400).

5.2 On-Device and Resource-Constrained Adaptation

Methods like PockEngine (Zhu et al., 2023) and MEFT (Hao et al., 7 Jun 2024) target efficient deployment on edge and low-memory systems by:

• Compiling static sparse computation graphs, pruning backward passes and leveraging sensitivity–cost tradeoffs;
• Offloading large sparse adapter parameters to CPU memory and limiting GPU transfers to highly activated routes (using MoE-like selection and FFN activation sparsity).

6. Quantization and Zeroth-Order Finetuning with Sparse Updates

Zeroth-order (ZO) optimization techniques combined with sparsity and quantization have further expanded the feasibility of memory-bounded finetuning. Sparse MeZO (Liu et al., 24 Feb 2024) and similar works (Guo et al., 5 Jun 2024) focus ZO updates on dynamically selected or precomputed “sensitive” parameter subsets (e.g., top 0.1% by gradient norm), leaving the rest quantized and frozen, enabling efficient finetuning on hardware with less than 8GiB memory without sacrificing performance. The updated ZO gradient estimate for a masked parameter set m is:

$$g_m(\theta) = \frac{\mathcal{L}(\theta + \epsilon (m \odot z)) - \mathcal{L}(\theta - \epsilon (m \odot z))}{2\epsilon}$$

where $\odot$ denotes elementwise multiplication and $\epsilon$ is the perturbation scale. Empirically, these methods achieve accuracy improvements and speedups over full ZO tuning, making LLM adaptation more broadly accessible.

7. Comparative Empirical Results and Applications

Extensive evaluations show that sparse memory finetuning, when implemented with the aforementioned techniques, consistently matches (and sometimes exceeds) dense/full finetuning and LoRA-style baselines across a variety of benchmarks and downstream tasks. Highlights include:

• SPARTAN (Deshpande et al., 2022) achieves a 90% inference speedup and ~89% storage reduction on edge hardware over adapter baselines, with slight improvements on GLUE.
• SPT (Gui et al., 2023) attains up to 50% peak memory reduction and ≥2× speedups versus full-parameter and LoRA tuning.
• SET-LSTM (Liu et al., 2019), in sentiment analysis, outperforms fully connected LSTM with <4% parameters, and maintains strong results up to 99% sparsity.
• Grass (Muhamed et al., 25 Jun 2024) enables 13B parameter LLaMA pretraining on a single 40GB GPU, with throughput up to 2× that of full-rank training.
• Sparse memory finetuning in continual learning (Lin et al., 16 Oct 2025) reduces catastrophic forgetting from 71–89% (LoRA/full finetuning) to 11% F1 drop, while acquiring equally or more new knowledge.

The convergence of structured sparsity, adaptive update selection, and architectural innovation is enabling scalable, resource-efficient, and robust model adaptation across tasks and hardware platforms.

---

Sparse memory finetuning is a rapidly evolving paradigm at the intersection of efficient adaptation, architectural compression, and continual learning. By shifting the locus of adaptation to compact, task-specific subspaces—whether realized through memory modules, neuron selection, structured projections, or dynamic masking—these techniques unlock model adaptability for next-generation, resource-constrained, and continually learning systems.