Dynamic Corpus-Aware Pruning Concepts

• The paper introduces dynamic corpus-aware pruning, which recalibrates importance metrics using live corpus data to optimize both model weights and training samples.
• This approach leverages gradients, Fisher information, and activation statistics to make adaptive, context-sensitive pruning decisions during training and inference.
• Empirical results demonstrate that dynamic pruning methods achieve high sparsity with minimal performance drop, enhancing domain adaptation for large models.

Dynamic corpus-aware pruning is a family of model and data compression techniques that dynamically leverage statistics, gradients, or activations computed over the current target corpus or input context to make heterogeneous, adaptive pruning decisions. Unlike static approaches that determine the pruning mask or retained data subset once—typically using fixed importance scores derived from either pre-trained weights or calibration sets—dynamic corpus-aware pruning adaptively recalibrates the pruning configuration in response to the specific domain, evolving input, or training trajectory. This paradigm is motivated by the observation that parameter and data importance can shift significantly under domain adaptation, fine-tuning, or even during inference, especially for large models such as LLMs or domain-specialized embedding models.

1. Motivation and Conceptual Foundations

Static pruning techniques historically assume that parameter, substructure, or data-point importance is fixed or can be properly evaluated in a one-shot calibration phase. However, when models are adapted to new domains or corpora, or when deployed in open-ended generative or retrieval settings, the critical subspace for maximal performance can drift over time. Two common failure modes are recognized:

• Pruning solely by general-domain importance can eliminate parameters specializing in domain-specific semantics, compromising downstream specialization.
• Pruning exclusively by domain importance risks removing parameters core to general-language or foundational capability, reducing transferability and overall robustness.

Dynamic corpus-aware pruning directly addresses these deficiencies by recomputing importance statistics, pruning masks, or sample selection weights in a corpus- or context-sensitive manner—often at regular intervals or triggered by detected context shifts. Mechanistically, "dynamic" refers to recomputation/refinement during adaptation, fine-tuning, training, or inference; "corpus-aware" emphasizes explicit conditioning on characteristics of the current domain data or input distribution (Tang et al., 13 Sep 2025, Lu et al., 2024, Tyagi et al., 30 Jan 2026).

2. Dynamic Corpus-Aware Pruning for Model Weights

Recent work has advanced several variants of dynamic corpus-aware pruning for model weights:

Gradient-Alignment and Fisher Importance: GAPrune

GAPrune introduces a principled hybrid scoring function, Domain Alignment Importance (DAI), integrating Fisher information estimated over both the general-domain and domain corpus, parameter magnitude, and domain-general gradient alignment. For each parameter $\theta_j$,

$$\text{DAI}(\theta_j) = \left( (F_{jj}^{dom} - \beta F_{jj}^{gen}) |\theta_j| + \gamma \sqrt{|\theta_j|} \right) (1 + \alpha s_g^j)$$

where $F_{jj}^\ast$ are diagonal Fisher information terms, and $s_g^j$ is the cosine-similarity between gradients on the general and domain corpora. Pruning decisions are recomputed dynamically on representative corpus samples and adaptively combine domain and general signals to retain parameters that are simultaneously relevant to domain-specialized as well as foundational semantic objectives. This preserves performance at high sparsity (≤2.5% drop at 50% sparsity, Qwen3-4B on FinMTEB/ChemTEB), outperforming magnitude and Fisher-only pruning. After one-shot pruning, brief retraining on the domain corpus enables recovery and further enhancement of performance (FinMTEB: +4.51%; ChemTEB: +1.73%) (Tang et al., 13 Sep 2025).

Dynamic Structural Masking during Fine-Tuning: ATP

ATP (All-in-One Tuning and Structural Pruning) co-optimizes a differentiable, trainable pruning-decision generator $\mathbf{G}(\mathbf{M})$ and domain-adapted LoRA weights throughout the fine-tuning trajectory. The pruning substructure is iteratively resampled at every training step, driven by current domain-loss and a group-structured sparsity constraint, ensuring that the pruned architecture flexibly adjusts to the evolving importance profile. After convergence, fixed groups are pruned, and adapter weights in deactivated groups are norm-shrunk via regularization. Experiments in legal and healthcare domains with LLaMA2/3 models demonstrate that ATP recovers 88–91% of dense performance at 40% sparsity, dramatically outperforming two-stage static approaches, with mask changes of up to 55% per layer relative to pruning frozen at pre-trained weights (Lu et al., 2024).

3. Dynamic Corpus-Aware Pruning in Inference and Contexts

Dynamic pruning is also employed at inference time to enhance efficiency and adaptability in real-time generative settings:

Context-Driven Pruning via Tracing and Mask Refresh: DART

DART (Dynamic Attention-Guided Runtime Tracing) periodically recomputes neuron-level binary masks for FFNs based on aggregated activation statistics over recent context windows (e.g., trailing $T=64$ tokens). A drift detector monitors real-time shifts in semantic context via attention centroid alignment, triggering mask recomputation when significant deviation is detected. Layers receive individually allocated sparsity budgets, reflecting heterogeneous knowledge density estimates. This adaptation enables DART to maintain near-dense performance at aggressive sparsity (≤70% FFN pruning), outperforming static pruning by up to 14.5% on LLAMA-3.1-8B (knowledge tasks) and realizing 3× higher ROUGE-L preservation on long-horizon summarization. The total additional memory is <10MB (LLAMA-3.1-8B), and the runtime FLOPs overhead is negligible (0.1%) (Tyagi et al., 30 Jan 2026).

Batch-Wise Probing and Structured Pruning: Probe Pruning

Probe Pruning (PP) addresses per-batch context adaptation during inference in LLMs. In each batch, a small probe (e.g., top 5% of samples × 50% of tokens by residual importance) is run through selected model blocks. Probe activations are fused with historical activation statistics (EMA), and these are used to compute channel-importance scores for pruning (PPsp) on the fly. Only the most important channels are retained for the remainder of the batch, with mask updates adapting as batch and sequence statistics evolve. PP achieves faster inference and substantially better trade-off in perplexity degradation per runtime saved compared to static and calibration-based methods (e.g., 2.56× lower PRR at 40% pruning, LLaMA-2-7B/WikiText2) (Le et al., 21 Feb 2025).

4. Dynamic Corpus-Aware Pruning for Training Data

While parameter pruning dominates model compression, analogous ideas are leveraged for dataset reduction:

Distributional Adaptive Sample Pruning: SCDP

Swift Cross-Dataset Pruning (SCDP) ranks dataset samples by their L₂ distance from the TF–IDF-based geometric median of the current corpus, yielding a simple proxy for underrepresented or "core" samples. A size-adaptive protocol—retaining farthest points for small datasets, stratified sampling for larger ones—efficiently balances diversity and coverage with minimal compute. Empirically, SCDP sustains or increases task accuracy across six NLU benchmarks (1–5 points improvement over random, up to 10× faster ranking than model-based baselines) and is robust across model families (Nguyen et al., 5 Jan 2025). A plausible implication is that online variants, tracking moving geometric medians and periodically pruning based on updated statistics, could enable continual learning or streaming data regimes to benefit from dynamic corpus-aware pruning.

Trajectory-Aware Dynamic Data Pruning

Dynamic data pruning treats data selection as an evolving decision process, re-evaluating sample utility at regular checkpoints based on model trajectory statistics: per-sample loss EMA, variance, and selection frequency. Reinforcement learning-inspired agents (e.g., $\epsilon$-greedy, UCB) can further exploit corpus-aware features (term frequency, semantic embeddings, cluster identity) and reward semantic diversity and coverage in the retained batch. Uniform random dynamic pruning already cycles through "sometimes" samples (∼60% of CIFAR-10), yielding superior accuracy at high pruning rates and up to 2× reduction in training time compared to traditional static methods (Raju et al., 2021).

5. Algorithmic Summaries and Workflow Patterns

The dynamic corpus-aware paradigm admits several characteristic workflow structures:

Technique	Pruning Target	Dynamic Trigger	Main Corpus Signal
GAPrune (Tang et al., 13 Sep 2025)	Model weights	Per-corpus rescore	Fisher, gradient alignment (domain vs. general)
ATP (Lu et al., 2024)	Model parameter groups	Every fine-tuning step	Joint mask optimization vs. calibration loss
DART (Tyagi et al., 30 Jan 2026)	FFN neurons (inference)	Attention shift/periodic	Contextual activation statistics
Probe Pruning (Le et al., 21 Feb 2025)	Channels (per batch)	Per-batch stats	Probed/intermediate residuals + history
SCDP (Nguyen et al., 5 Jan 2025)	Training samples	Batch or epoch	Distance-to-median in TF-IDF space
Dynamic data pruning (Raju et al., 2021)	Training samples	Checkpoint	Loss/loss variance trajectory + corpus meta

Adaptive thresholds (e.g., for gradient alignment, drift, diversity), corpus/statistics recomputation frequency, and the degree of structural granularity (per-weight, neuron, group, channel, sample) are typically chosen based on task and resource constraints.

6. Empirical Results and Impact

Dynamic corpus-aware pruning has consistently demonstrated improved trade-offs over static and global methods:

• On embedding models (Qwen3-4B, E5-mistral-7B), one-shot GAPrune (50% sparsity) incurs only 2–2.5% mean performance drop, while domain Fisher drops exceed 8% and general Fisher pruning degrades >30%. Retraining after pruning not only recovers but can surpass the dense baseline (+4.51% on FinMTEB) (Tang et al., 13 Sep 2025).
• For domain-specific LLMs, ATP outperforms LLM-Pruner and SliceGPT by 7-50% (relative performance) in legal/healthcare tasks at matching sparsity (Lu et al., 2024).
• At inference, DART achieves up to 14.5-point gains (accuracy) vs. static pruning on LLAMA-3.1-8B at 70% FFN sparsity, while preserving up to 95–100% of dense ROUGE on summarization (Tyagi et al., 30 Jan 2026).
• For data pruning, SCDP matches or exceeds random and coverage-based subsampling across GLUE-style tasks, with dramatically better compute efficiency (Nguyen et al., 5 Jan 2025). Dynamic data pruning methods halve wall-clock time at pruning rates up to 80% with <1% accuracy loss compared to dense training (Raju et al., 2021).

7. Limitations, Extensions, and Outlook

Key limitations include dependence on adequate corpus statistics or context windows to accurately estimate importance under drift, threshold sensitivity in context detectors, and untested generalization to generative or graph modalities (for data pruning) (Tyagi et al., 30 Jan 2026, Nguyen et al., 5 Jan 2025). Possible extensions include reinforcement-learned context drifters, joint attention and FFN dynamic pruning, dynamic tuning of diversity constraints, and hybridization with lightweight calibration.

Dynamic corpus-aware pruning is a unifying framework that adapts parameter, substructure, and data selection in real time, grounded in measurable characteristics of the target corpus or input context. It provides a principled method to align model and data reduction strategies with genuine shifts in semantic or statistical relevance, enabling scalable deployment and rapid adaptation of large models without sacrificing performance.