MPFT: Masking-Based Pre-trained Fine-Tuning

• MPFT is a parameter-efficient technique that employs binary or continuous masks to select specific pre-trained parameters, significantly reducing fine-tuning overhead.
• It enables low-latency, storage-conscious deployment by freezing most weights and adapting only a targeted subset across various domains.
• Empirical studies show that MPFT achieves comparable accuracy to full fine-tuning while dramatically lowering computational and communication costs.

Masking-based Pre-trained Model Fine-Tuning (MPFT) is a collective term for a family of parameter-efficient adaptation techniques for large-scale neural networks, unified by their reliance on binary or continuous masks to select or gate a subset of pre-trained parameters for downstream learning. Instead of updating the entire parameter set as in standard fine-tuning, MPFT either learns which weights to tune (masking selection) or exploits structurally-induced sparsity to restrict and regularize adaptation, thereby achieving high accuracy with dramatically reduced storage, computational, or communication overhead. MPFT has emerged as a core paradigm across language, vision, audio, and multimodal domains, and underpins recent advances in federated learning, memory- and latency-constrained inference, and robust generalization under distribution shift.

1. Foundations and Key Mechanisms

MPFT operates by freezing the majority or entirety of a pre-trained model's weights $\Theta^{(0)}$ and introducing a mask $m \in \{0, 1\}^D$ or $m \in [0, 1]^D$ (hard/binary or soft/continuous) that gates parameter updates, model activations, or connectivity. The mask can be learned or statically selected, applied globally or group-wise (e.g., per-layer, block, expert, or matrix partition), and can impact weights, biases, or task-specific heads. Two primary classes of MPFT have crystallized:

1. Sparse Fine-Tuning via Masking: Given a frozen $\Theta^{(0)}$, a mask $m$ selects a (typically small) subset of parameters for tuning, with the remainder fixed. Training updates only the positions where $m_i = 1$ (Liao et al., 2023), and storage requirements are limited to the trained subset and the indices or mask.
2. Structural Masking for Gated Subnetwork Discovery: Rather than modifying weights directly, a trainable mask score $S$ (per weight or block) defines a gating mask $M$ via thresholding or activation (e.g. $M_{ij} = \mathbb{I}\{ |S_{ij}| \geq \tau_k \}$ or $M_{ij} = \sigma(S_{ij}/T)$), yielding a masked model $W' = W \odot M$ (Zhang et al., 28 Dec 2025, Zhang et al., 27 Mar 2025).

Architectural integration varies: MPFT can be applied directly to linear layers (as in vision transformers or FFN modules in LLMs), to adapter matrices (e.g. Masked LoRA updates (Wang et al., 2024)), to binary gating in self-supervised clustering (Self-Masking Networks (Warmerdam et al., 2024)), and to expert partitions (Masked LoRA Experts (Wang et al., 2024)). The central unifying motif is compressive, interpretable adaptation via mask learning—sacrificing none, or only tiny fractions, of accuracy per task while yielding superlinear savings in memory, compute, or communication bytes.

2. Algorithmic Strategies for Mask Selection and Learning

Multiple algorithmic routes enable MPFT, depending on the desired level of task-specificity, statistical grounding, and practical constraints:

• Magnitude-based Masking (PaFi): Selects parameters with the smallest absolute values (bottom-$k$ by $|\Theta^{(0)}|$) for fine-tuning, either globally or per-group (Liao et al., 2023). The mask is computed once and reused across tasks, requiring only seconds at initialization and no downstream data—enabling task-agnostic or federated regimes.
• Random Masking: Samples a random binary mask $M_j \sim \text{Bernoulli}(p)$ for each coordinate (Xu et al., 2024). Fine-tuning proceeds solely on the unmasked subset, enabling surprising matches to LoRA and full fine-tuning at $p \leq 0.1\%$, contingent on aggressive learning rate scaling.
• Expert-Level Masking in LoRA: Decomposes a low-rank update into $r$ rank-1 “experts,” applies binary dropout per expert to maximize independence and diversity, and adaptively scales surviving experts (Wang et al., 2024).
• Mask Learning via Straight-Through Estimator (STE): Mask scores are trained by passing the downstream loss gradient directly through the non-differentiable mask threshold or top-$k$ selection (Zhang et al., 27 Mar 2025, Zhang et al., 28 Dec 2025, Zhao et al., 2020). This yields binary or soft masks without high-overhead continuous relaxation (e.g. Gumbel-Softmax is generally not used).
• Probabilistic/Stochastic Masking for Federated Learning: Per-client mask probabilities $\theta_i = \sigma(s_i)$ induce Bernoulli masks, which are communicated via compressed difference-encoding with beta-aggregation on the server, achieving ultra-low bit-per-parameter communication (Tsouvalas et al., 2023).
• Gradient-Sign Masking for Task Vector Transport: To rebase a source task vector $\Delta \theta_s$ onto a target model $\theta_t$, a mask is derived from few-shot gradient sign votes at the target, filtering the transported delta for local loss descent (Rinaldi et al., 7 Oct 2025).
• Mask-Aware Adaptation in Zero-Shot Segmentation/Robustness: Structural masking (CAM-driven, counterfactual patch masking or mask-aware token handling) is used to break spurious correlations or drive improved proposal sensitivity within frozen vision-language backbones (Xiao et al., 2023, Jiao et al., 2023).

3. Practical Deployment, Storage, and Zero-Latency Adaptation

A defining feature of MPFT is its capacity for low-latency, low-memory, and communication-efficient deployment. Representative practical aspects include:

• Storage Efficiency: Binary masks require only 1 additional bit per parameter per task. For multi-task NLP (BERT/RoBERTa), this reduces memory by up to 32×, enabling massively-multitask scenarios with a fixed backbone and per-task masks (e.g., 13 MB per task vs 438 MB for full fine-tuned copies) (Zhao et al., 2020). In vision, similar ratios appear with CLIP, ViT, and ResNet backbones (Warmerdam et al., 2024).
• Inference Latency: When the mask is static (binary), and adaptation is achieved by zeroing or gating weights, inference is unchanged relative to the original backbone. Adapter-augmented masking (HiWi) merges nonlinear updates into the frozen weights pre-inference, yielding identical computational footprints with no runtime overhead (Liao et al., 2023).
• Federated and Distributed Learning: MPFT masking enables federated adaptation with mask-communication as low as 0.09–0.25 bpp (DeltaMask) (Tsouvalas et al., 2023), far under gradient compression baselines and with negligible aggregation overhead.
• Interpretability and Structured Regularization: MPFT masks illuminate which parameters are essential per task, facilitating analysis of redundancy, transferability, and subnetwork connectivity. For instance, MFT in VLMs reveals highly prunable projections (Key/Query) and supplies task-specific “winning subnets” (Zhang et al., 28 Dec 2025).

4. Comparative Performance and Ablation Studies

Quantitative experiments across NLP, vision, audio, and multimodal tasks consistently demonstrate that MPFT can match or exceed full fine-tuning and structured adapter methods in accuracy, while using only a tiny fraction of parameters:

Method	% Tuned	% Stored	GLUE Avg	VariousGLUE Avg	BLEU	Storage Notes
Full Fine-Tune	100	100	88.0	88.0	37.3	No overhead
Diff Pruning	0.5	0.5	87.9	87.9	—	Mask per task
PaFi (MPFT)	0.5	0.5	89.3	86.8	37.7	Mask shared, min. compute
HiWi (bias)	0.5	0.03	—	87.3	—	Zero latency, minimal storage
HiWi (ffn-weight)	2.0	0.03	—	88.2	36.9–38.3	Zero latency
LoRA	0.5	0.5	88.6	87.2	—	Foldable
Random Masking	0.1	0.1	—	—	—	Matches LoRA at 1/10th params (Xu et al., 2024)

Ablations in (Liao et al., 2023, Wang et al., 2024, Warmerdam et al., 2024, Zhang et al., 28 Dec 2025) identify key mask selection mechanisms (bottom-$k$ versus top-$k$ magnitude, group-wise versus global, structured versus random), optimal ratios ($p$ values for sparse tuning, expert counts for LoRA extensions), and trade-offs between diversity and independence of mask components. Learning rate scales logarithmically with decreasing sparsity, and correct tuning is critical for optimization convergence.

5. Extensions Across Modalities and Tasks

MPFT paradigms have proliferated in:

• Language: Binary mask learning over frozen BERT/RoBERTa weights yields task-specific subnetworks (Zhao et al., 2020), selective masking based on task-relevant word lists and scores improves transfer (Lad et al., 2022), and pragmatic masking bridges social meaning tasks (Zhang et al., 2021).
• Vision: Masked sub-branch strategies regularize supervised and fine-tuning recipes (MaskSub) for ViT, ResNet, CLIP, Swin (Heo et al., 2023); counterfactual masked image generation for causal OOD robustness (Xiao et al., 2023); permutation-masked token pre-training (MaPeT) closes fine-tune/distribution gaps (Baraldi et al., 2023); and mask-aware adaptation sensitizes CLIP to proposal regions for zero-shot segmentation (Jiao et al., 2023).
• Multimodal: Structural mask learning in VLMs reorganizes existing knowledge without weight drift, yielding higher accuracy than LoRA or full fine-tuning (Zhang et al., 28 Dec 2025).
• Speech/Audio: Biased masked prediction pre-training and streaming variants synchronize self-supervised labels and task requirements, reducing WER by 15–44% over unbiased baselines (Kreyssig et al., 2022).
• Continuous/Probabilistic Masking: Bernoulli mask learning and Bayesian aggregation enable scalable federated fine-tuning over distributed clients (Tsouvalas et al., 2023).

6. Theoretical Guarantees, Mode Connectivity, and Generalization Bounds

Theoretical analyses in (Rinaldi et al., 7 Oct 2025, Zhao et al., 2020, Xu et al., 2024, Zhang et al., 28 Dec 2025) establish:

• First-Order Descent: Gradient-sign masking in “task vector transport” across releases provably yields loss descent under local Taylor approximation, and empirical results consistently outperform naive task vector transfer and few-shot target tuning (Rinaldi et al., 7 Oct 2025).
• Mode Connectivity: Masked and full fine-tuned minima reside in connected regions of the loss landscape, with linear and Bezier curve interpolation maintaining constant accuracy, suggesting intrinsic redundancy and robust validity of MPFT minima (Zhao et al., 2020).
• PAC-Bayes Generalization: Structural mask learning reduces the complexity bound $C(h_{MFT})$ relative to full fine-tuning when task sparsity is moderate ($1\%$–$3\%$ active), yielding strictly tighter generalization upper bounds (Zhang et al., 28 Dec 2025).

7. Limitations, Open Directions, and Best Practices

While MPFT excels in memory, latency, and mult-task deployment, several practical considerations remain:

• Mask and sparsity selection must be tuned per task or domain; adaptive or learnable mask strategies may further reduce redundancy.
• Unstructured binary masking does not inherently accelerate inference; structured block-masking or hardware-aligned pruning should be investigated for runtime gains.
• The relationship of MPFT to intrinsic model dimension and generalization plateau in extreme-sparsity regimes remains open for study.
• Best practices include group-wise bottom-$k$ magnitude selection, aggressive learning rate scaling in sparser settings, modular mask storage, and interpretability analysis over mask patterns.

In sum, Masking-based Pre-trained Model Fine-Tuning constitutes a principled, theoretically-sound, and empirically-validated paradigm for highly parameter-efficient, low-latency, and robust adaptation of large-scale models, offering practical recipes and unifying structural insights across modalities and domains (Liao et al., 2023, Zhang et al., 28 Dec 2025, Xu et al., 2024, Warmerdam et al., 2024, Zhang et al., 27 Mar 2025, Wang et al., 2024, Zhao et al., 2020, Tsouvalas et al., 2023, Rinaldi et al., 7 Oct 2025).