Target-Specific Fine-Tuning (TSFT)

• Target-Specific Fine-Tuning (TSFT) is a method that adapts large pre-trained models by updating only task-relevant parameters and token representations.
• It leverages the Fisher Information Matrix and attention-based token scoring to identify and update crucial model components for efficient task adaptation.
• Empirical results demonstrate that TSFT achieves higher accuracy with minimal parameter updates and lower resource usage compared to full fine-tuning and generic PEFT methods.

Target-Specific Fine-Tuning (TSFT) is a paradigm that adapts large pre-trained models to downstream tasks by selectively updating only those components—parameters and token representations—that are most relevant to the target task. TSFT contrasts with traditional full fine-tuning, which incurs prohibitive computational and memory costs by updating all model parameters indiscriminately, and with task-agnostic parameter-efficient fine-tuning (PEFT) mechanisms that process the same subset of parameters for all tasks. The TSFT approach advocates for targeted updates at both the parameter and token representation levels, thereby maximizing efficiency while often achieving or surpassing full fine-tuning accuracy across diverse vision benchmarks and model architectures (Luo et al., 30 Jul 2025).

1. Concept and Motivation

TSFT formalizes adaptation as the identification and update of only those model elements—parameters and token sequences—that concentrate the task’s discriminative signal. In architectures such as Vision Transformers (ViTs) or BERT, TSFT seeks a minimal, task-relevant subset Θ_T of the parameters θ and a task-refined set X_ref of tokens:

• Only the most informative parameters for the target are updated; all others remain frozen.
• Input sequences are dynamically pruned to retain only those tokens regarded (by internal attention mechanisms) as salient for the target decision. This perspective is grounded in empirical findings that different tasks rely on distinct parts of the network and input space, a fact that is neglected by global or uniform fine-tuning (Luo et al., 30 Jul 2025).

This targeted approach is especially consequential in large models (hundreds of millions or billions of parameters), where updating every parameter is not scalable and can result in unnecessary computation on irrelevant features. Task-agnostic strategies such as LoRA, Adapters, and prompt tuning select the same parameter or prompt subspaces across all tasks without regard to the actual distribution of task-relevant information, yielding suboptimal adaptation both in efficiency and in downstream accuracy.

2. Task-Relevant Parameter Selection (TR-PS)

TR-PS is the method by which TSFT identifies, on a per-task basis, a sparse subset of model parameters whose adaptation is maximally beneficial to the target task.

Fisher Information Matrix (FIM) for Parameter Importance

The Fisher Information Matrix (FIM) quantifies the sensitivity of the model’s output distribution $p(y|x;\theta)$ to perturbations in each parameter $\theta$, providing a principled framework for importance estimation.

$$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$

For classification models with cross-entropy loss $L_{CE}$, the diagonal of $F$ is approximated by the expected squared gradient: $$F_{ii} \approx \mathbb{E}_{(x,y)\sim D} [( \partial L_{CE}(x, y; \theta) / \partial \theta_i )^2 ]$$

Importance scores $S_i$ for each parameter are computed by accumulating $g_i^2$ (gradient squared), typically over a short “probe” pass with a small subset of target data.

Layer-wise Selection and Freezing

Parameters are ranked by $S_i$ within each layer. Globally, the top $M\%$ by $\theta$0 are selected to form the task-relevant set $\theta$1. For each layer $\theta$2, the fraction $\theta$3 of selected top-$\theta$4 parameters is normalized such that each neuron retains at least one updated connection. Subsequently, for each neuron, connections are sorted by FIM score, and only the top $\theta$5 are updated:

• All other parameters are frozen.
• The binary mask $\theta$6 over $\theta$7 encodes the task-relevant subset $\theta$8.

During fine-tuning, only the active subset is updated: $\theta$9

3. Task-Relevant Token Selection (TR-TS)

TSFT further optimizes efficiency by dynamically selecting tokens that carry discriminative content for the target.

Attention-Based Token Scoring

Within each Transformer layer, attention scores from the special [CLS] token aggregate information about the contribution of each input token. The attention of token $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$0 to [CLS] is computed as: $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$1 where $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$2 and $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$3 are the query and key vectors.

Larger $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$4 indicates greater task relevance. For a given retention rate $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$5, the top $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$6 tokens according to $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$7 are retained, and remaining tokens are merged into a single aggregate: $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$8 Thus, the input sequence at each layer evolves as $$F(\theta) = \mathbb{E}_{x,y \sim D} [ \nabla_{\theta} \log p(y|x;\theta)\;\nabla_{\theta} \log p(y|x;\theta)^T ]$$9.

Dynamic Pruning and Merging Workflow

At each application layer:

• Compute $L_{CE}$0 for all tokens.
• Select top $L_{CE}$1 tokens and merge the rest as described.
• Pass the refined sequence to the next layer.

This procedure allows early layers to preserve local information, while deeper layers concentrate computational resources on the most discriminative patches, yielding a balance between representational richness and computational budget.

4. Joint Optimization and Objective

TSFT optimizes a cross-entropy objective over the pruned token sequence and masked parameter set: $L_{CE}$2 subject to updating only $L_{CE}$3.

During backpropagation, a binary mask over parameters propagates gradients solely through the task-relevant set. Token selection operates online, without introducing auxiliary losses; thus, the adaptation of tokens and parameters is co-evolved throughout fine-tuning, focusing the model’s capacity on the most informative dimensions of both parameter and token space.

5. Empirical Findings and Computational Benefits

TSFT, as operationalized by the TR-PTS framework, yields substantial gains in both accuracy and efficiency across a range of vision benchmarks:

• On fine-grained visual classification (FGVC; 5 datasets), TSFT achieves 91.94% average accuracy, surpassing full fine-tuning (88.54%) and prior task-agnostic PEFT (91.78%), while updating only 0.60% of parameters.
• On the VTAB-1k suite (19 tasks), TSFT attains a mean of 75.92% versus 65.57% for full fine-tuning and 75.18% for GPS, with only 0.34% of parameters updated.

Resource usage is similarly reduced:

• FLOPs: 12G (TR-PTS) vs. 16G (full fine-tune), with comparable or lower inference time and memory consumption.
• Inference time per batch (ViT-B/16, 224×224, batch=32): 25ms (TR-PTS) vs. 30ms+ for dense PEFT baselines.

These gains result from prioritizing learning in the most informative subspaces of both parameters and tokens, which, in aggregate, outperforms global adaptation both in discriminative accuracy and resource requirements (Luo et al., 30 Jul 2025).

6. Analysis, Limitations, and Extensions

The core effectiveness of TSFT stems from:

• FIM-guided parameter masking, which ensures that adaptation occurs where it most directly impacts target loss.
• Progressive token selection, enabling the model to reason over only the most discriminative (often semantic or spatially local) cues present in the data.
• Implicit alignment of parameter and token sparsity: sparser parameter updates naturally correlate with greater token redundancy, which the architecture exploits via token merging in low-activity layers.

Current limitations include:

• Application primarily to classification settings with a ViT backbone. Generalization to object detection, segmentation, and non-visual domains remains unproven.
• The per-layer token retention rate $L_{CE}$4 is fixed; future research may implement adaptive or learned token budgets for greater flexibility.
• FIM is computed using first-order gradients; more sophisticated importance metrics (e.g., block-Hessian, low-rank approximations) could further sharpen task-relevance estimation.

Despite these limitations, TR-PTS demonstrates that a small, carefully curated subset of parameters and tokens suffices for high-fidelity, scalable adaptation. This advances the broader vision of TSFT as a resource-efficient, task-adaptive, and high-accuracy approach to model adaptation that is particularly suitable for large-scale pre-trained models facing diverse real-world tasks (Luo et al., 30 Jul 2025).