Self-Optimized Fine-Tuning (SOFT)

• Self-Optimized Fine-Tuning (SOFT) is a family of methods that uses a model’s own predictions, via self-distillation and pseudo-supervision, to enhance data efficiency and robustness.
• It applies tailored strategies such as OSFT for LLMs, curriculum-based approaches for recommendations, and weighted objectives in vision to balance source and target data.
• The approach yields improved training stability, reduced overfitting, and better generalizability by leveraging intrinsic prior knowledge and careful objective design.

Self-Optimized Fine-Tuning (SOFT) encompasses a family of methods for model adaptation that exploit a model's own predictions, knowledge, or prior representations to facilitate efficient, robust, and generalizable fine-tuning. Across diverse contexts—including vision, language, and recommendation—SOFT methods replace or augment classic supervised or reward-based fine-tuning by leveraging self-distillation, curriculum learning, or self-generated pseudo-supervision. These strategies not only improve data efficiency and performance, but also confer stability and regularization benefits when transferring to new domains or tasks.

1. Conceptual Foundations and Core Variants

Self-Optimized Fine-Tuning originated as a response to the limitations of conventional transfer learning and fine-tuning methods, particularly where labeled data in the target domain is scarce or expensive, or where standard optimization is insufficiently robust to domain shift.

Three principal instantiations of SOFT are established in the literature, each corresponding to a distinct problem setting and mechanism:

• Online Supervised Fine-Tuning (OSFT) for LLMs: The model generates predictions for a set of prompts at a low temperature (i.e., high confidence), then immediately fine-tunes itself on these self-sampled outputs without any external supervision or reward signal (Li et al., 21 Oct 2025).
• SOFT for LLM-based Recommender Systems: A two-stage curriculum-learning approach, where the model first learns from a self-distilled, easy-to-learn dataset (generated by its own predictions) before gradually shifting to real, harder recommendation data via a self-adaptive scheduling function (Tang et al., 27 May 2025).
• Soft Fine-Tuning for Vision: A weighted objective gradually transitions from the source-domain loss to the target-domain loss, ensuring retention of general discrimination capabilities while accelerating fine-tuning and mitigating overfitting to small target sets (Zhao et al., 2019).

A schematic table contrasts these approaches:

Context	Self-Optimization Mechanism	Key Objective Formulation
LLM Reasoning (OSFT)	Self-generated sharp outputs, NLL	$$L_{\mathrm{OSFT}}(\theta) = -\mathbb{E}_{q,o}[\log\pi_\theta(o|q;\tau_t)]$$
Recommendation with LLMs	Self-distillation, curriculum mix	$$L_{\mathrm{SOFT}}(t) = \lambda_t L_{SD} + (1-\lambda_t) L_{SFT}$$
Vision Transfer	Joint source/target, soft switching	$$\ell_{\mathrm{SOFT}}(\theta;t) = (1-\alpha(t)) \ell_S + \ell_T$$

2. Algorithmic Structure and Objective Formulations

2.1 OSFT for LLM Reasoning

The OSFT variant employs the following routine:

1. For each batch of prompts from pretraining (e.g., math problems), sample outputs using the model at temperature $\tau_s<1$ to concentrate probability on high-confidence generations.
2. Fine-tune the model via negative log-likelihood loss at $\tau_t=1$ on these self-generated outputs.
3. Iterate for a single epoch with default $G=1$ generation per prompt for maximal efficiency.

The optimization target is:

$$L_{\mathrm{OSFT}}(\theta) = - \mathbb{E}_{q\sim\mathcal{D},\, o\sim\pi_\text{old}(\cdot|q; \tau_s)} \left[ \log \pi_\theta(o|q; \tau_t) \right].$$

Fine-tuning proceeds with $\tau_s < \tau_t$, which ensures nonzero gradient and systematic amplification of the model's latent preferred trajectories. Ablations confirm that if $\tau_s = \tau_t$ (or $\tau_s > \tau_t$), learning stalls or degrades (Li et al., 21 Oct 2025).

2.2 SOFT for LLM-Based Recommendation

The curriculum-based SOFT for recommendation involves:

1. Initial SFT on the real dataset to warm start the model.
2. Generating self-distilled pseudo-labels $\hat{y}_i$ for each input $x_i$, forming a dataset $\hat{\mathcal{D}}$.
3. Minimizing a weighted sum of the self-distillation loss $L_{SD}$ and the real-data SFT loss $L_{SFT}$:

$$L_{\mathrm{SOFT}}(t) = \lambda_t L_{SD} + (1-\lambda_t)L_{SFT},$$

with $\lambda_t = \exp[\alpha (d_t/d_0 - 1)]$, where $d_t$ measures the current model's average embedding distance to ground-truth, and $\alpha$ controls the curriculum pace.

Early epochs emphasize easy data (large $\lambda_t$), while later epochs focus on real data as the model's readiness improves (Tang et al., 27 May 2025).

2.3 Soft Fine-Tuning for Vision

Vision SOFT introduces a two-headed architecture (source and target classifiers) and decays the influence of the source loss over a scheduled number of epochs:

$$\ell_{\mathrm{SOFT}}(\theta;t) = (1-\alpha(t))\,\ell_S(\theta) + \ell_T(\theta),$$

with $\alpha(t) = \min(1, t/E)$ and $E$ specifying the transition schedule (Zhao et al., 2019). Early training leverages stable gradients from the source task, improving convergence and mitigating catastrophic forgetting.

3. Empirical Performance and Comparison

3.1 LLM Reasoning

On six mathematical reasoning benchmarks (Math500, AMC, Olympiad, Minerva, AIME24, AIME25) using Qwen2.5-Math-7B, OSFT (with $G=1$, $\tau_s=0.6$) attains mean pass@1 of 36.0% and pass@8 of 55.6%, slightly outperforming GRPO RLVR methods on pass@1 while using 8$\times$ fewer generations per prompt (Li et al., 21 Oct 2025).

3.2 LLM-Based Recommendation

On three Amazon recommendation datasets, SOFT delivers an average improvement of 37.59% across hit ratio and NDCG metrics, substantially outperforming SFT-only, guidance-only, and both LLM-based and traditional recommenders (Tang et al., 27 May 2025).

3.3 Computer Vision Transfer

Across action recognition (Stanford-40), fine-grained recognition (Stanford Dogs, FGVC Aircraft), and face verification (Oulu-CASIA), SOFT consistently achieves higher accuracy than standard fine-tuning and specialized state-of-the-art models, e.g., mAP of 92.2% vs. 91.2% (ResNet-50, Stanford-40), and TAR@FAR=1e-3 of 76.1% vs. 64.5% for NIR-VIS verification (Zhao et al., 2019).

4. Ablation Analyses and Theoretical Properties

Ablation studies confirm key properties:

• Temperature Decoupling (OSFT): $\tau_s<\tau_t$ is critical; score-function identity ensures zero gradient for $\tau_s=\tau_t$. This regime amplifies already preferred outputs rather than broadening distributional uncertainty (Li et al., 21 Oct 2025).
• Rollout Count: Increasing $G$ in OSFT yields marginal pass@1 gains at significant computational cost. $G=1$ is thus optimal for most uses.
• Scheduler Tuning (RecSys): The choice of curriculum decay schedule ($\alpha$) significantly affects the ultimate balance between guidance and challenge in data, influencing final accuracy (Tang et al., 27 May 2025).
• Source Data Influence (Vision): Retaining source loss (even as a minority term) delays overfitting and sustains the model's "general discrimination" on unseen patterns (Zhao et al., 2019).

Theoretically, SOFT can be understood as an online self-distillation or regularized transfer mechanism: it constrains optimization to remain in a favorable region anchored by intrinsic prior knowledge, resulting in smoother gradients, reduced variance in updates, and enhanced generalizability.

5. Advantages, Limitations, and Extensions

Advantages:

• Data Efficiency: SOFT methods (notably OSFT) achieve competitive or superior results with drastically reduced external supervision and sample complexity (Li et al., 21 Oct 2025).
• Training Stability: Guided objective mixing (curriculum or soft-decay) prevents catastrophic forgetting and stabilizes early-phase optimization (Zhao et al., 2019, Tang et al., 27 May 2025).
• Broad Applicability: SOFT can be instantiated for LLM reasoning, recommendation, and vision, independent of base model architecture.

Limitations:

• Reliance on Intrinsic Knowledge: SOFT cannot correct major errors when the model's prior assigns negligible probability mass to correct solutions (Li et al., 21 Oct 2025).
• Risk of Self-Reinforcement: There is potential for overfitting to self-generated artifacts or hallucinations, especially in the absence of external ground truth.
• Hyperparameter Sensitivity: Curriculum pace and mixing parameters require careful tuning for optimal performance (Tang et al., 27 May 2025).

Potential Extensions:

• Hybridization with small amounts of external reward or oracle feedback to address unrecoverable biases.
• Dynamic curriculum variants (e.g., annealing $\tau_s$ or curriculum weights) to balance exploitation and exploration.
• Application to cross-domain settings, structured prediction (e.g., code, logic), and theoretical analysis of convergent properties (Li et al., 21 Oct 2025).

6. Contexts of Application and Broader Implications

SOFT frameworks have immediate utility for tasks where sample efficiency and transfer robustness are critical, such as mathematical reasoning, personalized recommendation, and vision adaptation under domain shift. Empirical validation across domains highlights the generality of the self-optimization principle: leveraging a model's latent capabilities accelerates and stabilizes adaptation to new tasks, often without incurring the cost or complexity of alternative reward-based or multi-stage procedures.

This suggests that, given sufficient pretraining or prior information, self-optimized strategies can serve as powerful, scalable alternatives to mainstream fine-tuning, especially in resource-constrained or rapidly evolving applied research contexts.