Contrastive Fine-Tuning (CFT): Methods & Insights

• Contrastive Fine-Tuning (CFT) is a family of techniques that adapts pretrained models using contrastive objectives to explicitly structure the latent space.
• CFT methods integrate contrastive loss with task-specific losses like cross-entropy, ranking, and multimodal alignment to improve model robustness and transferability.
• Empirical results across domains confirm that CFT enhances sample efficiency and generalization by enforcing semantic separation and reducing intra-class scattering.

Searching arXiv for papers on contrastive fine-tuning and closely related formulations. Contrastive Fine-Tuning (CFT) denotes a family of fine-tuning procedures in which a pretrained model is adapted using a contrastive objective, either alone or in combination with task losses such as cross-entropy, ranking loss, reinforcement-learning objectives, or CTC. Across modalities, the common aim is to reshape the representation space so that semantically similar examples are closer and dissimilar examples are farther apart, thereby improving discrimination, robustness, transferability, or sample efficiency. In the literature, CFT appears both as a direct supervised contrastive stage before or during downstream adaptation and as a broader design pattern that incorporates hard negative selection, label-aware weighting, adversarial pairs, or multimodal alignment (Ma et al., 2021, Pan et al., 2022, Ukarapol et al., 2024, Sanchez et al., 25 Apr 2025).

1. Conceptual scope and defining properties

A central motivation for CFT is that standard fine-tuning objectives often optimize decision boundaries without explicitly regularizing global geometry in the learned feature space. Several papers identify this as a limitation of conventional pipelines. In self-supervised visual pretraining, images of the same class are treated as negatives during pretraining, so same-class samples can remain scattered in feature space and hamper downstream adaptation; "Contrastive Initialization" introduces an extra supervised contrastive stage on the target dataset before standard fine-tuning to enrich semantics (Pan et al., 2022). In contrastive self-supervised visual models, cross-entropy fine-tuning is likewise argued to have limited capability for reducing intra-class feature scattering, motivating "Contrast-regularized tuning (Core-tuning)" (Zhang et al., 2021).

In practice, CFT is not a single algorithm but a set of related strategies. Some methods add a contrastive term to a standard downstream loss, as in neural ranking where a contrastive loss is combined with a standard ranking loss during fine-tuning (Ma et al., 2021), or in text classification where adversarially perturbed and clean examples are paired with a SimCLR-style objective alongside cross-entropy (Pan et al., 2021). Other methods place contrastive learning in a distinct stage before classification, such as COIN’s contrastive initialization stage (Pan et al., 2022) and the three-phase transformer adaptation pipeline that performs denoising autoencoding, then contrastive learning, then final fine-tuning (Lopez-Avila et al., 2024). A further class of methods treats contrastive structure as the primary adaptation target for embeddings or multimodal alignment, including smaller LLM embedding optimization on NLI triplets (Ukarapol et al., 2024), audio classification via PairTune (Wang et al., 2023), and language-image alignment with efficient prompt fine-tuning in medical imaging (Du et al., 2024).

This suggests that the most stable encyclopedic definition of CFT is procedural rather than architectural: it is fine-tuning in which contrastive constraints are used to regularize or restructure latent representations during downstream adaptation.

2. Core objectives and mathematical forms

Many CFT methods use supervised contrastive or InfoNCE-style losses. In COIN, the initialization stage uses a supervised contrastive loss over labeled target-domain mini-batches,

$L_{\text{sup} = -\frac{1}{n} \sum_{i=1}^{n} \frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(\mathbf{v}_i^\top \mathbf{v}_p / \tau)}{\sum\limits_{a \in A(i)} \exp(\mathbf{v}_i^\top \mathbf{v}_a / \tau)}$

where $\mathbf{v}_i$ is the normalized feature, $P(i)$ indexes same-class positives, $A(i)$ all other samples, and $\tau$ is the temperature (Pan et al., 2022). Core-tuning uses an analogous supervised contrastive term and further introduces a focal reweighting over hard positives, together with a mixed-data cross-entropy term (Zhang et al., 2021). AudioConFit also adopts the supervised contrastive loss during its first-stage PairTune procedure, again operating over positives from the same class and negatives from different classes (Wang et al., 2023).

Other instances use triplet or task-specific contrastive forms. For neural rankers, the contrastive component is a Triplet Margin Loss,

$$\mathcal{L}_{TML}(a, k^+, k^-; \theta) = \max\{0,\, m + D(a, k^+; \theta) - D(a, k^-; \theta)\}$$

and is combined with a ranking loss through

$$\mathcal{L} = w_1 \cdot \mathcal{L}_{ranking} + w_2 \cdot \mathcal{L}_{contrastive}$$

with relevance labels used to define similar and dissimilar pairs across queries (Ma et al., 2021). In contrastive adversarial training for text classification, the total objective combines clean and adversarial cross-entropy with an InfoNCE-style contrastive loss,

$$\mathcal{L} = \frac{1-\lambda}{2} \left( \mathcal{L}_{CE}^{V} + \mathcal{L}_{CE}^{V+r} \right) + \lambda \mathcal{L}_{ctr}$$

where $$r = -\epsilon \cdot \mathrm{sign}\left( \nabla_{V} \mathcal{L}_{CE} \right)$$ is an FGSM perturbation on the embedding matrix (Pan et al., 2021).

A notable variation is label-aware weighting. In fine-grained text classification, Label-aware Contrastive Loss reweights negatives according to class confusability rather than treating all negatives equally, using a weighting network that predicts which labels are most plausible confusions for each example (Suresh et al., 2021). This is a direct response to the fine-grained setting in which large numbers of semantically adjacent classes make uniform treatment of negatives suboptimal (Suresh et al., 2021).

CFT is also used in multimodal and generative settings. CLEFT employs a symmetric language-image InfoNCE objective,

$$\mathcal{L} = -\frac{1}{2N}\left[ \sum_{i=1}^{N}\log\frac{\exp(I_i\cdot T_i/\tau)}{\sum_{j=1}^{N}\exp(I_i\cdot T_j/\tau)} + \sum_{i=1}^{N}\log\frac{\exp(T_i\cdot I_i/\tau)}{\sum_{j=1}^{N}\exp(T_i\cdot I_j/\tau)} \right]$$

with largely frozen encoders and parameter-efficient adaptation modules (Du et al., 2024). SLM4Offer fine-tunes T5-Small with a dual objective in which an InfoNCE loss aligns customer personas with accepted offers and separates them from rejected offers, combined with a supervised generation loss via

$\mathbf{v}_i$0

with $\mathbf{v}_i$1 set to $\mathbf{v}_i$2 (Challapalli et al., 21 Aug 2025).

3. Design patterns in CFT pipelines

A recurrent design distinction is whether contrastive learning is integrated jointly with downstream optimization or staged before it. COIN explicitly "breaks the standard fine-tuning pipeline" by inserting a contrastive initialization stage before standard fine-tuning and keeping the total number of epochs unchanged by reallocating training time (Pan et al., 2022). The three-phase transformer pipeline similarly treats contrastive learning as a separate stage between denoising autoencoding and supervised classification, and reports that staged training is superior to a joint $\mathbf{v}_i$3 alternative (Lopez-Avila et al., 2024). AudioConFit is likewise two-stage: PairTune for supervised contrastive representation learning, followed by a frozen-encoder linear classifier (Wang et al., 2023).

Joint-loss approaches are at least as common. Core-tuning minimizes

$\mathbf{v}_i$4

thereby coupling mixed-data cross-entropy with focal supervised contrastive loss (Zhang et al., 2021). Neural ranking combines ranking and triplet contrastive loss in a weighted objective (Ma et al., 2021). Accent-robust ASR adds supervised contrastive regularization to CTC fine-tuning through

$\mathbf{v}_i$5

and discards the projection head at inference, so there is no additional inference overhead (Thai et al., 5 May 2026). CARFT adds contrastive signals over Chain-of-Thought representations to RL fine-tuning, using

$\mathbf{v}_i$6

together with an embedding-enhanced partial reward for negative CoTs (Zhu et al., 21 Aug 2025).

Another recurring pattern is the use of lightweight adaptation modules or frozen backbones. CLEFT trains PEFT parameters and projection heads while keeping most core LLM and vision encoder weights frozen (Du et al., 2024). Smaller LLMs are contrastively fine-tuned with LoRA on NLI data to improve text embeddings (Ukarapol et al., 2024). SCoFT fine-tunes Stable Diffusion with LoRA while using memorization, perceptual, and self-contrastive losses to prevent overfitting on a small cultural dataset (Liu et al., 2024). This suggests that CFT is often compatible with parameter-efficient adaptation, especially when the contrastive objective is intended to reshape geometry rather than relearn all task-specific parameters.

4. Pair construction, hard negatives, and semantic structure

The effectiveness of CFT is frequently tied to how positives and negatives are chosen. Several methods emphasize that not all negatives are equally informative. In semantic-aware multimodal malware classification, hard negative samples are selected based on cosine similarity, with high-similarity negatives used to enhance discriminative power and mid-tier negatives used to increase embedding diversity (Sanchez et al., 25 Apr 2025). The paper reports that similarity-based selection is superior to random sampling, with ablation gains of $\mathbf{v}_i$7-$\mathbf{v}_i$8, and that the resulting embeddings improve few-shot malware family classification when integrated into a multimodal classifier within a MAML framework (Sanchez et al., 25 Apr 2025).

Core-tuning uses a novel hard pair mining strategy. For each anchor, it mines the hardest positive and hardest negative, generates mixed hard pairs, and combines this with focal reweighting and decision-boundary smoothing via mixup (Zhang et al., 2021). AudioConFit also emphasizes hard pair mining, defining hard positives as same-class samples least similar to the anchor and hard negatives as different-class samples most similar to the anchor (Wang et al., 2023). In smaller LLM embedding optimization, the InfoNCE formulation explicitly includes a contradiction example as a hard negative, and an ablation shows that removing the hard negative penalty reduces performance (Ukarapol et al., 2024).

Other work refines pair construction using task semantics rather than similarity alone. In neural ranking, positives are different relevant query-document pairs and negatives are non-relevant pairs, allowing contrastive learning to operate across queries rather than within a single ranking list (Ma et al., 2021). In accent-robust ASR, positives are utterances with identical transcripts, enabling utterance-level supervised contrastive learning without explicit accent labels (Thai et al., 5 May 2026). In legal provision classification with SetFit, positive pairs are sentences from the same class and negative pairs are sentences from different classes; the contrastive setup is reported to perform better than vanilla fine-tuning while using a fraction of the training samples (Kilic et al., 2023). In CARFT, positive and negative CoTs are defined by whether they generate correct or incorrect answers, with additional structure derived from annotated CoTs and the longest common subsequence between annotated and rollout CoTs (Zhu et al., 21 Aug 2025).

A plausible implication is that CFT’s empirical behavior depends at least as much on supervision design and mining strategy as on the generic choice of contrastive loss.

5. Reported effects across modalities and tasks

The empirical literature portrays CFT as a general-purpose adaptation mechanism rather than a domain-specific trick. The table summarizes representative instances that explicitly frame their method as contrastive fine-tuning or as a closely allied fine-tuning regime.

Domain	Method	Reported effect
Neural ranking	Contrastive loss + ranking loss	Improved robustness to query reformulations, noise perturbations, and zero-shot transfer; outperformed data augmentation (Ma et al., 2021)
SSL vision fine-tuning	COIN	Outperformed existing methods without introducing extra training cost; improved cluster quality and downstream accuracy (Pan et al., 2022)
SSL vision fine-tuning	Core-tuning	Improved image classification, semantic segmentation, domain generalization, and adversarial robustness (Zhang et al., 2021)
Audio classification	AudioConFit	Outperformed vanilla fine-tuning on all datasets and all backbones; faster convergence (Wang et al., 2023)
Small LM embeddings	LoRA-based CFT on NLI	Improved embedding quality for MiniCPM, Phi-2, and Gemma; MiniCPM showed an average 56.33% performance gain (Ukarapol et al., 2024)
Malware classification	Semantic-aware CFT	Achieved 63.15% accuracy with as few as 20 samples on CIC-AndMal-2020; outperformed baselines by 11--21 percentage points (Sanchez et al., 25 Apr 2025)
LLM reasoning	CARFT	Improved robustness, performance up to 10.15%, and efficiency up to 30.62% (Zhu et al., 21 Aug 2025)
Accent-robust ASR	SupCon regularization during CTC FT	Up to 25 -- 29% relative WER reduction under unseen-accent evaluation (Thai et al., 5 May 2026)

These results span classification, ranking, retrieval, speech recognition, reasoning, and multimodal generation. In neural IR, contrastive fine-tuning improves robustness to out-of-domain data and noisy inputs for both BERT and BART rankers, and outperforms data augmentation for robustification (Ma et al., 2021). In text classification, contrastive adversarial training improves BERT Large by $\mathbf{v}_i$9 on average and RoBERTa Large by $P(i)$0 on average on several GLUE tasks, with additional gains on intent classification datasets (Pan et al., 2021). In fine-grained text classification, label-aware contrastive loss produces consistent gains over CE and ordinary supervised contrastive learning when the number of classes is larger or the classes are more confusable (Suresh et al., 2021).

In audio and speech, AudioConFit outperforms vanilla fine-tuning on all datasets and all backbones, with particularly strong improvements in large-class scenarios such as TIMIT with 630 classes (Wang et al., 2023). ContrastASC combines supervised contrastive fine-tuning with contrastive representation distillation and reports preserved closed-set performance together with improved few-shot adaptation to unseen acoustic scene categories (Yuan et al., 4 Oct 2025). For ASR, supervised contrastive regularization during CTC fine-tuning improves unseen-accent evaluation and reduces within-transcript cosine dispersion, which is interpreted as more compact and stable representation geometry under accent variability (Thai et al., 5 May 2026).

In multimodal and generative settings, CLEFT reports state-of-the-art results on multiple chest X-ray and mammography datasets while reducing total trainable model size by $P(i)$1 and reducing the trainable LLM to only $P(i)$2 compared with the current BERT encoder (Du et al., 2024). SLM4Offer reports a 17 percent improvement in offer acceptance rate over a supervised fine-tuning baseline on a synthetic dataset (Challapalli et al., 21 Aug 2025). SCoFT uses self-contrastive fine-tuning to improve cultural relevance and reduce stereotypes in generated images relative to Stable Diffusion, with the full variant achieving the best user-study rankings across description alignment, cultural representation, stereotypes, and offensiveness (Liu et al., 2024).

6. Theoretical interpretations and recurring claims

Some papers offer explicit theoretical or geometric interpretations for why CFT helps. Core-tuning argues analytically that minimizing the supervised contrastive loss is proportional to minimizing $P(i)$3 and maximizing $P(i)$4,

$P(i)$5

which is interpreted as encouraging same-class compactness and different-class separation (Zhang et al., 2021). The paper also states that the contrastive loss is proportional to the infimum of the conditional cross-entropy across classifiers, yielding an optimization benefit in addition to a representational one (Zhang et al., 2021).

Other work frames the benefit geometrically. COIN reports that enriched semantics lead to lower S_Dbw cluster scores, with an example decrease on CIFAR-10 from $P(i)$6 to $P(i)$7 (Pan et al., 2022). Accent-robust ASR measures within-transcript cosine dispersion and reports a reduction in mean $P(i)$8 from $P(i)$9 to $A(i)$0, with the interpretation that transcript-matched utterances become more compact under accent variability (Thai et al., 5 May 2026). AudioConFit reports that both vanilla fine-tuning and PairTune yield nearly isotropic embeddings, and attributes its gains to improved separability rather than purely embedding isotropy (Wang et al., 2023). MIMIC shows through t-SNE that mix-supervised contrastive fine-tuning yields better class separation for facial expression recognition and interprets this as bridging the domain gap between general-image pretraining and FER data (Zhang et al., 2024).

A recurring claim is that CFT improves generalization by encoding semantics at the representation level rather than relying only on task-level supervision. In neural ranking, the contrastive term is said to help models learn underlying matching semantics across different query-document pairs (Ma et al., 2021). In legal text classification, SetFit is reported to focus more strongly on legally informative features in LIME explanations and to achieve better or comparable performance with much less labeled data (Kilic et al., 2023). In malware classification, fine-tuned LLMs generate attribute-aware descriptions that generalize to unseen variants and bridge textual and binary feature gaps (Sanchez et al., 25 Apr 2025).

7. Limitations, variants, and points of ambiguity

The literature also shows that "CFT" is polysemous. In most of the papers cited here, CFT refers to contrastive fine-tuning. However, the acronym is also used for "Critical Token Fine-tuning" in LLM reasoning, where only functionally indispensable tokens are updated via a counterfactual mask and no contrastive loss is involved (Ruan et al., 13 Oct 2025). The overlap in acronym means that the term requires contextual disambiguation in current usage.

Another limitation is that the precise benefit of CFT is task-dependent. Some papers report large gains primarily in data-scarce, fine-grained, noisy, or distribution-shifted settings rather than universally. SetFit’s main advantage is strongest under data scarcity and imbalance (Kilic et al., 2023). Label-aware contrastive loss reports only minor improvements on binary SST-2, with larger benefits on SST-5 and multi-class emotion tasks (Suresh et al., 2021). The three-phase transformer method reports that contrastive learning is especially useful when class imbalance and representation robustness are prominent (Lopez-Avila et al., 2024). This suggests that the strongest case for CFT arises when downstream supervision alone is insufficient to induce a robust geometry.

There are also overhead and implementation trade-offs. Label-aware contrastive loss doubles training-time memory and compute requirements by using both an encoder and a weighting network, though the weighting network is not needed at inference (Suresh et al., 2021). Hard pair mining can increase per-batch cost in AudioConFit, although the paper argues that the number of epochs and instability are reduced (Wang et al., 2023). CARFT requires annotated Chain-of-Thoughts and rollout-based reasoning signals, which implies a stronger supervision burden than ordinary SFT (Zhu et al., 21 Aug 2025). CLEFT reduces trainable parameters but still depends on large pretrained encoders and prompt tuning infrastructure (Du et al., 2024).

Finally, not every claimed CFT method is equally well documented in the supplied source material. For "Semantic-Aware Contrastive Fine-Tuning: Boosting Multimodal Malware Classification with Discriminative Embeddings" the abstract reports a detailed method and numerical results, but the supplied document details explicitly state that the reviewed document appears to be a generic IEEE LaTeX template and does not contain the relevant technical content (Sanchez et al., 25 Apr 2025). The concrete claims cited here therefore derive from the abstract-level description rather than the missing full paper details.

In aggregate, the arXiv literature presents Contrastive Fine-Tuning as a broad adaptation paradigm for imposing semantically meaningful structure during model specialization. Its concrete instantiations differ in loss formulation, scheduling, pair construction, and parameterization, but they repeatedly target the same underlying problem: standard fine-tuning often optimizes predictions without sufficiently organizing the latent space for robustness, transfer, or fine-grained discrimination (Ma et al., 2021, Zhang et al., 2021, Pan et al., 2022).