SCUF: Semantically Consistent Unsupervised Fine-tuning

• SCUF is a method that optimizes pretrained model parameters using unlabeled data to maintain semantic fidelity across augmented and perturbed inputs.
• It employs techniques like cosine similarity, mutual information maximization, and cycle consistency to align outputs under adversarial and domain shifts.
• SCUF demonstrates state-of-the-art results in adversarial robustness, domain adaptation, low-light enhancement, and spurious correlation removal in vision and language tasks.

Semantically Consistent Unsupervised Fine-tuning (SCUF) refers to a family of frameworks in which model parameters are optimized using unlabeled data, with the explicit objective of maintaining semantic fidelity while adapting—either for robustness, adaptation to new domains or tasks, or counteracting undesirable correlations. SCUF techniques are now pervasive across vision, language, and multi-modal systems, spanning adversarial robustness, domain adaptation, removal of spurious correlations, and even cross-modal image enhancement. The principal commonality is the enforcement of “semantic consistency”: the representations (or outputs) of the model should remain aligned, both across augmented views and in the presence of domain shifts or adversarial signals, such that the semantic content (e.g., class identity, high-level features, or downstream utility) is preserved.

1. Core SCUF Paradigms and Formalization

Several instantiations of SCUF exist, tailored to data modality and target adaptation problem. Essential ingredients include:

• An unlabeled dataset (or weakly labeled, as in some semi-supervised settings),
• A pretrained model $\theta_0$ (or sequence of models),
• Transformations (adversarial perturbations, domain transfers, or augmentations) $T_1, T_2$,
• A semantic consistency criterion $C$ that measures representational or output similarity,
• Optimization of $\theta$ or additional modules to minimize discrepancies according to $C$.

For example, in vision encoders, SCUF may use two perturbed versions of the same image and enforce embedding alignment under adversarial augmentation (Hossain et al., 2024); in open-set domain adaptation for classification, semantic consistency is enforced between class predictions on strongly augmented pairs (Feng et al., 2020); in low-light enhancement, high-level consistency operates via cross-modal captions and reflectance preservation (Wang et al., 11 Jul 2025).

A high-level SCUF objective is:

$$\min_{\theta} \quad L_P(\theta) + \alpha_{\text{cons}} L_{\text{cons}}(\theta)$$

where $L_P$ is a pseudo-label or auxiliary loss and $L_{\text{cons}}$ quantifies semantic alignment between transformed instances or representations.

2. Methodologies and Architectures

The technical instantiations of SCUF are domain-specific, with notable examples including:

• Siamese Adversarial Fine-tuning: Sim-CLIP, a representative method, fine-tunes a frozen CLIP vision encoder $f_\theta$ without labels. Each batch involves both a clean image $x_c$ and an $T_1, T_2$0-perturbed PGD adversarial view $T_1, T_2$1. Both are encoded and their embeddings $T_1, T_2$2 are aligned using a symmetric stop-gradient cosine similarity loss to prevent collapse and avoid the need for negative samples or auxiliary momentum encoders (Hossain et al., 2024).
• Cycle/Consistency-based Diffusion Enhancement: In low-light image enhancement for zero-shot vision applications, SCUF employs a diffusion model conditioned on an illumination map derived from the image. It syndicates a cycle-training regime (lighten $T_1, T_2$3 normal $T_1, T_2$4 darken $T_1, T_2$5 original), caption-based feature consistency, and reflectance regularization to maintain both structural and semantic fidelity. LoRA adapters and cross-attention modules are fine-tuned while core weights remain frozen (Wang et al., 11 Jul 2025).
• Contrastive Representation Learning: SCUF in LLMs leverages contrastive losses across retrieved or augmented data points, enforcing consistency between representations that are semantically related either by domain or class, facilitating robust and generalizable features in absence of full supervision (Su et al., 2021).
• Pseudo-label and Mutual Information Regularization: For open-set domain adaptation, SCUF applies pseudo-labeling with entropy-based confidence selection and mutual information maximization on model predictions across transformations to tightly couple output distributions for augmented samples (Feng et al., 2020).
• Gradient Projection for Spurious Correlation Removal: Recent work uses unsupervised SVD extraction of latent spurious directions in LoRA fine-tuning and orthogonally projects gradients to exclude them, preventing the acquisition of new spurious relations, thus achieving semantically consistent adaptation without task signal loss (Gilligan-Lee et al., 26 May 2026).

3. Semantic Consistency Objectives and Losses

All SCUF methods are unified by their use of semantic consistency objectives, tailored via:

• Cosine Similarity: Embedding vectors for clean and perturbed or augmented samples are aligned in high-dimensional space. In Sim-CLIP, a symmetric stop-gradient cosine loss explicitly prevents trivial collapse by only propagating gradients to one branch at a time (Hossain et al., 2024).
• Mutual Information/Consistency: Maximization of mutual information or minimization of KL divergence between predictions for augmented pairs; this drives model outputs to remain deterministic and consistent under semantic-preserving transformations (Feng et al., 2020).
• Caption and Reflectance Consistency: Cycle loss for image-to-image models, enforcing that high-level semantic (caption-derived) and low-level reflectance (physical) features remain invariant around the enhancement cycle, preserving both object identity and scene integrity (Wang et al., 11 Jul 2025).
• Contrastive Losses: InfoNCE or related objectives across retrieved semantically related or unrelated instance pairs to shape decision boundaries in LLMs or vision encoders (Su et al., 2021).

4. Applications and Representative Domains

SCUF cuts across a range of domains and applications:

• Adversarial Robustness in Vision-LLMs: Sim-CLIP’s SCUF can immunize the CLIP vision backbone against adversarial attacks, sharply increasing robust accuracy under strong PGD/APGD $T_1, T_2$6 attacks, while maintaining zero-shot classification and retrieval fidelity (Hossain et al., 2024).
• Zero-Shot Low-Light Enhancement: GEFU/SCUF marries low-light image enhancement to high-level visual utility, demonstrating significant gains in downstream classification, detection, and segmentation without retraining, via strict semantic-preserving cycle fine-tuning of diffusion backbones (Wang et al., 11 Jul 2025).
• Spurious Correlation Removal in LLMs: GRASP detects and nullifies unsupervised spurious factors that emerge during fine-tuning (e.g., political bias, persona drift) using gradient projection, cleanly separating spurious and genuine signal (Gilligan-Lee et al., 26 May 2026).
• Open-Set and Domain Adaptation: SCUF-regularized hypothesis transfer enables robust adaptation to target domains with unseen classes, outperforming source-dependent and prior state-of-the-art domain adaptation methods (Feng et al., 2020).
• Low-resource or Few-Shot NLP: CSS-LM’s semi-supervised contrastive learning with explicit semantic-level retrieval confirms and extends the value of SCUF for robust generalization with limited labeled data (Su et al., 2021).

5. Empirical Results and Evaluation Metrics

Empirical benchmarks for SCUF are metric-dependent, typically evaluating both semantic fidelity and robustness/generality:

Task/Domain	Key Metrics	Example Results
Vision-Language CLIP Robustness	Robust accuracy under APGD, zero-shot retrieval, Silhouette score	Substantial $T_1, T_2$7 adversarial robustness improvements ($T_1, T_2$8) while preserving zero-shot classification (Hossain et al., 2024)
Low-Light Visual Understanding	PSNR, SSIM, LPIPS; downstream mAP$T_1, T_2$9, mIoU, Top-1	Up to 60.92% Top-1 classification (vs. 53.24%), 16.9% mAP (vs. 10.8%) in face detection under zero-shot transfer (Wang et al., 11 Jul 2025)
Open-Set Domain Adaptation	OS, OS* (overall/unknown class accuracy)	Office-31: OS = 95.8%, OS* = 96.6% (prior 93.4/94.5%) (Feng et al., 2020)
Spurious Correlation Suppression (LM)	Misalignment %, task performance, political drift	0.0% misalignment (from 59.1%) in code LMs without task score loss (Gilligan-Lee et al., 26 May 2026)

These results consistently show that enforcing semantic consistency within SCUF frameworks yields state-of-the-art results across modalities, without reliance on labeled data or large-scale negatives.

6. Limitations, Adaptations, and Extensions

Noted limitations and adaptation considerations include:

• Pseudo-label fidelity: Methods dependent on pseudo-labels (e.g., SCGAN, open-set SCUF) can propagate errors if the initial predictions are not sufficiently accurate, necessitating high confidence selection or iterative refinement (Rahman et al., 2021, Feng et al., 2020).
• Computational Overhead: Use of adversarial augmentation and cycle-consistency can increase training costs; additional discriminators or cross-modal modules may introduce latency or instability.
• Domain Generalization: Hyperparameters and architectural details (e.g., batch size, augmentation strength, cycle adapter design) may require empirical tuning across new backbones, tasks or modalities (Hossain et al., 2024, Wang et al., 11 Jul 2025).
• Scale and Dataset Transferability: Demonstrations on large-scale or long-tailed classes may require further engineering; generalization beyond tested domains demands additional validation (Rahman et al., 2021, Wang et al., 11 Jul 2025).

7. Theoretical Insights and Future Directions

Theoretical analyses underpinning SCUF’s efficacy include proofs that mutual information or cosine-based alignment maximizes agreement on true classes under reasonable augmentation assumptions (Feng et al., 2020), and that unsupervised SVD-based recovery of spurious directions is possible under simple low-rank models (Gilligan-Lee et al., 26 May 2026). A notable property is that semantic consistency constraints selectively retain meaningful, task-relevant features while suppressing superficial/misaligned patterns, as empirically confirmed by ablation and t-SNE visualizations across works (Hossain et al., 2024, Su et al., 2021, Feng et al., 2020).

A plausible implication is that future SCUF frameworks will increasingly operate across modalities (vision, language, audio) and tasks (robustness, fairness, generalization), with greater emphasis on tailored consistency objectives, minimal reliance on explicit supervision, and direct intervention on model representations or gradients to guarantee semantic fidelity. This trajectory is supported by the cross-domain adoption and extension of the core SCUF principle: semantically anchored, label-free fine-tuning remains central to robust and scalable machine learning.