CLIP-Fed: Lightweight Unsupervised Federated Learning

• The paper introduces CLIP-Fed, a framework that fixes CLIP’s image encoder and learns only a linear classifier to significantly reduce computation and communication costs.
• It refines soft pseudo-labels through momentum-based self-training while augmenting local learning with synthetic, class-balanced feature samples derived from CLIP text prototypes.
• The approach effectively addresses challenges of unlabeled data, non-i.i.d. data distribution, and class imbalance, achieving competitive accuracy on benchmarks like CIFAR-10 and CIFAR-100.

CLIP-Fed is a lightweight unsupervised federated learning framework built around a pretrained vision–LLM, specifically CLIP, for the setting in which privacy-preserving clients hold only unlabeled images and operate under strict computation and communication limits. In the formulation introduced in "Lightweight Unsupervised Federated Learning with Pretrained Vision LLM" (Yan et al., 2024), CLIP-Fed—also called FST-CBDG—fixes CLIP’s pretrained image encoder, initializes a linear classifier from CLIP text prototypes, refines soft pseudo-labels by self-training, augments local learning with synthetic class-balanced feature samples, and federated-averages only the linear head parameters. This combination is designed to address unlabeled local data, non-i.i.d. data heterogeneity, class imbalance, and the cost of transmitting deep models (Yan et al., 2024).

1. Definition and problem setting

CLIP-Fed targets a specific variant of federated learning in which each client holds unlabeled images and cannot afford either the labeling burden of supervised federated learning or the training and transmission cost of deep models. The framework uses CLIP’s zero-shot prediction capability and its pretrained image encoder to avoid full-model training on edge devices, while still permitting collaborative learning across isolated clients (Yan et al., 2024).

The method is motivated by three constraints stated explicitly in the original formulation: unlabeled data at each client, non-i.i.d. and class imbalance, and device constraints. The first removes the ordinary supervised signal. The second means that local pseudo-labels may be biased and some classes may be under-represented. The third makes deep-model back-propagation and large parameter exchanges impractical. CLIP-Fed addresses these constraints by reducing the trainable component to a linear head on top of fixed CLIP features, thereby turning federated optimization into the aggregation of a small parameter set rather than a full vision backbone (Yan et al., 2024).

A common misconception is to treat CLIP-Fed as a generic label for any CLIP-based federated method. In the literature, however, several related but distinct frameworks exist. FedCLIP fine-tunes a lightweight attention-based adapter for CLIP in labeled federated classification (Lu et al., 2023); CLIP2FL uses CLIP-guided knowledge distillation and federated feature synthesis on heterogeneous and long-tailed labeled data (Shi et al., 2023); and a later work reused the name CLIP-Fed for a federated backdoor-defense framework rather than for unsupervised learning (Gai et al., 14 Aug 2025). In this primary usage, CLIP-Fed refers to the unsupervised linear-head method of (Yan et al., 2024).

2. Model construction and pseudo-label self-training

The core architectural choice in CLIP-Fed is to keep CLIP’s image encoder fixed and learn only a linear classifier. On the server, for each class $k=1,\dots,K$, a prompt such as “a photo of a $\text{class}_k$” is encoded by CLIP’s text encoder $E_{\rm text}$ to obtain a text prototype $T_k\in\mathbb{R}^d$. These prototypes initialize the classifier parameters as

$$W^0 = \begin{bmatrix} T_1^\top \ T_2^\top \ \vdots \ T_K^\top \end{bmatrix}, \qquad b^0 = \mathbf 0.$$

The server then distributes $\{T_k\}_{k=1}^K$ and the fixed image encoder $E_{\rm img}$ to the clients (Yan et al., 2024).

For each unlabeled image $x_j$ on client $c$, CLIP-Fed extracts a feature

$z_j = E_{\rm img}(x_j)\in\mathbb{R}^d,$

computes zero-shot scores

$\text{class}_k$0

and obtains a soft pseudo-label

$\text{class}_k$1

These zero-shot probabilities serve as the initial pseudo-labels, $\text{class}_k$2. The trainable head is a standard softmax classifier on fixed CLIP features,

$\text{class}_k$3

During local training, the pseudo-labels are refined by a momentum update,

$\text{class}_k$4

This moving-average rule is central to the framework’s self-training behavior: it avoids discarding the initial zero-shot prior while allowing the local linear head to reshape class assignments over time (Yan et al., 2024).

The local loss on real unlabeled features is a self-training cross-entropy loss denoted $\text{class}_k$5. Only $\text{class}_k$6 are updated; CLIP’s image encoder remains fixed. This sharply constrains the optimization problem, which is a deliberate design choice for edge-side feasibility. A plausible implication is that CLIP-Fed treats CLIP not as a model to be adapted deeply, but as a high-quality frozen feature map whose geometry is already useful for federated self-training.

3. Class-balanced text feature sampling

To mitigate class imbalance and non-i.i.d. effects, CLIP-Fed augments local training with synthetic feature samples built directly around CLIP’s text prototypes. The method first estimates class counts from current pseudo-label assignments: if $\text{class}_k$7 is the number of images whose current pseudo-label $\text{class}_k$8 is class $\text{class}_k$9, and $E_{\rm text}$0, then synthetic counts $E_{\rm text}$1 are determined by

$E_{\rm text}$2

where $E_{\rm text}$3 is a balance factor (Yan et al., 2024).

For each class $E_{\rm text}$4, CLIP-Fed then draws synthetic features from a Gaussian centered at the corresponding text prototype:

$E_{\rm text}$5

These synthetic samples receive one-hot labels $E_{\rm text}$6, and their contribution is optimized through a text-sampling loss $E_{\rm text}$7 (Yan et al., 2024).

This mechanism is distinctive because it does not generate images. It generates feature-space samples around CLIP text embeddings, which are treated as class prototypes. Relative to image-space synthesis, this preserves the lightweight character of the method and avoids any need to train or communicate a generative model. The original ablation results report that using only $E_{\rm text}$8 or only $E_{\rm text}$9 gives no gain, whereas balanced rather than equal sampling yields a 5-point accuracy lift (Yan et al., 2024). That pattern indicates that the framework depends on the interaction between pseudo-label refinement and class-balanced prototype-based augmentation rather than on either ingredient in isolation.

4. Federated protocol and resource profile

CLIP-Fed runs for $T_k\in\mathbb{R}^d$0 federated rounds. In each round, the server samples a subset of clients and sends them the current $T_k\in\mathbb{R}^d$1. Each selected client downloads the linear head, runs the local self-training and class-balanced sampling routine for one epoch, and uploads updated $T_k\in\mathbb{R}^d$2. The server aggregates by plain FedAvg:

$T_k\in\mathbb{R}^d$3

The communication cost is therefore limited to approximately $T_k\in\mathbb{R}^d$4 floats per round, with the paper giving $T_k\in\mathbb{R}^d$5 for CIFAR-10 as an example, in contrast to tens of millions when transmitting a full ResNet. Likewise, local computation is reduced to back-propagation through one linear layer (Yan et al., 2024).

This protocol places CLIP-Fed in a broader family of CLIP-based federated methods that freeze large pretrained backbones and communicate only small task-specific parameters. FedCLIP, for example, communicates only an attention-based adapter and reports approximately $T_k\in\mathbb{R}^d$6 reductions relative to full CLIP fine-tuning (Lu et al., 2023). TriplePlay similarly freezes CLIP’s backbone, updates only an adapter, and adds quantization and LoRA to reduce communication overhead (Imteaj et al., 2024). CLIP-Fed is more minimal than either: it transmits only the parameters of a linear head.

The minimality of this communication pattern is important for interpreting the method’s scope. CLIP-Fed is not presented as a general-purpose replacement for all federated vision-language adaptation. Rather, it is a highly constrained solution for unlabeled and resource-limited clients, where even adapter fine-tuning may be less attractive than linear-head aggregation.

5. Experimental setting and reported performance

The experimental evaluation uses CIFAR-10, CIFAR-100, and CINIC-10, split across $T_k\in\mathbb{R}^d$7 clients, with 10% clients participating per round, under both i.i.d. and non-i.i.d. partitions. Non-i.i.d. settings include sharding with $T_k\in\mathbb{R}^d$8 shards and LDA with Dirichlet $T_k\in\mathbb{R}^d$9. The reported metric is test accuracy after 10 rounds, with 1 local epoch per round (Yan et al., 2024).

The main homogeneous-setting results are summarized below.

Dataset	CLIP-ZS	CLIP-Fed / FST-CBDG
CIFAR-10	68.7	74.0
CIFAR-100	39.0	43.2
CINIC-10	63.2	66.3

In the same setting, the paper reports supervised federated baselines of 73.3 for FedAvg on CIFAR-10, 37.8 on CIFAR-100, and 66.0 on CINIC-10, so CLIP-Fed not only improves over CLIP zero-shot predictions but also exceeds those supervised federated baselines under limited computational and communication overhead (Yan et al., 2024).

Under heavy non-i.i.d. partitioning on CIFAR-10 with sharding $$W^0 = \begin{bmatrix} T_1^\top \ T_2^\top \ \vdots \ T_K^\top \end{bmatrix}, \qquad b^0 = \mathbf 0.$$0, CLIP-Fed improves CLIP-ZS from 68.7 to 72.0, while FedAvg collapses to approximately 32% (Yan et al., 2024). The reported convergence behavior is also notable: one or two rounds suffice to surpass CLIP zero-shot and match supervised FL. This suggests that the initialization from CLIP text prototypes and the momentum-smoothed pseudo-label updates produce a strong early optimization trajectory.

The paper’s ablations identify two salient dependencies. First, using only the self-training loss or only the text-sampling loss gives no gain. Second, class-balanced sampling yields a 5-point accuracy lift over equal sampling. These results reinforce the interpretation that CLIP-Fed’s performance depends on the joint use of CLIP-derived pseudo-label priors and synthetic class balancing in feature space (Yan et al., 2024).

6. Relationship to adjacent methods and nomenclature

CLIP-Fed belongs to a rapidly expanding line of federated learning methods that use CLIP or other vision–LLMs to cope with heterogeneity, long-tail structure, and the cost of large pretrained backbones. Within this line, different methods allocate adaptation capacity at different points.

FedCLIP attaches an attention-based adapter immediately after the frozen CLIP image encoder and trains only the adapter in a standard federated-averaging loop, reporting strong generalization and personalization on PACS, VLCS, and Office-Home, as well as approximately $$W^0 = \begin{bmatrix} T_1^\top \ T_2^\top \ \vdots \ T_K^\top \end{bmatrix}, \qquad b^0 = \mathbf 0.$$1 savings in compute and communication relative to full fine-tuning (Lu et al., 2023). CLIP2FL uses client-side CLIP-guided knowledge distillation plus server-side federated feature synthesis with prototype contrastive learning to retrain a balanced server classifier on heterogeneous and long-tailed labeled data (Shi et al., 2023). TriplePlay uses CLIP as a modular adapter within FedAvg, adds GAN-based data balancing, and combines quantization with QLoRA; on PACS at 500 rounds it reports 80.2% final accuracy for TriplePlay versus 75.8% for FedCLIP+QLoRA and 70.4% for FedCLIP, with GPU utilization reduced from approximately 60–70% to approximately 35% (Imteaj et al., 2024).

Against this backdrop, CLIP-Fed is distinguished by three properties. First, it is explicitly unsupervised at the client level. Second, it freezes CLIP completely and learns only a linear head. Third, its synthetic augmentation occurs in feature space around text prototypes rather than through an image generator or an adapter-specific balancing module (Yan et al., 2024). This makes it one of the most lightweight formulations in the CLIP-based federated literature.

A separate nomenclature issue complicates the term. In 2025, "A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learning" introduced a different framework also named CLIP-Fed, centered on pre-aggregation PCA and HDBSCAN filtering, server-side dataset construction using an MLLM, and post-aggregation purification through prototype contrastive loss and KL-divergence knowledge transfer (Gai et al., 14 Aug 2025). For technical clarity, CLIP-Fed in the unsupervised federated learning sense should therefore be identified with the 2024 FST-CBDG formulation (Yan et al., 2024), while the 2025 usage should be treated as a distinct backdoor-defense framework that happens to share the same name.