---
title: Prototype-Oriented Clustering with Distillation (PCD)
url: https://www.emergentmind.com/topics/prototype-oriented-clustering-with-distillation-pcd
type: topic
---

# Prototype-Oriented Clustering with Distillation (PCD)

Prototype-Oriented Clustering with Distillation (PCD) is an advanced federated optimization methodology designed to address the challenges of label skew and data heterogeneity in federated learning (FL) settings, particularly within AI-native Radio Access Network (AI-RAN) enabled Multi-Access Edge Computing (MEC) systems. PCD as realized in the Multi-Prototype-Guided Federated Knowledge Distillation (MP-FedKD) framework leverages conditional hierarchical agglomerative clustering (CHAC), multi-prototype representations, and multi-component prototype alignment losses to enable robust, communication-efficient, and high-performance distributed model training in environments with non-independent and identically distributed (non-IID) data [2603.09727].

## 1. Motivation and Federated Context

In AI-RAN-enabled MEC, edge devices jointly train a global model without sharing raw data, making FL an attractive paradigm. Traditional FL approaches relying on single mean prototypes per class fail to adequately capture intra-class variability and can lead to information loss, especially under non-IID data distributions. PCD proposes to replace the single mean-prototype per class with a small set of cluster centroids. These multi-prototypes, obtained via CHAC, enhance the fidelity of local representations and their alignment with global prototypes, enabling superior mitigation of heterogeneity-induced performance degradation [2603.09727].

## 2. Conditional Hierarchical Agglomerative Clustering (CHAC)

For each client $m$ and class label $c$, the representation network $\Upsilon(\cdot)$ maps input data $x_{m}^{n}$ to local class-specific embeddings
\[
\varrho_{m,c}^n = \Upsilon(x_m^n; y_m^n=c).
\]
To construct a set of $\zeta_m^c$ representative prototypes per class, the client performs CHAC, initialized with each embedding as its own cluster. Using Ward’s linkage criterion, the algorithm iteratively merges pairs of clusters $(B_1, B_2)$ with minimal increase in within-cluster sum of squares (SSQ):
\[
\Delta SSQ_{B_1,B_2} = \frac{v_1 v_2}{v_1 + v_2} \sum_{q=1}^Q (\bar E_{B_1}^q - \bar E_{B_2}^q)^2,
\]
where $v_1, v_2$ are cluster sizes and $\bar E_{B_z}^q$ is the mean along dimension $q$. Merging continues until $\zeta_m^c = \min(\widehat{\zeta}, |\mathcal{D}_{m,c}|)$ clusters remain, after which cluster centroids $P_{m,c}^i$ are extracted as the local class prototypes [2603.09727].

## 3. Prototype Alignment and Aggregation

Prototypes $\{P_{m,c}^i\}$ from each client are communicated to the central server, which aggregates them to form global class prototypes $\{\overline{P}_c^t\}$ using a weighted average:
\[
\overline P_c^t = \frac{1}{|\mathcal{M}_c|\,\zeta_m^c} \sum_{m\in\mathcal{M}_c} \sum_{i=1}^{\zeta_m^c} \frac{|\,\mathcal{D}_{m,c}|}{|\mathcal{D}_c|} P_{m,c}^i.
\]
Alignment of local and global prototypes is quantitatively enforced by minimizing the Euclidean distance between local embeddings (evaluated using the previous-round representation) and global class prototypes, as codified by the Prototype Alignment (PA) loss [2603.09727].

## 4. Knowledge Distillation and Local Objective Design

PCD integrates multiple loss components in the federated optimization loop. The total local objective for client $m$ at round $t$ is given by
\[
\mathcal{L}_m^{loss} = \mu_1\,\mathcal{L}_m^{CE} + (1-\mu_1)\,\mathcal{L}_m^{SKD} + \mu_2\,\mathcal{L}_m^{PA} + \mu_3\,\mathcal{L}_m^{LEMGP},
\]
with the following components:
- **Cross-Entropy Loss ($\mathcal{L}_m^{CE}$):** Standard supervised criterion.
- **Self-Knowledge-Distillation (SKD) Loss ($\mathcal{L}_m^{SKD}$):** KL divergence between current and previous local model logits, smoothed by temperature $\tau$.
- **Prototype Alignment Loss ($\mathcal{L}_m^{PA}$):** Quadratic penalty enforcing local embedding proximity to global prototypes.
- **LEMGP Loss ($\mathcal{L}_m^{LEMGP}$):** An "attractive–repulsive" objective comprised of an intraclass attraction term and an interclass repulsion term, promoting compactness and discriminability [2603.09727].

Hyperparameters controlling the loss mixture are typically set to $\mu_1=0.9,\,\mu_2=1.0,\,\mu_3=0.1$; initial training rounds use only $\mathcal{L}_m^{CE}$.

## 5. End-to-End Workflow and Implementation Pseudocode

The federated procedure is organized as an alternating sequence of server and client operations, summarized below:

| Step                        | Description                                                                                                   | Reference                           |
|-----------------------------|---------------------------------------------------------------------------------------------------------------|-------------------------------------|
| Initialization              | Global model $\omega^0$, global prototypes $\{\overline P_c^0\}$                                              |                                    |
| Server broadcast            | $(\omega^{t-1}, \{\overline P_c^{t-1}\})$ to selected clients                                                |                                    |
| Client local update         | Calculation of $\mathcal{L}_m^{CE}$, $\mathcal{L}_m^{SKD}$, CHAC clustering, $\mathcal{L}_m^{PA}$, LEMGP loss | MP-FedKD workflow; eqs. (5,9,10–12) |
| Server aggregation          | FedAvg for $\omega^t$; prototype update via weighted sum (eq. 4)                                             |                                    |

The full procedural logic, including conditional CHAC, is specified in the pseudocode block in the source [2603.09727].

## 6. Hyperparameters and Design Insights

Key hyperparameters and their standard settings include:
- $\widehat\zeta = 3$ (max clusters per class).
- $\tau = 0.1$ (distillation temperature).
- $\Lambda = \lambda = 0.5$ (LEMGP attractive/repulsive weighting).
- Learning rate $\eta = 10^{-3}$, batch size 32, local epochs 5, rounds 50.

By design, PCD enables modeling intra-class diversity through multi-prototypes, stabilizes local updates on non-IID data by self-distillation, and ensures global-local consistency with prototype alignment and attraction–repulsion mechanisms. This approach provides robustness to data heterogeneity while maintaining communication efficiency, as only prototype vectors and model parameters are transmitted [2603.09727].

## 7. Performance Implications and Practical Significance

Experimental results on benchmark datasets under various non-IID conditions demonstrate that the MP-FedKD framework incorporating PCD consistently achieves higher accuracy, average accuracy, and lower RMSE/MAE compared to state-of-the-art baselines. Utilizing CHAC-based multi-prototype construction mitigates information loss characteristic of mean-prototype strategies and preserves intra-class manifold structure. The combination of self-knowledge distillation, PA, and LEMGP losses fosters an optimization dynamic that is less susceptible to the adverse effects of data skew and local overfitting, suggesting broad utility for federated deployments in real-world AI-RAN and MEC scenarios [2603.09727].

Source: https://www.emergentmind.com/topics/prototype-oriented-clustering-with-distillation-pcd