---
title: Class-wise Prototype Sparsification (CPS)
url: https://www.emergentmind.com/topics/class-wise-prototype-sparsification-cps
type: topic
---

# Class-wise Prototype Sparsification (CPS)

Searching arXiv for recent papers directly relevant to Class-wise Prototype Sparsification and closely related prototype-sparsification work.
Class-wise Prototype Sparsification (CPS) is a structured sparsification scheme for class prototypes in prototype-based federated learning (PBFL), introduced in TinyProto as a mechanism for reducing communication overhead by assigning each class a fixed binary support mask and transmitting only the retained coordinates of that class’s prototype [2507.04327]. In the formulation given in TinyProto, CPS acts on class-representative prototype vectors rather than on model weights or gradients, and its defining property is class-wise support consistency: clients and server share the same mask for a given class, so sparse prototypes remain aligned during aggregation [2507.04327]. Closely related work has extended the broader design space around class-wise prototype communication, alignment, and representation control, although these methods generally do not implement prototype sparsification in the strict sense. RefProtoFL combines class-wise prototype alignment with sparsified adapter updates, but does not sparsify prototypes themselves [2601.14746]. FedProtoKD refines class-wise prototypes through adaptive margin-preserving server-side learning rather than pruning or masking prototype dimensions [2508.19009]. In continual learning, YONO can be interpreted as an extreme form of class-wise prototype compression in which one condensed prototype is stored per class, but its objective is memory-efficient replay rather than communication-efficient federated aggregation [2305.16143]. A superficially similar acronym appears in Conditional Point Sparsification for SAM-based few-shot segmentation, but that work is unrelated to prototype sparsification and instead sparsifies prompt points [2602.05218]. A further adjacent line is SGDS, which uses class-mean prototypes only to organize sparse activation subspaces; it does not define sparse prototypes as the primary object [2601.21345].

## 1. Concept and scope

In TinyProto, CPS is defined as a class-wise, fixed-support prototype compression mechanism for PBFL [2507.04327]. Let a class prototype be $\bar{c} \in \mathbb{R}^d$ and let a binary mask be
\[
m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.
\]
CPS constructs a structured sparse prototype
\[
\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,
\]
and a compressed prototype
\[
\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,
\]
where
\[
s=\sum_{i=1}^d m_i.
\]
The structured sparse form preserves the original ambient dimension while zeroing inactive coordinates, whereas the compressed form contains only the retained entries and is the object communicated between clients and server [2507.04327].

The defining feature of CPS is that sparsity is not generic or unstructured. It is class-specific, fixed for a class, and shared across participating nodes [2507.04327]. This distinguishes it from per-round top-\(k\) sparsification, random coordinate dropping, quantization, model pruning, or gradient compression. The paper explicitly frames CPS as “allocating specific dimensions to class prototypes” and selecting “only a subset of elements where mask values are 1s” for communication [2507.04327].

A strict usage of the term therefore refers to prototype sparsification itself, not merely to sparse communication in a prototype-based system. This distinction matters in adjacent literature. RefProtoFL is prototype-driven but sparsifies shared adapter updates, not prototypes [2601.14746]. FedProtoKD uses trainable server prototypes and adaptive class-wise margins, but does not impose sparse prototype supports [2508.19009]. YONO stores one prototype per class, which can be viewed as prototype compression at the object level, yet it does not introduce masked sparse coordinates or a PBFL communication protocol [2305.16143].

## 2. Federated learning setting and motivation

TinyProto introduces CPS in the setting of communication-efficient federated learning for resource-constrained and heterogeneous environments [2507.04327]. In PBFL, clients exchange class prototypes—mean activations in a shared representation space—instead of model parameters. This is often cheaper than full-model communication, but the paper identifies a remaining bottleneck: prototype traffic still scales with both the number of classes and the feature dimension [2507.04327].

For prototype-based methods such as FedProto and FedTGP, the per-round communication cost is given as
\[
\sum_{i=1}^M (K_i + K)\times d,
\]
where \(M\) is the number of clients, \(K_i\) is the number of classes present on client \(i\), \(K\) is the total number of classes, and \(d\) is prototype dimension [2507.04327]. CPS replaces the dense dimension \(d\) by the retained support size \(s\), yielding
\[
\sum_{i=1}^M (K_i + K)\times s.
\]
This makes the communication burden depend on sparse support size rather than full representation width [2507.04327].

The motivation is not only algebraic but also structural. TinyProto argues that penultimate-layer activations often exhibit sparse behavior under ReLU, yet this sparsity is inconsistent across clients because of model heterogeneity and data heterogeneity [2507.04327]. As a result, naïvely aggregated dense prototypes tend to occupy most coordinates globally, even if individual client activations are locally sparse. CPS addresses this inconsistency by enforcing a shared per-class support pattern across clients, so that all prototype messages for class \(j\) lie in the same coordinate subset [2507.04327].

This suggests a broader interpretation: CPS is a coordination mechanism as much as a compression mechanism. Sparse communication becomes effective only when the support pattern is semantically aligned across participants. A plausible implication is that, without such coordination, prototype sparsity would be difficult to exploit in heterogeneous PBFL.

## 3. Formal construction and masking scheme

The local class prototype in TinyProto is defined as the mean feature activation in the penultimate or decision-layer representation space:
\[
\bar{c}_{i,j}^{L}= \frac{1}{n_{i,j}} \sum_{(x,y)\in \mathcal{D}_{i,j}} f_i(\theta_i;x),
\]
where \(\mathcal{D}_{i,j}\subseteq \mathcal{D}_i\) is the subset of client \(i\)’s local dataset containing class \(j\), \(n_{i,j}=|\mathcal{D}_{i,j}|\), and \(f_i(\theta_i;x)\in\mathbb{R}^d\) is the feature extractor output [2507.04327]. In standard PBFL, the global prototype is described either by weighted averaging,
\[
\bar{c}_j^{G} = \frac{\sum_{i=1}^{M} n_{i,j}\bar{c}_{i,j}^{L}}{\sum_{i=1}^{M} n_{i,j}},
\]
or by simple averaging,
\[
\bar{c}_{j}^{G}=\frac{1}{|\mathcal{N}_j|}\sum_{i\in\mathcal{N}_j}\bar{c}_{i,j}^{L},
\]
where \(\mathcal{N}_j\) is the set of clients containing class \(j\) [2507.04327].

Under CPS, each class \(j\) is assigned a class-specific binary mask
\[
m_j \in \{0,1\}^d.
\]
The retained dimension count is
\[
s = \sum_{k=1}^d [m_j]_k.
\]
The paper states that the server initializes the masking vector set \(\{m_j\}\) and shares masks with clients [2507.04327]. These masks are fixed per class and shared between server and clients. The paper does not present a learning procedure for masks, nor does it specify a detailed mask-construction algorithm. It states that, to ensure inter-class distinctiveness, it typically maximizes pairwise Hamming distances between masking vectors [2507.04327]. This means masks may be disjoint or overlapping; full disjointness is not a formal requirement.

The sparse prototype for class \(j\) is
\[
\tilde{c}_j = S(\bar{c}_j;m_j)=\bar{c}_j \odot m_j,
\]
and the communicated compressed prototype is
\[
\hat{c}_j = C(\bar{c}_j;m_j)=\left(\bar{c}_{j,k}: [m_j]_k=1\right)\in\mathbb{R}^s.
\]
The appendix further denotes the support set by
\[
\Omega_j \subset \{1,\ldots,d\},
\]
with fixed-support assumption
\[
\operatorname{supp}(S(\bar{C}^{(j)}))=\Omega_j
\]
in the convergence analysis [2507.04327].

This formalism makes CPS a support-allocation method over prototype coordinates. The paper’s emphasis on Hamming-distance separation between masks indicates that inter-class distinctiveness is encoded partly through support geometry, not only through prototype values. This suggests that CPS seeks discrimination by combining semantic content and coordinate allocation.

## 4. Aggregation, scaling, and training pipeline

TinyProto couples CPS with adaptive prototype scaling [2507.04327]. After computing dense local prototypes, a client compresses them using the shared mask:
\[
\hat{c}^{L}_{i,j}=C(\bar{c}^{L}_{i,j};m_j).
\]
Before transmission, the client scales the compressed local prototype by the class sample count:
\[
n_{i,j}\hat{c}^{L}_{i,j}.
\]
The server then aggregates scaled sparse prototypes according to
\[
\hat{c}_{j}^{G} = \frac{1}{|\mathcal{N}_j|} \sum_{i \in \mathcal{N}_j} n_{i,j}\hat{c}_{i,j}^{L}.
\]
Clients reconstruct the structured sparse prototype \(\tilde{c}^{G}_{j}\) from \(\hat{c}^{G}_{j}\) using the shared mask, and then apply global scaling in the regularization term:
\[
\mathcal{R}_i = \sum_j \rho(\bar{c}^{L}_{i,j}, \mu \tilde{c}^{G}_{j}),
\]
where \(\rho(\cdot,\cdot)\) is Euclidean distance [2507.04327]. The local objective becomes
\[
\tilde{\mathcal{L}_i(w_i)}=\mathcal{L}_i(w_i)+\lambda \mathcal{R}_i.
\]

The training procedure is PBFL augmented with CPS and scaling. Initially, clients train without regularization. Once global prototypes become available, each round follows the pattern: local training, dense prototype extraction, sparsification/compression, local scaling and upload, server aggregation, download and reconstruction, and local regularized training with \(\mu\tilde{c}^{G}_j\) [2507.04327]. Algorithm 1, “TinyProto-FP,” specifies that the server initializes mask vectors \(\{m_j\}\), sends masks to newly participating clients when needed, and aggregates returned \(n_{i,j}\hat{c}^{L}_{i,j}\) each round [2507.04327].

The scaling mechanism is presented as a way to preserve class-wise contribution importance while avoiding direct transmission of raw \(n_{i,j}\) as explicit metadata [2507.04327]. The paper states that this “prevents privacy leakage by obfuscating \(n_{i,j}\) during communication and at the server level.” In context, scaling is therefore not separate from CPS but a compensatory device: sparse communication reduces cost, while scaling helps preserve aggregation quality under class imbalance.

## 5. Communication efficiency and empirical behavior

The central communication claim is that TinyProto changes prototype exchange from scaling with \(d\) to scaling with \(s\) [2507.04327]. With \(d=500\) and \(s=50\), the paper states that this corresponds to a 90% compression rate. It further reports that TinyProto can achieve up to \(10\times\) reduction versus original PBFL methods and up to \(4\times\) lower communication than efficient baselines such as FedDistill in reported experiments [2507.04327].

The paper’s main quantitative results for \(d=500\), \(s=50\) are summarized below.

| Dataset | Method | Accuracy / Communication |
|---|---|---|
| CIFAR-100 | FedProto | \(29.97\%\), \(1.46\)M |
| CIFAR-100 | TinyProto-FP (50) | \(31.82\%\), \(0.15\)M |
| CIFAR-100 | FedTGP | \(36.92\%\), \(1.46\)M |
| CIFAR-100 | TinyProto-FT (50) | \(45.94\%\), \(0.15\)M |
| TinyImageNet | FedProto | \(13.30\%\), \(2.93\)M |
| TinyImageNet | TinyProto-FP (50) | \(16.01\%\), \(0.29\)M |
| TinyImageNet | FedTGP | \(19.44\%\), \(2.93\)M |
| TinyImageNet | TinyProto-FT (50) | \(27.29\%\), \(0.29\)M |

These results show that CPS-based communication can reduce dense PBFL prototype traffic by roughly \(10\times\) while maintaining or improving performance when combined with adaptive scaling [2507.04327]. The paper also varies CPS dimension \(s\in\{50,150,250,350,450\}\) for \(d=500\), reporting that reducing communicated dimensions via CPS does not necessarily hurt accuracy and can slightly improve it [2507.04327]. Figure 6 further indicates that CPS can outperform reducing the decision-layer dimension to reach the same communication budget, because the full \(d\)-dimensional model capacity is retained during training and only communication is compressed [2507.04327].

TinyProto is also evaluated under heterogeneous and non-IID settings. The setup uses Dirichlet partitioning with \(\alpha=0.1\), with additional tests at \(\alpha=0.01\), \(\alpha=0.5\), and \(M=50\) clients [2507.04327]. For CIFAR-100, TinyProto-FT (50) remains strongest with \(43.51\%\) at \(M=50\), \(72.74\%\) at \(\alpha=0.01\), and \(24.35\%\) at \(\alpha=0.5\) [2507.04327]. These are not isolated CPS-only effects, because adaptive scaling is simultaneously active, but they support the practical viability of the CPS design under heterogeneity.

## 6. Relationship to adjacent prototype methods

The most direct neighboring literature does not generally implement class-wise prototype sparsification in the strict TinyProto sense. RefProtoFL is highly relevant to prototype-based federated learning because it defines explicit class-wise local prototypes, public-data-induced prototypes, external reference prototypes, and global fallback prototypes [2601.14746]. Its local class prototype is
\[
p_{k,c}^{t} = \frac{1}{|\mathcal{D}_{k,c}|} \sum_{x\in\mathcal{D}_{k,c}} F\!\left(x;\theta^{b,t}_{k},\theta^{f,t}\right),
\]
and its public-data-induced class prototype is
\[
p^{\mathrm{pub},t}_{k,c} = \frac{1}{|\mathcal{D}^{\mathrm{pub}}_c|} \sum_{x\in\mathcal{D}^{\mathrm{pub}}_c} F\!\left(x;\theta^{b,t}_{k},\theta^{f,t}\right)
\]
[2601.14746]. However, RefProtoFL’s sparsification module APUD applies magnitude-based top-\(K\) to shared adapter parameters,
\[
u_k^t = \left| \theta_k^{a,t} - \theta^{a,t} \right|,
\]
rather than to prototypes [2601.14746]. From a CPS perspective, this makes RefProtoFL adjacent but not direct.

FedProtoKD is similarly adjacent. It is prototype-based and introduces class-wise adaptive prototype margins to counter prototype margin shrinking under heterogeneous FL [2508.19009]. It defines local class prototypes as mean class features and replaces naïve server-side averaging with trainable server prototypes:
\[
\tilde{P}^c = \mathcal{F}(\hat{P}^c; \mathcal{F}_{\theta}),
\]
together with adaptive class-wise margin
\[
\xi^c(t) = \min_{c,c' \in [C], c' \neq c} ( \delta(Q^c_t, Q^{c'}_t), \zeta )
\]
[2508.19009]. This is not sparsification of prototype coordinates or class transmissions. Instead, it is a geometry-preserving refinement of class-wise prototype alignment.

YONO, in contrast, can be read as an extreme form of class-wise prototype compression in class-incremental learning [2305.16143]. It stores one condensed prototype \(p_k\) per class and updates it using an attentional mean-shift rule
\[
p_k\leftarrow (1-\lambda)p_k+\lambda\sum_{i:y_i=k} a_{k,i}\cdot\frac{z_i}{\|z_i\|_2},\quad p_k\leftarrow\frac{p_k}{\|p_k\|_2},
\]
with weights derived from cosine similarity [2305.16143]. YONO is not a federated communication protocol, but it demonstrates that extreme class-wise prototype compression can be effective when representation learning is jointly reshaped to support it.

Outside prototype communication, SGDS in class-incremental learning is relevant only by analogy. It computes class prototypes \(\mu_y\) by averaging image embeddings from a frozen backbone and uses them to decide whether new classes should reuse or avoid sparse activation subspaces [2601.21345]. What is sparsified there is the adapter input activation vector, not the prototype itself. Conditional Point Sparsification in SAM-based few-shot segmentation is unrelated to class-wise prototype sparsification and sparsifies matched prompt points instead [2602.05218].

## 7. Interpretation, strengths, and limitations

CPS, in its direct TinyProto form, has several clear strengths. It replaces dense prototype exchange with fixed-support sparse communication, reduces communication from \(\sum_i (K_i+K)d\) to \(\sum_i (K_i+K)s\), preserves full model capacity during training, supports heterogeneous architectures so long as clients produce prototypes in a shared \(\mathbb{R}^d\) space, and avoids pruning-specific client-side fine-tuning overhead [2507.04327]. The paper also claims that structured sparsification can improve class distinctiveness, suggesting that sparsity may be beneficial not only for compression but also for discrimination [2507.04327].

The main limitations are equally explicit. Mask design is not fully specified: the server initializes masks and pairwise Hamming distances are typically maximized, but the paper does not provide a detailed constructive algorithm [2507.04327]. Additional hyperparameters are introduced, including the support size \(s\), the regularization weight \(\lambda\), and the scaling parameter \(\mu\), increasing deployment complexity [2507.04327]. Heterogeneous architectures still require a compatible prototype dimensionality; in the reported experiments, \(d=500\) is fixed [2507.04327]. The convergence appendix requires a fixed-support sparsity assumption, indicating that theoretical treatment depends on stable class-wise supports [2507.04327].

A common misconception is to treat any prototype-based communication method with some sparse component as CPS. The literature surveyed here does not support that broad equivalence. RefProtoFL sparsifies adapter updates rather than prototypes [2601.14746]. FedProtoKD preserves class margins through trainable server prototypes rather than masked sparse supports [2508.19009]. SGDS structures activation supports rather than sparse prototypes [2601.21345]. Conditional Point Sparsification concerns SAM prompt points and is only an acronym collision [2602.05218]. In a narrower and technically precise sense, CPS refers to class-wise sparse prototype supports as formalized in TinyProto [2507.04327].

A broader implication of the surrounding literature is that prototype communication has at least three separable axes: coordinate sparsity, class-wise geometric alignment, and server-side prototype refinement. TinyProto emphasizes the first axis [2507.04327], RefProtoFL the second [2601.14746], and FedProtoKD the third [2508.19009]. This suggests that future CPS-style systems may benefit from combining fixed class-wise sparse supports with geometry-preserving class margins or external reference anchors. Such a synthesis is not yet specified in the cited works, but it is a plausible implication of their complementary design choices.

Source: https://www.emergentmind.com/topics/class-wise-prototype-sparsification-cps