Class-wise Prototype Sparsification (CPS)

• The paper introduces CPS as a fixed-support compression mechanism for prototype-based federated learning to reduce communication overhead by sparsifying class prototypes.
• CPS enforces class-wise support consistency by using shared binary masks, ensuring that clients and the server transmit aligned sparse prototype vectors.
• Empirical results indicate up to a 10× reduction in communication cost while maintaining or improving performance compared to dense prototype exchanges.

Searching arXiv for papers directly relevant to Class-wise Prototype Sparsification and closely related prototype-sparsification work. Class-wise Prototype Sparsification (CPS) is a structured sparsification scheme for class prototypes in prototype-based federated learning (PBFL), introduced in TinyProto as a mechanism for reducing communication overhead by assigning each class a fixed binary support mask and transmitting only the retained coordinates of that class’s prototype (Lee et al., 6 Jul 2025). In the formulation given in TinyProto, CPS acts on class-representative prototype vectors rather than on model weights or gradients, and its defining property is class-wise support consistency: clients and server share the same mask for a given class, so sparse prototypes remain aligned during aggregation (Lee et al., 6 Jul 2025). Closely related work has extended the broader design space around class-wise prototype communication, alignment, and representation control, although these methods generally do not implement prototype sparsification in the strict sense. RefProtoFL combines class-wise prototype alignment with sparsified adapter updates, but does not sparsify prototypes themselves (Wu et al., 21 Jan 2026). FedProtoKD refines class-wise prototypes through adaptive margin-preserving server-side learning rather than pruning or masking prototype dimensions (Hossen et al., 26 Aug 2025). In continual learning, YONO can be interpreted as an extreme form of class-wise prototype compression in which one condensed prototype is stored per class, but its objective is memory-efficient replay rather than communication-efficient federated aggregation (Kong et al., 2023). A superficially similar acronym appears in Conditional Point Sparsification for SAM-based few-shot segmentation, but that work is unrelated to prototype sparsification and instead sparsifies prompt points (Nie et al., 5 Feb 2026). A further adjacent line is SGDS, which uses class-mean prototypes only to organize sparse activation subspaces; it does not define sparse prototypes as the primary object (Liu et al., 29 Jan 2026).

1. Concept and scope

In TinyProto, CPS is defined as a class-wise, fixed-support prototype compression mechanism for PBFL (Lee et al., 6 Jul 2025). Let a class prototype be $\bar{c} \in \mathbb{R}^d$ and let a binary mask be

$m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$

CPS constructs a structured sparse prototype

$\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$

and a compressed prototype

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$

where

$s=\sum_{i=1}^d m_i.$

The structured sparse form preserves the original ambient dimension while zeroing inactive coordinates, whereas the compressed form contains only the retained entries and is the object communicated between clients and server (Lee et al., 6 Jul 2025).

The defining feature of CPS is that sparsity is not generic or unstructured. It is class-specific, fixed for a class, and shared across participating nodes (Lee et al., 6 Jul 2025). This distinguishes it from per-round top-$k$ sparsification, random coordinate dropping, quantization, model pruning, or gradient compression. The paper explicitly frames CPS as “allocating specific dimensions to class prototypes” and selecting “only a subset of elements where mask values are 1s” for communication (Lee et al., 6 Jul 2025).

A strict usage of the term therefore refers to prototype sparsification itself, not merely to sparse communication in a prototype-based system. This distinction matters in adjacent literature. RefProtoFL is prototype-driven but sparsifies shared adapter updates, not prototypes (Wu et al., 21 Jan 2026). FedProtoKD uses trainable server prototypes and adaptive class-wise margins, but does not impose sparse prototype supports (Hossen et al., 26 Aug 2025). YONO stores one prototype per class, which can be viewed as prototype compression at the object level, yet it does not introduce masked sparse coordinates or a PBFL communication protocol (Kong et al., 2023).

2. Federated learning setting and motivation

TinyProto introduces CPS in the setting of communication-efficient federated learning for resource-constrained and heterogeneous environments (Lee et al., 6 Jul 2025). In PBFL, clients exchange class prototypes—mean activations in a shared representation space—instead of model parameters. This is often cheaper than full-model communication, but the paper identifies a remaining bottleneck: prototype traffic still scales with both the number of classes and the feature dimension (Lee et al., 6 Jul 2025).

For prototype-based methods such as FedProto and FedTGP, the per-round communication cost is given as

$\sum_{i=1}^M (K_i + K)\times d,$

where $M$ is the number of clients, $K_i$ is the number of classes present on client $i$, $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$0 is the total number of classes, and $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$1 is prototype dimension (Lee et al., 6 Jul 2025). CPS replaces the dense dimension $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$2 by the retained support size $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$3, yielding

$m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$4

This makes the communication burden depend on sparse support size rather than full representation width (Lee et al., 6 Jul 2025).

The motivation is not only algebraic but also structural. TinyProto argues that penultimate-layer activations often exhibit sparse behavior under ReLU, yet this sparsity is inconsistent across clients because of model heterogeneity and data heterogeneity (Lee et al., 6 Jul 2025). As a result, naïvely aggregated dense prototypes tend to occupy most coordinates globally, even if individual client activations are locally sparse. CPS addresses this inconsistency by enforcing a shared per-class support pattern across clients, so that all prototype messages for class $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$5 lie in the same coordinate subset (Lee et al., 6 Jul 2025).

This suggests a broader interpretation: CPS is a coordination mechanism as much as a compression mechanism. Sparse communication becomes effective only when the support pattern is semantically aligned across participants. A plausible implication is that, without such coordination, prototype sparsity would be difficult to exploit in heterogeneous PBFL.

3. Formal construction and masking scheme

The local class prototype in TinyProto is defined as the mean feature activation in the penultimate or decision-layer representation space: $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$6 where $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$7 is the subset of client $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$8’s local dataset containing class $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$9, $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$0, and $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$1 is the feature extractor output (Lee et al., 6 Jul 2025). In standard PBFL, the global prototype is described either by weighted averaging,

$\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$2

or by simple averaging,

$\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$3

where $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$4 is the set of clients containing class $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$5 (Lee et al., 6 Jul 2025).

Under CPS, each class $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$6 is assigned a class-specific binary mask

$\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$7

The retained dimension count is

$\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$8

The paper states that the server initializes the masking vector set $\tilde{c} = S(\bar{c};m)=\bar{c}\odot m,$9 and shares masks with clients (Lee et al., 6 Jul 2025). These masks are fixed per class and shared between server and clients. The paper does not present a learning procedure for masks, nor does it specify a detailed mask-construction algorithm. It states that, to ensure inter-class distinctiveness, it typically maximizes pairwise Hamming distances between masking vectors (Lee et al., 6 Jul 2025). This means masks may be disjoint or overlapping; full disjointness is not a formal requirement.

The sparse prototype for class $$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$0 is

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$1

and the communicated compressed prototype is

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$2

The appendix further denotes the support set by

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$3

with fixed-support assumption

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$4

in the convergence analysis (Lee et al., 6 Jul 2025).

This formalism makes CPS a support-allocation method over prototype coordinates. The paper’s emphasis on Hamming-distance separation between masks indicates that inter-class distinctiveness is encoded partly through support geometry, not only through prototype values. This suggests that CPS seeks discrimination by combining semantic content and coordinate allocation.

4. Aggregation, scaling, and training pipeline

TinyProto couples CPS with adaptive prototype scaling (Lee et al., 6 Jul 2025). After computing dense local prototypes, a client compresses them using the shared mask: $$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$5 Before transmission, the client scales the compressed local prototype by the class sample count: $$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$6 The server then aggregates scaled sparse prototypes according to

$$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$7

Clients reconstruct the structured sparse prototype $$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$8 from $$\hat{c}=C(\bar{c};m)=(\bar{c}_i: m_i=1)\in\mathbb{R}^s,$$9 using the shared mask, and then apply global scaling in the regularization term: $s=\sum_{i=1}^d m_i.$0 where $s=\sum_{i=1}^d m_i.$1 is Euclidean distance (Lee et al., 6 Jul 2025). The local objective becomes

$s=\sum_{i=1}^d m_i.$2

The training procedure is PBFL augmented with CPS and scaling. Initially, clients train without regularization. Once global prototypes become available, each round follows the pattern: local training, dense prototype extraction, sparsification/compression, local scaling and upload, server aggregation, download and reconstruction, and local regularized training with $s=\sum_{i=1}^d m_i.$3 (Lee et al., 6 Jul 2025). Algorithm 1, “TinyProto-FP,” specifies that the server initializes mask vectors $s=\sum_{i=1}^d m_i.$4, sends masks to newly participating clients when needed, and aggregates returned $s=\sum_{i=1}^d m_i.$5 each round (Lee et al., 6 Jul 2025).

The scaling mechanism is presented as a way to preserve class-wise contribution importance while avoiding direct transmission of raw $s=\sum_{i=1}^d m_i.$6 as explicit metadata (Lee et al., 6 Jul 2025). The paper states that this “prevents privacy leakage by obfuscating $s=\sum_{i=1}^d m_i.$7 during communication and at the server level.” In context, scaling is therefore not separate from CPS but a compensatory device: sparse communication reduces cost, while scaling helps preserve aggregation quality under class imbalance.

5. Communication efficiency and empirical behavior

The central communication claim is that TinyProto changes prototype exchange from scaling with $s=\sum_{i=1}^d m_i.$8 to scaling with $s=\sum_{i=1}^d m_i.$9 (Lee et al., 6 Jul 2025). With $k$0 and $k$1, the paper states that this corresponds to a 90% compression rate. It further reports that TinyProto can achieve up to $k$2 reduction versus original PBFL methods and up to $k$3 lower communication than efficient baselines such as FedDistill in reported experiments (Lee et al., 6 Jul 2025).

The paper’s main quantitative results for $k$4, $k$5 are summarized below.

Dataset	Method	Accuracy / Communication
CIFAR-100	FedProto	$k$6, $k$7M
CIFAR-100	TinyProto-FP (50)	$k$8, $k$9M
CIFAR-100	FedTGP	$\sum_{i=1}^M (K_i + K)\times d,$0, $\sum_{i=1}^M (K_i + K)\times d,$1M
CIFAR-100	TinyProto-FT (50)	$\sum_{i=1}^M (K_i + K)\times d,$2, $\sum_{i=1}^M (K_i + K)\times d,$3M
TinyImageNet	FedProto	$\sum_{i=1}^M (K_i + K)\times d,$4, $\sum_{i=1}^M (K_i + K)\times d,$5M
TinyImageNet	TinyProto-FP (50)	$\sum_{i=1}^M (K_i + K)\times d,$6, $\sum_{i=1}^M (K_i + K)\times d,$7M
TinyImageNet	FedTGP	$\sum_{i=1}^M (K_i + K)\times d,$8, $\sum_{i=1}^M (K_i + K)\times d,$9M
TinyImageNet	TinyProto-FT (50)	$M$0, $M$1M

These results show that CPS-based communication can reduce dense PBFL prototype traffic by roughly $M$2 while maintaining or improving performance when combined with adaptive scaling (Lee et al., 6 Jul 2025). The paper also varies CPS dimension $M$3 for $M$4, reporting that reducing communicated dimensions via CPS does not necessarily hurt accuracy and can slightly improve it (Lee et al., 6 Jul 2025). Figure 1 further indicates that CPS can outperform reducing the decision-layer dimension to reach the same communication budget, because the full $M$5-dimensional model capacity is retained during training and only communication is compressed (Lee et al., 6 Jul 2025).

TinyProto is also evaluated under heterogeneous and non-IID settings. The setup uses Dirichlet partitioning with $M$6, with additional tests at $M$7, $M$8, and $M$9 clients (Lee et al., 6 Jul 2025). For CIFAR-100, TinyProto-FT (50) remains strongest with $K_i$0 at $K_i$1, $K_i$2 at $K_i$3, and $K_i$4 at $K_i$5 (Lee et al., 6 Jul 2025). These are not isolated CPS-only effects, because adaptive scaling is simultaneously active, but they support the practical viability of the CPS design under heterogeneity.

6. Relationship to adjacent prototype methods

The most direct neighboring literature does not generally implement class-wise prototype sparsification in the strict TinyProto sense. RefProtoFL is highly relevant to prototype-based federated learning because it defines explicit class-wise local prototypes, public-data-induced prototypes, external reference prototypes, and global fallback prototypes (Wu et al., 21 Jan 2026). Its local class prototype is

$K_i$6

and its public-data-induced class prototype is

$K_i$7

(Wu et al., 21 Jan 2026). However, RefProtoFL’s sparsification module APUD applies magnitude-based top-$K_i$8 to shared adapter parameters,

$K_i$9

rather than to prototypes (Wu et al., 21 Jan 2026). From a CPS perspective, this makes RefProtoFL adjacent but not direct.

FedProtoKD is similarly adjacent. It is prototype-based and introduces class-wise adaptive prototype margins to counter prototype margin shrinking under heterogeneous FL (Hossen et al., 26 Aug 2025). It defines local class prototypes as mean class features and replaces naïve server-side averaging with trainable server prototypes: $i$0 together with adaptive class-wise margin

$i$1

(Hossen et al., 26 Aug 2025). This is not sparsification of prototype coordinates or class transmissions. Instead, it is a geometry-preserving refinement of class-wise prototype alignment.

YONO, in contrast, can be read as an extreme form of class-wise prototype compression in class-incremental learning (Kong et al., 2023). It stores one condensed prototype $i$2 per class and updates it using an attentional mean-shift rule

$i$3

with weights derived from cosine similarity (Kong et al., 2023). YONO is not a federated communication protocol, but it demonstrates that extreme class-wise prototype compression can be effective when representation learning is jointly reshaped to support it.

Outside prototype communication, SGDS in class-incremental learning is relevant only by analogy. It computes class prototypes $i$4 by averaging image embeddings from a frozen backbone and uses them to decide whether new classes should reuse or avoid sparse activation subspaces (Liu et al., 29 Jan 2026). What is sparsified there is the adapter input activation vector, not the prototype itself. Conditional Point Sparsification in SAM-based few-shot segmentation is unrelated to class-wise prototype sparsification and sparsifies matched prompt points instead (Nie et al., 5 Feb 2026).

7. Interpretation, strengths, and limitations

CPS, in its direct TinyProto form, has several clear strengths. It replaces dense prototype exchange with fixed-support sparse communication, reduces communication from $i$5 to $i$6, preserves full model capacity during training, supports heterogeneous architectures so long as clients produce prototypes in a shared $i$7 space, and avoids pruning-specific client-side fine-tuning overhead (Lee et al., 6 Jul 2025). The paper also claims that structured sparsification can improve class distinctiveness, suggesting that sparsity may be beneficial not only for compression but also for discrimination (Lee et al., 6 Jul 2025).

The main limitations are equally explicit. Mask design is not fully specified: the server initializes masks and pairwise Hamming distances are typically maximized, but the paper does not provide a detailed constructive algorithm (Lee et al., 6 Jul 2025). Additional hyperparameters are introduced, including the support size $i$8, the regularization weight $i$9, and the scaling parameter $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$00, increasing deployment complexity (Lee et al., 6 Jul 2025). Heterogeneous architectures still require a compatible prototype dimensionality; in the reported experiments, $m = (m_1,m_2,\ldots,m_d)\in\{0,1\}^d.$01 is fixed (Lee et al., 6 Jul 2025). The convergence appendix requires a fixed-support sparsity assumption, indicating that theoretical treatment depends on stable class-wise supports (Lee et al., 6 Jul 2025).

A common misconception is to treat any prototype-based communication method with some sparse component as CPS. The literature surveyed here does not support that broad equivalence. RefProtoFL sparsifies adapter updates rather than prototypes (Wu et al., 21 Jan 2026). FedProtoKD preserves class margins through trainable server prototypes rather than masked sparse supports (Hossen et al., 26 Aug 2025). SGDS structures activation supports rather than sparse prototypes (Liu et al., 29 Jan 2026). Conditional Point Sparsification concerns SAM prompt points and is only an acronym collision (Nie et al., 5 Feb 2026). In a narrower and technically precise sense, CPS refers to class-wise sparse prototype supports as formalized in TinyProto (Lee et al., 6 Jul 2025).

A broader implication of the surrounding literature is that prototype communication has at least three separable axes: coordinate sparsity, class-wise geometric alignment, and server-side prototype refinement. TinyProto emphasizes the first axis (Lee et al., 6 Jul 2025), RefProtoFL the second (Wu et al., 21 Jan 2026), and FedProtoKD the third (Hossen et al., 26 Aug 2025). This suggests that future CPS-style systems may benefit from combining fixed class-wise sparse supports with geometry-preserving class margins or external reference anchors. Such an overview is not yet specified in the cited works, but it is a plausible implication of their complementary design choices.