TinyProto: Efficient Federated Learning

• TinyProto is a prototype-based federated learning framework that replaces dense class prototypes with structured sparse prototypes to reduce communication overhead.
• It employs adaptive prototype scaling to recover count-weighted aggregation without transmitting explicit client counts, enhancing performance in heterogeneous settings.
• Experiments on CIFAR and TinyImageNet demonstrate up to 10× communication reduction while maintaining or improving accuracy compared to baseline methods.

Searching arXiv for the TinyProto paper and closely related prototype-based federated learning work. TinyProto is a communication-efficient prototype-based federated learning (PBFL) framework designed for resource-constrained and heterogeneous environments. It replaces dense class prototypes with class-wise structured sparse prototypes and augments aggregation with adaptive prototype scaling, targeting the communication bottleneck that arises when clients exchange per-class mean activations in a shared $d$-dimensional penultimate feature space (Lee et al., 6 Jul 2025). In the reported experiments, TinyProto reduces communication costs by up to $4\times$ compared to efficient baselines and by up to $10\times$ relative to original PBFL configurations, while maintaining or improving accuracy, especially in larger-class and heterogeneous settings (Lee et al., 6 Jul 2025).

1. Problem setting and motivation

In PBFL, a client $i$ uses a feature extractor $f_i(\cdot)$ with output dimension $d$, and for each class $c \in \{1,\dots,C\}$ forms a local prototype $p_{i,c} \in \mathbb{R}^d$, typically the mean activation in the penultimate layer. The server aggregates these into global prototypes $p_c \in \mathbb{R}^d$. This reduces communication relative to full model exchange, but the per-round cost still scales with $C \cdot d$, because each class contributes a length-$4\times$0 prototype on both uplink and downlink (Lee et al., 6 Jul 2025).

The TinyProto formulation is motivated by two empirical observations. First, decision-layer features under ReLU are sparse and exhibit many “dead units” (approximately $4\times$1 per class empirically), so local prototypes already contain many zeros. Second, heterogeneity across clients, in both data and model, makes zero locations inconsistent, so naïve global aggregation fills in most coordinates and the aggregate prototype remains dense. The framework also addresses a second limitation of prototype aggregation: PBFL often uses simple averaging to avoid transmitting per-class client counts $4\times$2, but that choice sacrifices fidelity because client importance is not reflected (Lee et al., 6 Jul 2025).

These observations define the central problem that TinyProto addresses. Dense prototypes are structurally wasteful when the feature geometry is already sparse, and simple averaging under heterogeneous class frequencies can degrade prototype quality. TinyProto therefore imposes consistent sparsity patterns across clients and restores class-frequency weighting without directly sending $4\times$3 as metadata (Lee et al., 6 Jul 2025).

2. Class-wise Prototype Sparsification

The core mechanism is Class-wise Prototype Sparsification (CPS). For each class $4\times$4, TinyProto defines a fixed support $4\times$5 of size $4\times$6, together with a binary mask $4\times$7, where $4\times$8 iff $4\times$9. The structured sparse prototype is

$10\times$0

and the compressed prototype is the coordinate subvector restricted to $10\times$1,

$10\times$2

Clients and server share these class-specific masks once, so subsequent rounds transmit only the $10\times$3 selected values rather than all $10\times$4 coordinates (Lee et al., 6 Jul 2025).

A defining property of CPS is that zero locations are consistent across clients. This is the mechanism by which sparsity survives aggregation: each client sends values only on the same support $10\times$5 for class $10\times$6, and the server aggregates only those coordinates. In contrast, if each client were sparse in different coordinates, global averaging would refill the vector and destroy sparsity. The paper states that masks are typically designed to maximize pairwise Hamming distance between $10\times$7. Disjoint supports $10\times$8 are described as a simple, strong choice, while overlapping supports are allowed provided that masks remain sufficiently separated (Lee et al., 6 Jul 2025).

This design is structured rather than adaptive at each round. A plausible implication is that TinyProto trades per-round optimization freedom for deterministic communication cost and zero index overhead during training. The reported practical guidance is to start with $10\times$9, corresponding to $i$0 compression, and to prefer disjoint or near-disjoint supports when feasible (Lee et al., 6 Jul 2025).

3. Prototype aggregation, adaptive scaling, and local training

TinyProto retains the standard prototype definition at the client. If $i$1 is the subset of local samples with label $i$2, and $i$3, then the local class prototype is

$i$4

The count-weighted global prototype would be

$i$5

but many PBFL implementations replace this with simple averaging,

$i$6

to avoid sending $i$7 (Lee et al., 6 Jul 2025).

TinyProto’s adaptive prototype scaling recovers the effect of importance weighting without explicitly transmitting counts. Each client multiplies its compressed prototype by the local class count before uplink and sends

$i$8

The server then performs simple averaging on these scaled values,

$i$9

and reconstructs $f_i(\cdot)$0 by placing $f_i(\cdot)$1 back at $f_i(\cdot)$2 and zeros elsewhere. The paper describes this as achieving the effect of count-weighted aggregation without explicitly transmitting $f_i(\cdot)$3 as metadata (Lee et al., 6 Jul 2025).

A second scaling step is global. The server scales reconstructed global prototypes by a hyperparameter $f_i(\cdot)$4, and clients use $f_i(\cdot)$5 in prototype regularization. Local optimization is

$f_i(\cdot)$6

with

$f_i(\cdot)$7

where $f_i(\cdot)$8 is Euclidean distance. In the reported experiments, $f_i(\cdot)$9, and $d$0 is dataset-dependent: $d$1 for CIFAR-10 and $d$2 for CIFAR-100 and TinyImageNet (Lee et al., 6 Jul 2025). Prediction for PBFL methods uses nearest-prototype classification,

$d$3

Because only prototypes are exchanged, TinyProto is architecture-agnostic provided that clients emit features in a common $d$4-dimensional space. Different backbones are therefore supported without shared weights. The paper notes that if some clients have different native feature dimensions $d$5, a lightweight projection to a common $d$6 can be added, although TinyProto does not prescribe a specific projection operator (Lee et al., 6 Jul 2025).

4. Communication scaling and computational profile

The communication analysis is one of TinyProto’s main contributions. With $d$7 bytes per float, the baseline PBFL byte cost is written as

$d$8

which the paper identifies as the byte version of the parameter-count expression for dense-prototype methods. This notation suggests that $d$9 denotes the number of classes represented on client $c \in \{1,\dots,C\}$0 (Lee et al., 6 Jul 2025).

Under CPS, only $c \in \{1,\dots,C\}$1 coordinates per class are transmitted, so the per-round cost becomes

$c \in \{1,\dots,C\}$2

The compression ratio is therefore approximately

$c \in \{1,\dots,C\}$3

When $c \in \{1,\dots,C\}$4, the paper reports $c \in \{1,\dots,C\}$5 relative to original PBFL (Lee et al., 6 Jul 2025).

TinyProto avoids a common source of sparse-communication overhead: per-round index transmission. Because masks are pre-shared once, the round-wise index overhead is zero. The one-time mask broadcast cost is $c \in \{1,\dots,C\}$6 bits, amortized across rounds. On the client, prototype computation remains $c \in \{1,\dots,C\}$7 because prototypes are still mean-pooled over the full feature space; CPS compression is then only a fixed subselect of size $c \in \{1,\dots,C\}$8, i.e., $c \in \{1,\dots,C\}$9 per class, with no top-$p_{i,c} \in \mathbb{R}^d$0 search, auxiliary pruning, or additional optimization. On the server, reconstruction is $p_{i,c} \in \mathbb{R}^d$1 (Lee et al., 6 Jul 2025).

This distinction is important for interpretation. TinyProto compresses communication while preserving the full training-time and inference-time representation dimension $p_{i,c} \in \mathbb{R}^d$2. The paper’s ablations state that CPS consistently outperforms shrinking the actual layer width to match the same communication budget. The stated reason is that TinyProto keeps full $p_{i,c} \in \mathbb{R}^d$3 capacity in training and inference while communicating only $p_{i,c} \in \mathbb{R}^d$4 coordinates; reducing $p_{i,c} \in \mathbb{R}^d$5 directly harms representational power (Lee et al., 6 Jul 2025).

5. Experimental results and empirical behavior

The reported evaluation uses CIFAR-10 ($p_{i,c} \in \mathbb{R}^d$6), CIFAR-100 ($p_{i,c} \in \mathbb{R}^d$7), and TinyImageNet ($p_{i,c} \in \mathbb{R}^d$8); backbones include ResNet-8, EfficientNet, ShuffleNet v2, and MobileNet v2, all with global average pooling and prototype dimension $p_{i,c} \in \mathbb{R}^d$9. The federated configuration uses $p_c \in \mathbb{R}^d$0 clients, all participating per round, over 300 rounds, with learning rate $p_c \in \mathbb{R}^d$1, batch size $p_c \in \mathbb{R}^d$2, and one local epoch per round. Data heterogeneity is generated with Dirichlet $p_c \in \mathbb{R}^d$3, with additional tests at $p_c \in \mathbb{R}^d$4 and $p_c \in \mathbb{R}^d$5. The paper evaluates TinyProto-FP, integrated into FedProto, and TinyProto-FT, integrated into FedTGP/ACL (Lee et al., 6 Jul 2025).

At $p_c \in \mathbb{R}^d$6 and $p_c \in \mathbb{R}^d$7, corresponding to $p_c \in \mathbb{R}^d$8 compression, the reported communication and accuracy are as follows. On CIFAR-10, FedProto attains $p_c \in \mathbb{R}^d$9 with communication $C \cdot d$0, FedTGP $C \cdot d$1 with $C \cdot d$2, FedDistill $C \cdot d$3 with $C \cdot d$4, TinyProto-FP $C \cdot d$5 with $C \cdot d$6, and TinyProto-FT $C \cdot d$7 with $C \cdot d$8. On CIFAR-100, FedProto attains $C \cdot d$9 with $4\times$00, FedTGP $4\times$01 with $4\times$02, FedDistill $4\times$03 with $4\times$04, TinyProto-FP $4\times$05 with $4\times$06, and TinyProto-FT $4\times$07 with $4\times$08. On TinyImageNet, FedProto attains $4\times$09 with $4\times$10, FedTGP $4\times$11 with $4\times$12, FedDistill $4\times$13 with $4\times$14, TinyProto-FP $4\times$15 with $4\times$16, and TinyProto-FT $4\times$17 with $4\times$18 (Lee et al., 6 Jul 2025).

The paper highlights two specific communication reductions. TinyProto-FT achieves approximately $4\times$19 reduction relative to FedProto on CIFAR-100, from $4\times$20 to $4\times$21 million parameters per round, and approximately $4\times$22 reduction relative to FedDistill on TinyImageNet, from $4\times$23 to $4\times$24 (Lee et al., 6 Jul 2025). These are the central reported efficiency comparisons.

Robustness experiments further emphasize heterogeneous settings. On CIFAR-100 with $4\times$25, TinyProto-FT reaches $4\times$26, ahead of TinyProto-FP at $4\times$27, FedDistill at $4\times$28, and FedTGP at $4\times$29. With $4\times$30, TinyProto-FT reaches $4\times$31, while TinyProto-FP reaches $4\times$32 and FedTGP $4\times$33. With $4\times$34, TinyProto-FT reaches $4\times$35, described as best among PBFL-type methods, and TinyProto-FP reaches $4\times$36 (Lee et al., 6 Jul 2025).

Taken together, the results indicate that TinyProto is most advantageous when $4\times$37 and $4\times$38 are large enough that dense-prototype exchange becomes expensive. The paper’s own summary recommends it when communication is the primary constraint, when models are heterogeneous but can agree on a shared embedding dimension, and when the raw $4\times$39 cost of PBFL is otherwise prohibitive (Lee et al., 6 Jul 2025).

6. Theoretical properties, limitations, and privacy considerations

TinyProto extends FedProto’s convergence analysis by introducing a fixed-support, non-expansive sparsification operator $4\times$40 for each class. The stated assumption is that for class $4\times$41, $4\times$42 maintains a fixed support $4\times$43 and is non-expansive:

$4\times$44

Under this assumption, the components of FedProto’s Lemma 2 that involve global prototypes are modified by inserting $4\times$45, and because masking is linear on fixed locations, the bound is preserved. The paper gives the key inequality

$4\times$46

and states that this supports carrying over FedProto’s non-convex convergence rate to CPS (Lee et al., 6 Jul 2025).

The framework’s limitations are explicit. Even with $4\times$47, total communication still grows with $4\times$48, so very large class counts can make mask storage and broadcast overhead non-trivial, and disjoint support allocation may become impossible when $4\times$49. Extreme class imbalance or rapid distribution drift can destabilize distance-based training because local scaling depends on $4\times$50; the paper states that server-side normalization and $4\times$51 scaling mitigate this, but further adaptive normalization may be helpful. Fixed supports $4\times$52 can also become stale if informative feature directions drift substantially over training (Lee et al., 6 Jul 2025).

Mask overlap is another design risk. If supports overlap excessively, inter-class discriminativeness can degrade; the recommended mitigation is to maintain large pairwise Hamming distances between masks. Heterogeneous models with differing native feature dimensions $4\times$53 require an added projection to a common $4\times$54, which the method allows but does not specify in detail. The paper also notes that class-wise prototypes may leak label distribution information. Proposed mitigations include Gaussian or Laplace noise on prototypes, differential privacy accounting for the scaling step, and secure aggregation (Lee et al., 6 Jul 2025).

Future directions follow naturally from these constraints. The paper identifies adaptive support allocation, combination with quantization or sketching such as randomized projections, and privacy-preserving variants that integrate differential privacy noise and secure aggregation of compressed vectors (Lee et al., 6 Jul 2025).

7. Other technical usage of the name

The name “TinyProto” also appears in a distinct systems-design context derived from the Bebop serialization and RPC work reported in "Simplicity Scales" (Sampson et al., 4 Mar 2026). In that usage, TinyProto denotes a proposed minimal, high-performance serialization/RPC system that mirrors Bebop’s fixed-width wire rules, fixed $4\times$55-byte length prefixes, packed structs, and transport-independent RPC framing. The guiding principle is that every data type uses a fixed number of bytes: for example, a $4\times$56-bit integer is always $4\times$57 bytes, a float64 is always $4\times$58 bytes, and length prefixes are always $4\times$59 bytes. The stated purpose is to eliminate data-dependent branches in decoding, replacing varint and text parsing with single loads, pointer assignment, and fixed strides (Sampson et al., 4 Mar 2026).

In that systems context, the proposed wire format uses little-endian fixed sizes for primitive and extended numeric types, strings encoded as $4\times$60-byte byte-length plus UTF-8 bytes plus a $4\times$61-byte NUL terminator, arrays encoded as a $4\times$62-byte count followed by contiguous elements, structs encoded positionally with no tags or padding, and messages encoded as a $4\times$63-byte total length followed by a stream of $4\times$64-byte tags and a single $4\times$65 end marker. Unknown tags are skipped deterministically using fixed widths or fixed-width length prefixes, preserving message evolvability without per-byte continuation checks (Sampson et al., 4 Mar 2026).

The same source also specifies a transport-agnostic RPC layer with a fixed $4\times$66-byte frame header composed of length ($4\times$67), flags (byte), and stream identifier ($4\times$68); method dispatch uses a $4\times$69-byte hash of “/ServiceName/MethodName”, and the protocol supports HTTP/1.1, HTTP/2, HTTP/3, TCP, WebSocket, and IPC. Batch pipelining is a central feature: a BatchCall carries dependency information through input_from, the server builds a DAG of dependent calls, partitions it into layers, and executes layers concurrently, collapsing $4\times$70 dependent round trips into one. Cursors, absolute deadlines, and async futures are also described (Sampson et al., 4 Mar 2026).

The reported performance claims for Bebop motivate that usage of the name. Across $4\times$71 decode workloads, Bebop decodes $4\times$72–$4\times$73 faster than Protocol Buffers; on a $4\times$74-dimension embedding vector, it decodes in $4\times$75 ns versus $4\times$76 ns for Protocol Buffers and $4\times$77 $4\times$78s for simdjson; and for records above approximately $4\times$79 KB, cold-cache utilization reaches $4\times$80 of peak DRAM bandwidth, summarized as $4\times$81 (Sampson et al., 4 Mar 2026). This systems-oriented “TinyProto” is unrelated in purpose to the federated-learning framework, but both usages share a common design theme: reducing overhead by constraining representation structure rather than adding runtime complexity.