EfficientFSL: Efficient Few-Shot Learning

• The paper introduces EfficientFSL, a framework that minimizes learnable parameters and data transfer to achieve near-SOTA results in few-shot and federated split learning.
• It leverages model decomposition, auxiliary networks, and asynchronous updates to optimize communication, computation, and storage efficiency across devices.
• The approach extends to hardware-efficient designs and query-only tuning for Vision Transformers, enabling scalable, energy-efficient few-shot learning.

EfficientFSL encompasses a family of frameworks and architectural strategies for achieving communication, computation, and storage efficiency in distributed and few-shot learning paradigms. The term "EfficientFSL" refers to several independent developments targeting either federated split learning (FSL) systems (Mu et al., 2023, Mu et al., 21 Jul 2025, Penedo et al., 10 Jul 2025), end-to-end hardware-efficient few-shot learning pipelines (Yang et al., 2024), and extremely parameter-efficient query-only fine-tuning for few-shot classification with large Vision Transformers (ViTs) (Liao et al., 13 Jan 2026). These techniques share a common emphasis on minimizing learnable parameters, reducing inter-device data transfer, and leveraging architectural decomposition or auxiliary modules to achieve nearly state-of-the-art (SOTA) accuracy at a fraction of legacy resource consumption.

1. Core Concepts and Problem Setting

EfficientFSL models emerge from critical bottlenecks in distributed machine learning and few-shot learning. In classical Federated Learning (FL), clients train models locally on private data and periodically synchronize with a server, but this incurs high computational, memory, and communication costs—especially at the edge. Split Learning (SL) further reduces edge hardware requirements by partitioning model layers so that only the initial layers reside on the client, but traditional SL involves high data traffic ("smashed data" transmissions) and, in some FSL variants, imposes linear server storage growth with the number of clients.

EfficientFSL, as instantiated in the Communication and Storage Efficient Federated Split Learning (CSE-FSL) framework (Mu et al., 2023, Mu et al., 21 Jul 2025), resolves these limitations by:

• Partitioning the model at a judiciously chosen "cut layer" into client-side and server-side subnetworks.
• Equipping each client with a lightweight auxiliary network that provides a fully local surrogate loss, allowing several steps of unsupervised or semi-supervised local optimization.
• Having the server maintain only a single copy of server-side parameters updated asynchronously as clients send batched activations.
• Communicating activations and model parameters less frequently via mini-batch scheduling.

In a distinct domain, EfficientFSL is also realized as "query-only" tuning for ViTs, where only a minimal head is adapted per task, efficiently extracting information from a fixed backbone (Liao et al., 13 Jan 2026). Finally, a hardware-oriented EfficientFSL realization achieves on-chip gradient-free few-shot learning with clustered-weight CNNs and hyperdimensional computing (HDC) (Yang et al., 2024).

2. System Architecture and Algorithms

2.1 Federated Split Learning Variants

Model Decomposition: A typical global neural network is split into client-side parameters $\theta_{(c)}$ (first layers) and server-side parameters $\theta_{(s)}$ (remaining layers). Each client device holds $\theta_{(c)}$, while the server hosts a single $\theta_{(s)}$. In EfficientFSL, each client also maintains a small auxiliary network $a$ that enables local loss computation:

$$F_c(\theta_{(c)}, a) = \frac{1}{N} \sum_{i=1}^N \mathbb{E}_{z\sim D_i} [\ell_c(\theta_{(c)}, a; z)]$$

Training Protocol (summarized):

• Clients perform $h$ mini-batch forward-backward steps over $(\theta_{(c)}, a)$ on-device, using only local losses.
• After $h$ steps, clients send “smashed data”—activations at the cut layer plus labels—to the server.
• The server computes its own forward-backward step over $\theta_{(s)}$ for each incoming batch, with updates processed immediately (first-come, first-serve).
• Every $\theta_{(s)}$0 server updates, federated averaging (FedAvg) aggregates client parameters and the auxiliary net.

Pseudocode snippet (see (Mu et al., 2023)):

$\theta_{(c)}$6

2.2 Secure and Decentralized Orchestration

HLF-FSL implements EfficientFSL atop Hyperledger Fabric, coordinating FSL without a single point of failure. Smart contracts ("chaincode") manage protocol steps—model and data hash registration, updates, and aggregation—while all raw data and activations traverse secure, off-ledger channels (Penedo et al., 10 Jul 2025). Privacy is strictly enforced via transient fields and private data collections; only cryptographic references or hashes are committed to the blockchain.

2.3 Hardware-Efficient Few-Shot Learning

EfficientFSL (FSL-HDnn) employs:

• CNNs with quantized weight-clustering (e.g., $\theta_{(s)}$1 centroids per filter via k-means) to reduce multiply-accumulate (MAC) operations and parameter storage by $\theta_{(s)}$2 and $\theta_{(s)}$3, respectively.
• A hyperdimensional classifier with fast bitwise (XOR, popcount) operations replaces floating-point MACs for class assignment.
• On-chip, gradient-free learning via integer hypervector updates enables real-time, memory-efficient FSL (Yang et al., 2024).

2.4 Query-Only Fine-Tuning for Transformers

The EfficientFSL transformer approach freezes all ViT backbone weights and learns a small trainable head:

• Forward Block: Injects trainable prompts and projections at each layer’s input, then computes attention/MLP with all new weights isolated in a "query" head.
• Combine Block: Fuses $\theta_{(s)}$4 multi-layer features adaptively by weighting and alignment for robust representation.
• Support-Query Attention: Realigns class prototypes with the query feature distribution for improved generalization. Total trainable parameters are reduced to 2.9–5.8% of backbone size, yet achieve SOTA few-shot accuracy (Liao et al., 13 Jan 2026).

3. Formal Guarantees and Theoretical Properties

For the communication- and storage-efficient versions:

• Assume $\theta_{(s)}$5-smoothness, unbiased gradients with bounded variance, diminishing step sizes.
• Let $\theta_{(s)}$6.
• Then, expected squared gradient norms on both client and server sides decay as $\theta_{(s)}$7 (see propositions in (Mu et al., 2023, Mu et al., 21 Jul 2025)).

Convergence rate bounds incorporate an additional drift term $\theta_{(s)}$8 representing client-to-global model mismatch due to infrequent communication. When learning rates and parameter split schedules are suitably tuned, this term is dominated and convergence is assured even under non-convex objectives.

4. Empirical Evaluation and Comparative Performance

Federated Split Learning Benchmarks:

For CIFAR-10 with 5 clients (IID partition, 200 epochs) (Mu et al., 2023, Mu et al., 21 Jul 2025):

Method	Accuracy (%)	Up+Down Comm. (GB)	Server Storage (M params)
FSL_MC	~80.6	172.5	5.3
FSL_OC	~73.7	172.5	1.5
FSL_AN	~77.8	94.0	5.5
CSE-FSL (h=5)	~76.5	18.1	1.6
CSE-FSL (h=10)	~75.8	9.6	1.6

On F-EMNIST (partial, non-IID), CSE-FSL achieves 70–72% after ∼4,000 rounds at dramatically reduced communication volume (Mu et al., 2023).

Decentralized EfficientFSL:

On CIFAR-10, HLF-FSL matches centralized FSL accuracy (94.14%) but cuts per-epoch training time (30 min vs. over an hour for Ethereum-based approaches) and eliminates gas fees and single points of failure (Penedo et al., 10 Jul 2025).

Hardware-Efficient:

FSL-HDnn exhibits 5.7 TOPS/W (feature extraction) and 0.78 TOPS/W (FSL classification), improving energy efficiency by 2.6×–6.6× over previous designs (Yang et al., 2024).

ViT Few-Shot Classifier:

EfficientFSL achieves 97.4% (1-shot) and 99.05% (5-shot) on miniImageNet (ViT-S), universally outperforming LoRA, AdaptFormer, and other PETL baselines with only 1.25–2.48 M trainable parameters. Ablations demonstrate necessity of each head module for SOTA results (Liao et al., 13 Jan 2026).

5. Trade-Offs, Limitations, and Best Practices

Parameter Split Point ($\theta_{(s)}$9):

• Larger $\theta_{(c)}$0 means fewer smashed-data communications, lower bandwidth cost, but may provoke model drift and slow convergence on non-IID distributions. Empirical sweet spots are $\theta_{(c)}$1–$\theta_{(c)}$2 depending on data and model depth (Mu et al., 2023, Mu et al., 21 Jul 2025).

Auxiliary Network Design:

• Small MLPs (2–4 layers) or shallow 1x1 CNNs suffice; smaller $\theta_{(c)}$3 reduces compute while approximating server gradient signals is critical for performance.

Failure Modes:

• Extreme non-IID settings or small client datasets can cause drift term $\theta_{(c)}$4 to dominate. Solutions include lowering $\theta_{(c)}$5 or more frequent global aggregations.

Enterprise and Privacy:

• HLF-FSL’s fully decentralized logic, with Fabric channels and private data collections, offers strong privacy and no single point of attack compared to Ethereum and legacy FSL (Penedo et al., 10 Jul 2025).

Transformer Fine-Tuning:

• Query-only EfficientFSL enables massive reduction in learnable parameters with no drop in accuracy, and block ablations affirm the necessity of learnable prompts, adaptive combination, and SQ prototype refinement (Liao et al., 13 Jan 2026).

6. Significance and Impact

EfficientFSL demonstrates that careful architectural decomposition—via split learning, minimal local loss proxies, selective communication, and modular fine-tuning heads—permits SOTA distributed, few-shot, and privacy-preserving learning at vastly lower communication, computation, and storage cost. These frameworks are directly relevant for edge/cloud ML deployment, secure cross-enterprise collaboration, and for rapidly customizing large transformers to low-resource domains.

The methodology generalizes: auxiliary networks, asynchronous parameter updates, blockchain-based consensus, and gradient-free HDC offer orthogonal levers to tackle similar bottlenecks in emerging ML systems. In hardware, EfficientFSL approaches unlock true on-chip, embedded few-shot learning with energy and speed unattainable by conventional architectures. In large-model few-shot applications, parameter isolation via query-only heads ensures scalability and minimizes overfitting, establishing new reference points for fairness, cost, and empirical accuracy (Mu et al., 2023, Penedo et al., 10 Jul 2025, Mu et al., 21 Jul 2025, Yang et al., 2024, Liao et al., 13 Jan 2026).