TransFed: Transformer-Enhanced Federated Learning

• TransFed is a family of federated paradigms that integrates transformer designs and transfer mechanisms to improve privacy, adaptability, and efficiency.
• In its clearest form, TransFed employs a lightweight one-patch transformer with FedAvg for human activity recognition, achieving near 99% accuracy with minimal computational overhead.
• Other variants incorporate server-side hypernetworks and client-adaptive focal modulation to personalize models and manage non-IID data across diverse federated settings.

Searching arXiv for papers relevant to “TransFed” and closely related uses of the term. TransFed denotes a family of federated paradigms centered on transformer or transfer mechanisms, but the term is not used uniformly across the literature. In its clearest named usage, TransFed is a federated human activity recognition framework built around a lightweight one-patch transformer and FedAvg, designed for privacy-sensitive home-healthcare sensor data (Raza et al., 2021). In later work, the name also refers to a transformer-based federated learning framework with client-adaptive focal modulation generated by a server-side hypernetwork (Ashraf et al., 14 Aug 2025). Other papers use “TransFed” more broadly, as an umbrella for transformer-enhanced federated learning, federated transfer learning, transductive federation, or even transport for federation, rather than as a single canonical algorithm (Shi et al., 19 Aug 2025).

1. Terminological scope and usage

In the supplied literature, “TransFed” appears in multiple, partially overlapping senses. Some papers use it as a proper method name, some as shorthand for transformer-based federated learning, and others as a paradigm label for transfer-oriented or transport-oriented federation. This suggests that the term functions less as a standardized protocol name than as a family resemblance across methods that combine federation with transformers, transfer, or transductive structure.

Usage of “TransFed”	Representative paper	Defining characteristic
Named HAR framework	(Raza et al., 2021)	lightweight one-patch transformer trained with FedAvg
Personalized transformer FL	(Ashraf et al., 14 Aug 2025)	server-side hypernetwork generates client-specific focal modulation
Transformer-enhanced FL umbrella	(Shi et al., 19 Aug 2025)	transformer encoder inside FedProx with CKKS, PBCS, and XAI
Transformer-based VFL closest match	(Wu et al., 2024)	distributed transformer for multi-party fuzzy VFL
Transport-layer federation usage	(C. et al., 30 Jun 2026)	synchronization boundary with compile-time confinement and $O(\Delta)$ synchronization

The broadest explicit mapping appears in the Trans-XFed paper, which states that “TransFed” is a broader idea, namely transformer-enhanced FL, and that Trans-XFed is a concrete realization for supply chain credit risk with F1-based client selection, CKKS-secured aggregation, and post-hoc integrated gradients explanations (Shi et al., 19 Aug 2025). A closely related position appears in the Federated Transformer paper, which states that the name “TransFed” does not appear in that paper but that, if it is intended to mean transformer-based federated learning, FeT is the closest match for multi-party vertical settings with fuzzy linkages (Wu et al., 2024).

2. TransFed as a lightweight federated HAR framework

The most explicit method named TransFed is introduced in “Lightweight Transformer in Federated Setting for Human Activity Recognition” (Raza et al., 2021). Its target problem is human activity recognition in home healthcare, where centralized training on smart-home and body-sensor data raises privacy concerns and where CNNs and RNNs are described as computationally expensive, sequential, or limited in sequence modeling. The proposed response is a lightweight “one-patch” transformer coupled with federated training.

The HAR model uses tri-axial accelerometer, gyroscope, and magnetometer streams, giving $F=9$ features sampled at $115$ Hz. Sliding windows covering approximately $2$ seconds are reshaped into inputs of shape $140 \times 9$, which the paper reports as the best trade-off from grid search. The architecture uses two transformer layers, five attention heads, normalization sub-layers, residual additions, and a flatten-plus-softmax classifier head, for a total of 14,697 trainable parameters (Raza et al., 2021).

The core attention mechanism is standard scaled dot-product attention,

$$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

with multi-head composition

$$\text{MHA}(Q,K,V)=\text{Concat}(\text{head}_1,\dots,\text{head}_H)W^O,$$

where $H=5$. The “one-patch” design is not a ViT-style many-patch tokenization. Instead, the entire $140 \times 9$ window is treated as one semantic unit during patch encoding, after which self-attention operates over the temporal dimension within the window. The stated rationale is reduced complexity and preservation of natural one-dimensional temporal structure.

Federation uses five Raspberry Pi edge devices and one aggregation server. All five clients participate in each round, local training runs for 100 epochs per round with batch size 30, Adam, learning rate 0.01, and weight decay 0.001, and aggregation is standard FedAvg:

$$\mathbf{GM}^{r+1}=\sum_{k=1}^{K}\frac{n_k}{n}\mathbf{LM}_k^{r+1}.$$

The data are non-IID: for each client, one activity is underrepresented by 40–50% compared with the others. Communication is protected with SSL, and raw sensor data remain local, but the implementation does not use secure aggregation, differential privacy, or homomorphic encryption (Raza et al., 2021).

Empirically, the paper reports 98.74% overall accuracy for the federated global model on the new 15-class dataset, 99.14% for centralized training on the same dataset, and 98.89% on WISDM in centralized evaluation. The reported per-class results are near-perfect for many activities, with lower but still strong values for classes with similar motion patterns such as Swiping, Vacuuming, and Dusting (Raza et al., 2021).

3. TransFed as hypernetwork-personalized focal modulation

A distinct and later use of the name appears in “Generalizable Federated Learning using Client Adaptive Focal Modulation,” whose abstract describes TransFed as prior work that introduced a transformer-based FL framework with a learn-to-adapt hypernetwork generating personalized focal modulation layers per client (Ashraf et al., 14 Aug 2025). In this line, TransFed addresses personalization under severe heterogeneity, including non-IID data, cross-domain shifts, source-free domain adaptation, and cross-task settings.

The architecture keeps a transformer backbone shared while personalizing only the focal modulation projection parameters. For an embedded representation $F=9$0, the modulation projections are

$F=9$1

and the client-specific projection bundle is generated by a server-side hypernetwork,

$F=9$2

where $F=9$3 are hypernetwork weights and $F=9$4 is a learnable client embedding. The personalized model for client $F=9$5 is therefore $F=9$6, with $F=9$7 denoting shared backbone parameters. The client objective is written as

$F=9$8

and server aggregation updates only the shared part by weighted averaging, while the hypernetwork and client embeddings are updated on the server through the observed local changes in modulation parameters (Ashraf et al., 14 Aug 2025).

The paper’s theoretical discussion states that TransFed reduces the effective hypothesis-class complexity by replacing $F=9$9 separate personalized modulation stacks with a shared generator plus lightweight embeddings. Its extended version, AdaptFED, adds task-aware client embeddings, refined adaptation, tighter bounds, and an efficient low-rank hypernetwork conditioning variant to further reduce communication. The same paper reports broad empirical validation for the extension on RSNA Pneumonia, Kermany, CIFAR-10, CIFAR-100, Office-Caltech, and Shakespeare, but those numerical results are presented for AdaptFED rather than for the original TransFed alone (Ashraf et al., 14 Aug 2025).

4. Transformer-enhanced federated systems commonly grouped with TransFed

Several papers explicitly place their methods in a broader transformer-enhanced federated category that overlaps with TransFed.

System	Federated setting	Core mechanism
Trans-XFed	horizontal FL for supply chain credit assessment	transformer encoder + FedProx + CKKS + PBCS + IG
FedAdapter	FedNLP	adapters with progressive upgrade and trial-group profiling
FedRA	federated tuning of foundation models	random layer allocation with LoRA adapters
FeT	multi-party fuzzy VFL	distributed transformer + PE for identifiers + DP+MPC

Trans-XFed combines FedProx, CKKS homomorphic encryption, a lightweight transformer encoder, integrated gradients, attention analysis, weighted NLL, and a performance-based client selection strategy that ranks clients by local F1 and selects the top $115$0 with $115$1 (Shi et al., 19 Aug 2025). The experimental setup uses four clients, 21 tabular features, 50 communication rounds, and a separate test set of 17,664 records. Under weighted NLL, the paper reports recall 0.8889, precision 0.5961, F1 0.7137, and best round 10 for the minority class, outperforming FedProx and FedAvg variants using the same local transformer architecture (Shi et al., 19 Aug 2025).

FedAdapter addresses transformer fine-tuning in federated NLP by training only bottleneck adapters, progressively upgrading adapter depth and width, profiling future configurations through trial groups, and exploiting activation caching on clients (Cai et al., 2022). It reports time-to-accuracy reductions to no more than several hours, up to 155.5$115$2 faster than vanilla FedNLP and 48$115$3 faster than strong baselines, with 126.7$115$4 lower network traffic and 18.4$115$5 lower energy on average (Cai et al., 2022).

FedRA targets heterogeneous clients that cannot support the full transformer backbone. It randomly allocates layer subsets and fine-tunes only corresponding LoRA adapters, while the backbone remains frozen (Su et al., 2023). On feature-skew DomainNet with ViT, the paper reports average top-1 accuracy improving from the best baseline’s 55.97 to 63.90, a gain of +7.93, and in an extreme heterogeneity setting where all clients have 6 layers, from 20.58 to 70.38, a gain of +49.80 (Su et al., 2023).

FeT, the Federated Transformer for multi-party fuzzy VFL, distributes transformer encoders across parties, encodes fuzzy identifiers through learnable positional encodings, and secures local representations with differential privacy plus secure multi-party computation (Wu et al., 2024). It reports that FeT surpasses baseline models by up to 46% in terms of accuracy when scaled to 50 parties, and also improves performance and privacy in two-party fuzzy VFL settings (Wu et al., 2024).

5. Transfer-oriented, vertical, and transductive interpretations

A second broad interpretation of TransFed is federated transfer learning. TinyFedTL exemplifies this usage on resource-constrained microcontrollers: it freezes a pretrained feature extractor, trains only a small fully connected head on-device, and aggregates head parameters by FedAvg (Kopparapu et al., 2021). The implementation runs on an Arduino Nano 33 BLE Sense with 1 MB flash and 256 KB SRAM, uses a 514-parameter binary head for perf-mobilenet, occupies 210 KB dynamic memory and 657 KB program storage, and reports communication costs of more than 30 seconds one-way for more than 6000 bytes in the serial prototype (Kopparapu et al., 2021).

FedGTST treats transferability itself as the main federated objective and tunes cross-client Jacobian statistics rather than local-domain proxies (Ma et al., 2024). Each client computes a scalar Jacobian norm, the server broadcasts the maximum guide norm, and local training adds a regularizer of the form

$115$6

The paper argues that increasing the average Jacobian norm while reducing cross-client Jacobian variance tightens an explicit upper bound on target loss, and reports a 9.8% accuracy improvement over FedSR and 7.6% over FedIIR on CIFAR-10$115$7SVHN with LeNet (Ma et al., 2024).

FedLED extends the transfer-oriented meaning to unsupervised vertical federated transfer learning for equipment fault diagnosis (Shen et al., 2023). It assumes heterogeneous source and target feature spaces, no overlapping samples, and zero labels in the target domain, and combines source supervised loss, conditional adversarial alignment, and joint output-space alignment in a split-learning-style protocol. The paper describes FedLED as the first unsupervised vertical FTL equipment fault diagnosis method and reports up to 4.13 times higher diagnosis accuracy than state-of-the-art approaches in some settings (Shen et al., 2023).

MrTF pushes the term toward transductive federated learning rather than transformer design (Li et al., 2023). It assumes that the server has pre-available unlabeled test samples and refines aggregated models on that test set through stabilized teachers, rectified distillation, and clustered label refinery. The setup remains privacy-preserving in the usual FL sense because clients send only parameters, but the learning objective is explicitly transductive: the server incorporates structural information from the to-be-inferred data during training (Li et al., 2023).

6. Shared themes, misconceptions, and open issues

A common misconception is that TransFed denotes one universally accepted algorithm. The literature instead shows several distinct but related constructions. Some are transformer architectures for horizontal FL, some are transfer-learning or source-free adaptation schemes, some operate in vertical or fuzzy-linked regimes, and one systems paper even uses “TransFed” for federation transport rather than model training (C. et al., 30 Jun 2026). A plausible implication is that the stable core of the term is not a fixed optimization rule but the attempt to make federation more adaptive under heterogeneity, privacy constraints, and deployment asymmetries.

Privacy assumptions vary sharply across these uses. The original HAR TransFed keeps raw data local and uses SSL, but does not implement secure aggregation, differential privacy, or homomorphic encryption (Raza et al., 2021). Trans-XFed adds CKKS-secured aggregation and explainability, while also cautioning that “attention is not explanation” in a strict causal sense (Shi et al., 19 Aug 2025). FeT combines clipping, subsampling, Gaussian noise, and MPC-secured SplitAvg to protect party representations in fuzzy VFL (Wu et al., 2024). In the personalized focal-modulation line, client embeddings are explicitly noted as potentially encoding sensitive distributional signatures, and the paper recommends privacy-preserving construction such as DP noise or secure aggregation in practice (Ashraf et al., 14 Aug 2025).

The systems interpretation makes the boundary even clearer. FSTP treats “TransFed” as transport for federation and formalizes a synchronization agent whose output type set is closed, with compile-time confinement enforced by the Rust type system, contextual unlinkable identities, a Blocklace event substrate, and synchronization cost proportional to the symmetric difference $115$8 (C. et al., 30 Jun 2026). This is not a learning algorithm, but it shows that the label can extend from model design to the federation substrate itself.

Open issues recur across the family. Communication remains dominant on constrained hardware, as TinyFedTL’s serial prototype and FedAdapter’s network-dominated FL measurements illustrate (Kopparapu et al., 2021). Heterogeneity can still destabilize layer coverage or personalization quality, motivating designs such as PBCS, random layer allocation, and hypernetwork-generated modulation (Shi et al., 19 Aug 2025). Explainability is often post-hoc and partial, especially when attention weights are used as relational probes rather than causal evidence (Shi et al., 19 Aug 2025). The literature therefore suggests that TransFed is best understood as an evolving research descriptor: a cluster of methods that place transformer or transfer structure at the center of federated adaptation, while leaving the exact learning objective, privacy model, and systems assumptions specific to each paper.