FedReFT: Federated Representation Fine-Tuning

• FedReFT is a federated learning framework that fine-tunes lightweight intervention modules in the hidden representations of a frozen backbone, ensuring efficient model adaptation.
• It introduces All-But-Me aggregation using the geometric median, which reduces semantic misalignment by excluding self-updates during parameter aggregation.
• The method demonstrates high parameter efficiency and robust performance on tasks like commonsense and arithmetic reasoning, outperforming PEFT baselines with 7×–15× fewer trainable parameters.

FedReFT, short for Federated Representation Fine-Tuning, is a federated learning framework for adapting large pre-trained models by modifying hidden representations directly rather than updating model weights. In this formulation, each client fine-tunes lightweight intervention modules inserted into a frozen backbone, and the server coordinates collaboration through All-But-Me (ABM) aggregation, a personalized aggregation rule designed to reduce aggregation mismatch under heterogeneous client data, tasks, and resources. The method is presented as a representation-level alternative to parameter-efficient fine-tuning (PEFT) in federated settings, and is evaluated on commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE, where it is reported to outperform state-of-the-art PEFT baselines in federated learning while using substantially fewer trainable parameters (Siddika et al., 27 Aug 2025).

1. Conceptual setting and antecedents

FedReFT is situated at the intersection of two developments. The first is the rise of parameter-efficient fine-tuning, in which adaptation is restricted to a small subset of trainable parameters such as LoRA modules, adapters, prompts, or biases. Earlier federated studies of pre-trained Transformers organized the design space along these lines and showed that lightweight tuning can deliver fast convergence and low communication cost in federated learning (Chen et al., 2022). The second is an increasing focus on representations as the locus of transfer. A theoretical analysis of FedAvg with local fine-tuning showed that local updates can learn shared representations across heterogeneous clients in a multi-task setting, thereby helping generalization to new tasks (Collins et al., 2022).

FedReFT adopts the latter perspective explicitly. Instead of treating federation as the exchange of low-rank weight deltas or adapter parameters alone, it treats hidden-state intervention as the primitive of adaptation. The motivating claim is that internal representations encode rich semantic information, so steering them directly can be both lightweight and semantically meaningful. This reorientation matters in federated learning because client heterogeneity does not merely perturb optimization; it can also induce disagreement about the semantic effect of an update. FedReFT is therefore framed not only as a PEFT method, but as a federated mechanism for representation-level personalization under heterogeneous data distributions, model capacities, and computational resources (Siddika et al., 27 Aug 2025).

A common misconception is to regard FedReFT as simply another low-rank PEFT baseline. The central distinction is architectural and semantic: PEFT methods such as LoRA alter trainable weights attached to the model, whereas FedReFT applies learnable edits to hidden representations in a frozen model. Its novelty lies less in reducing parameter count per se than in changing the object that is tuned and aggregated.

2. Intervention architecture and representation-level adaptation

At the client side, FedReFT injects sparse intervention layers at specific hidden states of a frozen model. The standard design uses low-rank projection matrices $W \in \mathbb{R}^{r \times d}$, $R \in \mathbb{R}^{r \times d}$, and $b \in \mathbb{R}^r$, and trains only $\{W, R, b\}$ while leaving the backbone unchanged (Siddika et al., 27 Aug 2025).

This intervention design gives FedReFT two stated properties. First, it is parameter efficient because only a tiny fraction of total model parameters is trained. Second, it offers semantic control because the intervention operates on hidden representations rather than on full model weights. In the formulation described for FedReFT, the intervention layers are lightweight enough to be suitable for edge devices, and the resulting updates are intended to be semantically rich rather than merely syntactic perturbations of the weight space (Siddika et al., 27 Aug 2025).

The architectural consequences are important. Because the backbone remains frozen, the local optimization problem is concentrated in a small representation-editing subspace. This reduces memory and transmission burden, but it also changes the failure mode of federation. In weight-centric PEFT, aggregation error is often discussed in terms of parameter drift. In FedReFT, the corresponding concern is semantic misalignment: two clients may learn interventions that are individually meaningful but mutually incompatible when averaged. The method’s aggregation design follows directly from this observation.

3. Aggregation mismatch and All-But-Me aggregation

FedReFT identifies two coupled challenges for representation-level federation. The first is semantic misalignment and aggregation mismatch. Under heterogeneous clients, direct averaging of intervention parameters can cause what the paper describes as semantic collapse, because the meaning of a representation edit on one client may conflict with that of another. The second is the standard federated tension between personalization and global knowledge sharing (Siddika et al., 27 Aug 2025).

To address these issues, FedReFT introduces All-But-Me (ABM) aggregation. For a client $k$, the server aggregates parameters from all other clients except $k$, using the geometric median over the uploaded intervention parameters $X \in \{W, R, b\}$. The resulting aggregate is then partially incorporated into client $k$’s local parameters through an interpolation step with mixing coefficient $\alpha \in [0,1]$. The geometric median is defined as the point minimizing the sum of Euclidean distances to the client updates, and the paper states that it is computed in practice with Weiszfeld’s algorithm (Siddika et al., 27 Aug 2025).

ABM has three intended effects. It is personalized, because a client’s own update does not dominate the aggregate returned to that client. It is robust, because the geometric median is less sensitive than the mean to outliers or misaligned updates. And it is balanced, because the interpolation coefficient $\alpha$ allows the client to trade local specialization against shared knowledge. In the empirical ablation reported for FedReFT, ABM with geometric median consistently outperforms both FedAvg and mean-ABM, especially in heterogeneous settings (Siddika et al., 27 Aug 2025).

A second misconception is that ABM is merely a cosmetic variant of federated averaging. FedReFT’s formulation treats exclusion of the self-update and robust aggregation by geometric median as the key mechanism for preventing corruption of semantic alignment, not as a minor optimization detail.

4. Empirical results and parameter efficiency

FedReFT is evaluated on commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE, using models including LLaMA-7B/2/3/8B, RoBERTa-large, and TinyLlama-1B (Siddika et al., 27 Aug 2025). Across these settings, the method is reported to use only 0.01–0.05% of model parameters for fine-tuning and to achieve 7×–15× higher parameter efficiency than leading LoRA-based federated baselines (Siddika et al., 27 Aug 2025).

The paper reports the following representative benchmark results.

Setting	FedReFT	Comparator
Commonsense reasoning, LLaMA-3 3B, MT setup	rank 8, 0.0857% parameters, 75.66% avg. acc.	Fed-SB: 0.0884% parameters, 75.66% avg. acc.; FFA-LoRA: 0.76% parameters, 71.11% acc.
Arithmetic reasoning, GSM8K, LLaMA-3 8B	0.0622% parameters, 49.68% acc.	LoRA: 0.38% parameters, 46.23% acc.
GLUE, RoBERTa-large	0.015% parameters, 90.93 avg.	Best FL LoRA: 0.405–0.551% parameters, 89.39–90.43 avg.

The model-level parameter counts reported for FedReFT further illustrate the scale of the intervention modules. For LLaMA-7B, the total parameter count is listed as 6,738M with 2.1M trainable parameters, or 0.031% trainable. For LLaMA-3 8B, the count is 8,030M total with 2.1M trainable, or 0.026%. For RoBERTa-large, the count is 355M total with 0.049M trainable, or 0.0138% (Siddika et al., 27 Aug 2025).

These numbers position FedReFT as a method that combines competitive or superior task performance with unusually small trainable modules in federated settings. A plausible implication is that the representation-level intervention has enough expressivity to remain competitive even when its parameter budget is below that of LoRA-based alternatives.

5. Relation to adjacent federated fine-tuning methods

FedReFT belongs to a broader federated fine-tuning literature, but its representation-level focus differentiates it from several neighboring lines of work. FeDeRA improves LoRA through SVD-based initialization of low-rank adapters to stabilize federated updates under non-IID data (Yan et al., 2024). SLoRA also addresses the PEFT-versus-full-fine-tuning gap under heterogeneity by combining sparse fine-tuning with SVD-primed LoRA initialization (Babakniya et al., 2023). FedTT and FedTT+ use tensorized adapters and adaptive freezing of tensor factors to reduce communication and improve robustness to heterogeneity, achieving up to a 10× reduction in communication cost relative to competing federated PEFT approaches (Ghiasvand et al., 2024). All of these remain fundamentally parameter-centric: they compress, initialize, sparsify, or selectively freeze trainable weight modules.

By contrast, FedReFT shifts the object of adaptation from weight updates to hidden representations. This makes it closer in spirit to methods that emphasize feature alignment, although its mechanism is different from the explicit representation matching regularizer proposed in earlier federated learning for heterogeneous data (Mostafa, 2019). It is also distinct from SFPrompt, which combines split learning and prompt tuning so that resource-limited clients do not need to run the complete pre-trained model locally; SFPrompt reports competitive performance while using only 0.46% of local computing resources and 53% less communication cost than federated full fine-tuning (Cao et al., 2024). That comparison is important because it highlights a practical boundary: parameter efficiency and representation efficiency do not automatically solve local execution cost.

Later work in federated fine-tuning has also widened the heterogeneity model beyond what FedReFT directly targets. H2Tune studies hybrid heterogeneous federated fine-tuning, where clients differ simultaneously in model architecture and downstream task, and introduces matrix decomposition, layer alignment, and alternating task-knowledge disentanglement to cope with that setting (Guo et al., 30 Jul 2025). LEGEND adapts LoRA depth and rank across heterogeneous devices and reports 1.5–2.8× speedup with about 42.3% communication savings on 80 commercial devices (Liu et al., 2024). These works indicate that client heterogeneity can be architectural and systems-level, not only distributional or semantic.

FedReFT also sits within an evolving debate over how much communication is really necessary. A one-shot federated fine-tuning study argues that, for models larger than 1 billion parameters, a single communication round can achieve performance comparable to multi-round aggregation, and explicitly suggests that FedReFT could potentially be restructured as a one-shot process (Wang et al., 2024). At the opposite extreme, semantic consensus methods argue that collaboration should occur through generated outputs on public prompts rather than through parameter aggregation at all, reducing communication by orders of magnitude and accommodating heterogeneous black-box models (Abourayya et al., 12 May 2026). FedReFT therefore occupies a middle position: it departs from full weight tuning, but still remains within a parameterized, multi-round aggregation paradigm.

6. Practical significance, scope, and unresolved issues

The practical case for FedReFT is straightforward in the source description. Its intervention layers are computationally lightweight, require transmission of far fewer parameters, and are therefore well suited to edge and federated environments. Because raw data never leave the clients, the method is described as naturally compatible with privacy-preserving deployment, with prospects for integration with DP techniques. The paper also points to settings such as consumer devices, healthcare, and finance, where personalization and privacy are especially important (Siddika et al., 27 Aug 2025).

At the same time, adjacent literature identifies several concerns that are orthogonal to FedReFT’s core contribution. FedRef introduces a server-side reference model and Bayesian proximal regularization to mitigate catastrophic forgetting across rounds (Yoon et al., 29 Jun 2025). FedBE addresses forgetting through adaptive transformer block expansion and dynamic block allocation, reporting 12–74% higher accuracy retention on general tasks after fine-tuning and 1.9–3.1× faster convergence than prior federated fine-tuning methods (Huo et al., 6 Jun 2025). On the security side, AugMP shows that federated fine-tuning of LLMs remains vulnerable to model-manipulation attacks that can reduce global LLM accuracy by up to 26% while evading distance- and similarity-based defenses (Cai et al., 8 May 2026). These results do not contradict FedReFT; rather, they locate it within a larger systems problem.

This suggests that FedReFT’s main contribution is best understood as a solution to semantic alignment of representation interventions under federated heterogeneity, not as a complete answer to every issue in federated foundation-model adaptation. Its importance lies in demonstrating that hidden-state intervention can be federated effectively, provided aggregation is redesigned around personalization and robustness. Whether future systems combine FedReFT-style representation steering with one-shot communication, split execution, behavior-level consensus, stronger Byzantine robustness, or explicit forgetting mitigation remains an open direction indicated by the surrounding literature (Wang et al., 2024).