---
title: 'FedReFT: Federated Representation Fine-Tuning'
url: https://www.emergentmind.com/topics/federated-representation-fine-tuning-fedreft
type: topic
---

# FedReFT: Federated Representation Fine-Tuning

FedReFT, short for **Federated Representation Fine-Tuning**, is a federated learning framework for adapting large pre-trained models by modifying hidden representations directly rather than updating model weights. In this formulation, each client fine-tunes lightweight intervention modules inserted into a frozen backbone, and the server coordinates collaboration through **All-But-Me (ABM) aggregation**, a personalized aggregation rule designed to reduce aggregation mismatch under heterogeneous client data, tasks, and resources. The method is presented as a representation-level alternative to parameter-efficient fine-tuning (PEFT) in federated settings, and is evaluated on commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE, where it is reported to outperform state-of-the-art PEFT baselines in federated learning while using substantially fewer trainable parameters [2508.20295].

## 1. Conceptual setting and antecedents

FedReFT is situated at the intersection of two developments. The first is the rise of **parameter-efficient fine-tuning**, in which adaptation is restricted to a small subset of trainable parameters such as LoRA modules, adapters, prompts, or biases. Earlier federated studies of pre-trained Transformers organized the design space along these lines and showed that lightweight tuning can deliver fast convergence and low communication cost in federated learning [2211.08025]. The second is an increasing focus on **representations** as the locus of transfer. A theoretical analysis of FedAvg with local fine-tuning showed that local updates can learn shared representations across heterogeneous clients in a multi-task setting, thereby helping generalization to new tasks [2205.13692].

FedReFT adopts the latter perspective explicitly. Instead of treating federation as the exchange of low-rank weight deltas or adapter parameters alone, it treats hidden-state intervention as the primitive of adaptation. The motivating claim is that internal representations encode rich semantic information, so steering them directly can be both lightweight and semantically meaningful. This reorientation matters in federated learning because client heterogeneity does not merely perturb optimization; it can also induce disagreement about the semantic effect of an update. FedReFT is therefore framed not only as a PEFT method, but as a federated mechanism for **representation-level personalization** under heterogeneous data distributions, model capacities, and computational resources [2508.20295].

A common misconception is to regard FedReFT as simply another low-rank PEFT baseline. The central distinction is architectural and semantic: PEFT methods such as LoRA alter trainable weights attached to the model, whereas FedReFT applies learnable edits to hidden representations in a frozen model. Its novelty lies less in reducing parameter count per se than in changing the object that is tuned and aggregated.

## 2. Intervention architecture and representation-level adaptation

At the client side, FedReFT injects **sparse intervention layers** at specific hidden states of a frozen model. The standard design uses low-rank projection matrices
\(W \in \mathbb{R}^{r \times d}\),
\(R \in \mathbb{R}^{r \times d}\),
and \(b \in \mathbb{R}^r\),
and trains only \(\{W, R, b\}\) while leaving the backbone unchanged [2508.20295].

This intervention design gives FedReFT two stated properties. First, it is **parameter efficient** because only a tiny fraction of total model parameters is trained. Second, it offers **semantic control** because the intervention operates on hidden representations rather than on full model weights. In the formulation described for FedReFT, the intervention layers are lightweight enough to be suitable for edge devices, and the resulting updates are intended to be semantically rich rather than merely syntactic perturbations of the weight space [2508.20295].

The architectural consequences are important. Because the backbone remains frozen, the local optimization problem is concentrated in a small representation-editing subspace. This reduces memory and transmission burden, but it also changes the failure mode of federation. In weight-centric PEFT, aggregation error is often discussed in terms of parameter drift. In FedReFT, the corresponding concern is **semantic misalignment**: two clients may learn interventions that are individually meaningful but mutually incompatible when averaged. The method’s aggregation design follows directly from this observation.

## 3. Aggregation mismatch and All-But-Me aggregation

FedReFT identifies two coupled challenges for representation-level federation. The first is **semantic misalignment and aggregation mismatch**. Under heterogeneous clients, direct averaging of intervention parameters can cause what the paper describes as **semantic collapse**, because the meaning of a representation edit on one client may conflict with that of another. The second is the standard federated tension between **personalization** and **global knowledge sharing** [2508.20295].

To address these issues, FedReFT introduces **All-But-Me (ABM) aggregation**. For a client \(k\), the server aggregates parameters from all other clients except \(k\), using the **geometric median** over the uploaded intervention parameters \(X \in \{W, R, b\}\). The resulting aggregate is then partially incorporated into client \(k\)’s local parameters through an interpolation step with mixing coefficient \(\alpha \in [0,1]\). The geometric median is defined as the point minimizing the sum of Euclidean distances to the client updates, and the paper states that it is computed in practice with **Weiszfeld’s algorithm** [2508.20295].

ABM has three intended effects. It is **personalized**, because a client’s own update does not dominate the aggregate returned to that client. It is **robust**, because the geometric median is less sensitive than the mean to outliers or misaligned updates. And it is **balanced**, because the interpolation coefficient \(\alpha\) allows the client to trade local specialization against shared knowledge. In the empirical ablation reported for FedReFT, ABM with geometric median consistently outperforms both FedAvg and mean-ABM, especially in heterogeneous settings [2508.20295].

A second misconception is that ABM is merely a cosmetic variant of federated averaging. FedReFT’s formulation treats exclusion of the self-update and robust aggregation by geometric median as the key mechanism for preventing corruption of semantic alignment, not as a minor optimization detail.

## 4. Empirical results and parameter efficiency

FedReFT is evaluated on **commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE**, using models including **LLaMA-7B/2/3/8B, RoBERTa-large, and TinyLlama-1B** [2508.20295]. Across these settings, the method is reported to use only **0.01–0.05%** of model parameters for fine-tuning and to achieve **7×–15×** higher parameter efficiency than leading LoRA-based federated baselines [2508.20295].

The paper reports the following representative benchmark results.

| Setting | FedReFT | Comparator |
|---|---|---|
| Commonsense reasoning, LLaMA-3 3B, MT setup | rank 8, 0.0857% parameters, 75.66% avg. acc. | Fed-SB: 0.0884% parameters, 75.66% avg. acc.; FFA-LoRA: 0.76% parameters, 71.11% acc. |
| Arithmetic reasoning, GSM8K, LLaMA-3 8B | 0.0622% parameters, 49.68% acc. | LoRA: 0.38% parameters, 46.23% acc. |
| GLUE, RoBERTa-large | 0.015% parameters, 90.93 avg. | Best FL LoRA: 0.405–0.551% parameters, 89.39–90.43 avg. |

The model-level parameter counts reported for FedReFT further illustrate the scale of the intervention modules. For **LLaMA-7B**, the total parameter count is listed as **6,738M** with **2.1M trainable parameters**, or **0.031%** trainable. For **LLaMA-3 8B**, the count is **8,030M** total with **2.1M trainable**, or **0.026%**. For **RoBERTa-large**, the count is **355M** total with **0.049M trainable**, or **0.0138%** [2508.20295].

These numbers position FedReFT as a method that combines competitive or superior task performance with unusually small trainable modules in federated settings. A plausible implication is that the representation-level intervention has enough expressivity to remain competitive even when its parameter budget is below that of LoRA-based alternatives.

## 5. Relation to adjacent federated fine-tuning methods

FedReFT belongs to a broader federated fine-tuning literature, but its representation-level focus differentiates it from several neighboring lines of work. **FeDeRA** improves LoRA through SVD-based initialization of low-rank adapters to stabilize federated updates under non-IID data [2404.18848]. **SLoRA** also addresses the PEFT-versus-full-fine-tuning gap under heterogeneity by combining sparse fine-tuning with SVD-primed LoRA initialization [2308.06522]. **FedTT** and **FedTT+** use tensorized adapters and adaptive freezing of tensor factors to reduce communication and improve robustness to heterogeneity, achieving up to a 10× reduction in communication cost relative to competing federated PEFT approaches [2410.13097]. All of these remain fundamentally parameter-centric: they compress, initialize, sparsify, or selectively freeze trainable weight modules.

By contrast, FedReFT shifts the object of adaptation from weight updates to hidden representations. This makes it closer in spirit to methods that emphasize feature alignment, although its mechanism is different from the explicit representation matching regularizer proposed in earlier federated learning for heterogeneous data [1912.13075]. It is also distinct from **SFPrompt**, which combines split learning and prompt tuning so that resource-limited clients do not need to run the complete pre-trained model locally; SFPrompt reports competitive performance while using only **0.46%** of local computing resources and **53% less communication cost** than federated full fine-tuning [2407.17533]. That comparison is important because it highlights a practical boundary: parameter efficiency and representation efficiency do not automatically solve local execution cost.

Later work in federated fine-tuning has also widened the heterogeneity model beyond what FedReFT directly targets. **H2Tune** studies **hybrid heterogeneous federated fine-tuning**, where clients differ simultaneously in model architecture and downstream task, and introduces matrix decomposition, layer alignment, and alternating task-knowledge disentanglement to cope with that setting [2507.22633]. **LEGEND** adapts LoRA depth and rank across heterogeneous devices and reports **1.5–2.8×** speedup with about **42.3%** communication savings on **80 commercial devices** [2412.20004]. These works indicate that client heterogeneity can be architectural and systems-level, not only distributional or semantic.

FedReFT also sits within an evolving debate over how much communication is really necessary. A one-shot federated fine-tuning study argues that, for models larger than **1 billion parameters**, a **single communication round** can achieve performance comparable to multi-round aggregation, and explicitly suggests that FedReFT could potentially be restructured as a one-shot process [2412.04650]. At the opposite extreme, **semantic consensus** methods argue that collaboration should occur through generated outputs on public prompts rather than through parameter aggregation at all, reducing communication by orders of magnitude and accommodating heterogeneous black-box models [2605.11857]. FedReFT therefore occupies a middle position: it departs from full weight tuning, but still remains within a parameterized, multi-round aggregation paradigm.

## 6. Practical significance, scope, and unresolved issues

The practical case for FedReFT is straightforward in the source description. Its intervention layers are computationally lightweight, require transmission of far fewer parameters, and are therefore well suited to **edge and federated environments**. Because raw data never leave the clients, the method is described as naturally compatible with privacy-preserving deployment, with **prospects for integration with DP techniques**. The paper also points to settings such as **consumer devices, healthcare, and finance**, where personalization and privacy are especially important [2508.20295].

At the same time, adjacent literature identifies several concerns that are orthogonal to FedReFT’s core contribution. **FedRef** introduces a server-side reference model and Bayesian proximal regularization to mitigate **catastrophic forgetting across rounds** [2506.23210]. **FedBE** addresses forgetting through adaptive transformer block expansion and dynamic block allocation, reporting **12–74% higher accuracy retention** on general tasks after fine-tuning and **1.9–3.1×** faster convergence than prior federated fine-tuning methods [2506.05977]. On the security side, **AugMP** shows that federated fine-tuning of LLMs remains vulnerable to model-manipulation attacks that can reduce global LLM accuracy by up to **26%** while evading distance- and similarity-based defenses [2605.07961]. These results do not contradict FedReFT; rather, they locate it within a larger systems problem.

This suggests that FedReFT’s main contribution is best understood as a solution to **semantic alignment of representation interventions under federated heterogeneity**, not as a complete answer to every issue in federated foundation-model adaptation. Its importance lies in demonstrating that hidden-state intervention can be federated effectively, provided aggregation is redesigned around personalization and robustness. Whether future systems combine FedReFT-style representation steering with one-shot communication, split execution, behavior-level consensus, stronger Byzantine robustness, or explicit forgetting mitigation remains an open direction indicated by the surrounding literature [2412.04650].

Source: https://www.emergentmind.com/topics/federated-representation-fine-tuning-fedreft