---
title: Robust Federated Encoder-Decoder Training
url: https://www.emergentmind.com/topics/robust-federated-encoder-decoder-training-framework
type: topic
---

# Robust Federated Encoder-Decoder Training

Robust federated encoder–decoder training denotes a class of federated optimization schemes in which the trainable system is decomposed into an encoder–decoder, dual-encoder–decoder, or modality-decoupled architecture, and robustness is enforced against data heterogeneity, malicious clients, adversarial attacks, biased cross-modal alignment, or optimization inconsistency. In the recent literature, this theme appears in several distinct but related formulations: embedding-space adversarial training plus robust aggregation for federated LLMs in "FedEAT: A Robustness Optimization Framework for Federated LLMs" [2502.11863], distributionally robust alignment with a two-stage federated procedure for medical vision-language pre-training in "Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity" [2404.03854], decoupled encoder training with local supervised and reinforcement fine-tuning in "Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning" [2605.27900], and encoder–decoder decomposition for multi-task collaboration in "Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks" [2504.09800]. Taken together, these works define a technical landscape in which robustness is not a single mechanism but a combination of objective design, architectural partitioning, local regularization, and server-side aggregation.

## 1. Formal objectives and optimization structure

A canonical robust federated objective is given in FedEAT. Let there be $K$ clients indexed by $k=1 \ldots K$. Client $k$ holds a private data distribution $D_k$ over input–output pairs $(x,y)$ and maintains a local copy of the model parameters $\theta_k$. The server maintains a global parameter vector $\theta$. The optimization problem is

$$
\min_{\theta} F(\theta), \qquad
F(\theta) = \sum_{k=1}^K p_k R_k(\theta),
$$

with $p_k = |D_k| / \sum_h |D_h|$. The local robust objective is

$$
R_k(\theta)
= (1-\alpha)\, \mathbb{E}_{(x,y)\sim D_k}[L(f_\theta(x),y)]
+ \alpha\, \mathbb{E}_{(x,y)\sim D_k}\!\left[\max_{\|\delta\|_p \le \epsilon} L(f_\theta(x+\delta),y)\right].
$$

Here $f_\theta$ denotes the entire encoder–decoder (LLM) mapping from input tokens to output tokens, $L(\cdot,\cdot)$ is the supervised loss, $\epsilon$ controls the size of allowed adversarial perturbations $\delta$ in the embedding space, and $\alpha \in [0,1]$ trades off clean-data performance vs. adversarial robustness [2502.11863].

FedDRA formulates robustness as a DRO problem over client mixtures. If each client $i$ has private distribution $P_i$ over $(x,y)$ pairs, their mixture is

$$
P^c = \sum_{i=1}^N w_i P_i,\quad w\in\Delta_{N-1},
$$

and the uncertainty ball is

$$
\mathcal{P}=\{\,Q : D(Q\|P^c)\le \rho\,\}.
$$

With $\theta=(f_v,f_\ell,g_v,g_\ell)$, the high-level DRO objective is

$$
\min_\theta \max_{Q\in\mathcal P} \mathbb E_{(x,y)\sim Q}[\mathcal L_{task}(\theta;x,y)].
$$

In practice the infinite $Q$-max is replaced by a finite weighted maximization over the $N$ client distributions,

$$
\min_{\theta}\max_{\substack{w\in\Delta_{N-1}\\D(Nw\|\mathbf 1)\le \rho}}
\sum_{i=1}^N w_i\,\mathbb E_{(x,y)\sim P_i}[\mathcal L_{task}(\theta;x,y)].
$$

This directly encodes robustness to test-time domains constructed from heterogeneous client datasets [2404.03854].

M-Fed encodes robustness to heterogeneity through encoder–decoder regularization rather than adversarial or distributional uncertainty. Each client model for task $k$ is written as

$$
F_k(w_i)=D_{k,i}\circ E_i,
$$

with parameter split $w_i=(w_i^*, w_i^\dagger)$, where $w_i^*$ are the encoder weights and $w_i^\dagger$ are the decoder weights. The client objective is

$$
\min_{w_i=(w_i^*,w_i^\dagger)} H_i(w_i;g,D_k)
=
L_i(x_i,y_i;w_i^*,w_i^\dagger)
+\lambda_{\rm enc}\|w_i^*-g^t\|_2^2
+\lambda_{\rm dec}\|w_i^\dagger-w_k^{t,\dagger}\|_2^2.
$$

The corresponding global objective sums local task loss, cross-task encoder regularization, and intra-task decoder consistency [2504.09800].

This suggests a common abstraction: robust federated encoder–decoder training is usually instantiated as a min–max or regularized federated objective in which the encoder carries transferable representation structure, while the decoder or alignment head absorbs task-specific or modality-specific variation.

## 2. Local robustness mechanisms

FedEAT places the adversary in embedding space. With continuous embedding $z=E(x)\in\mathbb R^d$, the worst-case perturbation is

$$
\delta^* = \arg\max_{\|\delta\|_p \le \epsilon} L(f(z+\delta), y),
$$

and the combined robust training loss is

$$
L_{\rm robust}(\theta; z,y)
= (1-\alpha)\cdot L(f_\theta(z), y)
+ \alpha \cdot L(f_\theta(z+\delta^*), y).
$$

In practice $\delta^*$ is approximated by $T$ steps of projected gradient ascent in embedding space,

$$
z^{(t+1)} = \operatorname{Proj}_{\|\cdot\|_p\le \epsilon}
\left[z^{(t)} + \eta_\delta \nabla_z L(f_\theta(z^{(t)}), y)\right],
$$

starting from $z^{(0)}=z$ [2502.11863].

FedDRA uses robust alignment rather than perturbation of embeddings. Given paired batch $\{(x_i,y_i)\}_{i=1}^B$, the client computes $v_i=f_v(x_i)$, $l_i=f_\ell(y_i)$, $z_{I,i}=g_v(v_i)$, and $z_{T,i}=g_\ell(l_i)$, then applies a CLIP-style contrastive loss

$$
\mathcal{L}_{CL} = -\frac{1}{B}\sum_{i=1}^B\Bigl[\log\frac{\exp\langle z_{I,i},z_{T,i}\rangle/\tau}
{\sum_{j=1}^B\exp\langle z_{I,i},z_{T,j}\rangle/\tau}
+
\log\frac{\exp\langle z_{T,i},z_{I,i}\rangle/\tau}
{\sum_{j=1}^B\exp\langle z_{T,i},z_{I,j}\rangle/\tau}\Bigr].
$$

After standard federated averaging of the encoders, the framework fixes $f_v^*,f_\ell^*$ and learns a single teacher aligner $(g_v^*,g_\ell^*)$ on worst-case client weights. The client-level robust loss is

$$
\mathcal{L}_{dro}^{(c)} =
\beta\sum_{i=1}^{B_c}\|\tilde z_{I,i}^* - z_{I,i}^*\|^2
+\beta\sum_{i=1}^{B_c}\|\tilde z_{T,i}^* - z_{T,i}^*\|^2
+\mathcal{L}_{CL}(\{\tilde z_I^*\},\{\tilde z_T^*\})
+\mathcal{L}_{task}(\hat f^*,g^*;x,y).
$$

FedDRA also introduces anchor representation regularization,

$$
\mathcal{L}_{guide}
= \sum_{i=1}^{B_c}\|g_v^*(f_v(x_i)) - g_v(f_v(x_i))\|^2
+\sum_{i=1}^{B_c}\|g_\ell^*(f_\ell(y_i)) - g_\ell(f_\ell(y_i))\|^2,
$$

which keeps local aligners from drifting too far and prevents over-fitting to the client’s idiosyncratic data [2404.03854].

FedDTL makes robustness a property of local training dynamics. At round $t\le M$, local optimization uses supervised fine-tuning with

$$
L_{ce}^k = - \frac{1}{N_k}\sum_{(x,y)\in D_k}\log p(\hat y=y|x),
$$

where the classifier is induced by cosine similarity between normalized visual and text embeddings. After SFT saturates, the framework switches to a GRPO-inspired RL stage. For each image $x_i$, Gaussian noise $\epsilon\sim\mathcal N(0,\sigma^2 I)$ is injected into the vision token before normalization, rewards are $r_{i,j}=1[a_{i,j}=y_i]$, advantages are

$$
A_{i,j} = \frac{r_{i,j} - \operatorname{mean}_j r_{i,j}}{\operatorname{std}_j(r_{i,j})},
$$

and the policy update uses a clipped surrogate with KL regularization to a reference policy,

$$
L_{rl}^k = - \frac{1}{G\cdot bs}\sum_{j=1}^G \sum_{i=1}^{bs} [L_p - \beta L_{kl}].
$$

The stated purpose is to mitigate intra-client over-specialization while preserving global task adaptation [2605.27900].

## 3. Aggregation, communication, and server-side robustness

In FedEAT, each round proceeds by broadcasting $\theta^{(t)}$ to a subset of clients, solving approximate local robust objectives, and aggregating the returned parameters. The server-side robustness mechanism is geometric median aggregation. Given client parameter vectors $w_1,\ldots,w_m$, the robust aggregate is

$$
g^* = \arg\min_g \sum_{k=1}^m \|w_k-g\|_2.
$$

This non-smooth optimization can be solved efficiently by Weiszfeld’s algorithm, initialized at $g^{(0)}$ and iterated as

$$
g^{(t+1)} =
\frac{\sum_{k=1}^m w_k / \|w_k-g^{(t)}\|_2}
{\sum_{k=1}^m 1 / \|w_k-g^{(t)}\|_2},
$$

until convergence. The final iterate becomes the new global parameter vector [2502.11863].

FedDRA uses a two-stage communication pattern. In Stage 1, each client receives global encoders and the global teacher aligner, initializes local aligners from the teacher, optimizes $\mathcal L_{task}+\alpha \mathcal L_{guide}$ for $K$ local steps, and sends back $(f_v^c,f_\ell^c,g_v^c,g_\ell^c)$. The server aggregates only the encoders by FedAvg,

$$
f_v^t=\sum_c \frac{n_c}{\sum n_c} f_v^c,\qquad
f_\ell^t=\sum_c \frac{n_c}{\sum n_c} f_\ell^c.
$$

In Stage 2, with fixed encoders, clients compute $\mathcal L_{dro}^{(c)}$; the server collects these scalar losses, updates the client weights by mirror descent,

$$
w_c^{(t+1)} \propto w_c^{(t)} \exp(\gamma \mathcal L_{dro}^{(c)}),
$$

projects onto $\{w: D(Nw\|\mathbf 1)\le \rho\}$, and updates the teacher aligner by minimizing $\sum_c w_c \mathcal L_{dro}^{(c)}$ [2404.03854].

FedDTL also decouples what is aggregated. Clients upload only class-token embeddings plus $\Delta W$, and the server aggregates visual LoRA by

$$
\Delta W_g^t = \sum_{k=1}^K \left(\frac{N_k}{\sum N_m}\right)\cdot \Delta W_k^t,
$$

then updates text LoRA on the server by minimizing a server-side cross-entropy loss over collected visual class embeddings and text prompts. The global text embeddings are then broadcast in the next round [2605.27900].

M-Fed performs two nested averages. First, for each task $k$, the server aggregates clients doing that task into a task-global model $w_k^{t+1}=A(\{w_i^t\}_{i\in N_k})$. It then extracts the encoder portion $(w_k^{t+1,*})$ from each task-global model and aggregates across all tasks to produce a single global encoder,

$$
g^{t+1}=A(\{w_k^{t+1,*}\}_{k=1}^K).
$$

The server broadcasts both the task-specific global model $w_{k(i)}^t$ and the global encoder $g^t$ to each client at the start of the next round [2504.09800].

A plausible implication is that robust federated encoder–decoder systems increasingly separate the aggregation target from the full local model: encoder-only aggregation, teacher-only aggregation, LoRA-only aggregation, and robust aggregation of complete parameter vectors are all used to control heterogeneity and attack surface.

## 4. Architectural decomposition patterns

FedEAT uses the LLM embedding layer as the interface between adversarial robustness and federated training. The method works with $z=E(x)\in\mathbb R^d$ and treats $f_\theta$ as the entire encoder–decoder (LLM) mapping from input tokens to output tokens. The experimental model set is "gemma-1.1-2b-it, PHI-3-MINI, MISTRAL-7B, ZEPHYR-7B (all encoder–decoder or decoder-only LLMs that support embedding-level backprop)" [2502.11863].

FedDRA adopts a dual-encoder–decoder, described as "backbone + aligner" for each modality. On the vision side, it uses encoder $f_v:\mathcal X\to \mathbb R^d$ and decoder/alignment head $g_v:\mathbb R^d\to \mathbb R^k$; on the language side, encoder $f_\ell:\mathcal Y\to \mathbb R^d$ and decoder/alignment head $g_\ell:\mathbb R^d\to \mathbb R^k$. The paired outputs are aligned in a shared $k$-dim joint embedding space [2404.03854].

FedDTL explicitly decouples the image encoder and text encoder across clients and the server. The vision encoder is a frozen ViT-B/16 backbone plus LoRA adapters $\Delta W^v$ inserted in layers $\ell=10\ldots$, the text encoder is a paired CLIP text branch plus LoRA adapters $\Delta W^t$, and the decoder head uses cosine similarity plus temperature $\tau$ to produce class-probabilities. For each self-attention weight $W_0\in\mathbb R^{d\times d}$, only the low-rank update

$$
\Delta W = B\cdot A,\qquad A\in\mathbb R^{r\times d},\; B\in\mathbb R^{d\times r},\; r\ll d
$$

is trained [2605.27900].

M-Fed generalizes the encoder–decoder split beyond robustness to adversarial or multimodal settings. Every client has an encoder $E_i:\mathcal X\to \mathbb R^d$ with identical architecture across all clients and tasks, and a decoder $D_{k,i}:\mathbb R^d\to \mathcal Y_k$ that is task-specific and may vary across clients of that task. Decoder aggregation is performed only among clients that share both the same task and the same decoder architecture [2504.09800].

These decompositions differ operationally, but they converge on the same design principle: robustness is easier to engineer when the transferable representation component is isolated from the client-specific prediction or alignment component.

## 5. Empirical evaluation and reported effects

FedEAT evaluates local adaptation on the `vicgalle/alpaca-gpt4` fine-tuning set and uses clean/adversarial splits of SST2, QQP, MNLI and QNLI from the TrustLLM benchmark. Utility is test accuracy on clean (benign) data, and robustness is Attack Success Rate (ASR) defined as the number of originally-correct samples misclassified under adversarial perturbations; lower ASR is better. The main averaged results report "FedAVG baseline avg. ASR ≈ 0.259 → FedEAT avg. ASR ≈ 0.252 (PHI-3) and 0.144 (ZEPHYR)." The paper also reports "Clean accuracy drop ≤ 3% in most cases; sometimes small ↑ due to better generalization from moderate perturbation." In ablation, "EAT-only and GM-only both reduce ASR vs. FedAVG, but the combination yields the best overall robustness" [2502.11863].

FedDRA is evaluated on MIMIC-CXR with 227 835 image–report pairs, partitioned into $N=5$ clients via LDA on a 5-way disease-category attribute, with heterogeneity index set to 1. The backbones are ViT-Base and BERT-Base with one extra transformer layer for each aligner. Evaluation uses image–text retrieval, few-shot classification on RSNA and COVIDx at 1%/10% shots, segmentation on RSNA via frozen encoder plus U-Net decoder, and out-of-domain retrieval on EHRXQA. The reported summary is that "FedDRA consistently raises average recall and worst-case recall vs. FedAvg, reduces client performance disparity, and approaches the centralized upper bound on all downstream tasks" [2404.03854].

FedDTL reports robustness to label skew, feature shift, and full-data FL settings. Under inter-client inconsistency, the empirical claim is that "FedDTL base accuracy outperforms baselines by 3–5% under Dir(0.1) & Non-IID." For domain shift, "Office-Caltech10 & DomainNet experiments (Table 2): FedDTL outperforms baselines by ~2–4% average under 'one' and 'Dir(0.1)'." For intra-client over-specialization, "Table 3 ablation: adding RL lifts novel HM by +5–7 points in full-data Non-IID" [2605.27900].

M-Fed evaluates on PASCAL-Context and MS-COCO. On PASCAL-Context, the table reports: Local only gives 42.07 on semantic segmentation, 50.74 on human part segmentation, 58.50 on saliency estimation, and 62.60 on edge detection; FedAvg-per-task gives 53.14, 55.76, 61.40, and 66.20; M-Fed gives 53.98, 55.93, 61.56, and 66.20. The stated summary is that "M-Fed improves over local by up to +12.4% on average, and beats FedAvg by ~0.6% on average." On MS-COCO, with ResNet-50, Local gives 18.97 instance AP, 25.78 panoptic PQ, 26.61 keypoint AP, and 5.37 stuff IoU; FedAvg gives 21.65, 23.67, 31.17, and 7.25; M-Fed gives 20.83, 24.21, 32.69, and 7.40. With ResNet-101, Local gives 20.38, 28.02, 33.40, and 15.45; FedAvg gives 23.25, 25.92, 39.89, and 16.26; M-Fed gives 23.22, 25.96, 40.33, and 16.62. The paper summarizes this as exceeding local training by "~16% (R50) / ~9% (R101)" and outperforming FedAvg per task in keypoint and "stuff" segmentation while being slightly below on instance segmentation [2504.09800].

## 6. Theory, limitations, and recurrent points of clarification

The strongest explicit convergence statement in this group of methods appears in FedEAT’s server-side optimizer: "Weiszfeld’s algorithm for geometric median is known to converge to a stationary point under mild conditions (no client at the current estimate)." At the same time, the paper states that "adversarial training and the nonconvexity of LLMs preclude strong global convergence proofs" and that "No closed-form convergence rate is given in the paper" [2502.11863]. Robustness claims in this setting are therefore empirical and algorithmic rather than fully asymptotic.

A recurrent misconception is that one robustness mechanism is sufficient. FedEAT directly contradicts that simplification: "EAT-only and GM-only both reduce ASR vs. FedAVG, but the combination yields the best overall robustness" [2502.11863]. A second misconception is that encoder–decoder decomposition removes structural constraints. M-Fed explicitly retains hard compatibility requirements: "All clients must adopt the same encoder architecture," and "Within a task, all decoders must share the same architecture to allow decoder aggregation" [2504.09800]. A third misconception is that local specialization is always beneficial under FL. FedDTL states that "Long SFT in full-data regimes causes CLIP to 'memorize' local biases → novel accuracy ↓," and the RL stage is introduced specifically to alleviate this over-fit [2605.27900].

FedDRA identifies a different failure mode: "client data heterogeneity in real-world scenarios could cause models to learn biased cross-modal alignment during local pre-training." Its anchor representation and DRO teacher updates are therefore not merely regularizers but mechanisms to bridge "the gap between pre-training samples and downstream applications" while avoiding over-fitting on client-specific information [2404.03854].

Across these frameworks, robustness is best understood as a layered property. One layer acts on the local objective through adversarial examples, anchor losses, or RL regularization; another acts on aggregation through geometric median, FedAvg over selected submodules, or DRO-weighted teacher updates; a third acts on model topology by partitioning encoder and decoder roles. This suggests that the robust federated encoder–decoder paradigm is less a single algorithm than a reproducible systems pattern for distributed training under heterogeneous, adversarial, or task-diverse conditions.

Source: https://www.emergentmind.com/topics/robust-federated-encoder-decoder-training-framework