Robust Federated Encoder-Decoder Training

• Robust federated encoder–decoder training is a framework that separates transferable encoder representations from task-specific decoders to counteract data heterogeneity and adversarial challenges.
• It employs local mechanisms such as adversarial embedding perturbation, contrastive losses, and regularization, alongside robust server-side aggregation methods like geometric median and DRO-based updates.
• Empirical evaluations across NLP, vision, and multi-task settings demonstrate improvements in accuracy and reduced attack success rates, highlighting practical benefits in real-world federated applications.

Robust federated encoder–decoder training denotes a class of federated optimization schemes in which the trainable system is decomposed into an encoder–decoder, dual-encoder–decoder, or modality-decoupled architecture, and robustness is enforced against data heterogeneity, malicious clients, adversarial attacks, biased cross-modal alignment, or optimization inconsistency. In the recent literature, this theme appears in several distinct but related formulations: embedding-space adversarial training plus robust aggregation for federated LLMs in "FedEAT: A Robustness Optimization Framework for Federated LLMs" (Pang et al., 17 Feb 2025), distributionally robust alignment with a two-stage federated procedure for medical vision-language pre-training in "Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity" (Shuai et al., 2024), decoupled encoder training with local supervised and reinforcement fine-tuning in "Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning" (Ma et al., 27 May 2026), and encoder–decoder decomposition for multi-task collaboration in "Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks" (Zhou et al., 14 Apr 2025). Taken together, these works define a technical landscape in which robustness is not a single mechanism but a combination of objective design, architectural partitioning, local regularization, and server-side aggregation.

1. Formal objectives and optimization structure

A canonical robust federated objective is given in FedEAT. Let there be $K$ clients indexed by $k=1 \ldots K$. Client $k$ holds a private data distribution $D_k$ over input–output pairs $(x,y)$ and maintains a local copy of the model parameters $\theta_k$. The server maintains a global parameter vector $\theta$. The optimization problem is

$$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$

with $p_k = |D_k| / \sum_h |D_h|$. The local robust objective is

$$R_k(\theta) = (1-\alpha)\, \mathbb{E}_{(x,y)\sim D_k}[L(f_\theta(x),y)] + \alpha\, \mathbb{E}_{(x,y)\sim D_k}\!\left[\max_{\|\delta\|_p \le \epsilon} L(f_\theta(x+\delta),y)\right].$$

Here $k=1 \ldots K$0 denotes the entire encoder–decoder (LLM) mapping from input tokens to output tokens, $k=1 \ldots K$1 is the supervised loss, $k=1 \ldots K$2 controls the size of allowed adversarial perturbations $k=1 \ldots K$3 in the embedding space, and $k=1 \ldots K$4 trades off clean-data performance vs. adversarial robustness (Pang et al., 17 Feb 2025).

FedDRA formulates robustness as a DRO problem over client mixtures. If each client $k=1 \ldots K$5 has private distribution $k=1 \ldots K$6 over $k=1 \ldots K$7 pairs, their mixture is

$k=1 \ldots K$8

and the uncertainty ball is

$k=1 \ldots K$9

With $k$0, the high-level DRO objective is

$k$1

In practice the infinite $k$2-max is replaced by a finite weighted maximization over the $k$3 client distributions,

$k$4

This directly encodes robustness to test-time domains constructed from heterogeneous client datasets (Shuai et al., 2024).

M-Fed encodes robustness to heterogeneity through encoder–decoder regularization rather than adversarial or distributional uncertainty. Each client model for task $k$5 is written as

$k$6

with parameter split $k$7, where $k$8 are the encoder weights and $k$9 are the decoder weights. The client objective is

$D_k$0

The corresponding global objective sums local task loss, cross-task encoder regularization, and intra-task decoder consistency (Zhou et al., 14 Apr 2025).

This suggests a common abstraction: robust federated encoder–decoder training is usually instantiated as a min–max or regularized federated objective in which the encoder carries transferable representation structure, while the decoder or alignment head absorbs task-specific or modality-specific variation.

2. Local robustness mechanisms

FedEAT places the adversary in embedding space. With continuous embedding $D_k$1, the worst-case perturbation is

$D_k$2

and the combined robust training loss is

$D_k$3

In practice $D_k$4 is approximated by $D_k$5 steps of projected gradient ascent in embedding space,

$D_k$6

starting from $D_k$7 (Pang et al., 17 Feb 2025).

FedDRA uses robust alignment rather than perturbation of embeddings. Given paired batch $D_k$8, the client computes $D_k$9, $(x,y)$0, $(x,y)$1, and $(x,y)$2, then applies a CLIP-style contrastive loss

$(x,y)$3

After standard federated averaging of the encoders, the framework fixes $(x,y)$4 and learns a single teacher aligner $(x,y)$5 on worst-case client weights. The client-level robust loss is

$(x,y)$6

FedDRA also introduces anchor representation regularization,

$(x,y)$7

which keeps local aligners from drifting too far and prevents over-fitting to the client’s idiosyncratic data (Shuai et al., 2024).

FedDTL makes robustness a property of local training dynamics. At round $(x,y)$8, local optimization uses supervised fine-tuning with

$(x,y)$9

where the classifier is induced by cosine similarity between normalized visual and text embeddings. After SFT saturates, the framework switches to a GRPO-inspired RL stage. For each image $\theta_k$0, Gaussian noise $\theta_k$1 is injected into the vision token before normalization, rewards are $\theta_k$2, advantages are

$\theta_k$3

and the policy update uses a clipped surrogate with KL regularization to a reference policy,

$\theta_k$4

The stated purpose is to mitigate intra-client over-specialization while preserving global task adaptation (Ma et al., 27 May 2026).

3. Aggregation, communication, and server-side robustness

In FedEAT, each round proceeds by broadcasting $\theta_k$5 to a subset of clients, solving approximate local robust objectives, and aggregating the returned parameters. The server-side robustness mechanism is geometric median aggregation. Given client parameter vectors $\theta_k$6, the robust aggregate is

$\theta_k$7

This non-smooth optimization can be solved efficiently by Weiszfeld’s algorithm, initialized at $\theta_k$8 and iterated as

$\theta_k$9

until convergence. The final iterate becomes the new global parameter vector (Pang et al., 17 Feb 2025).

FedDRA uses a two-stage communication pattern. In Stage 1, each client receives global encoders and the global teacher aligner, initializes local aligners from the teacher, optimizes $\theta$0 for $\theta$1 local steps, and sends back $\theta$2. The server aggregates only the encoders by FedAvg,

$\theta$3

In Stage 2, with fixed encoders, clients compute $\theta$4; the server collects these scalar losses, updates the client weights by mirror descent,

$\theta$5

projects onto $\theta$6, and updates the teacher aligner by minimizing $\theta$7 (Shuai et al., 2024).

FedDTL also decouples what is aggregated. Clients upload only class-token embeddings plus $\theta$8, and the server aggregates visual LoRA by

$\theta$9

then updates text LoRA on the server by minimizing a server-side cross-entropy loss over collected visual class embeddings and text prompts. The global text embeddings are then broadcast in the next round (Ma et al., 27 May 2026).

M-Fed performs two nested averages. First, for each task $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$0, the server aggregates clients doing that task into a task-global model $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$1. It then extracts the encoder portion $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$2 from each task-global model and aggregates across all tasks to produce a single global encoder,

$$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$3

The server broadcasts both the task-specific global model $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$4 and the global encoder $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$5 to each client at the start of the next round (Zhou et al., 14 Apr 2025).

A plausible implication is that robust federated encoder–decoder systems increasingly separate the aggregation target from the full local model: encoder-only aggregation, teacher-only aggregation, LoRA-only aggregation, and robust aggregation of complete parameter vectors are all used to control heterogeneity and attack surface.

4. Architectural decomposition patterns

FedEAT uses the LLM embedding layer as the interface between adversarial robustness and federated training. The method works with $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$6 and treats $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$7 as the entire encoder–decoder (LLM) mapping from input tokens to output tokens. The experimental model set is "gemma-1.1-2b-it, PHI-3-MINI, MISTRAL-7B, ZEPHYR-7B (all encoder–decoder or decoder-only LLMs that support embedding-level backprop)" (Pang et al., 17 Feb 2025).

FedDRA adopts a dual-encoder–decoder, described as "backbone + aligner" for each modality. On the vision side, it uses encoder $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$8 and decoder/alignment head $$\min_{\theta} F(\theta), \qquad F(\theta) = \sum_{k=1}^K p_k R_k(\theta),$$9; on the language side, encoder $p_k = |D_k| / \sum_h |D_h|$0 and decoder/alignment head $p_k = |D_k| / \sum_h |D_h|$1. The paired outputs are aligned in a shared $p_k = |D_k| / \sum_h |D_h|$2-dim joint embedding space (Shuai et al., 2024).

FedDTL explicitly decouples the image encoder and text encoder across clients and the server. The vision encoder is a frozen ViT-B/16 backbone plus LoRA adapters $p_k = |D_k| / \sum_h |D_h|$3 inserted in layers $p_k = |D_k| / \sum_h |D_h|$4, the text encoder is a paired CLIP text branch plus LoRA adapters $p_k = |D_k| / \sum_h |D_h|$5, and the decoder head uses cosine similarity plus temperature $p_k = |D_k| / \sum_h |D_h|$6 to produce class-probabilities. For each self-attention weight $p_k = |D_k| / \sum_h |D_h|$7, only the low-rank update

$p_k = |D_k| / \sum_h |D_h|$8

is trained (Ma et al., 27 May 2026).

M-Fed generalizes the encoder–decoder split beyond robustness to adversarial or multimodal settings. Every client has an encoder $p_k = |D_k| / \sum_h |D_h|$9 with identical architecture across all clients and tasks, and a decoder $$R_k(\theta) = (1-\alpha)\, \mathbb{E}_{(x,y)\sim D_k}[L(f_\theta(x),y)] + \alpha\, \mathbb{E}_{(x,y)\sim D_k}\!\left[\max_{\|\delta\|_p \le \epsilon} L(f_\theta(x+\delta),y)\right].$$0 that is task-specific and may vary across clients of that task. Decoder aggregation is performed only among clients that share both the same task and the same decoder architecture (Zhou et al., 14 Apr 2025).

These decompositions differ operationally, but they converge on the same design principle: robustness is easier to engineer when the transferable representation component is isolated from the client-specific prediction or alignment component.

5. Empirical evaluation and reported effects

FedEAT evaluates local adaptation on the vicgalle/alpaca-gpt4 fine-tuning set and uses clean/adversarial splits of SST2, QQP, MNLI and QNLI from the TrustLLM benchmark. Utility is test accuracy on clean (benign) data, and robustness is Attack Success Rate (ASR) defined as the number of originally-correct samples misclassified under adversarial perturbations; lower ASR is better. The main averaged results report "FedAVG baseline avg. ASR ≈ 0.259 → FedEAT avg. ASR ≈ 0.252 (PHI-3) and 0.144 (ZEPHYR)." The paper also reports "Clean accuracy drop ≤ 3% in most cases; sometimes small ↑ due to better generalization from moderate perturbation." In ablation, "EAT-only and GM-only both reduce ASR vs. FedAVG, but the combination yields the best overall robustness" (Pang et al., 17 Feb 2025).

FedDRA is evaluated on MIMIC-CXR with 227 835 image–report pairs, partitioned into $$R_k(\theta) = (1-\alpha)\, \mathbb{E}_{(x,y)\sim D_k}[L(f_\theta(x),y)] + \alpha\, \mathbb{E}_{(x,y)\sim D_k}\!\left[\max_{\|\delta\|_p \le \epsilon} L(f_\theta(x+\delta),y)\right].$$1 clients via LDA on a 5-way disease-category attribute, with heterogeneity index set to 1. The backbones are ViT-Base and BERT-Base with one extra transformer layer for each aligner. Evaluation uses image–text retrieval, few-shot classification on RSNA and COVIDx at 1%/10% shots, segmentation on RSNA via frozen encoder plus U-Net decoder, and out-of-domain retrieval on EHRXQA. The reported summary is that "FedDRA consistently raises average recall and worst-case recall vs. FedAvg, reduces client performance disparity, and approaches the centralized upper bound on all downstream tasks" (Shuai et al., 2024).

FedDTL reports robustness to label skew, feature shift, and full-data FL settings. Under inter-client inconsistency, the empirical claim is that "FedDTL base accuracy outperforms baselines by 3–5% under Dir(0.1) & Non-IID." For domain shift, "Office-Caltech10 & DomainNet experiments (Table 2): FedDTL outperforms baselines by ~2–4% average under 'one' and 'Dir(0.1)'." For intra-client over-specialization, "Table 3 ablation: adding RL lifts novel HM by +5–7 points in full-data Non-IID" (Ma et al., 27 May 2026).

M-Fed evaluates on PASCAL-Context and MS-COCO. On PASCAL-Context, the table reports: Local only gives 42.07 on semantic segmentation, 50.74 on human part segmentation, 58.50 on saliency estimation, and 62.60 on edge detection; FedAvg-per-task gives 53.14, 55.76, 61.40, and 66.20; M-Fed gives 53.98, 55.93, 61.56, and 66.20. The stated summary is that "M-Fed improves over local by up to +12.4% on average, and beats FedAvg by ~0.6% on average." On MS-COCO, with ResNet-50, Local gives 18.97 instance AP, 25.78 panoptic PQ, 26.61 keypoint AP, and 5.37 stuff IoU; FedAvg gives 21.65, 23.67, 31.17, and 7.25; M-Fed gives 20.83, 24.21, 32.69, and 7.40. With ResNet-101, Local gives 20.38, 28.02, 33.40, and 15.45; FedAvg gives 23.25, 25.92, 39.89, and 16.26; M-Fed gives 23.22, 25.96, 40.33, and 16.62. The paper summarizes this as exceeding local training by "~16% (R50) / ~9% (R101)" and outperforming FedAvg per task in keypoint and "stuff" segmentation while being slightly below on instance segmentation (Zhou et al., 14 Apr 2025).

6. Theory, limitations, and recurrent points of clarification

The strongest explicit convergence statement in this group of methods appears in FedEAT’s server-side optimizer: "Weiszfeld’s algorithm for geometric median is known to converge to a stationary point under mild conditions (no client at the current estimate)." At the same time, the paper states that "adversarial training and the nonconvexity of LLMs preclude strong global convergence proofs" and that "No closed-form convergence rate is given in the paper" (Pang et al., 17 Feb 2025). Robustness claims in this setting are therefore empirical and algorithmic rather than fully asymptotic.

A recurrent misconception is that one robustness mechanism is sufficient. FedEAT directly contradicts that simplification: "EAT-only and GM-only both reduce ASR vs. FedAVG, but the combination yields the best overall robustness" (Pang et al., 17 Feb 2025). A second misconception is that encoder–decoder decomposition removes structural constraints. M-Fed explicitly retains hard compatibility requirements: "All clients must adopt the same encoder architecture," and "Within a task, all decoders must share the same architecture to allow decoder aggregation" (Zhou et al., 14 Apr 2025). A third misconception is that local specialization is always beneficial under FL. FedDTL states that "Long SFT in full-data regimes causes CLIP to 'memorize' local biases → novel accuracy ↓," and the RL stage is introduced specifically to alleviate this over-fit (Ma et al., 27 May 2026).

FedDRA identifies a different failure mode: "client data heterogeneity in real-world scenarios could cause models to learn biased cross-modal alignment during local pre-training." Its anchor representation and DRO teacher updates are therefore not merely regularizers but mechanisms to bridge "the gap between pre-training samples and downstream applications" while avoiding over-fitting on client-specific information (Shuai et al., 2024).

Across these frameworks, robustness is best understood as a layered property. One layer acts on the local objective through adversarial examples, anchor losses, or RL regularization; another acts on aggregation through geometric median, FedAvg over selected submodules, or DRO-weighted teacher updates; a third acts on model topology by partitioning encoder and decoder roles. This suggests that the robust federated encoder–decoder paradigm is less a single algorithm than a reproducible systems pattern for distributed training under heterogeneous, adversarial, or task-diverse conditions.