FedSODA: Dual Federated Learning Methods

• FedSODA is a term that encapsulates two distinct federated learning methods, addressing both histopathology segmentation and resource-efficient LLM fine-tuning.
• In histopathology segmentation, FedSODA employs synthetic cross-assessment and dynamic stratified-layer aggregation to mitigate data heterogeneity and sample imbalance.
• For LLM fine-tuning, FedSODA leverages similarity group pruning, orchestrated distillation alignment, and QLoRA to reduce communication overhead and enhance efficiency.

Searching arXiv for FedSODA and closely related federated methods to ground the article in the current literature. arxiv_search(query="FedSODA federated", max_results=10, sort_by="submittedDate") FedSODA denotes at least two distinct federated learning methods in the arXiv literature rather than a single universally standardized algorithm. One usage refers to a framework for federated histopathology segmentation based on synthetic-driven cross-assessment operation and dynamic stratified-layer aggregation (Zhang et al., 2023). A later usage refers to a resource-efficient framework for federated fine-tuning of LLMs based on similarity group pruning and orchestrated distillation alignment (Zhu et al., 18 Aug 2025). The shared name creates a nomenclatural ambiguity: both methods address heterogeneous federated settings, but they target different model classes, data regimes, and systems constraints.

1. Scope and naming

In current arXiv usage, the label FedSODA is attached to two technically unrelated systems. The earlier method is defined for multi-site histopathology image segmentation, with an explicit focus on nuclei and tissue segmentation under sample imbalance and cross-domain heterogeneity (Zhang et al., 2023). The later method is defined for federated fine-tuning of 7B-class LLMs under client-side resource constraints, where clients cannot store or backpropagate through the full model (Zhu et al., 18 Aug 2025). This suggests that the name has become overloaded across subfields.

FedSODA paper	Domain	Core modules
"FedSODA: Federated Cross-assessment and Dynamic Aggregation for Histopathology Segmentation" (Zhang et al., 2023)	Histopathology segmentation	SO, DA, segmentation consistency loss
"FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment" (Zhu et al., 18 Aug 2025)	Federated fine-tuning of LLMs	SGP, ODA, QLoRA

The ambiguity matters bibliographically because nearby names in federated learning are also easy to confuse. "FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization" (Yu et al., 14 May 2025), "FedSTO" for semi-supervised federated object detection on SODA10M (Kim et al., 2023), and "FedSOL" for stabilized orthogonal learning with proximal restrictions (Lee et al., 2023) are separate methods and should not be conflated with either FedSODA variant.

2. FedSODA for histopathology segmentation

The histopathology FedSODA is proposed for federated image segmentation across multiple medical sites where raw data cannot be shared. Its experimental federation has seven clients, each corresponding to one dataset: CoNSeP, CPM-17, CRAG, CryoNuSeg, Glas, Kumar, and TNBC (Zhang et al., 2023). Within this federation, CRAG and Glas are tissue datasets with relatively more samples, while the remaining five are nuclei datasets with smaller sample sizes. The resulting non-IID structure is unusually severe because clients differ simultaneously in organ/domain, target scale, appearance or intensity distribution, and sample count.

The paper emphasizes two principal obstacles. The first is data heterogeneity induced by disparate organs, staining patterns, and segmentation targets ranging from small nuclei to larger tissue or gland structures. The second is sample imbalance, with client dataset sizes reported as roughly 30 to 210 samples. In this setting, standard federated baselines such as FedAvg, FedProx, FedBN, and HarmoFL are described as inadequate because they either average entire models too coarsely or fail to account for cross-client differences in segmentation semantics (Zhang et al., 2023).

Architecturally, the model backbone is U-Net, and local optimization uses Adam with learning rate $1\times 10^{-4}$, momentum 0.9 and 0.95, batch size 4, total training of 300 epochs, and 5 local update epochs per communication round (Zhang et al., 2023). The explicit local objectives include standard cross-entropy loss $\mathcal{L}_{ce}$ and a segmentation consistency loss $\mathcal{L}_{sc}$. The consistency parameter is written as

$\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$

and the consistency loss is written as

$\mathcal{L}_{sc}=\mathcal{L}_{ce}(\xi y, \hat{y}).$

The paper does not provide a final explicit combined local loss with a weighting coefficient, and several implementation details remain under-specified.

3. Synthetic cross-assessment and dynamic aggregation

The core of the histopathology method is the combination of SO and DA. SO, or synthetic-driven cross-assessment operation, is designed to mitigate representation bias arising from sample imbalance. For each client dataset, the paper states that the dataset has distribution mean $\mu$ and standard deviation $\delta$, while the synthetic information is sampled as

$v^t \sim \mathcal{N}(\mu,\sigma).$

The notation is explicitly inconsistent: the text mentions $\delta$ as the standard deviation but writes the Gaussian with $\sigma$ (Zhang et al., 2023). A memory mechanism then mixes current and historical synthetic information according to

$\mathcal{L}_{ce}$0

with best reported performance at $\mathcal{L}_{ce}$1.

The server feeds synthetic information $\mathcal{L}_{ce}$2 associated with client $\mathcal{L}_{ce}$3 into every client model, producing layer-wise feature sets $\mathcal{L}_{ce}$4. For layer $\mathcal{L}_{ce}$5, cross-client similarity is computed by cosine similarity,

$\mathcal{L}_{ce}$6

and normalized as

$\mathcal{L}_{ce}$7

The summation index is omitted in the paper; the intended normalization is over clients for a fixed probe client and layer.

DA, or dynamic stratified-layer aggregation, uses these similarities to aggregate model layers separately rather than averaging the entire network with a single weight. The server aggregation for layer $\mathcal{L}_{ce}$8 is written as

$\mathcal{L}_{ce}$9

This formula is intended to combine a self-retention term weighted by $\mathcal{L}_{sc}$0 with a similarity-weighted cross-client term. The paper later reports best performance at $\mathcal{L}_{sc}$1 (Zhang et al., 2023). The exact U-Net layer partition into $\mathcal{L}_{sc}$2 strata, however, is not specified.

The ablation profile supports the modular design. The baseline without SO, DA, or $\mathcal{L}_{sc}$3 achieves 82.62 average Dice. Adding SO only yields 82.80, DA only yields 82.97, SO + DA yields 83.11, and SO + DA + $\mathcal{L}_{sc}$4 yields 83.41 (Zhang et al., 2023). This pattern indicates that both modules contribute independently, with DA providing the slightly larger standalone gain and the consistency loss adding a further increment.

4. Empirical performance of the histopathology method

On the seven-client histopathology benchmark, the segmentation FedSODA reports the best average performance among the compared methods, with Dice: 83.41% and Accuracy: 92.53% (Zhang et al., 2023). The corresponding averages for FedAvg are 82.62 Dice and 91.87 Accuracy; for FedProx, 82.21 Dice and 91.71 Accuracy; for FedBN, 82.73 Dice and 91.83 Accuracy; and for HarmoFL, 81.94 Dice and 91.33 Accuracy. Relative to FedAvg, the average gain is about +0.79 Dice and +0.66 Accuracy.

Per-client Dice results are reported as 81.10 on CoNSeP, 87.06 on CPM-17, 86.49 on CRAG, 82.61 on CryoNuSeg, 87.98 on Glas, 80.17 on Kumar, and 78.43 on TNBC (Zhang et al., 2023). A particularly emphasized result is on CRAG, where FedSODA improves Dice by +6.16 over HarmoFL and Accuracy by +2.78 over FedAvg. The authors interpret this as evidence that the method better handles heterogeneity between larger tissue structures and smaller nuclei datasets.

Several limitations are also explicit. The paper does not fully specify the synthetic-data construction beyond Gaussian sampling, the exact meaning of $\mathcal{L}_{sc}$5, the exact memory-bank implementation, the exact total local loss formula, the exact U-Net layer partition used in DA, the exact communication-round count, or the communication/computational overhead (Zhang et al., 2023). It also does not provide a formal privacy guarantee or a privacy analysis beyond standard FL assumptions. As a result, the method is best understood as an empirically effective but partially under-specified segmentation framework.

5. FedSODA for federated fine-tuning of LLMs

The later FedSODA is a resource-efficient framework for federated fine-tuning (FFT) of LLMs. It is designed for the case where clients are resource-constrained and cannot access, store, or backpropagate through the full model (Zhu et al., 18 Aug 2025). The full model is decomposed as

$\mathcal{L}_{sc}$6

where $\mathcal{L}_{sc}$7 is the emulator and $\mathcal{L}_{sc}$8 is the adapter, consisting of the last 3 transformer layers. The server compresses the emulator using Similarity Group Pruning (SGP) to obtain $\mathcal{L}_{sc}$9, and the client-side deployed model becomes

$\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$0

SGP prunes redundant layers by grouping layers and comparing endpoint hidden states with angular distance. If the original emulator has $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$1 layers and each of $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$2 groups removes $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$3 layers, then the sub-emulator has

$\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$4

In the main experiments, the pruning configuration is $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$5 and $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$6, so 12 total layers are pruned (Zhu et al., 18 Aug 2025).

Alignment between the full model and the pruned sub-model is handled by Orchestrated Distillation Alignment (ODA). ODA has two phases: pre-alignment before FFT begins and realignment every 5 rounds during training, with 1 round of pre-alignment in the reported experiments. The representation objective is written as

$\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$7

and ODA also includes a KL-divergence term between the outputs of the full LLM and the sub-LLM (Zhu et al., 18 Aug 2025). During ODA, the adapter parameters and the sub-emulator backbone are frozen, and only LoRA modules on selected retained sub-emulator layers are trainable.

Client efficiency is further improved by QLoRA. The framework uses 4-bit NF4 quantization and trains only lightweight LoRA parameters. The LoRA update is written as

$\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$8

with LoRA rank $\xi = \max\left(0, |y-\hat{y}| - \epsilon \right),$9 and alpha $\mathcal{L}_{sc}=\mathcal{L}_{ce}(\xi y, \hat{y}).$0 (Zhu et al., 18 Aug 2025). During client training, only adapter LoRA is updated. During ODA, only sub-emulator LoRA is updated. The server retains the full LLM and the public dataset used for pruning and alignment, while clients receive only the quantized sub-LLM.

The experimental setup uses LLaMA3-8B, Qwen2-7B, and Mistral-7B; downstream tasks GSM8K, SST-2, and BoolQ; and WikiText-2 as the public dataset for SGP and ODA (Zhu et al., 18 Aug 2025). Non-IID experiments use 10 clients with Dirichlet-based label skew, and the software stack includes PyTorch 2.5 on NVIDIA H800 GPUs.

6. Efficiency results, comparative position, and recurrent ambiguities

The LLM FedSODA reports three headline efficiencies: 70.6% average communication reduction, 75.6% storage reduction, and 3.1% task accuracy improvement relative to key baselines (Zhu et al., 18 Aug 2025). The main comparison set is FedPETuning, FedOT, and FedBiOT. Concrete system numbers show that FedPETuning communicates about 30.4M–30.9M parameters per round and has model size 13.5–15.0 GB, while FedSODA communicates about 3.0M–3.3M parameters per round and has model size 3.2–3.8 GB. Trainable parameters decrease from about 15.2M–15.5M in FedPETuning to about 1.3M–1.4M in FedSODA (Zhu et al., 18 Aug 2025).

In task performance, FedSODA is consistently much better than FedOT, usually competitive with or slightly better than FedBiOT, and often approaches FedPETuning despite deploying only a compressed, quantized sub-model to clients (Zhu et al., 18 Aug 2025). On BoolQ, the ODA ablation is especially diagnostic: without pre-alignment and realignment, scores are 62.4 / 63.9 / 61.3 for LLaMA3-8B / Qwen2-7B / Mistral-7B; with both pre-alignment and realignment, they rise to 68.6 / 72.8 / 66.2. The paper also reports that FedSODA’s computation and communication are on average only 8.55% and 9.87% of those of FedPETuning.

Across both meanings of the name, FedSODA occupies a recognizable design space in federated learning: it modifies client interaction and aggregation to handle strong non-IID structure, but it does so through domain-specific machinery. In histopathology segmentation, the relevant axes are sample imbalance, target scale, and cross-organ heterogeneity (Zhang et al., 2023). In LLM fine-tuning, the dominant constraints are model size, gradient mismatch between sub-model and full model, and communication cost (Zhu et al., 18 Aug 2025). The commonality is therefore not algorithmic identity but a shared federated-systems orientation toward heterogeneous clients.

A persistent source of confusion is the proximity of the name to other federated methods. The semantic-segmentation framework FedSaaS is explicitly not FedSODA (Yu et al., 14 May 2025). The object-detection method on SODA10M is FedSTO, not FedSODA (Kim et al., 2023). The optimizer FedSOL targets heterogeneous FL through proximal perturbation and stabilized orthogonal learning, again under a different name (Lee et al., 2023). A practical implication is that citations to “FedSODA” are most precise when accompanied by the full title or the arXiv identifier, since the term now refers to multiple unrelated methods.