Fed-MedLoRA: Federated Medical Fine-Tuning

• Fed-MedLoRA is a model-agnostic federated framework that fine-tunes large language models using low-rank LoRA adapters to handle privacy and heterogeneity in medical data.
• It reduces communication by over 98.5% by transmitting only adapter parameters, making multi-institution collaboration both efficient and secure.
• Empirical evaluations in clinical information extraction show significant F1 improvements, rivaling centralized training while enabling low-resource new-site adaptation.

Searching arXiv for the cited Fed-MedLoRA paper and closely related federated LoRA work to ground the article in current literature. Fed-MedLoRA is a model-agnostic federated fine-tuning framework for adapting LLMs to medical applications under privacy, bandwidth, and cross-site heterogeneity constraints. Its defining design choice is to keep the pretrained backbone frozen and to federate only low-rank LoRA adapters, so that institutions collaborate without exchanging raw clinical text or full model weights. In the same line of work, Fed-MedLoRA+ adds adaptive, data-aware aggregation by reweighting client updates with a server-side validation signal, with the stated goal of improving convergence under heterogeneous clinical data. The framework is instantiated for clinical information extraction, including named entity recognition and relation extraction, and is evaluated in in-domain, external-validation, and low-resource new-site settings (Li et al., 29 Jan 2026).

1. Concept and problem setting

Fed-MedLoRA is motivated by two barriers that recur in medical LLM development. First, clinical text usually cannot be pooled across hospitals because of privacy and governance constraints. Second, conventional federated learning communicates full model weights each round, which is impractical for multi-billion-parameter LLMs and is especially brittle when sites differ in demographics, disease prevalence, documentation style, annotation completeness, and label distributions (Li et al., 29 Jan 2026).

Within that setting, each institution acts as a federated client with local data, while the server coordinates training by distributing and aggregating LoRA modules rather than the full backbone. This preserves the standard federated-learning property that raw patient data are not transferred. A central claim of the framework is that the relevant unit of communication for medical LLM adaptation is not the full model but the low-rank adapter parameters. The paper states that, for LLaMA3-8B, full-parameter fine-tuning would require roughly 29.92 GB per site per round in float32, whereas Fed-MedLoRA transmits only LoRA parameters, reducing total communication by 98.5%; the transmitted parameter count is 41,943,040 versus 8,030,261,248 total model parameters (Li et al., 29 Jan 2026).

A closely related healthcare study formulates the same general problem as federated LoRA fine-tuning under extreme non-IID conditions, with a central aggregator server and $K$ client institutions, each with dataset $\mathcal{D}_k$, sample size $n_k$, and total data $N=\sum_k n_k$. That work likewise argues that privacy-sensitive medical data, non-IID heterogeneity, and cross-institution collaboration jointly motivate LoRA-based federated adaptation rather than centralized fine-tuning (Chen et al., 1 Oct 2025). This suggests that Fed-MedLoRA is best understood not only as a single framework, but also as a representative point in a broader class of medical federated LoRA methods.

2. LoRA parameterization and federated optimization

Fed-MedLoRA adopts the standard LoRA parameterization. Let the pretrained backbone be $W^{(0)} \in \mathbb{R}^{d \times l}$. Instead of updating the full weight matrix, the trainable update is factorized as

$\Delta W = BA,$

where

$$B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times l}, \quad r \ll \min(d,l).$$

The adapted weight becomes

$W^{(0)} + BA.$

Only $A$ and $B$ are updated; the pretrained backbone remains frozen (Li et al., 29 Jan 2026).

The round-based federated procedure is a LoRA-restricted variant of FedAvg. At communication round $\mathcal{D}_k$0, the server samples a subset of clients $\mathcal{D}_k$1, transmits the current global LoRA modules $\mathcal{D}_k$2, and each selected client performs local fine-tuning for $\mathcal{D}_k$3 epochs on its own clinical data. The client then returns only the updated adapter parameters

$\mathcal{D}_k$4

The server aggregates them using dataset-size weights: $\mathcal{D}_k$5 where

$\mathcal{D}_k$6

The final model is then

$\mathcal{D}_k$7

This is the core Fed-MedLoRA update rule (Li et al., 29 Jan 2026).

A related healthcare formulation expresses the local adapter update as

$\mathcal{D}_k$8

with a FedAvg-style global aggregation

$\mathcal{D}_k$9

That paper also presents an equal-weight workflow variant,

$n_k$0

and interprets the iterative broadcast-and-merge loop as privacy-preserving knowledge flow across institutions (Chen et al., 1 Oct 2025). In the Fed-MedLoRA context, the same logic supports the claim that LoRA adapters provide a tractable communication interface for medical collaboration.

3. Fed-MedLoRA+ and adaptive, data-aware aggregation

Fed-MedLoRA+ retains the same LoRA-based client training protocol but modifies the server aggregation rule. The stated motivation is that healthcare institutions are not equally representative: some may have unusual patient populations, incomplete annotations, or noisier updates. Plain averaging by sample size can therefore overweight harmful updates under non-IID clinical data (Li et al., 29 Jan 2026).

To address this, Fed-MedLoRA+ introduces a small validation set held by the server,

$n_k$1

with $n_k$2. For each client $n_k$3, the server evaluates the model built from that client’s updated adapters on $n_k$4, using the validation loss

$n_k$5

Losses are converted into influence scores through a softmax over negative loss: $n_k$6 The final aggregation weight combines client data size and validation influence: $n_k$7 The server then aggregates adapters as

$n_k$8

If all $n_k$9 are equal, the method reduces to standard sample-size weighting. If a client’s update contributes less on the validation set, that client is down-weighted (Li et al., 29 Jan 2026).

This adaptive aggregation has clear parallels in medical LoRA literature. In cardiac MRI segmentation, Rate-My-LoRA also uses validation-informed weighting to penalize aggregated updates that harm client validation accuracy. There the adaptive weight is

$N=\sum_k n_k$0

and the server aggregates

$N=\sum_k n_k$1

with a decayed penalty $N=\sum_k n_k$2 (He et al., 6 Jan 2025). Fed-MedLoRA+ is not identical to that formulation, but the comparison indicates a recurring design principle in medical federated LoRA: validation-aware reweighting is used to counteract cross-site heterogeneity.

4. Empirical evaluation in clinical information extraction

The principal application studied for Fed-MedLoRA is clinical information extraction. The task includes named entity recognition (NER) and relation extraction (RE) for transforming patient narratives into structured medical entities and relations. The study uses five cohorts totaling 42,198 entities and 41,570 relations: MIMIC-III, MTSamples, UTP, I2B2, and YNHH. Four entity types are defined—Problem, Treatment, Test, and Drug—along with 16 relation types, including Severity, Temporal, Negation, Dosage, Strength, Reference range, Uncertain, Lab value, Route, Frequency, Subject, Form, Condition, Duration, Body location, and Course (Li et al., 29 Jan 2026).

The evaluation protocol comprises three scenarios. The first is in-domain training/testing, with two-site experiments on MIMIC-III + MTSamples and three-site experiments on MIMIC-III + MTSamples + UTP. The second is external validation, training on some cohorts and testing on unseen cohorts, including two-site training tested on UTP and I2B2, and three-site training tested on I2B2. The third is low-resource new-site adaptation, using YNHH notes as a new hospital with limited annotations. Metrics are precision, recall, and micro F1 under both Strict and Lenient matching criteria (Li et al., 29 Jan 2026).

The reported baselines include Bio_ClinicalBERT, LLaMA-3, DeepSeek-R1-Distill, GPT-4o in zero-shot comparisons, single-site fine-tuning, centralized training, and the prior federated LoRA method FedSA-LoRA (Li et al., 29 Jan 2026). The quantitative findings are explicit. The paper reports up to 65% absolute F1 improvement over zero-shot LLMs, with especially large gains for RE. In the two-site LLaMA3 experiment on MIMIC-III, NER strict F1 improves from 0.345 to 0.850 with Fed-MedLoRA+, and RE strict F1 improves from 0.203 to 0.860 (Li et al., 29 Jan 2026).

Fed-MedLoRA+ is also reported to outperform single-site fine-tuning by around ~25% higher F1 in many in-domain comparisons, again with especially large gains on RE. For example, in the two-site setting, LLaMA3 single-site RE on MIMIC-III is 0.648, whereas Fed-MedLoRA+ reaches 0.860. The federated methods also approach centralized training: in LLaMA3 two-site MIMIC-III NER, centralized training reaches 0.856 and Fed-MedLoRA+ reaches 0.850 (Li et al., 29 Jan 2026).

The new-site adaptation case study on YNHH is central to the framework’s medical relevance. For LLaMA3-8B, the zero-shot baseline yields strict F1 0.397 and lenient F1 0.524. Fed-MedLoRA improves these to strict F1 0.708 and lenient F1 0.794, while Fed-MedLoRA+ further improves them to strict F1 0.720 and lenient F1 0.809. In the three-site setting, Fed-MedLoRA+ reaches strict F1 0.730 and lenient F1 0.854 (Li et al., 29 Jan 2026). A plausible implication is that the framework is designed not only for multi-site training among existing members of a consortium, but also for bringing a newly participating hospital into a trained federated model family with limited labeled data.

5. Relation to other federated LoRA methods

Fed-MedLoRA belongs to a rapidly developing family of federated LoRA methods, but its formulation differs from several adjacent approaches. One important comparison concerns the algebraic mismatch created when LoRA factors are averaged independently. FedEx-LoRA formalizes the point that

$N=\sum_k n_k$3

and fixes this by adding a residual correction to the frozen base weight matrix so that the post-aggregation model exactly matches the average of client-side products. The method computes

$N=\sum_k n_k$4

and updates

$N=\sum_k n_k$5

This exact-aggregation view directly targets a limitation of factor-wise FedAvg that Fed-MedLoRA inherits in its basic form (Singhal et al., 2024).

A second line of work emphasizes asymmetry between LoRA factors under non-IID data. FedSA-LoRA argues that the $N=\sum_k n_k$6 matrix tends to encode more general or shared knowledge, whereas $N=\sum_k n_k$7 is more client-specific. It therefore keeps both factors trainable locally but shares only $N=\sum_k n_k$8 with the server, so that the client model after aggregation becomes

$N=\sum_k n_k$9

That design is motivated by both an asymmetry analysis and empirical cosine-similarity findings, and it stands in contrast to fully symmetric FedAvg-style exchange of $W^{(0)} \in \mathbb{R}^{d \times l}$0 and $W^{(0)} \in \mathbb{R}^{d \times l}$1 (Guo et al., 2024).

A third family modifies the training schedule rather than the aggregation target. RoLoRA uses alternating minimization: in odd rounds it freezes $W^{(0)} \in \mathbb{R}^{d \times l}$2, updates $W^{(0)} \in \mathbb{R}^{d \times l}$3, and aggregates $W^{(0)} \in \mathbb{R}^{d \times l}$4; in even rounds it freezes $W^{(0)} \in \mathbb{R}^{d \times l}$5, updates $W^{(0)} \in \mathbb{R}^{d \times l}$6, and aggregates $W^{(0)} \in \mathbb{R}^{d \times l}$7. The effective updates in alternating rounds are

$W^{(0)} \in \mathbb{R}^{d \times l}$8

and

$W^{(0)} \in \mathbb{R}^{d \times l}$9

The stated purpose is robustness when the number of trainable LoRA parameters decreases and heterogeneity increases (Chen et al., 2024).

Medical imaging variants move further toward explicit personalization. Med-DualLoRA decomposes each adapted layer into a shared global LoRA component and a client-private local LoRA component,

$\Delta W = BA,$0

and aggregates only the global component: $\Delta W = BA,$1 Local adapters remain private throughout training and inference (Perramon-Llussà et al., 11 Mar 2026). Compared with Fed-MedLoRA, which exchanges a single global adapter pair per round, this is a more explicitly personalized federated design.

At the opposite end of the design space, FedLEASE introduces clustered multi-expert LoRA with adaptive expert allocation and adaptive top-$\Delta W = BA,$2 expert selection. It clusters clients using representation similarity of LoRA $\Delta W = BA,$3 matrices, chooses the number of experts by silhouette score, and trains cluster-specific LoRA experts rather than a single shared adapter (Wang et al., 18 Sep 2025). This suggests that, in the broader federated-LoRA taxonomy, Fed-MedLoRA occupies the simpler global-adapter regime, while later work increasingly decomposes the problem into exactness, asymmetry, alternating optimization, personalization, or expert allocation.

6. Practical significance, limitations, and broader interpretation

The operational feasibility claims of Fed-MedLoRA are unusually concrete. The paper states that LLaMA3-8B training was feasible on a single RTX 4090 16 GB, that inference used around 6.79 GB memory, and that smaller 1B backbones could be trained on mid-range GPUs and deployed on lighter hardware. Simulations with up to 10 sites reportedly show that Fed-MedLoRA+ remains close to centralized performance at smaller numbers of clients and degrades only modestly at 10 sites, with a drop of about 3.9% for NER and 7.5% for RE versus centralized training (Li et al., 29 Jan 2026). Within the paper’s framing, these figures position the framework as a deployable rather than purely conceptual approach.

The privacy model is standard federated training rather than formal privacy protection. The paper explicitly states that no raw patient data are shared and only low-rank adapter updates are transmitted, but also notes that secure aggregation and differential privacy were not implemented (Li et al., 29 Jan 2026). A related healthcare federated-LoRA study adds a Cypherium blockchain-based identity and incentive mechanism with public-private key registration, identity verification, auditability, integrity, and reward distribution, while clarifying that this infrastructure does not change LoRA optimization itself (Chen et al., 1 Oct 2025). This suggests that Fed-MedLoRA’s privacy and governance claims are strongest at the systems level of data locality and weakest at the level of formal privacy guarantees.

Several limitations are explicit or readily implied by the reported experiments. Fed-MedLoRA is demonstrated on clinical information extraction, not on broader medical generation or multimodal tasks (Li et al., 29 Jan 2026). The related healthcare study is limited to medical QA and notes that transfer to clinical generation, multimodal medical reasoning, dialogue systems, and decision support remains to be shown (Chen et al., 1 Oct 2025). More generally, the fact that alternative methods such as FedEx-LoRA, FedSA-LoRA, RoLoRA, Med-DualLoRA, and FedLEASE target mathematically inexact aggregation, factor asymmetry, alternating optimization, client-aware decomposition, or clustered experts indicates that simple sample-size averaging of a single shared LoRA pair is only one point in a larger design space (Singhal et al., 2024, Guo et al., 2024, Chen et al., 2024, Perramon-Llussà et al., 11 Mar 2026, Wang et al., 18 Sep 2025).

In that broader perspective, Fed-MedLoRA’s central contribution is the claim that medical LLM adaptation can be made federated, parameter-efficient, and practically trainable by exchanging only low-rank adapters. Fed-MedLoRA+ extends that claim by making aggregation data-aware through validation-based influence scores. The broader literature suggests several possible next steps—exact aggregation, selective sharing, personalized local-global decomposition, and expert allocation—but the core Fed-MedLoRA formulation remains a concise reference design for privacy-preserving, cross-institutional medical LLM adaptation (Li et al., 29 Jan 2026).