FedKEI: Federated Knowledge-Enhanced Initialization

• FedKEI is a federated learning paradigm that initializes new tasks using historical, task-specific adapters and heads.
• It leverages server-side clustering and bi-level optimization to create informed starting points that mitigate data and system heterogeneity.
• Empirical results demonstrate improved AUC, LCA, and accelerated adaptation over baselines in dynamic healthcare applications.

Federated Knowledge-Enhanced Initialization (FedKEI) denotes a federated learning paradigm in which prior knowledge is converted into an informed starting state for later decentralized optimization. In the literature, the term is explicitly introduced for federated adapter tuning of foundation models in dynamic healthcare, where historical task-specific adapters and heads are clustered and reweighted to initialize learning of new diseases across clients (Peng et al., 14 Aug 2025). In a broader interpretive sense, closely related earlier studies treat server-side pre-training on proxy data, foundation-model distillation, and other knowledge-bearing latent states as instances of the same core idea: the initial condition of federated optimization materially shapes convergence speed, robustness to heterogeneity, and attainable performance (Nguyen et al., 2022).

1. Definition and scope

In its narrowest usage, FedKEI refers to the framework introduced for federated continual adaptation of foundation-model adapters in healthcare. There, each client observes a task stream $\{\mathcal{T}_i^1,\mathcal{T}_i^2,\cdots\}$, each task $\mathcal{T}_i^t \subset \mathcal{D}_i$ has a label set satisfying $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$ for all $j<t$, the foundation-model backbone $F^*(\cdot)$ is fixed, and only a lightweight adapter $g_\omega(\cdot)$ and task head $h_\phi(\cdot)$ are tuned. The trainable task module is $\theta_i^t=(\omega_i^t,\phi_i^t)$, and knowledge-enhanced initialization means that a new task is not started from random adapter/head parameters, nor from a single global average, but from a weighted aggregation of historical task-specific modules collected across clients and time (Peng et al., 14 Aug 2025).

In a broader usage suggested by adjacent work, FedKEI covers initialization schemes in which external or federatedly aggregated knowledge is injected before ordinary federated optimization begins. That broader family includes server-side pre-training on proxy data, distillation from a foundation model into a smaller federated student, and initialization of nonstandard latent objects such as synthetic knowledge variables or codebooks. This suggests that FedKEI is best understood as an initialization-centric design principle rather than as a single optimizer.

2. Precursors and empirical foundation

The empirical basis for FedKEI predates the term itself. A large-scale study of federated initialization showed that starting from a pre-trained model “reduces the training time required to reach a target error rate and enables the training of more accurate models (up to 40%)” and also “reduces the effect of both data and system heterogeneity” across CIFAR-10, FEMNIST, Stack Overflow, and Reddit benchmarks (Nguyen et al., 2022). That work kept the FL algorithm within the FedOpt framework and varied only the initial global model $x^0$, but still found changes in convergence, final accuracy, client drift, and even optimizer ranking. A complementary continual-learning line treated the global model sent at each round as the relevant initializer, and used server-side importance weights computed on a $1\%$ proxy dataset to regularize local updates, thereby improving the “initial accuracy” of later rounds under non-IID data (Yao et al., 2020). In medical segmentation, a distinct precursor distilled SAM-Med2D into a lightweight student and used the distilled weights as the initial global model for FedAvg, improving Dice from $\mathcal{T}_i^t \subset \mathcal{D}_i$0 to $\mathcal{T}_i^t \subset \mathcal{D}_i$1 under age skew and reaching strong performance in fewer than 40 rounds (Li et al., 2023).

Work	Knowledge source	Initialized object
Pre-trained FL baseline (Nguyen et al., 2022)	Proxy/public data or public checkpoints	Global model $\mathcal{T}_i^t \subset \mathcal{D}_i$2
Continual local training (Yao et al., 2020)	Proxy-data parameter importance	Round-wise global initializer
Foundation-model distillation (Li et al., 2023)	SAM-Med2D + proxy labels	Lightweight segmentation student
Meta knowledge condensation (Liu et al., 2022)	Other clients’ previous-round meta knowledge	Synthetic knowledge variable $\mathcal{T}_i^t \subset \mathcal{D}_i$3
FedKEI (Peng et al., 14 Aug 2025)	Historical adapters and heads across clients and tasks	New-task adapter/head initialization

These precedents establish three recurrent claims. First, initialization is not a cosmetic detail in FL benchmarking. Second, knowledge injected before federated rounds can partially absorb the role of heterogeneity-specific optimization machinery. Third, the initialized object need not always be the full model parameter vector; some methods initialize synthetic support sets or latent codebooks instead.

3. Core formulation of FedKEI

FedKEI is formulated for adapter tuning with a fixed foundation-model backbone $\mathcal{T}_i^t \subset \mathcal{D}_i$4. An adapter $\mathcal{T}_i^t \subset \mathcal{D}_i$5 is attached to the backbone, giving $\mathcal{T}_i^t \subset \mathcal{D}_i$6, and the task-specific predictor is $\mathcal{T}_i^t \subset \mathcal{D}_i$7. For task $\mathcal{T}_i^t \subset \mathcal{D}_i$8, the module is $\mathcal{T}_i^t \subset \mathcal{D}_i$9, and the historical knowledge pool at time $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$0 is

$\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$1

The paper’s optimization target for current tasks is

$\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$2

with

$\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$3

The central premise is that past task-specific modules are the transferable knowledge objects. Because LoRA-style adapters can occupy less than $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$4 of a ViT, the framework stores one adapter/head pair per task and reuses the accumulated pool for future adaptation instead of continually overwriting one shared model (Peng et al., 14 Aug 2025).

Knowledge generalization begins with server-side clustering of the historical modules. If there are $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$5 historical task modules and $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$6 clusters, the cluster assignment is represented by a binary matrix $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$7. The cluster-specific module is the normalized average of the modules assigned to cluster $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$8. Intra-cluster weights $\mathcal{Y}_i^t \neq \mathcal{Y}_i^j$9 refine this average: $j<t$0 Across clusters, inter-cluster weights $j<t$1 define the new-task initializer

$j<t$2

After optimization of these weights, the final initialization for client $j<t$3’s new task is

$j<t$4

and local training proceeds from

$j<t$5

4. Bi-level optimization and training workflow

FedKEI does not treat inter-cluster and intra-cluster weights as independent hyperparameters. It uses a bi-level optimization scheme in which the client-specific inter-cluster weights are optimized in the inner level and the shared intra-cluster weights are optimized in the outer level. For client $j<t$6 and new task $j<t$7, one inner-loop update is

$j<t$8

The server then updates $j<t$9 through

$F^*(\cdot)$0

implemented by gradient descent on each cluster weight vector,

$F^*(\cdot)$1

With the learned $F^*(\cdot)$2, each client then solves the actual personalized inner problem

$F^*(\cdot)$3

The interpretation supplied by the method is meta-learning-like but not MAML-style: the server learns cluster compositions that make later local adaptation of cluster mixtures more effective (Peng et al., 14 Aug 2025).

The operational pipeline is correspondingly staged. The server maintains the historical module pool, clusters it with $F^*(\cdot)$4-means++, initializes $F^*(\cdot)$5 from the cluster assignment matrix, sends cluster-specific modules to clients, receives gradients after inner-loop updates of $F^*(\cdot)$6, performs one outer-loop update of $F^*(\cdot)$7, recomputes cluster modules, and finally lets each client optimize $F^*(\cdot)$8 and construct $F^*(\cdot)$9. In the reported implementation, the inner-loop learning rate is $g_\omega(\cdot)$0, the outer-loop learning rate is $g_\omega(\cdot)$1, the inter-cluster inner updates run for 1 epoch, the number of outer-loop steps is 1, and $g_\omega(\cdot)$2 is tuned in $g_\omega(\cdot)$3. Communication cost is $g_\omega(\cdot)$4 per client; with $g_\omega(\cdot)$5, the reported communication is about $g_\omega(\cdot)$6 MB. On Derm-FL, the method incurs $g_\omega(\cdot)$7 min/task and $g_\omega(\cdot)$8 TFLOPs/task, compared with $g_\omega(\cdot)$9 min/task and $h_\phi(\cdot)$0 TFLOPs/task for FedAvg (Peng et al., 14 Aug 2025).

5. Empirical behavior

FedKEI is evaluated on three medical benchmark suites with different modalities. Derm-FL is built from ISIC19, HAM10000, PAD-UFES, and Derm7pt, split into 10 clients with 37,607 images and 5 sequential tasks. CXR-FL uses NIH-CXR-14, CheXpert, and VinDr-CXR as 3 clients with 79,010 images and 6 sequential tasks. OCT contains 84,495 retinal images, is partitioned into 5 clients with $h_\phi(\cdot)$1 label skew, and is organized into 3 tasks. The backbone is mainly ViT-B/16 pretrained on ImageNet, the adapter is LoRA, local fine-tuning lasts 30 epochs with learning rate $h_\phi(\cdot)$2 and batch size 64, and evaluation uses final AUC together with LCA, the average AUC across training epochs (Peng et al., 14 Aug 2025).

Dataset	FedKEI	Strongest baseline cited
Derm-FL	AUC $h_\phi(\cdot)$3, LCA $h_\phi(\cdot)$4	APPLE AUC $h_\phi(\cdot)$5, FedProx LCA $h_\phi(\cdot)$6
CXR-FL	AUC $h_\phi(\cdot)$7, LCA $h_\phi(\cdot)$8	APPLE AUC $h_\phi(\cdot)$9, APPLE / Per-FedAvg LCA $\theta_i^t=(\omega_i^t,\phi_i^t)$0
OCT	AUC $\theta_i^t=(\omega_i^t,\phi_i^t)$1, LCA $\theta_i^t=(\omega_i^t,\phi_i^t)$2	FFA-LoRA AUC $\theta_i^t=(\omega_i^t,\phi_i^t)$3, Per-FedAvg LCA $\theta_i^t=(\omega_i^t,\phi_i^t)$4

The method’s strongest reported effect is on rapid adaptation to later tasks. On Derm-FL task 5, APPLE reaches final AUC $\theta_i^t=(\omega_i^t,\phi_i^t)$5 at epoch 30, whereas FedKEI exceeds that level with $\theta_i^t=(\omega_i^t,\phi_i^t)$6 AUC at epoch 4. The corresponding wall-clock figures are $\theta_i^t=(\omega_i^t,\phi_i^t)$7 s for APPLE and $\theta_i^t=(\omega_i^t,\phi_i^t)$8 s for FedKEI. The ablation sequence is also structurally informative: learning only $\theta_i^t=(\omega_i^t,\phi_i^t)$9 already improves over random initialization, adding clustering improves further, learning $x^0$0 gives another gain, and the full bi-level formulation improves again. For Derm-FL, the transition from direct $x^0$1-learning to the full bi-level method moves AUC from $x^0$2 to $x^0$3 and LCA from $x^0$4 to $x^0$5; for CXR-FL, AUC from $x^0$6 to $x^0$7 and LCA from $x^0$8 to $x^0$9; for OCT, AUC from $1\%$0 to $1\%$1 and LCA from $1\%$2 to $1\%$3 (Peng et al., 14 Aug 2025).

The reported behavior is not tied to one adapter or one backbone. FedKEI remains best with Swin-B + LoRA and with ViT-B/16 + IA3, remains best under reversed synchronous task order and under asynchronous shuffled task order across clients, and also improves medical-foundation-model settings such as RETFound + LoRA on OCT and VoCo + LoRA on CC-CCII. This suggests that the main contribution is not a dataset-specific heuristic but the initialization mechanism itself.

6. Mechanistic interpretation, related variants, and limitations

A theoretical account for why initialization matters in FL is provided by later work on pre-training in FedAvg for two-layer CNNs. That analysis defines a filter as aligned at initialization if $1\%$4, shows that data heterogeneity primarily affects learning on misaligned filters, and derives a test-error bound in which the aligned-filter fraction $1\%$5 appears as an explicit term. The key conclusion is that starting with a pre-trained model typically yields fewer misaligned filters, so the adverse effect of heterogeneity is reduced without changing FedAvg itself (Jhunjhunwala et al., 11 Feb 2025). This does not prove the FedKEI algorithm, but it gives a formal mechanism for its core premise: knowledge-rich initialization changes the optimization geometry before local drift accumulates.

At the same time, the surrounding literature shows that “initialization” in federated systems is not a single object. FedMK uses another client’s previous-round meta knowledge as conditional initialization for the synthetic knowledge variable $1\%$6, not for model parameters; the initialized object is the client’s condensed dataset variables in a bi-level condensation process (Liu et al., 2022). In federated graph foundation modeling, AncDAI computes one mean-pooled prototype $1\%$7 per client, perturbs it by $1\%$8, and uses the resulting synthetic embeddings to initialize the global gVQ-VAE codebook $1\%$9, again showing that initialization may target a semantic bottleneck rather than the full parameter set (Zhu et al., 19 May 2025). By contrast, frameworks such as Knowledge-Injected Federated Learning, FEDMEKI, and FedKIM inject knowledge during local prediction, server-side alignment, or iterative foundation-model adaptation rather than through a dedicated warm-start rule, and therefore occupy adjacent but distinct positions in the design space (Fan et al., 2022).

Current boundary conditions are explicit. The 2025 FedKEI paper assumes a useful history of prior task-specific modules, depends on clustering quality, has not yet been tested at very large real-world FL scale, and identifies first-order approximations and model compression as future directions (Peng et al., 14 Aug 2025). Earlier initialization studies likewise depend on public proxy data, pretrained checkpoints, or semantically related source tasks (Nguyen et al., 2022). Taken together, the literature supports a narrow conclusion with strong empirical backing: in federated systems, initialization is a substantive design lever. What remains open is how broadly the FedKEI principle extends beyond adapter tuning, how to construct similar initializers when history or proxy data are weak, and how to combine initialization-centric transfer with privacy, personalization, and robustness constraints at larger scale.