Multi-Model Federated Learning

• Multi-Model Federated Learning is an extension of federated learning that concurrently trains multiple independent models across distributed clients, enhancing resource efficiency.
• Advanced scheduling strategies like Ranklist-Multi-UCB and Pareto-Multi-UCB exploit per-client loss statistics to optimize model assignments and accelerate convergence.
• Empirical studies on synthetic and real-world datasets reveal that MMFL achieves faster convergence and higher accuracy, especially under stringent resource and heterogeneous data constraints.

Multi-Model Federated Learning (MMFL) is an extension of federated learning that supports the parallel, distributed training of multiple independent models across a shared set of clients. In MMFL, clients possess data and compute resources but may be limited in the number of models they can train concurrently due to compute, memory, or communication constraints. The server's orchestration challenge is not only to select which clients participate each round but also to determine the assignment of models to clients, a decision that critically impacts convergence, accuracy, fairness, and overall resource efficiency.

1. Core MMFL Problem Formulation and Baseline Strategies

MMFL simultaneously maintains several models $\{w^{(m)}\}_{m=1}^M$, each updated using distributed, privacy-preserving local training at clients. The setting diverges from standard single-model FL in that: (a) each client $k$ may support a distinct subset of models per round, and (b) models are typically "unrelated" (i.e., representing different tasks or datasets).

A canonical extension of FedAvg, termed Multi-FedAvg, samples $K$ clients and, for each, assigns a model uniformly at random from the $M$ models (Bhuyan et al., 2022). After local SGD, the server aggregates the updates via weighted averaging per model, using client dataset sizes. While straightforward, this method is agnostic to per-model training loss, client-model data suitability, and client heterogeneity, often resulting in suboptimal convergence, especially under tight resource limits.

More sophisticated policies—such as Ranklist-Multi-UCB and Pareto-Multi-UCB—formulate the assignment as a multi-armed bandit problem: for each client, a score vector $A_t(k, i)$ is computed that balances historic loss and exploration via an upper confidence bound (UCB). Models are allocated either by ranking clients per model or by selecting those with Pareto-optimal score vectors. These policies more efficiently allocate client efforts to models that can benefit most, accelerating convergence and improving test accuracy, especially in resource-constrained regimes where only a minority of clients participate per round (Bhuyan et al., 2022).

2. Convergence Theory, Resource Tradeoffs, and Efficiency Gains

Analysis of MMFL under these assignment policies reveals convergence bounds that generalize single-task FL theory to the multi-model case. For instance, with random partitioning (MFA-Rand), the expected error per model decays as $\mathcal{O}(1/\sqrt{t})$, with the error bound including a variance term $C = \frac{(M-1)}{(N-1)}E^2(G^{(m)})^2$—demonstrating that variance and thus suboptimality grows with the number of models, client heterogeneity, and local update noise (Bhuyan et al., 2022).

A round-robin partitioning strategy (MFA-RR) yields tighter guarantees, approaching $\mathcal{O}(1/t)$ convergence if clients perform full gradient updates per round. Defining the "gain" of MMFL over sequential FL as $$g_P(M,\epsilon) = (M \cdot T_1(\epsilon)) / T_P(M, \epsilon)$$ (with $T_1(\epsilon)$ the required single-model rounds for error $\epsilon$), the analysis shows that $g_P(M,\epsilon) > 1$ for practical regimes, i.e., simultaneous training achieves nontrivial compute and communication reductions while maintaining or even improving accuracy (Bhuyan et al., 2022).

Empirical findings on synthetic and real-world datasets (e.g., CelebA) confirm that MMFL not only matches but may surpass sequential single-model FL in both convergence speed and final accuracy, particularly when the number of models per client is well matched to client resource budgets.

3. Model Assignment and Intelligent Scheduling

Assigning models to resource-heterogeneous clients is central to MMFL efficiency. Heuristic or random assignments lead to training inefficiencies and unfairness; advanced policies use per-client, per-model loss statistics to guide assignments.

Loss-aware Scheduling

In Ranklist-Multi-UCB, each client maintains a time-discounted cumulative loss and selection count per model, combining these with UCB terms to produce $A_t(k,i)$. The server constructs a rank list per model and cyclically allocates the highest-impact clients to each model in a round-robin sequence.

Pareto-Multi-UCB evaluates the Pareto-optimal set of clients (i.e., those not dominated in all dimensions of $A_t(k)$), randomly samples from among them, and assigns each selected client the model for which it ranks highest.

Both strategies outperform random allocation, delivering higher accuracy especially when the server can engage only a limited number of clients per round. These policies exploit the diversity in local loss landscapes to bias the training effort towards the models needing it most, while balancing exploration (via the UCB).

Resource-aware Extension

In practice, client heterogeneity extends beyond data to include compute, memory, and communication limits. Model assignment must thus further be constrained by per-client availability and resource profile, often resulting in combinatorial optimization (knapsack-type) problems. These are addressed in part by extending score-based selection to incorporate client capability, feasibility, and batch size adaptation (Lin et al., 12 Oct 2025).

4. Empirical Results and Implications

Key experimental findings include:

• With a small number of participating clients per round, loss-based policies (Ranklist-Multi-UCB, Pareto-Multi-UCB) outperform both Multi-FedAvg and single-model FedAvg (where each model trains with only a subset of clients). Initial convergence is faster, and test accuracy at plateau is higher (Bhuyan et al., 2022).
• When the number of clients per round is greater (e.g. 10 or more), Multi-FedAvg performance approaches that of single-model FedAvg; in such settings, the advantage of sophisticated scheduling is less pronounced, but still nontrivial for some tasks (Bhuyan et al., 2022).
• On both synthetic IID and non-IID partitions and real-world CelebA classification, MMFL achieves average test accuracies at least as high as those from sequence-trained single-model FedAvg, with particularly strong improvements under resource constraints and data heterogeneity (Bhuyan et al., 2022, Bhuyan et al., 2022).

These observations underscore MMFL's ability to both reduce training and communication time (when measured per model) and, with intelligent assignment, improve the utility of each communication round.

5. Extensions, Limitations, and Open Directions

Current MMFL frameworks make strong assumptions about the independence of models and the ability of clients to provide loss statistics per model. In realistic deployments, practical limitations include:

• Resource constraints: Each client may only be able to process one (or a small number of) model(s) at a time. The server's assignment mechanism must respect such constraints, possibly requiring more complex combinatorial optimization.
• Data and system heterogeneity: Per-client data may not be balanced across models, and system heterogeneity can limit the benefit of naive averaging or uniform allocation.
• Communication and synchronization overhead: As the number of models grows, so does the potential for synchrony issues and divergence, especially if client participation is variable.

Future research directions suggested in (Bhuyan et al., 2022) include the extension of MMFL frameworks to dynamic resource-aware assignment, possibly using online clustering and model-adaptive learning rates, as well as the paper of fairness constraints and incentive mechanisms to balance client load and address the potential for imbalance in task difficulty or system cost across models.

6. Comparative Summary Table: MMFL Model Assignment Policies

Policy Name	Assignment Approach	Key Features / Outcomes
Multi-FedAvg	Random client and uniform random model	Simple, baseline, matches single-model FL as clients per round increases
Ranklist-Multi-UCB	Loss/UCB-based, round-robin	Loss-aware, exploits data/model/client diversity, best under resource constraints
Pareto-Multi-UCB	Multi-objective Pareto optimality	Fair, multi-metric, assignment based on dominance in loss-space
FedAvg (Single-model)	Random client selection per model	Sequential, serves as a baseline; less efficient in MMFL context

7. Significance and Broader Impact

The principled extension of FL to the multi-model case fundamentally broadens federated optimization's applicability, particularly for scenarios with edge devices that must serve multiple concurrent learning tasks. MMFL demonstrates that, with appropriate client-model assignment and aggregation strategies, efficiency gains in both convergence speed and final model accuracy are achievable—thus reducing cost, communication, and wall-clock time per task. Furthermore, the results establish a foundation for more advanced scheduling, fairness enforcement, and dynamic resource allocation strategies, which are essential as FL scales to increasingly heterogeneous, multi-task, and resource-constrained real-world environments.