---
title: Federated Mutual Learning (FML)
url: https://www.emergentmind.com/topics/federated-mutual-learning-fml
type: topic
---

# Federated Mutual Learning (FML)

Federated Mutual Learning (FML) defines a broad family of federated learning (FL) frameworks wherein participating clients collaborate by sharing model predictions, soft labels, or models for mutual knowledge distillation, rather than, or in addition to, exchanging raw gradient updates or full model weights. Motivated by the limitations of canonical FL (especially FedAvg) under data, model, and objective heterogeneity, FML algorithms employ distributed or bidirectional knowledge transfer based on Kullback-Leibler (KL) divergence over model outputs. FML encompasses both centralized (server-orchestrated) and decentralized (peer-to-peer) protocols, supports homogeneous or heterogeneous client architectures, and can be instantiated with or without auxiliary public data. This paradigm demonstrably improves generalization, personalization, robustness to data heterogeneity, and communication efficiency compared to classical parameter-averaging approaches.

## 1. Core Principles and Formal Objectives

In the canonical FML protocol—prototyped by "Federated Learning Framework via Distributed Mutual Learning"—a system consists of $K$ clients, each with private data $D_i$ and model parameters $\theta_i$ [2503.05803]. Clients also receive access to a small public dataset $X_\text{pub} = \{x_1,\dots,x_M\}$ for exchanging knowledge. The training objective for each client $i$ augments the vanilla local empirical loss with a KL-based mutual knowledge distillation term:

\[
\mathcal{L}_i^{\text{FML}}(\theta_i) = L_i(\theta_i) + \lambda \frac{1}{M} \sum_{m=1}^M \mathrm{KL}\left(p_i(x_m \mid \theta_i)~\|~\bar p_{-i}(x_m)\right)
\]

where $L_i(\theta_i)$ is the cross-entropy loss on $D_i$, $p_i(x_m \mid \theta_i)$ is the softmax output (with optional temperature $T$) of client $i$ on public example $x_m$, and $\bar p_{-i}(x_m)$ denotes the arithmetic mean of all other clients' outputs on $x_m$.

By setting $\lambda=0$, the formulation reduces to standard FL; for $\lambda > 0$, each client is encouraged to align its output distribution to the consensus of peers on the public set. This mechanism supports model-architecture heterogeneity (since only output logits are exchanged) and can incorporate variants such as bi-directional distillation, clustering-based peer selection, or applicability to decentralized topologies [2110.07868, 2012.13063, 2402.01863, 2506.10207, 2006.16765].

## 2. Algorithmic Instantiations and Protocol Variants

FML implementations differ in topology, communication medium, and aggregation strategies but share key stages:

1. **Local private update:** Each client performs one or more epochs of local data optimization.
2. **Prediction exchange:** Clients compute output vectors (typically softmax probabilities) on a shared public set or their own data (depending on the protocol).
3. **Aggregation:** Server or peer(s) aggregate these output vectors (average or otherwise).
4. **Distillation update:** Clients incorporate a KL-regularized objective based on peer outputs.

A representative centralized FML training loop [2503.05803]:

```text
For each round t = 1 ... R:
    1. Local private update (SGD on D_i)
    2. Clients compute p_i^(t) = softmax(f(x_m; θ_i)/T) for m=1..M
    3. Clients send p_i^(t) to server
    4. Server computes mean predictions \bar{p}_{-i}^{t} for each client
    5. Server broadcasts \bar{p}_{-i}^{t} to each client i
    6. Each client updates θ_i using ∇L_i(θ_i) + λ∇KL(·)
```

Variants include:

- **FedMe:** Clients exchange full models (of potentially different architectures), perform deep mutual learning (DML) over local data, and use validation loss-based model selection for automatic architectural tuning [2110.07868].
- **Def-KT and DFML:** Decentralized, peer-to-peer protocols where clients pair with others, exchange models, and conduct mutual distillation locally, obviating the need for a central server or public data [2012.13063, 2402.01863].
- **FedMLAC:** Clients maintain both a personalized local model and a lightweight, globally shared "Plug-in" model, enforcing bidirectional KL-based distillation and employing layer-wise pruning in aggregation to enhance robustness in heterogeneous and potentially adversarial environments [2506.10207].
- **FML ("meme+personal"):** Each client simultaneously trains a global ("meme") model and a local personalized model, exchanging meme updates with the server and mutually distilling between meme and local on private data [2006.16765].

## 3. Communication and Computational Efficiency

FML protocols typically achieve substantial reductions in communication volume compared to weight-sharing baselines. For example, in a standard centralized FML, the per-round per-client communication cost is $M \cdot C \cdot 32$ bits (sharing $M$ softmax vectors of dimension $C$), versus $P \cdot 32$ for full model exchange (where $P$ is the parameter count) [2503.05803]. A practical case showed a 50x reduction: $2\,000$ floats (soft outputs) versus $10^5$ floats (weights). Model-exchange variants (FedMe, Def-KT, DFML), by their nature, transmit model parameters but address heterogeneity and personalization tradeoffs not attainable with weight-averaging.

Computation overhead for each client may increase relative to FedAvg due to the need for dual-model forward and backward passes (e.g., meme and local in FML, personalized and plug-in in FedMLAC), most notably doubling per-batch cost in vanilla mutual learning [2006.16765, 2506.10207].

## 4. Privacy Properties and Security Considerations

FML mitigates many privacy and inference risks associated with gradient or weight sharing. Since only softpredictions on a public or non-sensitive set are exchanged (rather than gradients on private data or full weights), risk of model inversion attacks is reduced [2503.05803]. In research to date, no formal differential privacy proofs are provided; rather, privacy claims rely on the inability to infer private data from soft outputs on known public inputs. Some FML extensions propose the addition of noise to logit vectors or further aggregation protocols to amplify privacy [2503.05803, 2006.16765].

Robustness to adversarial and byzantine participants is addressed explicitly in recent frameworks, such as FedMLAC's layer-wise pruning aggregation, which filters outlier updates based on parameter deviation statistics to defend against poisoning or corrupted-label attacks [2506.10207].

## 5. Empirical Results and Application Domains

Across multiple studies, FML frameworks yield superior accuracy, generalization, and personalization under both IID and non-IID data splits, often with faster or more stable convergence than FedAvg or parameter-averaging protocols.

For instance, on a face-mask detection task, distributed FML achieved 94.45% average accuracy, outperforming vanilla FedAvg (92.65%) and asynchronous weight aggregation (92.74%), with markedly lower communication burden [2503.05803]. In decentralized FML (e.g., Def-KT, DFML), accuracy gains of 2–5% over baselines are observed under severe heterogeneity in both data and model space [2012.13063, 2402.01863]. FedMLAC demonstrates 1–5% gains in F1 or accuracy for federated audio classification scenarios, particularly robust to noisy or adversarial data [2506.10207].

Application domains include computer vision (MNIST, CIFAR-10/100), speech and audio recognition (GSC, IEMOCAP), environmental sound recognition, and natural language tasks (Shakespeare) [2110.07868, 2506.10207, 2006.16765].

## 6. Extensions, Limitations, and Open Questions

FML algorithms have been extended to address various axes of heterogeneity:

- **Model-architecture heterogeneity:** By aligning soft predictions rather than layers or weights, FML natively accommodates clients running distinct neural architectures [2110.07868, 2506.10207, 2402.01863, 2006.16765].
- **Data heterogeneity and personalization:** Bidirectional distillation and per-client model components (local/personal/model exchange/plug-in) ensure that clients benefit from global knowledge without sacrificing performance on idiosyncratic data [2110.07868, 2506.10207, 2006.16765].
- **Decentralization and serverless protocols:** Fully peer-to-peer implementations eliminate single-point failures and increase robustness [2012.13063, 2402.01863].

Key limitations include the reliance, in some variants, on public datasets for prediction exchange, which may not be feasible for certain privacy settings [2503.05803]. Most theoretical analyses rely on FedAvg's convergence properties; formal non-convex convergence and privacy amplification guarantees remain largely open [2006.16765, 2506.10207]. The management of distillation hyperparameters (e.g., $\lambda$, KL annealing, mutual learning weights) and interaction with advanced privacy-preserving mechanisms (DP, encrypted aggregation) are active areas of inquiry. Adaptive scheduling and integration with robust aggregation further strengthen these frameworks [2503.05803, 2506.10207].

## 7. Representative Algorithms and Comparison Table

The FML landscape includes multiple algorithmic variants, distinguished by architecture, communication, and privacy strategies:

| Framework             | Centralized/Decentralized | Heterogeneous Model Support | Public Data Req. | Robustness Components                    | Reference      |
|-----------------------|--------------------------|----------------------------|------------------|-------------------------------------------|---------------|
| FML (loss exchange)   | Centralized              | Yes                        | Yes              | Soft-prediction only                      | [2503.05803]  |
| FedMe                 | Centralized              | Yes                        | Optional (for clustering) | Deep mutual learning + model selection  | [2110.07868]  |
| Def-KT                | Decentralized            | Yes                        | No               | Peer-to-peer bidirectional distillation   | [2012.13063]  |
| DFML                  | Decentralized            | Yes (non-restrictive)      | No               | WSM, cyclic distillation, peer aggregation| [2402.01863]  |
| FedMLAC               | Centralized              | Partial (Plug-in fixed)    | No               | Layer-wise pruning aggregation (LPA)      | [2506.10207]  |
| FML (meme+personal)   | Centralized              | Yes                        | No               | Personalized-local + meme mutual learning | [2006.16765]  |

Each framework demonstrates improved generalization and/or personalization under conditions of data and/or model heterogeneity, with varying privacy and robustness properties. FML has shifted the focus of federated optimization from direct parameter fusion to output-level consensus, providing a flexible foundation for next-generation FL systems.

Source: https://www.emergentmind.com/topics/federated-mutual-learning-fml