Complementary-Based Teacher Selection

• The paper introduces a budgeted coverage formulation that selects non-overlapping teacher models to achieve near-uniform class representation.
• It employs a greedy algorithm that iteratively chooses teachers based on marginal coverage gain measured by divergence metrics.
• Empirical results show that using a complementary teacher subset accelerates convergence and improves accuracy while mitigating catastrophic forgetting.

A complementary-based teacher selection mechanism is a strategy for assembling a subset of teacher models within multi-teacher knowledge distillation, federated learning, and related distributed training frameworks to maximize knowledge diversity and minimize redundancy. The mechanism seeks teachers whose contributions are most "complementary"—i.e., who offer non-overlapping, collectively balanced coverage of the knowledge or label space. This selection aims to enhance the efficacy of distillation, mitigate knowledge dilution, reduce communication and computational overhead, and address issues such as catastrophic forgetting under data or expertise heterogeneity (Xu et al., 11 Jul 2025).

1. Formal Problem Definition: Budgeted Coverage Formulation

At its core, complementary-based teacher selection formalizes the teacher selection problem as a variant of the budgeted coverage problem. Consider a federated learning round $r$ with $M$ candidate teacher models (clients), each characterized by a local class-frequency distribution: $$\Pi = \{\pi_1, \pi_2, ..., \pi_M\}, \qquad \mathbf D_{\pi_i} \in \mathbb R^C.$$ Here, $C$ denotes the number of classes; $\mathbf D_{\pi_i, c}$ is the empirical class $c$ frequency at client $\pi_i$, and $\mathbf U = (1/C, \dots, 1/C)$ represents the ideal uniform class distribution.

The selection objective is to pick a subset $\mathcal{T} \subseteq \Pi$ of size $K$ such that the sum of their distributions is as close as possible to uniformity according to a divergence metric $d$ (e.g., $L_1$, KL, or JSD):

$$\min_{\mathcal{T} \subseteq \Pi, |\mathcal{T}| = K} d\left(\sum_{\pi \in \mathcal{T}} \mathbf D_\pi, \, \mathbf U \right).$$

When each $\mathbf{D}_\pi$ is one-hot, this reduces to the classical maximum coverage problem, an NP-hard combinatorial problem; for general distributions, it remains NP-hard. The complementary-based approach thus leverages greedy approximate solvers to attain tractable, near-optimal solutions (Xu et al., 11 Jul 2025).

2. Greedy Teacher Selection Algorithm

To circumvent intractability, a greedy algorithm is employed. The server maintains an aggregated coverage vector $\mathbf D_{\rm agg}$ and iteratively adds the candidate teacher that most reduces the distance to uniformity.

Algorithm Outline:

• Input: Candidate set $\Pi$, their distributions $\{\mathbf D_\pi\}$, target uniform $\mathbf U$, selection budget $K$.
• Initialization: $\mathcal T \leftarrow \emptyset$, $\mathbf D_{\rm agg} \leftarrow 0$.
• For $i = 1$ to $K$:

1. For each $\pi \in \Pi \setminus \mathcal T$, compute $$\Delta_\pi = d(\mathbf D_{\rm agg} + \mathbf D_\pi, \mathbf U)$$.
2. Select $\pi^* = \arg\min_\pi \Delta_\pi$.
3. Update $\mathcal T \leftarrow \mathcal T \cup \{\pi^*\}$, $$\mathbf D_{\rm agg} \leftarrow \mathbf D_{\rm agg} + \mathbf D_{\pi^*}$$.

• Return: $\mathcal T$.

This procedure ensures that each newly selected teacher is maximally complementary, optimally covering gaps left by the current selection under marginal gain in objective.

3. Quantification of Complementarity

Complementarity is rigorously defined in terms of distributional dissimilarity and marginal coverage gain: $$\text{Compl}(\pi_i, \pi_j) = d(\mathbf D_{\pi_i}, \mathbf D_{\pi_j}),$$ where $d$ is a suitable divergence metric.

The broader notion for a candidate with respect to a current set $\mathcal{T}$ is the marginal coverage gain: $$\mathcal{G}(\pi \mid \mathcal{T}) = d\left(\sum_{\tau \in \mathcal T} \mathbf D_\tau, \mathbf U\right) - d\left(\sum_{\tau \in \mathcal T} \mathbf D_\tau + \mathbf D_\pi, \mathbf U\right).$$ A higher $\mathcal{G}$ indicates that $\pi$ supplies knowledge areas currently under-represented in $\mathcal{T}$. The greedy step at each stage selects the teacher with maximal marginal gain.

4. Computational and Communication Complexity

The greedy strategy entails, per round:

• For $K$ selection steps and $M$ candidates, $O(KMC)$ divergences are computed per round, where computing $d(\cdot, \cdot)$ is $O(C)$.
• For communication, each candidate's $C$-length distribution vector is sent to the server ($O(MC)$), and after selection, only the $K$ teacher models ($O(K|w|)$, $|w|$ = model size) are transmitted to the active learner.

This procedure achieves substantial reduction in both communication footprint and server-side computational load compared to naive full-ensemble distillation (Xu et al., 11 Jul 2025).

5. Performance Guarantees and Empirical Evaluation

The greedy algorithm benefits from classical approximation guarantees: when the objective is submodular (applicable for many coverage-type objectives), it attains a $(1-1/e)$-approximation to the global optimum (Nemhauser–Wolsey result).

Empirical results for federated learning:

• The mechanism ensures near-uniform class representation, combating catastrophic forgetting under heterogeneous data.
• Ablation studies demonstrate final test accuracy improvements of $0.7\%$ to $4.4\%$ over random teacher sampling in various heterogeneity regimes.
• Deploying only $K \ll M$ teachers via the greedy mechanism accelerates convergence (2×–2.5× faster) with no significant loss in final accuracy (Xu et al., 11 Jul 2025).

6. Practical Implications and Broader Context

Complementary-based teacher selection is especially pertinent for sequential federated learning and heterogeneous distributed learning, where naively aggregating or randomly selecting teacher models can induce catastrophic forgetting and dilute minority-class knowledge. By formulating teacher selection as a coverage optimization, leveraging distributional complementarity, and employing scalable greedy algorithms, this mechanism reduces redundancy, adaptively maintains diversity, and curtails resource overhead.

A plausible implication is that this approach can generalize to other multi-source distillation contexts, including reinforcement learning from heterogeneous human feedback, where the aim is to optimize diversity and informativeness of supervision under budget and redundancy constraints (Xu et al., 11 Jul 2025, Freedman et al., 2023).

7. Related Developments and Open Questions

While complementary-based teacher selection is prominently developed for federated distillation (Xu et al., 11 Jul 2025), related ideas arise in settings such as reinforcement learning from human feedback, where the selection of distinct, complementary human teachers can be automated by modeling teacher informativeness, rationality, and cost as latent variables (Hidden Utility Bandit and Active Teacher Selection frameworks) (Freedman et al., 2023).

Open questions persist regarding theoretical regret bounds, extensions to non-class frequency–based definitions of complementarity, and the design of selection strategies for hierarchically structured or dynamically changing teacher pools. Empirical evidence supports the practical advantage of complementarity-driven selection, but deeper theoretical analysis for generalization and optimal sample-complexity remains ongoing.