Federated Learning with Private Label Sets

• Federated learning with private label sets is a collaborative machine learning paradigm where each client trains on a distinct label subset, preserving data confidentiality.
• Adaptations like restricted FedAvg and tailored classifier combination schemes allow for effective model updates without sharing full label information.
• Empirical studies reveal that centralized tuning using MSE loss can mitigate accuracy gaps between private and public label settings while maintaining privacy.

Federated learning with private label sets is a collaborative machine learning paradigm in which individual clients—each holding data annotated with potentially disjoint or partially overlapping class labels—train a joint model without revealing the identities or full distribution of their label sets to other clients or, in the strongest sense, even to the aggregation server. This scenario arises naturally in domains where label privacy is paramount, such as personalized recommendation systems, healthcare consortia, and cross-organization collaboration against competitive or regulatory constraints. The technical challenges are severe: standard methods for federated learning, which assume universal label set knowledge and global classifier heads, are ill-posed or suboptimal under private label constraints. Recent research systematically explores protocol adaptations, representation alignment, differential privacy mechanisms, and classifier combination schemes to achieve privacy-preserving, accurate learning despite these limitations.

1. Problem Formulation and Privacy Motivations

In federated learning with private label sets, each client $k$ is associated with an individual label set $Y_k \subset Y$, where $Y$ is the union of all task labels in the federation. The knowledge of $Y_k$ is restricted: in the public label regime, all clients are aware of $Y$, while in the private label regime, only the server knows $Y$, and each client knows only its own $Y_k$ (Breitholtz et al., 26 Aug 2025). This setting is motivated by real-world constraints where exposing one’s label set may leak intellectual property, medical specialization, or reveal user identity, undermining privacy or regulatory compliance.

The key technical challenge is that classifier parameters for classes not present on a client, and the corresponding embedding or output structures, are undefined. As a result, standard federated methods that exchange or aggregate the full set of classifier parameters, or which require all clients to be aware of every possible class, cannot be naively applied.

2. Algorithmic Adaptations for Private Label Aggregation

One core methodological advance is adapting aggregation algorithms such as FedAvg to operate on "restricted" classifier supports. In the approach of (Breitholtz et al., 26 Aug 2025), each client receives and updates only the subset of classifier parameters for its local label set $Y_k$. Let $\theta_p$ denote the classifier head parameters; the update rule for class $y \in Y$ after aggregation is:

$$\theta_p^{(t+1)}(y) = \sum_{k: y \in Y_k} \frac{n_k}{n'_y} \theta_{p_k}(y)$$

where $n_k$ is the sample count for client $k$ and $n'_y = \sum_{k: y \in Y_k} n_k$. This ensures that each classifier parameter is averaged only over clients that observe the corresponding label.

This mechanism necessitates careful handling of model structures: if a client does not have label $y$, it neither receives nor updates parameters for $y$, thus preventing accidental leakage of both model outputs and label set composition.

3. Classifier Combination and Representation Alignment

To further enhance generalization and enable a coherent global classifier, adaptation of classical classifier combination principles is employed. Under the “subset-consistent labeling mechanism,” the model for client $k$ is restricted to

$$p(Y = y \mid X = x, Y \in Y_k) = \frac{\exp(-\theta(y)^\top \phi(x))}{\sum_{y' \in Y_k} \exp(-\theta(y')^\top \phi(x))}$$

where $\phi(x)$ is the shared representation. Global prediction aggregation can then leverage pairwise coupling equations, enforcing consistency among ratios of predictions on overlapping label pairs between clients (Breitholtz et al., 26 Aug 2025).

Further, centralized tuning at the server using an unlabeled dataset enables representational alignment. Two primary objectives are considered:

• Pairwise Tuning Loss:

$$\sum_k w_k \sum_{y,y'\in Y_k, y\ne y'} \Big[ h_k(y|x)h(y'|x) - h_k(y'|x)h(y|x) \Big]^2$$

• MSE Tuning Loss:

$$\sum_k w_k \sum_{y\in Y_k} \mathbb{E}_x \big[ (h_k(y|x) - h(y|x))^2 \big]$$

where $h_k$ is the client model, $h$ is the global classifier, and $w_k$ are aggregation weights.

Empirical evidence suggests that the MSE loss offers more stable improvement in the private label setting, whereas the pairwise loss may yield higher, but more variable, accuracy in public label regimes.

4. Empirical Evaluation and Performance Impact

Experiments on standard benchmarks such as CIFAR10 and Fashion-MNIST assessed the effect of label set heterogeneity and privacy on federated learning performance (Breitholtz et al., 26 Aug 2025). Main findings are:

• As the number of labels per client is reduced (e.g., from 10 to 2), global accuracy declines, even though the per-client sample count is kept constant.
• The adapted FedAvg (restricted aggregation) and its extension to FedProx remain competitive with or without private label sets, especially when combined with centralized tuning based on the MSE objective.
• Centralized tuning can partially recover global accuracy in the private label regime to match that of public label baseline methods, but may introduce increased variance, particularly for pairwise losses when effective support for some labels is small or zero in a given aggregation window.
• The privacy cost—defined as the gap in accuracy between private and public label settings—is small, especially when the label set per client is not extremely restricted.

These results highlight that with appropriate algorithmic modifications, federated learning can preserve client privacy with respect to label sets without incurring a prohibitive loss in predictive performance.

5. Broader Implications and Practical Considerations

The practical relevance of this research is substantial:

• In scenarios such as medical federations, competing enterprises, or regulatory-constrained environments, it is infeasible to disclose complete label sets to all participants.
• The restricted parameter exchange and aggregation protocol provides privacy by design, as no client learns about label sets it does not encounter.
• The requirement for centralized tuning with an unlabeled dataset at the server introduces minimal external data dependency, and this dataset can be disjoint from any client’s population, further preserving privacy.
• The adaptation cost to standard FL frameworks is low: only classifier head parameters and aggregation logic must be modified, and the protocols are compatible with existing weighted averaging approaches.

6. Theoretical Justification and Mathematical Underpinnings

The theoretical rationale rests on the subset-consistent model assumption and properties of softmax classifiers. Given per-client observations and restricted label sets, one can precisely model the likelihoods and ensure that the federated classifier, aggregated as in (restricted) FedAvg, is a consistent estimator provided each class receives sufficient coverage across the population.

Key equations as in (Breitholtz et al., 26 Aug 2025):

• Subset-Conditional Model:

$$p(Y = y | X = x, Y \in Y_k) = \frac{\exp( -\theta(y)^\top \phi(x) )}{\sum_{y' \in Y_k} \exp( -\theta(y')^\top \phi(x) )}$$

• Pairwise Tuning Objective:

$$\min_h \sum_k w_k \sum_{y, y' \in Y_k, y \ne y'} \mathbb{E}_x \left[ (h_k(y|x) h(y'|x) - h_k(y'|x) h(y|x))^2 \right]$$

• MSE Tuning Objective:

$$\min_h \sum_k w_k \sum_{y \in Y_k} \mathbb{E}_x \left[(h_k(y|x) - h(y|x))^2 \right]$$

These loss structures provide both a recipe for practical implementation and a blueprint for further theoretical paper regarding the consistency and optimality of federated learning under private label set constraints.

7. Conclusion and Future Directions

Federated learning with heterogeneous and private label sets introduces substantial statistical and algorithmic challenges not encountered in conventional public-label FL. The adaptation of model aggregation, classifier combination, and centralized representation alignment detailed in (Breitholtz et al., 26 Aug 2025) demonstrates that highly privacy-sensitive FL is feasible, incurring only a modest utility cost relative to traditional (public label) practices. Increased variance from representation misalignment remains a challenge; hybrid tuning strategies may mitigate this in future work.

The protocols and theoretical foundations provided offer a reproducible and generalizable approach for deployment in privacy-critical collaborative environments, providing a principled compromise between privacy constraints and model utility.