Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning (2006.12097v3)

Abstract: While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning. The code is available at https://github.com/wyjeong/FedMatch.

• The paper introduces FedMatch to overcome sparse labeling in federated systems by integrating inter-client consistency loss and disjoint parameter learning.
• It employs inter-client consistency to align predictions across helper agents, significantly improving model accuracy in non-i.i.d. environments.
• Experimental results on CIFAR-10 and Fashion-MNIST demonstrate that FedMatch outperforms baseline methods while reducing communication overhead in both Labels-at-Client and Labels-at-Server scenarios.

An Overview of Federated Semi-Supervised Learning with Inter-Client Consistency and Disjoint Learning

The paper introduces a novel approach to tackle Federated Semi-Supervised Learning (FSSL) by proposing a method called Federated Matching (FedMatch). The motivation behind this work arises from real-world scenarios where data collected at the client side often lacks sufficient labels due to factors like high labeling costs or the requirement for specialized expertise. This paper posits two primary scenarios dictated by the location of labeled data: one where the labeled data is available at the clients (Labels-at-Client) and another where it is available only at the server (Labels-at-Server).

FedMatch Framework

FedMatch innovates beyond current federated and semi-supervised approaches by introducing two key components: inter-client consistency loss and parameter decomposition for disjoint learning. The inter-client consistency loss aims to enforce prediction alignment across models trained by different clients. This is accomplished by selecting helper agents based on model similarity from different clients and ensuring consistency in prediction across these agents.

Furthermore, the parameter decomposition technique involves splitting the model parameters into two sets: one dedicated to supervised learning (σ) and one dedicated to unsupervised learning (ψ). This decomposition is specifically designed to prevent interference between learning from labeled and unlabeled data and to tackle the unique challenge posed by having labeled data only at the server or only at the client.

Experimental Validation

The efficacy of FedMatch is validated through extensive experiments on realistic datasets such as CIFAR-10 and Fashion-MNIST under different non-i.i.d. conditions, affirming its robustness. In the Labels-at-Client scenario, FedMatch shows significant performance advancement over baselines that combine federated learning with semi-supervised learning techniques naively. In the more complex Labels-at-Server scenario, the proposed method significantly mitigates the problem of inter-task interference, where traditional models may forget the knowledge acquired from labeled data due to simultaneous learning from both labeled and unlabeled data.

Numerical Results and Implications

Results indicate that FedMatch achieves superior accuracy across tasks while optimizing communication costs with its parameter decomposition strategy. Notably, for both Labels-at-Client and Labels-at-Server scenarios, the proposed method displays robustness to class imbalances across different clients, a non-trivial issue in federated learning settings.

Theoretical and Practical Implications

Theoretically, the paper provides an effective framework for addressing challenges in federated learning environments where labeled data is sparse or unevenly distributed between clients and the server. Practically, this work enhances the potential for deploying federated learning models in domains such as healthcare and personal mobile devices where privacy concerns traditionally limit the availability of labeled data.

Future Prospects

The future of AI development in federated learning could involve further enhancement of adaptive learning with imbalanced data distributions, exploration of inter-client knowledge transfer mechanisms beyond parameter sharing, and strengthening the system against adversarial attacks. FedMatch lays a strong foundation for these developments, contributing to the growing body of literature that aims to make federated learning feasible for semi-supervised tasks with minimal labeled data.

In conclusion, the paper effectively addresses the challenge of limited labels in federated learning scenarios by proposing FedMatch that not only reduces communication overhead but also outperforms traditional and state-of-the-art methods in retaining and leveraging knowledge across a federated network. This work, with its strong empirical results and practical relevance, is a notable advancement in the field of federated learning.