FedSIR: Spectral Client Identification and Relabeling for Federated Learning with Noisy Labels

Abstract: Federated learning (FL) enables collaborative model training without sharing raw data; however, the presence of noisy labels across distributed clients can severely degrade the learning performance. In this paper, we propose FedSIR, a multi-stage framework for robust FL under noisy labels. Different from existing approaches that mainly rely on designing noise-tolerant loss functions or exploiting loss dynamics during training, our method leverages the spectral structure of client feature representations to identify and mitigate label noise. Our framework consists of three key components. First, we identify clean and noisy clients by analyzing the spectral consistency of class-wise feature subspaces with minimal communication overhead. Second, clean clients provide spectral references that enable noisy clients to relabel potentially corrupted samples using both dominant class directions and residual subspaces. Third, we employ a noise-aware training strategy that integrates logit-adjusted loss, knowledge distillation, and distance-aware aggregation to further stabilize federated optimization. Extensive experiments on standard FL benchmarks demonstrate that FedSIR consistently outperforms state-of-the-art methods for FL with noisy labels. The code is available at https://github.com/sinagh72/FedSIR.

• The paper introduces FedSIR, a multi-stage framework that exploits spectral geometry to accurately identify and relabel noisy clients in federated learning.
• It utilizes SVD-based spectral analysis and a Gaussian mixture model to partition clients without requiring access to raw data or ground-truth noise transitions.
• Empirical evaluations on CIFAR-10 and CIFAR-100 benchmarks demonstrate that FedSIR significantly outperforms existing methods in accuracy and stability under extreme label noise.

Spectral Client Identification and Relabeling for Federated Learning with Noisy Labels

Background and Motivation

Federated learning (FL) suffers substantial degradation under label noise, especially when data remain distributed, label noise levels vary across clients, and raw data sharing is prohibited. Existing robust FL strategies primarily operate via loss-driven dynamics, client performance heuristics, or rely on a limited validation set, resulting in sub-optimal performance due to their sensitivity to inherent FL heterogeneity and lack of direct access to ground-truth noise transitions. The paper "FedSIR: Spectral Client Identification and Relabeling for Federated Learning with Noisy Labels" (2604.20825) introduces FedSIR, a multi-stage FL framework leveraging the spectral geometry of learned feature representations for client noise assessment and remediation.

Methodology

FedSIR consists of three tightly integrated modules: spectral client identification, spectral relabeling, and noise-aware federated optimization, introducing minimal communication overhead and requiring no trusted validation data.

Spectral Signature-Based Client Identification

FedSIR begins by extracting the spectral structure of each client's class-wise representation space. For each local dataset, class-wise feature matrices are constructed and decomposed using SVD; the leading right singular vector for each class estimates its dominant direction. Similarity matrices $\mathbf{S}_k$ (with entries $$[\mathbf{S}_k]_{c,c'} = |\mathbf{v}_{k,c}^\top \mathbf{v}_{k,c'}|$$) are used to measure subspace overlaps.

Clean clients exhibit low off-diagonal similarity (well-separated class features), while noisy clients display significant cross-class mixing.

Figure 1: Class subspace similarity matrices for clean, 40\% noise, and 80\% noise settings, demonstrating the spectral separation loss induced by noise.

FedSIR aggregates the off-diagonal mean and squared energy of $\mathbf{S}_k$ and employs a Gaussian mixture model to partition clients into noisy and clean subsets.

Figure 2: FedSIR framework: (A) Local spectral statistics computation and transmission; (B) GMM-based client partitioning; (C) Global spectral aggregation and reference distribution to noisy clients.

Spectral Relabeling

Clean client directions and residual subspaces are aggregated to form class anchors. For each noisy client sample, two spectral scores are computed relative to each class anchor:

• Dominant direction alignment
• Residual subspace projection energy

A relabel is adopted only if both criteria agree, enforcing a conservative correction regime that prioritizes precision in relabeling. This prevents overcorrection and ensures that only strongly mismatched noisy data are relabeled, leveraging stable spectral geometry across clean clients.

Noise-Aware FL Optimization

After relabeling, the FL process continues with a hybrid objective:

• Clean clients optimize standard logit-adjusted cross-entropy (addressing local class imbalance).
• Noisy clients interpolate between teacher-soft targets (knowledge distillation from the evolving clean global model) and corrected hard targets, weighted to stabilize training under residual noise and weak relabeling signals.
• Client model updates are aggregated via distance-aware weighting (DaAgg), suppressing anomalous updates from outlier clients (typically noisy or stragglers) by downweighting their influence based on their parameter proximity to the clean population.

Empirical Evaluation

Extensive experiments were performed on federated CIFAR-10 and CIFAR-100 benchmarks, under varying percentages of symmetric label noise, Dirichlet $\alpha$ for heterogeneity, and mixtures of clean and noisy clients. Strong comparisons were provided against state-of-the-art FL approaches (FedNoRo, FedELC, FedLSR, RHFL, FedNed, FedCorr, RoFL, and legacy FL methods).

FedSIR consistently achieves the highest accuracy across all noise and heterogeneity conditions. For instance, under CIFAR-10 with $\alpha=0.1$ (strongly non-IID), 5 clean clients, and up to 90% symmetric noise, FedSIR outperforms all baselines by a non-trivial margin, with improvements being even more pronounced in severely noisy regimes. On CIFAR-100, FedSIR similarly surpasses all alternatives even at extreme label noise levels ($\geq80\%$), demonstrating clear scalability to larger output spaces.

Ablation studies highlight the criticality of the spectral relabeling mechanism, with its removal causing the largest drop in high-noise performance. Logit adjustment and KD also yield measurable gains, and DaAgg confers additional robustness under heavy corruption. The conservative agreement-based relabeling rule is shown to provide better accuracy and more stable noise reduction than single-metric corrections.

Figure 3: Average relabeling noise reduction for three spectral variants. The agreement-based criterion yields the most reliable noise suppression.

Theoretical and Practical Implications

The principal theoretical advance is the introduction of a geometric lens—one that exploits emergent spectral structure within client-restricted feature spaces—enabling accurate and communication-efficient partitioning of noisy and clean clients without auxiliary supervision. This approach is robust to both label noise and FL heterogeneity, in contrast to loss-based heuristics that are confounded by system non-IIDness.

Practically, FedSIR obviates the need for central clean data, ground-truth transition matrices, or exclusion of noisy clients, enabling more inclusive and reliable FL deployments in real-world noisy environments (e.g., medical or sensor data collection where label quality cannot be guaranteed). The communication overhead is minimal and privacy properties are retained, as only local spectral statistics (and not raw data) are ever exposed.

Moreover, the modular design—combining spectral client selection, cautious sample correction, KD, and robust aggregation—provides a readily extensible architecture, with ablations confirming that each building block contributes positively to system reliability.

Limitations and Future Directions

While robust to random (symmetric) noise, extension to structured (asymmetric or instance-dependent) label corruption is not addressed. Covering highly heterogeneous and imbalanced tasks in open-vocabulary or extremely low-resource regimes remains an open question. Further work could focus on:

• Modeling structured label noise (possibly via spectral perturbation analysis)
• Integrating sample-level uncertainty scores for more granular correction
• Extending to privacy-preserving or personalized FL variants

Conclusion

FedSIR establishes a new standard for federated learning under noisy labels by leveraging spectral signatures of class geometry for client identification and relabeling. The empirical results demonstrate strong gains in accuracy, stability, and robustness over prior approaches on canonical benchmarks under challenging settings. This work highlights the importance of geometric and representation-level signals in robust federated systems, opening paths to more principled alternatives for trust assessment and correction under distributed noisy supervision.