FlashbackCL: Mitigating Temporal Forgetting in Federated Learning

Abstract: Federated Learning (FL) of foundation and edge models increasingly targets deployments where client data distributions drift over time, yet existing forgetting-mitigation methods assume each client's distribution is stationary. Flashback, the strongest recent FL method against cross-client (spatial) forgetting, uses monotonically accumulating per-class label counts as a knowledge proxy; this proxy becomes miscalibrated under temporal distribution shift and anchors the global model to an outdated class balance. We formalise temporal forgetting in FL with a per-phase metric isolated from protocol-level fluctuations and propose Flashback Continual Learning (FlashbackCL), a drop-in extension of Flashback with (i) temporally-decayed label counts; (ii) a device-aware replay buffer with Class-Balanced Reservoir Sampling (CBRS); and (iii) server-side active coreset curation on the public distillation set. The results show that FlashbackCL achieves 6.9% to 10.0% relative improvement relative to Flashback, on CIFAR-10 with 50 clients and three controlled temporal shift modes, while simultaneously reducing temporal forgetting by up to 68%. A 5-variant ablation identifies CBRS replay as the critical component. FlashbackCL also improves Flashback by 3.5 points on stationary CIFAR-100, suggesting that class-balanced replay regularises spatial heterogeneity as well as temporal shift.

• The paper introduces FlashbackCL, a method that mitigates temporal forgetting in federated learning by applying temporally-decayed label counts, device-aware replay buffers, and server-side coreset curation.
• It demonstrates 6.9% to 12.7% accuracy improvements and up to 68% reduced temporal forgetting across various temporal shift modes on CIFAR-10 and CIFAR-100.
• Ablation studies underline that the CBRS replay component is critical for performance, ensuring robustness in non-stationary client data environments with minimal computational overhead.

Authoritative Summary of "FlashbackCL: Mitigating Temporal Forgetting in Federated Learning"

Problem Formulation and Motivation

Federated learning (FL) is increasingly deployed in environments where client data distributions evolve temporally, resulting in a novel form of heterogeneity—temporal drift—distinguished from the extensively studied spatial heterogeneity. Current methods for mitigating knowledge forgetting in FL, notably dynamic distillation approaches such as Flashback, depend on proxies (monotonically accumulating per-class label counts) which are only reliable under the assumption of stationary per-client distributions. In deployment scenarios (e.g., on-device foundation models and autonomous mobility systems), this stationarity is violated: user or sensor behavior, and thus the underlying data distribution, evolves over time.

Flashback cannot adapt its knowledge proxy to distribution shifts, leading to persistent anchoring to outdated class balances and impaired ability to update or forget as the client population and data regime change. This leads to what is defined as temporal forgetting, a phenomenon orthogonal to the more familiar spatial forgetting.

Proposed Method: FlashbackCL

The paper introduces FlashbackCL, which extends Flashback via three components explicitly designed to mitigate temporal forgetting in FL under non-stationary, domain-incremental regimes:

1. Temporally-Decayed Label Count

A decay factor $\lambda \in (0,1]$ is applied to the global label-count vector before new per-client contributions are added. This enables the effective forgetting of classes no longer present in recent data, preventing persistence of outdated priors and allowing the distillation weights to rapidly adapt to newly emergent or transient class distributions.

2. Device-Aware Replay Buffer with Class-Balanced Reservoir Sampling (CBRS)

Each client maintains a replay buffer whose capacity is inversely proportional to its participation frequency, as clients participating less often are at greater risk of forgetting. Admission follows CBRS, which is invariant to arrival order and prevents buffer flushing at distribution boundaries by evicting only from overrepresented classes. Standard random reservoir sampling is insufficient as it exhibits catastrophic buffer flushing for newly emerging classes.

3. Server-Side Active Coreset Curation

The server aggregates label-count influxes with a sliding window and assigns per-class sampling weights to the public distillation set based on normalized slopes of label-count trends. This mechanism biases distillation toward classes that are declining in incidence, with a uniform lower-bound to prevent starvation of any class under rapid shift.

The cumulative effect is that FlashbackCL’s distillation proxies continuously reflect recent exposure, and replay buffers maintain class balance robust to arbitrary patterns of temporal shift.

Empirical Results

CIFAR-10 (50 Clients): Robustness Across Temporal Shift Modes

On CIFAR-10 with 50 clients and three controlled shift modes (abrupt, gradual, cyclic), FlashbackCL achieved 6.9% to 10.0% relative improvement in final accuracy over Flashback, with significantly reduced temporal forgetting—up to 68% lower $\overline{\text{TF}}$ in gradual shift, and 42% lower in abrupt shift.

Figure 1: Final test accuracy on CIFAR-10 with 50 clients across three temporal shift modes. FlashbackCL consistently achieves the highest accuracy.

Cross-Dataset Validation: CIFAR-100 (100 Clients, 100 Classes)

The advantage of FlashbackCL carries over to the more challenging CIFAR-100 setup with 100 clients and 100 output classes, again across all three shift modes:

Figure 2: FlashbackCL (green) vs Flashback (blue) on CIFAR-10 (left) and CIFAR-100 (right), across three shift modes; the relative advantage is maintained at both dataset scales.

FlashbackCL improved over Flashback by 9.7%–12.7%, confirming the robustness of its mechanisms even as the class and client space expands by an order of magnitude.

Ablation Studies

A comprehensive five-variant ablation (decay, replay, coreset, and their combinations) identified CBRS replay as the primary contributor to the gains. Ablations without CBRS underperformed Flashback; those with CBRS approached the full method’s accuracy but exhibited higher seed-to-seed variance and worse temporal forgetting. The full configuration (decay + CBRS + coreset) consistently offered the best stability and minimal forgetting.

Figure 3: 5-variant ablation on CIFAR-100 abrupt shift: left, final accuracy; right, temporal forgetting $\overline{\text{TF}}$. Replay (CBRS) is critical for the performance gains.

Public Data Independence

Crucially, when all server-side public dataset distillation is disabled, FlashbackCL still yields >20-point accuracy gains over both Flashback and FedAvg (e.g., 39.0% vs ≈15.7% in abrupt mode), demonstrating that its core mechanisms—temporal decay and class-balanced replay—are responsible for the observed benefits, not reliance on a public dataset.

Sensitivity and Generalizability

A decay hyperparameter sweep confirmed performance is stable with $\lambda \in [0.95, 0.99]$, dropping only with over-aggressive decay. Additionally, preliminary results with an 11M-parameter ResNet-18 indicate that the temporal forgetting mitigation persists at larger model scales, though full multi-seed evaluation is left to future work.

Figure 4: Decay-$\lambda$ sensitivity for CIFAR-100 with abrupt shift. Performance remains robust in the optimal range and drops under over-decay.

Theoretical and Practical Implications

FlashbackCL formalizes temporal forgetting as a failure mode distinct from spatial heterogeneity, and contributes actionable metrics for its quantification in FL. The use of temporally decayed statistics and class-balanced buffer replay not only mitigates this failure mode but regularizes standard spatial heterogeneity, providing accuracy improvements even in stationary regimes.

From a practical standpoint, FlashbackCL is straightforward to deploy as it is additive over Flashback, incurs negligible additional communication, and is robust to moderate environments shifts. In realistic edge and FM scenarios (e.g., on-device continual tuning, sensor domain shift, dynamic user populations), the mechanisms introduced—especially CBRS—are anticipated to be robust, pending evaluation on real-world nonstationary data such as nuScenes or persistent user-behavior logs.

Limitations and Avenues for Future Research

• Evaluation is primarily on image classification with moderate-scale CNNs; larger architectures and other modalities warrant further study.
• Experiment protocols use phase-based (abrupt) shift patterns; future work should consider “soft drift” models with continuous temporal distributional evolution.
• Joint optimization of temporal-aware client selection with decayed-label distillation remains an open direction.
• Extensive evaluation on real-world natural drift settings and FL foundation model fine-tuning is a high-priority extension.

Conclusion

FlashbackCL advances the state of knowledge in federated learning by formally introducing and addressing temporal forgetting. Its composite mechanism—temporal decay of label statistics, device-aware class-balanced replay, and server-side coreset curation—yields strong empirical gains and theoretical robustness to temporal nonstationarity, supported by comprehensive ablations isolating CBRS as essential for stable long-term knowledge retention. The practical implication is clear: maintaining class-balance under continually shifting client data is necessary to enable robust, real-world FL deployments.