Scaling RegionFed to Larger Transformer Models

Empirically validate whether RegionFed maintains its personalization effectiveness and stable adaptation intensity when applied to larger transformer models, specifically T5-Base with 220 million parameters and T5-Large with 770 million parameters.

Background

The experiments evaluate T5-Small and T5-3B, but the future-directions discussion specifically identifies T5-Base and T5-Large as untested intermediate-scale models. The paper hypothesizes that gradient conflicts will remain informative, golden-section search will adjust the adaptation intensity, and architecture-agnostic gradient operations will avoid the parameter-count-dependent failures observed in parameter-level methods. These claims require direct empirical validation on the stated model sizes.

References

We hypothesize that RegionFed will maintain effectiveness on T5-Base (220M) and T5-Large (770M) because: (1) gradient conflicts remain informative regardless of model size; (2) the optimal $\rho$ discovered via golden section search will self-adjust to appropriate magnitudes; (3) architecture-agnostic operations avoid the parameter-count-dependent failures observed in SCAFFOLD/pFedMe. Empirical validation on larger models remains future work.

— RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments  (2609.05403 - Nguyen et al., 4 Sep 2026) in Appendix, Section 'Supplementary Algorithms and Extensions', subsection 'Future Directions and Extensions', subsubsection 'Scaling to Larger Transformers'