Range Penalization: Theoretical Insights with Applications in Federated Learning

Abstract: This paper introduces range regularization for federated learning with linear systematic components to enhance statistical accuracy and induce cross-client regularity conducive to quantization, coding, and resource efficiency. Our approach identifies features with shared weights across different clients and adaptively clusters the weights of personalized features at extreme values, a process we refer to as polar clustering. Theoretical analysis of the associated estimators poses significant challenges due to the seminorm nature and non-decomposability of the regularizer. We develop new proof techniques for the nonasymptotic analysis of statistical accuracy and faithful pattern recovery. Moreover, a fast optimization algorithm that leverages varying degrees of local strong convexity is proposed to reduce iteration complexity. Experiments support the efficacy and efficiency of the proposed approach.

• The paper introduces a range-based penalty that shrinks model coefficient ranges to achieve robust statistical and communication performance.
• It leverages a nondecomposable seminorm with novel proof techniques to optimize federated learning under heterogeneous data and limited samples.
• Empirical results demonstrate superior estimation precision, resource efficiency, and privacy benefits compared to conventional regularizers.

Range Penalization for Federated Learning: Statistical Guarantees, Optimization Regimes, and Practical Implications

Introduction and Motivation

Federated learning (FL) architectures pose statistical and algorithmic challenges driven by the need to reconcile privacy-preserving distributed data storage with shared learning objectives. While conventional regularizers such as $\ell_1$-type or pairwise-difference penalties enable some degree of within-client or cross-client structure exploitation, they offer only partial solutions to the dual requirements of statistical efficiency and communication/resource efficiency. This work introduces and comprehensively analyzes a range-based penalty—a seminorm regularization that directly shrinks the dynamic range of model coefficients across clients. Beyond improved statistical fidelity, range penalization endows models with numerically compact solutions, robust to coding and quantization constraints, and potentially more privacy-preserving.

Theoretical Foundations of Range Penalization

Seminorm Structure and Nondecomposability

The range seminorm, defined for a vector $\beta \in \mathbb{R}^m$ as $$\|\beta\|_{\mathrm{range}} = \max_k \beta_k - \min_k \beta_k$$, is not a norm but a seminorm; its kernel is the one-dimensional subspace spanned by all-ones vectors (shared parameters across clients). The nondecomposability of this seminorm precludes the direct application of classical high-dimensional techniques reliant on decomposable norms, such as standard $\ell_1$-theory for Lasso-type estimators. This necessitates development of new proof tools for effective nonasymptotic analysis and pattern recovery.

The paper delivers a geometric and variational analysis of the range seminorm, characterizing its subdifferential structure and deriving closed-form expressions for its proximity operator. These results enable rigorous treatment of both optimization and statistical aspects across heterogeneous FL regimes.

Figure 1: The solution path of the range-penalized problem as the regularization parameter $\lambda$ varies, revealing distinct regimes: distinct components, polar clustering at the extremes, and total uniformity. Note the shrinkage at extremes and formation of polar clusters.

Block Range and Partial Personalization

The range penalty is naturally promoted to groups of parameters, e.g., rows of the client-wise coefficient matrix in FL, via a group-range (block seminorm) penalty. Block range penalization provides adaptive clustering of coefficients: rows corresponding to fully shared parameters are shrunk to the kernel, rows corresponding to personalized parameters are encouraged to polar cluster at two extremes, and intermediate values are left unshrunk. This construction admits automatic discovery of shared/personalized features and polar clusters—an expressivity lacking in center-oriented regularizers.

Nonasymptotic Statistical Analysis and Minimax Rates

The paper establishes sharp oracle inequalities and minimax lower bounds for estimation and pattern recovery under range penalization in FL. Distinct features of the analysis include:

• Minimax Lower Bounds: For a regime with $m$ clients and $J^*$ personalized features (rows with client-varying coefficients), the fundamental error rates have leading term $J^* m$, indicating that the cost of identifying latent cross-client clustering is intrinsically linear in both the number of personalized features and clients. No existing pairwise-difference-based regularizer achieves this rate in full generality.
• Seesaw Proof Device: Traditional dual-norm-based approaches yield suboptimal rates and excessively conservative regularity conditions for nondecomposable seminorms. The authors develop an integrated approach that orthogonally decomposes the stochastic term into model and estimator components and bounds them in a reciprocal ("seesaw") fashion. This construction enables optimal choice of regularization parameters and less stringent cone-restriction conditions.
• Error and Pattern Recovery Guarantees: In finite-sample settings, the range-penalized estimator enjoys oracle inequalities for squared estimation error and complexity penalties in terms of block-cluster structure, and achieves support/pattern recovery for shared/personalized features and extrema under incoherence and signal-gap assumptions.

Optimization and Acceleration Regimes

Standard proximal gradient methods are naturally adapted to handle the range regularizer, leveraging its closed-form proximal operator. However, communication and computation cost is a central operational concern in FL. The paper proposes an all-in-one momentum-based acceleration exploiting local strong convexity (restricted to the iterates) and adaptively tunes step sizes and momentum parameters based on empirical convexity at each step. The convergence guarantees, expressed in terms of generalized Bregman divergences, yield:

• Optimal Iteration Complexity: In the strongly convex regime, iteration complexity achieves the optimal $\mathcal{O}(\sqrt{\kappa} \log(1/\epsilon))$ dependence on the condition number $\kappa$.
• Adaptive Regime Handling: The scheme remains efficient in the absence of strong convexity, adapting to locally variable restricted strong convexity parameters estimated during optimization.

Figure 2: Log-scaled difference between the objective function value and the optimal value vs. number of iterations, showing faster convergence for the proposed acceleration compared to standard proximal gradient.

Empirical Results and Practical Implications

A comprehensive empirical evaluation confirms that range penalization systematically outperforms center-based and pairwise-difference regularizers in estimation precision, generalization performance, model parsimony, and robustness. Salient findings include:

• Statistical Efficiency: Across classification and regression tasks, RFL (range-penalized FL) achieves lower estimation/prediction error and higher pattern recovery accuracy, particularly in regimes with moderate or high heterogeneity and limited per-client sample sizes.
• Resource Efficiency & Range Reduction: Range penalization indisputably narrows the communication range of transmitted parameters—across both uplink and downlink channels—enabling effective quantization and encoding.

  Figure 3: Parameter ranges across iterations for the downlink and uplink channels under RFL for the 20 clients; both exhibit similarly reduced numerical range, affirming range-reduction for both communication directions.

• Robustness to Heterogeneous Clients: RFL maintains stable error as individual clients become increasingly outlying, whereas group $\beta \in \mathbb{R}^m$0-based methods suffer significant performance degradation with outliers.

Figure 4: Estimation and symmetric KL divergence under increasing outlier magnitude in client 1; RFL remains robust as corruption increases, while group lasso degrades sharply.

• Application to Real-World FL Tasks: On public datasets for crime prediction and air quality classification, RFL yields models that are more compact (fewer free parameters), have smaller parameter ranges, and deliver superior predictive accuracy compared to baselines (including clustering, center-based group lasso, and various personalization stratagems).

  Figure 5: Geographical locations of 12 air-quality monitoring sites in Beijing, supporting client-level personalization experiments.

Theoretical and Practical Implications

This work demonstrates that range penalization provides a unified framework for simultaneous statistical, communication, and privacy advantages in federated and distributed learning. The non-decomposable, boundary-focused regularization introduces new statistical proof devices and optimization methods, broadening the theoretical landscape for regularized empirical risk minimization in nonclassical norms or seminorms.

Contradictory to prevailing intuition, the paper reveals that the statistical cost of identifying cross-client clusters is not mitigated by parsimonious clustering assumptions in existing $\beta \in \mathbb{R}^m$1- or pairwise-difference-based formulations. The range-penalized approach attains minimax-optimality in this setting, setting a new statistical benchmark.

On the practical side, range penalization addresses key challenges in FL, including quantization/communication efficiency, privacy leakage reduction, and robustness to heterogeneity or corruption, without reliance on heuristic parameter selection or ad hoc weightings.

Conclusion

Range penalization represents a theoretically grounded and practically effective approach for federated learning, reconciling the needs for statistical accuracy, model/communication parsimony, and robust, privacy-conscious deployment. The nondecomposable nature of the range seminorm prompts new statistical insights and algorithmic innovation, and the proposed "seesaw" analysis and all-in-one acceleration scheme provide templates for further exploration in other nontraditional regularization regimes. Future directions include extension to nonlinear models, adaptation to heavy-tailed noise, and further integration with privacy-enhanced or communication-constrained FL architectures.

Reference:

"Range Penalization: Theoretical Insights with Applications in Federated Learning" (2606.10916)