Depersonalized Federated SGD

• Depersonalized Federated SGD is a class of methods that remove client-specific information from gradient updates, protecting individual privacy and mitigating data heterogeneity.
• Alternating updates and adaptive noise schemes decouple local personalization from global model aggregation, enhancing both fairness and convergence.
• Techniques such as private top-k selection, adaptive clipping, and subspace projection offer strong differential privacy guarantees while maintaining high model utility.

Depersonalized Federated Stochastic Gradient Descent (SGD) denotes a class of methods for distributed optimization in federated settings that explicitly suppress the leakage of client- or data-specific information from stochastic gradient updates. These algorithms are motivated by the privacy and fairness risks inherent in classical federated learning, where client updates can encode distinguishing statistical, behavioral, or identity-bearing signatures. The literature differentiates “depersonalization” from mere differential privacy by targeting the removal (or masking) of both explicit identifiers and implicit client-level heterogeneity within the aggregated optimization process. Approaches range from architectural decoupling of personalized and shared updates to local or adaptive perturbation schemes and compressed, subspace-projected communications.

1. Formal Principles and Motivation

Depersonalized Federated SGD fundamentally aims at preventing statistical or algorithmic attribution of shared model updates to individual clients or their specific data properties, even in adversarial aggregation or leakage scenarios. This objective arises from two fronts:

• Privacy Leakage and Re-identification: Collected gradients can encode features sensitive to re-identification—directly exposing client behaviors or indirectly facilitating instance reconstruction (Wei et al., 2023, Liu et al., 2020).
• Statistical Heterogeneity and Personalization Suppression: Heterogeneous data distributions across clients introduce variance in updates that can be exploited for de-anonymization, while simultaneously impeding global convergence by emphasizing idiosyncratic over population-wide optima (Zhou et al., 2022).

Depersonalized mechanisms intervene either by explicitly removing client-specific statistical drift before aggregation or by masking potential signature-bearing elements via local or adaptive perturbation with strong privacy guarantees.

2. Architectures and Algorithmic Approaches

2.1 Alternating SGD with Decoupled Personalization

The depersonalized framework articulated in "Depersonalized Federated Learning: Tackling Statistical Heterogeneity by Alternating Stochastic Gradient Descent" (Zhou et al., 2022) bifurcates the local client model into two submodels:

• Personalized model $v_i$: updated to track the local objective.
• Globalized model $y_i$: targets the global population optimum, regularized via a quadratic penalty coupling $v_i$ and $y_i$ to the consensus model $x$.

An alternating SGD procedure updates $y_i$ using a surrogate objective that penalizes divergence from $x$ adjusted by $v_i$, then independently updates $v_i$. After $\tau$ local steps, $v_i$ and $y_i$ are mixed with a tunable mixing rate $\lambda$. Only a depersonalized correction term ($y_{i,\tau}^k-x^k$) is returned to the server for aggregation. This structure ensures:

• Suppression of client-level variance in the global update.
• Retention of local adaptation capacity strictly in $v_i$, preventing its leakage to the shared model.

2.2 Local Differential Privacy and Sparse/Randomized Update Schemes

FedSel (Liu et al., 2020) introduces a two-stage, locally differentially private protocol:

• Private Dimension Selection: Only coordinates in the gradient vector deemed “important” under a private Top-$k$ selection (using exponential mechanism, perturbed encoding, or perturbed sampling) are eligible for transmission, thus minimizing exposure.
• Value Perturbation: The value in the selected dimension is further privatized with $\epsilon_2$-LDP before being sent, and the rest of the vector is set to zero.

This strategy exploits both sparse communication (single coordinate per client per round) and LDP, resulting in:

• Sharply reduced information leakage via dimension-wise sub-sampling and local randomization.
• Strict LDP guarantees by sequential composition of coordinate selection and value perturbation per client per round.

2.3 Adaptive Differential Privacy, Clipping, and Projections

Recent frameworks achieve depersonalization by dynamically adapting noise and clipping policies to match per-round or per-example gradient statistics:

• Fed-αCDP (Wei et al., 2023) applies instance-level clipping and adaptive noise addition (using decaying Gaussian noise calibrated to sensitivity) to each client's gradient before local SGD steps, with privacy budget accounting aligned to true gradient magnitudes.
• Dynamic DP-SGD (Du et al., 2021) modulates both clipping norms and noise variance in response to the global state, leveraging GDP accounting. This ensures both early and late optimization steps are protected without unnecessary over-noising, hence stabilizing aggregated updates and reducing distinguishability across clients.
• PCDP-SGD (Sha et al., 2023) integrates a pre-clipping projection to a top-$k$ principal subspace, discovered from a small public dataset, with subsequent DP noise addition and gradient clipping performed in the lower-dimensional subspace. This approach restricts the directions in which private gradients are exposed, reducing both communication and client-specific signal.

3. Theoretical Guarantees: Privacy, Error, and Convergence

The various depersonalized FedSGD protocols provide strong mathematical guarantees.

• Differential Privacy: All approaches quantify $(\varepsilon,\delta)$-DP or LDP at the instance or client level. For example, FedSel achieves $\epsilon$-LDP per client over $E$ rounds by composition; Fed-αCDP, Dynamic DP-SGD, and PCDP-SGD employ Moments Accountant or GDP frameworks for tight budgeting (Wei et al., 2023, Du et al., 2021, Sha et al., 2023).
• Error and Rate Bounds: Error bounds for FedSel show the variance of noisy sparse updates scales as $O(1/d)$ (unlike $O(d)$ for classical per-coordinate schemes), with batch size requirements that decrease in dimension. Alternating SGD in (Zhou et al., 2022) achieves $O(1/\sqrt{m\tau K})$ convergence under broad nonconvexity, matching or beating more communication-intensive schemes.
• Utility–Privacy Trade-offs: Dynamic schedules (adaptive clipping and noise) reduce variance and bias terms, leading to strictly improved constants in convergence rates as compared to static-DP schedules (Du et al., 2021).
• Communication and Computation: Dimension selection, sparse projections, and one-dimensional uplinks (in Top-$k$-sparse schemes) sharply reduce per-client communication to $O(\log d + 1)$ bits (Liu et al., 2020, Sha et al., 2023).

4. Empirical Evaluation and Practical Considerations

Empirical studies consistently demonstrate that depersonalized FedSGD variants achieve competitive or superior accuracy to non-private or naively privatized baselines across standard federated and privacy-sensitive benchmarks.

• FedSel achieves $1$–$8$ percentage point improvements in test accuracy over “flat” LDP competitors, and in some cases outperforms non-private Top-$k$ selection due to noise-induced robustness (Liu et al., 2020).
• Fed-αCDP (adaptive instance-level privacy) maintains close-to-non-private test accuracy (e.g., $98.3\%$ on MNIST under $\epsilon \approx 0.82$), while “completely foiling gradient-leakage reconstructions” (Wei et al., 2023).
• Dynamic DP-SGD outperforms static DP-FedSGD by $0.5$–$1.4\%$ in stringent privacy regimes ($\epsilon=0.1$ to $1.0$) and reduces profile-based linkage risks by flattening gradient-norm evolution across rounds (Du et al., 2021).
• PCDP-SGD and its federated extension deliver $2$–$10$ point accuracy gains over DP-SGD and compress uplink communication by $d/k$ in non-IID settings (Sha et al., 2023).

The adapted accumulation and dynamic schedule mechanisms are especially crucial for maintaining stability and suppressing client‐identifiable traces. Choice of hyperparameters—e.g., privacy budget allocation, projection dimension $k$, and adaptive scaling parameters—substantially influence the privacy–utility balance.

5. Extensions: Asynchrony, Decentralization, and Robustness

The AGRAF-SGD framework (Even et al., 2023) generalizes federated SGD to fully asynchronous operations while preserving classical convergence rates. In this setup, individual client updates and server aggregations are delayed or staggered arbitrarily, but the depersonalizing properties of DP and update structures hold—error bounds remain $O(1/\sqrt{nR})$ even under unbounded staleness. Such frameworks can be combined with privacy-preserving mechanisms for greater real-world resilience.

Depersonalization through adaptive DP noise, subspace projection, or architectural segregation of personalization can be extended to decentralized or asynchronous topologies, broadening applicability to networked or heterogeneous federated environments.

6. Limitations and Open Directions

Key limitations persist:

• Fine-tuning of privacy budgets, projection ranks, and mixing rates remains delicate and context-specific, often requiring cross-validation (Zhou et al., 2022, Sha et al., 2023).
• Some methods introduce additional computation per client—for instance, two gradient steps per local epoch or top-$k$ subspace estimation (Zhou et al., 2022, Sha et al., 2023).
• Theoretical understanding of optimal parameter choices (especially in nonconvex or highly skewed data settings) is incomplete.
• Extensions to asynchronous, quantized, or more statistically heterogeneous regimes are ongoing but not fully developed.

A plausible implication is that integrating adaptive privacy calibration, robust subspace compression, and architectural decoupling offers the strongest path forward in balancing communication, privacy, and convergence.

7. Summary Table of Notable Depersonalized FedSGD Approaches

Protocol	Depersonalization Principle	Privacy Guarantee
FedDeper (Zhou et al., 2022)	Alternating global/personal SGD, quadratic coupling	Suppresses client drift/statistical variance; not explicit DP
FedSel (Liu et al., 2020)	Private Top‑k selection + LDP value noise	$(\epsilon)$-LDP per client, O(1/d) noise variance
Fed-αCDP (Wei et al., 2023)	Adaptive instance-level DP with sensitivity tracking	$(\epsilon,\delta)$-DP (per-instance, per-client)
Dynamic DP-SGD (Du et al., 2021)	Time-varying clipping/noise via GDP	Central $(\epsilon,\delta)$-DP, stable profile
PCDP-SGD (Sha et al., 2023)	Projection—clipping/noise in Top‑$k$ subspace	Record- or client-level $(\epsilon,\delta)$-DP

Depersonalized Federated SGD thus encompasses a growing family of theoretically principled, empirically validated mechanisms—all converging on the dual objective of protecting client identity and ensuring high-utility, robust distributed optimization in federated machine learning.