Differentially Private Training

• Differentially private training is a privacy-preserving paradigm that integrates calibrated noise and gradient clipping to ensure minimal influence of any individual data point.
• Key methodologies include DP-SGD, federated learning, and specialized architectures like transformers and generative models, all balancing privacy with model accuracy.
• Advanced techniques such as adaptive noise allocation and architecture-specific methods improve convergence and reduce the inherent privacy-utility trade-off.

Differentially private training is a framework for learning models from sensitive data while providing formal, quantifiable privacy guarantees for each individual in the training set. Differential privacy (DP) ensures that the presence or absence of any single data point only minimally influences the model, typically by injecting calibrated noise at strategic points in the optimization workflow. Over the last decade, differentially private training has become the de facto standard in privacy-preserving deep learning and has been deployed across domains such as large-scale vision, language, recommendation, and federated learning.

1. Formal Foundations and Privacy Accounting

A randomized algorithm $\mathcal{M}$ is said to provide $(\epsilon, \delta)$-differential privacy if for any pair of neighboring datasets $D, D'$ (differing in one example), and for all $S \subseteq \text{Range}(\mathcal{M})$,

$$\Pr[\mathcal{M}(D) \in S] \leq e^\epsilon \Pr[\mathcal{M}(D') \in S] + \delta.$$

The canonical privacy mechanism is the Gaussian Mechanism, which adds noise $\mathcal{N}(0, \sigma^2 S^2 I)$ to a function $f$ with $\ell_2$-sensitivity $S$. In deep learning, the per-example gradient is clipped to norm $C$, setting $S \leq C$, and noise is injected according to the above distribution (Kurakin et al., 2022).

Privacy loss per iteration is amplified by subsampling batches and accumulated over $T$ steps via composition bounds. Modern procedures use Rényi Differential Privacy (RDP) or moments accountant techniques for tight accounting, converting the final RDP cost to $(\epsilon, \delta)$ with

$$\epsilon = \min_{\alpha > 1} \left\{ \sum_{t=1}^T \epsilon_t(\alpha) + \frac{\ln(1/\delta)}{\alpha-1} \right\}$$

(Kurakin et al., 2022, McMahan et al., 2018).

2. Core Methodologies in Deep Learning

DP-SGD Baseline

The fundamental algorithm underlying most differentially private training is DP-SGD (Differentially Private Stochastic Gradient Descent) [Abadi et al., 2016]:

1. Compute per-example gradients $g_i = \nabla \ell(\theta; x_i)$ for each example in the minibatch.
2. Clip each $g_i$ to norm $C$:

$\bar{g}_i = g_i / \max(1, \|g_i\|_2 / C)$

1. Aggregate and add Gaussian noise:

$$\hat{g} = \frac{1}{N}\sum_i \bar{g}_i + \mathcal{N}(0, \sigma^2 C^2 I)$$

2. Update parameters using a standard optimizer: $\theta \leftarrow \theta - \eta \hat{g}$.

This approach is directly applicable at scale, but incurs significant accuracy loss in large models or with tight privacy budgets (Kurakin et al., 2022, Mehta et al., 2022).

Extensions and Alternatives

• Direct Feedback Alignment (DP-DFA): Replaces backprop with layerwise error propagation via fixed random matrices, enabling tighter sensitivity bounds and improved utility due to earlier clipping (Lee et al., 2020).
• Sharpness-Aware Minimization (DP-SAT): Steers optimization to flat minima, mitigating distortion from clipping and noise, and achieves better privacy-utility trade-offs without extra privacy cost per iteration (Park et al., 2023).
• Semi-Sensitive Feature DP: When some features are public, noise injection is confined to the private subset, yielding improved empirical accuracy compared to full DP or label-DP-only baselines (Chua et al., 2024).
• Heterogeneous Noise Allocation (DP-Hero): Adapts per-coordinate noise scales using spectral information from previously trained models, improving convergence and utility, especially at high noise (Zheng et al., 2024).

3. DP Training in Specialized Architectures and Modalities

Neural Networks at Scale

• ResNets and Convolutional Models: Novel normalization strategies such as ScaleNorm (GN after each residual addition) alleviate instability from DP noise, enabling state-of-the-art top-1 accuracy (82.5% on CIFAR-10 at $\varepsilon=8$) (Klause et al., 2022).
• Transformers and Transfer Learning: Pre-training on massive public corpora (e.g., JFT-300M) and DP fine-tuning of only the last (classification) layer via a single noisy full-batch step yields >81% ImageNet accuracy at $\varepsilon=10$, closing much of the non-private gap while dramatically reducing computational cost (Mehta et al., 2022).

Federated and Split Learning

• Federated News Recommendation: Projecting user embeddings onto low-dimensional public bases, perturbing only the combination coefficients, and applying label permutation yields a substantially lower utility loss than naive DP on full gradients (Liu et al., 2022).
• Split Learning: Perturbing the final hidden representation gradients in their critical projection direction enables transcript-level $(\varepsilon,0)$-DP across embeddings, gradients, and model updates, eliminating label leakage risks (Yang et al., 2022).

Graph Neural Networks

• Random Walk Subgraph Sampling: Partitioning the training graph into disjoint subgraphs and treating each as an independent DP-SGD sample restores standard $2C$ sensitivity, permitting deeper GNNs and large-batch training with competitive accuracy at $\epsilon \leq 8$ (Ayle et al., 2023).

Generative Modeling

• Generative Models: Non-adversarial optimal transport objectives (Sinkhorn divergence) enable DP generative models to avoid the instability of DP-GANs and maintain robust downstream utility under $(\varepsilon=10,\,\delta=10^{-5})$ (Cao et al., 2021). Retrieval-augmented DP diffusion models leverage public trajectory knowledge bases to drastically reduce the number of noisy privatized steps and outperform previous DP-diffusion approaches in both sample quality and computational efficiency (Jiang et al., 18 Feb 2025).
• Private Language Modeling: Fine-tuning public LLMs (e.g., trained on Brown Corpus) on private data under DP-SGD produces practical $(\varepsilon=10)$ guarantees while maintaining nontrivial quality, which is not possible for DP-from-scratch runs (Kerrigan et al., 2020).
• Latent Dirichlet Allocation: Both centralized DP (with Laplace noise on sufficient statistics) and local DP (randomized response over document words) have been developed to privatize LDA training; online and streaming versions utilize Bayesian denoising for improved utility (Zhao et al., 2020).

4. Utility-Privacy Trade-offs and Predictive Multiplicity

Differentially private training mechanisms introduce a characteristic trade-off: decreasing $\varepsilon$ (increasing privacy) monotonically increases the test error and predictive multiplicity, where predictive multiplicity quantifies the chance that retraining from scratch will yield highly variable predictions for the same input (Kulynych et al., 2023). Empirical studies demonstrate that for many points, particularly at small $\varepsilon$, the model's output is essentially randomized, raising equity and justifiability issues for high-stakes applications.

Key empirical recommendations include:

• Use the highest tolerable privacy parameter for critical use cases;
• Estimate and report per-input and group-level multiplicity (disagreement rates);
• Where possible, aggregate over multiple DP-trained models to reduce arbitrariness, accepting additional privacy budget cost (Kulynych et al., 2023).

5. Privacy Analysis Techniques and Implementation Practices

Privacy loss must be tightly accounted for across iterations and across multiple vector-valued aggregates (such as layerwise gradients). Modern implementations employ modular architectures that separate training logic, privacy mechanism (clipping and noise), and post-training privacy accounting (e.g., via a PrivacyLedger and accountant) (McMahan et al., 2018).

Standard best practices across diverse domains include:

• Prefer large batch sizes to amplify privacy via subsampling;
• Replace batch normalization with per-example (e.g., group) normalization compatible with per-example gradient clipping (Kurakin et al., 2022);
• Pre-train on public (non-sensitive) data, then fine-tune privately to minimize the number of costly DP steps (Mehta et al., 2022, Kerrigan et al., 2020);
• Carefully tune clipping norm $C$ and optimizer step size $\eta$ jointly to maintain learning stability under DP noise;
• Track total privacy loss with RDP- or moments accountant-based tools, and optimize utility through hyperparameter sweeps conducted on disjoint public data (McMahan et al., 2018, Kurakin et al., 2022).

6. Frontiers, Limitations, and Open Challenges

Differentially private training has reached maturity in tabular, image, and text tasks, yet several research frontiers and practical obstacles remain:

• High-Utility Large-Scale DP: Even with all best practices, a non-negligible accuracy gap remains in large models at strict $\varepsilon$ targets (e.g., $\sim20$–$30\%$ loss vs non-private on ImageNet at $\varepsilon=10$) (Kurakin et al., 2022). Highly overparameterized models under DP-SGD lose more utility than smaller, well-regularized counterparts (Mehta et al., 2022).
• Specialized Mechanisms: Novel architectures (e.g., DP-KAN, which uses Kolmogorov-Arnold decomposition and univariate splines) can match standard MLPs in degradation profile, suggesting rich structure to exploit for DP-under-architectural design (Kalinin et al., 2024).
• Adaptive Noise and Heterogeneous Mechanisms: Guidance-driven noise scaling (DP-Hero) and block coordinate approaches with adaptive/decaying noise (under the hidden-state assumption) allow tighter privacy for the same utility (Zheng et al., 2024, Chen et al., 2024).
• Auditing and Explainability: Documenting and communicating predictive multiplicity is necessary for equitable deployment in consequential domains (Kulynych et al., 2023).
• Computation and Memory: Efficient vectorized per-example gradient computation, virtual batching, and model partitioning are critical to scaling DP training, especially when hardware is limited.

A central goal for ongoing research is closing the remaining privacy-utility gap for deep models—through architecture-aware privacy amplification, federated composition, and algorithm-specific sensitivity reductions—such that state-of-the-art non-private accuracy is achievable at tight DP budgets.

---

References

• (Knolle et al., 2021) NeuralDP Differentially private neural networks by design
• (Kulynych et al., 2023) Arbitrary Decisions are a Hidden Cost of Differentially Private Training
• (Lee et al., 2020) Differentially Private Deep Learning with Direct Feedback Alignment
• (Chua et al., 2024) Training Differentially Private Ad Prediction Models with Semi-Sensitive Features
• (Klause et al., 2022) Differentially private training of residual networks with scale normalisation
• (Cao et al., 2021) Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence
• (Kerrigan et al., 2020) Differentially Private LLMs Benefit from Public Pre-training
• (Jiang et al., 18 Feb 2025) RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models
• (Park et al., 2023) Differentially Private Sharpness-Aware Training
• (Liu et al., 2022) PrivateRec: Differentially Private Training and Serving for Federated News Recommendation
• (Ayle et al., 2023) Training Differentially Private Graph Neural Networks with Random Walk Sampling
• (Kalinin et al., 2024) DP-KAN: Differentially Private Kolmogorov-Arnold Networks
• (Zheng et al., 2024) Revisiting Privacy-Utility Trade-off for DP Training with Pre-existing Knowledge
• (Kurakin et al., 2022) Toward Training at ImageNet Scale with Differential Privacy
• (Zhao et al., 2020) Latent Dirichlet Allocation Model Training with Differential Privacy
• (McMahan et al., 2018) A General Approach to Adding Differential Privacy to Iterative Training Procedures
• (Mehta et al., 2022) Large Scale Transfer Learning for Differentially Private Image Classification
• (Yang et al., 2022) Differentially Private Label Protection in Split Learning
• (Chen et al., 2024) Hidden State Differential Private Mini-Batch Block Coordinate Descent for Multi-convexity Optimization