Inverse Weierstrass Private SGD

• The paper proposes IWP-SGD, a method that explicitly inverts the functional bias from local privatization to recover the original population risk minimizer.
• It leverages inverse Weierstrass and Bernoulli transforms to correct nonlinear distortions in gradient estimates caused by Gaussian and randomized response noise.
• The algorithm utilizes a single-pass projected SGD on privatized data, offering convergence guarantees under strong convexity while managing increased variance.

Searching arXiv for the named method and closely related private SGD baselines. Inverse Weierstrass Private SGD (IWP-SGD) is a stochastic gradient descent algorithm for learning from examples released once and for all under noninteractive Local Differential Privacy (LDP). It is developed for binary classification in a task-agnostic, noninteractive, “release once and for all” setting, where each user privatizes a record locally before any centralized learner sees it and does so without knowing the downstream task. The method’s central claim is that, unlike naive SGD run directly on noisy released data, it asymptotically recovers the true population risk minimizer by explicitly inverting the functional bias induced by local privatization (Dufraiche et al., 18 Feb 2026).

1. Problem formulation and target of estimation

The setting is binary classification with bounded feature space $\mathcal X\subset \mathbb R^p$, label space $\mathcal Y=\{-1,1\}$, bounded convex parameter set $\Theta\subset \mathbb R^k$, data distribution $P$ over $\mathcal X\times\mathcal Y$, and loss $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$. The target population risk is

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$

and the desired target is the true population risk minimizer

$\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$

The privatization model is local. A randomized mechanism $Q:E\to F$ is $(\epsilon,\delta)$-LDP if for all inputs $\mathcal Y=\{-1,1\}$0 and all measurable $\mathcal Y=\{-1,1\}$1,

$\mathcal Y=\{-1,1\}$2

For continuous features, the method uses a Gaussian mechanism

$\mathcal Y=\{-1,1\}$3

with

$\mathcal Y=\{-1,1\}$4

For binary labels, it uses Randomized Response (RR): $\mathcal Y=\{-1,1\}$5 The released example is

$\mathcal Y=\{-1,1\}$6

and the total privacy budget is $\mathcal Y=\{-1,1\}$7 (Dufraiche et al., 18 Feb 2026).

The difficulty is not merely that released data are noisy. For nonlinear downstream computations, especially losses and gradients, direct evaluation on $\mathcal Y=\{-1,1\}$8 is biased: $\mathcal Y=\{-1,1\}$9 for a general nonlinear function $\Theta\subset \mathbb R^k$0. Consequently, SGD run directly on privatized examples typically follows a stochastic gradient whose expectation is not the true gradient. In the formulation of IWP-SGD, this implies convergence to the minimizer of a privacy-distorted objective rather than to $\Theta\subset \mathbb R^k$1 (Dufraiche et al., 18 Feb 2026).

2. Privatization as a functional transform

The method’s key conceptual step is to represent local privatization as an operator acting on functions of the data. For $\Theta\subset \mathbb R^k$2, define

$\Theta\subset \mathbb R^k$3

This operator captures the bias induced by privatization.

For features, additive Gaussian noise yields the generalized Weierstrass transform

$\Theta\subset \mathbb R^k$4

The paper gives the series representation

$\Theta\subset \mathbb R^k$5

where $\Theta\subset \mathbb R^k$6 is the Laplacian and $\Theta\subset \mathbb R^k$7 its $\Theta\subset \mathbb R^k$8-fold composition. In this view, feature privatization acts as Gaussian smoothing of the downstream function (Dufraiche et al., 18 Feb 2026).

For labels, RR induces the Bernoulli transform. For $\Theta\subset \mathbb R^k$9,

$P$0

Thus label privatization mixes the function value at the true label and the flipped label.

For $P$1, the total transform is the composition

$P$2

The noisy risk therefore becomes

$P$3

which differs from $P$4 because Gaussian smoothing in $P$5 and RR mixing in $P$6 distort the loss surface (Dufraiche et al., 18 Feb 2026).

The exponential-loss example makes the distortion explicit. For

$P$7

the noisy risk is

$P$8

This exhibits both a feature-noise regularization-like distortion and a label-flip distortion. A common misconception is therefore to treat direct learning on released LDP examples as a variance problem alone; in the IWP-SGD analysis it is fundamentally a bias problem as well (Dufraiche et al., 18 Feb 2026).

3. Inverse transforms and unbiased gradient recovery

IWP-SGD addresses the preceding bias by explicitly inverting the transforms induced by the privacy mechanism. For labels, the inverse Bernoulli transform is defined using

$P$9

and

$\mathcal X\times\mathcal Y$0

For features, the inverse Weierstrass transform is

$\mathcal X\times\mathcal Y$1

This alternating Laplacian series formally undoes Gaussian smoothing for regular enough functions. The composite inverse is

$\mathcal X\times\mathcal Y$2

The paper defines the Inverse Weierstrass Private loss estimator

$\mathcal X\times\mathcal Y$3

and the corresponding IWP gradient estimator

$\mathcal X\times\mathcal Y$4

The key unbiasedness statements are

$\mathcal X\times\mathcal Y$5

and

$\mathcal X\times\mathcal Y$6

This is the core mechanism by which IWP-SGD targets the original risk $\mathcal X\times\mathcal Y$7 rather than the transformed risk $\mathcal X\times\mathcal Y$8 (Dufraiche et al., 18 Feb 2026).

A plausible implication is that IWP-SGD can be understood as de-biasing at the level of the stochastic oracle rather than modifying the objective indirectly. In the formulation of the paper, each released example becomes a source of an unbiased estimate of the original gradient once the inverse operator is applied.

4. Algorithmic structure and model specializations

IWP-SGD is a single-pass projected SGD algorithm that uses only the already released privatized dataset

$\mathcal X\times\mathcal Y$9

There is no further interaction with data holders. Its inputs are the privatized dataset $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$0, an initial parameter $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$1, step size $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$2, the loss $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$3, and the projection $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$4 onto bounded convex $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$5.

At iteration $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$6, the algorithm takes privatized example $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$7, computes

$$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$8

and updates by projected SGD: $$\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R$$9 The output is $$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$0. The pseudocode is exactly this sequence: compute the IWP gradient from each released example, then run ordinary projected SGD with that unbiased gradient estimator (Dufraiche et al., 18 Feb 2026).

For generalized linear model losses of the form

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$1

the method becomes more explicit. The IWP loss estimator is

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$2

with an analogous expression for the gradient (Dufraiche et al., 18 Feb 2026).

Two closed-form cases are given. For quadratic loss,

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$3

the estimator becomes

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$4

and

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$5

For exponential loss,

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$6

the estimator is

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$7

with the corresponding explicit gradient expression given in the paper (Dufraiche et al., 18 Feb 2026).

5. Guarantees, assumptions, and limitations

The theoretical foundation has three principal components: unbiasedness, variance control, and convergence. The unbiasedness theorem is the central statement: $$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$8

The price of de-biasing is increased variance. With

$$L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],$$9

the paper bounds the variance of the IWP gradient estimator by

$\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$0

Variance worsens when feature privacy is stronger, when label privacy is stronger, and when the loss class yields a large $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$1 (Dufraiche et al., 18 Feb 2026).

The convergence theorem assumes the regularity needed for the Weierstrass inverse, $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$2-strong convexity and smoothness of $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$3, bounded convex $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$4 and $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$5, and $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$6. Under these assumptions, IWP-SGD is $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$7-LDP by post-processing, and the last iterate satisfies

$\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$8

With a suitable step size $\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).$9, this becomes

$Q:E\to F$0

so the optimization and statistical error decays on the order of $Q:E\to F$1 up to logarithmic factors (Dufraiche et al., 18 Feb 2026).

The scope is restricted. The method applies when $Q:E\to F$2 and $Q:E\to F$3 belong to a class $Q:E\to F$4 of infinitely differentiable functions with controlled Gaussian growth and controlled iterated Laplacians. The paper explicitly discusses exponential loss, quadratic loss, generalized linear model losses, and, via truncation, log loss. The main development is specific to binary classification, Gaussian noise for continuous features, and RR for binary labels (Dufraiche et al., 18 Feb 2026).

The limitations stated in the paper are: mechanism-specific construction, regularity assumptions for the inverse Weierstrass series, increased variance, and computational cost when $Q:E\to F$5 is not available in closed form. In the latter case, the appendix allows truncated approximations

$Q:E\to F$6

with exponentially decreasing bias in the regular case. A plausible implication is that practical deployment depends strongly on whether the loss admits a tractable inverse-transform representation (Dufraiche et al., 18 Feb 2026).

6. Empirical behavior and position within private SGD research

The experiments compare three procedures: SGD on real data, SGD on noisy LDP-released data, and IWP-SGD on the same noisy LDP-released data. Tasks are binary classification with linear models, mainly using exponential loss with $Q:E\to F$7 regularization. Synthetic data are generated with make_classification, with $Q:E\to F$8 and $Q:E\to F$9, features in $(\epsilon,\delta)$0, $(\epsilon,\delta)$1, and privacy levels $(\epsilon,\delta)$2-LDP for $(\epsilon,\delta)$3 and $(\epsilon,\delta)$4-LDP for $(\epsilon,\delta)$5. Real data use Folktables tasks ACSIncome and ACSPublicCoverage, with features including AGEP, SCHL, and task-specific variables such as WKHP or PINCP, constructed from the five largest states with hundreds of thousands of rows and an 80/20 train-test split (Dufraiche et al., 18 Feb 2026).

Across synthetic and real data, SGD on noisy data converges to a different solution than SGD on real data, showing the predicted bias. IWP-SGD tracks the behavior of SGD on real data much more closely. On real data, a small remaining gap in average fitted-model loss is attributed to variance rather than bias, and the appendix reports similar behavior for log loss using truncated inverse-transform approximations (Dufraiche et al., 18 Feb 2026).

Within the broader literature, IWP-SGD occupies a distinct point. “Non-Asymptotic Analysis of Online Local Private Learning with SGD” studies online locally private SGD with additive privacy noise and provides non-asymptotic convergence theory, but it does not propose or mention IWP-SGD, an inverse-Weierstrass operator, or deconvolution-style bias correction (Shi et al., 9 Jul 2025). This establishes a useful baseline contrast: in that framework, privacy enters as additive noise in the update, whereas IWP-SGD treats privacy noise as a transform and debiases the stochastic oracle.

Other nearby work addresses different obstacles. “Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification” develops Projected DP-SGD for central $(\epsilon,\delta)$6-DP ERM by projecting noisy gradients onto a publicly estimated low-dimensional gradient subspace, thereby targeting ambient-dimension dependence rather than noninteractive LDP release bias (Zhou et al., 2020). “Learning with Differentially Private (Sliced) Wasserstein Gradients” derives explicit sensitivity bounds for Wasserstein and sliced-Wasserstein gradients and shows how to privatize non-finite-sum gradient queries with clipping and Gaussian noise, but it is not about Weierstrass transforms or inverse smoothing (Rodríguez-Vítores et al., 3 Feb 2025).

A common confusion is therefore to treat IWP-SGD as an instance of standard private SGD with a different noise calibration. The formulation in (Dufraiche et al., 18 Feb 2026) is narrower and more specific: it is a noninteractive, task-agnostic LDP method for binary classification in which Gaussian feature noise induces a Weierstrass transform, RR label noise induces a Bernoulli transform, and the learning algorithm is built by explicitly inverting those transforms. Under the stated smoothness, boundedness, and strong convexity assumptions, it converges to the true population risk minimizer rather than to the minimizer of a privacy-distorted objective (Dufraiche et al., 18 Feb 2026).