---
title: Inverse Weierstrass Private SGD
url: https://www.emergentmind.com/topics/inverse-weierstrass-private-sgd-iwp-sgd
type: topic
---

# Inverse Weierstrass Private SGD

Searching arXiv for the named method and closely related private SGD baselines.
Inverse Weierstrass Private SGD (IWP-SGD) is a stochastic gradient descent algorithm for learning from examples released once and for all under noninteractive Local Differential Privacy (LDP). It is developed for binary classification in a task-agnostic, noninteractive, “release once and for all” setting, where each user privatizes a record locally before any centralized learner sees it and does so without knowing the downstream task. The method’s central claim is that, unlike naive SGD run directly on noisy released data, it asymptotically recovers the true population risk minimizer by explicitly inverting the functional bias induced by local privatization [2602.16436].

## 1. Problem formulation and target of estimation

The setting is binary classification with bounded feature space \(\mathcal X\subset \mathbb R^p\), label space \(\mathcal Y=\{-1,1\}\), bounded convex parameter set \(\Theta\subset \mathbb R^k\), data distribution \(P\) over \(\mathcal X\times\mathcal Y\), and loss \(\ell:\Theta\times\mathcal X\times\mathcal Y\to\mathbb R\). The target population risk is
\[
L(\theta)=\mathbb E_{(x,y)\sim P}\big[\ell(\theta,x,y)\big],
\]
and the desired target is the true population risk minimizer
\[
\theta^*\in\arg\min_{\theta\in\Theta} L(\theta).
\]

The privatization model is local. A randomized mechanism \(Q:E\to F\) is \((\epsilon,\delta)\)-LDP if for all inputs \(z,z'\in E\) and all measurable \(A\subseteq F\),
\[
\Pr(Q(z)\in A)\le e^\epsilon \Pr(Q(z')\in A)+\delta.
\]
For continuous features, the method uses a Gaussian mechanism
\[
Q_{\epsilon_x,\delta}(x)=x+w,\qquad w\sim \mathcal N(0,\sigma^2 I_p),
\]
with
\[
\sigma^2=\frac{8\log(1.25/\delta)\|\mathcal X\|^2}{\epsilon_x^2}.
\]
For binary labels, it uses Randomized Response (RR):
\[
Q_{\epsilon_y}(y)=
\begin{cases}
y & \text{with probability } S(\epsilon_y),\\
-y & \text{with probability } 1-S(\epsilon_y),
\end{cases}
\qquad
S(\epsilon_y)=\frac{1}{1+e^{-\epsilon_y}}.
\]
The released example is
\[
(\tilde x,\tilde y)=\big(Q_{\epsilon_x,\delta}(x),\,Q_{\epsilon_y}(y)\big),
\]
and the total privacy budget is \(\epsilon=\epsilon_x+\epsilon_y\) [2602.16436].

The difficulty is not merely that released data are noisy. For nonlinear downstream computations, especially losses and gradients, direct evaluation on \((\tilde x,\tilde y)\) is biased:
\[
\mathbb E[h(\tilde x,\tilde y)\mid x,y]\neq h(x,y)
\]
for a general nonlinear function \(h\). Consequently, SGD run directly on privatized examples typically follows a stochastic gradient whose expectation is not the true gradient. In the formulation of IWP-SGD, this implies convergence to the minimizer of a privacy-distorted objective rather than to \(\theta^*\) [2602.16436].

## 2. Privatization as a functional transform

The method’s key conceptual step is to represent local privatization as an operator acting on functions of the data. For \(h:\mathcal X\times\mathcal Y\to\mathbb R\), define
\[
T_{\epsilon,\delta}[h]:(x,y)\mapsto \mathbb E_{(\tilde x,\tilde y)}[h(\tilde x,\tilde y)].
\]
This operator captures the bias induced by privatization.

For features, additive Gaussian noise yields the generalized Weierstrass transform
\[
W_{\sigma^2}[f](x)=\mathbb E_{w\sim\mathcal N(0,\sigma^2 I_p)}[f(x+w)].
\]
The paper gives the series representation
\[
W_{\sigma^2}[f] = \sum_{k=0}^{\infty}\frac{\sigma^{2k}}{2^k k!}\,\Delta^k[f],
\]
where \(\Delta\) is the Laplacian and \(\Delta^k\) its \(k\)-fold composition. In this view, feature privatization acts as Gaussian smoothing of the downstream function [2602.16436].

For labels, RR induces the Bernoulli transform. For \(g:\{-1,1\}\to\mathbb R\),
\[
B_\epsilon[g](y) = \mathbb E_{Q_\epsilon}[g(Q_\epsilon(y))] = S(\epsilon)g(y)+(1-S(\epsilon))g(-y).
\]
Thus label privatization mixes the function value at the true label and the flipped label.

For \(h:\mathcal X\times\mathcal Y\to\mathbb R\), the total transform is the composition
\[
T_{\epsilon,\delta}[h](x,y)
=
B_{\epsilon_y}\!\left[z\mapsto W_{\sigma^2}[h(\cdot,z)](x)\right](y).
\]
The noisy risk therefore becomes
\[
\tilde L(\theta)=\mathbb E_{(x,y)\sim P}\mathbb E_{(\tilde x,\tilde y)}[\ell(\theta,\tilde x,\tilde y)],
\]
which differs from \(L(\theta)\) because Gaussian smoothing in \(x\) and RR mixing in \(y\) distort the loss surface [2602.16436].

The exponential-loss example makes the distortion explicit. For
\[
\ell(\theta,x,y)=e^{-\theta^\top x y},
\]
the noisy risk is
\[
\tilde L(\theta) = e^{\sigma^2\|\theta\|^2/2} \left( S(\epsilon_y)L(\theta)+(1-S(\epsilon_y))L(-\theta) \right).
\]
This exhibits both a feature-noise regularization-like distortion and a label-flip distortion. A common misconception is therefore to treat direct learning on released LDP examples as a variance problem alone; in the IWP-SGD analysis it is fundamentally a bias problem as well [2602.16436].

## 3. Inverse transforms and unbiased gradient recovery

IWP-SGD addresses the preceding bias by explicitly inverting the transforms induced by the privacy mechanism. For labels, the inverse Bernoulli transform is defined using
\[
\tilde S(\epsilon)=\frac{1}{1-e^{-\epsilon}},
\]
and
\[
B_\epsilon^{-1}[g](\tilde y) = \tilde S(\epsilon)g(\tilde y)+\bigl(1-\tilde S(\epsilon)\bigr)g(-\tilde y).
\]

For features, the inverse Weierstrass transform is
\[
W_{\sigma^2}^{-1}[f](\tilde x) = \sum_{k=0}^{\infty} \frac{(-1)^k\sigma^{2k}}{2^k k!}\Delta^k[f](\tilde x).
\]
This alternating Laplacian series formally undoes Gaussian smoothing for regular enough functions. The composite inverse is
\[
T_{\epsilon,\delta}^{-1}[h](\tilde x,\tilde y)
=
B_{\epsilon_y}^{-1}\left[z\mapsto W_{\sigma^2}^{-1}[h(\cdot,z)](\tilde x)\right](\tilde y).
\]

The paper defines the Inverse Weierstrass Private loss estimator
\[
\tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
T_{\epsilon,\delta}^{-1}[\ell(\theta,\cdot,\cdot)](\tilde x,\tilde y),
\]
and the corresponding IWP gradient estimator
\[
\nabla_\theta \tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
T_{\epsilon,\delta}^{-1}[\nabla_\theta \ell(\theta,\cdot,\cdot)](\tilde x,\tilde y).
\]
The key unbiasedness statements are
\[
\mathbb E_{(\tilde x,\tilde y)}\big[\tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)\big] = \ell(\theta,x,y),
\]
and
\[
\mathbb E_{(\tilde x,\tilde y)}\big[\nabla_\theta \tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)\big] = \nabla_\theta \ell(\theta,x,y).
\]
This is the core mechanism by which IWP-SGD targets the original risk \(L\) rather than the transformed risk \(\tilde L\) [2602.16436].

A plausible implication is that IWP-SGD can be understood as de-biasing at the level of the stochastic oracle rather than modifying the objective indirectly. In the formulation of the paper, each released example becomes a source of an unbiased estimate of the original gradient once the inverse operator is applied.

## 4. Algorithmic structure and model specializations

IWP-SGD is a single-pass projected SGD algorithm that uses only the already released privatized dataset
\[
\tilde D_n=\{(\tilde x_i,\tilde y_i)\}_{i=1}^n.
\]
There is no further interaction with data holders. Its inputs are the privatized dataset \(\tilde D_n\), an initial parameter \(\theta_0\in\Theta\), step size \(\gamma>0\), the loss \(\ell\), and the projection \(\Pi_\Theta\) onto bounded convex \(\Theta\).

At iteration \(t\), the algorithm takes privatized example \((\tilde x_t,\tilde y_t)\), computes
\[
g_t=\nabla_\theta\tilde \ell_{\epsilon,\delta}(\theta_{t-1},\tilde x_t,\tilde y_t)
=
T_{\epsilon,\delta}^{-1}[\nabla_\theta\ell(\theta_{t-1},\cdot,\cdot)](\tilde x_t,\tilde y_t),
\]
and updates by projected SGD:
\[
\theta_t=\Pi_\Theta\bigl(\theta_{t-1}-\gamma g_t\bigr).
\]
The output is \(\theta_n\). The pseudocode is exactly this sequence: compute the IWP gradient from each released example, then run ordinary projected SGD with that unbiased gradient estimator [2602.16436].

For generalized linear model losses of the form
\[
\ell(\theta,x,y)=f(\theta^\top x y),
\]
the method becomes more explicit. The IWP loss estimator is
\[
\tilde\ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
\tilde S(\epsilon_y) W_{\sigma^2\|\theta\|^2}^{-1}[f](\theta^\top \tilde x\tilde y)
+
(1-\tilde S(\epsilon_y)) W_{\sigma^2\|\theta\|^2}^{-1}[f](-\theta^\top \tilde x\tilde y),
\]
with an analogous expression for the gradient [2602.16436].

Two closed-form cases are given. For quadratic loss,
\[
f(z)=\frac12(z-1)^2,
\]
the estimator becomes
\[
\tilde\ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
\tilde S(\epsilon_y)f(\theta^\top \tilde x\tilde y)
+
(1-\tilde S(\epsilon_y))f(-\theta^\top \tilde x\tilde y)
-\frac{\sigma^2}{2}\|\theta\|^2,
\]
and
\[
\nabla_\theta \tilde\ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
\tilde S(\epsilon_y)\nabla_\theta f(\theta^\top \tilde x\tilde y)
+
(1-\tilde S(\epsilon_y))\nabla_\theta f(-\theta^\top \tilde x\tilde y)
-\sigma^2\theta.
\]

For exponential loss,
\[
f(z)=e^{-z},
\]
the estimator is
\[
\tilde\ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)
=
e^{-\sigma^2\|\theta\|^2/2}\tilde S(\epsilon_y)f(\theta^\top \tilde x\tilde y)
+
e^{-\sigma^2\|\theta\|^2/2}(1-\tilde S(\epsilon_y))f(-\theta^\top \tilde x\tilde y),
\]
with the corresponding explicit gradient expression given in the paper [2602.16436].

## 5. Guarantees, assumptions, and limitations

The theoretical foundation has three principal components: unbiasedness, variance control, and convergence. The unbiasedness theorem is the central statement:
\[
\mathbb E_{(\tilde x,\tilde y)}[\tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)] = \ell(\theta,x,y),
\qquad
\mathbb E_{(\tilde x,\tilde y)}[\nabla_\theta\tilde \ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)] = \nabla_\theta\ell(\theta,x,y).
\]

The price of de-biasing is increased variance. With
\[
C=\sup_{(\theta,x,y)\in\Theta\times\mathcal X\times\mathcal Y,\ s<\sigma^2}
\max\Bigl\{
\|\nabla_\theta \ell(\theta,x,y)\|,
\ \|W_s^{-1}[\nabla_\theta\nabla_x\ell(\theta,\cdot,y)](x)\|
\Bigr\},
\]
the paper bounds the variance of the IWP gradient estimator by
\[
\|\nabla_\theta \tilde\ell_{\epsilon,\delta}(\theta,\tilde x,\tilde y)-\nabla_\theta \ell(\theta,x,y)\|^2
\le
C^2\left( \sigma^2 + 4\tilde S(\epsilon_y)(\tilde S(\epsilon_y)-1)(1+\sigma^2) \right).
\]
Variance worsens when feature privacy is stronger, when label privacy is stronger, and when the loss class yields a large \(C\) [2602.16436].

The convergence theorem assumes the regularity needed for the Weierstrass inverse, \(\mu\)-strong convexity and smoothness of \(L\), bounded convex \(\mathcal X\) and \(\Theta\), and \(\sigma^2<1/(4a)\). Under these assumptions, IWP-SGD is \((\epsilon,\delta)\)-LDP by post-processing, and the last iterate satisfies
\[
\|\theta_n-\theta^*\|^2 \le (1-\gamma\mu)^n\|\theta_0-\theta^*\|^2 + \left( \frac{\gamma C^2}{\mu} \tilde S(\epsilon_y)(\tilde S(\epsilon_y)-1) \frac{\log(1.25/\delta)}{\epsilon_x^2} \right).
\]
With a suitable step size \(\gamma=\tilde O(\log n/n)\), this becomes
\[
\|\theta_n-\theta^*\|^2
\le
\tilde O\!\left( \|\theta_0-\theta^*\|^2 e^{-\mu n/2} \right)
+
\tilde O\!\left(
\frac{C^2}{\mu^2 n}\tilde S(\epsilon_y)(\tilde S(\epsilon_y)-1)\frac{\log(1.25/\delta)}{\epsilon_x^2}
\right),
\]
so the optimization and statistical error decays on the order of \(1/n\) up to logarithmic factors [2602.16436].

The scope is restricted. The method applies when \(x\mapsto \ell(\theta,x,y)\) and \(x\mapsto \partial_{\theta_j}\ell(\theta,x,y)\) belong to a class \(\Phi_{M_a,a}\) of infinitely differentiable functions with controlled Gaussian growth and controlled iterated Laplacians. The paper explicitly discusses exponential loss, quadratic loss, generalized linear model losses, and, via truncation, log loss. The main development is specific to binary classification, Gaussian noise for continuous features, and RR for binary labels [2602.16436].

The limitations stated in the paper are: mechanism-specific construction, regularity assumptions for the inverse Weierstrass series, increased variance, and computational cost when \(W_{\sigma^2}^{-1}\) is not available in closed form. In the latter case, the appendix allows truncated approximations
\[
g_t^K(x)=\sum_{k=0}^K \frac{\Delta^k f(x)}{k!}(-t)^k,
\]
with exponentially decreasing bias in the regular case. A plausible implication is that practical deployment depends strongly on whether the loss admits a tractable inverse-transform representation [2602.16436].

## 6. Empirical behavior and position within private SGD research

The experiments compare three procedures: SGD on real data, SGD on noisy LDP-released data, and IWP-SGD on the same noisy LDP-released data. Tasks are binary classification with linear models, mainly using exponential loss with \(\ell_2\) regularization. Synthetic data are generated with `make_classification`, with \(p=2\) and \(p=10\), features in \([-1,1]^p\), \(n=10^6\), and privacy levels \((2,10^{-5})\)-LDP for \(p=2\) and \((5,10^{-5})\)-LDP for \(p=10\). Real data use Folktables tasks ACSIncome and ACSPublicCoverage, with features including AGEP, SCHL, and task-specific variables such as WKHP or PINCP, constructed from the five largest states with hundreds of thousands of rows and an 80/20 train-test split [2602.16436].

Across synthetic and real data, SGD on noisy data converges to a different solution than SGD on real data, showing the predicted bias. IWP-SGD tracks the behavior of SGD on real data much more closely. On real data, a small remaining gap in average fitted-model loss is attributed to variance rather than bias, and the appendix reports similar behavior for log loss using truncated inverse-transform approximations [2602.16436].

Within the broader literature, IWP-SGD occupies a distinct point. “Non-Asymptotic Analysis of Online Local Private Learning with SGD” studies online locally private SGD with additive privacy noise and provides non-asymptotic convergence theory, but it does not propose or mention IWP-SGD, an inverse-Weierstrass operator, or deconvolution-style bias correction [2507.07041]. This establishes a useful baseline contrast: in that framework, privacy enters as additive noise in the update, whereas IWP-SGD treats privacy noise as a transform and debiases the stochastic oracle.

Other nearby work addresses different obstacles. “Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification” develops Projected DP-SGD for central \((\varepsilon,\delta)\)-DP ERM by projecting noisy gradients onto a publicly estimated low-dimensional gradient subspace, thereby targeting ambient-dimension dependence rather than noninteractive LDP release bias [2007.03813]. “Learning with Differentially Private (Sliced) Wasserstein Gradients” derives explicit sensitivity bounds for Wasserstein and sliced-Wasserstein gradients and shows how to privatize non-finite-sum gradient queries with clipping and Gaussian noise, but it is not about Weierstrass transforms or inverse smoothing [2502.01701].

A common confusion is therefore to treat IWP-SGD as an instance of standard private SGD with a different noise calibration. The formulation in [2602.16436] is narrower and more specific: it is a noninteractive, task-agnostic LDP method for binary classification in which Gaussian feature noise induces a Weierstrass transform, RR label noise induces a Bernoulli transform, and the learning algorithm is built by explicitly inverting those transforms. Under the stated smoothness, boundedness, and strong convexity assumptions, it converges to the true population risk minimizer rather than to the minimizer of a privacy-distorted objective [2602.16436].

Source: https://www.emergentmind.com/topics/inverse-weierstrass-private-sgd-iwp-sgd