---
title: Deep Variational Privacy Funnel
url: https://www.emergentmind.com/topics/deep-variational-privacy-funnel
type: topic
---

# Deep Variational Privacy Funnel

Searching arXiv for recent and foundational papers on Deep Variational Privacy Funnel and closely related privacy-funnel formulations.
Deep Variational Privacy Funnel denotes a family of privacy-preserving representation-learning methods that instantiate the information-theoretic Privacy Funnel with deep stochastic encoders, variational bounds, and, in several formulations, adversarial training. Across these formulations, one observes private information $S$, observed data $X$, utility information $Y$ or $U$, and a released representation $Z$, and seeks an encoder or release mechanism that minimizes leakage such as $I(S;Z)$ while retaining utility such as $I(Y;Z)$, $I(X;Z)$, or $I(X;Y\mid S)$ under tractable learning objectives [2401.14792][2006.06332]. In end-to-end settings this yields latent Gaussian models with utility and privacy decoders; in distributed settings a shared VAE produces a compact latent code and a per-user filter perturbs that code under a divergence budget; and in more recent extensions related bottleneck constructions are applied to federated release and noisy transformer embeddings with Rényi-divergence and Bayesian Differential Privacy guarantees [1904.09415][2605.03069][2601.02307].

## 1. Information-theoretic formulation

The classical Privacy Funnel considers a stochastic release mechanism from useful data to a released variable and optimizes a privacy–utility trade-off in mutual-information terms. In the formulation summarized by Rodríguez-Gálvez et al., the classical problem is
\[
\min_{P_{Y|X}} I(S;Y)
\quad\text{s.t.}\quad
I(X;Y)\ge r',
\]
where $S$ is the private attribute, $X$ is the public data, and $Y$ is the released representation [2006.06332]. Their Conditional Privacy Funnel refines this by preserving only the non-private part of $X$:
\[
\min_{P_{Y|X}} I(S;Y)
\quad\text{s.t.}\quad
I(X;Y\mid S)\ge r.
\]

The end-to-end DVPF formulation in face-recognition and representation-learning settings is typically written as
\[
\sup I(Y;Z)\quad \text{subject to}\quad I(S;Z)\le R_s,
\]
or, equivalently,
\[
\min_{P(Z|X)} I(S;Z)-\beta I(Y;Z),
\]
with a Lagrange multiplier controlling the privacy–utility trade-off [2401.14792]. Razeghi et al. further separate a discriminative formulation,
\[
\mathcal L_{\mathsf{DisPF}} = I(X;Z)-\alpha I(S;Z),
\]
from a generative formulation,
\[
\mathcal L_{\mathsf{GenPF}} = I(X;X')-\alpha I(S;X'),
\]
where $X'=G(Z)$ is a synthetic output [2404.02696].

An alternative but equivalent operational view appears in the Gaussian Privacy Protector, where the constrained problem
\[
\text{minimize } I(Z;S)\quad \text{subject to } I(Z;U)\ge \gamma
\]
is rewritten as
\[
\mathcal L(\theta)=I(Z;S)+\beta H(U\mid Z),
\]
with $\beta>0$ trading off privacy against utility [2605.03069]. Earlier adversarial neural implementations use the same privacy-funnel logic with a distortion penalty:
\[
L_{\rm PF}(\theta)=I(S;Y)+\lambda\,\mathbb E[d(x,y)],
\]
for a randomized mechanism $p_\theta(y|x)$ [1712.07008].

These formulations share the same structural principle: privacy is defined as low information flow from sensitive attributes to the released representation, while utility is defined as retention of task-relevant or reconstruction-relevant information.

## 2. Variational bounds and tractable objectives

Direct evaluation of $I(S;Z)$, $I(Y;Z)$, $I(X;Y\mid S)$, or $H(U\mid Z)$ is generally intractable in high-dimensional models, so DVPF replaces these quantities with variational upper or lower bounds. In the supervised DVPF summarized in the face-recognition paper, the utility term admits the lower bound
\[
I(Y;Z)\ge \mathbb E_{P(X,Y)}\mathbb E_{P_\phi(Z|X)}[\log q_\theta(Y|Z)]-H(Y),
\]
which leads to maximizing
\[
L_{\rm util}(\phi,\theta)=\mathbb E_{data}\mathbb E_{Z\sim P_\phi(Z|X)}[\log q_\theta(Y|Z)].
\]
The same work gives two complementary bounds on the leakage term $I(S;Z)$: a classification-style bound using a private-attribute decoder $q_\xi(s|z)$, and a complexity–uncertainty bound based on the identity
\[
I(S;Z)=I(X;Z)-H(X|S)+H(X|S,Z),
\]
with a KL-based information-complexity term and a conditional-decoder uncertainty term [2401.14792].

The Conditional Privacy Funnel of Rodríguez-Gálvez et al. uses an encoder $p_\theta(y|x)$, a marginal proxy $q_\phi(y)$, and a conditional decoder $q_\phi(x|s,y)$ to bound
\[
I(X;Y)\le \mathbb E_{p(x)}\bigl[D_{\mathrm{KL}}(p_\theta(y|x)\|q_\phi(y))\bigr]
\]
and
\[
I(X;Y\mid S)\ge \mathbb E_{p(s,x,y)}[\ln q_\phi(x|s,y)]-\mathbb E_{p(s,x)}[\ln p(x|s)].
\]
Substituting these into the Lagrangian yields the stochastic loss
\[
\widetilde{\mathcal J}_{\text{CPF}}(\theta,\phi;\gamma)
=
\mathbb E_{p(x,s)}
\Bigl[
D_{\mathrm{KL}}(p_\theta(y|x)\|q_\phi(y))
-
\gamma\,\mathbb E_{p(e)}[\ln q_\phi(x\mid s,y=f_\theta(x,e))]
\Bigr],
\]
with $\gamma=\lambda+1$ [2006.06332].

The Gaussian Privacy Protector introduces variational families $Q_\phi(S|Z)$ and $Q_\psi(U|Z)$ and derives a lower bound on $I(Z;S)$ and an upper bound on $H(U|Z)$, producing the saddle-point objective
\[
\mathcal L_{\mathrm{GPP}}
=
\beta\,\mathbb E[\mathrm{CE}(u,Q_\psi(z))]
-
\mathbb E[\mathrm{CE}(s,Q_\phi(z))]
+
\lambda\,D_{KL}(q_\theta(z|x)\|\mathcal N(0,I)).
\]
Here the privacy term is adversarial and the utility term is predictive, but both remain grounded in mutual-information bounds [2605.03069].

Privacy-Preserving Adversarial Networks use the same variational device in an earlier form. Introducing an adversary $q_\phi(s|y)$ yields
\[
I(S;Y)\le H(S)+\mathbb E_{p(s,x)p_\theta(y|x)}[\log q_\phi(s|y)],
\]
while an optional decoder $r_\psi(x|y)$ lower-bounds $I(X;Y)$ [1712.07008]. In this sense, DVPF can be viewed as a deep variational generalization of the Privacy Funnel in which privacy leakage is approximated either through adversarial classification, decoder-based bounds, or both.

## 3. Architectural realizations

A common end-to-end DVPF architecture uses a Gaussian encoder $P_\phi(Z|X)$, utility decoder $q_\theta(Y|Z)$, private-attribute decoder $q_\xi(S|Z)$, and a prior or proposal $q_\psi(Z)$ that is either fixed isotropic $\mathcal N(0,I)$ or learned adversarially [2401.14792]. In the generative variant, a decoder or generator reconstructs or synthesizes $X'$ from $Z$, and privacy is enforced either on the latent code or on the generated output [2404.02696].

A distinct architectural pattern, introduced for distributed user customization, decouples representation learning from privatization. A VAE first learns a shared latent representation through
\[
\log p_d(x)\ge
\mathbb E_{z\sim q_e(z|x)}[\log p_d(x|z)]
-
KL[q_e(z|x)\|p(z)],
\]
and once trained the encoder $q_e(\cdot|\cdot)$ is frozen and shared by all users [1904.09415]. Each user then trains only a small generative filter $f_\theta$ in latent space. In the detailed formulation of Chen, Navidi and Rajagopal, the filter uses the deterministic posterior mean embedding $z=\mu_z(x)$, auxiliary noise $w\sim\mathcal N(0,I_p)$, and a one-hot private label $y$, and produces
\[
\tilde z = z + \psi[w;y].
\]
The filter is trained against a privacy adversary and a utility predictor under an $f$-divergence budget between $p_{\tilde z}$ and $p_z$ [2012.01467].

The same decoupled design appears in a slightly more general notation as a perturbation filter $q_f(z'|z,s,y)$ mapping $Z\to Z'$, with a typical architecture
\[
z\oplus s\oplus \epsilon \rightarrow \mathrm{FC}(H)\rightarrow \mathrm{ELU}\rightarrow \mathrm{FC}(H)\rightarrow \mathrm{ELU}\rightarrow \mathrm{FC}(\dim(Z))=\Delta z,
\]
and output $z'=z+\Delta z$ [1904.09415].

In federated settings, the Gaussian Privacy Protector replaces the shared fixed VAE with client-side stochastic encoders. Each client retains raw $x_m$ and sensitive labels $s_m$ locally, sends only sanitized $(z_m,u_m)$ to the server, and updates its own adversary $\phi_m$ without sharing sensitive labels [2605.03069].

For text, the Nonparametric Variational Information Bottleneck inserts an “NVIB layer” into a frozen BERT encoder. The posterior is a Dirichlet-process mixture of impulse vectors,
\[
q(S|x)=DP(\alpha_0^q,G_0^q),
\]
with parameters $\{\alpha_i^q,\mu_i^q,\sigma_i^q\}_{i=1}^{n+1}$ produced by a small projection network. A sampled noisy representation $S=\{(\pi_i,Z_i)\}_{i=1}^{n+1}$ is fed into a denoising multi-head attention block, and the residual skip around that block is removed to ensure that all information passes through the noisy bottleneck [2601.02307].

Taken together, these implementations show that DVPF is not tied to a single architecture: it appears as end-to-end latent Gaussian models, decoder-conditioned CPF models, per-user latent filters, federated encoders, and nonparametric transformer bottlenecks.

## 4. Optimization, estimation, and privacy accounting

Optimization is typically cast as alternating minimization or minimax training. In PPAN one solves
\[
\min_\theta \max_{\phi[, \psi]} L(\theta,\phi[, \psi]),
\]
alternating adversary updates with mechanism updates [1712.07008]. The user-customized latent filter uses a robust min–max objective
\[
\min_\theta \max_\phi \mathbb E[\ell_{\rm priv}(h_\phi(z'),s)]
-
\alpha_{\rm util}\min_\psi \mathbb E[\ell_{\rm util}(v_\psi(z'),y)]
\quad \text{subject to } D_f(q_f(z')\|q_e(z))\le b,
\]
or its Lagrangian-relaxed version with weight $\lambda$ on the divergence term [1904.09415]. GPP uses two phases: first training the adversary and utility decoder on encoded samples, then updating the encoder using the combined loss with CE terms and Gaussian-prior KL regularization [2605.03069].

The reparameterization trick is standard in Gaussian-latent variants:
\[
z=\mu_\theta(x)+\sigma_\theta(x)\odot \epsilon,\qquad \epsilon\sim\mathcal N(0,I),
\]
allowing gradients to propagate through stochastic samples [2605.03069]. The Conditional Privacy Funnel likewise uses $y=f_\theta(x,e)$ with $e\sim p(e)$ [2006.06332].

Empirical leakage is commonly assessed through mutual-information estimators. The distributed latent-filter framework uses a $K$-nearest-neighbor estimator from Gao et al. (2015),
\[
\hat I_{KNN}(X;Y)=\psi(k)+\psi(N)-\frac{1}{N}\sum_{i=1}^N[\psi(n_x(i)+1)+\psi(n_y(i)+1)],
\]
and also validates with sample-based variational bounds of the form
\[
I(X;Y)\ge \mathbb E[\log q_\phi(y|x)]-H(Y)
\]
[1904.09415]. Chen, Navidi and Rajagopal report the same estimator in a Kozachenko–Leonenko style form for $\widehat I(\tilde z;y)$ [2012.01467].

A central distinction in this literature is between mutual-information privacy objectives and formal differential privacy guarantees. In the linear-filter case, Chen, Navidi and Rajagopal show that if
\[
\tilde z=z+Aw+Vy,
\]
with $z\sim \mathcal N(\mu,\Sigma)$ and $w\sim\mathcal N(0,I)$, then under Theorem 3 the filter yields $(\epsilon,\alpha)$-Rényi differential privacy in $(z,y)$ under an explicit bound involving $\|z-z'\|_2$, $\|V\|_1$, and $\min \mathrm{eig}(AA^T)$ [2012.01467]. The transformer-based NVIB method goes further by directly controlling Rényi divergence
\[
D_\lambda(M(x)\|M(x'))\le \epsilon
\]
and converting it to a Bayesian Differential Privacy guarantee using Theorem 2 of Triastcyn and Faltings (ICML 2020) [2601.02307].

Federated privacy accounting is treated differently in GPP. Under IID data and an honest aggregator, the bound
\[
I(\{Z^m,U^m\}_{m=1}^t;S^{(m')})
\le
I(Z^{(m')};S^{(m')}) + I(U^{(m')};S^{(m')}\mid Z^{(m')})
\le \epsilon+\delta
\]
shows how client-level privacy composes with any residual leakage carried by utility labels [2605.03069].

A common misconception is that any DVPF objective automatically yields differential privacy. The cited formulations do not support that conclusion in general. Formal Rényi-DP or BDP guarantees arise only in models that explicitly analyze divergence, such as the linear-filter case or the NVIB transformer mechanism [2012.01467][2601.02307].

## 5. Empirical behavior across modalities

Empirical studies consistently report a trade-off curve rather than privacy improvement at fixed utility for all operating points. On MNIST, in the distributed latent-filter setup with digit identity privatized and “circle vs non-circle” preserved, the raw encoder output $Z$ yields private accuracy $\approx 97\%$ and utility $\approx 95\%$, while after the filter with $b\approx 1.5$ the private-label accuracy drops to $\approx 30\%$ and utility stays $\approx 95\%$; UMAP shows ten distinct clusters collapsing into two well-separated “circles vs non-circles” [1904.09415]. In a second MNIST setting that privatizes “$\ge 5?$” and preserves parity, private-label accuracy moves from $95\%\to 65\%$ as $b$ grows, while utility stays around $90\%$ [1904.09415]. Chen, Navidi and Rajagopal report a closely related MNIST experiment in which as the KL-budget $b$ grows $0\to 5$, private-ID accuracy drops from $0.95\to 0.2$ while utility stays above $0.9$, with a knee around $b\approx 1.5$ [2012.01467].

On tabular data, the UCI-Adult benchmark privatizes gender and preserves income. The reported comparison is:

| Model | Privacy metric | Utility metric |
|---|---|---|
| Plain VAE | private acc. $0.850 \pm 0.007$, private AUROC $0.843 \pm 0.004$ | utility acc. $0.837 \pm 0.009$, utility AUROC $0.755 \pm 0.005$ |
| VFAE | private acc. $0.802 \pm 0.009$, private AUROC $0.703 \pm 0.013$ | utility acc. $0.851 \pm 0.004$, utility AUROC $0.761 \pm 0.011$ |
| LMIFR | private acc. $0.728 \pm 0.014$, private AUROC $0.659 \pm 0.012$ | utility acc. $0.829 \pm 0.009$, utility AUROC $0.741 \pm 0.013$ |
| filter | private acc. $0.717 \pm 0.006$, private AUROC $0.632 \pm 0.011$ | utility acc. $0.822 \pm 0.005$, utility AUROC $0.731 \pm 0.015$ |

The reported interpretation is that the filter achieves the lowest gender leakage with only minor loss in income prediction [1904.09415]. In UCI-Abalone, with utility Rings $\ge 10$ and private Sex, a KL-budget $b=0.01$ lowers private-sex accuracy from $0.546$ to $0.321$ while utility stays $\sim 0.733$ [2012.01467].

On CelebA, multiple papers report similar patterns. In the distributed latent-filter setting, classifiers achieve $\sim 87\%$ accuracy on raw pixels and $\sim 83\%$ on raw 50-D VAE embeddings for eight private attributes; after filtering, private-attribute accuracy falls to $\sim 73\%$ on average, close to random-guess $\sim 67\%$, while smiling accuracy drops only $\sim 3\%$ from $\sim 86\%\to \sim 83\%$ [1904.09415]. Chen, Navidi and Rajagopal report that on a 100-D VAE code, private-label accuracy is $\approx 0.90$ before filtering and $\approx 0.73$ after filtering on average, while smiling remains $\approx 0.83$ [2012.01467]. The Gaussian Privacy Protector gives, for Smiling as utility and Gender as sensitive attribute with $d_z=256$ and $\beta=1$, utility $0.924$ AUC and adversary $0.547$ AUC under a 48$\times$ compression from $12\,288\to 256$ dimensions [2605.03069].

On MNIST, GPP reports utility within roughly one percentage point of an unconstrained autoencoder baseline while reducing the adversary’s AUC to near random guessing: No-Privacy AE gives Utility $0.987$ AUC and Adversary $0.943$ AUC, whereas GPP at $\beta=1.0$ gives Utility $0.978$ and Adversary $0.531$ [2605.03069]. On HAPT-Recognition, centralized GPP reaches Utility $0.891$, Adversary $0.523$, and the distributed version with five clients reaches $0.887/0.518$, or $0.883/0.512$ with heterogeneous $\beta_m$ [2605.03069].

Face-recognition experiments emphasize the cost of aggressive privacy protection. Before DVPF, raw embeddings carry nearly full sensitive information, with $I(X;\text{gender})\approx 0.62$ bits and accuracy $\approx 99.6\%$, and $I(X;\text{race})\approx 0.93$ bits with accuracy $\approx 99\%$ [2401.14792]. After DVPF on IResNet-50 embeddings, one reported setting gives at $\alpha=0.1$ a drop to $I(Z;\text{gender})=0.464$ bits, $\mathrm{Acc}(S|Z)=99.2\%$, and TMR $=93.6\%$ versus $99.4\%$ baseline; at $\alpha=10$, $I(Z;\text{gender})\to 0.388$ bits, $\mathrm{Acc}(S|Z)=84.3\%$, and TMR $=30.8\%$ [2401.14792]. A related face-recognition study reports that as $\alpha$ grows from $0.1\to 10$, attribute-classification accuracy drops from $\approx 99\%\to \approx 50\%$ and utility degrades from $\approx 99.4\%\to \approx 90\%$ in DisPF or $\approx 94\%$ in GenPF, with stronger effects at lower latent dimension $d_z$ [2404.02696].

In text classification, NVIB-based noisy transformer embeddings show a privacy–utility frontier on GLUE. For MRPC, the best reported NVDP result is $83.0\%$ accuracy with $\mathrm{RD}_{\max}=0.34$ and BDP $\epsilon_\mu=10.70$, compared with VTDP at $81.1\%/1.20/11.50$ and VIB-fixed at $82.4\%/2.98/12.58$; for SST-2, NVDP gives $91.7\%/0.19/10.90$ [2601.02307]. The reported accuracy-versus-Bayesian-DP curves show that for any target $\epsilon_\mu$, NVDP consistently achieves $\ge 1$–$2$ points higher accuracy than VTDP and $2$–$10\times$ lower Rényi divergence than single-vector VIB baselines [2601.02307].

## 6. Relations to adjacent frameworks and recurrent points of confusion

DVPF sits at the intersection of Privacy Funnel, Variational Information Bottleneck, VAE-style latent-variable modeling, adversarial learning, and, in recent extensions, differential privacy analysis. Rodríguez-Gálvez et al. state that the approach can be comfortably incorporated into common representation learning algorithms such as the VAE, the $\beta$-VAE, the VIB, or the nonlinear IB [2006.06332]. Razeghi et al. explicitly connect DVPF to VAEs, GANs, and Diffusion models, noting that the first two terms of their discriminative and generative objectives coincide with a standard VAE ELBO and that latent-space and output-space matching can be implemented adversarially [2404.02696].

One recurrent design issue concerns where to inject the sensitive attribute $S$. The Conditional Privacy Funnel emphasizes that, unlike earlier VAE-based PF variants, DVPF does not feed $S$ into the encoder $p(y|x,s)$ but only into the decoder. The stated reason is that this guarantees the encoder must remove all $S$-information from $Y$ [2006.06332]. In contrast, the user-customized latent-filter models do condition the small filter on the private label because the goal there is post hoc, user-specific perturbation of an already learned shared latent space rather than end-to-end encoder learning [1904.09415][2012.01467].

Another recurrent misconception is that local or federated storage alone solves the privacy problem. The federated GPP paper states that in privacy-sensitive deployments such as medical sensors, IoT devices, and wearables, the protection offered by keeping data local is incomplete because gradients, model updates, and the released representations themselves can leak sensitive attributes [2605.03069]. This motivates DVPF-style sanitization even when raw data never leave the device.

The literature also distinguishes discriminative protection from generative protection. DisPF obfuscates the latent code $Z$, whereas GenPF synthesizes $X'$ with privacy constraints on $I(S;X')$ [2404.02696]. This suggests that “privacy-preserving representation learning” and “privacy-preserving data generation” are not competing notions within this family but two realizations of the same information-theoretic trade-off.

A final point concerns scope. The empirical record summarized across Adult, Colored-MNIST, COMPAS, MNIST, CelebA, HAPT, face-recognition benchmarks, and GLUE shows that DVPF-like methods can support classification, reconstruction, generation, federated release, and sanitized transformer embeddings [2006.06332][2404.02696][2605.03069][2601.02307]. A plausible implication is that the principal modeling choice is less the data modality than the definition of utility, the threat model, and the form of privacy accounting—mutual information, adversarial predictability, divergence budgets, or explicit Rényi/Bayesian DP guarantees.

Source: https://www.emergentmind.com/topics/deep-variational-privacy-funnel