NVDP: Nonparametric Variational Differential Privacy

• NVDP is a framework that uses a Dirichlet Process-based nonparametric variational bottleneck to protect multi-vector transformer embeddings.
• It integrates an NVIB layer into transformer architectures, enforcing privacy by regulating per-token representations without bypass via residual connections.
• Empirical results on benchmarks like MRPC show that NVDP outperforms baseline methods, offering superior accuracy with reduced privacy leakage.

Nonparametric Variational Differential Privacy (NVDP) is a framework for privacy-preserving representation sharing in deep learning, specifically targeting the leakage of sensitive information in transformer embeddings of text. NVDP combines a nonparametric variational information bottleneck within the transformer model architecture with rigorous differential privacy guarantees, enabling strong privacy protection for multi-vector embeddings while maintaining high utility in downstream tasks (Zein et al., 5 Jan 2026).

1. Nonparametric Variational Information Bottleneck Layer

The Nonparametric Variational Information Bottleneck (NVIB) is the core building block of NVDP. NVIB addresses the challenge that transformer embeddings are multi-vector (one vector per token), and can encode sensitive input details—an adversarial concern for privacy when sharing representations.

Standard Variational Information Bottleneck (VIB) approaches are parametric and act independently on each embedding, disregarding sequence-length variability and inter-token dependency. In contrast, NVIB employs a Bayesian nonparametric approach using a Dirichlet Process (DP) prior over sets of weighted vectors. This architecture results in a flexible information bottleneck whose “capacity” adjusts in response to input sequence length.

Let $x \in \mathbb{R}^{n \times d}$ represent the sequence of token embeddings. NVIB constructs a random mixture of $m$ weighted impulse distributions $$(\pi \in \mathbb{R}^m, Z \in \mathbb{R}^{m \times d})$$, governed by a DP prior and posterior:

• Prior: $\mathrm{DP}(G_0^p, \alpha_0^p)$, with $G_0^p = \mathcal{N}(\mu^p = 0, (\sigma^p)^2 = 1)$, $\alpha_0^p = 1$
• Posterior: $\mathrm{DP}(G_0^q, \alpha_0^q)$, where

$$\alpha_0^q = \sum_{i=1}^{n+1} \alpha_i^q \qquad G_0^q = \sum_{i=1}^{n+1} \frac{\alpha_i^q}{\alpha_0^q} \mathcal{N}(\mu_i^q, (\sigma_i^q)^2)$$

• For optimization, $$\pi \sim \mathrm{Dir}(\alpha_1^q, \ldots, \alpha_{n+1}^q)$$ and $Z_i \sim \mathcal{N}(\mu_i^q, (\sigma_i^q)^2)$; the noisy representation $S = (\pi, Z)$ forms the privatised embedding.

NVIB regularisation is governed by the loss:

$L = L_T + \lambda_D L_D + \lambda_G L_G$

where $L_T$ is the task loss (e.g., cross-entropy), $L_D$ is the KL-divergence between posterior and prior Dirichlet distributions, and $L_G$ is a sum of KL-divergences for the Gaussian components. The hyperparameters $\lambda_D, \lambda_G$ balance utility and privacy.

2. Joint Objective: Utility and Privacy

The overall training objective of NVDP introduces the NVIB layer as a stochastic bottleneck within the transformer, enforcing that information flow passes exclusively through this privacy-capable interface. The transformed noisy sequence $S$ is passed through a denoising multi-head attention block (with residual skip connections disabled to prevent bypass), a feed-forward layer, and then the task classifier.

The loss minimized during training is:

$$\mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ -\log p(y \mid S \sim Q(\cdot \mid x)) \right] + \lambda_D \mathrm{KL}\left[ \mathrm{Dir}(\alpha^q(x)) \parallel \mathrm{Dir}(\alpha^p) \right] + \lambda_G \sum \mathrm{KL}\left[ \mathcal{N}(\mu_i^q(x), (\sigma_i^q(x))^2) \parallel \mathcal{N}(\mu^p, (\sigma^p)^2) \right]$$

In practice, a single regularisation weight $\lambda$ is often employed with grid search to select the optimal privacy-utility tradeoff (Zein et al., 5 Jan 2026).

3. Differential Privacy Guarantees: Rényi and Bayesian Formulations

NVDP adopts Rényi Differential Privacy (RDP) as the main analytical tool, measuring privacy leakage via Rényi divergence:

$$D_\lambda(Q \parallel Q') = \frac{1}{\lambda - 1} \log \int Q(z) \left( \frac{Q(z)}{Q'(z)} \right)^{\lambda-1} dz, \quad \lambda > 1$$

A mechanism $M$ is $(\lambda, \varepsilon)$-RDP if $D_\lambda(M(x) \parallel M(x')) \leq \varepsilon$ for all adjacent $x, x'$.

To improve semantic interpretability and match distributional priors over data, RDP bounds are mapped to Bayesian Differential Privacy (BDP) guarantees:

• $M$ is $(\varepsilon_\mu, \delta_\mu)$-BDP if for all $x$ and measurable $S$,

$$\Pr[M(x) \in S] \leq e^{\varepsilon_\mu} \Pr_{x' \sim \mathcal{X}}[M(x') \in S] + \delta_\mu$$

The worst-case pairwise $D_\lambda$ is computed over $Q(\cdot|x)$ and $Q(\cdot|x')$ to select $\varepsilon$, and Theorem 2 of Triastcyn & Faltings (2020) is used to translate RDP into BDP, with $\delta_\mu = 10^{-5}$ in experiments.

A closed-form bound is provided for two DP posteriors; for $Q = \mathrm{DP}(G_0^q, \alpha_0^q)$, $Q' = \mathrm{DP}(G_0^{q'}, \alpha_0^{q'})$:

$$D_\lambda(Q \parallel Q') \leq - \left[ \frac{1}{\lambda-1} \log \Gamma(\lambda \alpha_0^q - (\lambda-1) \alpha_0^{q'}) + \log \Gamma(\alpha_0^{q'}) - \frac{\lambda}{\lambda-1} \log \Gamma(\alpha_0^q) \right] + \sum_{i=1}^{n+1} \kappa_i [\text{Dirichlet-term}_{\alpha_i^q, \alpha_i^{q'}}] + \sum_{i=1}^{n+1} \kappa_i \left[ \frac{\lambda}{2} \left\| \frac{\mu_i^q - \mu_i^{q'}}{\sigma_i'} \right\|^2 + \frac{1}{1-\lambda} \mathbf{1}^T \log \left( \frac{\sigma_i'}{(\sigma^p)^{1-\lambda} (\sigma_i^q)^\lambda} \right) \right]$$

Here, $$\sigma_i' = \sqrt{(1-\lambda)(\sigma_i^{q'})^2 + \lambda (\sigma_i^q)^2}$$, $\kappa_i=1$ in practice, and vectors are aligned by token position.

4. Integration with Transformer Architectures

Practically, NVDP extends standard transformer encoders (e.g., BERT-base) by introducing a single NVIB layer directly on the per-token output vectors $x_i \in \mathbb{R}^d$. Each token is independently projected to $(\alpha_i^q, \mu_i^q, \sigma_i^q)$ to parameterize the DP posterior.

At both training and inference, noisy sampled sequences $S=(\pi, Z)$ are generated and passed through a denoising multi-head attention block—where residual skip connections are omitted to enforce passage through the NVIB bottleneck—followed by a traditional feed-forward network and classification head. All downstream computations are thus strictly mediated by NVIB-provided representations, ensuring all private information is constrained by the privacy bottleneck.

Noise calibration in the output distribution emerges adaptively from training. The network learns the parameters $(\alpha^q, \mu^q, \sigma^q)$ via the combination of task loss and KL-based regularisation.

5. Empirical Evaluation and Privacy–Utility Tradeoff

NVDP has been evaluated on the GLUE benchmark—including MRPC, STS-B, RTE, QQP, QNLI, and SST-2 tasks—using a BERT-base encoder (12 layers, 110M parameters, sequence length 512, batch sizes 64/8, learning rate $2 \times 10^{-7}$, Stable Adam optimizer, 0.2 warm-up).

Comparison baselines include:

• Non-private BERT-base and +REG (dropout 0.1, weight decay 0.01)
• VIB-fixed: isotropic Gaussian noise added to the pooled embedding ($\sigma=0.55$)
• VIB-learned: learnable dimension-wise noise on pooled embedding
• VTDP: per-token parametric Gaussian VIB

For privacy accounting, RDP order $\lambda=1.1$ is fixed, and the maximal pairwise Rényi divergence determines the DP budget. Conversion to BDP $(\varepsilon_\mu, \delta_\mu=10^{-5})$ follows.

Key MRPC results:

Method	Accuracy	BDP $\varepsilon_\mu$	RDP
Non-private +REG	82.4	–	–
VIB-fixed	82.4	12.58	2.98
VIB-learned	82.2	11.00	2.14
VTDP	81.1	11.50	1.20
NVDP	83.0	10.70	0.34

NVDP consistently dominates previous approaches along the privacy-utility Pareto frontier: for any given privacy budget, NVDP yields higher accuracy. Sweeping the regularisation strength $\lambda$ demonstrates the expected tradeoff—higher $\lambda$ reduces RDP and BDP values, but excessive regularization degrades task utility.

For instance, for MRPC:

$\lambda$	Accuracy	Max-RDP
$10^{-3}$	82.5	0.89
$10^{-2}$	83.0	0.34
$10^{-1}$	68.3	0.04
$1$	66.5	0.008

Optimal tradeoff is observed near $\lambda=10^{-2}$, where privacy gains are significant and accuracy loss is minimal (Zein et al., 5 Jan 2026).

6. Relation to Other Nonparametric Differential Privacy Approaches

While NVDP targets transformer-based embeddings using Dirichlet Process mixtures, nonparametric approaches to differential privacy have also been explored in Gaussian Process models (Honkela et al., 2021). In this line, privacy is enforced by injecting calibrated Gaussian noise into sufficient statistics of sparse variational posteriors. Both cases leverage nonparametric Bayesian structures to handle variable input complexity, but the operational domain (representation bottleneck vs. regression inference) and noise injection modalities differ substantially. A plausible implication is that the NVIB framework could in principle be adapted to other sequential or structured data settings where nonparametric bottlenecks and privacy guarantees are desired.

7. Significance and Impact

NVDP integrates a nonparametric variational bottleneck into transformers to provide rigorous, task-adaptive privacy guarantees for high-dimensional, sequence-based representations. By leveraging RDP and BDP analytics and learning noise calibration jointly with the task objective, it achieves significantly stronger privacy-utility tradeoffs than baseline methods, especially on multi-vector embeddings. The framework establishes new empirical and methodological benchmarks for privacy-preserving NLP representation sharing and provides a foundation for further extensions to nonparametric privacy mechanisms in deep learning (Zein et al., 5 Jan 2026).