AlignDP: Hybrid Differential Privacy

• AlignDP is a hybrid differential privacy mechanism that partitions user data into rare events shielded by PAC indistinguishability and non-rare events privatized using RAPPOR.
• It employs effective zero-ε LDP for rare events and standard ε-LDP for frequent events, ensuring unbiased frequency estimation under strong privacy guarantees.
• The framework balances privacy and utility through rigorous theoretical bounds, empirical metrics, and global aggregation, making it ideal for secure LLM deployments.

AlignDP is a hybrid differential privacy (DP) mechanism developed to mitigate the risks posed by extraction, distillation, and unauthorized fine-tuning of LLMs. Distinct from post-hoc watermarking or monitoring strategies, AlignDP operates at the data interface by partitioning user data into rare and non-rare components, shielding rare events via PAC indistinguishability (effectively yielding zero-ε local DP) and privatizing non-rare events using RAPPOR. This two-tier framework enforces strong privacy guarantees while retaining statistical utility for frequent categories, with composition and budget constraints enforced by a global aggregator. The theoretical underpinnings establish limits on PAC extensions, tight bounds for RAPPOR estimation error, and utility trade-offs for each privacy regime (Gaikwad, 19 Dec 2025).

1. Two-Tier Architecture of AlignDP

Let each user record be $X = (X_1,\dots,X_d)$, with marginal distributions $\mu_i$ over their respective domains $\mathcal{D}_i$. Fixing a threshold $\alpha>0$, each field $i$ is partitioned as

$$R_i = \{x\in\mathcal{D}_i:\mu_i(x)<\alpha\},\quad N_i = \mathcal{D}_i\setminus R_i.$$

• Rare events ($x\in R_i$) are processed by a PAC indistinguishability shield. The mechanism $M$ outputs the symbol $x$, but only aggregate counts are released, bounded by a PAC-style indistinguishability parameter $\delta(n, \alpha)$.
• Non-rare events ($x\in N_i$) are encoded via $k$-ary randomized response (RAPPOR). Each $x$ is mapped to a one-hot vector $v\in\{0,1\}^k$, bits flipped independently with probability $p$, yielding privatized vector $y$ sent to the aggregator.

This architecture ensures that rare events are hidden with “effective zero–$\epsilon$” LDP, while non-rare events support unbiased frequency estimation under standard LDP.

2. Formal Privacy Guarantees

PAC-Indistinguishability (Rare Events)

Define mechanism $M_{\text{rare}}$ for rare categories. $M_{\text{rare}}$ is said to satisfy PAC-indistinguishability with parameter $\delta(n, \alpha)$ if, for any $x, x' \in R_i$ and any (possibly randomized) distinguisher $\mathcal{A}$ observing $n$ outputs,

$$|\Pr[\mathcal{A}\ \text{outputs “}x\text{”} \mid x] - \Pr[\mathcal{A}\ \text{outputs “}x'\text{”} \mid x']| \leq \delta(n, \alpha).$$

A Hoeffding-type bound yields

$$\delta(n,\alpha) = \exp(-2n(\alpha-\mu_i(x))^2),\quad x\in R_i.$$

As $\delta\to 0$, this approaches $(0,\delta)$-DP, i.e., “zero–$\epsilon$” LDP for rare events.

Local Differential Privacy for Non-Rare Events (RAPPOR)

For non-rare $x\in N_i$, the $k$-ary randomized response mechanism $M_{\text{rr}}: N_i \to \{0,1\}^k$ is $\epsilon$-LDP if

$$\Pr[M_{\text{rr}}(x) = y] \leq e^{\epsilon}\,\Pr[M_{\text{rr}}(x') = y] \quad \forall x, x'\in N_i, \forall y\in\{0,1\}^k.$$

RAPPOR with bit-flip probability $p$ achieves

$\epsilon = \ln \frac{1-p}{p}.$

Each $n$-user aggregate yields, for each category $j$,

$$q = 1-p+\frac{p}{k},\quad \hat{\mu}_i(j) = \frac{y_j - \frac{1}{k}(1-q)}{q-\frac{1}{k}(1-q)}.$$

Resulting in unbiased estimates with variance $\mathrm{Var}[\hat{\mu}_i(j)] \leq \frac{p(1-p)}{n}$.

3. Fundamental Theoretical Results

Theorem 1: PAC Shielding of Rare Events

For $x \in R_i$ with $\mu_i(x)<\alpha$, $n$ i.i.d. samples yield:

$$\delta(n, \alpha) = \exp\big(-2n(\alpha-\mu_i(x))^2\big)$$

No adversary can distinguish $x$ from another rare value with advantage exceeding $\delta(n, \alpha)$. This bound follows from Hoeffding's inequality applied to empirical frequencies and thresholding at $\alpha$.

Theorem 2: $\epsilon$-LDP for RAPPOR

For non-rare categories, symmetric bit-flip RAPPOR with probability $p$ satisfies

$\epsilon = \ln \frac{1-p}{p}.$

Frequency estimators $\hat{\mu}_i(j)$ are unbiased, with variance upper bound $\leq p(1-p)/n$.

Theorem 3: Global Composition

Aggregating up to $k$ RAPPOR reports, each with privacy loss $\epsilon$, yields:

$\epsilon_{\mathrm{tot}} \leq k\epsilon$

(Basic composition.) For any $\delta > 0$,

$$\epsilon_{\mathrm{tot}} \leq \sqrt{2k\ln(1/\delta)}\,\epsilon + k\epsilon(e^\epsilon - 1)$$

(Pinsker–type advanced composition).

PAC shielding does not compose beyond the rare domain. If $\mu_i(x)\geq\alpha$, the adversary’s distinguishing probability increases with $n$, requiring DP to control leakage.

4. Analysis of Utility–Privacy Trade-offs

• Non-Rare (RAPPOR): Mean-squared error per category:

$\mathrm{MSE} \leq \frac{p(1-p)}{n}$

With privacy budget $\epsilon$, set $p=(1+e^{\epsilon})^{-1}$; thus $p(1-p) \approx e^{-\epsilon}/(1+e^{-\epsilon})^2$, yielding

$$\mathrm{MSE} \approx \frac{e^{-\epsilon}}{(1+e^{-\epsilon})^2}\frac{1}{n}$$

MSE decreases exponentially in $\epsilon$ and as $1/n$ with user count.

• Rare (PAC Shielding): Utility loss is the suppression of frequency estimation in $R_i$. Since $\sum_{x\in R_i}\mu_i(x)\le |R_i|\alpha$, the suppressed probability mass is at most $|R_i|\alpha$. For small $\alpha$ (e.g., $1\%$), overall impact is minimal.
• Hybrid Choice: Reducing $\alpha$ lowers the suppressed mass but increases the proportion of categories privatized by RAPPOR, increasing estimation error. Typically, $\alpha$ is chosen small enough for $|R_i|$ to remain modest, balancing the risk of leaking low-frequency identifiers and the noise introduced to moderately frequent events.

5. Empirical Performance and Metrics

Simulations with $n=1000$ users, $d=10$ fields (each size $k=20$), and threshold $\alpha=0.01$ yield:

Metric	Rare ($R_i$)	Non-rare ($N_i$)
Categories per field	$\approx 4$	$\approx 16$
MAE (est. freq.)	$\approx 0.001$	matches MSE bound
Top-5 accuracy ($n=10^4$)	n/a	$\approx 80\%$
KL divergence ($n=10^4$)	n/a	$\approx 0.0013$
Spearman's $\rho$ ($n=10^4$)	n/a	$\approx 0.798$

PAC shielding keeps rare event estimates at noise floor (MAE $\approx 0.001$), invariant to query repetition. Non-rare RAPPOR outputs (with $p=0.25$, $\epsilon\simeq1.1$) are consistent with theoretical MSE bounds, decaying as $1/n$. Repeated querying (up to 100) demonstrates that rare category estimation remains at noise floor, and non-rare recovery saturates at correlation coefficient $\rho\approx0.99$. No repetition permits the adversary to breach the shield or exceed the RAPPOR noise ceiling.

6. Context and Significance in LLM Privacy

AlignDP introduces a principled interface-level defense for LLMs, contrasting with reactive watermarking or monitoring approaches. By enforcing PAC indistinguishability for rare values and LDP for frequent values, it ensures robust mitigation of low-frequency signal leakage—often the locus of identification risk—while supporting meaningful aggregate analytics. The systematic integration of two privacy regimes, composition-aware aggregation, and explicit utility analysis positions AlignDP as a primary candidate for data sharing and queryable LLM deployments under privacy constraints (Gaikwad, 19 Dec 2025).