UniAud: O(1) DP Auditing Framework

• UniAud is a differential privacy auditing framework that estimates an empirical privacy lower bound in a single training run.
• It leverages synthetic, uncorrelated canaries and a self-comparison mechanism to mitigate data dependency and the auditing-utility conflict.
• UniAud++ extends the approach with a multi-task objective to efficiently preserve model utility while strengthening data-dependent privacy auditing.

UniAud is a black-box, $O(1)$ differential privacy auditing framework for models trained with differentially private optimization. Its purpose is to estimate an empirical privacy lower bound from a single training run rather than from repeated retraining, while avoiding the loss of auditing tightness that earlier one-run methods incurred through data dependency and an implicit conflict between auditing and utility. The framework is split into two regimes: UniAud for data-independent auditing, where utility is not a criterion and auditing power is maximized through synthetic uncorrelated canaries and self-comparison; and UniAud++ for data-dependent auditing, where utility matters and auditing is integrated through a separate auxiliary objective and audit head in the same training run (Liu et al., 6 Jul 2025).

1. Problem setting and formal perspective

UniAud is formulated in the setting of DP auditing for models trained with differentially private optimization, especially DP-SGD. The paper adopts the standard definition that a training algorithm $\mathcal A$ is $(\epsilon,\delta)$-DP if, for any neighboring datasets $D,D'$ differing in one example and any event $S$,

$$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$

In this setting, auditing is not a replacement for analytical privacy accounting; rather, it is an empirical verification mechanism intended to detect implementation failures such as incorrect clipping, miscalibrated noise, or flawed subsampling (Liu et al., 6 Jul 2025).

The object estimated by the audit is an empirical privacy lower bound, denoted $\epsilon_L$. This quantity is derived from a hypothesis test: an auditor tries to distinguish whether training used $D$ or $D'$ by constructing membership guesses $\hat S_i$ from scores $\mathcal A$0, typically score functions based on per-example loss. The paper writes this as

$\mathcal A$1

with a common choice

$\mathcal A$2

Low loss is therefore treated as evidence of membership. The audit then converts the resulting binary test behavior into $\mathcal A$3, which lower-bounds actual leakage.

A central distinction is between traditional $\mathcal A$4 auditing and one-run $\mathcal A$5 auditing. Traditional auditing retrains the model $\mathcal A$6 times on neighboring datasets and obtains $\mathcal A$7 effectively independent observations. One-run auditing instead includes or excludes many audit samples in a single training run, attempting to obtain $\mathcal A$8 observations at once. UniAud is explicitly designed to preserve the efficiency advantage of the second regime without inheriting its characteristic looseness.

2. Why prior one-run auditing was loose

The paper’s starting point is a “no free lunch” claim: prior $\mathcal A$9 frameworks improve efficiency, but they do not do so for free. Two mechanisms are identified.

The first is data dependency. In $(\epsilon,\delta)$0 auditing, each observation arises from a separate training run and is effectively independent. In $(\epsilon,\delta)$1 auditing, many audit samples coexist in the same run, so their training dynamics interact. Similar samples can regularize one another, weakening the sample-specific memorization signal on which membership inference depends. Dependency also appears at inference time because earlier one-run auditors frequently rely on relative ranking of scores across samples rather than on independent per-sample decisions. In that regime, increasing the number of audit samples $(\epsilon,\delta)$2 does not necessarily tighten the audit monotonically (Liu et al., 6 Jul 2025).

The second is the conflict between auditing and utility. The paper argues that good utility typically comes from learning generalizable feature-label correlations, whereas strong auditing depends on amplifying sample-specific memorization. To illustrate the tension, it introduces a synthetic dataset

$(\epsilon,\delta)$3

where the first term is label-dependent structure and the second is sample-specific noise. Increasing the relative sample-specific component improves membership inference but harms generalization. This framing is important because it motivates the bifurcation of the framework: UniAud drops utility as a criterion in the data-independent setting, whereas UniAud++ separates auditing and utility explicitly in the data-dependent setting.

A plausible implication is that one-run DP auditing cannot be made tight merely by increasing sample count or by reusing standard training objectives. The paper’s contribution is to redesign both the audit data and the audit decision rule so that one-run observations behave more like independent trials.

3. UniAud: data-independent auditing through uncorrelated canaries and self-comparison

In the data-independent regime, the goal is to evaluate the DP implementation itself rather than the privacy leakage of one particular deployed model. Because utility is not required, UniAud treats auditing as a membership encoding problem: construct data such that members are easy to fit and non-members remain hard to predict. Its encoding loss is

$(\epsilon,\delta)$4

with $(\epsilon,\delta)$5 drawn independently from $(\epsilon,\delta)$6 and $(\epsilon,\delta)$7. Under this construction, the model can memorize member pairs, driving their cross-entropy toward zero, whereas non-member pairs have no exploitable structure and satisfy

$(\epsilon,\delta)$8

The resulting member/non-member loss gap is the core signal UniAud seeks to maximize (Liu et al., 6 Jul 2025).

To instantiate this idea, UniAud uses synthetic canaries rather than real examples or mislabeled real examples. Two constructions are described. In Gaussian mode, the features are sampled from a Gaussian distribution and paired with uniformly random labels. In orthogonal mode, the features are generated from a QR-based construction so that they are approximately independent or separated when dimension is sufficiently large. In both cases, the labels are uniformly random. The paper repeatedly emphasizes that the canaries are designed to satisfy three requirements simultaneously: independence, easy-to-spot memorization, and scalable generation.

The second pillar is self-comparison. Instead of ranking a sample’s loss against all other samples, UniAud compares each canary against a counterfactual version of the same feature paired with a fresh random label. If $(\epsilon,\delta)$9 is the candidate member and $D,D'$0 is its paired counterfactual, the score is

$D,D'$1

If $D,D'$2 is the trained pair, the difference should be positive; if $D,D'$3 is the untrained counterfactual pair, it should be negative. This removes cross-sample ranking and turns membership inference into a calibrated, per-sample comparison. The theoretical result is stated as Theorem 2: if $D,D'$4 is $D,D'$5-DP and the proposed canary generator is used together with the self-comparison auditing procedure, then the inequality from the original $D,D'$6 theorem still holds for $D,D'$7 (Liu et al., 6 Jul 2025).

A concise summary of the two variants is useful.

Variant	Setting	Core mechanism
UniAud	Data-independent auditing	Synthetic uncorrelated canaries + self-comparison
UniAud++	Data-dependent auditing	Separate audit objective/head + self-comparison

The practical workflow follows directly from these ideas: generate $D,D'$8, build a comparison set $D,D'$9 with fresh labels, train once on $S$0, randomly choose whether the real or counterfactual pair is the candidate member for each sample, compute score differences, infer membership, and estimate $S$1.

4. UniAud++: utility-aware auditing in the data-dependent regime

UniAud++ addresses the harder case in which the audit concerns a specific training dataset and trained model, so utility must be preserved. The paper’s core claim is that auditing and utility should not be forced to share one implicit objective. Instead, they should be separated.

The main mechanism is a multi-task objective with a main-task predictor $S$2 and a separate audit head $S$3: $S$4 Here, $S$5 is the ordinary task label, $S$6 is an auxiliary audit tag, and $S$7 indicates that the example is selected for auditing. The design is explicit: $S$8 preserves utility, $S$9 encodes membership into a disjoint auxiliary label space, and the trade-off is controlled by $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$0 and the audit fraction $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$1 (Liu et al., 6 Jul 2025).

To realize this objective, UniAud++ augments selected training examples with a trigger $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$2 and an auxiliary tag $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$3. A fresh comparison tag $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$4 is also sampled. Selected examples are modified as $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$5, placed into the multi-task training set with their main label and audit tag, and paired at audit time with the comparison tag. The same self-comparison mechanism used in UniAud is then applied, but on the auxiliary audit loss rather than on the main-task loss.

The paper also analyzes the capacity of the auxiliary tag space. If too many audit examples share the same trigger-tag pair, independence degrades. The collision probability is approximated by

$$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$6

To alleviate this, UniAud++ proposes multi-bit membership encoding, associating each trigger with multiple tags $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$7 so that the number of distinct codes grows combinatorially without increasing output-head size proportionally. This suggests a structured way to scale audit capacity while constraining parameter growth.

5. Empirical results and efficiency–utility trade-offs

The experiments span both vision and language. For image classification, the paper evaluates on CIFAR10, CIFAR100, and GTSRB, using CNN, ViT-Small, ViT-Base, and a default 2-layer ReLU MLP for UniAud. For language modeling, it studies the GPT-2 family, PersonaChat, and a PubMed subset from 2023. Baselines include in-distribution one-run auditing, mislabeled canaries, a poisoned baseline using warmed-up initialization, and NewToken for LLMs (Liu et al., 6 Jul 2025).

The principal empirical claim is that UniAud’s black-box $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$8 framework matches the state of the art of $$\Pr[\mathcal{A}(D) \in S] \le e^\epsilon \Pr[\mathcal{A}(D') \in S] + \delta.$$9 auditing while using only one run, saving roughly $\epsilon_L$0 training runs. The paper notes that prior black-box $\epsilon_L$1 methods achieved only around $\epsilon_L$2 when the claimed privacy budget is $\epsilon_L$3. Under the configuration “Orthogonal w/ comp, 2-layer ReLU” with $\epsilon_L$4, UniAud reports:

• $\epsilon_L$5: $\epsilon_L$6
• $\epsilon_L$7: $\epsilon_L$8
• $\epsilon_L$9: $D$0

The $D$1 case is especially important in the paper’s interpretation, because it indicates that UniAud nearly reaches the statistical limit $D$2; the remaining gap is attributed primarily to finite-sample statistics rather than to weak membership inference. Ablations further show that both design elements matter: synthetic uncorrelated canaries already outperform real-data canaries, and self-comparison improves substantially over the same canaries without comparison.

The reported sensitivities are also structured. Auditing strength first improves and then degrades as $D$3 grows. Orthogonal canaries become განსაკუთრებით effective when feature dimension $D$4 is sufficiently large. The encoding-space size $D$5 should generally exceed $D$6, with the sweet spot appearing after the critical point $D$7. Random initialization outperforms pretrained initialization for synthetic canary tasks because pretrained models are less prone to memorize random patterns.

For UniAud++, the paper’s claim is not maximal auditing power in isolation, but the best utility-auditing trade-off among one-run baselines. In image classification, mislabeled canaries can increase $D$8 but often severely degrade utility; in-distribution auditing preserves utility but is weak; UniAud++ improves $D$9 while preserving much more utility than mislabeled canaries. For pretrained ViTs, the paper states that UniAud++ gets “the best of both worlds”: near-zero utility loss relative to in-distribution one-run auditing but with much stronger privacy lower bounds. In language modeling, utility degradation is described as especially small because audit tokens or embeddings occupy a naturally separate part of the token space.

6. Assumptions, limitations, and disambiguation

UniAud operates under a deliberately practical threat model. The auditor cannot modify the training implementation, assumes normal hyperparameters rather than settings chosen only to maximize leakage, has only black-box access to final model outputs or losses, and can specify the architecture and supply data before training. In this sense, the framework is designed for realistic auditing rather than for privileged white-box analysis (Liu et al., 6 Jul 2025).

Its limitations follow from the same design choices. The strongest gains of UniAud rely on synthetic canaries and synthetic-friendly architectures, which is well suited to diagnosing a DP implementation but is not intended to estimate privacy leakage for a specific real-data deployment. UniAud++ assumes that triggers and tags can be integrated before training and that the auxiliary audit loss remains observable at audit time. The self-comparison mechanism depends on the label-independence property of the synthetic canary construction. The paper also provides a theorem preserving the validity of the $D'$0 guarantee, but it does not claim optimality of the canary construction or furnish a theorem for explicit finite-sample gains.

A common source of confusion is nomenclature. UniAud in this sense is a privacy-auditing framework, not an audio generation or audio understanding system. It should therefore be distinguished from works such as “UniAudio: An Audio Foundation Model Toward Universal Audio Generation” (Yang et al., 2023), “UniAudio 1.5: LLM-driven Audio Codec is A Few-shot Audio Task Learner” (Yang et al., 2024), “UniAudio 2.0: A Unified Audio LLM with Text-Aligned Factorized Audio Tokenization” (Yang et al., 4 Feb 2026), and “Contextual AD Narration with Interleaved Multimodal Sequence” (Wang et al., 2024), whose scope is audio generation, audio-language modeling, or audio-description narration rather than differential privacy auditing.

In synthesis, UniAud reframes efficient DP auditing around two principles: engineer independence rather than assume it, and separate auditing from utility rather than conflate them. UniAud applies these principles in the data-independent setting through synthetic uncorrelated canaries and self-comparison; UniAud++ extends them to utility-aware data-dependent auditing through a separate audit channel and multi-task learning. The result is a one-run black-box auditing framework that, according to the reported experiments, attains the strongest efficiency–auditing trade-off while preserving utility far better than earlier one-run baselines (Liu et al., 6 Jul 2025).