Black-Box Privacy Auditing

• Black-box privacy auditing is a systematic evaluation method that tests observable outputs to detect privacy leakage without inspecting internal parameters.
• It encompasses diverse access models—including query-only and randomized sampling—to empirically assess differential privacy guarantees.
• Recent frameworks leverage techniques like canary-based attacks and adversarial query designs to derive empirical lower bounds on privacy loss.

Searching arXiv for papers on black-box privacy auditing and related auditing frameworks. Black-box privacy auditing is the systematic evaluation of whether a mechanism, model, or training pipeline leaks information about inputs or training data when the auditor cannot inspect internal implementation and must rely on controlled inputs and observed outputs. In recent work, the term covers query-only access to a final trained model, repeated sampling access to a randomized mechanism on neighboring datasets, and settings with Train and Infer APIs but no access to parameters, gradients, circuit structure, or other internals. Across these variants, the central objectives are to test claimed privacy guarantees, detect privacy violations, and derive empirical lower bounds on privacy loss from observable behavior (Cen et al., 2024, González et al., 8 Sep 2025, Song et al., 16 Dec 2025).

1. Access models and problem formulations

Black-box access is not a single threat model. In the most general AI-auditing discussion, it means query access to a trained model without exposing its internal implementation, and it is argued to be the minimum access level that should be granted to auditors because it balances trade secrets, data privacy, audit standardization, and audit efficiency (Cen et al., 2024). In final-model privacy auditing for DP-SGD, black-box access means observing only the outputs of the final trained model, not parameters or gradients (Yoon et al., 2024). In sequential DP auditing, it means i.i.d. samples $X_t \sim A(S)$ and $Y_t \sim A(S')$ from a randomized mechanism on a fixed neighboring pair $S \sim S'$, again without any knowledge of the mechanism’s internals (González et al., 8 Sep 2025). In quantum machine learning, black-box access is formalized through Train and Infer APIs with no access to circuits, parameters, gradients, or Kraus operators, and the auditor is modeled as a function

$$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$

that returns an empirical lower bound on privacy loss (Song et al., 16 Dec 2025).

These settings share a common structural feature: the auditor must turn behavioral evidence into a privacy claim. In the hypothesis-testing language proposed for AI auditing more broadly, one fixes a functional $g(f)$ encoding a compliance property and tests either $H_0: g(f)\le 0$ or its converse, with false-positive and true-positive rates interpreted as evidentiary error rates (Cen et al., 2024). In DP-specific work, the functional is usually a divergence or trade-off quantity between neighboring datasets or states. A practical implication is that black-box privacy auditing is best viewed not as a single attack, but as a family of statistically calibrated tests whose power depends on the access model, the chosen neighboring pair, and the observables available to the auditor.

2. Statistical and privacy-theoretic foundations

Several recent lines of work formalize black-box privacy auditing directly in privacy-theoretic terms. For approximate DP, one formulation uses Hockey-Stick divergence: $$D_\alpha(P\|Q) := \mathbb{E}_Q\!\Big[\max\{0, \tfrac{dP}{dQ} - \alpha\}\Big],$$ with

$$A \text{ is }(\varepsilon,\delta)\text{-DP} \iff D_{e^\varepsilon}(A(S)\|A(S')) \le \delta\quad\forall S\sim S'$$

(González et al., 8 Sep 2025). Another line works in $f$-DP, where a mechanism is audited through the trade-off function

$$T(P,Q)(\alpha) = \inf_{\phi} \big\{ Q[\phi=1] : P[\phi=1] \ge \alpha \big\},$$

and the null is that $Y_t \sim A(S')$0 for the target privacy profile (Shi et al., 15 Mar 2025). In QML, the neighbor relation is defined on quantum states via trace distance,

$Y_t \sim A(S')$1

and $Y_t \sim A(S')$2-QDP bounds output distinguishability for all measurable events (Song et al., 16 Dec 2025).

A distinctive contribution of the sequential auditing literature is to convert a DP claim into a testable kernel discrepancy condition. If $Y_t \sim A(S')$3 is $Y_t \sim A(S')$4-DP and $Y_t \sim A(S')$5, then

$Y_t \sim A(S')$6

so rejecting the null $Y_t \sim A(S')$7 certifies a violation of the claimed DP level on that neighboring pair (González et al., 8 Sep 2025). The resulting test supermartingale yields anytime-valid inference with Type I error control via Ville’s inequality.

Empirical lower bounds on privacy loss are typically obtained from attack error rates. In final-model DP-SGD auditing, false positive and false negative rates from a loss-threshold attack are converted into an empirical GDP parameter

$Y_t \sim A(S')$8

which is then mapped to an $Y_t \sim A(S')$9-lower bound through the Gaussian DP conversion

$S \sim S'$0

(Yoon et al., 2024). In DP-ICL, the black-box auditor similarly measures $S \sim S'$1 and $S \sim S'$2, forms $S \sim S'$3, and also computes GDP-based lower bounds from one-sided confidence bounds on false positive and false negative rates (Xia et al., 17 Nov 2025). This common structure links black-box auditing to the general principle that privacy guarantees constrain distinguishability, and any observed distinguishability can be inverted into a privacy lower bound.

3. Audit signals: canaries, adversarial inputs, and query design

The dominant empirical strategy is canary-based auditing. In one-run DP auditing, the auditor inserts $S \sim S'$4 designated examples, independently randomizes whether each canary is included, runs training once, and then scores the final model on the canaries. The number of correct inclusion guesses is then compared to a Binomial benchmark with success probability $S \sim S'$5, yielding a lower bound on $S \sim S'$6 from a single training run (Steinke et al., 2023). In later work, canary design itself becomes an optimization variable: metagradient descent is used to construct canaries that are memorizable when included and non-generalizable when excluded, improving empirical lower bounds for DP-SGD image classifiers by over $S \sim S'$7 in certain instances, while remaining transferable from non-private SGD on a small architecture to larger DP-SGD models (Boglioni et al., 21 Jul 2025).

A complementary direction replaces static canaries by optimized test inputs. In final-model-only DP-SGD auditing, worst-case adversarial samples are crafted in input space so as to maximize the separation between loss distributions under neighboring datasets. The proposed method yields tighter empirical lower bounds than traditional canary heuristics in black-box settings and is effective when only final model outputs are observable (Yoon et al., 2024). This makes explicit that black-box auditing power depends not only on the statistical test, but also on the geometry of the probes used to interrogate the model.

Other domains adapt the same basic idea to different observables. In black-box data provenance auditing, a clean-label backdoor serves as a stealthy membership signal: the user publishes slightly perturbed but label-consistent samples, and later tests whether a queried model exhibits the backdoor behavior using only hard labels. The attack is formalized as a hypothesis test with

$S \sim S'$8

where $S \sim S'$9 is the random-guess threshold for $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$0 classes (Chen et al., 2024). In DP-ICL, black-box query design is cast as a binary decision problem: prompts are constructed so that the final output is a single “Yes” or “No,” or one of two fixed strings $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$1, and membership inference is then run directly on the text outputs (Xia et al., 17 Nov 2025). In QML, the analogue is the quantum canary: a classical canary $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$2 is encoded as a quantum state $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$3, and an offset encoding $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$4 generates a neighboring state. For one-qubit rotations,

$$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$5

which provides an explicit calibration rule between the canary perturbation and the QDP neighbor relation (Song et al., 16 Dec 2025).

4. Efficiency regimes: batch, sequential, one-run, and lifted auditing

A major recent theme is reducing the sample and compute cost of black-box privacy auditing. Traditional batch auditors fix a sample size in advance, estimate a divergence or attack success rate after collecting all data, and then test the privacy claim. Sequential auditing instead maintains a nonnegative supermartingale or e-process and stops as soon as sufficient evidence accumulates. In the MMD-based sequential DP auditor, the wealth process

$$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$6

supports anytime-valid rejection with

$$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$7

while under violations the stopping time scales with the MMD excess $$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$8 (González et al., 8 Sep 2025). Empirically, this reduces sample requirements from tens or hundreds of thousands to a few hundred examples for several classical mechanisms, and detects DP-SGD violations in less than one training run in a white-box canary-gradient setting (González et al., 8 Sep 2025).

One-run auditing attacks the same cost problem differently. Rather than adaptively collecting outputs over time, it parallelizes many independent canary inclusion decisions into a single training run and then aggregates the resulting membership guesses (Steinke et al., 2023). Although the strongest current one-run method, "Let's Ask Gauss" (Agrawal et al., 10 Jun 2026), is white-box rather than black-box, its diagnosis of existing one-run methods is relevant: collapsing each per-canary score to a binary membership decision discards distributional information. This suggests that black-box one-run audits may also benefit from using continuous score distributions rather than only thresholded guesses, provided the resulting test can still be calibrated.

In QML, Lifted QDP provides a third efficiency mechanism. By randomizing over many neighboring canary pairs and rejection sets, a single trained model can be reused to evaluate many Bernoulli tests. The sample complexity improves from

$$\mathcal{F}\colon (Train, Infer, d, n, \beta) \mapsto \hat\epsilon$$9

for one canary per run to

$g(f)$0

for $g(f)$1 canaries per run (Song et al., 16 Dec 2025). This makes explicit a general principle: lifted or multiplexed auditing definitions are particularly well aligned with black-box settings because they amortize expensive training or querying over many probes.

5. Representative audited settings

Recent work applies black-box privacy auditing across substantially different mechanisms and access models.

Setting	Black-box observation	Audit signal
DP-SGD final model	Final outputs or losses	Canary loss, adversarial loss
DP-SDG	Synthetic datasets only	DCR, Querybased MIA
DP-ICL	Final text output	Binary “Yes/No” or $g(f)$2
QML	Train/Infer APIs	Loss on offset quantum canaries
Social media relevance estimation	DP group histograms	Fairness gap over score buckets
Group distribution shifts	Model predictions only	Inter-group attack-performance gap

In DP synthetic data generation, black-box MIAs such as DCR and Querybased are commonly used, but current black-box MIAs are found to be severely limited in power for tightly estimating privacy leakage in correctly implemented DP-SDGs. At the same time, black-box auditing is sufficient to detect several implementation failures, including metadata violations, a new violation in the DPWGAN implementation submitted to the NIST DP Synthetic Data Challenge, and other bugs that manifest in released synthetic data (Annamalai et al., 2024). This establishes a characteristic pattern: black-box audits are effective at finding gross violations but often loose as estimators of the true worst-case privacy loss.

In DP-ICL, a tight and efficient auditing framework supports both black-box and white-box threat models by reducing classification and generation to a binary decision problem and translating membership inference success rates into empirical privacy guarantees using Gaussian DP. On classification tasks, empirical leakage estimates closely match theoretical DP budgets; on generation tasks, they are consistently lower, which the paper attributes to conservative embedding-sensitivity bounds (Xia et al., 17 Nov 2025). In QML, black-box auditing based on Lifted QDP and quantum canaries is evaluated on simulated systems and on IBM Quantum hardware, where the method produces empirical lower bounds on effective privacy loss from Train and Infer access alone (Song et al., 16 Dec 2025).

Black-box privacy auditing also extends beyond formal DP verification. One line treats membership inference itself as an auditing tool for unauthorized data use, using clean-label backdoors that retain natural labels and require only predicted labels from the target model (Chen et al., 2024). Another treats black-box access to deployed models as a means of inferring underrepresentation of demographic groups in the training distribution: by extending shadow-model techniques from membership and property inference, external auditors can detect group distribution shifts solely by querying the model, with reported AUC-ROC between 80% and 100% in several settings (Juarez et al., 2022). A more infrastructure-oriented extension is platform-supported auditing of social media relevance estimators, where the platform exposes only DP-protected histograms of group-level relevance scores; for one fairness metric, ensuring privacy imposes only a small constant factor increase in sample complexity, with $g(f)$3 as an upper bound and $g(f)$4 for typical parameters (Imana et al., 2022).

6. Limitations, failure modes, and open directions

Black-box privacy auditing is constrained by both statistical and epistemic limits. The most general impossibility result states that, under minimal assumptions, one cannot simultaneously control Type I and Type II errors in black-box DP auditing; the conformal-inference framework in $g(f)$5-DP therefore prioritizes robust Type I error control, and only under a monotone likelihood ratio assumption does it recover effective control of both errors and finite-sample confidence bands for the trade-off function (Shi et al., 15 Mar 2025). Sequential MMD auditing exhibits a related limitation in weak privacy regimes: as $g(f)$6 becomes large, both $g(f)$7 and the DP threshold saturate near $g(f)$8, making the excess $g(f)$9 tiny and the required sample size effectively $H_0: g(f)\le 0$0 (González et al., 8 Sep 2025).

A second failure mode arises from rare events and tail behavior. "Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power" argues that sampling-based black-box auditors systematically ignore small probabilities or densities because they cannot estimate them accurately; this creates false-positive regions in which a curator can overstate privacy guarantees and still pass the audit (Wang et al., 2024). The analysis is framed as a second-order hypothesis test about the correctness of the auditor itself and implies that black-box acceptance should not be interpreted as a proof of the claimed $H_0: g(f)\le 0$1-DP level.

A third limitation is explanatory. Black-box tests provide direct behavioral evidence about the deployed system, but they cannot by themselves reveal why leakage happens, validate claimed training procedures, or establish compliance with upstream obligations such as lawful basis, DPIAs, or privacy-by-design steps (Cen et al., 2024). This is one reason the access-and-evidence literature argues for black-box access as a minimum rather than a complete solution, and for escalating to gray-box or white-box access when the property being tested cannot be identified with sufficient power from outputs alone (Cen et al., 2024).

Current empirical work therefore converges on a layered view. Black-box auditing is indispensable because it is deployable, minimally intrusive, and directly tied to observable behavior. Yet tight auditing often requires additional ingredients: stronger query design, optimized or adversarial canaries, worst-case neighboring datasets, sequential or one-run amortization, and, in some domains, escalation to white-box access. For DP-SDGs, the explicit conclusion is that black-box MIAs alone are often too loose and that tight estimation currently requires white-box MIAs plus worst-case datasets (Annamalai et al., 2024). Open directions named across the literature include integrating worst-case dataset search with sequential tests, extending sequential auditing beyond approximate DP to Rényi DP and related notions, adaptive quantum canary generation, joint modeling of multiple hardware noise sources, and broader inference threat models beyond differential privacy leakage alone (González et al., 8 Sep 2025, Song et al., 16 Dec 2025).