Reference-Based Membership Inference

• Reference-Based Membership Inference is a technique that calibrates model scores against auxiliary reference measures to distinguish true membership signals from intrinsic confounders.
• The approach employs diverse reference sources—such as shadow models, synthetic neighbors, and public probes—to account for sample difficulty and domain typicality.
• Recent advancements demonstrate that calibrated loss differences enhance attack accuracy and reduce computational costs, thereby strengthening privacy auditing assessments.

Reference-based membership inference is a class of membership inference attacks (MIAs) in which the evidence for membership is not taken from a target model’s score in isolation, but from that score relative to a reference quantity intended to account for sample difficulty, domain typicality, or counterfactual behavior. In the surveyed literature, the reference may be a separately trained reference model, a distribution over shadow models, a set of reference population records, synthetic neighboring samples, a self-prompted proxy corpus, or a counterfactual output from the same system. The unifying goal is to distinguish true membership effects from confounders such as intrinsic easiness, domain frequency, or broad semantic similarity (Mattern et al., 2023, Zarifzadeh et al., 2023).

1. Core concept and formal structure

A standard reference-free attack thresholds a target-model score directly. In the language-model formulation, this is written as

$$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) < \gamma].$$

Reference-based attacks replace this with a calibrated statistic,

$$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$

where $d(x)$ is a reference-derived difficulty term. In classification-oriented formulations, the same idea appears as a threshold test on a membership score,

$MIA = \mathbbm{1}_{Score \ge \beta}.$

The essential move is calibration: the target score is judged against a baseline rather than absolutely (Mattern et al., 2023, Zarifzadeh et al., 2023).

The “reference” is therefore not a single mechanism. In one family, $d(x)$ is the loss under a reference model $f_{\theta'}$. In another, the reference is a population sample $z \sim \pi$, and the attack asks whether the target point dominates that sampled alternative under a likelihood-ratio test. In another, the reference is local rather than global: synthetic neighbors $\tilde{x}_1,\ldots,\tilde{x}_n$ around the queried sample act as interchangeable non-members. In recommender systems, a reference can even be a counterfactual output from the same deployed system, such as an attribute-only recommendation for the same user (Mattern et al., 2023, Zarifzadeh et al., 2023, Chi et al., 10 Dec 2025).

Reference source	Representative statistic	Typical purpose
Reference model(s)	$\mathcal{L}(f_\theta,x)-\mathcal{L}(f_{\theta'},x)$	Difficulty calibration
Reference population records	$\Pr_{z\sim\pi}(LR \ge \gamma)$	Fine-grained null modeling
Synthetic neighbors or counterfactual outputs	$$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$0, or $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$1	Local or personalized calibration

This broadening of the reference object is one of the field’s main developments. A plausible implication is that “reference-based” is best understood as a relational notion: membership is inferred from how a candidate behaves relative to an auxiliary baseline, not from any single score alone.

2. Shadow models and the classical reference-model paradigm

The classical black-box formulation decomposes the attack into three stages: construct a shadow dataset $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$2, train one or more shadow or reference models on $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$3, and then train an attack model on the posterior vectors emitted by those models for known members and non-members. In this formulation, the target classifier is

$$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$4

and the attack model learns from posterior vectors $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$5 labeled as $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$6 or $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$7. This makes explicit what the reference object is in the classical setting: a learned distribution of posterior behavior under membership and non-membership (Truex et al., 2018).

Subsequent work made this paradigm more scalable and less dependent on exact target imitation. Bagged GMIA assigns each candidate point to exactly $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$8 of $$A_{f_\theta}(x) = \mathbf{1}[\mathcal{L}(f_\theta, x) - d(x) < \gamma],$$9 reference training sets, yielding expected reference-set size

$d(x)$0

It then trains a separate logistic attack model for each point from the confidence vectors produced by reference models on that same point. Empirically, smaller reference-model training sets often strengthened the attack, and the reference models did not need to share the target architecture (Felps et al., 2020).

Large-scale benchmarking later showed that some classical shadow-model assumptions are weaker than often claimed. In particular, “same-architecture and same-distribution between shadow and target models are unnecessary,” and even Internet-collected auxiliary data can support strong attacks. The benchmark also organized attacks into neural-network-based and metric-based families, both still rooted in reference behavior learned from shadow models (He et al., 2022).

3. LLMs: calibrated likelihoods, distribution mismatch, and sequence-aware references

In language modeling, the main justification for reference-based calibration is that raw likelihood is confounded by sample complexity. Easy texts—short, repetitive, formulaic, or otherwise broadly probable—can receive low loss regardless of membership. A reference model therefore serves as a baseline notion of how likely the same sample is “in the domain” more generally. In a grey-box language-model setting, LiRA is instantiated as

$d(x)$1

so the attack statistic becomes

$d(x)$2

or equivalently a likelihood ratio. Yet this calibration is highly sensitive to reference quality. On AG News, for example, the LOSS baseline achieved $d(x)$3 TPR at $d(x)$4 FPR, LiRA with the base reference model reached $d(x)$5, while the oracle-reference attack jumped to $d(x)$6. The same paper shows that reference-attack performance peaks only when the reference model’s validation perplexity is very close to that of the target model, which is the central empirical statement of brittleness under imperfect attacker knowledge (Mattern et al., 2023).

This fragility motivated two distinct responses in LLM work. One response was local synthetic calibration. The neighbourhood attack replaces a separately trained reference model with synthetic neighboring texts $d(x)$7, using

$d(x)$8

In the reported experiments, many highly local one-word replacements worked best, and the method outperformed realistic imperfect-knowledge reference attacks on AG News, Twitter, and Wikipedia (Mattern et al., 2023).

A second response was self-prompt calibration for fine-tuned LLMs. SPV-MIA constructs a synthetic reference dataset $d(x)$9 by querying the target LLM itself, fine-tunes one reference model on that dataset, and then calibrates a memorization-oriented score called probabilistic variation: $MIA = \mathbbm{1}_{Score \ge \beta}.$0 Across Wikitext, AG News, and XSum, average AUCs reported for SPV-MIA were $MIA = \mathbbm{1}_{Score \ge \beta}.$1, $MIA = \mathbbm{1}_{Score \ge \beta}.$2, and $MIA = \mathbbm{1}_{Score \ge \beta}.$3, compared with $MIA = \mathbbm{1}_{Score \ge \beta}.$4, $MIA = \mathbbm{1}_{Score \ge \beta}.$5, and $MIA = \mathbbm{1}_{Score \ge \beta}.$6 for LiRA-Candidate. The paper also reports that the self-prompt reference dataset is only slightly worse than an ideal identical-distribution dataset (Fu et al., 2023).

Reference distributions themselves also had to be adapted to the structure of sequence models. Sequence-aware LiRA replaces a scalar sequence score with a vector of per-token losses,

$MIA = \mathbbm{1}_{Score \ge \beta}.$7

and performs a multivariate likelihood-ratio test with IN and OUT Gaussian distributions over $MIA = \mathbbm{1}_{Score \ge \beta}.$8. The strongest variant uses covariance-aware modeling with Oracle Approximating Shrinkage, often with shared covariance. On Pythia average-case canaries at $MIA = \mathbbm{1}_{Score \ge \beta}.$9 FPR, Univariate Shared achieved $d(x)$0, whereas Independent Shared and OAS Shared reached $d(x)$1 and $d(x)$2, respectively. The reference-based mechanism remains LiRA-style calibration, but the reference distributions are defined over token-level conditional-loss structure rather than a single aggregated loss (Rossi et al., 5 Jun 2025).

4. Low-cost and uncertainty-aware reference mechanisms

A central engineering problem in reference-based MIA is computational cost. RMIA addresses this by combining reference models and reference population records in a single likelihood-ratio framework. Using Bayes’ rule, it computes

$d(x)$3

then aggregates across population samples,

$d(x)$4

with the final decision

$d(x)$5

This makes RMIA reference-based in two senses at once: reference models estimate $d(x)$6 and $d(x)$7, while reference population points instantiate the null world. The paper reports that with only 2 reference models, RMIA achieves 5–10% higher AUC and $d(x)$8 to $d(x)$9 higher TPR at low FPRs than prior methods, and that with only one reference model it shows at least 26% higher AUC and 100% more TPR at low FPRs than LiRA over all datasets (Zarifzadeh et al., 2023).

BMIA pursues the same efficiency goal differently. Instead of many reference models, it trains one reference model and converts it into a Bayesian neural network via Laplace approximation. For a query $f_{\theta'}$0, it computes a target hinge score,

$f_{\theta'}$1

samples posterior weights from the Bayesianized reference model, evaluates corresponding reference scores, and performs a one-sided one-sample $f_{\theta'}$2-test on the score differences. The method is therefore still reference-based, but the ensemble of reference models is compressed into a posterior around a single reference model. In the main table, BMIA reached $f_{\theta'}$3 TPR@$f_{\theta'}$4 FPR on CIFAR-100, compared with $f_{\theta'}$5 for LiRA $f_{\theta'}$6 and $f_{\theta'}$7 for RMIA $f_{\theta'}$8 (Liu et al., 10 Mar 2025).

Taken together, these methods show two different routes to low-cost reference-based inference: richer null modeling with fewer reference models, or probabilistic uncertainty modeling around one reference model.

5. Beyond global auxiliary models: public probes, personalized references, and external memory

Reference-based membership inference has expanded beyond globally trained auxiliary models. In fine-tuned LLMs, ICP-MIA-Ref uses semantically similar public examples as in-context probes rather than as training data for a reference model. For a candidate sample $f_{\theta'}$9 and a probe context $z \sim \pi$0, it computes

$z \sim \pi$1

and aggregates by

$z \sim \pi$2

The reference data are therefore public, semantically retrieved, and used to simulate an extra optimization step in context. The paper’s ablations show that complete distributional alignment works best, but task structure and semantic similarity independently matter (Lu et al., 18 Dec 2025).

In hybrid recommender systems, the reference can be a counterfactual recommendation from the same target system. The attacker queries once with the user’s interactions and attributes to obtain the target recommendation, and once with attributes only to obtain a reference recommendation. Historical interactions, target recommendation, and reference recommendation are embedded as latent vectors $z \sim \pi$3, $z \sim \pi$4, and $z \sim \pi$5, and membership is inferred through

$z \sim \pi$6

On DropoutNet with ML-1M, the reported attack success rate was 0.9340, with 99.84% TPR@1%FPR (Chi et al., 10 Dec 2025).

A further extension shifts membership from model training data to external reference stores. In long-context LLMs, the attack problem becomes

$z \sim \pi$7

where membership means that a target document $z \sim \pi$8 is present in the external context $z \sim \pi$9. Lower generation loss and higher semantic similarity between the model’s continuation and the candidate reference become the operative signals, and a meta-classifier over multi-granularity features reaches 90.66% F1 on 30-document multi-document QA with LongChat-7b-v1.5-32k (Wang et al., 2024). In image-based retrieval-augmented generation, the membership target is the hidden retrieval database rather than the model weights; ImageAuditor decomposes each query into a retrieval segment and an extraction segment, and reports AUROC above 0.8 with only four queries per audited image (Zhang et al., 2 Jun 2026). In retrieval-based in-context learning for document QA, a separate reference-model attack uses a local reference LM plus prefix probes to estimate an otherwise unavailable loss metric from black-box outputs alone (Kulkarni et al., 5 May 2026).

These cases broaden the scope of reference-based membership inference from “training-set auditing” to “auditing hidden context or retrieval memory.” The reference object becomes a retrieved document, a database item, or a counterfactual system response.

6. Realism, brittleness, and the role of reference-based MI in privacy auditing

The principal controversy in this literature is not whether reference-based MIAs can be strong, but under what assumptions they are strong. Several papers argue that favorable results often rely on matched reference data, matched hyperparameters, known member fractions, or strong query interfaces. A systematic critique states that removing knowledge of training hyperparameters, same-distribution non-members, and the fraction of members in the evaluation set leads to “a significant drop in the performance of black-box attacks.” In the corresponding no-assumptions setting, LiRA on CIFAR-10 dropped to 0.55 TPR at $\tilde{x}_1,\ldots,\tilde{x}_n$0 FPR, whereas the white-box, reference-free ImpMIA reached 2.76 (Golbari et al., 12 Oct 2025). This does not invalidate reference-based MI, but it sharply restricts when reference-model results can be interpreted as realistic attack power.

A related development is the attempt to estimate model-level vulnerability without reference models. “The Tail Tells All” argues that strong attacks such as LiRA are expensive because they use many reference models, but that model-level vulnerability to those attacks is reflected in the shape of the target model’s own train/test loss distributions. For image models, the proposed statistic—the TNR of a simple loss attack at the relevant operating point—predicts LiRA’s TPR@FPR=$\tilde{x}_1,\ldots,\tilde{x}_n$1 with $\tilde{x}_1,\ldots,\tilde{x}_n$2 under a linear fit and $\tilde{x}_1,\ldots,\tilde{x}_n$3 under an exponential fit (Dodd et al., 22 Oct 2025). The paper is explicit that this estimates vulnerability to LiRA rather than replacing LiRA as a pointwise attack.

The literature therefore supports two simultaneous conclusions. First, reference-based MI remains one of the most powerful paradigms for pointwise privacy auditing, especially when calibration truly reflects the target model’s data-generating conditions. Second, its practical relevance depends on the realism of the reference object—whether that is a matched corpus, a shadow-model family, a public probe set, or a counterfactual system output. The field’s recent trajectory has been to weaken or replace the hardest assumptions: local synthetic references instead of oracle corpora, self-prompted corpora instead of naturally matched public data, single-reference Bayesianization instead of large ensembles, and context- or retrieval-specific references instead of global shadow models. That trajectory suggests that the enduring core of reference-based membership inference is not any one attack family, but the general principle of membership by calibrated comparison.