Model-Agnostic Reference Distribution

Updated 4 July 2026

Model-Agnostic Reference Distribution is a framework that defines external benchmarks for calibrating and comparing outputs independently of model internals.
It is applied in varied settings such as network statistics, embedding-space confidence filtering, regression calibration, self-supervised transfer, and data valuation.
Implementations leverage mixtures of simulated models, empirical residuals, and synthetic geometric targets to quantify uncertainty and standardize performance.

Model-agnostic reference distribution denotes a reference law used to standardize, calibrate, filter, compare, or align outputs while remaining independent of a particular model architecture or internal scoring rule. The literature uses the term in several technically distinct ways rather than as a single canonical formalism. In network analysis it appears as a mixture of simple random-graph models for size-correcting graph statistics; in image classification as the empirical distribution of training embeddings for confidence filtering; in regression calibration as the conditional residual law $P_{U\mid X=x}$ for a fixed black-box predictor; in self-supervised transfer as a predefined geometric target law $P_{\mathcal R}$ ; in data valuation as a trusted clean empirical measure $P_0$ ; and in preference evaluation as a reference pair distribution $P_{\mathrm{pair}}$ over $(x,y^+,y^-)$ triples (Smith et al., 2015, Kazanskii et al., 5 Sep 2025, Liu et al., 2023, Jiao et al., 20 Feb 2025, Zhu et al., 30 Jun 2025, Li, 29 May 2026).

1. General formulation and taxonomy

A reference distribution is model-agnostic when it is specified without committing to the internals of the evaluated or calibrated model. The precise meaning varies by domain. In the pairwise-alignment formulation, model-agnosticity is explicit: no model $M$ or scoring rule appears in the definition of $P_{\mathrm{pair}}$ (Li, 29 May 2026). In nonparametric regression calibration, the predictor $\hat f$ is treated as a black box, and calibration proceeds only through residuals $U=Y-\hat f(X)$ (Liu et al., 2023). In self-supervised transfer, the reference law $P_{\mathcal R}$ is fixed “once-and-for-all” and depends on $P_{\mathcal R}$ 0, $P_{\mathcal R}$ 1, $P_{\mathcal R}$ 2, $P_{\mathcal R}$ 3, and $P_{\mathcal R}$ 4, not on the neural-net architecture (Jiao et al., 20 Feb 2025).

Setting	Reference distribution	Primary role
Network statistics	$P_{\mathcal R}$ 5	Standardize graph-level measures across network sizes
Embedding-space confidence	$P_{\mathcal R}$ 6	Detect low-confidence or out-of-distribution test points
Regression calibration	$P_{\mathcal R}$ 7	Calibrate conditional quantiles for a fixed regressor
Self-supervised transfer	fixed $P_{\mathcal R}$ 8 on $P_{\mathcal R}$ 9	Match representation distribution to a predefined target
Data valuation	$P_0$ 0	Measure distributional influence relative to clean data
Pairwise alignment	$P_0$ 1 on $P_0$ 2	Define ordinal agreement observables

The same design pattern also appears in conditional language generation, where the empirical reference distribution for one context is $P_0$ 3, and the objective is to compare the generated empirical law $P_0$ 4 to $P_0$ 5 rather than scoring one hypothesis against one best-matching reference (Chan et al., 2022). This suggests that “model-agnostic reference distribution” is best understood as a family of constructions that externalize the comparison target.

2. Mixture reference distributions for network statistics

Smith, Calder and Browning study the problem of comparing graph-level summaries across networks with different numbers of nodes and show that direct comparison of raw and normalized statistics is often inappropriate because many summaries are strong functions of $P_0$ 6 (Smith et al., 2015). Their reference distribution is a mixture of simple random-graph models fitted to a subset of observed networks: $P_0$ 7 with weights $P_0$ 8, $P_0$ 9, and, in many examples, $P_{\mathrm{pair}}$ 0. Candidate component families include Bernoulli graphs, mean-degree-preserving Bernoulli graphs, and hierarchical Bernoulli graphs.

The operational procedure has two stages. First, one chooses $P_{\mathrm{pair}}$ 1, randomly samples $P_{\mathrm{pair}}$ 2 observed networks, and fits the chosen parametric model to each sampled network to obtain $P_{\mathrm{pair}}$ 3. Second, for each observed network $P_{\mathrm{pair}}$ 4 of size $P_{\mathrm{pair}}$ 5, one simulates $P_{\mathrm{pair}}$ 6 graphs of size $P_{\mathrm{pair}}$ 7 from each $P_{\mathrm{pair}}$ 8, pools the $P_{\mathrm{pair}}$ 9 simulated graphs, computes the statistic $(x,y^+,y^-)$ 0 on each draw, estimates $(x,y^+,y^-)$ 1 and $(x,y^+,y^-)$ 2, and reports the adjusted statistic

$(x,y^+,y^-)$ 3

Empirical percentiles can be used instead, but the z-score is the default.

The simulation study uses network sizes $(x,y^+,y^-)$ 4, six ground-truth models, five common statistics, and 200 replicates at each size. The six generating models are an Erdős–Rényi/ $(x,y^+,y^-)$ 5 graph with $(x,y^+,y^-)$ 6, a Bernoulli graph with $(x,y^+,y^-)$ 7, a mean-degree-preserving Bernoulli graph with the Krivitsky–Handcock offset, a Markov ERGM fit to the Florentine marriage data, a hierarchical Bernoulli HERGM, and a hierarchical Markov HERGM. The five statistics are degree centralization, closeness centralization, betweenness centralization, average path length, and transitivity. Without adjustment, the empirical distributions shift markedly with $(x,y^+,y^-)$ 8; a simple $(x,y^+,y^-)$ 9-model z-score helps centralization measures but fails badly on topological measures; and the Bernoulli-mixture adjustment yields histograms “almost perfectly overlapping” across sizes, KS statistics close to zero, and Anderson–Darling statistics reduced by an order of magnitude (Smith et al., 2015).

The practical guidance is correspondingly specific. The simple Bernoulli family is recommended for most real applications because it is fast to fit and already captures the dominant density effect. The mean-degree-preserving Bernoulli mixture is useful when average degree is expected to stay constant as networks grow. Stable results are reported for $M$ 0 on the order of $M$ 1– $M$ 2 and $M$ 3 around $M$ 4 per observed graph size. Computation can be reduced by simulating once per unique network size and reusing reference draws. The adjusted score is explicitly relative, not an absolute feature of the graph. Limitations are equally clear: a single Bernoulli mixture may be inadequate for extreme structural heterogeneity, and z-scores may be suboptimal when the reference distribution of $M$ 5 is highly non-Gaussian. The methodology is also applied to co-location networks derived from the Los Angeles Family and Neighborhood Survey activity location data (Smith et al., 2015).

3. Embedding-space reference distributions for confidence and out-of-distribution filtering

Kazanskii and Kasianov define the training-set embedding cloud as an empirical reference distribution

$M$ 6

where $M$ 7 are $M$ 8-dimensional embeddings of the training points under a chosen embedding model (Kazanskii et al., 5 Sep 2025). For smoothing, they describe a kernel density estimate

$M$ 9

and a Gaussian mixture model

$P_{\mathrm{pair}}$ 0

although the actual implementation is N-nearest-neighbors based.

Given a test embedding $P_{\mathrm{pair}}$ 1, uncertainty or out-of-distributionness is quantified by one of three surrogates: negative log-density, Mahalanobis distance

$P_{\mathrm{pair}}$ 2

or the $P_{\mathrm{pair}}$ 3-th nearest-neighbor distance $P_{\mathrm{pair}}$ 4. A prediction is accepted as high confidence when $P_{\mathrm{pair}}$ 5, equivalently when density is sufficiently high or the $P_{\mathrm{pair}}$ 6-th neighbor distance is below a threshold. The threshold is tuned on a held-out validation split, specifically a 75/25 split of ImageNet-V2, to maximize the area under the accuracy-versus-coverage curve, called the “Normalized Confidence Gain.”

The pipeline is deliberately post hoc. One precomputes reference embeddings on the training set, fits a reference distribution or stores the embeddings for nearest-neighbor lookup, tunes thresholds on validation data, and then filters test predictions. The method is therefore agnostic to the classifier itself; the classifier $P_{\mathrm{pair}}$ 7 is fixed, and only external embedding models are used to define the reference geometry. The framework also supports multiple embeddings. In one formulation, one may use

$P_{\mathrm{pair}}$ 8

while the paper’s implemented ensemble is a greedy coverage-allocation scheme across embeddings (Kazanskii et al., 5 Sep 2025).

Empirically, the NN-count filter improves accuracy at reduced coverage across all tested architectures. For ResNet-50, benchmark accuracy $P_{\mathrm{pair}}$ 9 increases to roughly $\hat f$ 0– $\hat f$ 1 after dropping the $\hat f$ 2– $\hat f$ 3 hardest samples. Stronger embedding models, especially DINOv2-B/14, yield higher confidence gains. On the more shifted external test set ObjectNet, performance drops, with best single-embedding NCG around $\hat f$ 4 and ensemble NCG around $\hat f$ 5, indicating sensitivity to out-of-distribution shift. The same recipe is proposed for NLP by replacing image embeddings with sentence or document encoders and applying the same reference-distribution estimation and threshold tuning to held-out text (Kazanskii et al., 5 Sep 2025).

4. Residual-space and parameter-space reference distributions for uncertainty quantification

In regression calibration, the reference object is the conditional residual distribution of a fixed black-box predictor. With i.i.d. samples $\hat f$ 6 and a pretrained regressor $\hat f$ 7, one defines residuals $\hat f$ 8 and seeks the conditional law $\hat f$ 9 in order to form calibrated quantiles

$U=Y-\hat f(X)$ 0

The proposed Simple Nonparametric Quantile Estimator constructs a local weighted empirical distribution

$U=Y-\hat f(X)$ 1

and then minimizes the weighted pinball loss. The resulting Nonparametric Regression Calibration procedure uses sample splitting: one half trains $U=Y-\hat f(X)$ 2, the other half supplies residuals for nonparametric estimation (Liu et al., 2023).

Theoretical guarantees are explicit. Under Lipschitzness of the true conditional quantile, boundedness, and a local density lower bound, the SNQ estimator is pointwise consistent and achieves mean-square error

$U=Y-\hat f(X)$ 3

with a matching minimax lower bound over a Lipschitz class. The paper also shows that dimensionality reduction through a summary $U=Y-\hat f(X)$ 4 can improve the rate to $U=Y-\hat f(X)$ 5 when $U=Y-\hat f(X)$ 6. This yields an explicitly model-agnostic reference distribution $U=Y-\hat f(X)$ 7, but not an assumption-free one; the analysis makes the curse of dimensionality central rather than incidental (Liu et al., 2023).

A distinct parameter-space formulation appears in probabilistic model-agnostic meta-learning. PLATIPUS learns a Gaussian prior

$U=Y-\hat f(X)$ 8

over model parameters and a variational posterior generated by a noisy gradient update,

$U=Y-\hat f(X)$ 9

where $P_{\mathcal R}$ 0 and $P_{\mathcal R}$ 1 are learned meta-parameters (Finn et al., 2018). At meta-test time, one samples $P_{\mathcal R}$ 2 and adapts with noisy gradient steps on the new task, obtaining an empirical posterior over plausible task-specific models. The reference distribution here is not an external data law but a learned prior over weights, and the paper characterizes it as model-agnostic because it can be used with any differentiable architecture. Its empirical role is to capture multimodal ambiguity in few-shot learning and to support active learning via posterior variance (Finn et al., 2018).

5. External reference sets and target laws in optimization and data valuation

KAIROS assumes two disjoint finite datasets: a trusted reference set $P_{\mathcal R}$ 3, drawn i.i.d. from a clean target distribution $P_{\mathcal R}$ 4, and a training set $P_{\mathcal R}$ 5, drawn from $P_{\mathcal R}$ 6 (Zhu et al., 30 Jun 2025). The empirical measures are

$P_{\mathcal R}$ 7

Their discrepancy is quantified by the RKHS Maximum Mean Discrepancy: $P_{\mathcal R}$ 8 The data-valuation score of a training point is defined through the influence of infinitesimally reweighting that point, which reduces to

$P_{\mathcal R}$ 9

The paper states that this score matches the true leave-one-out change in $P_{\mathcal R}$ 00 up to $P_{\mathcal R}$ 01 error in ranking, uniformly over all points. It further adds conditional kernels for label-error detection, proves symmetry and density-separation properties, and supports online updates in $P_{\mathcal R}$ 02 time for a new batch of size $P_{\mathcal R}$ 03. On benchmark corruption and poisoning tasks, KAIROS reports near-perfect overlap with true leave-one-out top- $P_{\mathcal R}$ 04 sets, specifically $P_{\mathcal R}$ 05 overlap for top-100 versus roughly $P_{\mathcal R}$ 06 for Lava, and up to $P_{\mathcal R}$ 07 speedup in online settings (Zhu et al., 30 Jun 2025).

Distribution Matching for self-supervised transfer learning uses a different reference construction. The encoder $P_{\mathcal R}$ 08 induces a representation distribution $P_{\mathcal R}$ 09, and the training objective combines augmentation alignment with Wasserstein matching to a fixed reference law: $P_{\mathcal R}$ 10 The reference law $P_{\mathcal R}$ 11 is a mixture of $P_{\mathcal R}$ 12 spherical-cap blobs on the radius- $P_{\mathcal R}$ 13 sphere, obtained by choosing centers $P_{\mathcal R}$ 14, sampling a small perturbation $P_{\mathcal R}$ 15 from the unit sphere, and setting

$P_{\mathcal R}$ 16

Because $P_{\mathcal R}$ 17 depends only on the geometric hyperparameters and not on the network architecture, it functions as a genuinely external target (Jiao et al., 20 Feb 2025).

The theoretical claims connect this target law to downstream classification. The population theorem bounds class overlap in the target domain by $P_{\mathcal R}$ 18, where $P_{\mathcal R}$ 19 measures domain shift. The end-to-end sample theorem states that, under augmentation and approximation assumptions, the target misclassification error decreases at a rate determined by $P_{\mathcal R}$ 20, even when labeled target data are limited. Empirically, with a ResNet-18 backbone and $P_{\mathcal R}$ 21, DM is competitive with SimCLR, Barlow Twins, and VICReg, reporting $P_{\mathcal R}$ 22 linear-probe/5-NN accuracy on CIFAR-10, $P_{\mathcal R}$ 23 on CIFAR-100, and $P_{\mathcal R}$ 24 on STL-10 (Jiao et al., 20 Feb 2025).

6. Reference distributions for evaluation, ranking, and alignment

In conditional natural language generation, the reference side is explicitly distributional. For a single conditioning context, the set of $P_{\mathcal R}$ 25 human utterances $P_{\mathcal R}$ 26 defines

$P_{\mathcal R}$ 27

and a model under evaluation provides $P_{\mathcal R}$ 28 samples $P_{\mathcal R}$ 29 with

$P_{\mathcal R}$ 30

The objective is to compare $P_{\mathcal R}$ 31 and $P_{\mathcal R}$ 32, jointly capturing sample quality and support dispersion. The proposed Triangle-Rank Metrics build mixed triangles with one in-distribution edge and two cross-distribution edges, count how often the in-distribution edge is smallest, middle, or largest, and define

$P_{\mathcal R}$ 33

Kernel-based alternatives include Fréchet BERT Distance and MMD in embedding space (Chan et al., 2022).

The method is explicitly model-agnostic: it requires only black-box sampling, not internal logits or likelihoods. It is also reference-aware in a stronger sense than max-over-references metrics because all references contribute to the empirical $P_{\mathcal R}$ 34. Complexity for TRM is $P_{\mathcal R}$ 35, which the paper describes as feasible for $P_{\mathcal R}$ 36. In a visual-description case study with MS-COCO and MSR-VTT, using $P_{\mathcal R}$ 37 samples from CLIPCap or VLP, TRM becomes significant with $P_{\mathcal R}$ 38– $P_{\mathcal R}$ 39 samples and yields approximately $P_{\mathcal R}$ 40– $P_{\mathcal R}$ 41 gain in statistical sensitivity relative to average METEOR. As temperature varies from $P_{\mathcal R}$ 42 to $P_{\mathcal R}$ 43, average METEOR declines monotonically, whereas $P_{\mathcal R}$ 44 is unimodal, diagnosing both underdispersion and overdispersion (Chan et al., 2022).

A more stripped-down evaluative formalism appears in pairwise reference alignment. One fixes a reference pair distribution $P_{\mathcal R}$ 45 on $P_{\mathcal R}$ 46, defines a model score $P_{\mathcal R}$ 47, and measures agreement by

$P_{\mathcal R}$ 48

The centered statistic

$P_{\mathcal R}$ 49

acts as an order-parameter-like summary, and a margin extension uses $P_{\mathcal R}$ 50 with optional thresholding. Finite-sample estimators are simple sample averages, and Hoeffding bounds yield

$P_{\mathcal R}$ 51

In an empirical illustration with Qwen2.5 models on 5,120 RewardBench pairs, sign agreement increases with model size and instruction tuning, and bootstrap half-widths closely track the conservative Hoeffding radius of approximately $P_{\mathcal R}$ 52 (Li, 29 May 2026).

7. Cross-cutting properties, misconceptions, and limitations

Across these formulations, model-agnosticity is implemented in at least three distinct ways. First, the reference can be empirical and external to the model, as in training embeddings, multi-reference text sets, trusted clean datasets, or pairwise human-preference corpora (Kazanskii et al., 5 Sep 2025, Chan et al., 2022, Zhu et al., 30 Jun 2025, Li, 29 May 2026). Second, it can be a black-box residual law that depends on the predictor only through observed errors, as in nonparametric regression calibration (Liu et al., 2023). Third, it can be a deliberately synthetic target distribution, such as the spherical-cap mixture used for self-supervised transfer, or a simple fitted mixture chosen to normalize observed objects rather than to model them mechanistically (Jiao et al., 20 Feb 2025, Smith et al., 2015).

A recurrent misunderstanding is to equate model-agnosticity with absence of assumptions. The cited methods do not support that interpretation. Network standardization still requires choosing mixture components and may fail under extreme structural heterogeneity; z-scores may be poor summaries when the reference distribution is highly non-Gaussian (Smith et al., 2015). Embedding-space filtering depends on the choice of embedding model, distance, and validation threshold, and its gains diminish under stronger external shift (Kazanskii et al., 5 Sep 2025). Nonparametric residual calibration is subject to the curse of dimensionality and relies on Lipschitz and density conditions for its finite-sample guarantees (Liu et al., 2023). KAIROS assumes access to a trusted clean reference set (Zhu et al., 30 Jun 2025). Distribution-aware NLG evaluation is most natural when multiple human references are available (Chan et al., 2022). Pairwise reference alignment is always relative to the chosen $P_{\mathcal R}$ 53 and the chosen scalar score $P_{\mathcal R}$ 54; high agreement under one pair distribution does not imply high agreement under another (Li, 29 May 2026).

A second common misconception is that a reference distribution must be a density model. The literature contradicts this directly. Some constructions are empirical measures with Dirac masses; some are simple parametric mixtures; some are induced by rank distributions; some are RKHS mean embeddings; and some are priors over parameter space. This suggests that the unifying idea is not density estimation per se, but the externalization of a comparison target against which outputs, statistics, or hypotheses can be standardized. Under that interpretation, the model-agnostic reference distribution is less a single method than a general statistical device for separating the question of what constitutes the relevant baseline from the question of how a particular model produces its outputs.