Unsupervised Conformal Inference

• The paper demonstrates that unsupervised conformal inference employs geometric conformity scores to yield distribution-free prediction sets without labeled data.
• It uses bootstrapped batch processing and conformal alignment to guarantee finite-sample coverage and robust risk control.
• The framework effectively reduces hallucination in LLM outputs, providing practical, label-free deployment in dynamic applications.

An unsupervised conformal inference framework provides finite-sample, distribution-free uncertainty quantification for predictions in settings where no labeled calibration data are available, where data are non-exchangeable or highly dynamic (e.g. under distribution shift), or where the predictive task—such as generation by LLMs—lacks intrinsic ground-truth reward. Such frameworks combine nonparametric calibration, black-box score extraction, and online adaptation to construct rigorous prediction sets or gates, utilizing only the geometric or statistical properties of the model outputs. The approach supports real-time, label-free filtering or risk control in demanding modern applications.

1. Core Principles and Architecture

Unsupervised conformal inference replaces the typical reliance on labeled calibration data with a geometric or distributional “compatibility” score, extracted directly from model outputs or intermediate representations. The fundamental workflow consists of:

• Computing a conformity or atypicality score, $S_i$, for each model prediction, often derived from intrinsic geometry (e.g., embedding Gram matrices) or energy-based similarities among a batch of outputs.
• Aggregating these scores to construct an empirical distribution, from which thresholds for acceptance (prediction set membership) are determined by quantile functions, thereby inducing prediction sets or gates.
• Applying conformal prediction theory (often via split, batched, or bootstrapped methods) to guarantee that, over finite samples, the marginal error rate (miscoverage) does not exceed a prescribed level %%%%1%%%%, without distributional or model-specific assumptions.
• Optionally, introducing an online, adaptive calibration strategy—updating critical thresholds continuously in response to empirical miscoverage rates, addressing arbitrarily nonstationary environments and supporting real-time deployment.

The key innovation is that no knowledge of the underlying data-generating process, label structure, or reward is required, so these methods are well-suited for black-box or API-only model access.

2. Geometric Conformity Scores and Batch Processing

A defining feature in practical unsupervised conformal inference—especially in LLMs or any high-dimensional generative model—is measurement of typicality via response-embedding interaction geometry. Let $n$ responses $\{Y_1, ..., Y_n\}$ have associated (unit-norm) embeddings $\{v_1, ..., v_n\}$. The $n\times n$ Gram matrix $G = VV^T$ captures pairwise inner products.

• The “energy” of each response is defined by

$$e(i; G) = \left(\sum_{j=1}^n \langle v_i, v_j \rangle^2\right)^{1/2}$$

• The normalized atypicality score is

$\Phi(i; G) = 1 - \frac{e(i; G)}{\sqrt{n}}$

where $\sqrt{n}$ is the maximum possible energy across the batch.

• Low $e(i;G)$ (hence high $\Phi$) indicates an outlier or novel (less-redundant) response.

Batchwise unsupervised conformal prediction (UCP) splits outputs into batches, computes scores in a leave-one-out fashion, and pools residuals across batches for robust quantile estimation. The “BB-UCP” variant introduces a bootstrapping stage: for each batch, generate multiple resampled sets of residuals $\{S_{j,k}\}$, pooling across all batches, and select the acceptance threshold $q$ as the calibrated quantile over all bootstrapped residuals. This procedure sharpens precision and stabilizes the threshold.

3. Conformal Alignment and Goal-Conditioned Filtering

A further extension, “conformal alignment,” calibrates the unsupervised threshold dynamically so that a user-specified predicate (e.g., a factuality measure or risk metric) is satisfied on unseen future batches with user-controllable coverage probability. Formally:

• For each calibration batch $j$, define a right-continuous batch predicate

$\mathcal{P}_j(\tau): [0,1] \rightarrow \{0,1\}$

taking value 1 if, at strictness $\tau$, the batch passes a target criterion (e.g., the conditional Value-at-Risk (CVaR) of a severity metric improves).

• The minimal passing strictness per batch is

$S_j = \inf \{ \tau : \mathcal{P}_j(\tau) = 1 \}$

• Across $B$ calibration batches, select the calibrated global threshold $\hat{\tau}$ as the $K$-th order statistic of $\{S_j\}$ (with $K$ set by the conformal calibration level).

This procedure guarantees that, on a new batch, $\mathbb{P}[\mathcal{P}(\hat{\tau})=1]\geq 1-\alpha$—ensuring, at inference, that goal-oriented constraints (e.g., low hallucination) are achieved with high probability using only unsupervised geometric signals as proxies.

4. Performance, Calibration, and Empirical Behavior

Empirical results on multiple LLM benchmarks (e.g., ASQA, NQ-Open, HotpotQA, AmbigQA) demonstrate:

• Near-nominal coverage rates for test-set acceptance (i.e., the fraction of LLM outputs passing the conformal gate is $\approx 1-\alpha$).
• Substantially improved threshold stability and interval width (i.e., batch acceptance quantiles $q$ are tighter and less volatile) with BB-UCP compared to classic split-UCP; the batched, bootstrapped approach utilizes data more efficiently.
• Marked reduction in output “hallucination severity” as quantified by quality metrics such as BERTScore-F1 on answer heads, primarily due to the filtering effect of higher strictness on outlier generations.
• Computational cost comparable to leading per-sample detectors; notably, the framework is API-compatible, requiring only output features, and is deployable label-free.

These improvements derive from systematically aggregating unsupervised geometric evidence at the batch level and integrating conformal risk control for universal, distribution-free guarantees.

5. Comparison with Classical Methods and Applicability

Traditional conformal prediction, in the absence of labels, cannot be directly applied. Even variants relying on surrogate losses or external signals (e.g., lightweight outlier detectors) generally lack rigorous finite-sample guarantees and struggle with instability when deployed over batched, high-dimensional generative outputs.

The BB-UCP and conformal alignment framework differ by:

Method	Label Requirement	Calibration Level	Main Score	Coverage Guarantee
Split-UCP	None	Split-batch	Geometric	Marginal (finite-$n$)
BB-UCP	None	Bootstrap, batch	Geometric	Marginal (tighter q)
Per-response det.	None	None	Heuristic	None

By requiring only exchangeability within sampled batches and leveraging intrinsic geometry, the unsupervised conformal inference gate is applicable in LLM API-based production, where retraining or label access is infeasible.

6. Practical Implications and Extensions

The framework provides a robust, label-free risk control mechanism for open-ended or generative tasks. It enables:

• Calibrated batch or stream-level decision gates translating embedding-based geometric signals into actionable accept/reject thresholds.
• Easily-defined, goal-aligned deployment targets: factuality enhancement, hallucination reduction, or application-specific risk control.
• Extensible design: practitioners can tailor the underlying geometric score to use richer representations, multi-layer ensembles, or domain-adapted batch predicates, provided the exchangeability and batched calibration regime is preserved.

A plausible implication is the applicability of this approach to dense retrieval, content moderation, model selection, or any setting where outputs are high dimensional and no reference labels exist, and groupwise error control is desirable.

7. Key Formulas

Principal statistical operations underpinning the method include:

• Response Gram matrix:

$G = V V^\top$

• Per-response energy:

$$e(i; G) = \left(\sum_j \cos^2\theta_{ij}\right)^{1/2}$$

• Atypicality score:

$\Phi(i; G) = 1 - \frac{e(i;G)}{\sqrt{n}}$

• Bootstrapped acceptance quantile:

$q = \text{Quantile}_{1-\alpha}(\{S_{j,k}\}_{j,k})$

• Calibrated global strictness (conformal alignment):

$$\hat{\tau} = S_{(K)}, \quad K = \lceil (B+1)\alpha \rceil$$

These statistical primitives together yield a practical blueprint for label-free, rigorous uncertainty quantification in the unsupervised deployment of large models (Pang et al., 26 Sep 2025).