MACI: Multi-LLM Adaptive Conformal Inference

• MACI is a conformal inference framework that employs multiplicative filtering to retain claims meeting a joint-factuality threshold, maximizing true claim retention.
• It uses adaptive ensemble scoring with learned weights and group-conditional calibration to meet finite-sample coverage guarantees in diverse, high-stakes applications.
• Empirical evaluations show MACI outperforms baseline methods by achieving higher retention rates and reduced computational latency across medical, biographical, and expert QA tasks.

Multi-LLM Adaptive Conformal Inference (MACI) is a conformal inference framework designed to provide reliable, distribution-free factuality guarantees for outputs of LLMs. MACI leverages a novel multiplicative filtering paradigm, group-conditional calibration, and adaptive ensemble LLM verification to maximize retention of true claims while enforcing user-specified finite-sample coverage guarantees in high-stakes applications such as medicine and law (Noh et al., 1 Feb 2026). MACI generalizes and extends strongly adaptive multi-model conformal prediction methods (Hajihashemi et al., 2024), integrating them with ensemble LLM scoring and advanced calibration strategies tailored to natural language factuality tasks.

1. Multiplicative Filtering for Joint Factuality

Conventional conformal inference frameworks filter atomic LLM-generated claims by thresholding individual factuality scores $p(c)\in[0,1]$ at a global level, retaining claims above a fixed threshold $\tau$. This procedure achieves marginal coverage but is typically excessively conservative, as the retention set for each output is governed by the least certain (worst) individual claim; this limits the number of true claims retained, resulting in low utility.

MACI replaces this paradigm with a multiplicative filtering rule: claims $c_{(1)},\dots,c_{(k)}$ (sorted by decreasing estimated factuality) are retained as the largest prefix such that the product of their scores, $\Pi_k = \prod_{j=1}^k p(c_{(j)})$, exceeds a specified threshold $\tau$. Under a conditional independence assumption, $\Pi_k$ equals the probability that all $k$ retained claims are true, thus maximizing utility (retention) subject to a joint-factuality constraint. For computational stability, the procedure uses additive log-scores: $\ell(c) = -\log(1-p(c)+\varepsilon),$ with cumulative sums used to determine retention.

This multiplicative perspective is the core methodological innovation; it ensures that the surviving set of claims has a calibrated probability of being fully factual, given the user-specified tolerance $\alpha$ for false positives (Noh et al., 1 Feb 2026).

2. Ensemble Scoring and Adaptive Model Selection

MACI utilizes an ensemble of $M$ independent verifier LLMs, each viewed as a black-box that estimates $\Pr(y=1\mid P,c)$ for any claim $c$ and input $P$. The verifiers often disagree on claim veracity, as indicated by low inter-model correlation. MACI combines verifier outputs through a weighted linear ensemble: $$p_{\mathrm{ens}}(P,c;w) = \sum_{m=1}^M w_m p_m(P,c),$$ where $w_m$ are non-negative, normalized weights.

The ensemble weights are learned by directly optimizing a surrogate objective that targets maximal retention (the proportion of claims kept) subject to high true-positive rate (TPR) constraints, while minimizing false-positive rate (FPR). Letting $\hat\tau_{w,\delta}$ denote the empirical $\delta$-quantile of scores on true claims, the optimization problem is: $$\min_{w}\; \frac1{n_{\mathrm{opt}}}\sum_{i=1}^{n_{\mathrm{opt}}} \widehat{\mathrm{FPR}_i}\left(p_{\mathrm{ens}}(\cdot;w),\,\hat\tau_{w,\delta}\right) \quad\text{s.t.}\quad \frac1{n_{\mathrm{opt}}}\sum_i \widehat{\mathrm{TPR}_i} \geq 1-\delta,$$ yielding, when performed per group, robust group-specific ensemble weights.

This adaptive ensemble selection generalizes the multi-model conformal prediction framework (e.g., MOCP, SAMOCP) to settings where each model is an LLM verifier, with weights dynamically informed by calibration and groupwise performance (Hajihashemi et al., 2024).

3. Group-Conditional Calibration and Error Guarantees

MACI enforces group-conditional coverage guarantees critical to high-stakes domains. For a grouping function $g:(P,C)\mapsto k\in\{1,\dots,K\}$ (e.g., by medical intent, question type, or risk group), calibration and threshold selection are performed separately within each group.

For each document $(P_i,C_i,Y_i)$ in group $k$, the Haldane-run conformity score is computed as

$$E_i = \inf\{\tau: F(\hat{p}_k, \tau, U_i; P_i, C_i) \subseteq A_i\}$$

where $U_i\sim \mathrm{Uniform}[0,1]$ introduces randomization for ties. The group-specific threshold is chosen as the empirical $(1-\alpha)$-quantile over the calibration set: $$\hat Q_{1-\alpha}^{(k)} = \mathrm{Quantile}_{1-\alpha}\left(\{E_i : g(P_i, C_i) = k\}\right).$$ With this selection, MACI guarantees

$$\Pr(\text{no false claim in output} \mid g=k) \geq 1-\alpha,$$

where $\alpha$ is fixed per group, providing finite-sample and deployable guarantees (Noh et al., 1 Feb 2026).

4. Algorithmic Workflow and Computational Complexity

MACI employs a three-pool split: an "opt" set for learning ensemble weights, a "cal" set for group-specific calibration, and a test set for evaluation. The main computational steps, per group, are:

1. Learn $w_k$ by optimizing the surrogate FPR/TPR objective over $\mathcal{D}_{\mathrm{opt},k}$.
2. Form groupwise ensemble scoring functions $\hat{p}_k(c)$.
3. Compute conformity scores $E_i$ and quantiles $\hat Q_{1-\alpha}^{(k)}$ over $\mathcal{D}_{\mathrm{cal},k}$.
4. On new data, assign group, compute claim scores, sort, and apply the multiplicative filter using the calibrated threshold.

The dominant complexity is $O(M\times|C|)$ (ensemble scoring for $|C|$ claims), with overhead from sorting and calibration steps remaining tractable. Weight optimization is fast on moderate calibration/optimization sets (Noh et al., 1 Feb 2026).

5. Empirical Behavior and Comparative Performance

MACI has been evaluated on MedLFQA (medical QA), WikiBio (biography generation), and ExpertQA (expert question answering), all decomposed into atomic claims and grouped according to application-relevant attributes. The ensemble LLM verifiers used include Llama-3.3-70B-Instruct, Qwen-2.5-72B, and DeepSeek-V3.

Key empirical observations include:

• For target $\alpha=0.1$ (90% coverage) on MedLFQA: BCI retention $\sim$2%, CCI $\sim$30%, MACI $\sim$50% (meeting coverage across all groups).
• On WikiBio: BCI/CCI retention $\sim$10–20%, MACI $\sim$25–30%.
• On ExpertQA: Sampling-based (SelfCheck, FSC-KG) methods achieve 80–90% retention but coverage $<$70%; MACI achieves 95% coverage at 15–20% retention.
• On 500 WikiBio tests, MACI yielded a 3× speedup (599s vs. 1644s for CCI) due to efficient single-pass scoring and lighter calibration.

This demonstrates that MACI achieves higher retention and reduced latency while strictly adhering to group-conditional coverage targets (Noh et al., 1 Feb 2026).

6. Comparative Analysis, Limitations, and Extensions

Compared to prior approaches:

• Baseline Conformal Inference (BCI): Based on worst-claim score; highly conservative, causing low retention.
• Conditional Conformal Inference (CCI): Employs adaptive linear rules for retention gains but still relies on worst-claim and linear groupings, and loses fixed $\alpha$ deployment.
• MACI: Achieves multiplicative aggregation over all retained claims, with exact per-group calibration and fixed $\alpha$ guarantees. The ensemble approach reduces estimation error, improving retention directly (as shown in Theorem 3.3).

Limitations include small-group calibration variance, handled via group-clustering at minor expense to guarantee strength. Under distributional (covariate) shift, MACI-DRE (density ratio estimation) can recover valid coverage. Empirically, switching from claim-wise to full joint scoring did not alter coverage substantially, but more advanced dependency modeling remains an open direction. Future extensions may involve deeper ensemble stacking or lighter calibration via techniques such as Jackknife+ (Noh et al., 1 Feb 2026).

7. Connections to Strongly Adaptive Multi-Model Conformal Prediction

The MACI architecture generalizes the MOCP/SAMOCP frameworks developed for robust conformal prediction in dynamic, non-stationary environments (Hajihashemi et al., 2024). In the multi-LLM context, each LLM is treated as an individual model whose outputs contribute to the ensemble factuality score. The adaptive machinery of MOCP/SAMOCP (exponential weighting, strongly adaptive regret minimization, and coverage calibration) underpins the ensemble weighting and dynamic filtering procedures in MACI, ensuring valid coverage across distributional shifts and heterogeneous subgroups.

MACI thus represents the synthesis of rigorous uncertainty quantification (conformal inference), adaptive multi-model selection, and ensemble LLM verification, yielding a scalable and reliable framework for factuality filtering in LLM-driven applications.