Dual-Set Conformal Prediction

• Dual-set conformal prediction encompasses techniques that combine evidence from multiple sources to maintain valid coverage under heterogeneity and non-exchangeability.
• These methods aggregate source-specific p-values through operators like averaging or max-aggregation to address constraints such as privacy and partial label information.
• Practical applications reveal a trade-off between efficiency and rigorous coverage, with tailored protocols for non-disclosed, hierarchical, and partial-label settings.

“Dual-set conformal prediction” is not presented in the cited literature as a single canonical method. The available work instead suggests an umbrella interpretation: conformal prediction procedures in which predictive evidence is constructed from more than one set, such as multiple non-disclosed datasets, multiple heterogeneous source distributions, hierarchical groups, or set-valued candidate labels. Across these settings, the unifying objective is to retain conformal-style coverage while respecting structural constraints that invalidate naive pooling, including privacy restrictions, source heterogeneity, partial exchangeability, and label ambiguity (Spjuth et al., 2018, Yang et al., 6 Jan 2026, Dunn et al., 2018, Javanmardi et al., 2023).

1. Conceptual scope and departure from standard conformal prediction

Conformal prediction produces valid prediction regions under mild conditions (Spjuth et al., 2018). In its standard form, however, the methodology assumes that the augmented sample is exchangeable and that training and calibration labels are precise rather than set-valued (Dunn et al., 2018, Javanmardi et al., 2023). These assumptions become problematic when data are split across non-disclosed repositories, when observations arise from multiple distinct source distributions, when only within-group exchangeability is available, or when labels are observed only through candidate sets.

A common formal template survives across these extensions. For a significance level $\epsilon$, conformal prediction regions take the form

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$

or an equivalent thresholded set construction based on conformal $p$-values or nonconformity scores (Spjuth et al., 2018). What changes is the mechanism by which those $p$-values or scores are constructed and combined. In some settings, the combination rule is arithmetic averaging of source-wise $p$-values; in others, it is a max operator, repeated subsampling average, or a transformation of set-valued calibration labels. This suggests that “dual-set” methods are better understood as a family of structured conformal constructions than as a single algorithmic object.

The principal technical difficulty is that naive pooling often destroys the conditions under which standard validity arguments apply. In hierarchical data, the full collection of observations is generally not exchangeable across groups (Dunn et al., 2018). In multi-source robustness, a test point may arise from an arbitrary member or mixture of several heterogeneous distributions, so source-wise guarantees must be combined into a single worst-case-valid set (Yang et al., 6 Jan 2026). In partial-label learning, the hidden precise labels underlying calibration data are exchangeable, but the observed objects are pairs $(x_i,S_i)$ with $y_i \in S_i$, so standard scalar calibration scores are unavailable (Javanmardi et al., 2023).

2. Non-disclosed aggregation across separate datasets

One direct realization of the multi-set idea is the non-disclosed aggregated conformal predictor (NDCP), which considers training data split across $K$ non-disclosed sources $D_1,\dots,D_K$, each of arbitrary size, with no raw training data shared between the sources or with the aggregation location $A$ (Spjuth et al., 2018). In the binary setup of that work, the object space is $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$0, the label space is $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$1, and the example space is $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$2.

Each source applies transductive conformal prediction locally. The local predictor is Mondrian TCP with random forest. For a candidate label $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$3, the source appends the query point $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$4, forms $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$5, retrains the random forest on $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$6, and computes nonconformity scores using the random forest output $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$7: $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$8 This yields local conformal $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$9-values $p$0, and the local prediction region is

$p$1

Aggregation is then performed centrally by simple arithmetic averaging of the source-specific $p$2-values: $p$3 with final prediction set

$p$4

The privacy rationale is explicit: only the query object and the resulting $p$5-values are exchanged (Spjuth et al., 2018).

The paper distinguishes sharply between ordinary TCP validity and aggregated validity. For test set $p$6 of size $p$7, it evaluates the observed error rate

$p$8

and the global validity metric

$p$9

For pooled TCP and individual local TCPs, validity is maintained in the standard conformal sense. For NDCP, exact finite-sample validity is not claimed; instead, calibration plots show that the method is conservatively valid for significance levels roughly in the range $p$0 to $p$1, while being overall less valid than pooled or individual TCPs (Spjuth et al., 2018).

Efficiency is assessed by observed fuzziness,

$p$2

where lower values are better. On five UCI datasets—Spambase, Breast Cancer Wisconsin, Mushroom, First-order theorem proving, and Phishing Websites—with $p$3 training, $p$4 testing, and five repeated splits, pooled data is significantly more efficient than all other models. NDCP is less efficient than pooled, but the loss is described as “not so large” in absolute terms. NDCP also does not significantly improve efficiency over a single small TCP, yet it reduces variance, especially when the number of sources is $p$5 or $p$6. The experiments further report no significant difference in validity or efficiency between equally sized partitions (EqSrc) and randomly sized partitions (RandSrc), suggesting robustness to unequal source sizes (Spjuth et al., 2018).

3. Uniformly valid prediction over multiple heterogeneous source distributions

A more recent and theoretically stronger multi-source construction is Multi-Distribution Conformal Prediction (MDCP), developed for the case in which labeled data come from $p$7 heterogeneous source distributions $p$8, while the test point may come from any one of them or from any mixture

$p$9

(Yang et al., 6 Jan 2026). Each source-specific dataset

$p$0

is split into a training fold and a calibration fold; the training folds are used to learn source-specific score functions $p$1, which are then treated as fixed during source-wise calibration.

The target guarantee is uniform validity across all sources. The paper states that the conformal set must achieve at least $p$2 coverage no matter which source generated the test point, and proves the exact finite-sample guarantee

$p$3

for any mixture $p$4, which immediately implies source-wise uniform coverage (Yang et al., 6 Jan 2026).

The aggregation rule differs fundamentally from NDCP. Instead of averaging $p$5-values, MDCP forms a source-wise conformal $p$6-value $p$7 from each calibration split and aggregates by

$p$8

The final prediction set is

$p$9

which is exactly the union of the source-wise conformal sets: $(x_i,S_i)$0 A central clarification follows from this formulation: the construction is not a dual-set intersection; it is a multi-set union induced by max-$(x_i,S_i)$1 aggregation (Yang et al., 6 Jan 2026).

The paper’s main theoretical development is an efficiency characterization under uniform coverage constraints. At fixed $(x_i,S_i)$2, the conditional optimization problem is

$(x_i,S_i)$3

Its dual uses nonnegative multipliers $(x_i,S_i)$4, with threshold function

$(x_i,S_i)$5

and optimal set

$(x_i,S_i)$6

The dual objective is

$(x_i,S_i)$7

Complementary slackness yields a tightness statement: if $(x_i,S_i)$8, then source $(x_i,S_i)$9 is tight at $y_i \in S_i$0, coverage equals $y_i \in S_i$1; if $y_i \in S_i$2, then source $y_i \in S_i$3 is slack; and at least one source is tight (Yang et al., 6 Jan 2026).

Efficiency improvements are obtained by learning conformity scores that approximate the oracle dual threshold

$y_i \in S_i$4

The practical estimator is

$y_i \in S_i$5

after which every source uses the shared score

$y_i \in S_i$6

while calibration remains source-specific. The nonnegative weights $y_i \in S_i$7 are parameterized by splines or neural networks and enforced via softplus. Under consistency of $y_i \in S_i$8 and $y_i \in S_i$9, the MDCP set converges to the oracle optimal set up to boundary ambiguity (Yang et al., 6 Jan 2026).

Experimentally, the reported pattern is that single-source conformal sets often under-cover on out-of-source groups, naive max-$K$0 aggregation is valid but often overly conservative, and MDCP maintains near-tight worst-case coverage while substantially reducing set size relative to naive aggregation. In classification simulations, MDCP produced sets about $K$1 smaller than naive max-$K$2 aggregation, with much lower size variance; in regression simulations, it achieved about $K$3 smaller intervals. On FMoW, PovertyMap, and MEPS, the method is reported to improve the size-coverage tradeoff under source heterogeneity (Yang et al., 6 Jan 2026).

4. Two-layer hierarchical constructions and partial exchangeability

Another line of work addresses two-layer hierarchical models, where observations are exchangeable within groups but not across all individual datapoints. The model first draws group-specific distributions

$K$4

then conditional on $K$5, draws group-$K$6 observations iid: $K$7 The tasks include predicting $K$8 for a new group and predicting a new observation from an already observed group. The same framework also has a supervised version with $K$9 pairs and group-specific regression relationships (Dunn et al., 2018).

The reason standard conformal prediction fails here is explicit: the full collection $D_1,\dots,D_K$0 is generally not exchangeable because different groups come from different $D_1,\dots,D_K$1’s. The paper therefore constructs procedures that use whichever level of exchangeability remains available—between-group exchangeability of the latent distributions and within-group iid sampling (Dunn et al., 2018).

For unsupervised prediction of a new group, four constructions are developed. Double conformal is valid when all group sizes are equal, $D_1,\dots,D_K$2. It first builds within-group conformal intervals $D_1,\dots,D_K$3, then forms

$D_1,\dots,D_K$4

with

$D_1,\dots,D_K$5

and guarantee

$D_1,\dots,D_K$6

The method is valid but typically overcovers and gives large intervals; nontriviality requires roughly $D_1,\dots,D_K$7 and $D_1,\dots,D_K$8 (Dunn et al., 2018).

CDF pooling averages empirical CDFs across groups. With

$D_1,\dots,D_K$9

the pooled quantile is

$A$0

and the interval becomes

$A$1

If $A$2 has a continuous distribution, then

$A$3

This method uses all observations and typically gives the smallest intervals, but may undercover for small or moderate $A$4, especially if groups are unbalanced (Dunn et al., 2018).

Single subsampling draws one observation from each group, creating $A$5 iid draws from the mixture $A$6, after which ordinary conformal prediction can be applied. The interval

$A$7

with

$A$8

satisfies

$A$9

Its limitation is that it uses only one point per group and is therefore noisy and less reproducible (Dunn et al., 2018).

Repeated subsampling improves reproducibility by averaging over many subsamples. For $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$00, the method samples one observation from each group and constructs a conformal $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$01-value $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$02. The set

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$03

satisfies

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$04

Although the formal guarantee is $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$05, the simulations reported coverage usually close to $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$06, with similar average set size to single subsampling and more stable behavior across repeated runs (Dunn et al., 2018).

The supervised case mirrors these constructions through residual-based calibration. CDF pooling on residuals is asymptotically valid if $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$07 is continuous; single subsampling satisfies

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$08

repeated subsampling satisfies

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$09

For existing-group prediction in the unsupervised setting, ordinary conformal prediction applied only to the target group gives exact validity, while a James–Stein shrinkage version preserves exact validity and can produce smaller intervals, especially when within-group variance is large relative to between-group variance (Dunn et al., 2018).

5. Conformal prediction with set-valued training and calibration labels

A different interpretation of “dual-set” arises when the training and calibration data themselves are set-valued. In partial label learning or superset learning, the observed sample is

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$10

with the true label satisfying $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$11. The conformal objective is then inverted relative to the standard setting: training data are imprecise, but the conformal predictor still outputs a set-valued prediction (Javanmardi et al., 2023).

The procedure begins by splitting partially labeled observations into proper training and calibration subsets,

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$12

training a probabilistic classifier with a partial label learning algorithm,

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$13

and calibrating a threshold using a set-derived score multiset $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$14. For any such $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$15, the threshold is the

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$16

and the prediction set is

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$17

Standard conformal prediction is recovered in the special case $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$18 (Javanmardi et al., 2023).

The paper develops several set-based score constructions. The pessimistic or max score is

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$19

Using all candidate-label scores gives

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$20

and the mean score is

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$21

Two natural baselines are also defined: $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$22 and

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$23

Validity results depend on the score construction. If the hidden precise calibration labels together with the test point are exchangeable, then the prediction set based on $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$24 satisfies

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$25

For $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$26, validity requires additional conditions: $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$27 and

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$28

For $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$29, validity holds if

$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$30

A supporting lemma shows that if enough added scores lie to the right of the original threshold, then the threshold cannot decrease, which underpins the $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$31 theorem (Javanmardi et al., 2023).

The empirical study uses synthetic partial-label versions of MNIST, KMNIST, and Fashion-MNIST, and real partial-label datasets BirdSong, Lost, MSRCv2, Soccer Player, and Yahoo!News. The reported pattern is a validity-efficiency tradeoff. $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$32 is always valid but inefficient, with coverage essentially always $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$33 or very near it and often huge prediction sets. $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$34 remains near or at the target level in many settings and is typically more efficient than $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$35. $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$36 often gives dramatically smaller prediction sets, but coverage degrades on instance-dependent contamination and real datasets. $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$37 produces the smallest sets but undercovers, while $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$38 interpolates between optimistic and pessimistic extremes without a universal guarantee (Javanmardi et al., 2023).

6. Comparative structure, guarantees, and recurring misconceptions

Taken together, these works reveal several distinct mechanisms for combining more than one set of conformal evidence. The specific aggregation operator is not incidental; it determines the form of the validity argument and the typical efficiency profile.

Setting	Combination rule	Coverage status
Non-disclosed multi-source TCP	Average source-wise $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$39-values	Conservatively valid in the significance interval $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$40 to $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$41
Multi-distribution robust CP	Max over source-wise $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$42-values; equivalent to union of source sets	Finite-sample valid for any mixture of the source distributions
Hierarchical repeated subsampling	Average subsample conformal $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$43-values	$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$44 guarantee
Partial-label CP	Transform candidate-label sets into calibration scores	Validity depends on the score construction

A first recurring theme is the tension between pooling and structural constraints. Pooled-data TCP is significantly more efficient than distributed alternatives in the non-disclosed setting, but it requires data sharing that NDCP is designed to avoid (Spjuth et al., 2018). CDF pooling in the hierarchical setting typically gives the smallest intervals, but only asymptotic validity is available and undercoverage may occur for small or moderate $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$45 (Dunn et al., 2018). In partial-label learning, aggressive score constructions can reduce set size, but robust validity is concentrated in the more conservative constructions (Javanmardi et al., 2023). In multi-distribution robustness, naive max-$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$46 aggregation is valid but often overly conservative, motivating score learning through the dual program (Yang et al., 6 Jan 2026).

A second theme is that exact finite-sample validity does not survive every aggregation rule. Source-wise conformal validity plus max-$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$47 aggregation yields exact worst-case coverage across heterogeneous sources (Yang et al., 6 Jan 2026). By contrast, arithmetic averaging of source-specific $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$48-values in NDCP is only empirically conservative, not exactly valid in the standard conformal sense (Spjuth et al., 2018). Repeated subsampling in hierarchical models achieves a formal $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$49 guarantee rather than $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$50 (Dunn et al., 2018). This suggests that “combining conformal predictors” is not itself a guarantee-preserving operation; the combination rule must be justified case by case.

A third theme concerns the meaning of “dual” in “dual-set conformal prediction.” One possible misconception is that the phrase denotes an intersection-based rule over two conformal sets. The surveyed literature does not support that as a general characterization. In MDCP, the operative set combination is a union induced by max-$\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$51 aggregation, and the “dual” aspect appears in the optimization program used to learn an efficient shared score (Yang et al., 6 Jan 2026). In NDCP, the defining operation is arithmetic averaging over source-wise $\Gamma^\epsilon = \{y \mid p_y > \epsilon\},$52-values rather than set intersection (Spjuth et al., 2018). In hierarchical and partial-label settings, the key issue is not geometric combination of two sets, but rather conformal calibration under partial exchangeability or set-valued supervision (Dunn et al., 2018, Javanmardi et al., 2023).

The literature therefore supports a narrower conclusion: “dual-set conformal prediction” is best viewed as a descriptive label for conformal prediction under multiple sets of evidence, not as a settled term for a single formal framework. The technically meaningful distinctions lie in the data-generating structure, the aggregation primitive, and the form of the coverage guarantee.