Conformal Prediction Procedures

• Conformal prediction procedures are nonparametric algorithms that construct set-valued predictions with finite-sample frequentist coverage guarantees.
• They use nonconformity scores and robust calibration methods—marginal and PAC—to ensure prediction reliability under minimal assumptions.
• Advanced techniques optimize structured prediction sets via integer programming and graph-based representations for complex tasks.

Conformal prediction procedures are a class of nonparametric algorithms that construct set-valued predictions—prediction intervals or sets—for predictive models while offering finite-sample frequentist coverage guarantees under minimal distributional assumptions. The foundation of conformal prediction is the use of a nonconformity score, typically derived from a base model, and an exchangeable calibration step which ensures that the probability of the true label falling outside the predicted set does not exceed a prespecified miscoverage rate, regardless of the model’s correctness. These procedures can be specialized for classical problems (regression, classification), structured outputs (e.g., hierarchies or intervals), and even for data with missing or partially observed labels.

1. Classical and Structured Conformal Prediction

A standard conformal prediction workflow consists of:

• A pre-trained scoring function $$g: \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}$$ (such as class probability or an unnormalized model score).
• Definition of prediction sets at threshold $\tau$:

$$C_\tau(x) = \{ y \in \mathcal{Y} : g(x, y) \geq \tau \}$$

• Calibration of $\tau$ via a hold-out calibration set $\mathcal{Z}$ to ensure that, for a target error $\epsilon$, the marginal coverage

$$\mathbb{P}_{\mathcal{Z},(X,Y)\sim D}[ Y \in C_{\hat{\tau}(\mathcal{Z})}(X) ] \geq 1 - \epsilon$$

is achieved.

The recent extension to structured prediction generalizes this paradigm. Instead of working with power set $2^{\mathcal{Y}}$, conformal prediction considers a more interpretable family $\hat{\mathcal{Y}}$ of structured sets (e.g., intervals, hierarchical subtrees) with an associated mapping $$\gamma: \hat{\mathcal{Y}} \rightarrow 2^{\mathcal{Y}}$$ representing the subset of labels implied by the structure, as well as a size function $\sigma: \hat{\mathcal{Y}} \rightarrow \mathbb{R}_+$. The conformal set at threshold $\tau$ is

$$h_\tau(x) \in \underset{\hat{y} \in \hat{\mathcal{Y}}}{\arg\min} \ \sigma(\hat{y}) \quad \text{s.t.} \quad \sum_{y \in \mathcal{Y}} g(x, y) \cdot 1[y \in \gamma(\hat{y})] \geq \tau.$$

This yields the smallest structured set whose cumulative score exceeds $\tau$, accommodating interpretable outputs for complex tasks (Zhang et al., 8 Oct 2024).

2. Calibration Algorithms and Coverage Guarantees

Two main calibration strategies are used:

• Marginal calibration:

For a calibration set $Z = \{(x_i, y_i)\}_{i=1}^n$ and candidate threshold $\tau$, define

$$\varphi_{\mathrm{marginal}}^\epsilon(Z, \tau) = 1 \text{ if } \sum_{(x,y^*) \in Z} 1[y^* \notin \gamma(h_\tau(x))] \leq \lfloor (n+1) \epsilon \rfloor.$$

A sequential search returns the largest $\tau$ accepted by the test, ensuring marginal miscoverage $\leq \epsilon$ for any data distribution.

• PAC (Probably Approximately Correct) calibration:

For specified $\epsilon, \delta$, derive the largest integer $\ell$ such that the binomial CDF $F(\ell; n, \epsilon) < \delta$. The PAC test is

$$\varphi_{\mathrm{PAC}}^{\epsilon, \delta}(Z, \tau) = 1 \text{ if } \sum_{i=1}^n 1[y_i \notin \gamma(h_\tau(x_i))] \leq \ell.$$

The resulting prediction set satisfies

$$\mathbb{P}_{Z \sim D^n}\left[ \mathbb{P}_{(X,Y)\sim D}[ Y \in \gamma(h_{\hat{\tau}(Z)}(X)) ] \geq 1-\epsilon \right] \geq 1-\delta.$$

These theorems provide rigorous control of coverage under exchangeability, with a learn-then-test logic (Zhang et al., 8 Oct 2024).

3. Implicit Representation and Optimization via Graph Structures

For structured outputs, prediction sets are often best represented implicitly via the structure of a directed acyclic graph (DAG) or hierarchy:

• Each node corresponds to a structured label (e.g., a hierarchical class or an interval).
• The prediction set is a subset of nodes, with leaves mapping to fine-grained labels.
• The optimization to find $h_\tau(x)$ is formulated as an integer program:
  • Minimize the size of the structured set,
  • Subject to a constraint on cumulative model score,
  • Ensuring that the structured set captures all descendants in the label space.

Example: In hierarchical image classification (ImageNet + WordNet), internal nodes represent coarse categories; selecting a node implicitly includes all its leaf descendants in the prediction set, allowing the construction of small, interpretable conformal prediction regions (Zhang et al., 8 Oct 2024).

4. Empirical Results and Interpretability Trade-offs

Structured conformal prediction has been empirically evaluated on tasks such as:

• MNIST digit strings (interpretable via confident prefixes),
• Hierarchical image classification (prediction sets as sets of hierarchical labels),
• Interval-based question answering (calendar years).

Key empirical findings include:

• Empirical coverage matches or exceeds nominal levels (marginal or PAC, as prescribed).
• Increasing $\epsilon$ (desired error), $\delta$ (PAC confidence), or $m$ (allowed set size) results in smaller average prediction sets—a direct trade-off between efficiency and conservatism.
• Structured prediction sets (e.g., unions of intervals or coarse hierarchical categories) are more interpretable and concise than unstructured lists of labels or atomic answers (Zhang et al., 8 Oct 2024).

5. Assumptions and Computational Considerations

The coverage guarantees in both classical and structured conformal prediction rest on minimal assumptions:

• Data (calibration and test) must be i.i.d. (or exchangeable).
• The model scoring function $g$ can be arbitrary; however, probabilistic scores (e.g., true probabilities) generally yield more efficient sets.
• A finite grid of candidate thresholds $T$ must be specified for threshold search.
• Solving the underlying integer program (for structured sets represented as subgraphs or intervals) can be computationally expensive for large $m$ or DAG size, limiting scalability.
• The method requires an explicit, correct mapping $\gamma: \hat{\mathcal{Y}} \to 2^{\mathcal{Y}}$.
• PAC guarantees require correct binomial parameterization (Zhang et al., 8 Oct 2024).

6. Extensions and Limitations

Structured conformal prediction extends classical conformal methods to arbitrary interpretable set families, including but not limited to hierarchical, interval, and prefix-encoded search spaces. This abstraction enables practical set-valued prediction for scenarios where the complexity or cardinality of the label space would render classical unstructured conformal sets uninterpretable or too large.

Limitations are primarily computational (for large or complex structures), reliance on the expressiveness and faithfulness of $\gamma$, and the non-adaptivity of the framework to settings with dependent data or additional structural constraints not captured in the DAG or interval encoding. PAC coverage, while stronger, may be less interpretable than marginal guarantees for end-users (Zhang et al., 8 Oct 2024).

---

References:

• "Conformal Structured Prediction" (Zhang et al., 8 Oct 2024)