Post-hoc L2D: Density-Ratio Deferral Framework

• The paper introduces a post-hoc L2D framework that employs density‐ratio estimation to compare fixed model and expert ideal distributions via a learned scalar scorer.
• It explains how the density‐ratio formulation recovers classical Chow reject rules and adapts to expert-aware conservatism under data shift and fairness constraints.
• Empirical evaluations demonstrate that DR CPE methods achieve competitive accuracy and enhanced robustness on benchmarks like CIFAR-100 and specialized medical datasets.

Post-hoc Learning to Defer (L2D) is a human–AI decision framework in which a predictor is fixed first and a separate post-processing mechanism is then used to decide whether the predictor should act or defer to an expert on a given input. In the density-ratio formulation of post-hoc L2D, deferral is defined by comparing divergence-regularized “ideal distributions” for the model and the expert, and the comparison is operationalized through a learned scalar scorer whose threshold can be adjusted at test time without retraining (Soen et al., 19 May 2026). This formulation places post-hoc deferral in direct relation to Chow-style reject rules, expert-aware posterior reweighting, and density-ratio estimation, while adjacent post-hoc lines of work study training-free multiple-expert routing, partial deferral for sequence outputs, and constrained post-processing under fairness or budget requirements (Bary et al., 16 Sep 2025).

1. Formal setting and scope

In the framework developed for density-ratio-based post-hoc L2D, the starting point is a data distribution

$P \in \Delta(\X\times\Y), \qquad Y\in\{1,\dots,L\},$

with decomposition

$$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$

A model–loss pair $(h,\ell)$ is compared against an expert–loss pair $(h^{(E)},\ell)$. The deferral decision is not built into the predictor itself; instead, a post-hoc scorer is trained after the predictor has been fixed, and deferral is produced by thresholding that scorer (Soen et al., 19 May 2026).

This post-hoc separation between prediction and deferral also appears in other variants of L2D. For sequences, a frozen autoregressive predictor $f$ is paired with either a token-level rejector or a one-time rejector learned afterward, rather than retraining the underlying sequence model (Rayan et al., 3 Feb 2025). In constrained L2D, the post-hoc object can be a simplex-valued decision rule estimated from hold-out data to satisfy constraints such as expert-budget or demographic-parity requirements (Charusaie et al., 2024). A training-free variant goes further by eliminating rejector training entirely and calibrating deferral from conformal prediction sets and expert confusion data (Bary et al., 16 Sep 2025).

The central scope of post-hoc L2D is therefore narrower than generic abstention and broader than a single reject-rule family. It includes learned scorers, calibrated set-based routing, and plug-in post-processing rules, provided the predictor itself is treated as fixed and the defer decision is imposed afterward.

2. Ideal distributions and deferral by density ratio

The density-ratio approach defines deferral through two divergence-regularized reweightings of the data distribution: one for the model and one for the expert. For a convex divergence $D$ and temperature $\gamma>0$, the model’s marginal ideal distribution on $\X$ is

$Q_{\mtype} = \arg\min_{Q\in\Delta(\X)} \Bigl\{ \E_{X\sim Q}\bigl[\E_{Y\sim\eta(X)}[\ell(X,Y,h(X))]\bigr] +\gamma\,D(Q\Vert P_{\rm x}) \Bigr\},$

while the joint ideal distribution is first defined on $\X\times\Y$,

$$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$0

and then marginalized as

$$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$1

Exactly analogous distributions $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$2 and $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$3 are defined for the expert (Soen et al., 19 May 2026).

Deferral is then encoded by the density ratio between the model’s and the expert’s ideals: $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$4 Under this construction, an input is deferred when it is sufficiently more compatible with the expert’s ideal distribution than with the model’s.

For $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$5, the joint ideals admit closed-form multiplicative weights. In particular,

$$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$6

This makes the model–expert comparison a comparison of reweighted local loss profiles rather than raw predictive confidence (Soen et al., 19 May 2026).

A common simplification is to treat post-hoc deferral as nothing more than thresholding confidence scores. The ideal-distribution formulation is stricter: the threshold is applied to a density ratio induced by low-loss reweightings of the original distribution, so the rule is explicitly relative to both model and expert performance.

3. DR CPE losses and the learned scorer

The operational step in the density-ratio framework is a reduction from density-ratio estimation (DRE) to class-probability estimation (CPE). Given distributions $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$7 and $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$8, a binary CPE problem is defined using a strictly proper composite loss $$P(x,y)=P_{\rm x}(x)\,\eta_y(x), \qquad \eta_y(x)=P[Y=y\mid X=x].$$9 with link $(h,\ell)$0: $(h,\ell)$1 At optimum,

$(h,\ell)$2

which yields

$(h,\ell)$3

Since deferral only requires thresholding the density ratio, it is sufficient to threshold the scorer: $(h,\ell)$4 In post-hoc L2D, $(h,\ell)$5 and $(h,\ell)$6 are specialized to the model and expert ideal distributions $(h,\ell)$7 and $(h,\ell)$8 (Soen et al., 19 May 2026).

By choosing

$(h,\ell)$9

the normalizers cancel, which yields the joint-ideal DR CPE loss

$(h^{(E)},\ell)$0

A corresponding marginal loss $(h^{(E)},\ell)$1 can also be derived, although it is described as harder to estimate. By Jensen’s inequality,

$(h^{(E)},\ell)$2

and under mild bounded-loss conditions their difference scales like the label-conditional loss-variance (Theorem 4.6) (Soen et al., 19 May 2026).

Once trained, the scorer is turned into a deferral policy by simple thresholding: $(h^{(E)},\ell)$3 The threshold $(h^{(E)},\ell)$4 can be chosen on a validation split to hit a desired deferral rate. This modularity is a defining property of the post-hoc approach: a single scalar scorer is learned once, and operating points are changed by varying the threshold rather than retraining.

4. Recovery of Chow’s rule and expert-tilted conservatism

For KL-based ideal distributions with equal temperatures, the marginal density-ratio rule recovers the classical Chow rule exactly. Specifically, when $(h^{(E)},\ell)$5 and $(h^{(E)},\ell)$6,

$(h^{(E)},\ell)$7

Thresholding at $(h^{(E)},\ell)$8 is then equivalent to a threshold on the expected loss difference, and choosing $(h^{(E)},\ell)$9 gives

$f$0

which is exactly the Chow rule (Soen et al., 19 May 2026).

The joint rule behaves differently. Its rejection region is contained in that of Chow’s rule under a reweighted distribution $f$1 with posterior

$f$2

Joint-ideal thresholding at $f$3 implies

$f$4

so deferral occurs whenever Chow’s rule under the expert-tilted distribution would defer. The paper characterizes this joint rule as more conservative, because it incorporates the expert’s per-label loss via tilting rather than only comparing expected losses under the original posterior (Soen et al., 19 May 2026).

This distinction is central to the interpretation of post-hoc L2D under distribution shift. The marginal construction matches Chow optimality in the unshifted KL setting; the joint construction modifies the effective posterior according to expert competence, which suggests a principled route to expert-aware conservatism when the label geometry of difficult examples matters.

5. Empirical methodology and observed behavior

The empirical study for density-ratio-based post-hoc L2D evaluates prediction accuracy versus deferral-rate curves and robustness across clean versus corrupted splits. The datasets and corruptions comprise CIFAR-100 model-cascade settings such as ResNet8→14, 14→56, and 8→32, together with PathMNIST and DermaMNIST variants featuring label noise on the first $f$5 classes, long-tail subsampling, and specialist-only subsampling such as melanocytic versus non-melanocytic cases (Soen et al., 19 May 2026).

The baselines are: confidence-thresholding without training; a two-stage surrogate with exp-loss from Mao et al. 2023; and expert-comparison regressions, namely Estimate-Diff01, which regresses $f$6, and Estimate-MaxProb, which regresses $f$7. The key findings are that all methods compete on clean data, whereas under corruption DR CPE is consistently among the top two and degrades least; joint DR CPE recovers Chow optimality when there is no shift and yields an expert-aware tilt under shift; and DR CPE handles extreme specializations, including PathMNIST specialists, far more robustly than the two-stage surrogate or naive confidence thresholding (Soen et al., 19 May 2026).

These results position post-hoc density-ratio learning as a robustness-oriented alternative to baselines that either ignore expert structure or encode expert comparison through direct regression targets. A plausible implication is that the ideal-distribution reweighting acts as a regularized comparison mechanism across heterogeneous dataset conditions, especially when the expert is specialized or the data are corrupted.

6. Broader post-hoc L2D landscape

The density-ratio perspective explicitly connects post-hoc L2D to hypothesis testing and anomaly detection. Thresholding $f$8 is a likelihood-ratio test between the model’s ideal and the expert’s ideal, in the spirit of Neyman–Pearson testing. The same paper notes that classical anomaly detection thresholds $f$9 against a nominal measure, so the distinction between “typical” and “anomalous” can be reinterpreted here as a distinction between the model’s ideal and the expert’s ideal (Soen et al., 19 May 2026). It also identifies multi-expert deferral as a multi-source DRE/CPE problem and points to DRO/GVI-style regularized extensions involving fairness or budget constraints.

Other post-hoc lines of work make these adjacent directions concrete. A training-free multiple-expert framework based on conformal prediction constructs a prediction set

$D$0

accepts singleton sets, and otherwise defers to the expert with highest segregativity score restricted to $D$1. It provides the marginal coverage guarantee

$D$2

under exchangeability, requires no retraining when experts join or leave the pool, and reports $D$3 accuracy on CIFAR10-H and $D$4 on ImageNet16-H while reducing expert workload by up to a factor of $D$5 (Bary et al., 16 Sep 2025).

For structured prediction, post-hoc L2D has been generalized from whole-output rejection to partial deferral for sequences. In that setting, a frozen predictor is paired either with a token-level rejector $D$6, which defers specific token predictions, or with a one-time rejector $D$7, which defers the remainder of the sequence from a selected point onward. The paper proves Bayes consistency of surrogate objectives for both settings and reports better cost-accuracy tradeoffs than whole deferrals on Traveling Salesman solvers and News summarization models (Rayan et al., 3 Feb 2025).

A separate post-processing framework addresses multi-objective L2D under constraints. Using a $D$8-dimensional generalization to the fundamental lemma of Neyman and Pearson, it derives a Bayes-optimal simplex-valued rule

$D$9

which can encode expert-budget constraints, demographic parity, equality of opportunity, equalized odds, OOD deferral, long-tail coverage, and Type-$\gamma>0$0 error bounds. On COMPAS, the reported accuracy/parity-gap/deferral-rate triple is $\gamma>0$1, compared with $\gamma>0$2 and $\gamma>0$3 for the listed baselines (Charusaie et al., 2024).

Taken together, these works show that post-hoc L2D is not a single algorithmic recipe but a family of post-processing strategies with different invariances and optimization targets. Density-ratio scoring emphasizes model–expert comparison through ideal distributions; conformal routing emphasizes marginal coverage and plug-and-play expert sets; partial sequence deferral emphasizes granularity of intervention; and constrained post-processing emphasizes Bayes-optimality under side conditions. The common structure is that the predictor is treated as given, while the defer decision is learned, calibrated, or optimized afterward.