Learning-to-Defer with Advice

• Learning-to-Defer with Advice is a framework where models jointly choose an expert and an advice action to optimize decision-making.
• The approach extends classical rejection learning by dynamically assessing the cost–benefit tradeoffs of acquiring expert-specific contextual information.
• It improves performance and fairness through composite action optimization, adapting to sequential decision and reinforcement learning scenarios.

Learning-to-Defer with Advice denotes a family of decision-theoretic frameworks in which a learner does not merely decide whether to act autonomously or hand off to an expert; it also exploits additional information tied to that handoff, such as expert-conditioned retrieved context, competence priors, small context sets describing the currently available expert, or multiple expert opinions. In classical learn-to-defer, routing is defined over fixed experts with fixed information. In the expert-conditioned formulation, the system selects both an expert $j$ and an advice action $k\in[K]_0$ before the realized advice is revealed, so the executed decision is the composite pair $(j,k)$ rather than routing alone (Madras et al., 2017, Montreuil et al., 15 Mar 2026).

1. From rejection learning to advice-aware deferral

Classical learn-to-defer was introduced as a generalization of rejection learning in which the model accounts for the quality of the downstream decision-maker rather than treating abstention as a fixed penalty. In the original two-stage formulation, the model prediction $\hat Y_M=f(X)$, the downstream decision-maker prediction $\hat Y_D=h(X,Z)$, and the defer indicator $s=g(X)\in\{0,1\}$ produce the system output

$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$

with system-level loss

$$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$

This formulation makes deferral adaptive to the downstream agent’s realized loss, and rejection learning is recovered when the downstream decision-maker has constant loss (Madras et al., 2017).

That classical formulation is routing-only. Later multi-objective work retained the same routing viewpoint while adding fairness, intervention-budget, anomaly-deferral, and error-control constraints. In that line, the Bayes-optimal constrained policy is characterized by an argmax over a Lagrangian-adjusted score vector of the form

$\psi_0(x)-\sum_i k_i\psi_i(x),$

but the collaboration protocol still consists only of “predict yourself” or “defer to the human.” It does not model recommendation content, explanations, confidence messages, or how advice changes the human’s final action (Charusaie et al., 2024).

The conceptual shift in advice-aware deferral is therefore not merely the presence of an external expert. It is the relaxation of the fixed-information assumption: after selecting an expert, the system may also choose what additional information that expert should receive. A plausible implication is that the statistical target changes from pure routing over experts to joint optimization over expert–information pairs.

2. Formalizing expert-conditioned advice

The most explicit formalization models advice as a post-routing but pre-execution information-acquisition decision. Let $X$ be the input, $k\in[K]_0$0 the target, and $k\in[K]_0$1 a vector of advice sources. The learner selects an expert $k\in[K]_0$2 and an advice action $k\in[K]_0$3, where $k\in[K]_0$4 means “no advice.” The selected advice source is revealed only after that choice. A masking operator $k\in[K]_0$5 keeps only the chosen advice coordinate and replaces the others with $k\in[K]_0$6, so the chosen expert receives $k\in[K]_0$7 and masked advice $k\in[K]_0$8 (Montreuil et al., 15 Mar 2026).

The realized cost of executing expert $k\in[K]_0$9 under advice action $(j,k)$0 is

$(j,k)$1

where $(j,k)$2 is expert $(j,k)$3’s prediction rule, $(j,k)$4 is the task loss, $(j,k)$5 is the expert fee, and $(j,k)$6 is the advice-acquisition fee with $(j,k)$7. A sequential policy can be written as a router $(j,k)$8 and an expert-conditional query function $(j,k)$9. The true loss is

$\hat Y_M=f(X)$0

The Bayes-optimal policy first computes, for each expert, the advice action minimizing conditional expected cost,

$\hat Y_M=f(X)$1

and then routes to the expert whose best advice-conditioned conditional risk is smallest,

$\hat Y_M=f(X)$2

Equivalently, one can define the composite action space

$\hat Y_M=f(X)$3

and view the executed decision as a single composite action $\hat Y_M=f(X)$4 (Montreuil et al., 15 Mar 2026).

This formalization yields a precise condition for when advice is worth acquiring. Under the additive cost model, $\hat Y_M=f(X)$5 iff there exists $\hat Y_M=f(X)$6 such that the expected reduction in task loss from revealing advice source $\hat Y_M=f(X)$7 exceeds its acquisition cost $\hat Y_M=f(X)$8. This directly distinguishes advice-aware deferral from ordinary L2D: the relevant comparison is no longer “which expert is best,” but “which expert is best under its own optimal advice policy.”

3. Surrogate design and statistical consistency

A central technical result is that natural separated surrogates are misaligned with the true protocol. In the smallest non-trivial setting with $\hat Y_M=f(X)$9 experts and $\hat Y_D=h(X,Z)$0 binary advice source, separated parameterizations use one score for routing and one advice score per expert. A broad family of such surrogates can be written so that each expert’s advice-conditioned row of costs is first profiled into a scalar summary $\hat Y_D=h(X,Z)$1. The difficulty is that Bayes compares row minima, whereas the separated surrogate compares these profiled summaries. The paper proves Fisher inconsistency by constructing a pointwise cost table

$\hat Y_D=h(X,Z)$2

for which Bayes chooses expert $\hat Y_D=h(X,Z)$3 because $\hat Y_D=h(X,Z)$4, but the unique minimizer of the separated surrogate decodes to expert $\hat Y_D=h(X,Z)$5 because the catastrophic entry $\hat Y_D=h(X,Z)$6 distorts the row summary (Montreuil et al., 15 Mar 2026).

The proposed remedy is to abandon route-then-query factorization and optimize directly over the composite action space $\hat Y_D=h(X,Z)$7. The true loss admits a mismatch decomposition

$\hat Y_D=h(X,Z)$8

with weights

$\hat Y_D=h(X,Z)$9

This reduces advice-aware deferral to a weighted multiclass problem over composite actions. The augmented surrogate is then

$s=g(X)\in\{0,1\}$0

where $s=g(X)\in\{0,1\}$1 is the comp-sum multiclass surrogate; for $s=g(X)\in\{0,1\}$2, it becomes the usual log-softmax form (Montreuil et al., 15 Mar 2026).

The resulting theory is $s=g(X)\in\{0,1\}$3-consistent. The excess true risk is bounded by a transfer function applied to the excess augmented-surrogate risk, with $s=g(X)\in\{0,1\}$4 in the log-softmax case. In the unrestricted class, the minimizability gaps vanish, yielding recovery of the Bayes-optimal deferral-advice policy in the limit (Montreuil et al., 15 Mar 2026). The methodological consequence is exact: once advice acquisition is expert-dependent, the correct prediction target is the executed expert–advice pair, not separate routing and query heads.

4. Broader notions of advice: expert side information, competence priors, and expert sets

The literature also uses “advice” in broader senses than expert-conditioned retrieved context. One major strand studies side information about the expert rather than additional task-specific evidence. In learning-to-defer to a population, the currently available expert is represented by a small context set

$s=g(X)\in\{0,1\}$5

and the rejector becomes expert-conditional,

$s=g(X)\in\{0,1\}$6

Optimization-based and model-based meta-learning variants adapt the rejector to unseen experts at test time, and attention over context points estimates local expert competence $s=g(X)\in\{0,1\}$7 for the current query (Tailor et al., 2024).

A second strand makes expert competence itself the advice channel. Expert-Agnostic Learning to Defer represents expert $s=g(X)\in\{0,1\}$8 by classwise posterior means

$s=g(X)\in\{0,1\}$9

derived from a Beta-Binomial model over sparse context predictions. Optional self-assessed accuracies $\hat Y=(1-s)\hat Y_M+s\hat Y_D,$0 and confidences $\hat Y=(1-s)\hat Y_M+s\hat Y_D,$1 are converted into Beta priors, so prior beliefs about competence enter the rejector in a principled Bayesian way. Across HAM10000, Blood Cells, Retinal OCT, and Liver Tumours, this yields up to a $\hat Y=(1-s)\hat Y_M+s\hat Y_D,$2 relative improvement on unseen experts while matching or exceeding state-of-the-art performance on seen experts (Strong et al., 14 Feb 2025).

A third strand treats advice as a set of expert opinions rather than a single handoff. Top-$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$3 L2D defines the rejector set

$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$4

and incurs summed cost over the selected committee,

$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$5

Top-$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$6 further learns the number of consulted experts per input via a selector $\hat Y=(1-s)\hat Y_M+s\hat Y_D,$7, with $\hat Y=(1-s)\hat Y_M+s\hat Y_D,$8, and supports majority vote, weighted vote, uniform average, and weighted average as downstream combination rules (Montreuil et al., 17 Apr 2025). The related one-stage Top-$\hat Y=(1-s)\hat Y_M+s\hat Y_D,$9 framework unifies labels and experts in a single entity set and learns one score model that can be reused across Top-$$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$0 regimes without retraining; on CIFAR-10, adaptive Top-$$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$1 reaches $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$2 majority-voting accuracy at budget $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$3, surpassing the best fixed Top-$$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$4 result of $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$5 at the higher budget $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$6 (Montreuil et al., 15 May 2025).

Taken together, these lines suggest that “advice” in L2D has at least three technically distinct meanings: additional information delivered to the chosen expert, side information about expert competence, and jointly queried outputs from multiple experts.

5. Sequential and reinforcement-learning variants

In sequential settings, advice and deferral interact with dynamics, adherence, and intervention costs. A direct sequential formulation augments the machine action space to

$$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$7

so the machine can either advise an action $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$8 or choose defer, meaning “temporarily refrain from giving advice and trust the human’s default behavior.” Human adherence is modeled by a state- and advice-dependent probability $$\mathcal L_{\mathrm{defer}}=-\sum_i\left[(1-s_i)\ell(Y_i,\hat Y_{M,i})+s_i\ell(Y_i,\hat Y_{D,i})\right].$$9, and the induced machine MDP uses transitions

$\psi_0(x)-\sum_i k_i\psi_i(x),$0

Selective advising is induced by a penalized reward

$\psi_0(x)-\sum_i k_i\psi_i(x),$1

and the optimal policy advises only at “critical time stamps,” formalized by the proposition that if $\psi_0(x)-\sum_i k_i\psi_i(x),$2, then $\psi_0(x)-\sum_i k_i\psi_i(x),$3 (Chen et al., 2023).

Reinforcement-learning work on action advising studies the same defer-or-act tradeoff under a consultation budget. One line uses a student policy $\psi_0(x)-\sum_i k_i\psi_i(x),$4, a teacher policy $\psi_0(x)-\sum_i k_i\psi_i(x),$5, a finite budget $\psi_0(x)-\sum_i k_i\psi_i(x),$6, and a learned teacher-imitation model. In “Methodical Advice Collection and Reuse in Deep Reinforcement Learning,” the student queries the teacher when student uncertainty $\psi_0(x)-\sum_i k_i\psi_i(x),$7 exceeds an adaptive threshold $\psi_0(x)-\sum_i k_i\psi_i(x),$8, and reuses the imitation model when its uncertainty $\psi_0(x)-\sum_i k_i\psi_i(x),$9 is below threshold $X$0, yielding a three-way routing rule among teacher, surrogate teacher, and student policy (Sahir et al., 2022). “Learning on a Budget via Teacher Imitation” similarly combines budgeted teacher queries with a behavioral cloning model

$X$1

and automatically tunes the uncertainty threshold $X$2 from the percentile of correctly imitated advised states, so that direct teacher advice and reused advice are both gated by imitation uncertainty (Ilhan et al., 2021).

Not all advice-based RL papers are L2D in the strict sense. “Teachable Reinforcement Learning via Advice Distillation” formalizes a coaching channel in a coaching-augmented MDP and learns an advice-conditioned surrogate policy $X$3, later distilling it into an advice-free policy $X$4. However, it contains no explicit abstention, delegation, query action, or switching-control operator, and the deployment target is autonomous rather than advice-reliant (Watkins et al., 2022). Likewise, sequential medical deferral via model-based RL learns a pre-emptive defer policy to an expert under non-stationary dynamics, but the expert is an alternative controller rather than an advice channel (Joshi et al., 2021). These distinctions delimit the sequential scope of advice-aware deferral.

6. Applications, boundaries, and unresolved issues

Empirically, the strongest direct evidence for expert-conditioned advice comes from FEVER retrieval, tabular fraud escalation, and CLIP-based prompt escalation. In FEVER, the augmented expert-conditioned-advice surrogate improves over standard L2D in $X$5 of $X$6 advice-cost regimes and converges to ordinary L2D behavior when advice becomes too expensive (Montreuil et al., 15 Mar 2026). In competence-aware unseen-expert deferral, Bayesian competence priors materially help when they are accurate, but inaccurate informative priors degrade performance, demonstrating that advice about expert quality is useful only when it is itself reliable (Strong et al., 14 Feb 2025).

Several boundary cases are now clear. Constrained post-processing frameworks solve the routing layer under fairness, expert-intervention budgets, and anomaly-deferral constraints, but they do not model recommendation content or how advice affects the human’s action (Charusaie et al., 2024). Advice-distillation methods internalize coaching signals so that deferral is no longer needed at deployment, which makes them adjacent mechanisms rather than direct instances of learning-to-defer with advice (Watkins et al., 2022).

The main technical limitations are also well specified. Direct expert-conditioned-advice training assumes full-information supervision over all expert–advice pairs $X$7 for each training example, and the composite action space grows as $X$8 (Montreuil et al., 15 Mar 2026). Context-based unseen-expert methods assume access to a representative context set for the current expert (Tailor et al., 2024). Bayesian competence-prior methods inherit the calibration problems of self-assessed expertise (Strong et al., 14 Feb 2025). Top-$X$9 formulations depend on the choice and calibration of the aggregation rule after expert selection (Montreuil et al., 17 Apr 2025, Montreuil et al., 15 May 2025).

This suggests three open directions. First, partial-feedback and bandit formulations are needed for settings where only the executed expert–advice pair is observed. Second, richer advice channels—retrievals, tool outputs, rationales, demographic metadata, escalation summaries, or multimodal context—require models of how advice changes expert behavior rather than merely expert costs. Third, online settings need joint treatment of routing, advice acquisition, and non-stationary expert competence. The current literature establishes that advice-aware deferral is not reducible to ordinary L2D with a larger action space; the unresolved question is how far the composite-action viewpoint can scale once advice is costly, delayed, strategic, or only partially observed.