Multi-Task Prediction-Powered Inference
- Multi-Task Prediction-Powered Inference is a framework that combines inexpensive proxy measurements with limited high-cost labels to yield statistically valid estimates across multiple related tasks.
- It optimally allocates resources by selecting predictor subsets, employing active routing and adaptive sampling to minimize variance under strict budget constraints.
- The approach demonstrates improved label efficiency and accuracy in fields like LLM evaluation and healthcare, supported by robust theoretical guarantees and practical applications.
Multi-task prediction-powered inference denotes a family of inferential methods that use abundant but inexpensive proxy measurements together with a small number of expensive, high-quality labels to perform statistically valid estimation across multiple predictors, multiple measurement designs, or many related tasks. Recent work develops this idea along three closely related axes: Multiple-Prediction-Powered Inference, which optimally allocates sampling across subsets of measurement sources under explicit cost constraints; Active Multiple-Prediction-Powered Inference, which performs per-instance routing, active labeling, and linear reweighting under a single deployment-time budget; and multi-task PPI across many tasks, which borrows strength through cross-task recalibration while retaining task-specific inference (Cowen-Breen et al., 28 Mar 2026, Brawand et al., 8 May 2026, Emmenegger et al., 28 May 2026).
1. From single-predictor PPI to multi-task structure
Classical prediction-powered inference (PPI) addresses a setting with one expensive target variable and one cheap predictor. In its basic form, with labeled pairs and additional predictor-only samples, the estimator is
PPI++ introduces a scalar tuning parameter ,
with chosen to minimize variance from labeled data (Cowen-Breen et al., 28 Mar 2026).
The single-predictor formulation becomes restrictive in several regimes emphasized in recent work. One regime has multiple predictors of differing cost and quality available at inference time, so that the relevant design question is not merely how much to trust one proxy, but which subset of proxies to query and how often. A second regime has many related tasks, each with only a small labeled sample, so that within-task rectification becomes unstable even though the proxy-ground-truth relationship may be partially shared across tasks. A third regime arises in post-deployment monitoring, where model-query cost and gold-label cost must be optimized jointly and where different instances may warrant different predictor subsets and different label-sampling probabilities (Cowen-Breen et al., 28 Mar 2026, Brawand et al., 8 May 2026, Emmenegger et al., 28 May 2026).
In this broader setting, “multi-task” has two distinct but compatible meanings. In Multiple-PPI, each coordinate of a random vector can be interpreted as a task or measurement type, and subsets of coordinates correspond to joint measurement designs. In the many-task finite-population framework, each task is a separate inferential target , but tasks share recalibration structure. This suggests a unifying perspective in which prediction-powered inference is extended by exploiting either cross-predictor covariance structure, cross-instance heterogeneity, or cross-task calibration structure.
2. Multiple-Prediction-Powered Inference as global cost-aware design
Multiple-Prediction-Powered Inference (MultiPPI) studies a general mean-estimation problem with multiple measurement sources of varying quality and cost. The basic object is a random vector
and the target is a linear functional of the mean,
In the experiments, , so the target is , with 0 representing the high-quality but expensive measurement and 1 representing cheaper proxies (Cowen-Breen et al., 28 Mar 2026).
The distinctive feature of MultiPPI is that it allows sampling any subset 2 of coordinates from a collection 3. If 4 denotes the number of i.i.d. samples of 5, then each 6 represents a joint query. This is the core generalization of multi-task or multi-model PPI: the estimator is built from a collection of joint measurement designs rather than a single predictor. With scalar costs 7, the budget constraint is
8
and the paper also treats vector-valued costs for multiple simultaneous constraints (Cowen-Breen et al., 28 Mar 2026).
For fixed allocations 9 and weights 0, the estimator class is
1
Unbiasedness reduces to linear constraints on the 2, and MultiPPI chooses the unbiased, budget-satisfying estimator with minimal mean squared error. Writing 3 for projection onto coordinates in 4, 5, and 6 for the Moore–Penrose pseudo-inverse, the paper defines
7
and characterizes the minimum achievable variance under known 8 through the optimization
9
The optimal weights are
0
The continuous relaxation has a convex structure. With a single budget constraint it yields a second-order cone program, and with multiple budget constraints it yields an SDP. The dual SOCP is
1
The MultiAllocate procedure uses an estimate 2 and the costs 3 to solve this program, recover a continuous allocation, scale it to the available budget, round to integer sample sizes, and compute the corresponding 4 (Cowen-Breen et al., 28 Mar 2026).
The resulting allocation is budget-adaptive. For small budgets, the method tends to emphasize cheap predictors and may approximate scalar PPI++ with a single best proxy. As budget increases, it can shift toward more expensive and more accurate predictors and toward joint subsets whose covariance structure provides stronger variance reduction. In the paper’s interpretation, this behavior is precisely what a multi-task or multi-model PPI procedure should do when costs and correlations are heterogeneous.
3. Active Multiple-Prediction-Powered Inference and per-instance routing
Active Multiple-Prediction-Powered Inference (AM-PPI) addresses post-deployment monitoring with multiple predictors available at inference time, each with known per-query cost. The observed covariates are 5, the expensive gold-standard label is 6, and the inferential target is the population mean
7
For each instance, one chooses a subset 8 of predictors 9, incurring cost
0
and one may also purchase the gold label with probability 1, paying 2. The expected per-instance budget constraint is
3
for 4 (Brawand et al., 8 May 2026).
The AM-PPI estimator belongs to the linear-prediction augmented inverse propensity weighted class. If 5 are linear weights and 6, then
7
This recovers the usual ASI or PPI++-style estimator when 8, 9, and 0 (Brawand et al., 8 May 2026).
Its asymptotic variance can be written in terms of the conditional residual variance
1
as
2
The optimization therefore couples three decisions: routing 3, sampling 4, and reweighting 5.
For a fixed subset 6, the optimal sampling rule is uncertainty-proportional: 7 followed by clipping,
8
Thus, instances with high residual uncertainty are labeled more often, and instances with very large residual variance are always labeled. For fixed sampling, the optimal 9 solves a weighted least-squares condition,
0
Routing is then performed pointwise by minimizing a per-instance Lagrangian criterion 1 that combines variance contribution, predictor-query cost, and expected label cost (Brawand et al., 8 May 2026).
A central technical result is that, despite non-joint convexity, the fixed point defined by the KKT conditions is globally optimal. The argument uses biconvexity, strong duality for the single scalar budget constraint, and finite outer minimization over routing. Operationally, AM-PPI extends Multiple-PPI from global per-predictor allocation to per-instance adaptive routing.
4. Cross-task recalibration across many tasks
A different line of work studies prediction-powered inference across many related tasks. Here each task 2 has a finite population of 3 items, covariates 4, a ground-truth outcome 5, and a proxy 6. The task-specific estimand is the finite-population mean
7
Only a small labeled subset 8 is observed for each task, typically under simple random sampling without replacement, while proxies are available for all items and all tasks (Emmenegger et al., 28 May 2026).
The starting point is the PPI++ estimator for mean estimation,
9
where 0 is a surrogate or recalibrated proxy. In standard single-task PPI, all rectification and power tuning are done within task. The multi-task framework changes how 1 is learned: it uses labels from related tasks to improve the proxy-ground-truth mapping while preserving per-task inference (Emmenegger et al., 28 May 2026).
The simplest method is GRePPI. For a target task 2, it fits a recalibration map 3 on pooled labeled data from all other tasks,
4
using a function class 5. The paper uses isotonic regression of 6 on 7, motivated by the observation that LLM scores tend to be miscalibrated but roughly monotone in human labels. The surrogate values 8 are then inserted into the usual task-specific PPI estimator, with either 9 or a locally estimated
0
ARePPI addresses task heterogeneity by adaptively mixing global and local recalibration. For each task 1, the labeled set 2 is split into two folds. Within each fold, the method constructs out-of-fold local recalibration predictions, chooses a mixing weight 3 to maximize squared correlation with 4, and forms an adaptive recalibrator
5
The final estimator uses cross-fitted adaptive surrogate values 6 in the same PPI form. The data-sharing mechanism is therefore deliberately narrow: tasks borrow strength only through the surrogate mapping, not by shrinking the target parameters 7 toward one another (Emmenegger et al., 28 May 2026).
5. Statistical guarantees and interpretive boundaries
The three frameworks provide distinct but complementary theoretical guarantees. For MultiPPI, when 8 is known, the estimator is minimax optimal in mean squared error among all budget-satisfying estimators over distributions sharing covariance 9: 0 With estimated covariance 1, the estimator is asymptotically normal,
2
and its risk is stable to covariance misspecification through the bound
3
The same paper also shows a low-budget regime in which the allocation concentrates on a single best subset and a high-budget regime in which continuous relaxations and rounded integer allocations are asymptotically equivalent (Cowen-Breen et al., 28 Mar 2026).
For AM-PPI, the estimator is asymptotically normal under bounded fourth moments, nuisance consistency, and overlap: 4 The variance estimator is consistent, so standard Wald intervals have asymptotic coverage at least 5. Within the class of linear-prediction AIPW estimators satisfying the overlap and budget constraints, AM-PPI uniquely achieves minimum asymptotic variance. Moreover, if the span of predictor features contains the true regression function 6 for some subset, then its influence function coincides with the semiparametric efficient influence function, and the variance equals the semiparametric efficiency bound (Brawand et al., 8 May 2026).
For multi-task PPI across many tasks, validity is task-specific. With simple random sampling without replacement, fixed recalibration, and fixed 7, the estimator is unbiased in finite samples. A finite-population CLT yields asymptotic normality, and the variance estimator includes the finite population correction: 8 The most distinctive theoretical result is the affine-invariance statement: if the recalibration is affine, 9 with 00, then its oracle variance is the same as that of the identity surrogate. Strict improvement over the raw proxy is possible if and only if the regression function 01 is not affine on the population support. In other words, power tuning already extracts all linear benefit from the proxy; additional gains require nonlinear proxy-ground-truth structure (Emmenegger et al., 28 May 2026).
These results resolve several common misunderstandings. Multi-task PPI is not merely “use more proxies.” In MultiPPI it is a joint measurement-design problem over costs and covariance. In AM-PPI it is a routing-and-sampling problem over instances. In the many-task framework it is a recalibration problem over related tasks. It is also not equivalent to hierarchical shrinkage of the target parameters. The many-task paper is explicit that information sharing occurs at the level of the surrogate mapping 02, while each 03 is still estimated from its own labeled sample and its own rectification step.
A second recurring boundary concerns coverage. MultiPPI notes that when the number of gold labels is fixed and the proxy budget grows, coverage of asymptotic confidence intervals can decay slightly below nominal because the tuning parameter is estimated from a fixed labeled set. The many-task paper reports a related small-04 issue: local power tuning can be slightly anti-conservative when labels are extremely scarce, whereas fixing 05 is conservative but less efficient. AM-PPI instead enforces overlap through 06 and proves asymptotic validity under nuisance consistency, but its guarantees are asymptotic rather than finite-sample (Cowen-Breen et al., 28 Mar 2026, Brawand et al., 8 May 2026, Emmenegger et al., 28 May 2026).
6. Empirical domains, limitations, and open directions
The empirical record spans LLM evaluation, healthcare monitoring, and social-science-style multi-task auditing. MultiPPI evaluates three LLM scenarios. In Chatbot Arena win-rate estimation, the target is the human win-rate of Claude-2.1 versus GPT-4-1106-Preview, with Gemini 2.5 Pro and Gemini 2.5 Flash as proxies. In ProcessBench, the target is whether a math solution contains a process error, and the proxies are four Gemini 2.5 Pro autoraters with Think budgets of 125, 250, 375, and 500 words under a non-additive cascading cost model. In biography factuality evaluation, the target is factual consistency of biography-fact pairs, with Gemini 2.0 Flash Lite debate configurations as proxies. Across these scenarios, MultiPPI achieves lower estimation error than existing baselines, coverage of 95% CIs is near nominal in the main regimes, and the learned allocation shifts from cheap single-autorater behavior at low budgets to joint multi-autorater behavior at higher budgets (Cowen-Breen et al., 28 Mar 2026).
AM-PPI is evaluated on synthetic regression, MIMIC-III lab-text consistency monitoring, hypothyroid detection, and VeriFact-BHC proposition consistency. Its headline empirical result is that it produces 10 to 40 percent narrower confidence intervals than single-predictor ASI in the budget regime where routing matters, and matches the better baseline elsewhere. The regime structure is explicit: at low budget, the expensive predictor may be infeasible and AM-PPI effectively reduces to the cheap baseline; at intermediate budget, mixed routing yields the largest gains; at high budget, routing matters less and the methods converge (Brawand et al., 8 May 2026).
The many-task recalibration framework is validated on synthetic and semi-synthetic datasets and on an audit of LLMs on election-related information during the 2024 U.S. presidential election. In the human-annotated case study, tasks are model–prompt-pair comparisons over 186 questions, producing 72 tasks, 07 labeled question pairs per task, and 08 annotations. Both GRePPI and ARePPI yield substantially narrower confidence intervals than classical labeled-only inference, raw-proxy PPI, and single-task RePPI. The observed benefit aligns with the paper’s theory because the proxy-label relationship is monotone but clearly nonlinear (Emmenegger et al., 28 May 2026).
The limitations are correspondingly heterogeneous. MultiPPI depends on a reliable covariance estimate 09, requires a reasonable number of fully labeled samples, and faces a combinatorial subset family if 10. The restricted family
11
works well empirically but lacks a formal optimality guarantee. AM-PPI assumes a single scalar budget constraint in its main theory, requires nuisance consistency for routing, uncertainty, and weights, and is developed for mean estimation even though the framework extends to general 12-estimators. The many-task recalibration framework assumes simple random sampling without replacement, stability of the proxy-label relationship across tasks, and enough shared structure that leave-one-task-out or adaptive recalibration is informative (Cowen-Breen et al., 28 Mar 2026, Brawand et al., 8 May 2026, Emmenegger et al., 28 May 2026).
The open directions named in these papers are also mutually reinforcing. MultiPPI identifies formal multi-task theory for vector parameters, Pareto-efficient allocations across tasks, structured subset selection, robustness beyond covariance assumptions, adaptive or sequential allocation, and connections to semiparametric efficiency, TMLE, and double ML. AM-PPI supplies a blueprint for extending multi-predictor routing to multi-task monitoring under shared budgets. The many-task paper highlights integration with empirical-Bayes shrinkage, extension beyond finite-population means, and more refined task-similarity learning. Taken together, these developments define multi-task prediction-powered inference as a rapidly consolidating area centered on one principle: task-specific valid inference can be made substantially more label-efficient when the design explicitly exploits shared cost structure, shared covariance structure, or shared nonlinear calibration structure.