---
title: Multi-Task Prediction-Powered Inference
url: https://www.emergentmind.com/topics/multi-task-prediction-powered-inference
type: topic
---

# Multi-Task Prediction-Powered Inference

Multi-task prediction-powered inference denotes a family of inferential methods that use abundant but inexpensive proxy measurements together with a small number of expensive, high-quality labels to perform statistically valid estimation across multiple predictors, multiple measurement designs, or many related tasks. Recent work develops this idea along three closely related axes: Multiple-Prediction-Powered Inference, which optimally allocates sampling across subsets of measurement sources under explicit cost constraints; Active Multiple-Prediction-Powered Inference, which performs per-instance routing, active labeling, and linear reweighting under a single deployment-time budget; and multi-task PPI across many tasks, which borrows strength through cross-task recalibration while retaining task-specific inference [2603.27414; 2605.08429; 2605.29249].

## 1. From single-predictor PPI to multi-task structure

Classical prediction-powered inference (PPI) addresses a setting with one expensive target variable and one cheap predictor. In its basic form, with labeled pairs \((X_1^{(j)}, X_2^{(j)})\) and additional predictor-only samples, the estimator is
\[
\hat{\theta}_{\mathrm{PPI}}
=
\frac{1}{n}\sum_{j=1}^n \bigl(X_1^{(j)}-X_2^{(j)}\bigr)
+
\frac{1}{N}\sum_{j=1}^N X_2^{(j)}.
\]
PPI++ introduces a scalar tuning parameter \(\lambda\),
\[
\hat{\theta}_{\mathrm{PPI++}}
=
\frac{1}{n}\sum_{j=1}^n \bigl(X_1^{(j)}-\lambda X_2^{(j)}\bigr)
+
\lambda \frac{1}{N}\sum_{j=1}^N X_2^{(j)},
\]
with \(\lambda\) chosen to minimize variance from labeled data [2603.27414].

The single-predictor formulation becomes restrictive in several regimes emphasized in recent work. One regime has multiple predictors of differing cost and quality available at inference time, so that the relevant design question is not merely how much to trust one proxy, but which subset of proxies to query and how often. A second regime has many related tasks, each with only a small labeled sample, so that within-task rectification becomes unstable even though the proxy-ground-truth relationship may be partially shared across tasks. A third regime arises in post-deployment monitoring, where model-query cost and gold-label cost must be optimized jointly and where different instances may warrant different predictor subsets and different label-sampling probabilities [2603.27414; 2605.08429; 2605.29249].

In this broader setting, “multi-task” has two distinct but compatible meanings. In Multiple-PPI, each coordinate of a random vector can be interpreted as a task or measurement type, and subsets of coordinates correspond to joint measurement designs. In the many-task finite-population framework, each task is a separate inferential target \(\theta^{(t)}\), but tasks share recalibration structure. This suggests a unifying perspective in which prediction-powered inference is extended by exploiting either cross-predictor covariance structure, cross-instance heterogeneity, or cross-task calibration structure.

## 2. Multiple-Prediction-Powered Inference as global cost-aware design

Multiple-Prediction-Powered Inference (MultiPPI) studies a general mean-estimation problem with multiple measurement sources of varying quality and cost. The basic object is a random vector
\[
X=(X_1,\dots,X_k)\in\mathbb{R}^k,
\qquad
\Sigma=\operatorname{Cov}(X),
\]
and the target is a linear functional of the mean,
\[
\theta^* = a^\top \mathbb{E}[X], \qquad a\in\mathbb{R}^k.
\]
In the experiments, \(a=(1,0,\ldots,0)\), so the target is \(\mathbb{E}[X_1]\), with \(X_1\) representing the high-quality but expensive measurement and \(X_2,\dots,X_k\) representing cheaper proxies [2603.27414].

The distinctive feature of MultiPPI is that it allows sampling any subset \(I\) of coordinates from a collection \(\mathcal{I}\subseteq 2^{\{1,\dots,k\}}\). If \(n_I\) denotes the number of i.i.d. samples of \(X_I\), then each \(I\) represents a joint query. This is the core generalization of multi-task or multi-model PPI: the estimator is built from a collection of joint measurement designs rather than a single predictor. With scalar costs \(c_I\), the budget constraint is
\[
\sum_{I\in\mathcal{I}} c_I n_I \le B,
\]
and the paper also treats vector-valued costs for multiple simultaneous constraints [2603.27414].

For fixed allocations \(n_I\) and weights \(\lambda_I\in\mathbb{R}^{|I|}\), the estimator class is
\[
\hat{\theta}(n,\Lambda)
=
\sum_{I\in\mathcal{I}:n_I>0}
\frac{1}{n_I}\sum_{j=1}^{n_I} \lambda_I^\top X_I^{(j)}.
\]
Unbiasedness reduces to linear constraints on the \(\lambda_I\), and MultiPPI chooses the unbiased, budget-satisfying estimator with minimal mean squared error. Writing \(P_I\) for projection onto coordinates in \(I\), \(\Sigma_I=P_I\Sigma P_I^\top\), and \(\Sigma_I^\dagger\) for the Moore–Penrose pseudo-inverse, the paper defines
\[
S(n)=\sum_{I\in\mathcal{I}} n_I \Sigma_I^\dagger,
\]
and characterizes the minimum achievable variance under known \(\Sigma\) through the optimization
\[
V_B = \min_{n:B(n)\ \mathrm{holds}} a^\top S(n)\, a.
\]
The optimal weights are
\[
\lambda_I^\star = \bigl(n_I^\star \Sigma_I^\dagger S(n^\star)a\bigr)_I.
\]

The continuous relaxation has a convex structure. With a single budget constraint it yields a second-order cone program, and with multiple budget constraints it yields an SDP. The dual SOCP is
\[
\max_{y\in\mathbb{R}^k} a^\top y
\quad\text{s.t.}\quad
\|P_I y\|_{\Sigma_I}\le c_I \ \text{for all } I.
\]
The MultiAllocate procedure uses an estimate \(\hat{\Sigma}\) and the costs \(c_I\) to solve this program, recover a continuous allocation, scale it to the available budget, round to integer sample sizes, and compute the corresponding \(\lambda_I\) [2603.27414].

The resulting allocation is budget-adaptive. For small budgets, the method tends to emphasize cheap predictors and may approximate scalar PPI++ with a single best proxy. As budget increases, it can shift toward more expensive and more accurate predictors and toward joint subsets whose covariance structure provides stronger variance reduction. In the paper’s interpretation, this behavior is precisely what a multi-task or multi-model PPI procedure should do when costs and correlations are heterogeneous.

## 3. Active Multiple-Prediction-Powered Inference and per-instance routing

Active Multiple-Prediction-Powered Inference (AM-PPI) addresses post-deployment monitoring with multiple predictors available at inference time, each with known per-query cost. The observed covariates are \(X_1,\dots,X_n\sim P_X\), the expensive gold-standard label is \(Y\), and the inferential target is the population mean
\[
\theta^*=\mathbb{E}[Y].
\]
For each instance, one chooses a subset \(I_i\subseteq\{1,\dots,k\}\) of predictors \(f_1,\dots,f_k\), incurring cost
\[
c_{I_i}=\sum_{j\in I_i} c_j,
\]
and one may also purchase the gold label with probability \(\pi_{I_i}(X_i)\), paying \(c_{\text{label}}\). The expected per-instance budget constraint is
\[
\mathbb{E}\!\left[c_{I(X)} + \pi_{I(X)}(X)c_{\text{label}}\right]\le b
\]
for \(b=B/n\) [2605.08429].

The AM-PPI estimator belongs to the linear-prediction augmented inverse propensity weighted class. If \(\lambda_{I_i}\in\mathbb{R}^{|I_i|}\) are linear weights and \(m_i=\lambda_{I_i}^\top f_{I_i}(X_i)\), then
\[
\hat{\theta}
=
\frac{1}{n}\sum_{i=1}^n
\left[
\lambda_{I_i}^\top f_{I_i}(X_i)
+
\left(Y_i-\lambda_{I_i}^\top f_{I_i}(X_i)\right)\frac{\xi_i}{\pi_{I_i}(X_i)}
\right].
\]
This recovers the usual ASI or PPI++-style estimator when \(k=1\), \(c_1=0\), and \(\lambda_1=1\) [2605.08429].

Its asymptotic variance can be written in terms of the conditional residual variance
\[
r_I(x) = \mathbb{E}\!\left[(Y-\lambda_I^\top f_I(x))^2\mid X=x\right],
\qquad
u_I(x)=\sqrt{r_I(x)},
\]
as
\[
\mathrm{Var}(\hat{\theta})
=
\frac{1}{n}\mathrm{Var}(Y)
+
\frac{1}{n}\,
\mathbb{E}\!\left[
r_{I(X)}(X)\left(\frac{1}{\pi_{I(X)}(X)}-1\right)
\right].
\]
The optimization therefore couples three decisions: routing \(I(\cdot)\), sampling \(\pi(\cdot)\), and reweighting \(\lambda\).

For a fixed subset \(I\), the optimal sampling rule is uncertainty-proportional:
\[
\pi_I^*(x)
=
\sqrt{\frac{r_I(x)}{n\mu c_{\text{label}}}}
=
\frac{u_I(x)}{\sqrt{n\mu c_{\text{label}}}},
\]
followed by clipping,
\[
\hat{\pi}_I(x)=\min\!\bigl(1,\pi_I^*(x)\bigr).
\]
Thus, instances with high residual uncertainty are labeled more often, and instances with very large residual variance are always labeled. For fixed sampling, the optimal \(\lambda_I\) solves a weighted least-squares condition,
\[
\mathbb{E}\!\left[w_I(X)\bigl(Y-\lambda_I^\top f_I(X)\bigr)f_I(X)\right]=0,
\qquad
w_I(X)=\frac{1}{\hat{\pi}_I(X)}-1.
\]
Routing is then performed pointwise by minimizing a per-instance Lagrangian criterion \(\ell_I(x)\) that combines variance contribution, predictor-query cost, and expected label cost [2605.08429].

A central technical result is that, despite non-joint convexity, the fixed point defined by the KKT conditions is globally optimal. The argument uses biconvexity, strong duality for the single scalar budget constraint, and finite outer minimization over routing. Operationally, AM-PPI extends Multiple-PPI from global per-predictor allocation to per-instance adaptive routing.

## 4. Cross-task recalibration across many tasks

A different line of work studies prediction-powered inference across many related tasks. Here each task \(t\) has a finite population of \(N\) items, covariates \(X_i^{(t)}\), a ground-truth outcome \(Y_i^{(t)}\), and a proxy \(\hat{Y}_i^{(t)}\). The task-specific estimand is the finite-population mean
\[
\theta^{(t)}=\frac{1}{N}\sum_{i=1}^N Y_i^{(t)}.
\]
Only a small labeled subset \(L^{(t)}\subset[N]\) is observed for each task, typically under simple random sampling without replacement, while proxies are available for all items and all tasks [2605.29249].

The starting point is the PPI++ estimator for mean estimation,
\[
\hat{\theta}_\lambda
=
\frac{1}{n}\sum_{i\in L} Y_i
+
\lambda
\left\{
\frac{1}{N}\sum_{i=1}^N s(X_i,\hat{Y}_i)
-
\frac{1}{n}\sum_{i\in L} s(X_i,\hat{Y}_i)
\right\},
\]
where \(s\) is a surrogate or recalibrated proxy. In standard single-task PPI, all rectification and power tuning are done within task. The multi-task framework changes how \(s\) is learned: it uses labels from related tasks to improve the proxy-ground-truth mapping while preserving per-task inference [2605.29249].

The simplest method is GRePPI. For a target task \(t\), it fits a recalibration map \(\hat{s}^{(-t)}\) on pooled labeled data from all other tasks,
\[
L^{(-t)}=\bigcup_{j\neq t} L^{(j)},
\]
using a function class \(\mathcal{H}\). The paper uses isotonic regression of \(Y\) on \(\hat{Y}\), motivated by the observation that LLM scores tend to be miscalibrated but roughly monotone in human labels. The surrogate values \(s_i^{(t)}=\hat{s}^{(-t)}(\hat{Y}_i^{(t)})\) are then inserted into the usual task-specific PPI estimator, with either \(\lambda=1\) or a locally estimated
\[
\hat{\lambda}_{L^{(t)}}
=
\frac{\mathrm{Cov}_{L^{(t)}}\!\left(Y^{(t)},\,\hat{s}^{(-t)}(\hat{Y}^{(t)})\right)}
{\mathrm{Var}_{L^{(t)}}\!\left(\hat{s}^{(-t)}(\hat{Y}^{(t)})\right)}.
\]

ARePPI addresses task heterogeneity by adaptively mixing global and local recalibration. For each task \(t\), the labeled set \(L^{(t)}\) is split into two folds. Within each fold, the method constructs out-of-fold local recalibration predictions, chooses a mixing weight \(\hat{\gamma}_F\in[0,1]\) to maximize squared correlation with \(Y\), and forms an adaptive recalibrator
\[
\hat{s}^{\mathrm{ada}_F}
=
\hat{\gamma}_F \hat{s}_F + (1-\hat{\gamma}_F)\hat{s}^{(-t)}.
\]
The final estimator uses cross-fitted adaptive surrogate values \(u_i^{(t)}\) in the same PPI form. The data-sharing mechanism is therefore deliberately narrow: tasks borrow strength only through the surrogate mapping, not by shrinking the target parameters \(\theta^{(t)}\) toward one another [2605.29249].

## 5. Statistical guarantees and interpretive boundaries

The three frameworks provide distinct but complementary theoretical guarantees. For MultiPPI, when \(\Sigma\) is known, the estimator is minimax optimal in mean squared error among all budget-satisfying estimators over distributions sharing covariance \(\Sigma\):
\[
\inf_{\hat{\theta}\in\mathcal{O}_B}\sup_{P\in\mathcal{P}_\Sigma}
\mathbb{E}_P[(\hat{\theta}-\theta^*)^2]
=
\operatorname{Var}(\hat{\theta}_{\mathrm{MultiPPI}(\Sigma)})
=
V_B.
\]
With estimated covariance \(\hat{\Sigma}\to\Sigma\), the estimator is asymptotically normal,
\[
\sqrt{B}\bigl(\hat{\theta}_{\mathrm{MultiPPI}(\hat{\Sigma})}-\theta^*\bigr)
\overset{d}{\to}\mathcal{N}(0,V^*),
\]
and its risk is stable to covariance misspecification through the bound
\[
\mathbb{E}\!\left[(\hat{\theta}_{\mathrm{MultiPPI}(\hat{\Sigma})}-\theta^*)^2\right]
\le
V_B + 4V_B^{\mathrm{classic}}\|\hat{\Sigma}-\Sigma\|_F.
\]
The same paper also shows a low-budget regime in which the allocation concentrates on a single best subset and a high-budget regime in which continuous relaxations and rounded integer allocations are asymptotically equivalent [2603.27414].

For AM-PPI, the estimator is asymptotically normal under bounded fourth moments, nuisance consistency, and overlap:
\[
\sqrt{n}(\hat{\theta}-\theta^*) \xrightarrow{d} \mathcal{N}(0,V).
\]
The variance estimator is consistent, so standard Wald intervals have asymptotic coverage at least \(1-\alpha\). Within the class of linear-prediction AIPW estimators satisfying the overlap and budget constraints, AM-PPI uniquely achieves minimum asymptotic variance. Moreover, if the span of predictor features contains the true regression function \(\mu(x)=\mathbb{E}[Y\mid X=x]\) for some subset, then its influence function coincides with the semiparametric efficient influence function, and the variance equals the semiparametric efficiency bound [2605.08429].

For multi-task PPI across many tasks, validity is task-specific. With simple random sampling without replacement, fixed recalibration, and fixed \(\lambda\), the estimator is unbiased in finite samples. A finite-population CLT yields asymptotic normality, and the variance estimator includes the finite population correction:
\[
\widehat{V}(\hat{\theta}^{(t)})
=
\frac{1}{n_t}\left(1-\frac{n_t}{N}\right)\hat{S}_r^2.
\]
The most distinctive theoretical result is the affine-invariance statement: if the recalibration is affine, \(\hat{s}(\hat{Y})=a\hat{Y}+b\) with \(a\neq 0\), then its oracle variance is the same as that of the identity surrogate. Strict improvement over the raw proxy is possible if and only if the regression function \(m(z)=\mathbb{E}_N[Y\mid \hat{Y}=z]\) is not affine on the population support. In other words, power tuning already extracts all linear benefit from the proxy; additional gains require nonlinear proxy-ground-truth structure [2605.29249].

These results resolve several common misunderstandings. Multi-task PPI is not merely “use more proxies.” In MultiPPI it is a joint measurement-design problem over costs and covariance. In AM-PPI it is a routing-and-sampling problem over instances. In the many-task framework it is a recalibration problem over related tasks. It is also not equivalent to hierarchical shrinkage of the target parameters. The many-task paper is explicit that information sharing occurs at the level of the surrogate mapping \(s\), while each \(\theta^{(t)}\) is still estimated from its own labeled sample and its own rectification step.

A second recurring boundary concerns coverage. MultiPPI notes that when the number of gold labels is fixed and the proxy budget grows, coverage of asymptotic confidence intervals can decay slightly below nominal because the tuning parameter is estimated from a fixed labeled set. The many-task paper reports a related small-\(n_t\) issue: local power tuning can be slightly anti-conservative when labels are extremely scarce, whereas fixing \(\lambda=1\) is conservative but less efficient. AM-PPI instead enforces overlap through \(\pi_{\min}\) and proves asymptotic validity under nuisance consistency, but its guarantees are asymptotic rather than finite-sample [2603.27414; 2605.08429; 2605.29249].

## 6. Empirical domains, limitations, and open directions

The empirical record spans LLM evaluation, healthcare monitoring, and social-science-style multi-task auditing. MultiPPI evaluates three LLM scenarios. In Chatbot Arena win-rate estimation, the target is the human win-rate of Claude-2.1 versus GPT-4-1106-Preview, with Gemini 2.5 Pro and Gemini 2.5 Flash as proxies. In ProcessBench, the target is whether a math solution contains a process error, and the proxies are four Gemini 2.5 Pro autoraters with Think budgets of 125, 250, 375, and 500 words under a non-additive cascading cost model. In biography factuality evaluation, the target is factual consistency of biography-fact pairs, with Gemini 2.0 Flash Lite debate configurations as proxies. Across these scenarios, MultiPPI achieves lower estimation error than existing baselines, coverage of 95% CIs is near nominal in the main regimes, and the learned allocation shifts from cheap single-autorater behavior at low budgets to joint multi-autorater behavior at higher budgets [2603.27414].

AM-PPI is evaluated on synthetic regression, MIMIC-III lab-text consistency monitoring, hypothyroid detection, and VeriFact-BHC proposition consistency. Its headline empirical result is that it produces 10 to 40 percent narrower confidence intervals than single-predictor ASI in the budget regime where routing matters, and matches the better baseline elsewhere. The regime structure is explicit: at low budget, the expensive predictor may be infeasible and AM-PPI effectively reduces to the cheap baseline; at intermediate budget, mixed routing yields the largest gains; at high budget, routing matters less and the methods converge [2605.08429].

The many-task recalibration framework is validated on synthetic and semi-synthetic datasets and on an audit of language models on election-related information during the 2024 U.S. presidential election. In the human-annotated case study, tasks are model–prompt-pair comparisons over 186 questions, producing 72 tasks, \(n_t=40\) labeled question pairs per task, and \(72\times 40\times 5 = 14{,}400\) annotations. Both GRePPI and ARePPI yield substantially narrower confidence intervals than classical labeled-only inference, raw-proxy PPI, and single-task RePPI. The observed benefit aligns with the paper’s theory because the proxy-label relationship is monotone but clearly nonlinear [2605.29249].

The limitations are correspondingly heterogeneous. MultiPPI depends on a reliable covariance estimate \(\hat{\Sigma}\), requires a reasonable number of fully labeled samples, and faces a combinatorial subset family if \(\mathcal{I}=2^{\{2,\dots,k\}}\). The restricted family
\[
\mathcal{I}=\{\{1,\dots,k\},\{2,\dots,k\}\}\cup\{\{2\},\dots,\{k\}\}
\]
works well empirically but lacks a formal optimality guarantee. AM-PPI assumes a single scalar budget constraint in its main theory, requires nuisance consistency for routing, uncertainty, and weights, and is developed for mean estimation even though the framework extends to general \(M\)-estimators. The many-task recalibration framework assumes simple random sampling without replacement, stability of the proxy-label relationship across tasks, and enough shared structure that leave-one-task-out or adaptive recalibration is informative [2603.27414; 2605.08429; 2605.29249].

The open directions named in these papers are also mutually reinforcing. MultiPPI identifies formal multi-task theory for vector parameters, Pareto-efficient allocations across tasks, structured subset selection, robustness beyond covariance assumptions, adaptive or sequential allocation, and connections to semiparametric efficiency, TMLE, and double ML. AM-PPI supplies a blueprint for extending multi-predictor routing to multi-task monitoring under shared budgets. The many-task paper highlights integration with empirical-Bayes shrinkage, extension beyond finite-population means, and more refined task-similarity learning. Taken together, these developments define multi-task prediction-powered inference as a rapidly consolidating area centered on one principle: task-specific valid inference can be made substantially more label-efficient when the design explicitly exploits shared cost structure, shared covariance structure, or shared nonlinear calibration structure.

Source: https://www.emergentmind.com/topics/multi-task-prediction-powered-inference