Meta-Learners: Frameworks for Learning to Learn

• Meta-learners are meta-algorithmic frameworks that decompose complex learning tasks into simpler supervised subproblems, facilitating rapid adaptation when data is scarce.
• They enable estimation of conditional effects through techniques like S-, T-, and X-learners to solve causal inference problems by rearranging standard regression tasks.
• These frameworks find applications in few-shot learning, weakly-supervised segmentation, and adversarial online learning, offering robust performance even in complex task distributions.

A meta-learner is a meta-algorithmic framework that leverages the solution of simpler subproblems—often standard supervised learning or regression problems—to perform “learning to learn” at the task or distributional level. The goal is to extract, transfer, or regularize statistical structure across tasks or environments, and thereby achieve rapid adaptation, robust estimation, or efficient predictive inference, even when base task data are limited or the underlying task distribution is complex. Meta-learners are prominent in modern causal inference, few-shot learning, weakly/sparsely-supervised segmentation, Bayesian sequence learning, adversarial online learning, and foundational theoretical studies of multi-task and representation learning.

1. Canonical Meta-Learner Frameworks in Causal Inference

Meta-learners in the context of heterogeneous treatment effect estimation decompose the conditional average treatment effect (CATE) problem—which cannot be directly posed as standard supervised learning—into one or several regression subproblems solvable by any regression or machine learning base learner (e.g., random forests (RF), Bayesian additive regression trees (BART), neural nets) (Künzel et al., 2017).

Let observed data be i.i.d. units $(Y_i(0),Y_i(1),X_i,W_i)$ with covariates $X_i\in\mathbb{R}^d$, binary treatment indicator $W_i\in\{0,1\}$, and observed outcome $Y_i=Y_i(W_i)$. The CATE is defined as $\tau(x)=\mathbb{E}[Y(1)-Y(0)\mid X=x]$.

Major meta-learners:

• S-Learner: Fit a single regression model $\hat{m}(x,w)$ for $Y$ on $(X,w)$. Estimate CATE as $\hat{\tau}_S(x)=\hat{m}(x,1)-\hat{m}(x,0)$. Pools all data, but can bias $\hat{\tau}$ toward zero if $w$ is a weak predictor.
• T-Learner: Fit two separate models, $\hat{m}_1(x)$ using treated units, $\hat{m}_0(x)$ using controls. Compute CATE as $\hat{\tau}_T(x)=\hat{m}_1(x)-\hat{m}_0(x)$. Handles highly non-overlapping response functions, but does not borrow strength when CATE is structurally simpler than the base outcome functions.

• X-Learner: A 3-stage procedure that exploits both shared structure and unbalanced designs:

  1. Fit $\hat{\mu}_1(x)$, $\hat{\mu}_0(x)$ by base learners.
  2. Impute pseudo-outcomes (difference-in-differences): (D_i^{(1)}=Y_i-\hat{\mu}_