Anchor Regression: Balancing Accuracy and Robustness

• Anchor regression is an estimation method that regularizes ordinary least squares by incorporating anchor variables to address heterogeneity and distribution shifts.
• It establishes a continuum between OLS and instrumental variable methods by tuning a regularization parameter, gamma, to balance in-sample fit and robustness.
• The approach is validated through theoretical guarantees and practical results in applications like gene expression and economic modeling, offering actionable insights for robust prediction.

Anchor regression is an estimation method designed to enhance distributional robustness and causal interpretability in prediction problems with heterogeneous data. Its core strategy involves regularizing ordinary least squares (OLS) regression by directly incorporating information from an exogenous “anchor” variable (or set of variables) that captures environmental or batch heterogeneity. Anchor regression yields estimators and variable selections that are robust to shifts in the data distribution aligned with the anchor, providing a continuous interpolation between OLS and instrumental variable (IV) solutions while maintaining computational efficiency and theoretical guarantees of minimax robustness against shift interventions.

1. Definition, Conceptual Motivation, and Formalism

Anchor regression introduces a loss function that decomposes the prediction error according to its alignment with the anchor variable. The population anchor regression estimator $b^{(\gamma)}$ for predicting a response $Y$ from predictors $X$ using anchor(s) $A$ is defined as:

$$b^{(\gamma)} = \underset{b}{\arg\min}\;\mathbb{E} \left[ ((I - P_A)(Y - X^\top b))^2 \right] + \gamma \,\mathbb{E} \left[ (P_A(Y - X^\top b))^2 \right]$$

Here, $P_A$ is the projection operator onto the linear span of $A$; $\gamma \geq 0$ is a regularization parameter. The loss splits the residuals into a component orthogonal to $A$ (invariant to anchor-induced shifts) and a component explained by $A$ (exposed to anchor-induced variability). The parameter $\gamma$ controls the trade-off between in-sample predictive accuracy and robustness under anticipated shifts aligned with $A$ (Rothenhäusler et al., 2018).

For $\gamma = 1$, the formulation reduces to OLS; for $\gamma = 0$, the estimator is “partialled out” with respect to $A$, focusing exclusively on anchor-invariant variation; as $\gamma \to \infty$, the estimator approaches two-stage least squares (IV), prioritizing immunity to anchor-induced perturbations, even at the cost of reduced in-sample fit.

2. Methodological and Computational Aspects

Loss Decomposition and Estimator Computation

The anchor regression loss function supports computationally efficient estimation. In practice, one can construct perturbed, anchor-regularized versions of the predictors and outcomes:

$$\tilde{X} = (I - \Pi_A) X + \sqrt{\gamma}\, \Pi_A X,\quad \tilde{Y} = (I - \Pi_A) Y + \sqrt{\gamma}\, \Pi_A Y$$

where $\Pi_A$ is the empirical projection matrix onto $A$. Anchor regression is implemented via (penalized) least squares on $(\tilde{X}, \tilde{Y})$. In high-dimensional settings ($d > n$), an $\ell_1$ penalty (“anchor Lasso”) can be added for sparsity.

Trade-off Summary Table

$\gamma$	Methodical Limit	Estimator Behavior
$0$	Partialling out	Focuses exclusively on anchor-invariant variation
$1$	Ordinary Least Squares (OLS)	Optimal in-sample prediction
$\infty$	Two-stage Least Squares (IV estimator)	Maximally robust to anchor-induced shifts

This continuum enables practitioners to tune robustness to distributional shifts versus predictive performance according to application-specific risk preferences.

3. Theoretical Guarantees and Distributional Robustness

A defining property of anchor regression is its explicit robustness to a predefined class of distributional shifts, characterized as “shift interventions” along directions influenced by the anchor $A$ through the system’s structural equations. Theoretical analysis establishes that the anchor regression objective is dual to minimizing the worst-case mean squared error under perturbations $C^{(\gamma)}$ consistent with the anchor’s effect:

$$\mathbb{E}\left[ ((I - P_A)(Y - X^\top b))^2 \right] + \gamma\, \mathbb{E}\left[ (P_A(Y - X^\top b))^2 \right] = \sup_{v \in C^{(\gamma)}} \mathbb{E}_v[(Y - X^\top b)^2]$$

This result provides a concrete guarantee that $b^{(\gamma)}$ is minimax optimal within the shift class $C^{(\gamma)}$ determined by the anchor’s connections to $(X, Y, H)$ via the system’s shift matrices.

If $b^{(0)} = b^{(\infty)}$, referred to as “anchor stability,” the OLS and IV solutions coincide, and the corresponding coefficient is invariant to anchor interventions. Under strong faithfulness and correct model specification, this coefficient can often be identified with the direct causal effect $\partial_x \mathbb{E}[Y \mid do(X=x)]$.

4. Relationship to Competing and Related Estimators

Anchor regression unifies and generalizes several standard estimation procedures:

• Ordinary Least Squares (OLS): Corresponds to $\gamma=1$, optimizing mean squared error without robustness considerations.
• Partialling Out the Anchor: $\gamma=0$, ignores any anchor-explained variation, risking inefficiency if anchor-induced shifts are modest.
• Instrumental Variable (IV): As $\gamma \to \infty$, the estimator converges to the IV solution, trading in-sample fit for maximal invariance to arbitrary anchor-based shifts.

Anchor regression enables systematic interpolation between these endpoints by adjusting $\gamma$, allowing tailored balancing of robustness and statistical efficiency.

Anchor stability is not always present; substantial discrepancies between $b^{(0)}$ and $b^{(\infty)}$ suggest that predictive accuracy and robustness cannot both be maximized, and the degree of sensitivity to $\gamma$ offers diagnostic insight into dependence on specific forms of observed heterogeneity.

5. Empirical Results and Applications

Empirical validation includes both simulation and real-world use cases:

• Simulation via SEMs: Demonstrations in three structural equation models (with anchors affecting $X$, $Y$, or latent confounders $H$) confirm that anchor regression maintains stable prediction error over increasing perturbation strengths (increasing $\gamma$).
• Gene Expression (GTEx): Prediction and variable selection for gene expression across different tissues (anchors) show improved replicability and feature stability when ranking by anchor regression (over an appropriate $\gamma$ range), compared to standard Lasso estimates.
• Bike Sharing Data: Time- or grouping-based anchor variables enable anchor regression to reduce worst-case prediction errors relative to OLS for hourly bike rental count prediction, confirming the expected robustness benefits.
• Practical Implementation: Data is transformed along anchor-induced directions, and anchor regression is computed via standard regression machinery. In high dimensions or with sparse signals, anchor Lasso provides a scalable solution.

6. Practical Implications, Extensions, and Limitations

Anchor regression is especially useful in heterogeneous data settings—multi-tissue or multi-batch omics data, temporally or spatially structured economic panels, or any scenario with observable grouping variables reflecting distributional changes.

In predictive modeling, it enables a principled trade-off between fit and robustness; as a diagnostic, the stability of the solution across $\gamma$ values indicates the degree of invariance in the underlying system, supporting causal interpretation subject to faithfulness assumptions.

Extensions include adaptation of anchor regression to non-linear models, although guarantees are most direct within a class of shift interventions aligned with the anchor’s span. Optimal choice of $\gamma$ is application-specific, with cross-validation (possibly targeting quantiles of conditional prediction error) being a practical recommendation. However, the method is limited by the linearity assumptions and the requirement that anticipated distributional changes correspond to the anchor’s effect directions; “black swan” or entirely novel distributional changes are not directly guarded against.

Anchor regression provides an operational framework for robust, causally-motivated estimation in the presence of heterogeneous data, with explicit theoretical minimax guarantees, efficient computation, and applicability to high-dimensional, real-world problems characterized by distribution shifts aligned with observable anchors.