Kernel Instrumental Variable Regression (1906.00232v6)

Abstract: Instrumental variable (IV) regression is a strategy for learning causal relationships in observational data. If measurements of input X and output Y are confounded, the causal relationship can nonetheless be identified if an instrumental variable Z is available that influences X directly, but is conditionally independent of Y given X and the unmeasured confounder. The classic two-stage least squares algorithm (2SLS) simplifies the estimation problem by modeling all relationships as linear functions. We propose kernel instrumental variable regression (KIV), a nonparametric generalization of 2SLS, modeling relations among X, Y, and Z as nonlinear functions in reproducing kernel Hilbert spaces (RKHSs). We prove the consistency of KIV under mild assumptions, and derive conditions under which convergence occurs at the minimax optimal rate for unconfounded, single-stage RKHS regression. In doing so, we obtain an efficient ratio between training sample sizes used in the algorithm's first and second stages. In experiments, KIV outperforms state of the art alternatives for nonparametric IV regression.

• The paper proposes Kernel Instrumental Variable Regression (KIV), a nonparametric generalization of the classical two-stage least squares method that handles nonlinear relationships using reproducing kernel Hilbert spaces.
• KIV operates in two stages: first learning the conditional mean embedding of features using operator estimation, and then performing nonlinear regression of the outcome variable on these learned embeddings.
• The authors prove KIV's consistency and minimax optimality under mild assumptions, and empirical results show it outperforms other nonparametric instrumental variable methods, particularly for smooth structural functions.

Kernel Instrumental Variable Regression: A Nonparametric Approach

Instrumental variable (IV) regression is a widely employed method for estimating causal relationships using observational data when there are confounding variables present. Traditionally, the two-stage least squares (2SLS) approach is adopted, which assumes linear relationships among the variables. However, in many real-world scenarios, the relationships between variables may not adhere to such linear assumptions. This paper proposes a novel approach called kernel instrumental variable regression (KIV), a nonparametric generalization of 2SLS, which relaxes the linearity constraint and models these relationships within a nonlinear framework using reproducing kernel Hilbert spaces (RKHSs).

The Kernel Instrumental Variable (KIV) Regression Framework

KIV addresses the limitations of the classical 2SLS method by allowing the modeling of nonlinear relationships in a flexible manner. It does so by leveraging the properties of RKHSs, a popular technique in the machine learning community used to handle nonlinearities. The core innovation in KIV is its ability to handle potentially infinite-dimensional feature spaces, which are often necessary for capturing complex, real-world causal structures that are nonlinear. The algorithm comprises two primary stages:

1. Stage 1 - Learning the Conditional Mean Embedding: This stage involves learning the conditional expectation of features using operator estimation in an RKHS. The conditional mean embedding $\mu(z)$ characterizes the full distribution of $X$ given %%%%2%%%%, rather than just approximating the conditional mean.
2. Stage 2 - Nonlinear Regression: The second stage employs kernel ridge regression to model the structural relationship between the conditional mean embeddings and the outcome variable. This stage is performed by regressing the output $Y$ on the learned embeddings from Stage 1.

Consistency and Optimality

The paper theoretically proves the consistency of the KIV algorithm under mild assumptions, providing convergence guarantees at the minimax optimal rate for unconfounded, single-stage RKHS regression. This development positions KIV as not only a practical alternative to linear IV methods but also an optimal one, given sufficient sample sizes.

The efficiency of the KIV estimator stems from its ability to dynamically adjust to the complexities of the underlying data-generating process. The authors highlight that KIV requires an optimal ratio between the sample sizes used in the algorithm's first and second stages, which depends on the problem's intrinsic difficulty. The efficient allocation of samples between these stages is critical for achieving minimax optimal rates.

Empirical Results

Experiments conducted demonstrate that KIV significantly outperforms other nonparametric IV regression methods across various simulation settings. It excels particularly when the true structural function to be learned is smooth, which aligns with its theoretical minimax optimal guarantee. Compared to its counterparts like sieve IV, Nadaraya-Watson IV, and deep IV approaches, KIV offers a clear advantage in terms of flexibility and inference accuracy.

Implications and Future Directions

The introduction of KIV has critical implications for both theoretical and applied causal inference research. By successfully merging kernel methods with instrumental variable estimation, KIV provides a robust framework that is well-suited for a wide range of applications in economics and beyond, where nonlinear relationships are prevalent. This work suggests that RKHS methods can serve as a bridge between econometrics and machine learning, offering promising new tools for understanding complex causal relationships in data-rich environments.

Looking forward, there is a range of potential research paths. Exciting directions include extending the KIV framework to handle high-dimensional data more efficiently, improving computational scalability, and exploring its integration with emerging deep learning approaches. As the landscape of causal inference continues to evolve, KIV stands as a potent example of the innovative intersection between statistical rigor and computational sophistication.