Difference in Differences with Time-Varying Covariates (2202.02903v3)

Abstract: This paper considers identification and estimation of causal effect parameters from participating in a binary treatment in a difference in differences (DID) setup when the parallel trends assumption holds after conditioning on observed covariates. Relative to existing work in the econometrics literature, we consider the case where the value of covariates can change over time and, potentially, where participating in the treatment can affect the covariates themselves. We propose new empirical strategies in both cases. We also consider two-way fixed effects (TWFE) regressions that include time-varying regressors, which is the most common way that DID identification strategies are implemented under conditional parallel trends. We show that, even in the case with only two time periods, these TWFE regressions are not generally robust to (i) time-varying covariates being affected by the treatment, (ii) treatment effects and/or paths of untreated potential outcomes depending on the level of time-varying covariates in addition to only the change in the covariates over time, (iii) treatment effects and/or paths of untreated potential outcomes depending on time-invariant covariates, (iv) treatment effect heterogeneity with respect to observed covariates, and (v) violations of strong functional form assumptions, both for outcomes over time and the propensity score, that are unlikely to be plausible in most DID applications. Thus, TWFE regressions can deliver misleading estimates of causal effect parameters in a number of empirically relevant cases. We propose both doubly robust estimands and regression adjustment/imputation strategies that are robust to these issues while not being substantially more challenging to implement.

• The paper critiques conventional TWFE models by showing that time-varying covariates can invalidate the parallel trends assumption in DID analysis.
• It proposes alternative estimation methods, including regression adjustments and doubly robust strategies, to enhance causal inference.
• The study introduces diagnostic tools to detect specification bias and underscores practical implications for empirical econometric research.

Analyzing Difference-in-Differences with Time-Varying Covariates: Methodology and Challenges

The paper "Difference-in-Differences with Time-Varying Covariates in the Parallel Trends Assumption" addresses critical methodological issues in applying difference-in-differences (DID) estimation techniques when covariates vary over time. The authors, Caetano, Callaway, Payne, and Rodrigues, provide a critical examination of the two-way fixed effects (TWFE) regression framework that is frequently employed in evaluating such models. Specifically, this paper meticulously highlights limitations of TWFE regressions, particularly under scenarios with time-varying covariates and how those might compromise the validity of causal estimates.

Key Contributions

1. Critique of TWFE Regressions:

The paper critiques prevalent reliance on TWFE regressions which assume that parallel trends exist after accounting solely for time-invariant covariates. It posits several scenarios where this assumption is violated: - Paths of untreated potential outcomes are contingent on the levels of time-varying covariates, not just their changes. - Linear models may incorrectly specify changes over time in the presence of complex covariate interactions.

1. Alternative Estimation Approaches: Proposed are alternative methodologies designed to overcome these limitations, such as regression adjustment and doubly robust estimation strategies that account for paths of untreated outcomes influenced by time-varying covariates.
2. Empirical Diagnostic Tools: The paper also derives straightforward diagnostic tools, beneficial for practitioners aiming to evaluate the extent of specification bias within TWFE models. These diagnostics involve scrutinizing the implicit regression weights to assess balance across covariates.

Numerical and Theoretical Implications

The authors underscore the weak interpretability and potential bias inherent in TWFE regressions when covariate trends are complex or misaligned with assumptions of linearity. They leverage detailed analytical proofs, illustrating the effects numerically with examples where TWFE could yield misleading estimates. They emphasize the context-specific factors influencing covariate effects on outcome trends over time, arguing that incorrect application of TWFE can lead to erroneous policy implications in empirical economics studies.

Challenges with TWFE and Proposed Remedies

This paper identifies several failure modes of TWFE regressions, such as negative weighting and weight-reversal issues, wherein ordinary regression adjustments may lead to distorted results due to imbalances in weighting units. By offering regression adjustments that incorporate historical levels of covariates, and advocating for doubly robust approaches that combine model-based extrapolations with these adjustments, they provide a comprehensive toolkit for achieving more reliable DID outcomes.

Future Directions

The implications of this research extend toward refining econometric models that incorporate more sophisticated treatment of time-varying covariates, emphasizing robustness against hidden biases. With increasing complexity in data structures due to technological advances and richer data collection methodologies, such insights are pivotal. The practical aspects of integrating machine learning techniques within these frameworks also suggest promising directions for future research, particularly in enhancing model flexibility without underwriting substantial parametric assumptions.

In summary, the paper provides a nuanced and technical examination of DID methodologies, urging careful reconsideration of TWFE regressions under conditions of time-variance in covariates. The methodical critique combined with alternative strategies marks a significant conceptual contribution to econometric analyses, offering empirical researchers a more robust analytic scaffolding to base causal inference.