Panel Data Estimation of Individual Demand in Markets with Many Consumers

Abstract: The purpose of this paper is to consider whether and how panel data can be used to estimate individual demand, as opposed to market-level demand, while accounting for simultaneity resulting from prices being determined in markets. We consider linear demand models and random coefficient demand models, together with linear supply models. We find that the bias of individual demand estimates obtained using familiar panel data methods, like differencing, disappears as the number of consumers in each market grows, as long as the time-varying, i.e. idiosyncratic, component of preferences is orthogonal to the unobserved, time-varying component of supply. This approximate control is assumed in many panel discrete choice models and is plausible in other models where idiosyncratic preferences represent random variation in preferences over time. Macroeconomic effects can be allowed for by including regressors characterizing time effects, such as trends and time period dummies, or fixed time effects.

• The paper establishes that bias in individual demand estimates vanishes at rate O(1/M) as the number of consumers increases.
• It shows that panel estimators yield near-consistent price sensitivity measures in both linear and random coefficient demand models without using instrumental variables.
• The analysis confirms that standard fixed effects techniques accurately estimate micro-level demand even amidst simultaneous price and quantity determination.

Panel Data Estimation of Individual Demand in Large Consumer Markets

Overview

This paper addresses the longstanding econometric challenge of estimating individual-level demand in markets where prices are determined simultaneously with quantities via market equilibrium, creating endogenous price determination even at the individual observation level. The research critically investigates whether standard panel data methods, particularly fixed-effects or first-differencing estimators, can yield unbiased estimates of individual price sensitivity without the use of instrumental variables, provided the market structure comprises a large number of consumers.

The authors analyze linear demand and supply models, as well as random coefficient (heterogeneous slope) extensions, both with and without explicit time effects. Strong theoretical results are established under general conditions, showing that bias from price endogeneity diminishes at an explicit rate as the number of consumers per market grows, provided idiosyncratic demand shocks are orthogonal to supply shocks.

Theoretical Contributions

The core contribution is a rigorous characterization of the rate at which the bias in fixed-effects estimates of individual demand (specifically, price elasticity) vanishes with increasing market size. For linear demand models with additive fixed effects, standard panel data estimators—such as differencing or within transformations—are shown to yield near-consistent estimates as $M$ (the number of consumers per market) increases, under the key assumption that the idiosyncratic, time-varying preference component is independent of time-varying supply shocks.

This is formalized in a sequence of theorems (see Theorem 1 & 2 in the text), quantifying the bias as $O(1/M)$:

$$plim(\hat{\beta}) = \beta + \frac{1}{M} \frac{-1}{\beta - \delta} \frac{Var(\Delta \eta_{it})}{Var(\Delta p_t)}.$$

With random coefficient demand models, analogous results hold for identification of the average price elasticity under similar orthogonality assumptions.

The framework is extended to accommodate macro-level time effects (trends, seasonal shocks), demonstrating that two-way (individual and time) fixed effects models retain the approximate unbiasedness result if individuals in the sample are drawn from separate markets, preserving sufficient cross-sectional variation in prices.

Additionally, in random coefficient settings, the OLS estimator for average price effects remains asymptotically unbiased as $M \to \infty$, again under appropriate independence restrictions.

Strong Numerical Results and Claims

The paper makes explicit quantitative claims regarding the rate at which endogeneity-induced bias decreases, specifying that as $M$ becomes large, the bias in estimated price effects approaches zero at rate $O(1/M)$. This result holds not only for mean price elasticities but also in the presence of random coefficient heterogeneity.

Further, the analysis shows that instrumental variables (IVs) are not needed for identification of individual demand in these large-market settings, contrasting sharply with much existing empirical IO literature that relies on IV strategies for market-level demand identification.

The assumptions underpinning this result are notably weaker than the no-aggregate endogeneity conditions usually imposed; only independence between idiosyncratic preference shocks and supply shocks is required for the time-varying components, while arbitrary correlation is permitted for time-invariant preference heterogeneity and supply. This directly generalizes several panel discrete choice models in empirical microeconomics.

Implications and Relevance

Practical Econometric Implications:

The primary implication is the justification for using panel data methods—without IVs—for individual-level demand estimation in scanner data or other consumer-level transaction datasets, provided the cross-sectional dimension is rich (large $M$). This result directly supports and extends estimation strategies used in recent empirical work ("Demand estimation with scanner data", Chernozhukov et al., 2019).

Theoretical Economic Implications:

From a structural economics perspective, the findings clarify the circumstances under which econometric endogeneity from simultaneous price determination is negligible at the individual level, formalizing the intuition that the individual's impact on equilibrium prices disappears in "large" markets. This clarifies the boundary between micro-level and market-level econometric challenges.

Implications for Model Extensions:

The current analysis focuses on linear demand and supply systems, with random coefficient extensions. The paper highlights pathways for future work on more general, potentially nonlinear, demand and supply relationships, suggesting that similar rates and identification arguments may hold with appropriate technical modifications.

Limitations and Future Directions

The results are contingent upon the key orthogonality assumption for idiosyncratic time-varying preferences and supply shocks. While plausible in many settings (e.g., households with shocks uncorrelated with industry-level supply conditions), violations could occur when aggregate shocks simultaneously affect both sides of the market. The inclusion of general time effects (e.g., dummies or trends) partially addresses this for aggregate shocks, but not for unobserved time-varying confounders with more complex dependence structures.

Future research is needed to:

• Generalize these findings to nonlinear and highly nonparametric settings, particularly discrete choice frameworks and general equilibrium models.
• Explore the finite-sample behavior and robustness to deviations from the orthogonality assumptions, particularly in markets where $M$ is not "large" relative to the panel length or the cross-market sample.

Conclusion

This paper delivers a rigorous econometric foundation for the use of standard panel data methods to estimate individual demand in large markets without recourse to IVs, establishing that the bias from price endogeneity vanishes at an explicit rate as market size grows, provided idiosyncratic preference shocks are exogenous to supply. The results challenge the necessity of IV identification in high-dimensional consumer panels and clarify modeling requirements for credible micro-level demand estimation. This has far-reaching implications for empirical research using consumer transaction data and informs future extensions to more sophisticated market models.