Treatment effect: a critique

Abstract: Two broad positions within statistics define a treatment effect, on the one hand, as a parameter of a statistical model, and on the other, as an appropriate population-level difference in outcomes or counterfactual outcomes under the different treatment regimes. This short expository paper presents some simple but consequential insights on the two formulations, contrasting the answers under the most favourable fictitious idealisation for the counterfactual framework. These observations clarify the relationship between Fisherian model-based inference and modern counterfactual formulations, and emphasise concerns, raised by Cox and others, regarding the suitability of model-free definitions as targets of inference when scientific conclusions are intended to generalise beyond the observed sample. Parts of the paper are necessarily controversial; we follow Cox (1958a) in not putting these forward in any dogmatic spirit.

• The paper critically examines whether the average treatment effect should be defined as a statistical model parameter (model-based) or a model-free population-level contrast of potential outcomes (counterfactual estimand).
• It demonstrates that under maximal observability, the standard average treatment effect recovers a stable, well-defined treatment parameter only within transformation families satisfying specific group-action conditions.
• The analysis reveals that model-free approaches often conflate the treatment effect with sample-specific nuisance structure, potentially undermining generalisability.

Overview

Battey and Edgar's expository paper examines a foundational tension in statistical methodology: whether a treatment effect should be defined as a parameter of a statistical model, in the Fisherian tradition, or as a model-free population-level contrast of potential outcomes, as in the Rubin causal model. The paper does not contest identifiability or consistency results for counterfactual estimands; rather, it asks whether such estimands are appropriate targets of inference when scientific conclusions are intended to generalise beyond the observed sample. Its central device is deliberately concessive to the counterfactual framework: the authors grant a fictitious idealisation under which both potential outcomes are observable and the mechanism generating them is fully known, so that objections about unobservability and unfalsifiability (Dawid, 2000; Popper, 1963) can be set aside entirely. Even under this maximally favourable setting, they show that the standard average treatment effect fails to recover a stable, well-defined treatment parameter except under restrictive conditions.

Model-based treatment effects

The model-based position treats the outcome distribution as belonging to a parametrised family $\mathcal{P}_\Theta$, with stable parameters capturing commonalities across units so that conclusions transfer to units not yet observed. Following McCullagh's formalisation, a treatment acts on the probability distribution of the outcome — not on the outcome itself — via the action of a group $\mathcal{G}$ on the baseline parameter space $\Gamma$. Additive shifts ($\Gamma = \mathbb{R}$) and multiplicative scalings ($\Gamma = \mathbb{R}^+$) are the canonical examples, with the null effect given by the group identity. Individual heterogeneity enters through baseline parameters $\gamma_i$ (e.g., $\gamma_i = w_i^\beta$ in regression), while the treatment effect itself remains a single stable group element across individuals; genuine variation by unit characteristics is accommodated through interaction terms while preserving stability.

The authors stress that this formalisation is detached from any causal mechanism: it is subject-matter context, not the mathematics, that determines whether causation is securely established.

The counterfactual estimand

The model-free alternative, characteristic of the Neyman–Rubin tradition, defines the treatment effect as

$$\tau_n := \frac{1}{n}\sum_{i=1}^{n} E_i\{Y_i(1) - Y_i(0)\},$$

estimated via imputation-based outcome models or propensity-score reweighting under no unmeasured confounding. Under the fictitious idealisation, this reduces to a matched-pair problem in which each individual is twinned with a perfect copy randomised to opposite arms. The question posed is then precise: when does $\tau_n$ recover the model-based treatment parameter?

Five examples

The paper's analytical core is a sequence of five idealised examples:

• Additive location model: $Y_{i0} = \gamma_i + \varepsilon_i$, $\mathcal{G}$0. Here $\mathcal{G}$1 exactly; the two formulations coincide.
• Exponential rate scaling: rates $\mathcal{G}$2 versus $\mathcal{G}$3. After log transformation, $\mathcal{G}$4, again compatible.
• Weibull with shape $\mathcal{G}$5: the same multiplicative structure yields $\mathcal{G}$6. The true effect is identified only up to sign unless $\mathcal{G}$7 is known — semi-compatible.
• Exponential additive shift: rates $\mathcal{G}$8 versus $\mathcal{G}$9. Then $\Gamma$0, which depends explicitly on the sample through $\Gamma$1: with $\Gamma$2, baselines near zero give large $\Gamma$3 while large baselines give negligible values. A well-defined, stable treatment problem is converted into one unstable under resampling of individuals, precluding extrapolation.
• Logistic binary outcomes: an additive log-odds effect $\Gamma$4 produces a complicated dependence of $\Gamma$5 on $\Gamma$6 except at the null.

The general lesson is crystallised in three conditions under which the two definitions agree for every sample: the group action must operate through the baseline parameter space; it must be describable by the same action on observations; and no nuisance parameters beyond $\Gamma$7 may enter the relevant expectation. These conditions characterise transformation families. Outside this class, $\Gamma$8 conflates the treatment effect with sample-specific nuisance structure.

A notable corollary concerns the "sharp null" $\Gamma$9 often attributed to Fisher. Citing Bailey's historical correction, the authors establish that Fisher's actual formulation rested on unit treatment additivity on an appropriate scale — a model-based notion — and that the Fisherian treatment effect typically violates the sharp null even when counterfactuals are fully observable. This also resolves Holland's earlier confusion regarding Cox's position.

Assessment of arguments favouring the model-free definition

Three standard defences of $\Gamma = \mathbb{R}$0 are addressed directly. Against A.1 (accommodation of individual-level effect heterogeneity), the authors agree with Ding and Dasgupta that constant effects on the expectation scale are unrealistic for binary outcomes, but argue the remedy is a stable parameter on an appropriate scale (log-odds) plus interactions, not a composite whose value drifts with the sample. On A.2 (model-freeness), they concede that misspecification risk is the most legitimate objection to model-based inference, but note that misspecification likewise destroys stability, underscoring the importance of model adequacy checks. On A.3 (double robustness), the point is blunt: double robustness is a property of estimators conditional on accepting $\Gamma = \mathbb{R}$1 as the target, and it is precisely that acceptance which the paper challenges.

Implications for the wider literature

The instability of $\Gamma = \mathbb{R}$2 is presented as the implicit driver of several recent developments: Jin and Rothenhäusler's conditionally valid finite-population inference responds to the fact that sub-population versions of $\Gamma = \mathbb{R}$3 are more relevant than population averages, a need the authors attribute to the estimand's inherent instability; Chernozhukov et al.'s conditional influence functions similarly refine covariate-conditional efficiency. Inverse probability weighting and imputation address imbalance and missingness problems that do not arise in the model-based formulation, where maximum-likelihood inference on treatment parameters requires no covariate balance absent unmeasured confounding, and where Fisherian conditioning on observed covariates (or ancillaries) already accounts for extreme imbalance in reference sets. Averaging over all observed covariate values is characterised, from the Fisherian standpoint, as too unconditional.

Limitations and open questions

The paper is candid about its own standing. It claims no originality for the underlying points, presenting them instead in sharpened form via idealised examples; parts are acknowledged as necessarily controversial and offered without dogmatism, following Cox (1958a). The fictitious idealisation, while strategically useful, is stronger than anything available in practice, so the analysis isolates measurement issues but cannot by itself adjudicate the falsifiability critique of counterfactuals, which the authors explicitly defer to prior work. The three compatibility conditions are stated for simple settings; their scope under more complex data structures (time-varying treatments, interference, continuous doses) is left unexamined. Nor does the paper offer a constructive programme for settings where no plausible stable parameter exists — arguably the regime the counterfactual framework was designed for.

Conclusion

The paper demonstrates that even under maximal observability of counterfactual outcomes, the model-free average treatment effect recovers a stable treatment parameter only within transformation families satisfying specific group-action conditions; otherwise it inherits sample-dependent nuisance structure that undermines generalisability. The contribution is clarificatory rather than novel in substance: it renders precise, through elementary examples, why Cox and others have resisted model-free averages as targets of scientific inference, and it situates several strands of contemporary causal methodology as responses to an instability inherent in the estimand itself.