Text-as-Treatment Experiments

• Text-as-Treatment experiments are causal designs where textual variations are directly manipulated to assess effects on perceptions, attitudes, and behaviors.
• They employ diverse interventions ranging from precise lexical substitutions to adjustments of latent textual features, ensuring semantic validity and controlling confounders.
• Advanced estimation techniques, including representation learning and design-based strategies, are used to mitigate issues like overlap failure and treatment leakage.

Searching arXiv for recent and foundational papers on text-as-treatment causal inference to ground the article. {"query":"text-as-treatment causal inference lexical substitution effect design-based solution LLM too large latent textual treatments arXiv", "max_results": 10} I found several relevant arXiv papers on text-as-treatment and adjacent causal-text methodology, including foundational and recent work. I’ll use these together with the supplied source papers to write the encyclopedia article. Text-as-treatment experiments are causal designs in which textual variation is the intervention and the outcome is an external response such as perception, attitude, behavior, or a downstream rating. In this formulation, text is neither merely a covariate nor merely an outcome; it is the object of intervention, whether as a whole document, a latent linguistic property, a lexical substitution, or a semantically structured generated variant. Across the literature, the central methodological problem is to define manipulable textual interventions precisely enough for causal interpretation while preserving overlap, controlling confounding textual content, and maintaining semantic validity of the intervention itself (Egami et al., 2018, Wang et al., 2018, Tierney et al., 9 Oct 2025, Shi et al., 24 Oct 2025).

1. Conceptual scope and object of intervention

A foundational statement of the field is that text can enter causal research in several distinct roles: as treatment, as outcome, or as the source of discovered measures used in causal analysis. When text is treatment, the assigned intervention is a document or message, but the scientific target is often not the raw text itself; it is a lower-dimensional representation $Z_i=g(T_i)$, where the codebook function $g$ maps a high-dimensional textual object into categories, topics, binary features, or latent scales. This move is motivated by the fact that researchers are usually interested not in whole-document identity, but in “some underlying feature of a document” or “some aspect or latent value of the text” (Egami et al., 2018).

The literature distinguishes several granularities of textual treatment. One line studies highly localized interventions, most clearly the lexical substitution $p=(w_i,w_j)$ applied to a sentence context, with the sentence as the unit of analysis and audience perception as the outcome. Another line studies latent linguistic properties such as intellectual humility, where the treatment is a feature of the text rather than exposure to one arbitrary document versus another. A third line treats AI-generated texts, prompts, headlines, ads, or recommendations as high-dimensional structured treatments endowed with semantic similarity, so that the experimental problem becomes one of learning over a large treatment space rather than comparing a few fixed arms (Wang et al., 2018, Tierney et al., 9 Oct 2025, Shi et al., 24 Oct 2025).

This scope matters because different formulations imply different causal questions. A design may ask for the effect of exposure to a labeled document, the effect of a latent textual feature holding all else fixed, the effect of a specific lexical replacement in context, or a conditional treatment effect $\tau(z,x)$ for a semantically structured treatment $z$ and user covariates $x$. The field is therefore unified less by a single estimand than by a shared commitment to treating textual variation as intervention rather than as passive measurement (Tierney et al., 9 Oct 2025, Shi et al., 24 Oct 2025).

2. Formal causal formulations and estimands

The formal basis of text-as-treatment work is standard potential-outcomes notation, adapted to high-dimensional language. In the general binary case, the individual treatment effect is

$$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$

with average treatment effect

$\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$

For text treatments, the difficult step is defining $\mathbf{X}$, $T$, and the intervention space in a way that is both causally coherent and linguistically meaningful (Wang et al., 2018).

In lexical text-as-treatment, the canonical estimand is the Lexical Substitution Effect, defined for a substitution pair $g$0 and sentence context $g$1 as

$g$2

Here, the context $g$3 is the sentence with the focal token removed, so the same substitution can have different causal effects in different sentence environments. This is the core move from average word association to context-specific causal effect estimation (Wang et al., 2018).

For latent textual properties, a central distinction is between the effect of document exposure and the effect of the feature itself. The former allows non-treatment textual content $g$4 to change with treatment status,

$g$5

whereas the latter holds that non-treatment content fixed,

$g$6

This distinction is fundamental because randomizing exposure to naturally occurring treated and control texts generally identifies $g$7, not $g$8 (Tierney et al., 9 Oct 2025).

A more general structured-treatment formulation models potential outcomes under treatment object $g$9 as

$p=(w_i,w_j)$0

with conditional average treatment effect relative to a control treatment $p=(w_i,w_j)$1,

$p=(w_i,w_j)$2

This accommodates settings where treatments are not binary but belong to a large semantic space, as with AI-generated headlines or prompts (Shi et al., 24 Oct 2025).

The field also emphasizes that when raw text is mapped to a lower-dimensional treatment representation $p=(w_i,w_j)$3, identification requires more than randomization of documents. One stated condition is the Sufficiency Assumption: for all $p=(w_i,w_j)$4 and $p=(w_i,w_j)$5 such that $p=(w_i,w_j)$6, one requires

$p=(w_i,w_j)$7

This is a hidden-versions-of-treatment condition for text: documents sharing the same latent treatment representation must be equivalent, at least on average, in all response-relevant respects (Egami et al., 2018).

3. Experimental design strategies

Text-as-treatment experiments span randomized, observational, and hybrid designs. A direct randomized strategy is to construct treatment and control texts that differ only in the target wording and collect human judgments on the resulting pairs. In lexical substitution work, this was implemented by selecting substitution pairs, sampling control sentences, generating treated versions by replacing one word, and collecting randomized ratings on Amazon Mechanical Turk. In that setting, lexical choice affected perceived neighborhood desirability in Airbnb and perceived author gender in Twitter and Yelp, and algorithmic estimates were benchmarked against 687 annotated tuples from randomized human judgment (Wang et al., 2018).

A design-based alternative for latent textual properties uses human writers, editors, and evaluators in sequence. The four-step design is: choose a latent treatment feature $p=(w_i,w_j)$8, topics, and outcomes; have writers generate original texts with or without the feature; have editors modify each text to switch treatment status while preserving everything else; and randomly assign evaluators to rate texts. Under the condition that edited versions satisfy $p=(w_i,w_j)$9 and $\tau(z,x)$0, the within-original-text estimator $\tau(z,x)$1 is unbiased for $\tau(z,x)$2 and can be identified with weighted least squares. In the humility application, this design was used to estimate effects on aggressiveness, informativeness, articulation, persuasiveness, and enjoyability of conversation (Tierney et al., 9 Oct 2025).

A different but related discipline is sample splitting. When treatment features or text-based outcomes are discovered from data, the training sample is used exclusively to learn the codebook function $\tau(z,x)$3, refine and label it, and the test sample is used only once for causal estimation with $\tau(z,x)$4 frozen. This is proposed as a defense against analyst-induced SUTVA violations and overfitting in both text-as-treatment and text-as-outcome settings (Egami et al., 2018).

Recent work extends design to LLM-mediated generation. One approach uses deterministic decoding in an open-source deep generative model so that the same prompt yields the same internal representation $\tau(z,x)$5 and the same text. Another uses sparse autoencoders to identify latent semantic features, steers generation along a decoder direction, measures realized concept intensity $\tau(z,x)$6, and then defines treatment from realized intensity rather than from nominal steering. A separate line treats semantically similar AI-generated treatments as a structured treatment space for adaptive experimentation rather than as unrelated experimental arms (Imai et al., 2024, Feldman et al., 17 Feb 2026, Shi et al., 24 Oct 2025).

4. Estimation methods and representation learning

One major family of estimators adapts individual treatment effect estimation to text. For lexical substitution, four observational or quasi-experimental estimators are proposed: K-nearest-neighbor matching on contextual similarity, Virtual Twins Random Forest, Counterfactual Random Forest, and Causal Forest. These methods use sentence-level observational data labeled with proxy outcomes, treating the substituted word as a binary treatment and the remaining context words as covariates. A second family reframes effect estimation as classification using a small amount of randomized-control-trial-labeled data and three domain-general features: context probability, control word probability, and treatment word probability. In Yelp, this causal perception classifier achieved Pearson correlation $\tau(z,x)$7 with human RCT estimates; in Airbnb, the best correlation was $\tau(z,x)$8, indicating a much harder and noisier domain (Wang et al., 2018).

A broader representation-learning response to apparent overlap violations is to avoid conditioning on the full text and instead learn a lower-dimensional representation sufficient for adjustment. One such proposal defines

$\tau(z,x)$9

where $z$0. The resulting Treatment Ignorant estimator combines outcome modeling and a propensity score on the two-dimensional representation $z$1, yielding asymptotic normality under $z$2-type nuisance-rate conditions. The key idea is that full text makes treatment deterministic, $z$3, whereas an appropriately chosen representation can restore overlap while preserving confounding information (Gui et al., 2022).

Another approach, GenAI-Powered Inference, assumes access to the “true internal representation” $z$4 from the generative model that produced or regenerated the text. The method learns a deconfounder $z$5 without a treatment-prediction loss, estimates arm-specific outcome regressions, and then applies a doubly machine-learned augmented inverse probability weighted estimator. Under stated separability and deterministic-decoding assumptions, the method identifies the average treatment effect of a binary text feature $z$6 while avoiding direct conditioning on $z$7, which would destroy overlap because treatment is deterministic in the text (Imai et al., 2024).

For semantically structured treatments, double kernel representation learning posits a low-rank factorization

$z$8

learned in RKHSs over treatment embeddings and user covariates. This supports both heterogeneous-effect estimation and adaptive experimentation. On Upworthy semi-synthetic experiments at rank 2, DKRL achieved test error $z$9 versus $x$0 for Structured Intervention Networks and $x$1 for a product-kernel baseline (Shi et al., 24 Oct 2025).

A separate deployment problem arises when structured confounders are available at training time but only text is available at inference time. There the target is

$x$2

identified via

$x$3

The proposed estimator uses doubly robust pseudo-outcomes computed from structured confounders $x$4, LLM-generated surrogate text $x$5, and a regression of pseudo-outcomes on text embeddings. On semi-synthetic medical data, this framework reduced PEHE from $x$6 to $x$7 on IST and from $x$8 to $x$9 on MIMIC-III relative to the best naive text-only baselines (Ma et al., 3 Jul 2025).

5. Identification threats, overlap failures, and treatment leakage

The major controversy in text-as-treatment methodology is that the treatment is often a component of the same object used for adjustment. If one conditions on the full text $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$0, then

$$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$1

so exact adjustment destroys overlap. This is the structural basis for the claim that “better prediction is not better causal identification”: a representation that captures treatment too well may create practical non-overlap even when it reduces omitted-variable bias (Tierney et al., 9 Oct 2025, Gui et al., 2022).

This tension is amplified in LLM-based adjustment. One design-based critique argues that methods learning representations predictive of both treatment and outcome can encode treatment itself and thereby induce overlap bias. On semi-synthetic data built from experimentally generated texts, BoW IPW often recovered the true ATE, whereas TextCause and Treatment Ignorant methods based on DistilBERT representations failed, with estimated propensities frequently near 0 or 1. The broader claim is not that bag-of-words always dominates, but that larger and more expressive models can be causally worse when the treatment is embedded in the text itself (Tierney et al., 9 Oct 2025).

A closely related problem is treatment leakage. If text $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$2 is used as a proxy for an unobserved confounder $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$3 but also contains information caused by treatment $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$4, then $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$5 is “both necessary for adjusting (as it is a proxy) yet it is also a post-treatment variable.” In the leakage formulation,

$$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$6

Simulation evidence shows that adjustment with non-distilled text can be worse than omitting the proxy entirely: with true treatment effect $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$7, omitting $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$8 gave $$\tau(\mathbf{x}) = \mathbb{E}[Y^{(1)} \mid \mathbf{X}=\mathbf{x}] - \mathbb{E}[Y^{(0)} \mid \mathbf{X}=\mathbf{x}],$$9, while adjusting with leaked text gave $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$0. Oracle distillation reduced bias to $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$1 (Daoud et al., 2022).

More recent work formalizes leakage with a distilled representation $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$2 satisfying

$\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$3

while preserving confounder information. Four mitigation strategies are proposed: similarity-based passage removal, distant supervision classification, salient feature removal, and iterative nullspace projection. The main empirical lesson is a bias-variance tradeoff: moderate distillation best balances leakage reduction against confounder retention, whereas overly stringent distillation degrades precision (Daoud et al., 30 Dec 2025).

Other threats are specific to learned treatment representations. Discovering $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$4 on the same data used for effect estimation can create Analyst Induced SUTVA Violations and overfitting; separability between treatment and confounding features may fail; and semantic substitutability or hidden versions of treatment may be poorly approximated by paraphrase resources, embeddings, or latent features. These concerns explain why several papers argue for sample splitting, human auditing, or explicit paired-text construction rather than purely post hoc adjustment (Egami et al., 2018, Wang et al., 2018, Imai et al., 2024).

6. Empirical domains, neighboring paradigms, and current direction

Empirical text-as-treatment work has moved across several domains. Lexical substitution experiments show that word choice alters perceived gender and desirability in context-dependent ways; for example, “shops” $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$5 “boutiques” or “boyfriend” $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$6 “buddy” can have very different effects depending on the sentence context (Wang et al., 2018). Design-based experiments on intellectual humility in political communication find that humility decreases aggressive ratings by $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$7 ($\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$8), informative by $\tau = \mathbb{E}[Y^{(1)}] - \mathbb{E}[Y^{(0)}].$9 ($\mathbf{X}$0), and persuades self by $\mathbf{X}$1 ($\mathbf{X}$2), yielding the substantive pattern that humility softens tone but weakens persuasive force (Tierney et al., 9 Oct 2025).

The expansion of AI-generated treatments has shifted attention toward treatment discovery and generation. One phrase-level discovery approach uses contextual embeddings plus CNN filters to recover clusters of influential phrases, such as “Wuhan virus” in censorship data and patterns like banking processes or use of contractions in CFPB complaints. Another end-to-end pipeline uses sparse autoencoders to identify steerable latent features, steers model activations by $\mathbf{X}$3, scores coherence with an LLM judge, and then estimates treatment effects after embedding residualization. In that setting, naive estimation suffers significant bias because text conflates treatment and covariate information; residualization of embeddings materially reduces ATE bias and CATE RMSE (Ayers et al., 2024, Feldman et al., 17 Feb 2026).

The neighboring literature clarifies the boundaries of the field. Text-as-mediator work studies paths such as $\mathbf{X}$4, emphasizing natural direct and indirect effects for “aspects of language” rather than treating text as treatment itself. Outcome-learning work in randomized trials uses LLMs to discover which latent outcomes are affected by treatment and then validates those themes with hold-out human coding and completeness diagnostics. Efficient inference for text outcomes combines machine prediction with a residual correction on a human-coded subset to preserve the human-defined estimand while improving power. These are not text-as-treatment designs, but they sharpen the broader lesson that discovered textual variables require explicit validation, sample splitting, and careful causal role assignment (Keith et al., 2021, Modarressi et al., 2 Mar 2025, Mozer et al., 2023).

Across this literature, a common direction is visible. Text-as-treatment experiments are moving from a small number of hand-designed message arms toward scalable causal analysis of lexical, semantic, and generated interventions; from brute-force randomized comparisons toward hybrid designs combining discovery, steering, and validation; and from generic predictive embeddings toward representations judged by overlap, semantic faithfulness, and causal admissibility rather than by predictive accuracy alone. The central unresolved problem remains the same: text interventions are high-dimensional, compositional, and meaning-dependent, so credible causal analysis requires jointly solving treatment definition, intervention validity, and estimation under severe representation-induced pathologies (Egami et al., 2018, Tierney et al., 9 Oct 2025, Shi et al., 24 Oct 2025, Feldman et al., 17 Feb 2026).