IV Co-Scientist: Multi-Agent LLM Framework for Causal Instrumental Variable Discovery

Abstract: In the presence of confounding between an endogenous variable and the outcome, instrumental variables (IVs) are used to isolate the causal effect of the endogenous variable. Identifying valid instruments requires interdisciplinary knowledge, creativity, and contextual understanding, making it a non-trivial task. In this paper, we investigate whether LLMs can aid in this task. We perform a two-stage evaluation framework. First, we test whether LLMs can recover well-established instruments from the literature, assessing their ability to replicate standard reasoning. Second, we evaluate whether LLMs can identify and avoid instruments that have been empirically or theoretically discredited. Building on these results, we introduce IV Co-Scientist, a multi-agent system that proposes, critiques, and refines IVs for a given treatment-outcome pair. We also introduce a statistical test to contextualize consistency in the absence of ground truth. Our results show the potential of LLMs to discover valid instrumental variables from a large observational database.

• The paper introduces IV Co-Scientist, a staged multi-agent pipeline that filters variables, generates instruments and confounders, critiques exclusion and independence, and grounds proposals in real data.
• Frontier models recovered canonical instruments with Conceptual Match scores up to 1.00, while critic agents often detected discredited candidates that proposer models reproduced.
• On five Gapminder applications, most instruments exceeded the F > 10 relevance threshold and normalized consistency scores of about 0.51–0.58 outperformed random proxies, but expert validation remains essential.

Motivation and problem statement

Instrumental variables (IVs) permit consistent estimation of causal effects under unobserved confounding, but identifying a valid instrument requires satisfying three conditions—relevance, exclusion, and independence—of which only relevance is statistically testable. The remaining conditions must be defended through domain knowledge, institutional insight, and creative hypothesis generation. The paper investigates whether LLMs, which encode broad interdisciplinary knowledge, can support this task. The authors frame LLMs as "thinking collaborators" for early-stage IV discovery rather than replacements for theoretical reasoning, and ground their evaluation in a real-world setting using the Gapminder database of over 500 socio-economic indicators.

Two-stage evaluation of LLM IV reasoning

The evaluation proceeds in two stages before any open-ended discovery. First, the authors test whether LLMs can recover canonical instruments from five well-studied treatment–outcome pairs drawn from economics and health sciences (e.g., military service → earnings, education → wages). Second, they test whether models avoid historically discredited instruments—such as rainfall as an instrument for GDP's effect on conflict, whose exclusion restriction has been extensively challenged—and whether a critic agent can detect such instruments when explicitly injected into a candidate list.

Five models are evaluated: GPT-4o, o3-mini, QwQ, Llama3.1 8B, and Llama3.1 70B. Recovery performance is scored with Exact Match (EM, embedding-based semantic similarity) and Conceptual Match (CM, an LLM judge rating conceptual equivalence to the literature instrument).

Model	Military service→Earnings (EM/CM)	Education→Wages (EM/CM)	Migration→Wages (EM/CM)
GPT-4o	0.74 / 1.00	0.82 / 1.00	0.40 / 0.74
o3-mini	0.73 / 1.00	0.82 / 1.00	0.45 / 0.81
QwQ	0.74 / 1.00	0.73 / 1.00	0.31 / 0.70
Llama3.1 70B	0.61 / 0.84	0.67 / 1.00	0.57 / 0.77
Llama3.1 8B	0.28 / 0.42	0.48 / 0.76	0.35 / 0.60

Frontier models achieve CM scores of 1.00 on several pairs, indicating that they reliably propose conceptually equivalent instruments even when phrasing differs from the literature. This supports the claim that strong LLMs have internalized standard causal reasoning about canonical settings.

On invalid instruments, results are more mixed. Powerful models (GPT-4o, o3-mini, QwQ) frequently propose the flawed instrument themselves—for example, all three suggest rainfall-type instruments for GDP → conflict—yet the CriticAgent often detects and rejects them post hoc. A notable asymmetry emerges with Llama3.1 8B: it rarely proposes flawed instruments but fails to flag them when injected, suggesting that conservative generation does not imply sound critique. The practical implication is that an automated critic stage provides meaningful protection even when the proposer reproduces invalidated examples from the literature.

The multi-agent discovery pipeline

IV Co-Scientist is a sequential pipeline in which each agent refines the previous stage's output:

• PreSelector: filters variable pairs by Pearson correlation ($|\rho| > 0.7$) and effective sample size.
• HumanProxy: an LLM selects socio-economically meaningful, policy-relevant pairs.
• CausalOracle: combines LLM causal-direction judgments with Granger causality tests; only pairs where both agree on direction are retained as $(T \rightarrow Y)$.
• HypothesisGenerator: proposes $i = 5$ candidate IVs and $j = 5$ candidate confounders per pair.
• CriticAgents: two independent agents evaluate each candidate on exclusion and independence; only candidates passing both enter $\mathcal{Z}_{\text{valid}}$.
• Grounder: maps abstract proposals to concrete proxy variables in the dataset; ungrounded candidates trigger re-invocation of the discovery stages.

Evaluation without ground truth

Because discovered triplets are novel, the authors introduce two metrics. Relevance is measured via the first-stage F-statistic with heteroskedasticity-consistent estimators, using the conventional threshold $F > 10$. For validity beyond relevance, they propose a consistency metric: if multiple instruments isolate genuine exogenous variation, their 2SLS estimates should agree, so $$\Delta_{\text{LLM}} = |\hat{\beta}^{(Z_1)} - \hat{\beta}^{(Z_2)}|$$ should be small relative to a null distribution built from randomly sampled proxy pairs. The normalized score $$\mathcal{C}_{\text{norm}} = |\Delta_{\text{LLM}} / \Delta_{\text{Rand}}|$$ compares against this negative control. An appendix decomposition shows the pairwise difference reflects both estimator variance (inflated by weak instruments) and bias differences between instruments, connecting the metric formally to instrument strength and validity. The approach is inspired by self-compatibility testing in causal discovery [(Jiralerspong et al., 2024) context; Faller et al., ICML 2024].

On five autonomously generated Gapminder pairs (GDP → Health, Income → Emissions, Sanitation → Mortality, Poverty → Cholesterol, Female literacy → Kids), most suggested IVs exceed the F-statistic threshold (values range roughly 10.3–19.8), and $\mathcal{C}_{\text{norm}}$ values fall consistently below 1 (roughly 0.51–0.58), indicating LLM-suggested instruments produce more internally consistent effect estimates than random proxies. Posterior density plots of the ATE visually corroborate this: LLM-suggested IVs yield coherent, overlapping posteriors while random proxies yield diffuse, inconsistent ones. These results provide indirect evidence of quality, though the authors are careful to note that consistency does not confirm any single instrument's validity—an important caveat given that exclusion and independence remain untestable from observational data alone.

An ablation over the number of proposed instruments $i$ and hypothesized confounders $(T \rightarrow Y)$0 shows that $(T \rightarrow Y)$1 leaves independence checks under-specified (F ≈ 8.27, $(T \rightarrow Y)$2 = 0.62), while $(T \rightarrow Y)$3 makes them overly restrictive; $(T \rightarrow Y)$4 raises relevance (F = 16.49) at substantially higher token cost with marginal consistency gains. The default $(T \rightarrow Y)$5 configuration offers the best trade-off.

A qualitative evaluation with a faculty-level economist found the CriticAgents' reasoning generally sound, with an instructive observation: accepted and rejected instruments often differ not in validity but in generality—accepted ones tend to be broader and less debated, while rejected ones are more specific and carry known critiques.

Limitations and open questions

The authors concede several constraints explicitly. Evaluation relies on proxy metrics because ground-truth causal effects are rarely observable, so the consistency score offers indirect rather than confirmatory evidence. The framework is tested only in relatively well-understood domains (macro-level socio-economic data), and generalization to noisier or less-documented fields remains untested. The linear 2SLS setting excludes heterogeneous or nonlinear effects; although the appendix discusses LATE interpretation, the main experiments assume linear noise. Additionally, the consistency metric depends on having at least two grounded instruments per pair, and the reliance on LLM-as-judge for Conceptual Match inherits known concerns about circularity in LLM-based evaluation. Whether the framework can discover instruments that survive adversarial expert scrutiny in less-documented domains is left open.

Conclusion

This paper makes two contributions: a staged empirical assessment showing that frontier LLMs can recover canonical instruments (CM up to 1.00) and that critic agents mitigate the tendency to reproduce discredited ones, and a multi-agent system, IV Co-Scientist, that performs end-to-end IV discovery on real observational data with a novel consistency-based evaluation metric. The empirical finding that LLM-suggested instruments exhibit normalized consistency below 1 against random controls positions LLMs as plausible early-stage collaborators in instrument discovery, while leaving the verification of exclusion and independence—the core difficulty of the task—firmly in the hands of domain experts.