LLMs for Causality: Methods & Applications

• LLMs for Causality is an emerging area that integrates causal inference techniques with language models to extract and explain cause-effect relationships from text and structured data.
• Studies reveal that performance on causal queries is highly sensitive to encoding schemes, with top models achieving F1 scores up to 0.77 on graph-level tasks.
• Hybrid approaches combining symbolic and neural methods, retrieval augmentation, and counterfactual reasoning are enhancing LLMs’ capabilities in causal discovery and explanation.

LLMs for causality constitute a rapidly developing area at the intersection of autonomous reasoning, knowledge extraction, and causal inference. Recent research evaluates LLMs’ capacity to answer, discover, and explain causal relationships—both from textual corpora and structured data—highlighting their promise, systematic limitations, encoding sensitivity, and the ongoing need for hybrid symbolic-neural and causality-aware approaches.

1. Formal Foundations: Causal Graphs, Interventional Logic, and LLM Task Formulation

LLMs are increasingly evaluated against the formal machinery of structural causal models (SCMs). An SCM is a tuple $$(\mathbf{U}, \mathbf{V}, \mathbf{F}, P(\mathbf{U}))$$, where $\mathbf{V}$ are observed variables, $\mathbf{U}$ exogenous noise, and $\mathbf{F}$ defines the directed acyclic graph (DAG) structure: $V_i = f_i(\textrm{Pa}(V_i), U_i)$ (Sheth et al., 2024, Wu et al., 2024, Zhang et al., 2023). The key inferential operator is Pearl’s do-calculus: $P(Y\,|\,do(X=x))$, the post-intervention distribution of $Y$ when $X$ is forcibly set to $x$ and confounders are blocked.

Task design for LLMs involves two main categories:

• Graph-level queries: extract sets of nodes by type (e.g. mediator, confounder) given a textual encoding of the DAG.
• Node-level (binary) queries: determine, for a node $v$, if it exhibits a given causal property (e.g., "Is node 3 a mediator?") (Sheth et al., 2024).

There is also considerable emphasis on LLM support for interventional reasoning (predicting the distributional shift under $\mathbf{V}$0) and counterfactual logic (predicting $\mathbf{V}$1 under hypothetical actions or alternate histories) (Sheth et al., 2024, Kıcıman et al., 2023, Ashwani et al., 2024).

2. Causal Query Benchmarks and Encoding Sensitivity

Recent large-scale benchmarks such as CausalGraph2LLM (Sheth et al., 2024), CausalBench (Zhou et al., 2024), and evaluations in "Causal Reasoning and LLMs" (Kıcıman et al., 2023) stress that the performance of LLMs on causal tasks is highly sensitive to how the causal graph or system is encoded textually.

Key encoding schemes tested include adjacency lists, adjacency matrices, JSON parent-lists, textual edge descriptions, and markup languages (GraphML, Graphviz). Encoding sensitivity, defined as the difference in average F1 between the best and worst formats for a given model ($\mathbf{V}$2), can reach 0.60 even for state-of-the-art models like GPT-4 (Sheth et al., 2024). JSON and GraphML formats, generally aligning with pretraining corpora, yield the best results for top models, while adjacency matrices often produce near-chance outcomes (F1 as low as 0.10–0.20).

Performance is stratified by query difficulty:

• Graph-level: F1 up to 0.77 for GPT-4 (GraphML), 0.71 for Gemini, 0.51 for GPT-3.5, and 0.34 for Mistral-7B (Sheth et al., 2024).
• Node-level (binary) queries: top models exceed 0.80 accuracy.

Encoding sensitivity persists even in downstream causal tasks such as interventional effect prediction, where the correct identification of post-intervention variable changes depends on the graph representation.

3. Strengths, Failure Modes, and the Nature of LLM "Causal Reasoning"

Though LLMs exhibit nontrivial performance when extracting or verifying memorized causal facts and chain-of-thought inferences, multiple studies highlight fundamental weaknesses:

• Surface Heuristic Reliance: LLMs often conflate mention order ("X before Y") or temporal proximity with causality (post hoc fallacy), overgeneralizing temporal associations to directional causation. This persists even as model scale increases, unless counteracted by direct supervision or explicit counterfactual statements (Joshi et al., 2024).
• Insensitivity to Counterfactuals: Standard model variants are markedly less accurate at inferring X→Y from "If X had not occurred, Y would not have occurred" templates relative to humans or even basic statistical rules (Joshi et al., 2024).
• Associational vs. Interventional Reasoning: LLMs predominantly model statistical association ($\mathbf{V}$3), performing poorly on interventional queries ($\mathbf{V}$4). This limitation aligns with Pearl’s Ladder of Causation, which places current LLMs at the associational rung rather than the interventional or counterfactual levels (Zhang et al., 2024).
• Bias from Parametric Memory: When variable names match commonsense priors, LLMs overestimate plausible edges, increasing F1 by 10–20 points but also systematically yielding false positives and overfitting to plausible but incorrect graph structures (Sheth et al., 2024).

Case study results show that small and open models barely exceed random-chance F1, while even strong closed-source models significantly degrade in larger or more ambiguous graphs (e.g., >50 nodes), colliders, or anti-commonsense edge-orientations (Zhou et al., 2024, Sheth et al., 2024).

4. Approaches to Causal Discovery and Hybrid Architectures

LLMs have been incorporated into causal discovery pipelines in several roles:

• Conditional Independence Oracle: LLMs are prompted to judge if "X is independent of Y given Z" and used as oracles inside the PC algorithm ("chatPC") (Cohrs et al., 2024). This yields variable accuracy and F1 depending on domain, graph size, and aggregation schema (majority vote vs. binomial test). GPT-4 with statistical aggregation approaches or exceeds mutual-information-based CI test baselines in benchmark systems but shows high variability and conservative bias.
• Score-Based Regularization: Multi-LLM collaboration frameworks inject edge priors from several LLMs into classical score-based methods (e.g., GES, NOTEARS) using soft $\mathbf{V}$5 or $\mathbf{V}$6 penalties on adjacency matrices, with mixture weights determined by data compatibility (Li et al., 2024). This consistently yields absolute improvements in Structural Hamming Distance (SHD), FDR, and TPR on classic Bayesian network datasets.
• Knowledge Graph Augmentation: Combining LLMs with metapath-based subgraphs from biomedical and open-domain knowledge graphs (KGs) and supervised Learning-to-Rank models improves the reliability and F1 of pairwise causal inference—by up to 44.4 points compared to standard retrieval or no-KG LLM prompting (Susanti et al., 10 Jun 2025).
• Retrieval-Augmented Prompting: Pattern-matched and embedding-based in-context example selection ("Pattern RAG", "kNN RAG") boosts F1, accuracy, and recall in sentence- and phrase-level causality extraction and detection by priming LLMs with highly relevant causal templates from a curated repository (Naduvilakandy et al., 29 May 2025).
• Symbolic-Neural World Models: Plug-in architectures integrating LLMs as planners with learned causal world models (latent-space SCMs admitting interventions) yield substantial gains in multi-step reasoned planning tasks, with causally aware approaches eclipsing pure LLM baselines as horizon grows (Gkountouras et al., 2024).

5. Applications: Explanation, Extraction, and Security

LLMs have been adapted to a variety of downstream applications that operationalize causal reasoning mechanisms:

• Counterfactual Explanation Generation: Multi-stage pipelines leverage LLMs for abductive reasoning over counterfactual sets, action-oriented explanation synthesis, and closed-loop verification, achieving validity and consistency upwards of 75–93% in tabular decision scenarios (Fredes et al., 2024).
• Biomedical Causality Extraction: In clinical guideline contexts, LLMs (GPT-4, Llama2) can extract phrase-level causal spans but lag domain-specialized transformers such as BioBERT in token-level precision/recall (F1 ≈ 0.60 for LLMs, 0.72 for BioBERT), partly due to prompt window limits, hallucination, and incomplete span alignment (Gopalakrishnan et al., 2024).
• Model Security and Interpretability: Causality-driven frameworks systematically analyze LLM safety mechanisms by modeling layers, neurons, or tokens as causal variables subject to interventions ($\mathbf{V}$7). Precision interventions (e.g., deactivating 1–2% of "critical" neurons in early layers) can deterministically break or restore safety constraints. Causal feature-based detection attains >95% F1 across various threat types (Zhao et al., 4 Dec 2025).

6. Integrating Causality Across the LLM Lifecycle and Future Directions

Systematic approaches for endowing LLMs with robust causal reasoning target multiple phases of model development and use:

• Token embedding debiasing: Counterfactual data augmentation and invariant risk minimization (IRM) objectives to reduce bias at low-level representations (Wu et al., 2024).
• Pretraining and Fine-tuning: Incorporating automatically generated intervention/counterfactual cases during pretraining; fine-tuning with causal effect or preference objectives; post-training methods (e.g., CAPT) that use event identification/intervention to remove spurious distributional shortcuts and enhance OOD generalization (Gui et al., 11 Jun 2025).
• Causal Modules and "Causal LLMs": Proposals for hybrid architectures where LLMs interface explicitly with SCM APIs for effect computation, causal discovery, and assumption verification, or where causal modules are embedded natively within transformer layers to support do-queries and counterfactual reasoning natively (Zhang et al., 2023, Wu et al., 2024).
• Explicit/Implicit Reasoning Fusion: CARE-CA style frameworks combining explicit KG-based causal edge scoring, contextual counterfactual statement generation, and LLM-driven hypothesis selection/explanation have achieved state-of-the-art F1 and interpretability on a suite of causal benchmarks (Ashwani et al., 2024).
• Evaluation: Extensive new benchmarks (e.g., CausalGraph2LLM, CausalBench, CausalNet) and rigorous closed-loop evaluation methods (actionable explanation → new test instance → rule compliance) are now central to measuring causal understanding beyond raw accuracy.

Future work envisions integrating symbolic reasoning, graph-theoretic constraints, and do-calculus within the LLM lattice, scaling to large, heterogeneous real-world graphs, and guiding LLMs with structured priors and adapters that mitigate parametric memory and semantic bias (Wu et al., 2024, Joshi et al., 2024, Sheth et al., 2024). Embedding causality in LLMs remains a critical goal for achieving both technical reliability and alignment with scientific, policy, and social objectives.