LLM Explainability with Counterfactual Chains and Causal Graphs

Abstract: Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses LLMs to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with $σ$-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.

• The paper introduces a causal framework for LLM explainability using model-driven counterfactual chains and discriminative concept discovery.
• The methodology constructs a process-level causal graph by iteratively intervening on high-level concepts to validate predictive fidelity and structural stability.
• Empirical findings show stable, model-specific causal structures that emphasize the importance of process-level auditing in high-stakes applications.

Concept-Based Causal Explainability of LLMs via Counterfactual Chains

Motivation and Problem Setting

Recent advancements in LLMs have surfaced a fundamental challenge: inference processes remain highly opaque, impeding trust and deployment in high-stakes domains. Existing interpretability paradigms, including feature attribution and attention-based methods, are fundamentally associational and inadequate for inferring causality in complex reasoning pathways. Methods rooted in mechanistic interpretability target low-level internals such as attention heads or circuits, but are misaligned with stakeholder needs for process-level, concept-driven explainability. The central objective of this work is to model LLM inference as a causal graph over high-level, human-interpretable concepts and to provide evidence for causal dependencies through iterative intervention using model-driven counterfactuals.

Methodological Framework

The proposed approach consists of a four-phase pipeline that automates the construction of a concept-level causal graph representing an LLM’s internal reasoning. The approach is fully model-driven: it leverages the target LLM to generate predictions, discover concepts, annotate examples, and produce counterfactuals.

1. Label Prediction: Ground-truth labels are supplanted by the LLM’s own predicted classes, ensuring that subsequent inference reflects the model’s epistemic stance.
2. Discriminative Concept Discovery: The LLM operates over class-balanced batches, proposing candidate concepts that differentiate between predicted classes. These concepts are filtered by relevance and discriminativeness, operationalized via distributions over possible class subsets for each concept.
3. MCMC-Inspired Counterfactual Expansion: To address sparse coverage in the observed concept state space, an MCMC-inspired algorithm generates chains of counterfactual texts. Each transition attempts a minimal intervention on a target concept’s alignment and is retained only if it induces the desired change without drifting non-target concepts above a given threshold.
4. Causal Graph Recovery: The expanded annotated dataset is used with the $\sigma$-CG algorithm, which is robust to discrete variables and cyclic dependencies, to recover a process-level directed graph from text through concepts to prediction.

This methodology is formalized over a vectorized concept annotation space, $\phi(x)$, that maps each input to a tuple of concept-dependent categorical variables, each representing alignment with subsets of possible classes.

Figure 1: The contrast between modeling real-world causal processes and the LLM-inference causal graph, which characterizes how the model internally maps text to prediction.

Evaluation Protocol

Ground-truth graphs for latent LLM inference do not exist, necessitating indirect validation methodologies. The framework introduces two central criteria:

• Predictive Fidelity: For each node in the causal graph (concepts and prediction), a multinomial logistic regression is trained to predict node states using only the states of its graph-based parents. This is compared exhaustively against regressors using all alternative concept subsets not containing the true parent set. The approach seeks to validate that discovered parents constitute the most informative predictors.
• Expansion Utility and Structural Stability: The KL-divergence between empirical distributions of joint concept states is tracked across iterations of MCMC expansion, with empirical convergence compared to theoretical bounds for perfectly overlapping and maximally orthogonal expansion. Structural stability is confirmed by measuring the Hamming distance between edge sets in discovered graphs over successive iterations.

  Figure 2: Four representative concept-level causal graphs illustrating model- and task-specific differences recovered by the framework.

  

  Figure 3: MCMC-based counterfactual expansion consistently improves mean predictive accuracy for both intermediate concepts and downstream predictions across all training regimes and datasets.

Empirical Findings

Causal Structure Consistency and Model-Specific Reasoning

On synthetic tasks (e.g., LIBERTY disease diagnosis), all models extract near-identical discriminative concepts and recover topologies consistent with the true generative process. However, on naturalistic tasks such as IMDB Sentiment Analysis and LAJ (Reddit preference judgment), both the concept inventories and the induced topological structures diverge across LLMs. Despite agreement on task-level outcomes, model-specific latent heuristics yield distinct chains of reasoning—a finding with implications for downstream fairness, accountability, and selection of LLMs for domain-specific deployment.

Validation of Causal Graphs

Across all combinations of benchmark, model, and variable, regressors conditioned on graph-based parents achieve the highest or tied classification accuracies in held-out cross-validation, outperforming all alternative predictor subsets in every setting. The parent sets discovered by the framework are especially robust under counterfactual expansion, maintaining ranking as top predictive combinations in the vast majority of folds.

Impact of MCMC Counterfactual Expansion

Counterfactual expansion is essential to robust causal discovery: augmenting observational data with targeted interventions vastly densifies the empirical distribution over the concept state space. Without these augmentations, causal discovery overfits spurious or sparsely realized dependencies. The KL-divergence analyses support the claim of true convergence: after sufficient iterations, the empirical distributional shift aligns with perfect-overlap bounds, and recovered causal graphs attain topological stability.

Figure 4: Per-concept predictive accuracy details; MCMC-based expansion results in a higher center-of-mass accuracy distribution compared to all-original or seed-only regimes for Gemini on Sentiment Analysis (IMDB).

Figure 5: Extracted causal graphs of the SA and DD task across models, demonstrating intra- and inter-model conceptual topology differences.

Figure 6: A selection of query-specific causal graphs for the LAJ (preference judgment) task reveals the heterogeneity of LLM-internal reasoning over diverse questions.

Practical and Theoretical Implications

The work operationalizes LLM explainability at the concept-mechanism level. By leveraging the LLM to both generate and audit its own interventions, this framework opens the pathway for scalable, model-native, process-level auditing with minimal reliance on external annotators or rigid a priori concept definitions. Critically, differences between model-recovered reasoning highlight that aggregate task accuracy is insufficient for model evaluation in regulated or safety-critical contexts; the validity of latent heuristics must be established and compared to stakeholder or domain expectations.

Theoretically, the approach demonstrates that concept-based causal discovery is possible with high specificity in operational domains, given sufficient intervention coverage and careful alignment of model view and data annotation. The data regime requirements for graph stability and the role of empirical convergence metrics reveal avenues for formal analysis in the claims of faithfulness and stability in LLM explanations.

Limitations and Future Directions

Key limitations include sensitivity to batch composition in concept extraction, a reliance on self-annotation and LLM-generated counterfactuals (which can propagate unfaithful rationales), and a validation approach that focuses on local Markov blankets. Extending the pipeline toward open-ended generation tasks, more challenging multimodal domains, and the integration of human audits remains future work. The framework must be extended to include external validation against ground-truth mechanisms when available and to support stakeholder-oriented, task-specific explainability constraints.

Conclusion

This work introduces a model-driven framework for extracting process-level, concept-based causal graphs from LLM inference, augmented via MCMC-inspired counterfactual chains to ensure stability and coverage. Empirical analysis demonstrates both predictive faithfulness and delineates model-specific causal structures on real-world and synthetic tasks. The methodology provides a foundation for next-generation explainability in LLMs, especially in applications where regulating the internal process is as crucial as the output.