Cross-Modal Causal Intervention in Code LLMs

• Cross-Modal Causal Intervention Module is a framework that quantifies and disentangles causal effects across multiple modalities using structural causal models and semantics-preserving interventions.
• It employs do-calculus, mediation analysis, and robust plug-in estimation to separate genuine semantic understanding from spurious pattern exploitation in code LLM outputs.
• The framework supports practical applications like prompt engineering, model evaluation, and causal tuning to enhance interpretability and robustness in multi-modal systems.

A Cross-Modal Causal Intervention Module is a structural component or suite of algorithms for systematically quantifying and disentangling the causal effects of multiple modalities (e.g., natural language, code syntax, input/output examples) on the output of large multi-modal LLMs. This module, as articulated in CodeSCM (Gupta et al., 7 Feb 2025), employs an explicit structural causal model augmented with do-calculus interventions, mediation analysis, robust estimation procedures, and empirical ablations to interpret and separate genuine multi-modal understanding from spurious pattern exploitation. It facilitates both theoretical insight and practical guidance for prompt engineering, model evaluation, and causal tuning in multi-modal code generation.

1. Structural Causal Model for Multi-Modal Code Generation

The CodeSCM module formalizes code generation with a directed acyclic graph linking observed modalities and latent mediators. The endogenous variable set is

$$\mathrm{V} = \{\mathrm{NL}, \mathrm{Code}_{al}, \mathrm{Code}_{nl}, \mathrm{I/O}, \mathrm{M}_{NL}, \mathrm{M}_{Code}, \mathrm{R}\}$$

• NL: natural language instructions (e.g., docstrings)
• Codeₐₗ: algorithmic code channel (function headers, code syntax)
• Codeₙₗ: natural-language code channel (descriptive function names)
• I/O: example input/output pairs
• M_NL: latent semantics of NL (mediator)
• M_Code: latent semantics of code (mediator)
• R: model’s generated code

The causal graph is:

• NL → M_NL → R
• Codeₐₗ → M_Code → R
• Codeₙₗ → {M_NL, M_Code} → R
• I/O → M_Code → R

Structural assignments are

$$\begin{align*} \mathrm{NL} &= f_{NL}(U_{NL}) \ \mathrm{Code}_{al} &= f_{AL}(U_{AL}) \ \mathrm{Code}_{nl} &= f_{NLcode}(U_{NLcode}) \ \mathrm{I/O} &= f_{IO}(U_{IO}) \ \mathrm{M}_{NL} &= g_{NL}(\mathrm{NL}, \mathrm{Code}_{nl}, U_1) \ \mathrm{M}_{Code} &= g_{Code}(\mathrm{Code}_{al}, \mathrm{Code}_{nl}, \mathrm{I/O}, U_2) \ R &= h(\mathrm{M}_{NL}, \mathrm{M}_{Code}, U_3) \end{align*}$$

where $U_{\cdot}$ are exogenous/noise variables.

2. Modality-specific Do-Interventions and Dead-Edits

Causal effects are probed via do-operator interventions on each modality:

• X ∈ {NL, Codeₐₗ, Codeₙₗ, I/O} is set to:
  • 1: original input plus a semantics-preserving "dead edit" (does not alter meaning)
  • 0: original input only
  • −1: removed entirely (set to NULL)

Operational definitions: | Modality | X=1 (dead edit) | X=0 (original) | X=−1 (removed) | |---------------|-----------------------|---------------------|------------------------| | NL | S+DS (dead string) | S | NULL | | Codeₐₗ | Cₐₗ+C_DC (dead code) | Cₐₗ | NULL | | Codeₙₗ | Cₙₗ+DN (dead name) | Cₙₗ | NULL | | I/O | inequality equiv. | original asserts | NULL |

"Dead" edits are designed to preserve the semantics of mediators, ensuring valid causal mediation analysis.

3. Causal Mediation Decomposition: TE, NDE, NIE

For each modality, the total effect (TE) of "adding back" (X:−1→0) is decomposed:

• TE (Total Effect): $TE_X = E[Y \mid do(X=0)] - E[Y \mid do(X=-1)]$
• NDE (Natural Direct Effect): $$NDE_X = E[Y_{X=0, M \leftarrow M_{X=-1}}] - E[Y_{X=-1}]$$
• NIE (Natural Indirect Effect): $$NIE_X = E[Y_{X=0}] - E[Y_{X=0, M \leftarrow M_{X=-1}}]$$

Here $M$ denotes the latent mediators and causal mediation analysis quantifies direct (spurious) and indirect (mediated semantic) pathways. Under the semantics-preserving design, natural direct effect is equivalent to the path-specific direct effect.

4. Estimation and Evaluation Procedures

Evaluation on fixed LLMs (no continued learning) is conducted by empirical plug-in estimation:

1. For each test example and modality X:
   • For x ∈ {−1, 0, 1}, construct prompt $P_x$ with intervention $X \leftarrow x$.
   • Generate code $R_x = \text{LLM}(P_x)$.
   • Record binary label $y_x = 1$ iff $R_x$ passes all correctness tests.
2. Estimate
   • $TE = \text{mean}(y_{X=0}) - \text{mean}(y_{X=-1})$
   • $DE = \text{mean}(y_{X=1}) - \text{mean}(y_{X=0})$

Assumptions:

• No omitted confounders in modality-mediator-output
• Dead edits fully preserve M
• Consistency: interventional outcome matches empirical result

5. Empirical Results: Modal Impact and Sensitivity

Main findings on pass@1 drop (percentage-points) for GPT-4-Turbo, WizardCoder-15B, LLaMa-3-8B on HumanEval+, mMBPP+, CoderEval-SCP:

Model ∖ Modality	HumanEval+	mMBPP+	CoderEval-SCP	Mean TE	Mean DE
NL	42.1, 1.2	19.1, 4.3	20.0, 2.9	27.7	2.8
Codeₐₗ	1.8, 1.2	1.3, 4.0	8.6, 0.0	3.9	1.7
Codeₙₗ	18.9, 1.8	42.9, 2.8	0.0, 2.9	20.6	2.5
I/O	5.5, 2.4	12.3, 6.3	N/A	8.9	4.3

• On HumanEval+, NL has the highest TE.
• On mMBPP+ Codeₙₗ (naming) exceeds NL.
• I/O pairs show highest DE.
• Codeₐₗ crucial for Java (CoderEval-SCJ), with pass@1 nearly 0 if removed.

Ablation and robustness checks:

• Different dead-edit strategies produce nearly identical DE values.
• Memorization is detected: pass@1 remains 5–10 pp even when NL is removed, indicating dataset leakage.

6. Insights for Model Design and Interpretability

The mediation module structure (M_NL, M_Code) isolates semantic understanding from spurious behavior, supporting interpretability of multi-modal LLMs. High I/O pair impact suggests it is beneficial to augment code LLMs with explicit I/O embeddings or specific “unit-test tokens”. The dead-edit intervention paradigm provides a generalizable blueprint for designing prompt interventions and causal-tuned objectives. Prompt engineering can be guided quantitatively: function headers, docstrings, and I/O pairs can be leveraged according to their causal marginal impact. These interventions are transferable to reinforcement learning or prompt-tuning regimes to target desired pathways (maximize NDE/minimize spurious DE).

7. Broader Context and Methodological Implications

CodeSCM establishes a rigorous toolkit for quantifying cross-modal causal effects in code LLMs—a methodology extendable to other domains such as multi-modal VLMs, medical report generation, and video question answering. The empirical isolation of direct versus mediated effects supports systematic deconfounding in model evaluation, model design, and adaptive training. The approach emphasizes semantics-preserving interventions for causal probing, and provides actionable insight for enhancing model robustness and fairness across modalities.