Causal Token Injection Experiments

• Causal Token Injection experiments are methodologies that manipulate token representations in language and vision models to reveal and leverage causal effects on downstream outputs.
• They utilize interventions such as token swaps, counterfactual generation, and causal attention tuning, validated by metrics like perturbation success rate and attack success rate.
• These techniques offer practical benefits in fairness improvement, hallucination reduction, and bias mitigation while highlighting challenges like latent token interpretability and scaling.

Causal Token Injection experiments investigate the direct manipulation or augmentation of token representations, activations, or generation pathways in neural language and vision models to isolate, quantify, or leverage their causal effects on downstream outputs. This family of methodologies spans causal reasoning in chain-of-thought modules, counterfactual editing in autoregressive generation, fairness interventions, mechanistic hallucination mitigation in vision-LLMs, tokeniser bias estimation, and adversarial prompt injection for security analysis. A central objective is to determine whether certain tokens or token sequences serve as genuine carriers of reasoning, causal knowledge, or target attributes, as opposed to functioning as spurious or merely representational shortcuts.

1. Formal Foundations and Causal Frameworks

The core paradigm of causal token injection is formalized via structural causal models (SCMs) that treat discrete tokens (or embeddings) as endogenous variables within networked graphs of dependencies. In LLMs, token-level causal interventions are framed as do-operator manipulations on specific representations or sampled outputs, directly altering $E_S$ by an engineered perturbation $\Delta_S$ or forcibly setting token values at chosen positions. In models such as Chain-of-Continuous-Thought (COCONUT), explicit CoT tokens $r_k$ and latent token sequences $z_k$ are mapped to representations $e_t \in \mathbb{R}^d$ across transformer layers, enabling precise identification and modification of reasoning-relevant positions (Zhang et al., 25 Dec 2025). Autoregressive models employ explicit SCMs with noise terms (Gumbel-Max noise $U_i$) guiding counterfactual sampling (Chatzi et al., 2024), while causal fairness architectures compute per-token average treatment effects (ATE) under pairwise counterfactual replacement (Madhavan et al., 2023). In CLIP-style vision-LLMs, token granularity affords block-identifiability results, enabling the distinction of compositional nonidentifiability under SWAP, REPLACE, and ADD interventions (Chen et al., 30 Oct 2025).

2. Experimental Protocols and Intervention Techniques

Token injection experiments are constructed around tightly controlled perturbations, swaps, or counterfactual augmentations. Key methodologies include:

• Steering/Perturbation: Applying vector perturbations $\Delta_S$ to select token positions (explicit CoT or latent $z$), with the steering direction defined adversarially (e.g., via logistic probe weight orthogonal to the decision boundary), and injected at configurable layers pre-attention (Zhang et al., 25 Dec 2025).
• Token Swaps: Directly transplanting reasoning tokens ($z$ or $r$-steps) between samples and analyzing the effect on answer output and inconsistency rate (IR) (Zhang et al., 25 Dec 2025).
• Counterfactual Generation: Using SCMs, storing per-token RNG states/Gumbel draws to regenerate sequence tails under imposed token edits, yielding factual vs. counterfactual outputs (Chatzi et al., 2024).
• Causal Pair Pretraining: Harvesting large-scale cause–effect tuples (sentence/word-level) and injecting them via augmented pretraining (auxiliary loss on [CLS]) to imprint causal structure (Li et al., 2021).
• FCCT/IRI in Vision-LLMs: Intervening at visual/textual token activations at critical network components (MHSA, FFN) and layers to gauge recovery rates or reinforce activation profiles for hallucination mitigation (Li et al., 8 Nov 2025).
• Regression Discontinuity for Tokenisation Bias: Utilizing BPE or WordPiece subword orderings, the RD design quantifies the local causal effect of a subword’s vocabulary inclusion on character-string probability via cutoff-based measurement (Lesci et al., 3 Jun 2025).
• Prompt Injection Attacks: Compressing mathematical reasoning chains into minimal token injections to trigger specific vulnerabilities (e.g., reasoning cessation via special end tokens) and systematically evaluating attack success rates and compression efficiency (Cui et al., 29 Apr 2025).
• Causal Attention Tuning (CAT): Automated annotation of causal token links and regularization of attention weights during training to boost the ratio of attention on causal vs non-causal tokens at each layer (Han et al., 1 Sep 2025).

3. Metrics and Causal Evaluation

Quantitative analysis of causal injection relies on tailored metrics:

Metric Name	Formula / Calculation	Purpose
Perturbation Success Rate (PSR)	$\frac{1}{N} \sum_{i} \mathbf{1}[Y_i^{do}=target_i]$	Measures steerability via token intervention
Answer Inconsistency Rate (IR)	$$\frac{1}{N} \sum_i \mathbf{1}[Y_i^{orig} \ne Y_i^{swap}]$$	Quantifies output changes under token swaps
Shortcut Exploitation Fraction	Fraction of errors relying on biased/injected option/context	Detects shortcut usage vs genuine reasoning
Recovery Rate (RR), FCCT	$$\mathrm{RR}_{t,c}^{(\ell)} = \frac{P_{\mathrm{patched}} - P_{\mathrm{corrupted}}}{P_{\mathrm{clean}} - P_{\mathrm{corrupted}}}$$	Estimates causal effect at components/layers
Local Average Treatment Effect (LATE), RD	$$\tau_{\text{LATE}} = \lim_{x\downarrow c} \mathbb{E}[Y|X=x] - \lim_{x\uparrow c} \mathbb{E}[Y|X=x]$$	Tokenisation bias estimator
ATE (Token-Level)	$$ATE(x_i) = \mathbb{E}[Y|do(X_i=x_i,Z_i=1)] - \mathbb{E}[Y|do(X_i=x_i,Z_i=0)]$$	Causal contribution of token
Attack Success Rate (ASR)	$\frac{1}{\lambda |D|} \sum_{i=1}^{|D|} d_i$	Prompt injection effectiveness
Compression Rate (CR)	$\frac{\text{Avg}(|A_c|)}{\text{Avg}(|A_o|)}$	Efficiency of attack prompt compression
Attention Ratio, CAT	$C_i/N_i$	Attention allocated to causal vs non-causal

These metrics distinguish genuine causality from shortcut reliance, measure faithfulness of reasoning, and assess security and interpretability contributions.

4. Key Findings and Interpretations

Causal token injection experiments have surfaced several robust patterns:

• Latent Tokens and Reasoning Faithfulness: COCONUT-style latent tokens, while resistant to direct steering, lack critical causal influence; their injection or swap does not affect answers, implying that these tokens act as placeholders, with apparent reasoning performance instead attributable to shortcut exploitation (dataset artifacts) (Zhang et al., 25 Dec 2025).
• Counterfactual Generation Stability: Replaying Gumbel draws preserves local stability in sequence tails after token intervention, enabling precise bias analysis; moderate income and occupation shifts following sex/race interventions demonstrate built-in bias in Llama 3 and Mistral 8B (Chatzi et al., 2024).
• Causal Knowledge Injection: Self-supervised causal pair pretraining (CausalBERT approach) reliably enhances downstream causal classification and inference benchmarks, with minimal supervision and low risk of catastrophic forgetting (Li et al., 2021).
• Mechanistic Interpretability and Hallucination Mitigation: In LVLMs, FCCT localizes causal bottlenecks for cross-modal fusion to middle MHSA layers, with IRI leveraging these findings to achieve state-of-the-art hallucination reductions (POPE, MME, CHAIR) via targeted activation injection (Li et al., 8 Nov 2025).
• Tokenisation Bias: Subword vocabulary inclusion delivers an estimated $17\times$ probability boost to corresponding character-strings at the cutoff in 57M LLaMA-style models; the effect persists, though attenuated, at larger scales and across various tokenisation algorithms (Lesci et al., 3 Jun 2025).
• Prompt Injection Security: Minimal arithmetic reasoning chains can trigger reasoning interruption vulnerabilities (“thinking-stopped”), with compression yielding CR $\approx 60\%$ and ASR $\approx 100\%$ on some operations; defense via output-prefix hardening is effective (Cui et al., 29 Apr 2025).
• Causal Attention Tuning: CAT regularization creates a significant improvement in OOD generalization on synthetic and real math/QA benchmarks by enforcing concentrated attention on true causal tokens, as visualized in attention distributions (Han et al., 1 Sep 2025).
• Compositionality in Vision-Language: SWAP/REPLACE/ADD token-injection protocols show that classical CLIP objectives admit pseudo-optimal text encoders, which are invariant to atomic concept composition and insensitive to hard negatives, explaining compositional brittleness (Chen et al., 30 Oct 2025).

5. Practical Applications and Limitations

Applications span reliability diagnostics, interpretability, fairness, bias detection, detoxification, security analysis, and compositional reasoning stress-testing. Notable use cases include:

• Validation of LLM reasoning integrity (via steering and swap metrics).
• Counterfactual world-model interrogation for bias and diversity analysis.
• Plug-and-play detoxification and bias mitigation in generative LMs.
• Hallucination reduction and faithfulness improvement in LVLMs using FCCT/IRI methodologies.
• Vocabulary optimization and fairness in tokenisation scheme selection.
• Robust model benchmarking and improvement of contrastive alignment via compounded negative mining.

Limitations observed across experiments include the non-interpretable nature of certain latent tokens, scaling challenges, empirical nature of some annotation pipelines (CAT), local rather than global identification in RD tokenisation bias, and reliance on exogenous annotation LLMs potentially susceptible to their own biases.

6. Future Research Directions

Ongoing and open research areas include:

• Development of intervention-based benchmarks for reasoning faithfulness and shortcut detection in latent token modules (Zhang et al., 25 Dec 2025).
• Scaling counterfactual token generation techniques beyond 8B-parameter models and ensuring consistency with human world knowledge (Chatzi et al., 2024).
• Extending causal knowledge injection protocols to contextual, token-level supervision and deep context-dependent reasoning (Han et al., 1 Sep 2025).
• Refinement and generalization of FCCT/IRI, moving from static global interventions to dynamic, sample-specific causal tracing and injection (Li et al., 8 Nov 2025).
• Multilingual and larger-scale study of tokenisation bias including interference-ridden subword structures (Lesci et al., 3 Jun 2025).
• Expansion of compositional intervention protocols in multi-modal models and bridging modality gaps via compounded token and image edits (Chen et al., 30 Oct 2025).

This research landscape defines causal token injection as an indispensable benchmark, diagnostic, and improvement tool for next-generation reasoning and generative architectures in both NLP and vision-language domains.