Causal Sentence-BERT (CSBert) Model

• The paper introduces a novel ternary contrastive learning approach that transforms semantic embeddings into a geometric space for explicit causal reasoning.
• By employing cosine-based loss functions for cause-to-rationale and rationale-to-effect links, CSBert effectively discriminates true causal chains from spurious ones.
• Empirical evaluations demonstrate that integrating CSBert within ECCoT improves reasoning accuracy by up to 7%, enhancing the interpretability of large language models.

Causal Sentence-BERT (CSBert) is a sentence embedding model explicitly designed to encode causal relationships between textual statements. Developed as a core component within the ECCoT framework, CSBert transforms ordinary sentence embeddings into a geometric space where cause-and-effect relations become explicit: if sentence A causes sentence B, their embeddings are proximate under a causal metric, whereas non-causal or spurious pairs are distanced. This embedding facilitates precise evaluation, filtering, and ranking of reasoning chains generated by LLMs, thereby enhancing the interpretability and trustworthiness of automated inference (Duan et al., 24 Jun 2025).

1. Architectural Design

CSBert adopts and extends the Siamese "twin-tower" Sentence-BERT design into a triplet-based, or ternary, architecture for causal contrastive learning. Each instance during training consists of three sentences: a putative cause ("Q"), an intermediate rationale or effect ("R"), and a consequent or answer ("A"). All three are encoded using shared BERT weights (typically BERT-base or RoBERTa-base, with 12 transformer layers and a hidden size $H=768$).

Following token-level encoding, the model applies mean-pooling (or alternative [CLS]-pooling) to produce a single $H$-dimensional vector for each sentence. This is further projected via a linear layer to a lower embedding dimension $d$ (e.g., $d=256$):

$$e_x = W \cdot {\rm pooling}({\rm BERT}(x)) + b, \quad W \in \mathbb{R}^{d \times H}$$

The three resulting vectors—$e_Q$, $e_R$, $e_A$—feed into a ternary contrastive module, enabling the model to optimize over multi-hop causal structures rather than isolated pairs.

2. Learning Objective and Loss Function

CSBert is trained on triplets $\{(Q_n, R_n, A_n, y_n)\}_{n=1}^{N}$, with $y_n = 1$ if $(Q_n \rightarrow R_n \rightarrow A_n)$ constitutes a true causal chain, and $y_n = 0$ otherwise. The model defines cosine-based losses for the "cause$\rightarrow$rationale" ($L_{qr}^{(n)}$) and "rationale$\rightarrow$effect" ($L_{ra}^{(n)}$) links:

$$L_{qr}^{(n)} = 1 - {\rm cos}(e_{Q_n}, e_{R_n}), \quad L_{ra}^{(n)} = 1 - {\rm cos}(e_{R_n}, e_{A_n}),$$

where ${\rm cos}(u, v) = \frac{u^\top v}{\|u\|\|v\|}$ is the cosine similarity. The overall piecewise loss function is:

$\mathcal{L}_{\mathrm{CSBert}} = \frac{1}{N} \sum_{n=1}^N \begin{cases} L_{qr}^{(n)} + L_{ra}^{(n)}, & y_n = 1,\[6pt] \max(L_{qr}^{(n)}, L_{ra}^{(n)}), & y_n = 0. \end{cases}$

This objective pulls together embeddings of causal triplets and enforces that at least one link in a negative triplet is separated, sharpening the alignment to true causal structure.

3. Causal Alignment in Embedding Space

Minimizing the above loss restructures the embedding manifold so that valid causal sequences form chains of high cosine similarity. Given consecutive steps $s_t \rightarrow s_{t+1}$ in any reasoning chain, the model computes the causal similarity:

$${\rm sim}_{\rm causal}(s_t, s_{t+1}) = {\rm cos}(e_{s_t}, e_{s_{t+1}})$$

High values serve as a proxy for genuine causal links. This allows for systematic chain evaluation: entire candidate reasoning chains can be scored according to the aggregate (average or product) of their stepwise causal similarities. Chains with weak links (i.e., low similarity at any step) are filtered out.

4. Distinction from Standard Sentence-BERT

Vanilla Sentence-BERT (Reimers & Gurevych 2019) employs a two-tower architecture with contrastive or classification loss—typically aligned with semantic similarity or NLI tasks. CSBert introduces the following distinctions:

• Ternary (Triplet) Structure: Enforces pairwise constraints both from cause$\rightarrow$rationale and rationale$\rightarrow$effect, as opposed to mere semantic paraphrase/alignment.
• Causal Annotations as Supervision: Training signals explicitly encode causal relations rather than entailment or paraphrase similarity.
• Piecewise Loss for Negatives: The loss for negative triplets only requires that at least one step is discouraged, ensuring robust discrimination of spurious multi-hop sequences.

These adaptations result in an embedding space optimized for directional, multi-hop causality, demonstrated to yield much stronger alignment with human causal judgments under eye-tracking and rating protocols (Duan et al., 24 Jun 2025).

5. Integration within ECCoT Framework

Within ECCoT, CSBert operates downstream of the Markov Random Field-Embedded Topic Model (MRF-ETM), which generates plausible, topic-conditioned reasoning chains. For each candidate chain $$s_1 \rightarrow s_2 \rightarrow \cdots \rightarrow s_T$$, CSBert is applied as follows:

• Step Embedding: Each sentence $s_t$ is embedded as $e_{s_t} = {\rm CSBert}(s_t)$.
• Chain Scoring: The average link similarity is computed:

$$S_{\mathrm{chain}} = \frac{1}{T-1} \sum_{t=1}^{T-1} {\rm cos}(e_{s_t}, e_{s_{t+1}})$$

• Filtering by Order Statistics: Chain scores are aggregated across all $N$ candidate chains. Chains with scores below a quantile threshold (e.g., 25th percentile) are discarded.

ECCoT further visualizes the interplay between MRF-ETM topic signals and CSBert attention via cross-interaction heatmaps, confirming that topic coherence and causal alignment reinforce one another. Only chains passing both criteria are advanced for final answer generation, resulting in more interpretable and trustworthy output (Duan et al., 24 Jun 2025).

6. Empirical Impact and Ablation

CSBert’s contribution is quantifiable via ablation on standard benchmarks. Removing CSBert from the ECCoT pipeline yields the following accuracy drops:

Dataset	W/ CSBert	W/O CSBert	$\Delta$ Accuracy
ANLI	72.23%	69.22%	–3.01
SVAMP	92.72%	90.38%	–2.34
CommonQA	86.54%	79.46%	–7.08

A paired bootstrap significance test ($p<0.05$) confirms the declines are statistically significant, highlighting CSBert’s role as a principal driver of reasoning improvements within ECCoT. A plausible implication is that the embedding's causal structure is not only machine-actionable but also aligns with human reasoning under realistic decision-making tasks (Duan et al., 24 Jun 2025).

7. Significance and Research Context

The incorporation of CSBert within the ECCoT framework represents a substantive step toward interpretable, criteria-driven reasoning with LLMs. By transforming the semantic embedding space to express geometric facts about causality, CSBert enables principled filtering of reasoning chains and exposes causal dynamics for further analysis. This explicit causal alignment distinguishes it from prior embedding methods, providing a robust mechanism for improving trust in automated reasoning systems—especially when paired with topic modeling and chain-of-thought protocols (Duan et al., 24 Jun 2025).