DepBERT: Dependency-Aware Cause-Effect Extractor

Updated 6 July 2026

DepBERT is a dependency-aware transformer that integrates dependency trees and POS tags to extract cause and effect phrases at both token and span levels.
It employs a two-tower architecture combining standard contextual self-attention with dependency-restricted attention to dynamically fuse semantic and syntactic information.
Empirical results show significant improvements over conventional models in precision, recall, F1, and exact match accuracy across diverse causality datasets.

DepBERT is a dependency-aware transformer model for extracting cause and effect phrases from a single sentence at the token and span level. It extends a transformer-based token classifier by incorporating a sentence’s dependency tree and POS tags directly into the model framework through a two-tower architecture that combines standard contextual self-attention with dependency-restricted attention, followed by gated fusion and multi-class token labeling over the classes Special, Cause, Effect, and Other (Kabir et al., 14 Jul 2025).

1. Task definition and problem setting

DepBERT is designed for sentence-level causality extraction. Given a sentence $S$ with tokens $s_1, s_2, \ldots, s_N$ , and two non-overlapping phrases $u$ and $w$ that occur in $S$ , the objective is to identify $u$ as the cause phrase and $w$ as the effect phrase by assigning each token $s_t$ one of four labels: Special, Cause, Effect, or Other (Kabir et al., 14 Jul 2025).

This formulation is explicitly token-level rather than sentence-level. As a consequence, it supports multi-word spans for both cause and effect. The motivation is that causal arguments are often realized as noun phrases or longer constituents, not isolated tokens. The problem is operationally important in medicine, law, education, and scientific research. Examples given for the broader task include statements such as “Vitamin D deficiency contributes to insulin resistance and diabetes” and “Failure to adhere to safety regulations resulted in an accident.” In such settings, dependable extraction of causal links supports hypothesis generation, liability assessment, and automated assistance.

The model is situated against two established methodological tendencies. Unsupervised approaches have long used linguistic tools such as dependency trees, dependency relations, and POS tags to capture syntactic patterns associated with causality. Supervised approaches, by contrast, have typically relied on transformer-based token classifiers such as fine-tuned BERT. DepBERT targets the gap between these lines of work by integrating sentence-level syntactic structure directly into transformer attention rather than using syntax only as an external heuristic or feature source.

2. Two-tower architecture and dependency integration

DepBERT is organized as a two-tower transformer for token classification. The left tower is a standard transformer encoder, BERT-style, that learns contextual token embeddings from token co-occurrences using the usual input IDs and positional encodings. The right tower is a transformer encoder whose self-attention is restricted to edges in the sentence’s dependency graph and which also incorporates POS tags into the token representation. The final prediction is produced by a multi-class token classification head trained with cross-entropy over the four labels (Kabir et al., 14 Jul 2025).

The dependency tree is obtained with spaCy and represented as $G = (V, E)$ , where $V$ indexes tokens and each token carries a POS tag, while $s_1, s_2, \ldots, s_N$ 0 contains directed head-dependent relations. The tokenizer is modified so that each token $s_1, s_2, \ldots, s_N$ 1 is assigned three IDs: an input token id $s_1, s_2, \ldots, s_N$ 2, a positional id $s_1, s_2, \ldots, s_N$ 3, and a POS tag id $s_1, s_2, \ldots, s_N$ 4. Their embeddings are summed to form the initial token vector $s_1, s_2, \ldots, s_N$ 5. Both towers receive POS-enhanced token embeddings, but only the right tower applies dependency-restricted attention.

In the right tower, attention is computed only over tokens connected through dependency edges. Operationally, all non-neighbor tokens are masked, so attention normalization is carried out over the dependency neighborhood $s_1, s_2, \ldots, s_N$ 6 of token $s_1, s_2, \ldots, s_N$ 7. This creates a graph-attention-like mechanism over syntactic neighbors rather than a fully connected token graph. The outputs of the left and right towers are then combined through a learned gate, allowing the model to weight semantic and syntactic channels dynamically before classification.

A central architectural point is that DepBERT does not use a CRF layer. This distinguishes it from the SCITE baseline, which includes a BiLSTM-CRF. The reported gains are attributed instead to dependency-aware attention and POS integration.

3. Mathematical specification

The left tower uses standard transformer attention. For token $s_1, s_2, \ldots, s_N$ 8 with query $s_1, s_2, \ldots, s_N$ 9 and token $u$ 0 with key $u$ 1 in a $u$ 2-dimensional space,

$u$ 3

and value aggregation gives

$u$ 4

This is followed by Add & Norm and a feed-forward block (Kabir et al., 14 Jul 2025).

The right tower modifies attention so that it operates over dependency neighbors only. With summed embedding $u$ 5 and trainable matrices $u$ 6 and $u$ 7,

$u$ 8

For connected tokens $u$ 9 and $w$ 0, the affinity is

$w$ 1

Attention is normalized only over $w$ 2:

$w$ 3

The neighborhood-restricted aggregation is

$w$ 4

The two towers are fused with a learned gate. If $w$ 5 is the left-tower output and $w$ 6 is the right-tower output, then

$w$ 7

The fused embedding $w$ 8 is passed to the classifier:

$w$ 9

With one-hot target vector $S$ 0, the multinomial cross-entropy objective is

$S$ 1

This formalization makes the architectural intent explicit. The left tower captures unrestricted contextual co-occurrence, the right tower captures syntactic neighborhood structure, and the gate mediates tokenwise interpolation between them. A plausible implication is that the model can shift reliance between semantic context and dependency-local evidence depending on whether the sentence expresses causality through direct verbal governance, nominal triggers, or more distributed constructions.

4. Data, training regimen, and evaluation protocol

The experiments use three datasets. SemEval is constructed from SemEval-2007 Task 4 and SemEval-2010 Task 8 and contains 1,427 cause-effect sentences. SCITE contains 1,079 sentences exhibiting cause-effect relations. CAUSALGPT is a newly created dataset with 22,273 sentences generated by LLMs such as ChatGPT and Bard, with a medical domain emphasis. All three datasets are split 60% train, 30% test, and 10% validation, and each sentence contains exactly one cause-effect pair (Kabir et al., 14 Jul 2025).

DepBERT has 227M trainable parameters. The left tower is initialized from “bert-base-uncased,” while the right tower is trained from scratch with dependency-aware attention and POS embeddings. Optimization uses Adam with batch size 128 and learning rate 0.001. Training uses early stopping up to 1000 epochs with tolerance 10, and most models converge in fewer than 100 epochs. Sequence length limits, dropout, and seed or hardware are not reported. Results are described as coming from the initial stable runs.

Evaluation includes exact match accuracy, precision, recall, and F1. Exact match accuracy is strict: a prediction is counted correct only if all tokens of both cause and effect spans are correctly labeled, and no partial matches are allowed. The paper does not report token-level versus span-level breakdowns and does not report statistical significance tests.

Dataset	Size and split	DepBERT (Gated)
CAUSALGPT	22,273 sentences; 60/30/10	Precision 0.967, Recall 0.969, F1 0.963, Exact Acc 0.858
SemEval	1,427 sentences; 60/30/10	Precision 0.942, Recall 0.962, F1 0.957, Exact Acc 0.842
SCITE	1,079 sentences; 60/30/10	Precision 0.932, Recall 0.943, F1 0.939, Exact Acc 0.834

The implementation description leaves several details unspecified. The paper does not detail how wordpieces are aligned to dependency nodes, how multi-word spans are aggregated, or how parser noise is handled explicitly. It states that using spaCy’s dependency trees suffices empirically to yield improvements, and that the format of the dependency tree is not critical to performance, based on prior findings.

5. Empirical performance and component analysis

Across all three datasets, DepBERT consistently outperforms the reported baselines, including BERT, BERT+Dependency, BERT+POS, Sentence-BERT, LLaMA, BiLSTM, and BiLSTM-CRF. On CAUSALGPT, DepBERT (Gated) reaches Precision 0.967, Recall 0.969, F1 0.963, and Exact Acc 0.858, compared with BERT at F1 0.938 and Acc 0.811, BERT+Dependency at F1 0.897 and Acc 0.831, BERT+POS at F1 0.948 and Acc 0.822, Sentence-BERT at F1 0.954 and Acc 0.828, LLaMA at F1 0.954 and Acc 0.828, BiLSTM at F1 0.847 and Acc 0.778, and BiLSTM-CRF at F1 0.849 and Acc 0.781. Relative improvements over Dasgupta are indicated as 13.7% for F1 and 10.3% for accuracy (Kabir et al., 14 Jul 2025).

On SemEval, DepBERT (Gated) obtains Precision 0.942, Recall 0.962, F1 0.957, and Exact Acc 0.842. The corresponding baselines include BERT at F1 0.932 and Acc 0.809, BERT+Dependency at F1 0.954 and Acc 0.841, BERT+POS at F1 0.941 and Acc 0.831, Sentence-BERT at F1 0.933 and Acc 0.818, LLaMA at F1 0.937 and Acc 0.819, BiLSTM at F1 0.844 and Acc 0.768, and BiLSTM-CRF at F1 0.860 and Acc 0.771. Improvements over Dasgupta are reported as +13.4% F1 and +9.63% accuracy.

On SCITE, DepBERT (Gated) reports Precision 0.932, Recall 0.943, F1 0.939, and Exact Acc 0.834, while BERT attains F1 0.893 and Acc 0.768, BERT+Dependency F1 0.906 and Acc 0.796, BERT+POS F1 0.916 and Acc 0.811, Sentence-BERT F1 0.886 and Acc 0.773, LLaMA F1 0.909 and Acc 0.790, BiLSTM F1 0.817 and Acc 0.747, and BiLSTM-CRF F1 0.831 and Acc 0.751. Improvements over Dasgupta are listed as +14.9% F1 and +11.6% accuracy.

The ablation narrative is limited but informative. The single-component baselines “BERT+Dependency” and “BERT+POS” isolate the effect of syntactic edges and POS tags. Across datasets, “BERT+Dependency” tends to outperform “BERT+POS,” which indicates a stronger effect from modeling explicit dependencies within attention. DepBERT combines POS-enhanced embeddings with dependency-restricted attention and achieves the best results. The paper reports the gated version of DepBERT but does not provide a version without gating, so the contribution of the gating mechanism is inferred from the overall model design rather than isolated experimentally.

The qualitative examples clarify how dependency-aware attention is intended to function. For “Vitamin D deficiency causes diabetes,” the dependency tree contains edges $S$ 2, $S$ 3, $S$ 4, and $S$ 5, and the POS tags are NOUN, NOUN, NOUN, VERB, NOUN. In the dependency-aware tower, “causes” attends primarily to its syntactic subject “deficiency” and object “diabetes” rather than to all tokens. This focused attention helps identify “Vitamin D deficiency” as the cause span and “diabetes” as the effect span. A second example, “Smoking is a major risk factor for developing lung cancer,” is presented to show that the mechanism can also capture causal relations expressed through nominal and prepositional structure rather than a simple verb like “cause” (Kabir et al., 14 Jul 2025).

Several limitations are explicit. CAUSALGPT contains only one cause-effect pair per sentence by construction, so multi-pair extraction is not evaluated. Experiments are in English only, and cross-lingual performance and parser transfer are not studied. Reliance on dependency parsing introduces sensitivity to parser errors, but no explicit noise-handling strategy is reported. Code and data release details are also not reported, and the paper does not specify seeds, hardware, or dropout settings. The complexity profile is only partially characterized: the right tower adds computational overhead through a second encoder and gating, although its attention is restricted to dependency neighbors rather than all tokens.

The broader relevance of the model is presented cautiously. Its combination of global token co-occurrence, dependency structure, and POS information suggests applicability beyond cause-effect extraction to relation extraction, event extraction, and semantic role labeling tasks where head-dependent relations and argument structure are central. This suggests a more general design pattern for integrating linguistic structure into transformer sequence labeling.

A common source of ambiguity is nomenclature. DepBERT, the dependency-aware causality extractor described here, is distinct from “DeeBERT,” which addresses dynamic early exiting for accelerating BERT inference rather than cause-effect extraction (Xin et al., 2020). It is also distinct from production query-title relevance systems centered on eBERT and BertBiLSTM for CPU-constrained deployment in e-commerce search (Dahlmann et al., 2021).