---
title: 'KGBERT4Eth: Integrated Ethereum Fraud Encoder'
url: https://www.emergentmind.com/topics/kgbert4eth
type: topic
---

# KGBERT4Eth: Integrated Ethereum Fraud Encoder

KGBERT4Eth is a **feature-complete pre-training encoder** for **multi-task Ethereum fraud detection** that jointly models three complementary feature families derivable from Ethereum native transaction data: **sequential semantic/contextual information** from transaction histories, **graph-structured fund-flow and inter-account relation information** from address interactions, and **expert-defined behavioral descriptors** such as transaction statistics, temporal patterns, and graph-theoretic indicators. It is presented as a dual-component framework consisting of a **Transaction Semantic Extractor**, implemented as an enhanced **Transaction Language Model (TLM)**, and a **Transaction Knowledge Graph (TKG)**, with the two modules jointly optimized through **Biased Mask Prediction (BMP)**, **Transaction Link Prediction (TLP)**, and a **mask-invariant attention synergy module (MiAS)** [2509.03860].

## 1. Problem setting and conceptual scope

The paper frames Ethereum fraud detection as a **representation learning problem over historical on-chain transaction data**. The downstream objective is to learn account representations that support two tasks: **phishing account detection** and **account de-anonymization**. In the paper’s formulation, the field had split into three largely separate methodological strands: **expert-defined feature models**, **graph-based methods**, and **sequential transaction modeling**. KGBERT4Eth is motivated by the claim that these strands collectively span the main information sources in Ethereum’s native data layer, yet prior systems do not integrate them endogenously during learning [2509.03860].

The paper uses **“feature-complete”** in a restricted technical sense. It does not mean using every raw field literally as-is. Rather, it means that the framework jointly covers the three major feature paradigms naturally available from Ethereum native transaction data: sequence semantics, fund-flow structure, and expert-curated behavioral knowledge. This positioning is important because the method is not solely a graph model, not solely a BERT-style sequence model, and not merely a hand-engineered feature pipeline.

The two downstream tasks are defined concretely. **Phishing account detection** is a binary classification problem over Ethereum accounts. **Account de-anonymization** aims to infer account identity categories from behavior, using labeled address sets with the categories **Airdrop Hunter**, **ICO Wallets**, **Mev Bots**, and **Synthetix accounts**. The paper explicitly notes that this is not de-anonymization in the strong legal sense of recovering a person’s name, but rather identifying the behavioral or organizational type behind an otherwise anonymous address [2509.03860].

A plausible implication is that KGBERT4Eth should be read less as a generic knowledge-graph-enhanced language model and more as a domain-specific pre-training framework for security analytics on Ethereum. Its knowledge graph is transactional and behavioral rather than encyclopedic or ontological.

## 2. Dual-component architecture

KGBERT4Eth combines two pre-training components. The first is a **Transaction Semantic Extractor**, implemented as an enhanced **Transaction Language Model (TLM)** over textualized transaction records. The second is a **Transaction Knowledge Graph (TKG)** that models Ethereum accounts, transaction relations, and expert-curated account features. These modules are jointly pre-trained and exchange information through **MiAS** [2509.03860].

At a high level, the pipeline is described as follows: build transaction sentences from each account’s transaction history; build a transaction knowledge graph from address interactions and initialize nodes with expert features; encode transaction text with a Transformer-based TLM; encode the TKG with a GNN-based graph encoder; exchange information between the two modalities through MiAS during pre-training; and train with **BMP** for the TLM and **TLP** for the TKG. The intended result is an account representation that fuses **semantic transaction context**, **fund-flow structure**, and **human/expert knowledge**.

The TLM begins from an account transaction history
$$
\hat{T}_i = \{t_1, t_2, \dots, t_n\},
$$
where \(n\) is the number of transactions. From the full set of native fields, the paper excludes fields that “lack inherent features” for language modeling, especially address-like identifiers such as `from` and `to`. It retains five essential fields: `value`, `timeStamp`, `IO`, `gas`, and `gasPrice`. These are textualized into readable strings, tokenized with the **BERT tokenizer**, and collected into a corpus
$$
\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.
$$
After masking, the sequence is represented as
$$
\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],
$$
with \(L < L_{\max}\), and the initial hidden representation is
$$
\boldsymbol{h}^{(0)}_j = \boldsymbol{e}_j + \boldsymbol{p}_j.
$$
The TLM itself is an **\(N\)-layer Transformer architecture** with a **multi-head self-attention** sublayer and a **feed-forward network** in each layer; the final \([\mathrm{CLS}]\) representation is used as the aggregated semantic embedding of the account [2509.03860].

The TKG is defined as a multi-relational graph
$$
G = (V, E),
$$
where \(E \subseteq V \times R \times V\), and each edge is a triple
$$
(h, r, t).
$$
Each node corresponds to an Ethereum address, including **Externally Owned Accounts (EOAs)** and **Contract Accounts (CAs)**. Relation types are divided into **external transactions** and **contract transactions**. The graph encoder is a **graph neural network** with a generic message-passing formulation:
$$
\boldsymbol{\tilde{v}}_1^{(l+1)}, \dots, \boldsymbol{\tilde{v}}_{|V|}^{(l+1)} = f_{\text{gnn}}\left(\boldsymbol{v}_1^{(l)}, \dots, \boldsymbol{v}_{|V|}^{(l)}\right),
$$
and node updates are given by
$$
\boldsymbol{v}_i^{(l+1)} = f_v \left( \sum_{s \in \mathcal{N}_i \cup \{i\}} \alpha_{s,i}\boldsymbol{m}_{si} \right),
$$
with
$$
\boldsymbol{m}_{si} = f_m(\boldsymbol{v}_s^{(l)}, \boldsymbol{r}_{si}),
\qquad
\alpha_{s,i} = \mathrm{Softmax}\left(\frac{\boldsymbol{q}_s \boldsymbol{k}_i^\top}{\sqrt{d}}\right).
$$

A defining architectural feature is that TKG nodes are initialized with **expert-engineered features**. These features are grouped into three categories.

| Category | Features |
|---|---|
| Statistical financial behavior | Node outdegree; Node indegree; Direction ratio; Max outgoing amount; Min outgoing amount; Max incoming amount; Min incoming amount; Average outgoing amount; Average incoming amount; Account balance |
| Temporal activity patterns | Account lifetime; Active days; Long-term transfer frequency; Short-term transfer frequency; Long-term incoming transfer frequency; Short-term incoming transfer frequency; Long-term outgoing transfer frequency; Short-term outgoing transfer frequency |
| Structural network metrics | Katz centrality; Betweenness centrality; Degree centrality; Closeness centrality; Clustering coefficient; Eigenvector centrality; Indegree centrality; Outdegree centrality |

The paper motivates these features empirically by stating that phishing account lifespan and transaction count follow **power-law distributions**, suggesting the relevance of heavy-tailed indicators. This suggests that expert priors are treated not as auxiliary metadata but as a core part of the representation-learning design [2509.03860].

## 3. Pre-training objectives and cross-modal coordination

A central claim of the paper is that ordinary masked language modeling is poorly matched to Ethereum transaction text because the text is **highly homogeneous**. Under random masking, the model may focus on common, repetitive, low-information tokens rather than the rare tokens that indicate atypical or suspicious behavior. KGBERT4Eth addresses this with **Biased Mask Prediction (BMP)**, in which masking is guided by **BM25-based token importance** [2509.03860].

For each token \(tok_j\) in account text \(\hat{X}_i\), the paper computes inverse document frequency
$$
\mathrm{IDF}(tok_j) = \log \frac{|\mathcal{C}|}{|\{\hat{X}_i \in \mathcal{C} : tok_j \in \hat{X}_i\}| + 1},
$$
and then the BM25 score
$$
\mathrm{BM25}(tok_j, \hat{X}_i) =
\frac{ f(tok_j, \hat{X}_i)\cdot \left( \mathrm{IDF}(tok_j)\cdot (z_1 + 1) \right) }
{ f(tok_j, \hat{X}_i) + z_1 \cdot \left(1 - b + b \cdot \frac{|\hat{X}_i|}{avgdl}\right) }.
$$
Masking is threshold-based:
$$
\mathrm{Mask}(tok_j) =
\begin{cases}
1, & \text{if } \displaystyle \frac{\mathrm{BM25}(tok_j, \hat{X}_i)}
{\sum_{tok_j' \in \hat{X}_i} \mathrm{BM25}(tok_j', \hat{X}_i)} > \tau, \\
0, & \text{otherwise}.
\end{cases}
$$
The BMP loss is
$$
\mathcal{L}_{\text{BMP}}(\theta^m) = - \sum_{i=1}^{L} \mathrm{Mask}(tok_i)\log P(tok_i \mid \boldsymbol{h}_i).
$$
The paper’s rationale is explicit: a token that is **rare globally** but **frequent locally** is more discriminative for that account and should be more likely to be masked and reconstructed.

The graph side uses **Transaction Link Prediction (TLP)**. For a triple \((h,r,t)\), the graph encoder yields embeddings \((\boldsymbol{v}_h, \boldsymbol{r}, \boldsymbol{v}_t)\), and the TLP objective is
$$
\mathcal{L}_{\text{TLP}}(\theta^l) =
\sum_{(h,r,t)\in S}
\left(
-\sigma(d(\boldsymbol{v}_h, \boldsymbol{r}, \boldsymbol{v}_t))
+ \sum_{(h',r,t')\in S'} \sigma(d(\boldsymbol{v}_{h'}, \boldsymbol{r}, \boldsymbol{v}_{t'}))
\right).
$$
Here, \(S\) is the set of positive triples, \(S'\) is the set of corrupted negative triples, and \((h',r,t')\) is generated by replacing head or tail with a random entity. The paper states that the scoring function can be instantiated with standard KG models, mentioning **TransE** as an example, and reports **RotatE** as the default “ours” in experiments [2509.03860].

The integration problem is handled by **Mask-invariant Attention Synergy (MiAS)**. The paper argues that naïve interaction between TLM and TKG can damage masked language modeling: if graph information is injected into all token positions, masked tokens may be indirectly “leaked” or their semantics altered. MiAS therefore imposes two constraints: **query-aligned dictionary restriction**, under which a semantic embedding only attends to TKG entities and triplets relevant to the same account, and **visibility masking through \([\mathrm{CLS}]\)**, under which semantic embeddings interact with graph entities only via the \([\mathrm{CLS}]\) token [2509.03860].

The semantic-to-graph attention is defined as
$$
\alpha_{h,v} = \mathrm{Softmax}\left(\frac{\boldsymbol{q}_h \boldsymbol{k}_v^\top + M}{\sqrt{d}}\right),
$$
with update
$$
\boldsymbol{\hat{h}}^{(l)} = \sum_{v \in \mathscr{D}(h)} \alpha_{h,v}\cdot \boldsymbol{v}_v.
$$
The paper states that the graph side is updated symmetrically, though it does not provide an explicit formula for \(\boldsymbol{\hat{v}}^{(l)}\). The joint optimization objective is
$$
\mathcal{L}(\boldsymbol{\theta}) = \mathcal{L}_{\text{BMP}}(\theta^m) + \mathcal{L}_{\text{TLP}}(\theta^l),
$$
with no weighting coefficients reported. This suggests that the representation fusion is intentionally simple at the loss level, with the main sophistication concentrated in masking, graph construction, and cross-modal visibility control.

## 4. Data, preprocessing, and evaluation protocol

The model is pre-trained on **3,251,082 Ethereum transaction records** involving **496,740 accounts**. The data sources are **Etherscan**, **Kaggle**, and **Xblock**. The preprocessing pipeline is described as: collect account transaction records, textualize selected fields into transaction sentences, tokenize using the **BERT-base tokenizer**, compute token BM25 scores using **BM25L**, apply threshold-based biased masking, and build the TKG from the same records [2509.03860].

From the same data, the paper constructs a TKG with approximately **500k entities** and **1M edges**. Nodes are addresses, edges are transaction relations, and node features are initialized with expert-engineered behavioral features. For downstream datasets, the paper uses **2-hop node retrieval** with a maximum of **100 nodes**. The exact mechanics are not elaborated, but the text identifies this as a graph sampling or retrieval heuristic for fine-tuning.

The sequence and masking hyperparameters that are explicitly reported are:
- maximum sequence length:
  $$
  L_{\max} = 512
  $$
- mask threshold:
  $$
  \tau = 0.1
  $$

The sensitivity analysis further studies \(\tau\) and \(L_{\max}\). For downstream tasks, the data split is **70% training**, **10% validation**, and **20% test**. The paper explicitly states that any accounts used in pre-training are excluded from downstream evaluation, and that all reported results are on the **test set** and **averaged over five runs** [2509.03860].

The evaluation tasks and metrics are summarized below.

| Task | Datasets / Labels | Metrics |
|---|---|---|
| Phishing account detection | MultiGraph, B4E, SPN; phishing vs non-phishing | F1, AUC, FNR |
| Account de-anonymization | 972 Airdrop Hunters, 178 ICO Wallets, 331 Mev Bots, 524 Synthetix accounts | Overall F1, Overall FNR, class-wise F1 and FNR |

The baseline suite covers fifteen methods across three groups: **DeepWalk**, **Role2Vec**, and **Trans2Vec** as random-walk or embedding baselines; **GCN**, **GAT**, **GSAGE**, **DiffPool**, **U2GNN**, **Graph2Vec**, **TSGN**, **GrabPhisher**, **GAE**, and **GATE** as GNN and graph-representation baselines; and **BERT4ETH** and **ZipZap** as pretrained Transformer baselines. The paper states that this baseline suite is intended to compare KGBERT4Eth against all three prevailing paradigms it seeks to unify [2509.03860].

## 5. Empirical performance and ablation evidence

Across all three phishing benchmarks, KGBERT4Eth is reported as the **top-performing model**. On **MultiGraph**, it achieves **F1 = 87.60**, **AUC = 96.01**, and **FNR = 14.11**. The best random-walk baseline, **Trans2Vec**, has **F1 = 68.50**, and the best GNN baseline, **GrabPhisher**, has **F1 = 78.89**. The paper therefore reports absolute gains of **+19.10 F1** over the best random-walk baseline and **+8.71 F1** over the best GNN baseline. On **B4E**, KGBERT4Eth achieves **F1 = 85.43**, **AUC = 95.34**, and **FNR = 16.03**. On **SPN**, it achieves **F1 = 88.02**, **AUC = 97.12**, and **FNR = 14.75**. These results are presented as supporting the claim of roughly **8–16% absolute F1 improvement** on phishing detection benchmarks [2509.03860].

For de-anonymization, KGBERT4Eth achieves **Overall F1 = 90.52** and **Overall FNR = 8.90**. The best random-walk baseline, **Trans2Vec**, reaches **74.91 F1**; the best GNN baseline, **GrabPhisher**, reaches **81.07 F1**; and the best Transformer baseline, **BERT4ETH**, reaches **76.13 F1**. The resulting absolute improvements are **+15.61 F1** over Trans2Vec, **+9.45 F1** over GrabPhisher, and **+14.39 F1** over BERT4ETH. The class-wise results are **Airdrop Hunter: F1 = 98.36, FNR = 0.11**; **ICO Wallets: F1 = 65.66, FNR = 42.76**; **Mev Bot: F1 = 84.07, FNR = 17.75**; and **Synthetix: F1 = 88.68, FNR = 9.07**. The paper highlights the ICO Wallets category as especially important because it is the sparsest category [2509.03860].

The ablation study isolates the contribution of the framework’s principal components. On phishing detection, the full model obtains **F1 = 88.02** and **FNR = 14.75**; **w/o BMP** falls to **F1 = 85.27**; **w/o TKG** to **86.95**; **w/o Expert** to **84.49**; **w/o MiAS** to **86.54**; **BERT** gives **82.80**; and **RoBERTa** gives **80.92**. On de-anonymization, the full model gives **F1 = 90.52** and **FNR = 8.90**; **w/o BMP** gives **86.67**; **w/o TKG** gives **88.91**; **w/o Expert** gives **88.22**; **w/o MiAS** gives **86.81**; **BERT** gives **82.73**; and **RoBERTa** gives **83.64**.

These ablations are interpreted directly in the paper. **BMP matters** because replacing biased masking with standard random masking hurts both tasks. **TKG matters** because removing graph knowledge reduces performance. **Expert knowledge matters** because randomizing node initialization degrades results. **MiAS matters strongly**, especially on de-anonymization, where removing it drops F1 from **90.52** to **86.81**. A plausible implication is that the method’s gains do not come from pre-training alone, but from the interaction among rare-token emphasis, graph structure, expert features, and mask-safe cross-modal coordination.

The paper also compares KG scoring functions for TLP: **DistMult**, **QuatE**, **TransE**, **ComplEx**, and **RotatE**. Results are described as close. **RotatE** is used in the main model and gives **phishing F1 = 88.02** and **de-anonymization F1 = 90.52**. The authors conclude that score function choice is less critical when large-scale pre-training is used [2509.03860].

## 6. Relation to knowledge-graph-enhanced BERT and stated constraints

KGBERT4Eth belongs to the broader family of models that combine Transformer-based language representations with structured knowledge, but its design is domain-specific. The paper explicitly characterizes part of its novelty, relative to broader KG-enhanced language models, as **applying a K-BERT-inspired visibility constraint to the fraud-detection setting**, **coupling transaction-sequence semantics with fund-flow graph structure**, **preserving MLM-style training integrity while enabling pre-training-time graph injection**, and **aiming for inference without explicit KG reconstruction** [2509.03860; 1909.07606].

This connection is conceptually important. **K-BERT** introduced the problem of **knowledge noise (KN)** and proposed structural controls such as a **visible matrix** to constrain attention when knowledge is injected into a Transformer [1909.07606]. KGBERT4Eth adopts a related visibility logic in MiAS, but repurposes it for a different setting: the objective is not sentence-level knowledge injection from a symbolic KG into ordinary text, but stable coordination between account-level transaction semantics and a transaction graph during joint pre-training. The paper’s term **mask-invariant** reflects this shift in emphasis from generic knowledge enrichment to protecting masked-token learning [2509.03860].

The paper also makes a practical deployment claim: during fine-tuning and evaluation, the model **directly processes textual transaction records without building or updating the KG**. In this sense, graph knowledge is intended to be distilled into the pretrained representation rather than queried dynamically at inference time. This differentiates the method from systems in which a knowledge graph remains an online dependency throughout deployment.

Several limitations and assumptions are visible in the paper’s description. It states that **architectural details are under-specified**, which constrains reproducibility. The **BERT context limit of 512** truncates long transaction histories. **Address fields are excluded from sequence modeling**, so graph modeling must carry all address-interaction information. **Expert features require manual design**, which may reduce portability to other fraud types or other chains. The paper also notes future work on **long-context language models** and **model compression for lightweight deployment** [2509.03860].

The sensitivity analysis sharpens these constraints. Performance is best when
$$
\tau \in [0.05, 0.2].
$$
When \(\tau > 0.3\), performance declines sharply because too few tokens are masked. The paper also reports that about **70% of account token sequences are shorter than 512 tokens**, corresponding to about **15 transaction records**, and that performance improves as \(L_{\max}\) increases up to **512**. This suggests that longer context windows could further improve results, although the authors did not test beyond **512** because of BERT’s architectural limit [2509.03860].

Taken together, KGBERT4Eth is best understood as a jointly pretrained Ethereum-specific encoder in which **transaction text**, **transaction graph structure**, and **expert-curated behavioral priors** are fused before downstream supervision. Its core contribution lies not in a new generic Transformer architecture, but in the particular way it integrates these three signal families into one pre-training framework for phishing detection and account de-anonymization [2509.03860].

Source: https://www.emergentmind.com/topics/kgbert4eth