KGBERT4Eth: Integrated Ethereum Fraud Encoder

• KGBERT4Eth is a feature-complete encoder that unifies sequential transaction semantics, fund-flow graph structure, and expert-defined behavioral descriptors for Ethereum fraud detection.
• It employs a dual-component architecture—combining a Transformer-based Transaction Language Model (TLM) and a Graph Neural Network-based Transaction Knowledge Graph (TKG)—to capture diverse on-chain features.
• Innovative training objectives like Biased Mask Prediction and mask-invariant attention synergy enable significant gains in phishing detection and account de-anonymization performance.

KGBERT4Eth is a feature-complete pre-training encoder for multi-task Ethereum fraud detection that jointly models three complementary feature families derivable from Ethereum native transaction data: sequential semantic/contextual information from transaction histories, graph-structured fund-flow and inter-account relation information from address interactions, and expert-defined behavioral descriptors such as transaction statistics, temporal patterns, and graph-theoretic indicators. It is presented as a dual-component framework consisting of a Transaction Semantic Extractor, implemented as an enhanced Transaction LLM (TLM), and a Transaction Knowledge Graph (TKG), with the two modules jointly optimized through Biased Mask Prediction (BMP), Transaction Link Prediction (TLP), and a mask-invariant attention synergy module (MiAS) (Jia et al., 4 Sep 2025).

1. Problem setting and conceptual scope

The paper frames Ethereum fraud detection as a representation learning problem over historical on-chain transaction data. The downstream objective is to learn account representations that support two tasks: phishing account detection and account de-anonymization. In the paper’s formulation, the field had split into three largely separate methodological strands: expert-defined feature models, graph-based methods, and sequential transaction modeling. KGBERT4Eth is motivated by the claim that these strands collectively span the main information sources in Ethereum’s native data layer, yet prior systems do not integrate them endogenously during learning (Jia et al., 4 Sep 2025).

The paper uses “feature-complete” in a restricted technical sense. It does not mean using every raw field literally as-is. Rather, it means that the framework jointly covers the three major feature paradigms naturally available from Ethereum native transaction data: sequence semantics, fund-flow structure, and expert-curated behavioral knowledge. This positioning is important because the method is not solely a graph model, not solely a BERT-style sequence model, and not merely a hand-engineered feature pipeline.

The two downstream tasks are defined concretely. Phishing account detection is a binary classification problem over Ethereum accounts. Account de-anonymization aims to infer account identity categories from behavior, using labeled address sets with the categories Airdrop Hunter, ICO Wallets, Mev Bots, and Synthetix accounts. The paper explicitly notes that this is not de-anonymization in the strong legal sense of recovering a person’s name, but rather identifying the behavioral or organizational type behind an otherwise anonymous address (Jia et al., 4 Sep 2025).

A plausible implication is that KGBERT4Eth should be read less as a generic knowledge-graph-enhanced LLM and more as a domain-specific pre-training framework for security analytics on Ethereum. Its knowledge graph is transactional and behavioral rather than encyclopedic or ontological.

2. Dual-component architecture

KGBERT4Eth combines two pre-training components. The first is a Transaction Semantic Extractor, implemented as an enhanced Transaction LLM (TLM) over textualized transaction records. The second is a Transaction Knowledge Graph (TKG) that models Ethereum accounts, transaction relations, and expert-curated account features. These modules are jointly pre-trained and exchange information through MiAS (Jia et al., 4 Sep 2025).

At a high level, the pipeline is described as follows: build transaction sentences from each account’s transaction history; build a transaction knowledge graph from address interactions and initialize nodes with expert features; encode transaction text with a Transformer-based TLM; encode the TKG with a GNN-based graph encoder; exchange information between the two modalities through MiAS during pre-training; and train with BMP for the TLM and TLP for the TKG. The intended result is an account representation that fuses semantic transaction context, fund-flow structure, and human/expert knowledge.

The TLM begins from an account transaction history

$\hat{T}_i = \{t_1, t_2, \dots, t_n\},$

where $n$ is the number of transactions. From the full set of native fields, the paper excludes fields that “lack inherent features” for language modeling, especially address-like identifiers such as from and to. It retains five essential fields: value, timeStamp, IO, gas, and gasPrice. These are textualized into readable strings, tokenized with the BERT tokenizer, and collected into a corpus

$$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$

After masking, the sequence is represented as

$$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$

with $L < L_{\max}$, and the initial hidden representation is

$$\boldsymbol{h}^{(0)}_j = \boldsymbol{e}_j + \boldsymbol{p}_j.$$

The TLM itself is an $N$-layer Transformer architecture with a multi-head self-attention sublayer and a feed-forward network in each layer; the final $[\mathrm{CLS}]$ representation is used as the aggregated semantic embedding of the account (Jia et al., 4 Sep 2025).

The TKG is defined as a multi-relational graph

$G = (V, E),$

where $E \subseteq V \times R \times V$, and each edge is a triple

$n$0

Each node corresponds to an Ethereum address, including Externally Owned Accounts (EOAs) and Contract Accounts (CAs). Relation types are divided into external transactions and contract transactions. The graph encoder is a graph neural network with a generic message-passing formulation:

$n$1

and node updates are given by

$n$2

with

$n$3

A defining architectural feature is that TKG nodes are initialized with expert-engineered features. These features are grouped into three categories.

Category	Features
Statistical financial behavior	Node outdegree; Node indegree; Direction ratio; Max outgoing amount; Min outgoing amount; Max incoming amount; Min incoming amount; Average outgoing amount; Average incoming amount; Account balance
Temporal activity patterns	Account lifetime; Active days; Long-term transfer frequency; Short-term transfer frequency; Long-term incoming transfer frequency; Short-term incoming transfer frequency; Long-term outgoing transfer frequency; Short-term outgoing transfer frequency
Structural network metrics	Katz centrality; Betweenness centrality; Degree centrality; Closeness centrality; Clustering coefficient; Eigenvector centrality; Indegree centrality; Outdegree centrality

The paper motivates these features empirically by stating that phishing account lifespan and transaction count follow power-law distributions, suggesting the relevance of heavy-tailed indicators. This suggests that expert priors are treated not as auxiliary metadata but as a core part of the representation-learning design (Jia et al., 4 Sep 2025).

3. Pre-training objectives and cross-modal coordination

A central claim of the paper is that ordinary masked language modeling is poorly matched to Ethereum transaction text because the text is highly homogeneous. Under random masking, the model may focus on common, repetitive, low-information tokens rather than the rare tokens that indicate atypical or suspicious behavior. KGBERT4Eth addresses this with Biased Mask Prediction (BMP), in which masking is guided by BM25-based token importance (Jia et al., 4 Sep 2025).

For each token $n$4 in account text $n$5, the paper computes inverse document frequency

$n$6

and then the BM25 score

$n$7

Masking is threshold-based:

$n$8

The BMP loss is

$n$9

The paper’s rationale is explicit: a token that is rare globally but frequent locally is more discriminative for that account and should be more likely to be masked and reconstructed.

The graph side uses Transaction Link Prediction (TLP). For a triple $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$0, the graph encoder yields embeddings $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$1, and the TLP objective is

$$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$2

Here, $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$3 is the set of positive triples, $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$4 is the set of corrupted negative triples, and $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$5 is generated by replacing head or tail with a random entity. The paper states that the scoring function can be instantiated with standard KG models, mentioning TransE as an example, and reports RotatE as the default “ours” in experiments (Jia et al., 4 Sep 2025).

The integration problem is handled by Mask-invariant Attention Synergy (MiAS). The paper argues that naïve interaction between TLM and TKG can damage masked language modeling: if graph information is injected into all token positions, masked tokens may be indirectly “leaked” or their semantics altered. MiAS therefore imposes two constraints: query-aligned dictionary restriction, under which a semantic embedding only attends to TKG entities and triplets relevant to the same account, and visibility masking through $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$6, under which semantic embeddings interact with graph entities only via the $$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$7 token (Jia et al., 4 Sep 2025).

The semantic-to-graph attention is defined as

$$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$8

with update

$$\mathcal{C} = \{\hat{X}_1, \hat{X}_2, \dots, \hat{X}_{N_a}\}.$$9

The paper states that the graph side is updated symmetrically, though it does not provide an explicit formula for $$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$0. The joint optimization objective is

$$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$1

with no weighting coefficients reported. This suggests that the representation fusion is intentionally simple at the loss level, with the main sophistication concentrated in masking, graph construction, and cross-modal visibility control.

4. Data, preprocessing, and evaluation protocol

The model is pre-trained on 3,251,082 Ethereum transaction records involving 496,740 accounts. The data sources are Etherscan, Kaggle, and Xblock. The preprocessing pipeline is described as: collect account transaction records, textualize selected fields into transaction sentences, tokenize using the BERT-base tokenizer, compute token BM25 scores using BM25L, apply threshold-based biased masking, and build the TKG from the same records (Jia et al., 4 Sep 2025).

From the same data, the paper constructs a TKG with approximately 500k entities and 1M edges. Nodes are addresses, edges are transaction relations, and node features are initialized with expert-engineered behavioral features. For downstream datasets, the paper uses 2-hop node retrieval with a maximum of 100 nodes. The exact mechanics are not elaborated, but the text identifies this as a graph sampling or retrieval heuristic for fine-tuning.

The sequence and masking hyperparameters that are explicitly reported are:

• maximum sequence length:

$$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$2

• mask threshold:

$$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$3

The sensitivity analysis further studies $$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$4 and $$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$5. For downstream tasks, the data split is 70% training, 10% validation, and 20% test. The paper explicitly states that any accounts used in pre-training are excluded from downstream evaluation, and that all reported results are on the test set and averaged over five runs (Jia et al., 4 Sep 2025).

The evaluation tasks and metrics are summarized below.

Task	Datasets / Labels	Metrics
Phishing account detection	MultiGraph, B4E, SPN; phishing vs non-phishing	F1, AUC, FNR
Account de-anonymization	972 Airdrop Hunters, 178 ICO Wallets, 331 Mev Bots, 524 Synthetix accounts	Overall F1, Overall FNR, class-wise F1 and FNR

The baseline suite covers fifteen methods across three groups: DeepWalk, Role2Vec, and Trans2Vec as random-walk or embedding baselines; GCN, GAT, GSAGE, DiffPool, U2GNN, Graph2Vec, TSGN, GrabPhisher, GAE, and GATE as GNN and graph-representation baselines; and BERT4ETH and ZipZap as pretrained Transformer baselines. The paper states that this baseline suite is intended to compare KGBERT4Eth against all three prevailing paradigms it seeks to unify (Jia et al., 4 Sep 2025).

5. Empirical performance and ablation evidence

Across all three phishing benchmarks, KGBERT4Eth is reported as the top-performing model. On MultiGraph, it achieves F1 = 87.60, AUC = 96.01, and FNR = 14.11. The best random-walk baseline, Trans2Vec, has F1 = 68.50, and the best GNN baseline, GrabPhisher, has F1 = 78.89. The paper therefore reports absolute gains of +19.10 F1 over the best random-walk baseline and +8.71 F1 over the best GNN baseline. On B4E, KGBERT4Eth achieves F1 = 85.43, AUC = 95.34, and FNR = 16.03. On SPN, it achieves F1 = 88.02, AUC = 97.12, and FNR = 14.75. These results are presented as supporting the claim of roughly 8–16% absolute F1 improvement on phishing detection benchmarks (Jia et al., 4 Sep 2025).

For de-anonymization, KGBERT4Eth achieves Overall F1 = 90.52 and Overall FNR = 8.90. The best random-walk baseline, Trans2Vec, reaches 74.91 F1; the best GNN baseline, GrabPhisher, reaches 81.07 F1; and the best Transformer baseline, BERT4ETH, reaches 76.13 F1. The resulting absolute improvements are +15.61 F1 over Trans2Vec, +9.45 F1 over GrabPhisher, and +14.39 F1 over BERT4ETH. The class-wise results are Airdrop Hunter: F1 = 98.36, FNR = 0.11; ICO Wallets: F1 = 65.66, FNR = 42.76; Mev Bot: F1 = 84.07, FNR = 17.75; and Synthetix: F1 = 88.68, FNR = 9.07. The paper highlights the ICO Wallets category as especially important because it is the sparsest category (Jia et al., 4 Sep 2025).

The ablation study isolates the contribution of the framework’s principal components. On phishing detection, the full model obtains F1 = 88.02 and FNR = 14.75; w/o BMP falls to F1 = 85.27; w/o TKG to 86.95; w/o Expert to 84.49; w/o MiAS to 86.54; BERT gives 82.80; and RoBERTa gives 80.92. On de-anonymization, the full model gives F1 = 90.52 and FNR = 8.90; w/o BMP gives 86.67; w/o TKG gives 88.91; w/o Expert gives 88.22; w/o MiAS gives 86.81; BERT gives 82.73; and RoBERTa gives 83.64.

These ablations are interpreted directly in the paper. BMP matters because replacing biased masking with standard random masking hurts both tasks. TKG matters because removing graph knowledge reduces performance. Expert knowledge matters because randomizing node initialization degrades results. MiAS matters strongly, especially on de-anonymization, where removing it drops F1 from 90.52 to 86.81. A plausible implication is that the method’s gains do not come from pre-training alone, but from the interaction among rare-token emphasis, graph structure, expert features, and mask-safe cross-modal coordination.

The paper also compares KG scoring functions for TLP: DistMult, QuatE, TransE, ComplEx, and RotatE. Results are described as close. RotatE is used in the main model and gives phishing F1 = 88.02 and de-anonymization F1 = 90.52. The authors conclude that score function choice is less critical when large-scale pre-training is used (Jia et al., 4 Sep 2025).

6. Relation to knowledge-graph-enhanced BERT and stated constraints

KGBERT4Eth belongs to the broader family of models that combine Transformer-based language representations with structured knowledge, but its design is domain-specific. The paper explicitly characterizes part of its novelty, relative to broader KG-enhanced LLMs, as applying a K-BERT-inspired visibility constraint to the fraud-detection setting, coupling transaction-sequence semantics with fund-flow graph structure, preserving MLM-style training integrity while enabling pre-training-time graph injection, and aiming for inference without explicit KG reconstruction (Jia et al., 4 Sep 2025, Liu et al., 2019).

This connection is conceptually important. K-BERT introduced the problem of knowledge noise (KN) and proposed structural controls such as a visible matrix to constrain attention when knowledge is injected into a Transformer (Liu et al., 2019). KGBERT4Eth adopts a related visibility logic in MiAS, but repurposes it for a different setting: the objective is not sentence-level knowledge injection from a symbolic KG into ordinary text, but stable coordination between account-level transaction semantics and a transaction graph during joint pre-training. The paper’s term mask-invariant reflects this shift in emphasis from generic knowledge enrichment to protecting masked-token learning (Jia et al., 4 Sep 2025).

The paper also makes a practical deployment claim: during fine-tuning and evaluation, the model directly processes textual transaction records without building or updating the KG. In this sense, graph knowledge is intended to be distilled into the pretrained representation rather than queried dynamically at inference time. This differentiates the method from systems in which a knowledge graph remains an online dependency throughout deployment.

Several limitations and assumptions are visible in the paper’s description. It states that architectural details are under-specified, which constrains reproducibility. The BERT context limit of 512 truncates long transaction histories. Address fields are excluded from sequence modeling, so graph modeling must carry all address-interaction information. Expert features require manual design, which may reduce portability to other fraud types or other chains. The paper also notes future work on long-context LLMs and model compression for lightweight deployment (Jia et al., 4 Sep 2025).

The sensitivity analysis sharpens these constraints. Performance is best when

$$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$6

When $$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$7, performance declines sharply because too few tokens are masked. The paper also reports that about 70% of account token sequences are shorter than 512 tokens, corresponding to about 15 transaction records, and that performance improves as $$\hat{X}_i^{\text{masked}} = [[\mathrm{CLS}], tok_1, tok_2, \dots, tok_L],$$8 increases up to 512. This suggests that longer context windows could further improve results, although the authors did not test beyond 512 because of BERT’s architectural limit (Jia et al., 4 Sep 2025).

Taken together, KGBERT4Eth is best understood as a jointly pretrained Ethereum-specific encoder in which transaction text, transaction graph structure, and expert-curated behavioral priors are fused before downstream supervision. Its core contribution lies not in a new generic Transformer architecture, but in the particular way it integrates these three signal families into one pre-training framework for phishing detection and account de-anonymization (Jia et al., 4 Sep 2025).