Transaction Language Model (TLM)

• Transaction Language Model is a domain-specific encoder that converts structured transaction records into tokenized sequences for contextual representation learning.
• Models like TLMG4Eth and KGBERT4Eth use transformer encoders with biased masking strategies to capture nuanced transactional semantics in Ethereum fraud analysis.
• Integrating graph structures such as TASG, AIG, and TKG with TLM improves anomaly detection by fusing semantic and relational features.

Transaction LLM (TLM) denotes a family of models that convert structured transaction records into text-like sequences and learn contextual representations over those sequences with language-modeling machinery, typically Transformer encoders and masked prediction objectives. In current arXiv usage, the term is most closely associated with Ethereum fraud analysis, where account transaction histories are serialized into “transaction sentences” or “conceptualized transaction records” and then encoded for downstream tasks such as phishing detection and de-anonymization (Sun et al., 2024, Jia et al., 4 Sep 2025). The acronym is, however, highly polysemous: earlier and much broader engineering literature uses TLM to mean Transaction Level Modeling rather than a LLM, and several NLP papers use TLM for unrelated objectives such as Task-Driven Language Modeling, Translation Language Modeling, or Test-Time Learning (0710.4808, Yao et al., 2021).

1. Terminological scope and disambiguation

In the narrow sense relevant to contemporary blockchain ML, a Transaction LLM is a domain-specific encoder that treats transactions as a language-like corpus. Its basic move is to serialize transaction attributes into token sequences so that contextual representation learning can capture regularity, rarity, and sequence-dependent behavior in a form closer to language modeling than to hand-engineered feature extraction. In the broader arXiv record, however, the same acronym denotes several unrelated concepts, so explicit disambiguation is usually required.

Meaning of TLM	Domain	Representative papers
Transaction LLM	Ethereum fraud detection	(Sun et al., 2024, Jia et al., 4 Sep 2025)
Transaction Level Modeling	SystemC, SoC, verification, FMI co-simulation	(0710.4808, 0710.4748, Jain et al., 2014, Qiu et al., 15 May 2025, Albu et al., 27 Aug 2025)
Other NLP meanings of TLM	Task-driven, translation, test-time, multilingual pretraining	(Yao et al., 2021, Lin et al., 2021, Fernando et al., 10 Jan 2025, Hu et al., 27 May 2025, Urteaga et al., 2022)

The hardware meaning is historically older and methodologically distinct. In that literature, TLM abstracts signal-level bus activity into transaction-level communication for SoC design, verification, and co-simulation, as in AMBA/AHB bus modeling, SystemC SHIP channels, UVM-TLM processor verification, and SystemC-to-FMI wrapping (0710.4808, 0710.4748). By contrast, the blockchain usage of Transaction LLM refers to semantic modeling of transaction histories as token sequences.

2. Transaction serialization and corpus construction

The defining operation of a Transaction LLM is the conversion of numerical or structured transaction records into textual or tokenized sequences. In TLMG4Eth, the raw unit is a transaction history for one account, with ordered transactions

$$\mathcal{T}=\{t_1,t_2,\dots,t_N\}, \qquad t_i=(v_i,d_i,\tau_i),$$

where $v_i$ is transaction amount, $d_i\in\{-1,1\}$ is direction, and $\tau_i$ is timestamp. Raw timestamps are not used directly as semantic tokens; instead, the model encodes inter-transaction time gaps

$$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$

and the final transaction representation is

$$\mathcal{L}(t_i)=\{\text{amount:}v_i,\text{direction:}d_i,\text{2-inter\_time:}\Delta\tau_{i,2},\dots,\text{5-inter\_time:}\Delta\tau_{i,5}\}.$$

Each element in $\mathcal{L}(t_i)$ is treated as a transaction word, and the account corpus is

$$\mathcal{C}=\{\mathcal{L}(t_1),\mathcal{L}(t_2),\dots,\mathcal{L}(t_N)\}.$$

This representation makes amount, flow direction, and multi-scale timing explicit lexical objects rather than leaving them as opaque numeric fields (Sun et al., 2024).

KGBERT4Eth uses a related but distinct textualization strategy. It is explicitly account-centric: an account $i$ has a transaction history

$\hat{T}_i=\{t_1,t_2,\dots,t_n\},$

which is converted into a textual transaction record

$v_i$0

The TLM side retains five essential fields—value, timeStamp, IO, gas, and gasPrice—and excludes fields such as from and to, which are described as long hexadecimal identifiers better handled by the graph module. After textualization, the model applies the BERT tokenizer, and the vocabulary is defined as

$v_i$1

where $v_i$2 is the set of selected fields and $v_i$3 is the union of tokenized words across accounts (Jia et al., 4 Sep 2025).

Across these formulations, the common principle is structural lexicalization: transaction records are turned into field-value tokens whose contextual combinations become learnable by sequence encoders. This suggests a domain-specific notion of “language” in which semantic units are not natural-language words but typed transaction attributes and their serialized values.

3. Core architectures and language-model objectives

The language-modeling core in this literature is BERT-like rather than autoregressive. TLMG4Eth continues pretraining BERT-base on a domain-specific corpus formed by all accounts’ transaction sentences,

$v_i$4

using masked language modeling:

$v_i$5

The contextual token embedding is written as

$v_i$6

Here the TLM is a continued-pretrained BERT-base operating on templated transaction sentences built from amount, direction, and interval-time fields (Sun et al., 2024).

KGBERT4Eth describes its TLM as an $v_i$7-layer Transformer architecture with a special $v_i$8 token and standard token-plus-position initialization:

$v_i$9

The encoder stack is written as

$d_i\in\{-1,1\}$0

and the final account representation is $d_i\in\{-1,1\}$1 (Jia et al., 4 Sep 2025).

Its pretraining objective is not standard random MLM but Biased Mask Prediction (BMP):

$d_i\in\{-1,1\}$2

Mask selection is driven by BM25 token importance rather than uniform sampling. The model computes

$d_i\in\{-1,1\}$3

$d_i\in\{-1,1\}$4

and then masks a token when its normalized BM25 score exceeds a threshold $d_i\in\{-1,1\}$5:

$d_i\in\{-1,1\}$6

The reported pretraining choice is $d_i\in\{-1,1\}$7, and the LLM’s maximum input sequence length is $d_i\in\{-1,1\}$8; about 70% of tokenized account sequences are shorter than 512, corresponding to roughly 15 transaction records per account (Jia et al., 4 Sep 2025).

A key distinction therefore emerges between two TLM design choices already present in the literature: generic MLM on transaction sentences and anomaly-sensitive masking driven by information-retrieval statistics.

4. Hybridization with graph structure and relational priors

Recent Transaction LLMs are not purely sequential encoders. Both major Ethereum implementations combine the language-model branch with graph-based components, but they do so in different ways.

TLMG4Eth augments semantic embeddings with two graph views. The first is a Transaction Attribute Similarity Graph (TASG), a vocabulary graph $d_i\in\{-1,1\}$9 whose edges are built either from normalized PMI,

$\tau_i$0

with an edge added when $\tau_i$1, or from TF-IDF sentence-word links,

$\tau_i$2

The second is an Account Interaction Graph (AIG), a weighted account graph $\tau_i$3 whose edge weight $\tau_i$4 is the number of transactions between accounts $\tau_i$5 and $\tau_i$6. Semantic and similarity embeddings are concatenated,

$\tau_i$7

then fused by a 12-layer, 12-head multi-head attention network. The resulting account representation initializes the AIG node features, after which a GCN is applied:

$\tau_i$8

Final prediction interpolates the MAN and GCN outputs:

$\tau_i$9

Thus, transaction semantics, transaction similarity, and account interaction structure are trained jointly rather than fused only after separate optimization (Sun et al., 2024).

KGBERT4Eth uses a different graph object: a Transaction Knowledge Graph (TKG) that incorporates expert-curated domain knowledge and fund-flow patterns. Its TLM and TKG communicate through Mask-invariant Attention Synergy (MiAS). The TLM hidden states are

$$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$0

and the TKG entity embeddings are

$$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$1

Cross-attention from graph to text is written as

$$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$2

The visibility matrix $$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$3 restricts semantic-graph interaction to the $$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$4 token only, so graph information enriches the account-level summary without directly corrupting masked token positions. Joint pretraining sums the TLM’s BMP loss and the TKG’s transaction link prediction loss:

$$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$5

This makes the TLM not an isolated encoder but one branch in a co-trained semantic-relational system (Jia et al., 4 Sep 2025).

5. Applications and empirical performance

The principal reported application domain for Transaction LLMs is Ethereum account-level fraud analysis. TLMG4Eth targets phishing account detection on three binary datasets—MulDiGraph, B4E, and SPN—and reports that the full model outperforms the strongest baseline by roughly 20.12% on MulDiGraph, 14.12% on B4E, and 10.32% on SPN in F1. Its ablation table shows that the TLM alone is already strong: F1 on MulDiGraph improves from 0.8804 for TLM Only to 0.9041 with the full similarity-enhanced model, on B4E from 0.7923 to 0.8123, and on SPN from 0.8018 to 0.8146. The TF-IDF-based similarity graph is the best TASG construction, and the best interpolation occurs at intermediate rather than extreme values of $$\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},$$6, around 0.7 on MulDiGraph and 0.8 on B4E and SPN (Sun et al., 2024).

KGBERT4Eth extends the scope from phishing detection to multi-task Ethereum fraud detection, specifically phishing account detection and account de-anonymization. The abstract reports absolute F1-score improvements of 8–16% on three phishing detection benchmarks and 6–26% on four de-anonymization datasets. In the phishing setting, the reported F1 scores are 87.60 on MultiGraph, 85.43 on B4E, and 88.02 on SPN, compared with BERT4ETH scores of 64.20, 69.05, and 72.98 and ZipZap scores of 65.25, 68.48, and 72.31. In de-anonymization, the overall F1 reaches 90.52, compared with 76.13 for BERT4ETH and 75.91 for ZipZap. The ablations further isolate the TLM contribution: removing the TKG yields phishing F1 86.95 and de-anonymization F1 88.91; removing BMP yields 85.27 and 86.67; removing MiAS yields 86.54 and 86.81. These numbers indicate that the sequential semantic branch is already strong on its own, while graph coupling, biased masking, and coordination supply additional gains (Jia et al., 4 Sep 2025).

Across both systems, the empirical pattern is consistent. Transaction text encoding is not treated as a replacement for graph structure, but as a high-value semantic branch that becomes stronger when paired with token-similarity graphs, account-interaction graphs, or knowledge graphs. A plausible implication is that the recent literature regards transaction semantics, similarity, and topology as complementary rather than competing feature spaces.

6. Misconceptions, adjacent meanings, and broader context

A common misconception is to treat Transaction LLM as synonymous with any work labeled “TLM.” The arXiv record does not support that reading. In hardware and system design, TLM most often means Transaction Level Modeling: a bus or processor abstraction used for SoC performance analysis, early software development, verification, or co-simulation rather than a LLM over transactions (0710.4808, Qiu et al., 15 May 2025). In NLP, TLM may instead denote Task-Driven Language Modeling, Translation Language Modeling, multilingual continual pretraining, Test-Time Learning for LLMs, or bandit-based optimization of Transformer LLM pre-training (Yao et al., 2021, Fernando et al., 10 Jan 2025).

There are also adjacent lines of research that are transaction-oriented without being Transaction LLMs in the Ethereum sense. TStreamLLM proposes a systems framework that applies transactional stream-processing principles—consistency, isolation, durability, concurrency control, and fault tolerance—to continuous LLM updates and usage, but it does not textualize transaction records or define a transaction-language encoder (Zhang et al., 2023). TACLA, by contrast, operationalizes Transactional Analysis as a multi-agent control architecture for dialogue simulation, where Parent, Adult, and Child ego-state agents are orchestrated according to contextual triggers and script-like priors; this is transaction-aware in the psychological sense, not in the sense of modeling financial transaction corpora (Zamojska et al., 19 Oct 2025).

The present state of the term is therefore narrow but technically specific. In its strict recent meaning, a Transaction LLM is an account-centric, Transformer-based semantic encoder over serialized transaction histories, typically trained with masked prediction and often integrated with graph learning. Its most developed uses are in Ethereum fraud detection, where the central research question is how to turn repetitive, structured, largely numerical transaction logs into contextual embeddings that preserve behavioral semantics, anomaly salience, and relational compatibility (Sun et al., 2024, Jia et al., 4 Sep 2025).