---
title: Transaction Language Model (TLM)
url: https://www.emergentmind.com/topics/transaction-language-model-tlm
type: topic
---

# Transaction Language Model (TLM)

Transaction Language Model (TLM) denotes a family of models that convert structured transaction records into text-like sequences and learn contextual representations over those sequences with language-modeling machinery, typically Transformer encoders and masked prediction objectives. In current arXiv usage, the term is most closely associated with Ethereum fraud analysis, where account transaction histories are serialized into “transaction sentences” or “conceptualized transaction records” and then encoded for downstream tasks such as phishing detection and de-anonymization [2409.07494][2509.03860]. The acronym is, however, highly polysemous: earlier and much broader engineering literature uses TLM to mean Transaction Level Modeling rather than a language model, and several NLP papers use TLM for unrelated objectives such as Task-Driven Language Modeling, Translation Language Modeling, or Test-Time Learning [0710.4808][2111.04130].

## 1. Terminological scope and disambiguation

In the narrow sense relevant to contemporary blockchain ML, a Transaction Language Model is a domain-specific encoder that treats transactions as a language-like corpus. Its basic move is to serialize transaction attributes into token sequences so that contextual representation learning can capture regularity, rarity, and sequence-dependent behavior in a form closer to language modeling than to hand-engineered feature extraction. In the broader arXiv record, however, the same acronym denotes several unrelated concepts, so explicit disambiguation is usually required.

| Meaning of TLM | Domain | Representative papers |
|---|---|---|
| Transaction Language Model | Ethereum fraud detection | [2409.07494], [2509.03860] |
| Transaction Level Modeling | SystemC, SoC, verification, FMI co-simulation | [0710.4808], [0710.4748], [1408.1150], [2505.10145], [2508.20223] |
| Other NLP meanings of TLM | Task-driven, translation, test-time, multilingual pretraining | [2111.04130], [2103.07040], [2501.05700], [2505.20633], [2203.13151] |

The hardware meaning is historically older and methodologically distinct. In that literature, TLM abstracts signal-level bus activity into transaction-level communication for SoC design, verification, and co-simulation, as in AMBA/AHB bus modeling, SystemC SHIP channels, UVM-TLM processor verification, and SystemC-to-FMI wrapping [0710.4808][0710.4748]. By contrast, the blockchain usage of Transaction Language Model refers to semantic modeling of transaction histories as token sequences.

## 2. Transaction serialization and corpus construction

The defining operation of a Transaction Language Model is the conversion of numerical or structured transaction records into textual or tokenized sequences. In TLMG4Eth, the raw unit is a transaction history for one account, with ordered transactions
$$
\mathcal{T}=\{t_1,t_2,\dots,t_N\}, \qquad t_i=(v_i,d_i,\tau_i),
$$
where $v_i$ is transaction amount, $d_i\in\{-1,1\}$ is direction, and $\tau_i$ is timestamp. Raw timestamps are not used directly as semantic tokens; instead, the model encodes inter-transaction time gaps
$$
\Delta\tau_{i,n}=\tau_i-\tau_{i-n}, \quad n\in\{1,2,\dots,N-1\},
$$
and the final transaction representation is
$$
\mathcal{L}(t_i)=\{\text{amount:}v_i,\text{direction:}d_i,\text{2-inter\_time:}\Delta\tau_{i,2},\dots,\text{5-inter\_time:}\Delta\tau_{i,5}\}.
$$
Each element in $\mathcal{L}(t_i)$ is treated as a transaction word, and the account corpus is
$$
\mathcal{C}=\{\mathcal{L}(t_1),\mathcal{L}(t_2),\dots,\mathcal{L}(t_N)\}.
$$
This representation makes amount, flow direction, and multi-scale timing explicit lexical objects rather than leaving them as opaque numeric fields [2409.07494].

KGBERT4Eth uses a related but distinct textualization strategy. It is explicitly account-centric: an account $i$ has a transaction history
$$
\hat{T}_i=\{t_1,t_2,\dots,t_n\},
$$
which is converted into a textual transaction record
$$
\hat{X}_i=\{x_1,x_2,\dots,x_n\}.
$$
The TLM side retains five essential fields—`value`, `timeStamp`, `IO`, `gas`, and `gasPrice`—and excludes fields such as `from` and `to`, which are described as long hexadecimal identifiers better handled by the graph module. After textualization, the model applies the BERT tokenizer, and the vocabulary is defined as
$$
\mathcal{D}=\{\hat{E},W\}, \qquad W=w_1\cup w_2\cup\dots\cup w_{N_a},
$$
where $\hat{E}$ is the set of selected fields and $W$ is the union of tokenized words across accounts [2509.03860].

Across these formulations, the common principle is structural lexicalization: transaction records are turned into field-value tokens whose contextual combinations become learnable by sequence encoders. This suggests a domain-specific notion of “language” in which semantic units are not natural-language words but typed transaction attributes and their serialized values.

## 3. Core architectures and language-model objectives

The language-modeling core in this literature is BERT-like rather than autoregressive. TLMG4Eth continues pretraining BERT-base on a domain-specific corpus formed by all accounts’ transaction sentences,
$$
\mathcal{D}=\bigcup_{a\in\mathcal{A}}\mathcal{C}_a,
$$
using masked language modeling:
$$
\mathcal{L}_{\text{MLM}}=\mathbb{E}_{t\sim\mathcal{C}}\left[-\sum_{i\in\mathcal{M}}\log P(t_i\mid \tilde{t})\right].
$$
The contextual token embedding is written as
$$
\mathbf{e}_i=\text{BERT}(x_i\mid t)\in\mathbb{R}^d.
$$
Here the TLM is a continued-pretrained BERT-base operating on templated transaction sentences built from amount, direction, and interval-time fields [2409.07494].

KGBERT4Eth describes its TLM as an $N$-layer Transformer architecture with a special $[CLS]$ token and standard token-plus-position initialization:
$$
\hat{X}_i^{\text{masked}}=[[CLS],tok_1,tok_2,\dots,tok_L], \qquad 
\boldsymbol{h}^{(0)}_j=\boldsymbol{e}_j+\boldsymbol{p}_j.
$$
The encoder stack is written as
$$
\boldsymbol{\tilde{h}^{(l+1)}_{CLS}}, \boldsymbol{\tilde{h}^{(l+1)}_1}, \ldots, \boldsymbol{\tilde{h}^{(l+1)}_L}
= f_{seq}(\boldsymbol{h}^{(l)}_{CLS}, \boldsymbol{h}^{(l)}_1, \ldots, \boldsymbol{h}^{(l)}_L),
$$
and the final account representation is $\boldsymbol{\tilde{h}^{(N)}_{CLS}}$ [2509.03860].

Its pretraining objective is not standard random MLM but Biased Mask Prediction (BMP):
$$
\mathcal{L}_{\text{BMP}}(\theta^m)=-\sum_{i=1}^{L}\textup{Mask}(tok_i)\log P(tok_i\mid \boldsymbol{h}_i).
$$
Mask selection is driven by BM25 token importance rather than uniform sampling. The model computes
$$
\text{IDF}(tok_j)=\log\frac{|\mathcal{C}|}{|\{\hat{X}_i\in\mathcal{C}: tok_j\in \hat{X}_i\}|+1},
$$
$$
\text{BM25}(tok_j,\hat{X}_i)=
\frac{f(tok_j,\hat{X}_i)\cdot (\text{IDF}(tok_j)\cdot (z_1+1))}
{f(tok_j,\hat{X}_i)+z_1\cdot \left(1-b+b\cdot \frac{|\hat{X}_i|}{avgdl}\right)},
$$
and then masks a token when its normalized BM25 score exceeds a threshold $\tau$:
$$
\text{Mask}(tok_j)=
\begin{cases}
1 & \text{if }\displaystyle \frac{\text{BM25}(tok_j,\hat{X}_i)}{\sum_{tok'_j\in \hat{X}_i}\text{BM25}(tok'_j,\hat{X}_i)} > \tau,\\[1.2ex]
0 & \text{otherwise}.
\end{cases}
$$
The reported pretraining choice is $\tau=0.1$, and the language model’s maximum input sequence length is $L_{\max}=512$; about 70% of tokenized account sequences are shorter than 512, corresponding to roughly 15 transaction records per account [2509.03860].

A key distinction therefore emerges between two TLM design choices already present in the literature: generic MLM on transaction sentences and anomaly-sensitive masking driven by information-retrieval statistics.

## 4. Hybridization with graph structure and relational priors

Recent Transaction Language Models are not purely sequential encoders. Both major Ethereum implementations combine the language-model branch with graph-based components, but they do so in different ways.

TLMG4Eth augments semantic embeddings with two graph views. The first is a Transaction Attribute Similarity Graph (TASG), a vocabulary graph $\mathcal{G}_w=(\mathcal{V}_w,\mathcal{E}_w)$ whose edges are built either from normalized PMI,
$$
\text{NPMI}(w_i,w_j)=
\frac{\log \frac{p(w_i,w_j)}{p(w_i)p(w_j)}}{-\log p(w_i,w_j)},
$$
with an edge added when $\text{NPMI}(w_i,w_j)>\theta$, or from TF-IDF sentence-word links,
$$
\text{TF-IDF}(w_i,d)=\text{TF}(w_i,d)\times \log\left(\frac{N}{|\{d\in D:w_i\in d\}|}\right).
$$
The second is an Account Interaction Graph (AIG), a weighted account graph $G=(V,E)$ whose edge weight $w_{ij}$ is the number of transactions between accounts $i$ and $j$. Semantic and similarity embeddings are concatenated,
$$
E_i=[E_i^s;E_i^g],
$$
then fused by a 12-layer, 12-head multi-head attention network. The resulting account representation initializes the AIG node features, after which a GCN is applied:
$$
\mathbf{H}^{(l+1)}=\sigma\left(\mathbf{\hat{A}}\mathbf{H}^{(l)}\mathbf{W}^{(l)}\right), \qquad
\mathbf{\hat{A}}=\mathbf{D}^{-\frac{1}{2}}(\mathbf{A}+\mathbf{I})\mathbf{D}^{-\frac{1}{2}}.
$$
Final prediction interpolates the MAN and GCN outputs:
$$
\text{Pred}=\lambda \mathbf{Z}_{\text{GCN}}+(1-\lambda)\mathbf{Z}_{\text{MAN}}.
$$
Thus, transaction semantics, transaction similarity, and account interaction structure are trained jointly rather than fused only after separate optimization [2409.07494].

KGBERT4Eth uses a different graph object: a Transaction Knowledge Graph (TKG) that incorporates expert-curated domain knowledge and fund-flow patterns. Its TLM and TKG communicate through Mask-invariant Attention Synergy (MiAS). The TLM hidden states are
$$
\boldsymbol{\tilde{h}^{(l)}}=[\boldsymbol{\tilde{h}^{(l)}_{CLS}},\boldsymbol{\tilde{h}^{(l)}_1},\dots,\boldsymbol{\tilde{h}^{(l)}_L}],
$$
and the TKG entity embeddings are
$$
\boldsymbol{\tilde{v}^{(l)}}=[\boldsymbol{\tilde{v}^{(l)}_1},\boldsymbol{\tilde{v}^{(l)}_2},\dots,\boldsymbol{\tilde{v}^{(l)}_{|V|}}].
$$
Cross-attention from graph to text is written as
$$
\alpha_{h,v}=\text{Softmax}\left(\frac{\boldsymbol{q}_h\boldsymbol{k}_v^\top+M}{\sqrt{d}}\right), \qquad
\boldsymbol{\hat{h}^{(l)}}=\sum_{v\in \mathscr{D}(h)} \alpha_{h,v}\cdot \boldsymbol{v}_v.
$$
The visibility matrix $M$ restricts semantic-graph interaction to the $[CLS]$ token only, so graph information enriches the account-level summary without directly corrupting masked token positions. Joint pretraining sums the TLM’s BMP loss and the TKG’s transaction link prediction loss:
$$
\mathcal{L}(\boldsymbol{\theta})=\mathcal{L}_{\text{BMP}}(\theta^m)+\mathcal{L}_{\text{TLP}}(\theta^l).
$$
This makes the TLM not an isolated encoder but one branch in a co-trained semantic-relational system [2509.03860].

## 5. Applications and empirical performance

The principal reported application domain for Transaction Language Models is Ethereum account-level fraud analysis. TLMG4Eth targets phishing account detection on three binary datasets—MulDiGraph, B4E, and SPN—and reports that the full model outperforms the strongest baseline by roughly 20.12% on MulDiGraph, 14.12% on B4E, and 10.32% on SPN in F1. Its ablation table shows that the TLM alone is already strong: F1 on MulDiGraph improves from 0.8804 for TLM Only to 0.9041 with the full similarity-enhanced model, on B4E from 0.7923 to 0.8123, and on SPN from 0.8018 to 0.8146. The TF-IDF-based similarity graph is the best TASG construction, and the best interpolation occurs at intermediate rather than extreme values of $\lambda$, around 0.7 on MulDiGraph and 0.8 on B4E and SPN [2409.07494].

KGBERT4Eth extends the scope from phishing detection to multi-task Ethereum fraud detection, specifically phishing account detection and account de-anonymization. The abstract reports absolute F1-score improvements of 8–16% on three phishing detection benchmarks and 6–26% on four de-anonymization datasets. In the phishing setting, the reported F1 scores are 87.60 on MultiGraph, 85.43 on B4E, and 88.02 on SPN, compared with BERT4ETH scores of 64.20, 69.05, and 72.98 and ZipZap scores of 65.25, 68.48, and 72.31. In de-anonymization, the overall F1 reaches 90.52, compared with 76.13 for BERT4ETH and 75.91 for ZipZap. The ablations further isolate the TLM contribution: removing the TKG yields phishing F1 86.95 and de-anonymization F1 88.91; removing BMP yields 85.27 and 86.67; removing MiAS yields 86.54 and 86.81. These numbers indicate that the sequential semantic branch is already strong on its own, while graph coupling, biased masking, and coordination supply additional gains [2509.03860].

Across both systems, the empirical pattern is consistent. Transaction text encoding is not treated as a replacement for graph structure, but as a high-value semantic branch that becomes stronger when paired with token-similarity graphs, account-interaction graphs, or knowledge graphs. A plausible implication is that the recent literature regards transaction semantics, similarity, and topology as complementary rather than competing feature spaces.

## 6. Misconceptions, adjacent meanings, and broader context

A common misconception is to treat Transaction Language Model as synonymous with any work labeled “TLM.” The arXiv record does not support that reading. In hardware and system design, TLM most often means Transaction Level Modeling: a bus or processor abstraction used for SoC performance analysis, early software development, verification, or co-simulation rather than a language model over transactions [0710.4808][2505.10145]. In NLP, TLM may instead denote Task-Driven Language Modeling, Translation Language Modeling, multilingual continual pretraining, Test-Time Learning for LLMs, or bandit-based optimization of Transformer language model pre-training [2111.04130][2501.05700].

There are also adjacent lines of research that are transaction-oriented without being Transaction Language Models in the Ethereum sense. TStreamLLM proposes a systems framework that applies transactional stream-processing principles—consistency, isolation, durability, concurrency control, and fault tolerance—to continuous LLM updates and usage, but it does not textualize transaction records or define a transaction-language encoder [2307.08225]. TACLA, by contrast, operationalizes Transactional Analysis as a multi-agent control architecture for dialogue simulation, where Parent, Adult, and Child ego-state agents are orchestrated according to contextual triggers and script-like priors; this is transaction-aware in the psychological sense, not in the sense of modeling financial transaction corpora [2510.17913].

The present state of the term is therefore narrow but technically specific. In its strict recent meaning, a Transaction Language Model is an account-centric, Transformer-based semantic encoder over serialized transaction histories, typically trained with masked prediction and often integrated with graph learning. Its most developed uses are in Ethereum fraud detection, where the central research question is how to turn repetitive, structured, largely numerical transaction logs into contextual embeddings that preserve behavioral semantics, anomaly salience, and relational compatibility [2409.07494][2509.03860].

Source: https://www.emergentmind.com/topics/transaction-language-model-tlm