---
title: Transaction-Token Contrastive Language Model (TxCLM)
url: https://www.emergentmind.com/topics/transaction-token-contrastive-language-model-txclm
type: topic
---

# Transaction-Token Contrastive Language Model (TxCLM)

Searching arXiv for the specified paper and closely related context papers.
arxiv_search.query({"search_query":"id:2509.03939","start":0,"max_results":5})
arxiv_search.query({"search_query":"all:\"BERT4ETH\" OR all:\"Ethereum fraud detection\"","start":0,"max_results":10})
Transaction-Token Contrastive Language Model (TxCLM) is the language-modeling component of LMAE4Eth, a multi-view learning framework for Ethereum fraud detection that combines transaction semantics, masked graph embedding, and expert knowledge [2509.03939]. TxCLM is designed to transform context-independent numerical transaction records into logically cohesive linguistic representations, and it does so with a standard BERT-style Transformer Encoder equipped with two pretraining heads: a Masked Language Model head and a Token-aware Contrastive Learning head. Within LMAE4Eth, TxCLM addresses two limitations identified for prior Ethereum fraud detection methods: reliance on flat, context-free numerical transaction tuples and embedding anisotropy caused by the homogeneity of raw transaction values [2509.03939].

## 1. Conceptual role and problem setting

TxCLM is motivated by the observation that prior methods such as BERT4ETH represent each transaction as a flat, context-free tuple of numbers—amount, timestamp, and direction—so the representation remains “context independent” and does not expose the underlying semantics or intent behind a transfer [2509.03939]. The same source identifies a second issue: the pervasive homogeneity of raw transaction values tends to collapse transaction-token embeddings into a narrow cone, producing embedding anisotropy and reducing discriminative power.

The model’s central intervention is semantic tokenization. Each numerical field is converted into a short linguistic phrase, allowing a Transformer to operate on transaction sequences as if they were sentence-like inputs. This permits masked-language modeling over transaction records and, in parallel, token-level contrastive learning that explicitly separates semantically distinct positions within a sequence.

Within the broader LMAE4Eth pipeline, TxCLM provides sequence-level semantic embeddings, while MAGAE provides graph embeddings derived from account relations. A cross-attention fusion network then unifies the two representations. This suggests that TxCLM is not intended as a standalone fraud detector in the full system, but as the semantic encoder for transaction histories within a multi-view architecture.

## 2. Transaction semantics and linguistic tokenization

TxCLM begins from an account-level transaction sequence with \(N\) transactions:
\[
\mathcal{T}=\{t_1,\dots,t_N\},\quad
t_i=(v_i,d_i,\tau_i),
\]
where \(v_i\) is the transfer amount, \(d_i\in\{-1,+1\}\) is the direction, and \(\tau_i\) is the timestamp [2509.03939].

The defining preprocessing step is a linguistic conversion of each tuple into three labeled tokens:
\[
\mathcal{L}(t_i)=\{\text{“amount: }v_i”,\;\text{“direction: }d_i”,\;\text{“timestamp: }\tau_i”\}.
\]

These per-transaction token triples are concatenated into a token sequence:
\[
\mathcal{C} = [\,\mathcal{L}(t_1)\;\Vert\;\mathcal{L}(t_2)\;\Vert\;\dots\;\Vert\;\mathcal{L}(t_N)\,],
\]
which is then fed into a standard subword tokenizer such as WordPiece and into the Transformer encoder.

The significance of this formulation is precise. TxCLM does not replace numeric transaction content with free-form text; rather, it prepends descriptive labels—“amount:”, “direction:”, and “timestamp:” —to the original fields. A common misconception is to treat the model as conventional natural-language understanding over blockchain narratives. The formulation instead remains tightly coupled to structured transaction data, with the linguistic surface serving as a mechanism for contextual encoding.

## 3. Anchor-enhanced pretraining objectives

TxCLM uses an “anchor-enhanced” setup with two models. An Anchor model \(A\), which is frozen, processes the full unmasked sequence \(\mathcal{C}\). An Enhanced model \(E\), which is trainable, processes a masked version \(\tilde{\mathcal{C}}\) [2509.03939].

For the unmasked sequence, the Anchor model produces token embeddings
\[
h=[h_1,\dots,h_n],\quad h_i\in\mathbb{R}^{d_{\rm LM}}.
\]
For the masked sequence, the Enhanced model produces
\[
\tilde h=[\tilde h_1,\dots,\tilde h_n].
\]

The first objective is the token-aware contrastive learning loss. For each masked position \(i\), the model pulls \(\tilde h_i\) toward its positive counterpart \(h_i\) and pushes it away from all other tokens \(\{h_j\}_{j\ne i}\) in the same sequence. Using cosine similarity
\[
\operatorname{sim}(x,y)=\frac{x^\top y}{\|x\|\|y\|},
\]
and temperature \(\tau\), the loss is
\[
\mathcal{L}_{\rm Ta}
= -\,\sum_{i=1}^n \mathbf{1}_{\{\tilde x_i\;\mathrm{masked}\}}
\,\log\frac{\exp\bigl(\operatorname{sim}(\tilde h_i,h_i)/\tau\bigr)}
{\sum_{j=1}^n \exp\bigl(\operatorname{sim}(\tilde h_i,h_j)/\tau\bigr)}.
\]

The second objective is masked transaction modeling, implemented as a standard BERT-style masked language modeling loss:
\[
\mathcal{L}_{\rm MLM}
= -\sum_{i\in\mathcal{M}} \log P\bigl(x_i \mid \tilde{\mathcal{C}}\bigr).
\]

The combined pretraining objective is
\[
\mathcal{L}_{\rm TxCLM}
= \mathcal{L}_{\rm MLM} + \mathcal{L}_{\rm Ta}.
\]

An important implementation detail is that no inter-sequence negatives are used: contrastive negatives come from other tokens in the same sequence. This sharply specifies the contrastive geometry. The model is not trained to separate accounts from one another at the contrastive stage; it is trained to distinguish masked token positions against alternative positions within the same account history.

## 4. Encoder architecture and optimization regime

The encoder follows BERT-base: 12 Transformer-encoder layers, self-attention heads, and hidden size \(d_{\rm LM}=768\) [2509.03939]. Input embeddings are the sum of token embeddings, segment embeddings, and positional embeddings. Dropout and layer normalization are used as in standard BERT.

The reported pretraining setup uses the entire unlabeled transaction corpora of MulDiGraph, B4E, and/or SPN. Training uses AdamW with learning rate \(1\mathrm{e}{-4}\), batch size 256 sequences, and 10–20 epochs over the unlabeled corpus. Warmup occupies the first 10% of total steps, followed by linear decay.

TxCLM-specific hyperparameters include a mask ratio \(\alpha=0.15\) and contrastive temperature \(\tau=0.1\) in the reported pretraining setup. Elsewhere in the same description, the mask ratio is given as typically 15–30%, with best values at 15% on MulDiGraph/SPN and 30% on B4E. Equal weighting is used between \(\mathcal{L}_{\rm MLM}\) and \(\mathcal{L}_{\rm Ta}\).

These details clarify the model’s design priorities. The architecture itself remains close to standard BERT, while the principal novelty lies in the transaction-specific tokenization and the addition of token-aware contrastive learning to masked modeling.

## 5. Interface with graph representations and cross-attention fusion

After pretraining, TxCLM encodes each account’s transaction text into a sequence of embeddings
\[
\mathbf{S}\in\mathbb{R}^{N\times d_s}.
\]
LABOR-MAGAE provides a graph embedding
\[
\hat{\mathbf{x}}\in\mathbb{R}^{d_g}
\]
for each account [2509.03939].

Cross-attention fusion proceeds in two stages. First, \(k_s\) learnable semantic aggregate tokens
\[
\mathbf{A}^s\in\mathbb{R}^{k_s\times d_s}
\]
compress the TxCLM sequence:
\[
\mathbf{Z}^s=\mathrm{softmax}\Bigl(\tfrac{(\mathbf{A}^sW_Q)(\mathbf{S}W_K)^\top}{\sqrt{d_s}}\Bigr)\,(\mathbf{S}W_V),
\]
yielding
\[
\mathbf{Z}^s\in\mathbb{R}^{k_s\times d_s}.
\]

Second, each compressed semantic token \(\mathbf{z}_i^s\) is fused with the graph vector \(\hat{\mathbf{x}}\):
\[
\mathbf{z}_i^{sg}
= \sigma\bigl(W_s\,\mathbf{z}_i^s + W_g\,\hat{\mathbf{x}} + b\bigr),
\quad i=1,\dots,k_s.
\]

Finally, \(k_f\) fusion tokens
\[
\mathbf{A}^f\in\mathbb{R}^{k_f\times d_f}
\]
cross-attend to \(\{\mathbf{z}_i^{sg}\}\) to produce a single fused representation
\[
\mathbf{F}\in\mathbb{R}^{d_f}.
\]

This division of labor is methodologically important. TxCLM contributes semantic structure derived from transaction histories; MAGAE contributes relational structure derived from the account graph. The fusion module is explicitly designed to preserve both views rather than collapsing one into the other before interaction.

## 6. Empirical behavior and interpretive significance

The reported ablation on B4E, in the fused MAGAE+CAFN setting, isolates the contribution of the token-aware contrastive objective. Full LMAE4Eth attains \( \text{F1}=0.8143 \) and \( \text{BAcc}=0.8641 \), whereas removing \(\mathcal{L}_{\rm Ta}\) yields \( \text{F1}=0.7846 \) and \( \text{BAcc}=0.8397 \). The paper summarizes this as a drop of approximately \(2.97\) percentage points in F1 [2509.03939].

| Variant | F1 | BAcc |
|---|---:|---:|
| Full LMAE4Eth | 0.8143 | 0.8641 |
| w/o contrastive \((-\mathcal{L}_{\rm Ta})\) | 0.7846 | 0.8397 |

A second result concerns representation geometry: self-similarity visualization shows that TxCLM produces more diverse token embeddings, with lower average pairwise cosine similarity, than standard BERT or RoBERTa. This directly addresses the anisotropy problem identified in the motivation.

The language-model comparison, again in the fused setting with MAGAE, reports the following metrics. BERT-base obtains \(P=0.8046\), \(R=0.7680\), \(F1=0.7859\), \( \text{BAcc}=0.8374\); RoBERTa-base obtains \(0.7792\), \(0.7668\), \(0.7730\), \(0.8291\); ALBERT-base obtains \(0.7888\), \(0.7894\), \(0.7891\), \(0.8419\); ELECTRA-base obtains \(0.7744\), \(0.7692\), \(0.7718\), \(0.8286\); VGCN-BERT obtains \(0.8002\), \(0.7859\), \(0.7930\), \(0.8439\); and TxCLM obtains \(P=0.7903\), \(R=0.8397\), \(F1=0.8143\), \( \text{BAcc}=0.8641\). The source states that TxCLM yields the highest recall and F1 once fused, demonstrating the standalone semantic gain.

At the full-framework level, LMAE4Eth is evaluated against 21 baseline approaches on three datasets, and the reported result is that it outperforms the best baseline by over 10% in F1-score on two of the datasets. Because TxCLM is one component of that system, this should not be interpreted as a direct standalone TxCLM result. A plausible implication is that TxCLM’s contribution is strongest when transaction semantics are combined with graph-derived account structure rather than used in isolation.

Source: https://www.emergentmind.com/topics/transaction-token-contrastive-language-model-txclm