Transaction-Token Contrastive Language Model (TxCLM)

• The paper introduces semantic tokenization and a token-aware contrastive learning objective to convert numerical transaction tuples into meaningful contextual embeddings.
• TxCLM leverages a BERT-style Transformer with dual pretraining heads—one for masked language modeling and one for contrastive learning—to address embedding anisotropy in transaction data.
• Empirical results show that integrating TxCLM with graph-based embeddings improves recall and F1 scores, underscoring the benefit of combining semantic and relational features in fraud detection.

Searching arXiv for the specified paper and closely related context papers. arxiv_search.query({"^^^^^^^^7search_query7 arxiv_search.query({"^^^^^^^^7search_query7 OR ^^^^^^^^7all:\7 fraud detection\"","^^^^^^^^7start7 Transaction-Token Contrastive LLM (TxCLM) is the language-modeling component of LMAE^^^^7search_query7^^^^Eth, a multi-view learning framework for Ethereum fraud detection that combines transaction semantics, masked graph embedding, and expert knowledge (&&&^^^^7search_query7^^^^&&&). TxCLM is designed to transform context-independent numerical transaction records into logically cohesive linguistic representations, and it does so with a standard BERT-style Transformer Encoder equipped with two pretraining heads: a Masked LLM head and a Token-aware Contrastive Learning head. Within LMAE^^^^7search_query7^^^^Eth, TxCLM addresses two limitations identified for prior Ethereum fraud detection methods: reliance on flat, context-free numerical transaction tuples and embedding anisotropy caused by the homogeneity of raw transaction values (&&&^^^^7search_query7^^^^&&&).

^^^^7id:(Jia et al., 4 Sep 2025)7^^^^. Conceptual role and problem setting

TxCLM is motivated by the observation that prior methods such as BERT^^^^7search_query7^^^^ETH represent each transaction as a flat, context-free tuple of numbers—amount, timestamp, and direction—so the representation remains “context independent” and does not expose the underlying semantics or intent behind a transfer (&&&^^^^7search_query7^^^^&&&). The same source identifies a second issue: the pervasive homogeneity of raw transaction values tends to collapse transaction-token embeddings into a narrow cone, producing embedding anisotropy and reducing discriminative power.

The model’s central intervention is semantic tokenization. Each numerical field is converted into a short linguistic phrase, allowing a Transformer to operate on transaction sequences as if they were sentence-like inputs. This permits masked-language modeling over transaction records and, in parallel, token-level contrastive learning that explicitly separates semantically distinct positions within a sequence.

Within the broader LMAE^^^^7search_query7^^^^Eth pipeline, TxCLM provides sequence-level semantic embeddings, while MAGAE provides graph embeddings derived from account relations. A cross-attention fusion network then unifies the two representations. This suggests that TxCLM is not intended as a standalone fraud detector in the full system, but as the semantic encoder for transaction histories within a multi-view architecture.

^^^^7start7^^^^. Transaction semantics and linguistic tokenization

TxCLM begins from an account-level transaction sequence with PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^ transactions: PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7^^^^ where PRESERVED_PLACEHOLDER_^^^^7start7^^^^ is the transfer amount, PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^ is the direction, and PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^ is the timestamp (&&&^^^^7search_query7^^^^&&&).

The defining preprocessing step is a linguistic conversion of each tuple into three labeled tokens: PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^

These per-transaction token triples are concatenated into a token sequence: PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^ which is then fed into a standard subword tokenizer such as WordPiece and into the Transformer encoder.

The significance of this formulation is precise. TxCLM does not replace numeric transaction content with free-form text; rather, it prepends descriptive labels—“amount:”, “direction:”, and “timestamp:” —to the original fields. A common misconception is to treat the model as conventional natural-language understanding over blockchain narratives. The formulation instead remains tightly coupled to structured transaction data, with the linguistic surface serving as a mechanism for contextual encoding.

^^^^7max_results7^^^^. Anchor-enhanced pretraining objectives

TxCLM uses an “anchor-enhanced” setup with two models. An Anchor model $A$, which is frozen, processes the full unmasked sequence $\mathcal{C}$. An Enhanced model $E$, which is trainable, processes a masked version PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7search_query7^^^^ (&&&^^^^7search_query7^^^^&&&).

For the unmasked sequence, the Anchor model produces token embeddings

PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7id:(Jia et al., 4 Sep 2025)7^^^^

For the masked sequence, the Enhanced model produces

PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7start7^^^^

The first objective is the token-aware contrastive learning loss. For each masked position PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7max_results7^^^^, the model pulls PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7search_query7^^^^ toward its positive counterpart PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7all:\7^^^^ and pushes it away from all other tokens PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7 OR all:\7^^^^ in the same sequence. Using cosine similarity

PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)77^^^^

and temperature PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7^^^^8, the loss is

PRESERVED_PLACEHOLDER_^^^^7id:(Jia et al., 4 Sep 2025)7^^^^9

The second objective is masked transaction modeling, implemented as a standard BERT-style masked language modeling loss: PRESERVED_PLACEHOLDER_^^^^7start7search_query7^^^^

The combined pretraining objective is

PRESERVED_PLACEHOLDER_^^^^7start7id:(Jia et al., 4 Sep 2025)7^^^^

An important implementation detail is that no inter-sequence negatives are used: contrastive negatives come from other tokens in the same sequence. This sharply specifies the contrastive geometry. The model is not trained to separate accounts from one another at the contrastive stage; it is trained to distinguish masked token positions against alternative positions within the same account history.

^^^^7search_query7^^^^. Encoder architecture and optimization regime

The encoder follows BERT-base: ^^^^7id:(Jia et al., 4 Sep 2025)7start7^^^^ Transformer-encoder layers, self-attention heads, and hidden size PRESERVED_PLACEHOLDER_^^^^7start7start7^^^^ (&&&^^^^7search_query7^^^^&&&). Input embeddings are the sum of token embeddings, segment embeddings, and positional embeddings. Dropout and layer normalization are used as in standard BERT.

The reported pretraining setup uses the entire unlabeled transaction corpora of MulDiGraph, B^^^^7search_query7^^^^E, and/or SPN. Training uses AdamW with learning rate PRESERVED_PLACEHOLDER_^^^^7start7max_results7^^^^, batch size ^^^^7start7all:\7 OR all:\7^^^^ sequences, and ^^^^7id:(Jia et al., 4 Sep 2025)7search_query7^^^^–^^^^7start7search_query7^^^^ epochs over the unlabeled corpus. Warmup occupies the first ^^^^7id:(Jia et al., 4 Sep 2025)7search_query7^^^^% of total steps, followed by linear decay.

TxCLM-specific hyperparameters include a mask ratio PRESERVED_PLACEHOLDER_^^^^7start7search_query7^^^^ and contrastive temperature PRESERVED_PLACEHOLDER_^^^^7start7all:\7^^^^ in the reported pretraining setup. Elsewhere in the same description, the mask ratio is given as typically ^^^^7id:(Jia et al., 4 Sep 2025)7all:\7^^^^–^^^^7max_results7search_query7 with best values at ^^^^7id:(Jia et al., 4 Sep 2025)7all:\7^^^^% on MulDiGraph/SPN and ^^^^7max_results7search_query7^^^^% on B^^^^7search_query7^^^^E. Equal weighting is used between PRESERVED_PLACEHOLDER_^^^^7start7 OR all:\7^^^^ and PRESERVED_PLACEHOLDER_^^^^7start77^^^^.

These details clarify the model’s design priorities. The architecture itself remains close to standard BERT, while the principal novelty lies in the transaction-specific tokenization and the addition of token-aware contrastive learning to masked modeling.

^^^^7all:\7^^^^. Interface with graph representations and cross-attention fusion

After pretraining, TxCLM encodes each account’s transaction text into a sequence of embeddings

PRESERVED_PLACEHOLDER_^^^^7start7^^^^8

LABOR-MAGAE provides a graph embedding

PRESERVED_PLACEHOLDER_^^^^7start7^^^^9

for each account (&&&^^^^7search_query7^^^^&&&).

Cross-attention fusion proceeds in two stages. First, PRESERVED_PLACEHOLDER_^^^^7max_results7search_query7^^^^ learnable semantic aggregate tokens

PRESERVED_PLACEHOLDER_^^^^7max_results7id:(Jia et al., 4 Sep 2025)7^^^^

compress the TxCLM sequence: PRESERVED_PLACEHOLDER_^^^^7max_results7start7^^^^ yielding

PRESERVED_PLACEHOLDER_^^^^7max_results7max_results7^^^^

Second, each compressed semantic token PRESERVED_PLACEHOLDER_^^^^7max_results7search_query7^^^^ is fused with the graph vector PRESERVED_PLACEHOLDER_^^^^7max_results7all:\7^^^^: PRESERVED_PLACEHOLDER_^^^^7max_results7 OR all:\7^^^^

Finally, PRESERVED_PLACEHOLDER_^^^^7max_results77^^^^ fusion tokens

PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^8

cross-attend to PRESERVED_PLACEHOLDER_^^^^7max_results7^^^^9 to produce a single fused representation

PRESERVED_PLACEHOLDER_^^^^7search_query7search_query7^^^^

This division of labor is methodologically important. TxCLM contributes semantic structure derived from transaction histories; MAGAE contributes relational structure derived from the account graph. The fusion module is explicitly designed to preserve both views rather than collapsing one into the other before interaction.

^^^^7 OR all:\7^^^^. Empirical behavior and interpretive significance

The reported ablation on B^^^^7search_query7^^^^E, in the fused MAGAE+CAFN setting, isolates the contribution of the token-aware contrastive objective. Full LMAE^^^^7search_query7^^^^Eth attains PRESERVED_PLACEHOLDER_^^^^7search_query7id:(Jia et al., 4 Sep 2025)7^^^^ and PRESERVED_PLACEHOLDER_^^^^7search_query7start7^^^^, whereas removing PRESERVED_PLACEHOLDER_^^^^7search_query7max_results7^^^^ yields PRESERVED_PLACEHOLDER_^^^^7search_query7search_query7^^^^ and PRESERVED_PLACEHOLDER_^^^^7search_query7all:\7^^^^. The paper summarizes this as a drop of approximately PRESERVED_PLACEHOLDER_^^^^7search_query7 OR all:\7^^^^ percentage points in F^^^^7id:(Jia et al., 4 Sep 2025)7^^^^ (&&&^^^^7search_query7^^^^&&&).

Variant	F^^^^7id:(Jia et al., 4 Sep 2025)7^^^^	BAcc
Full LMAE^^^^7search_query7^^^^Eth	^^^^7search_query7^^^^.8^^^^7id:(Jia et al., 4 Sep 2025)7search_query7max_results7	^^^^7search_query7^^^^.8^^^^7 OR all:\7search_query7id:(Jia et al., 4 Sep 2025)7^^^^
w/o contrastive PRESERVED_PLACEHOLDER_^^^^7search_query77^^^^	^^^^7search_query7^^^^.78^^^^7search_query7 OR all:\7^^^^	^^^^7search_query7^^^^.8^^^^7max_results7

A second result concerns representation geometry: self-similarity visualization shows that TxCLM produces more diverse token embeddings, with lower average pairwise cosine similarity, than standard BERT or RoBERTa. This directly addresses the anisotropy problem identified in the motivation.

The language-model comparison, again in the fused setting with MAGAE, reports the following metrics. BERT-base obtains PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^8, PRESERVED_PLACEHOLDER_^^^^7search_query7^^^^9, PRESERVED_PLACEHOLDER_^^^^7all:\7search_query7^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\7id:(Jia et al., 4 Sep 2025)7^^^^; RoBERTa-base obtains PRESERVED_PLACEHOLDER_^^^^7all:\7start7^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\7max_results7^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\7search_query7^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\7all:\7^^^^; ALBERT-base obtains PRESERVED_PLACEHOLDER_^^^^7all:\7 OR all:\7^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\77^^^^, PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^8, PRESERVED_PLACEHOLDER_^^^^7all:\7^^^^9; ELECTRA-base obtains PRESERVED_PLACEHOLDER_^^^^7 OR all:\7search_query7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7id:(Jia et al., 4 Sep 2025)7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7start7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7max_results7^^^^; VGCN-BERT obtains PRESERVED_PLACEHOLDER_^^^^7 OR all:\7search_query7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7all:\7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7 OR all:\7^^^^, PRESERVED_PLACEHOLDER_^^^^7 OR all:\77^^^^; and TxCLM obtains PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^8, PRESERVED_PLACEHOLDER_^^^^7 OR all:\7^^^^9, $A$^^^^7search_query7^^^^, $A$^^^^7id:(Jia et al., 4 Sep 2025)7^^^^. The source states that TxCLM yields the highest recall and F^^^^7id:(Jia et al., 4 Sep 2025)7^^^^ once fused, demonstrating the standalone semantic gain.

At the full-framework level, LMAE^^^^7search_query7^^^^Eth is evaluated against ^^^^7start7id:(Jia et al., 4 Sep 2025)7^^^^ baseline approaches on three datasets, and the reported result is that it outperforms the best baseline by over ^^^^7id:(Jia et al., 4 Sep 2025)7search_query7^^^^% in F^^^^7id:(Jia et al., 4 Sep 2025)7^^^^-score on two of the datasets. Because TxCLM is one component of that system, this should not be interpreted as a direct standalone TxCLM result. A plausible implication is that TxCLM’s contribution is strongest when transaction semantics are combined with graph-derived account structure rather than used in isolation.