ScamGPT-J: A Scam Simulation Model

• ScamGPT-J is a fine-tuned language model designed to simulate scammer conversational patterns for real-time scam prevention.
• It employs soft prompt tuning on GPT-J-6B using synthetic scam dialogues from categories like Authority, Job, Love, and Investment scams.
• The model uses cosine similarity between predicted scam replies and actual messages to provide continuous warnings, aiding user reflection.

ScamGPT-J is a fine-tuned generative LLM for messaging-scam prevention that simulates plausible scammer replies in real time and uses the resemblance between those simulated replies and an interlocutor’s actual messages as a user-facing warning signal. Unlike a binary scam classifier, it is designed to support self-identification of scams through persuasion and reflection: if an incoming message is markedly similar to ScamGPT-J’s predicted scammer continuation, that similarity is presented to the user as evidence that the conversation may be fraudulent. The system was introduced as a prompt-tuned GPT-J-6B model trained on curated scam dialogues across Authority, Job, Love, and Investment scams, and was later situated as the “Simulate” component within the broader Anticipate, Simulate, Reason framework for interpretable, human-centered scam defense (Tan et al., 2024, Tan et al., 23 Jul 2025).

1. Conceptual basis and problem setting

ScamGPT-J was motivated by the scale and structure of instant messaging fraud. One reported estimate places global scam losses at US$1.026 trillion in 2023, with approximately 58% attributed to messaging platform scams. The underlying threat model is not limited to overtly malicious messages: scams often begin with innocuous contact, then escalate through urgency, desirability, impersonation, and other manipulative tactics that exploit emotional state, anchoring effects, and sunk-cost fallacies. Within this setting, the paper presents ScamGPT-J as a “novel, user-centered paradigm for scam prevention” that emphasizes persuasion and reflection rather than prescriptive classification (Tan et al., 2024).

The system organizes scam behavior into four major categories: Authority, Job, Love, and Investment. These categories serve both as a dataset design scaffold and as a practical decomposition of common conversational fraud. Authority scams center on impersonation of officials; Job scams on fake recruitment and fee extraction; Love scams on trust-building followed by financial requests; Investment scams on guaranteed returns and deposit pressure. In later work, the same four categories define the scope of the simulation engine inside ASR, where ScamGPT-J supplies scammer-like continuations and a separate reasoning model explains cues and tactics to the user (Tan et al., 23 Jul 2025).

A recurring misconception is that ScamGPT-J is a scam classifier that emits a definitive probability of fraud. The original system does not enforce a classification threshold, does not provide a fixed cutoff $\tau$, and explicitly warns that the similarity scores are context-free and should not be interpreted as a definitive “scamness” measure. Its intended effect is cognitive: repeated convergence between simulated and observed scammer replies is meant to move users from heuristic to systematic processing.

2. Model architecture, prompt tuning, and corpus construction

ScamGPT-J starts from EleutherAI’s GPT-J-6B, chosen for openness and transparency. The base model remains frozen, and adaptation is performed with soft prompt tuning through the Hugging Face PEFT library. The learned soft prompt is appended to the model’s input embeddings and is seeded with the instruction: “Assuming you are a scammer, your goal is to trick a victim to give you money.” Only the soft prompt parameters are trained; the GPT-J weights are kept fixed. For technical evaluation, the generation context consists of the previous two dialogue turns. The paper describes soft-prompt tuning and comparative performance, but does not specify the exact loss function, optimizer, hyperparameters, tokenizer details, training hardware, runtime, or random seeds (Tan et al., 2024).

The training corpus is predominantly synthetic, reflecting the scarcity and privacy sensitivity of real scam transcripts. The seed set contains 90 high-quality conversation samples created using ChatGPT-4 from police advisories, news screenshots, and hand-crafted scenarios based on the authors’ interactions with scammers. Each seed conversation was then expanded with 10 GPT-3.5 Turbo variants that changed names and style while preserving the core scam mechanism. Outputs were manually vetted to correct inconsistent or illogical exchanges and to remove duplicates. Limited real messages and audio recordings were collected, but were used only as seeds for further synthesis because of privacy and confidentiality constraints. The resulting corpus contains 902 total conversations, split into 812 for training and 90 for validation; the paper does not specify multilingual coverage, and the examples are in English (Tan et al., 2024).

This construction strategy is central to ScamGPT-J’s identity. It is neither a general-purpose assistant nor a fully supervised detector trained on large-scale labeled production traffic. It is a domain-adapted generator built to internalize scammer conversational patterns while minimizing direct exposure of private victim data.

3. Runtime workflow and the semantics of “technical congruence”

The end-to-end workflow comprises data ingestion, message preprocessing, generation, similarity matching, decision signaling, and logging. In deployment terms, the system monitors an ongoing conversation between a potential victim and the messaging counterpart. During evaluation, conversations are parsed into turns, and ScamGPT-J receives the previous two interactions as context. It then generates the next plausible scammer reply, but that output is shown only to the user; the counterpart does not see it. The system computes semantic similarity between the generated reply and the actual incoming message using Sentence-BERT embeddings and cosine similarity, then displays the score immediately after each counterpart message (Tan et al., 2024).

Cosine similarity is defined in the paper as

$$s(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x}\cdot\mathbf{y}}{\|\mathbf{x}\|\,\|\mathbf{y}\|}.$$

For a conversation with $K$ generated turns and similarity scores $s_1,\dots,s_K$, two aggregate measures are reported:

$$\text{MeanSim} = \frac{1}{K}\sum_{j=1}^{K} s_j, \qquad \text{MaxSim} = \max_{1\le j \le K} s_j.$$

The paper calls this semantic alignment “technical congruence”: the closer ScamGPT-J’s simulated scammer reply is to the actual next message in a scam dialogue, the higher the congruence. In the representative job-scam example, the scammer offers flexible work and salary details, then moves to a registration fee request; ScamGPT-J privately generates analogous salary and fee-escalation continuations. Two consecutive similarity scores, 0.78 and 0.69, yield a conversation mean similarity of 0.74 and a maximum similarity of 0.78 (Tan et al., 2024).

This design matters because the decision signal is continuous rather than categorical. The system does not say “this is a scam”; it says, in effect, “this is what a scammer would plausibly say next, and the live conversation is behaving similarly.” In the user-study infrastructure, all interactions and ratings are logged through a custom survey web application, with a Hugging Face inference endpoint and Firestore storage.

4. Evaluation results and measured user effects

Two evaluation regimes define the empirical profile of ScamGPT-J. The first tests whether prompt tuning materially improves scammer-like next-turn generation relative to untuned GPT-J. The second tests whether the resulting simulation helps users identify scams. Later ASR work adds a third perspective by isolating the effect of showing users predicted scammer replies and scam likelihood scores in a controlled interface (Tan et al., 2024, Tan et al., 23 Jul 2025).

Evaluation	Setup	Reported result
Technical congruence	90 validation conversations; ScamGPT-J vs GPT-J	MeanSim 0.433 vs 0.329; MaxSim 0.622 vs 0.525
User simulation study	20 volunteers; three logs each; double-blind	Usefulness 4.4/5 vs 1.8/5
Anticipate UI in ASR	78 participants; treatment vs control	Macro accuracy 0.952 vs 0.896
Reason component in ASR	Same pool and design	No significant accuracy gain

On the 90 validation conversations, ScamGPT-J outperformed GPT-J in 80 of 90 conversations on MeanSim and in 73 of 90 on MaxSim. Paired t-tests gave $p=2.3\text{e-}10$ with $t=6.73$ for MeanSim, and $p=3.6\text{e-}07$ with $t=5.29$ for MaxSim. The paper interprets these results as showing, with 99% confidence, that soft prompt tuning materially improves scammer-like generative fidelity relative to the untuned base model (Tan et al., 2024).

The user study used a double-blind randomized survey with 20 volunteers, each contributing three conversation logs, including at least one known scam and one normal conversation. In scam contexts, ScamGPT-J produced context-suited responses in 14 of 16 conversations, whereas GPT-J did so in 3 of 19. In normal contexts, ScamGPT-J produced non-context-suited responses in 10 of 14 conversations, while GPT-J was context-suited in 9 of 11. Average usefulness ratings were 4.4 out of 5 for ScamGPT-J and 1.8 out of 5 for GPT-J. The stated interpretation is that ScamGPT-J is specialized to mimic scammers convincingly in scam conversations and to diverge when the conversation is legitimate, thereby supplying a discriminative cue for self-identification of scams (Tan et al., 2024).

The later ASR study separates the interface effect from the generator by evaluating an idealized “Anticipate” component with manually curated predicted replies and manually assigned scam-likelihood scores. In a between-subject design with 78 participants and eight scenarios, treatment improved macro precision from 0.896 to 0.953, macro recall from 0.896 to 0.952, macro F1 from 0.896 to 0.952, and macro accuracy from 0.896 to 0.952. Linear regressions estimate an overall AI-assisted gain of roughly six percentage points after demographic controls, with the largest improvements in job scams, where $\beta_{\text{AI-Assisted}} \approx 0.232$ to $$s(\mathbf{x}, \mathbf{y}) = \frac{\mathbf{x}\cdot\mathbf{y}}{\|\mathbf{x}\|\,\|\mathbf{y}\|}.$$0. By contrast, the “Reason” component, implemented with ChatGPT-4o Mini explanations, did not significantly improve accuracy by itself, although it was associated with perceived helpfulness in job scams (Tan et al., 23 Jul 2025).

A notable demographic finding is that younger participants, defined as under 25, performed worse on job scams and rated the assistance as substantially less helpful even where objective gains were largest. The paper describes this as a contradiction in which those most at risk are often least receptive to AI support.

5. Relation to adjacent research traditions

ScamGPT-J sits at the intersection of several research lines but is not reducible to any one of them. It differs from LLM-based scam detection systems that treat the task as direct classification or risk scoring, and it differs from automatic scam-baiting systems whose objective is to waste attacker time. Its distinctive move is to use generative simulation as a cognitive aid for the potential victim (Tan et al., 2024).

Automatic scam-baiting studies based on ChatGPT or GPT-3.5-turbo aim to maximize scammer engagement duration, rounds of replies, and attacker effort. One month-long experiment with 819 targeted scammer email addresses reported that an instruction-only ChatGPT-based replier achieved 93 valid conversations, 501 total scammer replies, a mean of 5.38 replies per conversation, and a mean engagement time of approximately 2.8 days, outperforming a template-based control on response rate and conversation length (Bajaj et al., 2023). A separate earlier study using ChatGPT through a human-in-the-loop email workflow found engagement lasting approximately 18 days on average, with some threads reaching approximately 27 days (Cambiaso et al., 2023). ScamGPT-J is related in its use of generative LLMs to mimic credible human or scammer behavior, but its deployment logic is different: the generated text is not sent to the scammer, and the primary target of persuasion is the user.

Later benchmark and measurement work suggests a broader technical context for ScamGPT-J. PsyScam introduces a benchmark for psychological techniques in real-world scams, with nine PT categories and downstream tasks for PT classification, scam completion, and scam augmentation (Ma et al., 21 May 2025). PreScam structures 11,573 conversational scam instances into a three-stage kill chain and shows that supervised encoders outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs (Sun et al., 12 May 2026). A human-centered study of job scams reports that urgency/time-pressure cues are significantly associated with payment behavior, while social proof operates more as a context-dependent legitimacy cue than a standalone predictor (Anagha et al., 27 Jan 2026). These results do not redefine ScamGPT-J, but they suggest adjacent axes along which simulation-based systems could be extended: PT-aware conditioning, progression modeling, and stronger representation of escalation cues.

6. Safety profile, limitations, and open directions

The ScamGPT-J papers frame safety as a design constraint rather than an afterthought. The corpus is predominantly synthetic to address privacy and confidentiality issues associated with real victim communications. Field studies were excluded to avoid potential harm in live scam situations. ScamGPT-J’s scam-like replies and similarity scores are shown only to the user, not to the scammer, reducing direct misuse risk. At the interface level, the authors explicitly warn against automation bias and emphasize that similarity scores are signals rather than recommendations (Tan et al., 2024).

The technical limitations are equally explicit. The original system was developed and evaluated with English-language synthetic dialogues in four scam domains, with no multilingual coverage or cross-regional benchmark. It does not report targeted adversarial evaluations such as obfuscation, paraphrase attacks, or code-switching, although the use of ten paraphrastic variants per seed conversation likely increases tolerance to stylistic variability. It does not provide latency, throughput, compute requirements, or on-device feasibility. Hyperparameters, runtime details, and several implementation particulars are omitted. The evaluation study is small, and the larger ASR interface study uses idealized manually curated predictions and scores rather than a fully deployed end-to-end model (Tan et al., 2024, Tan et al., 23 Jul 2025).

The papers also identify a deeper design tension. False positives are expected in practice because legitimate outreach, sales, or official communication can resemble scams, yet the harm profile is asymmetric: false negatives are often costlier than false positives. At the same time, younger users in the ASR study were both more vulnerable to job scams and less receptive to AI assistance. This suggests that future work is not only a matter of improving generation or calibration. It is also a matter of interface strategy, user trust, and how reflective prompts are presented without encouraging deference to a single score.

Within this trajectory, ScamGPT-J can be understood as an early but technically specific instantiation of a broader idea: generative models can be used not only to detect scam content after the fact, and not only to engage scammers offensively, but also to externalize the anticipated next move of the scammer so that the victim can recognize the manipulation while it is unfolding.