ThreatGPT: Dual Use in Cybersecurity

• ThreatGPT is a term referring to both maliciously configured GPTs that exploit system instructions, knowledge files, and tool integrations, and defensive AI systems for threat modeling.
• Adversarial ThreatGPT implementations use techniques like prompt injection, phishing, and social engineering to expand the attack surface of generative AI systems.
• Defensive applications of ThreatGPT integrate LLMs with live threat intelligence and structured risk assessment, reducing analysis time from hours to seconds.

ThreatGPT is a polysemous term in recent arXiv literature. In one line of work, it denotes maliciously configured custom GPTs that exploit builder-controlled instructions, knowledge files, tool integrations, and external actions to deceive users, inject insecure code, and exfiltrate sensitive data (Antebi et al., 2024). In a broader cybersecurity usage, it functions as a conceptual umbrella for the adversarial misuse of generative AI systems—including ChatGPT, FraudGPT, WormGPT, and related multimodal generators—for phishing, social engineering, malware scaffolding, prompt-based guardrail bypass, and other offensive activity (Falade, 2023, Gupta et al., 2023). A distinct literature uses ThreatGPT as the name of defensive systems: an agentic assistant for standards-aligned threat modeling in public safety settings and a retrieval-augmented threat intelligence assistant grounded in live CVE, CWE, EPSS, and KEV feeds (Zisad et al., 4 Sep 2025, Paul et al., 1 Apr 2025). The term therefore refers not to a single canonical system, but to a cluster of concepts and systems at the intersection of LLMs, cyber offense, cyber defense, and platform security.

1. Terminological divergence and conceptual scope

The literature uses “ThreatGPT” in at least three materially different senses. First, “GPT in Sheep’s Clothing” defines ThreatGPT as a maliciously crafted custom GPT that weaponizes customizable layers—system instructions, uploaded knowledge, capabilities, and actions—to deceive, exfiltrate, and compromise users (Antebi et al., 2024). Second, “From ChatGPT to ThreatGPT” and “Decoding the Threat Landscape” use the term more abstractly, as shorthand for the adversarial misuse of general-purpose generative AI in cyber offense, including jailbreaks, reverse psychology, prompt injection, phishing, malware ideation, and deepfake-enabled social engineering (Gupta et al., 2023, Falade, 2023). Third, later work assigns the name to defensive assistants, including an agentic threat-modeling system for public safety infrastructure and a GPT-4o–backed RAG pipeline for proactive threat intelligence (Zisad et al., 4 Sep 2025, Paul et al., 1 Apr 2025).

This divergence is not merely terminological. It reflects a split between two research agendas. One focuses on how GPT-based applications enlarge the attack surface of software ecosystems, especially when tool use, retrieval, and marketplace distribution are exposed to adversarial builders or users. Another focuses on how the same class of models can be operationalized for structured security analysis, threat prioritization, and mitigation. A plausible implication is that “ThreatGPT” now functions as a contested label for both the threat and the tool.

2. ThreatGPT as malicious custom GPTs and programmable attack surface

The most concrete use of the term appears in the custom-GPT security literature. “GPT in Sheep’s Clothing” characterizes customized GPTs as user-built variants of ChatGPT whose behavior is shaped by builder-specified properties, instructions, uploaded knowledge, capabilities, and actions, and argues that these levers create a new attack surface beyond ordinary LLM interaction (Antebi et al., 2024). System instructions can embed covert malicious agendas; uploaded knowledge files can seed harmful patterns or phishing content; capabilities such as web browsing, image generation, and code interpretation expand operational reach; and actions, defined through OpenAPI schemas, create a direct channel to third-party services and off-platform data exfiltration.

The paper’s threat taxonomy organizes risks into three primary classes: vulnerability steering, malicious injection, and information theft. Vulnerability steering includes N-day exploit attacks and insecure coding recommendations. Malicious injection includes destructive or exfiltrating code snippets and typosquatted libraries. Information theft includes both direct phishing through action calls and third-party phishing through deceptive links (Antebi et al., 2024). The same paper emphasizes that OpenAI’s statement that “Your chats with GPTs are not shared with builders” can be undermined in practice when a GPT is instructed to encapsulate sensitive inputs in an API action and transmit them off-platform.

Related ecosystem studies generalize this attack surface model. “Opening A Pandora’s Box” identifies five critical data exchange channels in custom GPTs—conversation, files, network, operational commands, and authentication—and, using STRIDE, enumerates 26 potential attack vectors, 19 of which were partially or fully validated in real-world settings (Tao et al., 2023). “An Empirical Study on the Security Vulnerabilities of GPTs” formalizes components as prompts/system instructions $P$, memory/chat history $C$, tools $T$, and knowledge $K$, and models attack paths such as $$\langle A0, C \rightarrow P/K/T \rightarrow C, G1 \rangle$$ for component exposure and $$\langle A2, C \rightarrow upload \rightarrow K \rightarrow C \rightarrow T, G2 \rangle$$ for knowledge-poisoning–driven tool misuse (Wu et al., 28 Nov 2025). “Assessing Prompt Injection Risks in 200+ Custom GPTs” narrows the focus to prompt injection, system prompt extraction, and file leakage, while “Privacy and Security Threat for OpenAI GPTs” scales instruction-leakage analysis to 10,000 deployed custom GPTs (Yu et al., 2023, Wenying et al., 4 Jun 2025).

3. Offensive misuse: social engineering, phishing, malware, and prompt-based evasion

A broader ThreatGPT literature treats generative AI as an offensive multiplier. “Decoding the Threat Landscape” frames ThreatGPT as an umbrella for generative AI models exploited to scale and sharpen social engineering, encompassing mainstream systems such as ChatGPT and specialized offerings such as FraudGPT and WormGPT, alongside deepfake generators such as DALL-E 2, Stable Diffusion, VALL-E, Sensory, and Resemble AI (Falade, 2023). The paper attributes to these systems human-like text, image, audio, and code generation; multilingual personalization at scale; memory retention in chats; removal of grammar and style telltales; and prompt-driven manipulation, including indirect prompt injection.

That paper reports a 135% surge in social engineering attacks using generative AI, cites more than 255 million phishing attacks in six months with a 61% year-over-year increase, and notes that pretexting incidents comprise nearly half of social engineering hacks (Falade, 2023). Its case examples include AI-generated CEO voice impersonation leading to approximately \$243,000 transfers, a \$35 million fraud in the UAE via deepfake audio, a €220,000 attempted transfer using deepfake audio, and a 4.3 million yuan transfer via AI-powered face swap. It also describes indirect prompt injection against a Bing chatbot to impersonate a Microsoft employee and solicit credit card details.

“From ChatGPT to ThreatGPT” places these behaviors within a broader offensive taxonomy that includes jailbreaks such as DAN, SWITCH, and role-play narratives; reverse psychology prompts that indirectly surface restricted information; prompt injection that leaks hidden system prompts; social engineering and phishing; SQL injection and WAF-evasion payload ideation; ransomware and malware sketches; and polymorphic malware concepts (Gupta et al., 2023). “Generative AI in Cybersecurity” extends this to malware generation and mutation, obfuscation, packaging and delivery, DDoS orchestration, and deepfakes, citing an approximately 80% click-through rate for AI-crafted phishing emails at Black Hat 2023, a 1,265% increase in malicious phishing emails associated with AI use, FraudGPT subscription pricing on the order of \$200 per month, and LoRA-based safety removal on Llama 2-Chat 70B for approximately \$200 in GPU rental (Metta et al., 2024).

The offensive literature repeatedly emphasizes that mainstream models retain some safety constraints, but those constraints are probed or skirted through reframing, persona assignment, negative framing, or narrative indirection. This suggests that ThreatGPT, in its offensive sense, is less a single artifact than a method for converting linguistic compliance and tool access into adversarial capability.

4. Empirical demonstrations and ecosystem-wide vulnerability measurements

Custom-GPT papers provide transcript-level and large-scale evidence for these risks. “GPT in Sheep’s Clothing” documents stepwise demonstrations of Log4Shell N-day exploitation, SQL injection in PHP, buffer overflow in C, destructive Python output that deletes the Windows folder on the C drive, typosquatted library substitution (“torchs” for “torch”), direct phishing through actions, and third-party phishing through a “disccrd.com” login page (Antebi et al., 2024). The Log4Shell example specifically recommends vulnerable Log4j versions—pre-2.17.0 for Java 8, pre-2.12.3 for Java 7, and pre-2.3.1 for Java 6—and injects LDAP exploit code.

Marketplace-scale studies show that these are not isolated failures. “Assessing Prompt Injection Risks in 200+ Custom GPTs” tested 216 GPTs and found a 97.2% overall success rate for system prompt extraction and 100% file leakage among GPTs with uploaded files; system prompt extraction succeeded in 90 of 96 GPTs without code interpreter and 120 of 120 with code interpreter (Yu et al., 2023). “Privacy and Security Threat for OpenAI GPTs” analyzed 10,000 GPTs and reported that 98.8% were vulnerable to instruction leaking via single-turn adversarial prompts, with 95.1% compromised in ILA-P1 and an additional 3.7% in ILA-P2; among GPTs with explicit defensive statements, 77.5% were still vulnerable to basic P1 attacks, and only 2.5% resisted all tested attacks (Wenying et al., 4 Jun 2025).

Knowledge-file leakage studies identify additional vectors beyond prompts. “When GPT Spills the Tea” analyzes 651,022 GPT metadata entries, 11,820 flows, and 1,466 responses, and identifies five leakage vectors: metadata, GPT initialization, retrieval, sandboxed execution environments, and prompts (Shen et al., 30 May 2025). It reports that 23.79% of GPTs had knowledge files, that 55.3% of knowledge files in tested GPTs leaked via myfiles_browser flows, and that enabling Code Interpreter created a privilege-escalation path by which original knowledge files could be downloaded with a 95.95% GPT-level success rate and a 92.97% file-level success rate. In a copyright analysis of 566 leaked PDFs, 28.80% were categorized as infringing.

“A Large-Scale Empirical Analysis of Custom GPTs’ Vulnerabilities in the OpenAI Ecosystem” studies 14,904 GPTs across seven attack classes and reports that more than 95% lack adequate protection (Ogundoyin et al., 13 May 2025). The most prevalent vulnerabilities are roleplay-based vulnerabilities at 96.51%, system prompt leakage at 92.90% in the detailed results, phishing at 91.22%, social engineering at 80.08%, malware code generation at 69.47%, reverse psychology at 51.38%, and DEN jailbreak at 5.98%. Only 0.47% of tested apps were fully resistant to all seven attacks, while 2.47% were compromised by all seven and 31.36% by six of seven. Category-specific rates are similarly high: Programming GPTs reached 98.70% for roleplay, 95.31% for system prompt leakage, 96.09% for phishing, and 88.28% for malware generation (Ogundoyin et al., 13 May 2025).

5. Defensive responses, detection, and governance

The defensive literature converges on layered controls rather than single-point guardrails. “GPT in Sheep’s Clothing” proposes GPT Self-Check and Configuration Verification, reporting that ChatGPT correctly flagged all attacks except information theft when analyzing malicious transcripts and identified malicious intent in every case when given instructions, knowledge, and actions for safety assessment (Antebi et al., 2024). The same paper recommends stricter review and vetting of custom GPTs, community reputation systems, prompt hardening, permission scoping and transparency for actions, display of raw URLs, and explicit surfacing of exactly what data is sent in each API call.

Platform and architectural controls recur across the ecosystem studies. “Opening A Pandora’s Box” argues for execution transparency, strict separation of user data, developer data, and instructions, granular action permissions, robust sandboxing, logging, and rate limiting (Tao et al., 2023). “An Empirical Study on the Security Vulnerabilities of GPTs” reports that prompt-level hardening reduced expert prompt leakage to 0%, lowered custom component leakage to 14.8% on average, and reduced tool misuse by 83.0% across attack families, though it notes that the defenses are lightweight and prompt-based rather than formal security guarantees (Wu et al., 28 Nov 2025). “Privacy and Security Threat for OpenAI GPTs” recommends layered policies, hierarchical constraints, explicit refusal templates, topic scoping, few-shot adversarial examples, rule-based pre-filters, least-privilege action schemas, data minimization, logging, and red-teaming with the paper’s three-phase instruction-leakage framework (Wenying et al., 4 Jun 2025).

The broader misuse literature adds process and identity controls. “Decoding the Threat Landscape” emphasizes MFA, passkeys, zero trust, verification of senders, phishing simulations, AI-powered anomaly detection, deepfake detectors, and public-private threat intelligence sharing (Falade, 2023). “From ChatGPT to ThreatGPT” emphasizes stronger alignment and RLHF, content filtering and prompt shielding, adversarial training and red teaming, watermarking and provenance, usage policies and monitoring, human-in-the-loop review, and the use of precision, recall, and F1 as future evaluation metrics for defensive deployments (Gupta et al., 2023). “Generative AI in Cybersecurity” adds Responsible Scaling Policies, AI Safety Levels, weight protection, red-team evaluations for dangerous capabilities, and stronger SOC instrumentation for AI-assisted abuse (Metta et al., 2024).

Taken together, these papers imply that defenses must operate simultaneously at the configuration layer, the execution layer, the distribution layer, and the governance layer. The recurring failure mode is that instruction-only defenses are brittle when tools, retrieval, client-visible flows, or off-platform actions remain weakly controlled.

6. ThreatGPT as a defensive system name

A separate set of papers uses ThreatGPT as the name of constructive security tooling. “ThreatGPT: An Agentic AI Framework for Enhancing Public Safety through Threat Modeling” presents an agentic assistant that combines a Google Gemini backend with curated knowledge from STRIDE, MITRE ATT&CK, CVE/NVD, NIST, and CISA, along with 50+ few-shot exemplars, to generate structured threat models from natural-language system descriptions (Zisad et al., 4 Sep 2025). Its architecture comprises a CLI layer, an AI agent layer, a knowledge base/training dataset layer, and a pretrained LLM layer. The reported workflow includes perception, planning, tool use, reasoning and self-verification, and reflection and refinement. In an evaluation on a Drone Delivery Management System, time-to-analysis was 20–30 seconds across prompt types, compared with a cited 40+ hours baseline for human-led threat modeling (Zisad et al., 4 Sep 2025).

“LLM-Assisted Proactive Threat Intelligence for Automated Reasoning” does not title its system ThreatGPT, but the detailed synthesis explicitly characterizes it as a proactive, GPT-4o–backed threat intelligence assistant built from continuous feeds, RAG, and automated reasoning (Paul et al., 1 Apr 2025). The system uses Patrowl for retrieval of CVE, CWE, EPSS, and KEV feeds; all-mpnet-base-v2 for 768-dimensional embeddings; Milvus for similarity search; and LangChain for orchestration. The dataset sizes are reported as 255,158 CVE entries, 963 CWE entries, 251,953 EPSS entries, and 1,126 KEV entries. In case studies, the RAG system answered correctly on recently disclosed CVEs such as CVE-2024-39471 and KEVs such as CVE-2024-38112, whereas the baseline GPT-4o response was “I don’t know” (Paul et al., 1 Apr 2025).

Other papers extend ThreatGPT into additional defensive or analytical roles. “ChatGPT and Other LLMs for Cybersecurity of Smart Grid Applications” describes an LLM-based IDS for IEC 61850 digital substations, with ChatGPT 4.0 reaching 98.18% TPR and 4% FPR on GOOSE traffic under full HITL training, and 96.67% TPR with 0% FPR on SV traffic under full HITL training (Zaboli et al., 2023). “HW-V2W-Map” is framed in the supplied details as a ThreatGPT-like capability for hardware and IoT security, combining ontology-driven vulnerability-to-weakness mapping, CVSS vector prediction, and GPT-based mitigation generation; reported results include accuracy up to 98.29% and recall 90.90% for ML-based severity vector prediction (Lin et al., 2023). “Unified Threat Detection and Mitigation Framework” is similarly presented as a technical foundation for an LLM-driven assistant that detects prompt injection, deceptive outputs, and fairness threats, reporting 92% prompt injection detection accuracy, 65% reduction in deceptive outputs, and 78% improvement in fairness metrics (KumarRavindran, 6 Oct 2025).

These defensive usages are conceptually distinct from the offensive and adversarial ones. They show that the same label has become attached to systems intended to structure threat modeling, prioritize vulnerabilities, detect abuse, or assist operators in critical infrastructure contexts.

7. Open questions and unresolved tensions

Several unresolved issues recur across the literature. One is evaluation scope. Many custom-GPT studies provide attack success rates, but formal risk models, longitudinal ecosystem measurements, and utility–security trade-off analyses remain limited (Antebi et al., 2024, Wenying et al., 4 Jun 2025). Another is provenance and execution-layer control. Knowledge-file leakage and action-based exfiltration demonstrate that client-visible flows, retrieval pipelines, and sandboxed tools can violate privacy and intellectual-property expectations even when prompt-level defenses appear strong (Shen et al., 30 May 2025, Wu et al., 28 Nov 2025).

A second tension concerns base-model versus application-layer responsibility. Large-scale studies argue that custom GPTs inherit or amplify vulnerabilities present in the underlying OpenAI models, including roleplay, reverse psychology, DEN-style jailbreaks, and malware generation (Ogundoyin et al., 13 May 2025). At the same time, several papers show that builder choices—tool enablement, action schemas, defensive prompts, knowledge-file handling, and marketplace presentation—substantially affect exploitability (Antebi et al., 2024, Yu et al., 2023). This suggests that ThreatGPT risk is jointly produced by foundation-model weaknesses and application-layer composition.

A third issue is epistemic ambiguity. In one literature, ThreatGPT names maliciously configured GPTs; in another, it names the misuse of generative AI more generally; in still another, it names defensive assistants for threat modeling or intelligence. This suggests that the term has evolved into a family resemblance concept rather than a stable technical designation. For researchers, that ambiguity makes citation context essential: a paper invoking ThreatGPT may be describing a malicious custom GPT, an AI-enhanced social engineering threat model, a retrieval-augmented threat intelligence system, or an agentic public-safety analysis tool.

The common denominator across these usages is not a specific implementation, but the coupling of LLMs to cyber threat processes—whether as attack surface, attack enabler, or defensive copilot.