TrojanPromptGuard: Securing Educational LLM Use

• TrojanPromptGuard is a prototype detection layer that identifies multi-turn Trojanized prompt chains in K–12 educational LLM applications.
• It integrates rule-, model-, and heuristic-based methods, leveraging conversation history, role cues, and semantic escalation to flag unsafe content.
• Evaluations using simulated child confusion and literary Trojanization reveal tradeoffs between underblocking and overblocking in educational settings.

TrojanPromptGuard (TPG) is a prototype detection and mitigation layer proposed for K–12 educational use of LLMs. It is designed to identify and manage “Trojanized prompt chains”: multi-turn prompts that appear benign, pedagogical, moral, or literary on the surface while gradually steering a model toward unsafe or unintended outputs. In "Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design" (Charles et al., 15 Jul 2025), TPG is positioned as an intermediate layer between student input and the LLM engine, with explicit emphasis on conversation history, role and persona cues, semantic escalation, and educator oversight. The acronym is not uniform across the literature: in "Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-LLMs," "TPG" denotes Targeted Prompt Guidance, not TrojanPromptGuard (Wang et al., 3 Aug 2025).

1. Educational setting and problem formulation

The system is motivated by K–12 classroom use of LLMs for tutoring, homework assistance, and lesson planning. The underlying deployment assumption is that teachers often supervise many students simultaneously and can only superficially supervise individual screens, so students may interact with classroom LLM tools in ways that are not visible in real time. Within that setting, TrojanPromptGuard addresses prompts that are crafted to look academically legitimate while embedding a hidden unsafe intent (Charles et al., 15 Jul 2025).

The paper uses “Trojanized prompts” for prompts or prompt sequences that are deliberately framed as academic, moral, or literary while carrying a concealed payload such as dangerous chemical information, extremist themes, or other restricted content. “Trojanized prompt chains” are specifically multi-turn dialogues in which the early turns remain plausible schoolwork and later turns introduce ambiguity that pressures the model into unsafe output. This is narrower than generic prompt injection or generic jailbreaks in two stated respects: the attacks are embedded in educational motifs, and they rely on multi-turn, persona-driven escalation rather than a single explicit jailbreak string (Charles et al., 15 Jul 2025).

The system is therefore not a general-purpose moderation API in the paper’s formulation. It is a context-aware middleware layer tailored to educationally framed, multi-turn misuse. A plausible implication is that the design objective is less about static keyword filtering than about recognizing when apparently ordinary classroom discourse becomes control content for unsafe generation.

2. Attack patterns, threat model, and empirical basis

The threat model centers on students using classroom or homework LLM tools. The assumed capabilities are that students can initiate multi-turn conversations, adopt personas such as “I’m in 6th grade” or “my teacher said…,” exploit moral, academic, or literary framing, and search online for jailbreak recipes. The main constraint is plausibility: prompts must still look like schoolwork, because overt requests are easier for moderation systems to catch (Charles et al., 15 Jul 2025).

Two attack families structure the paper’s analysis. Simulated Child Confusion (SCC) uses moral or safety framing, such as a student who claims to want to learn what is dangerous in order to avoid it. Prompt Chain Escalation via Literary Devices (PCELD) frames illicit material as fiction, satire, metaphor, or literary analysis. Both families are implemented as 3–4 turn chains with increasing semantic ambiguity. The dominant failure modes are correspondingly described as a Moral Framing Exploit and Literary Trojanization (Charles et al., 15 Jul 2025).

The empirical basis for TPG is a 500-trial experiment on GPT-3.5 and GPT-4 using SCC and PCELD chains across low-, medium-, and high-risk prompts. The reported vulnerability profile is concentrated in higher-risk settings. For GPT-3.5, high-risk bypass rates were 43.75% for PCELD and 45.45% for SCC; for GPT-4, high-risk bypass rates were 62.86% for PCELD and 34.88% for SCC. Low-risk prompts showed 0.00% bypass and 0.00% moderation across all listed conditions. The paper identifies turn 3 as the inflection point where the interaction typically moves from benign to ambiguous or unsafe. A chi-square test of independence over bypass outcomes gives a chi-square statistic of 0.450, degrees of freedom 3, and a p-value of 0.930, so the authors do not find a statistically significant difference between model types and chain categories overall, while still emphasizing practical differences such as GPT-4’s higher bypass rate in high-risk PCELD (Charles et al., 15 Jul 2025).

The study also records an opposing error mode: overly restrictive moderation. One noted example is GPT-3.5 refusing a benign request about slavery images and citing content policy. This matters because TrojanPromptGuard is motivated by a dual failure pattern: existing systems can underblock Trojanized educational chains and overblock legitimate educational queries (Charles et al., 15 Jul 2025).

3. Detection architecture and scoring pipeline

TrojanPromptGuard is described as a hybrid architecture that combines rule-based, model-based, and heuristic components. Its design target is session-level analysis rather than isolated prompt filtering, and its pipeline explicitly uses the current prompt, conversation history, and role or persona cues (Charles et al., 15 Jul 2025).

Module	Function	Primary signals
Input Preprocessor	Normalize and tokenize input; detect persona cues	“I’m in 6th grade”, “my teacher said”
Role Detection Engine	BERT-based classifier on annotated educational prompts	student, teacher, admin, adversary
Semantic Escalation Tracker	Monitor multi-turn topic drift with embeddings	drift toward chemical misuse, violence, extremism
Trojan Phrase Pattern Matcher	Regex and neural NER on red-teamed prompts	“what not to do”, “just hypothetically”
Guardrail Violation Scorer	Aggregate features into scalar risk score	$s_t \in [0,1]$, thresholding
User Interface and Feedback Loop	Teacher dashboard and feedback collection	risk ratings, escalation paths, reframing suggestions

The Role Detection Engine is specified as a BERT-based classifier fine-tuned on annotated educational prompts. Its purpose is to classify the user persona as student, teacher, admin, or adversary, and to flag role inconsistency, such as a purported sixth-grader asking for graduate-level biochemistry about weaponization. The Semantic Escalation Tracker uses embeddings and cosine similarity to detect topic drift across turns. The paper gives the similarity form

$$\cos(\theta) = \frac{\mathbf{e}_{t} \cdot \mathbf{e}_{t-1}}{\|\mathbf{e}_{t}\| \|\mathbf{e}_{t-1}\|}$$

and states that the tracker also compares current turns to known high-risk topic vectors (Charles et al., 15 Jul 2025).

The Trojan Phrase Pattern Matcher combines handcrafted regular expressions with a neural named-entity recognizer trained on red-teamed prompts. The examples explicitly listed are “what not to do,” “just hypothetically,” and “to help others avoid danger.” These patterns are meant to capture the SCC and PCELD structures uncovered in the experiments (Charles et al., 15 Jul 2025).

Feature aggregation is performed by the Guardrail Violation Scorer, which produces a scalar risk score

$$s_t = f(\text{role\_score}_t, \text{escalation\_score}_t, \text{trojan\_phrase\_score}_t, \ldots)$$

with $s_t \in [0,1]$. If $s_t > \tau$, the system flags or blocks the prompt; the paper gives $\tau = 0.75$ as an example threshold. It also states that the exact function $f(\cdot)$ and weight values are not specified, describing the scorer only as “a weighted ensemble of feature detectors” (Charles et al., 15 Jul 2025).

A central technical point is that TPG is session-level and multi-turn aware. It does not implement a detailed graph formalism or state machine in the paper, but it is explicitly described as monitoring semantic escalation across turns and as leveraging conversation context in role detection and phrase matching. This suggests a moderation model keyed to trajectory structure rather than isolated utterances.

4. Intervention logic, deployment modes, and human oversight

TrojanPromptGuard is placed as an intermediate layer between the student and the LLM engine. In the workflow described in the paper, the student enters a prompt through an educational platform, TPG intercepts the prompt and associated history, computes a risk score, and then decides whether to pass the prompt unchanged, flag it, rewrite or reframe it, block it, or surface it to an educator-facing dashboard (Charles et al., 15 Jul 2025).

The mitigation logic is intentionally high-level. The paper lists four main intervention modes. Flag and block prevents the prompt from reaching the LLM when the risk score exceeds the threshold. Flag and reframe proposes safer reinterpretations, for example redirecting a request from dangerous chemical combinations toward general safety. Educator dashboards allow teachers or administrators to inspect flagged prompts, see why they were flagged, and approve or reject borderline cases. Monitoring-only mode logs and tags interactions without blocking them, particularly for pilot deployments or auditing (Charles et al., 15 Jul 2025).

Two deployment models are named. In Inline Filtering (Middleware), TPG sits in front of the LLM API in a learning management system or tutoring application, so every student prompt is screened before inference. In Monitoring-Only Mode, TPG observes prompts and responses without enforcing a block decision. The paper also mentions deployment as a middleware service integrated with school servers or LMS back ends, and in some configurations as a local browser extension (Charles et al., 15 Jul 2025).

The human-oversight layer is not incidental. Teachers and administrators are expected to view individual and aggregate risk scores, inspect escalation paths, and provide feedback on false positives and false negatives. Logged feedback is intended for future retraining. Exact interface mockups are not provided, but the paper states that the dashboard shows escalation paths and automated suggestions for reframing unsafe input (Charles et al., 15 Jul 2025).

5. Evaluation status, limitations, and failure modes

The paper demonstrates vulnerabilities in GPT-3.5 and GPT-4 and uses those results to motivate TrojanPromptGuard, but it does not report a full empirical evaluation of TPG itself. Instead, it outlines a prototype evaluation plan. The proposed metrics are True Positive Rate (TPR), False Positive Rate (FPR), Intervention Accuracy, and Educator Usability Feedback. No numerical precision, recall, F1, confusion matrices, or ROC curves are reported for TPG, and no formally described labeled dataset of Trojan versus benign educational prompts is provided (Charles et al., 15 Jul 2025).

The listed limitations are consequential. Coverage is centered on two Trojanization families, SCC and PCELD, so other attack styles may evade detection. The Role Detection Engine, Semantic Escalation Tracker, and Trojan Phrase Pattern Matcher depend on annotated educational prompts and red-teamed adversarial corpora, creating sensitivity to incompleteness or bias in those resources. The paper also identifies risks of overblocking, model dependence in threshold calibration, and adversarial adaptation via euphemisms, indirect references, or multilingual prompts (Charles et al., 15 Jul 2025).

A further limitation is operational rather than algorithmic: TPG is motivated as a context-aware educational guard, but the paper does not present a real-world deployment case study. Consequently, its status is best understood as a prototype architecture grounded in an experimental misuse study rather than as a benchmarked production defense. A plausible implication is that its most mature contribution is the problem decomposition—role consistency, session-level escalation, phrase-pattern analysis, and teacher oversight—rather than validated classifier performance.

6. Relation to broader backdoor, prompt-security, and context-integrity research

TrojanPromptGuard sits within a wider literature on prompt security, trojan/backdoor detection, and context integrity, but its primary contribution is domain-specific: educationally framed, multi-turn misuse. "Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message" argues that many current systems are vulnerable because they scrutinize user turns more strongly than model-attributed history, a failure mode the paper terms Asymmetric Safety Alignment (Duan et al., 7 Jul 2025). This supports TPG’s emphasis on session-level analysis rather than isolated prompt filtering.

Research on agentic workspaces extends the same concern from classroom chat to persistent local state. "From Prompt Injection to Persistent Control: Defending Agentic Harness Against Trojan Backdoors" introduces ClawTrojan and DASGuard, where the critical problem is whether untrusted content becomes future control content in files, memories, or policies. DASGuard’s content-source graph, source labeling, and sanitized commits show that provenance tracking can be central in multi-step trojan defense; the reported collapse in performance when source labels are removed underscores that point (Tan et al., 29 May 2026). A plausible implication is that TPG’s session-level logic would need provenance mechanisms to scale from chat moderation to agentic educational systems.

Broader backdoor research provides a second axis of context. The TrojAI final report frames trojans and backdoors as hidden behaviors that remain dormant on clean inputs and activate on triggers, and surveys detection through weight analysis, trigger inversion, and mitigation strategies such as sample rejection, input purification, and model correction (Reese et al., 6 Feb 2026). "Trojan Detection Through Pattern Recognition for LLMs" operationalizes a three-stage LLM detector—token filtration, trigger identification, and trigger verification—and stresses that verification under perturbation is critical for distinguishing true triggers from merely high-likelihood adversarial strings (Bhasin et al., 20 Jan 2025). TDC2023 analysis reaches a complementary conclusion: recovering intended triggers is much harder than eliciting harmful behavior, so behavior-focused defense is more realistic than exact trigger recovery (Maloyan et al., 2024).

Prompt-level backdoor work further clarifies the threat model that TPG addresses at the interface layer. "TrojFSP: Trojan Insertion in Few-shot Prompt Tuning" shows that few-shot prompt tuning can achieve attack success rates over 99% while maintaining negligible decreases in clean data accuracy through target-class reweighting, selective token poisoning, and a Trojan-Trigger Attention objective (Zheng et al., 2023). "TrojLLM: A Black-box Trojan Prompt Attack on LLMs" shows that black-box attackers can construct universal and stealthy prompt-level triggers that retain high clean performance while producing high attack success across models including GPT-3.5 and GPT-4 (Xue et al., 2023). These results suggest that a prompt-side defense such as TrojanPromptGuard cannot rely on visible prompt oddity alone.

Finally, terminological ambiguity is itself notable. In "Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-LLMs," TPG denotes Targeted Prompt Guidance, a textual steering mechanism within the Simulated Ensemble Attack rather than a guard or detector (Wang et al., 3 Aug 2025). Confusing the two obscures a basic distinction: TrojanPromptGuard is presented as a defensive, educationally tuned middleware layer, whereas Targeted Prompt Guidance is an offensive prompt-construction strategy for transferable jailbreaks.