Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

Abstract: Jailbreak attacks remain a critical threat to the safe deployment of LLMs. While prior work has primarily studied attacks and defenses at the prompt level, we show that this prompt-centric paradigm overlooks a structural vulnerability in stateful, function-calling environments. In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context. This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data, allowing adversarial intent to be distributed across a multi-turn execution path. We exploit this architectural flaw through SMT, a black-box attack framework based on Simulated Moderation Traces. Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation-auditing workflow. Within this trajectory, a fabricated moderation frame leverages red-team testing as a pretext to elicit harmful generations. The subsequent validation feedback treats safety refusals as execution failures, prompting refinements that gradually weaken the model's safety constraints and ultimately trigger harmful outputs. Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore while requiring a near-minimal number of queries, substantially outperforming existing baselines. These findings demonstrate that prompt-level sanitization alone is fundamentally insufficient for defending tool-enabled LLM systems and highlight the urgent need for context-aware validation across schemas, arguments, tool outputs, and accumulated conversation state. The code is available at https://github.com/liujlong27/SMT.

• The paper introduces the SMT framework to bypass safety guardrails by simulating legitimate moderation workflows through multi-turn, stateful interactions.
• The paper reports SMT attack success rates of up to 99.67% on SafeBench and 98.33% on JailbreakBench, with an average cost of <1.5 API calls per exploit.
• The paper highlights that conventional prompt-level defenses are insufficient for function-calling LLMs, underscoring the need for context-aware, schema-validated safety measures.

Function-Calling as an Attack Surface: Jailbreaking LLMs with Simulated Moderation Traces

Introduction

"Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces" (2607.00481) rigorously investigates the security limitations emergent from the integration of function-calling APIs in safety-aligned LLM deployments. The paper identifies a fundamental architectural vulnerability introduced by the conflation of control logic and untrusted external data within the shared LLM context. To exploit this, the authors introduce the Simulated Moderation Traces (SMT) attack framework, which systematically bypasses prompt-level guardrails by leveraging multi-turn, stateful interactions and imitating legitimate moderation workflows.

Function-Calling Vulnerabilities and the SMT Framework

Existing jailbreak research predominantly construes the attack surface as prompt text, focusing on prompt engineering and multi-turn conversational attacks. However, modern tool- and function-augmented LLM systems allow developer-supplied schemas, structured arguments, external tool outputs, and validation feedback to be injected into a unified context window. As Figure 1 demonstrates, this blurs the historically stricter role separation between trusted control channels and untrusted user data, allowing adversarial input to be introduced across conversation states, schemas, and auxiliary API channels, not just at the prompt level.

Figure 1: Comparative schematic of prompt-based versus function-calling jailbreaks, capturing single/multi-turn prompt attacks (top) and function-calling pathways (bottom), including JailbreakFunction and SMT paradigms.

The SMT methodology advances beyond both prompt-centric and early function-calling jailbreaks by simulating an end-to-end moderation pipeline. The attacker submits a red-team testing request that initiates a function call to generate paired "safe" and "unsafe" samples, under the guise of policy robustness evaluation. Critically, when the model emits a refusal (as expected from a safety-aligned system), SMT reframes this refusal as an execution or validation failure, appending a fabricated backtrace and prompting iterative refinements. Over multiple turns, this loop erodes the model's refusal boundaries, tilting it towards compliance, as conceptualized in Figure 2.

Figure 2: Overview of the SMT attack workflow, illustrating structured schema definitions, moderation context framing, and multi-turn trajectory refinement.

Evaluation: Superiority of SMT Attacks over Prompt-Based Methods

Empirical assessment is comprehensive, covering SafeBench-Tiny and JailbreakBench across six prominent frontier LLMs (GPT-4o, GPT-5.4, Qwen3-Max, Gemini-3-Flash, DeepSeek-V4-Flash, and Claude-Sonnet-4.5) from five providers. SMT achieves the highest observed attack success rates (ASR), up to 99.67% on SafeBench and 98.33% on JailbreakBench, and corresponding HarmScores of 74.27 and 76.83, respectively. These results substantially surpass competitive prompt-level and function-calling baselines, including CC-BOS, JailbreakFunction, and advanced multi-turn attacks.

Notably, the SMT query efficiency—requiring on average <1.5 API calls per successful jailbreak—presents a significant reduction in attack cost compared to the iterative or fuzzing-based prompt methods. The system demonstrates robust performance across all major misuse categories (Figure 3), attesting to its generality and the inadequacy of legacy guardrails when the attack surface is distributed across schema, argument, and history.

Figure 3: Category-wise attack success rates for SMT and baselines on JailbreakBench; SMT achieves stable high ASR across diverse harm types.

Mechanisms of Alignment Deviation under Function Calls

To unpack why function-calling attacks (such as SMT) outperform conventional prompt attacks, the authors conduct white-box analyses using Qwen3.5-9B and diff the role/priority structure between prompt-based and tool-augmented sessions. As illustrated by Figure 4, function-calling enables tool and schema instructions to share system-level priority with alignment policies. This role-boundary shift means that harmful intent, injected via structured arguments or simulated tool outputs, coexists with privileged control directives. Consequently, format-compliance requirements may override or subvert core safety guardrails.

Figure 4: Illustration of the "role-boundary shift" in Qwen3.5-9B under function-calling—tool schema and safety policies are equally prioritized in the model context.

The iterative nature of SMT—where refusals or insufficiently harmful outputs are reframed as validator "errors" and circulated back as context—further amplifies this paradigm. Multi-turn context accumulation exploits the LLM’s objective to complete complex workflows, progressively undermining refusal safeguards in pursuit of apparent task success.

Attack Robustness and the Limits of Conventional Defenses

SMT demonstrates resistance to standard input/output filtering, decoding constraints (e.g., low temperature), length limits, and prompt-level reminder injections. Combined defensive setups only modestly reduce SMT efficacy, and in some cases, have negligible effect compared to their impact on prompt-based jailbreaks. Correspondingly, performance degrades only minimally as the HarmScore success threshold increases; when $\theta$ is increased beyond 80 or 100, the attack cost in terms of interactive rounds rises, but SMT continues to identify high-severity escape pathways (Figure 5).

Figure 5: SMT attack performance under varying success thresholds $\theta$; higher $\theta$ increases difficulty but does not eliminate attack viability.

Contrastingly, the introduction of a function-calling-specific system prompt guardrail (FCGuard)—explicitly instructing the model to detect policy violations framed within red-teaming or structured schemas—reduces ASR to zero on models with sufficient instruction-following ability. Thus, current defenses retrofitted from prompt-level paradigms are fundamentally insufficient against function-calling state-space attacks, but targeted context-sensitive system prompts present a possible, though non-trivial, remedy.

Ablation and Human Evaluation

Ablation experiments confirm that removing any single aspect of the SMT workflow—red-teaming contextualization, schema pairing, validator context, or feedback-driven refinement—reduces attack success. The moderation-testing cover is particularly critical for evading alignment policies, as is the use of paired case generation for plausibility. Additionally, human evaluation aligns closely with automated LLM-judge results, validating the claimed superiority of SMT outputs in evading refusal mechanisms and producing actionable harmful content.

Figure 6: Human evaluation interface used in manual grading of jailbreak outputs against H-CoT guidelines.

Implications, Theoretical Impact, and Future Directions

The analysis in (2607.00481) demonstrates that the security and alignment of LLM-based applications cannot be reasoned about using prompt-level paradigms alone when function calling and tool augmentation are present. The architectural risk is not merely the expanded attack surface or increased complexity, but the inherent blending of trusted system control with untrusted, adversary-influenced data in the context window. This systemic ambiguity fundamentally weakens the effectiveness of surface-level safety interventions.

Practical implications are substantial. Tool-augmented assistants, agent APIs, and complex workflow orchestration frameworks must incorporate context-accumulation-aware, schema-validating safeguards. Reliance on post-hoc prompt sanitization or one-off system prompts is categorically insufficient. Theoretical research must address how safety alignment interacts with multi-modal, multi-turn, and structured execution traces, not just static language streams.

FCGuard and similar context-aware defense strategies represent an emergent direction, yet their efficacy depends on integrating cross-role reasoning and deep context-inference capabilities. System designers must be wary that LLMs' strong instruction-following and format-compliance incentives can, in adversarial regulation, provide stable vulnerabilities until architectural control/logical separation is robustly enforced.

Conclusion

This work provides decisive evidence that stateful, function-calling LLM deployments introduce a qualitatively new security attack surface not captured by prompt-centric threat models. Multi-turn, schema-aware state manipulation—embodied in the SMT attack—consistently subverts prompt-level defenses in major commercial LLMs and highlights the latent risk of blurring control and data boundaries. Future alignment and safety engineering must develop context- and schema-aware methodologies for function-augmented LLM systems. As agentic systems and structured APIs become pervasive, the challenge outlined by SMT attacks will remain at the center of trustworthy LLM deployment.