Multi-Turn Stress Testing Framework
- Multi-Turn Stress Testing Framework is an evaluation architecture that simulates sustained, adaptive dialogues to reveal evolving vulnerabilities in LLM-based agents.
- It employs a modular design with distinct agents for question generation, evaluation, and context management to dynamically adapt based on feedback.
- The framework has demonstrated practical value by uncovering edge-case failures and achieving 10–12X higher throughput compared to manual testing.
A multi-turn stress testing framework is an evaluation architecture for LLM-based agents in which test generation, execution, and assessment unfold across adaptive, context-sensitive dialogues rather than fixed single-turn prompts. In this formulation, the target of evaluation is not merely a model’s response to an isolated input, but its behavior under sustained interaction, changing tone, shifting intent, and evaluation-conditioned follow-up. Neo, introduced as a configurable multi-agent framework for scalable and realistic testing of LLM-based agents, is a representative implementation: it automates realistic multi-turn evaluation through a Question Agent, an Evaluation Agent, and a shared Context Hub interacting with a Target Agent, and is explicitly designed around autonomy, context-awareness, and feedback-driven adaptation (Wang et al., 19 Jul 2025).
1. Problem setting and defining requirements
The framework arises from a mismatch between deployed LLM agents and the methods commonly used to evaluate them. Manual testing is described as too slow, expensive, subjective, and difficult to reproduce, while static benchmark datasets are described as too rigid to capture evolving, context-sensitive behavior and as becoming stale as systems, prompts, and guardrails change. For conversational and agentic systems, many important failures emerge only across multiple turns, including topic drift, incoherent follow-ups, brittle safety behavior, emotional escalation, context misuse, failure propagation, and adversarial combinations that exploit previous turns (Wang et al., 19 Jul 2025).
Within this problem framing, a realistic multi-turn stress testing framework must satisfy several simultaneous requirements. It must generate dynamic multi-turn interactions rather than fixed test cases, support scalable automated execution, maintain context-awareness across turns, permit goal-conditioned control over what kinds of behaviors are probed, and include feedback-driven adaptation so later turns react to earlier outcomes. It must also balance human realism with adversarial coverage, and remain modular enough to evolve as testing objectives change (Wang et al., 19 Jul 2025).
A recurrent misconception is that multi-turn stress testing is simply a longer benchmark. The framework’s defining property is not turn count alone, but the fact that later prompts are conditioned on prior dialogue history and prior evaluation outcomes. This suggests that the relevant test object is an interaction trajectory rather than a set of independent prompts.
2. Architectural structure and feedback loops
In Neo’s current implementation, the internal architecture has three foundational components—Question Agent, Evaluation Agent, and Context Hub—and interacts with an external Target Agent, the LLM-based system under test (Wang et al., 19 Jul 2025).
| Component | Role |
|---|---|
| Question Agent | Generates the next user-like input |
| Evaluation Agent | Judges whether the target output satisfies predefined criteria |
| Context Hub | Stores static and dynamic context and coordinates the test process |
| Target Agent | External system under test |
The Question Agent pulls both static and dynamic information from the Context Hub. Static information includes domain metadata, prompt templates, test intents, and configuration. Dynamic information includes conversation history, current emotional tone, prior evaluation outcomes, and adaptive instructions reflecting what has happened so far in the test session. Using this combined context, it generates the next user-like input and sends it to the Target Agent. The Target Agent returns a response, after which the full interaction state is sent to the Evaluation Agent. In the current version, the evaluation result is only a binary success/failure signal, though the design explicitly anticipates future multidimensional judgments such as factual accuracy, emotional appropriateness, completeness, and policy compliance (Wang et al., 19 Jul 2025).
The Context Hub is the framework’s centralized memory and orchestration layer. It is not itself an autonomous agent. Its function is to externalize both static context and dynamic context, including domain-specific configurations, modular prompt templates, behavioral expectations, accumulated history, and a structured evolving interaction state. This externalization is what makes the framework modular and configurable rather than hardcoded into a monolithic test generator (Wang et al., 19 Jul 2025).
The system contains two feedback loops. The inner loop operates turn by turn during simulation: generation depends on prior context and prior evaluation, and evaluation affects future generation policy. The outer loop connects human stakeholders and deployment traces back into future test policy: developers consume structured reports and can improve the target system, and post-launch user interactions may eventually be processed into traces that refine strategy (Wang et al., 19 Jul 2025).
3. Goal-driven and state-driven generation
Neo implements what it calls a goal-driven and state-driven testing paradigm. A test session begins from a high-level testing goal. The paper specifies four goal types: Security Testing — “Stress system boundaries and safety mechanisms”; Robustness Testing — “Evaluate system performance under ambiguous or malformed inputs”; Coverage Testing — “Ensure full topic/intent coverage across known scenarios”; and Realism Testing — “Mimic genuine user behavior across multi-turn conversations” (Wang et al., 19 Jul 2025).
These goals alter the probabilities used by the state controller. Security Testing increases the likelihood of adversarial inputs, while Realism Testing emphasizes human-like flows, tone variation, and conversational coherence. The framework therefore balances realism and adversariality by changing probabilistic controls over the same generative machinery rather than by swapping to a different architecture (Wang et al., 19 Jul 2025).
The generation process is centered on an explicit state vector: $\mathbf{S = \langle F, I, T, FB \rangle}$ where $F$ is Flow Type, $I$ is Intent Type, $T$ is Tone Index, and $FB$ is Feedback. The paper defines these dimensions as follows: $F$ takes Start, Follow-up, Switch, or Repeat transitions; $I$ takes Baseline, Edge Case, Adversarial, or Malicious; $T$ is emotional polarity, e.g., $-10$ (angry) to $+10$ (pleased); and $F$0 is the prior evaluation result, Success or Fail (Wang et al., 19 Jul 2025).
Conversation is initialized with a Start flow state and a goal-conditioned initial sampling over intent and tone. After the Target Agent answers, the Evaluation Agent emits $F$1, and this feedback influences the next transition. The paper gives adaptive logic in prose: a failure may increase the likelihood of repeating or rephrasing a question, while a success may result in a coherent follow-up or a shift in intent focus. Elsewhere it states that the feedback signal can increase the likelihood of a follow-up after success, or trigger retries or topic shifts after failure (Wang et al., 19 Jul 2025).
4. Formalization of controllable interaction space
The framework also formalizes the size of the controllable test space. Let $F$2 denote the configurable state dimensions, and let $F$3 be the number of discrete values each can take. The paper gives the total number of distinct interaction states, with notation described as malformed in the PDF source; the intended expression is: $F$4 with $F$5 the number of configurable state dimensions (Wang et al., 19 Jul 2025).
A full multi-turn session is modeled as a question tree. Each node is a generated test input; edges encode structural relations such as follow-up or switch. Every session begins with a unique Start node. A Follow-up extends from an existing node as a child, preserving continuity. A Switch starts a new topic and creates a new root disconnected from the previous thread. For a session of $F$6 rounds, the number of structurally distinct trees is given as: $F$7 because at round $F$8, the new question may either follow up on one of the $F$9 existing nodes or initiate a topic switch, yielding $I$0 attachment choices at round $I$1 (Wang et al., 19 Jul 2025).
Semantic diversity is then defined through intent and tone labeling. With $I$2 possible intents and $I$3 tone levels, assuming independent sampling per round, the number of labelings across $I$4 rounds is: $I$5 Combining structure and labeling yields the total number of distinct test sessions: $I$6 The paper illustrates this with $I$7, $I$8, and $I$9: $T$0 possible session variants (Wang et al., 19 Jul 2025).
At the same time, the paper does not provide an explicit transition matrix, exact sampling distribution, pseudocode for state transitions, a formal Markov kernel, or an objective function for generation. The procedural model is clearly described, but the transition dynamics remain deliberately configurable rather than fully specified (Wang et al., 19 Jul 2025).
5. Empirical behavior, performance, and scope
Neo was applied to a production-grade Seller Financial Assistant chatbot. In that evaluation it uncovered edge-case failures across five attack categories with a 3.3% break rate, close to the 5.8% achieved by expert human red-teamers. It also delivered 10–12X higher throughput, generating 180 coherent test questions in around 45 mins versus 16h of human effort (Wang et al., 19 Jul 2025).
Beyond security probing, the framework’s stochastic policies balanced topic coverage and conversational depth, yielding broader behavioral exploration than manually crafted scripts. This suggests that the framework’s value is not limited to adversarial prompting narrowly construed; it also functions as a coverage mechanism for conversational behaviors that are difficult to enumerate manually (Wang et al., 19 Jul 2025).
The architecture is also described as model-agnostic and extensible. Its agent interfaces, state controller, and feedback loops are positioned as a foundation for richer factual-grounding and policy-compliance checks. The current implementation is therefore best understood as an initial multi-turn stress-testing substrate rather than a closed evaluation protocol (Wang et al., 19 Jul 2025).
A second misconception is that automation eliminates human quality assurance. In this framework, automation scales generation and evaluation, but human stakeholders remain part of the outer feedback loop through report consumption, system iteration, and possible incorporation of post-launch traces.
6. Position within the broader research landscape
The broader literature reinforces the claim that multi-turn stress testing is not reducible to single-turn evaluation with more tokens. ToolHaystack distinguishes long-term multi-turn tool use from ordinary multi-turn evaluation by emphasizing entangled task histories, interruptions, and context dilution in continuous conversations (Kwak et al., 29 May 2025). THRD argues that safety in dialogue is trajectory-dependent and reports that over 70% of multi-turn attacks are first detected only at Turn 2 or later, making turn-wise independent analysis insufficient for jailbreak defense (Ma et al., 1 Jun 2026). MORTAR frames multi-turn dialogue testing around the oracle problem and uses dialogue-level perturbations plus metamorphic relations to evaluate context-sensitive behavior without relying on LLM judges (Guo et al., 2024).
Other frameworks extend the same general idea into specialized domains. STING treats agent misuse as sequential illicit $T$1-step goal execution, tracking time-to-first-jailbreak under adaptive follow-ups and judge-mediated phase completion (Talokar et al., 18 Feb 2026). PBSuite shows that models with less than 4% failure rates in single-turn policy adherence can exhibit up to 84% failure rates in multi-turn adversarial interactions under custom behavioral policies (Varshney et al., 7 Nov 2025). In mental-health dialogue, adaptive probing reduced the average number of turns to first boundary violation from 9.21 in static progression to 4.64, highlighting the importance of time-to-breach rather than breach/no-breach alone (Cheng et al., 2 Jan 2026). In coding-agent evaluation, StaminaBench operationalizes long-horizon robustness as the number of consecutive interaction turns survived before failure, showing that standard task-solved metrics miss a distinct capability the authors call stamina (Sobal et al., 17 Jun 2026).
Taken together, these works suggest that a multi-turn stress testing framework is less a single benchmark than a family of evaluation designs sharing several properties: persistent interaction state, adaptive or structured follow-up generation, trajectory-level failure criteria, and explicit mechanisms for relating later turns to earlier model behavior. Neo’s specific contribution within that family is a modular multi-agent architecture with a shared context hub, goal-conditioned state control, and evaluation-conditioned generation policy (Wang et al., 19 Jul 2025).
