Success-at-Turn in Multi-Turn Systems

• Success-at-Turn is a family of turn-indexed success metrics that quantify individual turn contributions across tutoring, jailbreak evaluation, planning, backdoor analysis, and reinforcement learning.
• The literature demonstrates that cumulative and per-turn metrics, such as ASR(k) and binary credit indicators, reveal differences between aggregate session outcomes and nuanced turn-level performance.
• Methodologies employing Success-at-Turn enhance model evaluation and optimization by integrating turn-aware credit assignment, minimal-turn planning, and targeted adversarial defenses.

Searching arXiv for papers on “success-at-turn” and related multi-turn evaluation/training. Searching arXiv for: success-at-turn multi-turn jailbreak ASR turn-by-turn verifier success probability minimal turns. Success-at-Turn denotes a family of turn-indexed success notions for multi-turn systems rather than a single canonical metric. In recent work, it appears as a binary indicator that a tutor utterance contributed to eventual task completion, as a cumulative success-by-turn curve for jailbreak evaluation, as a success probability under the minimal number of turns in task planning, as a per-turn attack success rate for structural backdoors, and as a turn-level credit signal for reinforcement learning. At the same time, some influential multi-turn attack papers do not define any explicit turn-wise metric and instead report only aggregate end-of-session Attack Success Rate (ASR). The resulting literature is therefore heterogeneous in both semantics and measurement (Zhao et al., 24 Jan 2025, Wang et al., 18 Feb 2025, Yang et al., 11 Aug 2025, Hu et al., 24 Sep 2025, Lu et al., 20 Jan 2026, He et al., 9 May 2026, Feng et al., 6 Feb 2026).

1. Terminological scope and formal variants

The available literature suggests that “Success-at-Turn” is best understood as a problem class: success is indexed by turn, but the object being indexed differs across domains. In tutoring, the turn is assessed for its contribution to eventual student success; in jailbreak evaluation, the turn often indexes cumulative attack success by or before turn $k$; in long-horizon planning, the turn indexes completion probability under minimal horizon constraints; in backdoor analysis, the turn itself can be the trigger; and in RL optimization, turn-level success is often operationalized through credit assignment or per-task success-rate estimates.

Setting	Formalization	Key quantity
Siren (Zhao et al., 24 Jan 2025)	No distinct Success-at-Turn metric	Final-turn ASR only
Traver (Wang et al., 18 Feb 2025)	Turn contribution indicator	$o_{s_t}$, $v_t$, TOC
Multi-turn jailbreak evaluation (Yang et al., 11 Aug 2025)	Cumulative success by turn $k$	$S(k)$
Minimal-turn planning (Hu et al., 24 Sep 2025)	Success under shortest completion horizon	$P_t^\pi(s_t)$
Structural backdoor (Lu et al., 20 Jan 2026)	Per-turn attack success rate	$\mathrm{ASR}(t)$
RL credit assignment (He et al., 9 May 2026)	Turn-level contribution multipliers	$c_{i,t}$, $m_{i,t}$

This variation is substantive rather than notational. Siren explicitly does not introduce or label a distinct “Success-at-Turn” metric anywhere in its equations, and all reported ASRs are aggregate over a multi-turn session (Zhao et al., 24 Jan 2025). By contrast, Traver introduces a binary turn indicator $o_{s_t}\in\{0,1\}$ and a cumulative turn-based reward $o_{s_t}$0 (Wang et al., 18 Feb 2025), while automated jailbreak evaluation defines success-by-turn as the fraction of harmful queries that have produced a “perfect jailbreak” by or before turn $o_{s_t}$1 (Yang et al., 11 Aug 2025).

2. End-of-session ASR versus cumulative success-by-turn in jailbreak evaluation

A central distinction in multi-turn jailbreak work is whether success is evaluated only at the end of an interaction or accumulated across turns. Siren adopts the former. A multi-turn attempt consists of up to $o_{s_t}$2 turns, with an optional 4th if the third turn fails to elicit a harmful response. At turn $o_{s_t}$3 the attacker issues query $o_{s_t}$4, the target returns response $o_{s_t}$5, and $o_{s_t}$6 is evaluated as “harmful” or “harmless.” However, the interaction is treated as successful only if the final response, $o_{s_t}$7 or $o_{s_t}$8, is judged harmful. The paper reports two flavors of ASR, KW-ASR and GPT-ASR, and defines overall ASR as

$o_{s_t}$9

It does not provide per-turn success curves, ASR after turn 1, turn 2, or turn 3, or any LaTeX definition of $v_t$0 (Zhao et al., 24 Jan 2025).

Despite that measurement choice, Siren reports high aggregate outcomes under this final-turn criterion: 90% ASR with LLaMA-3-8B as the attacker against Gemini-1.5-Pro, and 70% with Mistral-7B against GPT-4o. The paper also states that single-turn methods rarely exceed approximately 30–50% ASR on local models and often fall to zero on API models, and that Siren’s multi-turn strategies raise ASR into the 60–90% range. This suggests that multi-turn interactions materially change observed vulnerability, even when the evaluation remains trajectory-final rather than turn-wise (Zhao et al., 24 Jan 2025).

Other jailbreak papers make the turn dimension explicit. “Multi-Turn Jailbreaks Are Simpler Than They Seem” defines, for harmful queries $v_t$1 with $v_t$2, a running maximum score

$v_t$3

where $v_t$4 is the StrongREJECT score, and then defines Success-at-Turn $v_t$5 as

$v_t$6

Equivalently, with threshold $v_t$7,

$v_t$8

Here $v_t$9 is the single-turn success rate and $k$0 is the overall multi-turn success rate after $k$1 interactions, with $k$2 in the main experiments (Yang et al., 11 Aug 2025).

That paper also provides representative success-at-turn curves. For GPT-4o-mini, $k$3 rises from 0.32 at $k$4 to 0.71 at $k$5; for Claude 3.5 Sonnet, from 0.18 to 0.38; and for Gemini 2.5 Flash Lite, from 0.64 to 0.94. The paper argues that parallel single-turn baselines re-sampled 8 times closely match each multi-turn curve, and therefore that the principal driver of improved attack success across turns is the opportunity to retry rather than deep conversational steering. It further reports that higher reasoning effort often leads to higher attack success rates, and that the growth of $k$6 is well captured by an exponential saturation form $k$7 (Yang et al., 11 Aug 2025).

SEMA uses a related but not identical notion. Its main metric is $k$8, defined over a dataset $k$9 as

$S(k)$0

where $S(k)$1 is the success label from the judge for the $S(k)$2-th rollout. Here the turn-like index is an attempt index over multiple sampled rollouts rather than the within-dialogue position. On HarmBench to Llama-3.1-8B-Instruct, SEMA rises from $S(k)$3 to $S(k)$4 at $S(k)$5, $S(k)$6 at $S(k)$7, $S(k)$8 at $S(k)$9, and $P_t^\pi(s_t)$0 at $P_t^\pi(s_t)$1 (Feng et al., 6 Feb 2026).

3. Turn-level contribution in tutoring systems

In tutoring, Success-at-Turn is defined less as cumulative dialogue success and more as causal contribution by an individual tutor utterance. Traver introduces at turn $P_t^\pi(s_t)$2 a binary indicator $P_t^\pi(s_t)$3 for whether the tutor’s $P_t^\pi(s_t)$4-th utterance contributed to the student’s eventual correct solution, a guiding distance $P_t^\pi(s_t)$5, and a cumulative turn-based reward $P_t^\pi(s_t)$6. These are linked by

$P_t^\pi(s_t)$7

and

$P_t^\pi(s_t)$8

When $P_t^\pi(s_t)$9, the weighted reward is positive; when $\mathrm{ASR}(t)$0, it is negative. The dependence on $\mathrm{ASR}(t)$1 means that later turns can receive larger absolute weight (Wang et al., 18 Feb 2025).

This turn-wise formalism is coupled to an explicit verifier. At each turn $\mathrm{ASR}(t)$2, the verifier $\mathrm{ASR}(t)$3 takes as input the triplet $\mathrm{ASR}(t)$4 and predicts a scalar $\mathrm{ASR}(t)$5, intended to match the true cumulative reward. The implementation uses Mistral-7B as the base encoder, with a single linear head on top, and is fine-tuned via LoRA on the Q and V projection matrices with rank $\mathrm{ASR}(t)$6 and $\mathrm{ASR}(t)$7. Training minimizes mean-squared error between predicted and synthesized ground-truth $\mathrm{ASR}(t)$8 values, and inference samples $\mathrm{ASR}(t)$9 candidate responses, scores each with the verifier, and selects the one with highest predicted $c_{i,t}$0 (Wang et al., 18 Feb 2025).

Traver further embeds Success-at-Turn into evaluation through the DICT protocol. Tutoring Outcome (TO) is defined as relative improvement in Recall and Pass:

$c_{i,t}$1

To quantify turn-wise progress, the paper introduces the Tutoring Outcome Curve (TOC): at each turn $c_{i,t}$2, it runs a hypothetical post-test using only the dialogue up to turn $c_{i,t}$3 and plots Recall@k or Pass@k versus $c_{i,t}$4. The slope of this curve measures how rapidly the tutor is driving coding success at each turn. Figure 1 shows that Pass rate rises more steeply with Traver’s verifier than with vanilla instruct prompting, and Figure 2 reports that increasing the candidate pool $c_{i,t}$5 from 1 to 20 raises the turn-by-turn Pass rate from 35.1% to 39.3% (Wang et al., 18 Feb 2025).

A plausible implication is that tutoring work uses Success-at-Turn in a more interventionist sense than jailbreak work. The turn is not merely a checkpoint at which overall success is observed; it is an action whose local pedagogical effect is explicitly modeled and evaluated.

4. Minimal-turn completion and step-level success optimization

In multi-turn task planning, Success-at-Turn is formalized as success probability under the shortest feasible horizon. “Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning” defines, for a multi-turn MDP $c_{i,t}$6, the minimal turns from state $c_{i,t}$7 as

$c_{i,t}$8

For a policy $c_{i,t}$9, its success probability in the minimal number of turns is

$m_{i,t}$0

The paper proves that if a reference policy $m_{i,t}$1 is improved in the single-turn setting via GRPO to obtain $m_{i,t}$2, then $m_{i,t}$3 has no worse success probability in the multi-turn MDP under the minimal-turn constraint. Experimentally, on the Robotouille benchmark, a Qwen2.5-1.5B model with SFT+GRPO reaches success rates of 0.30 on Cheese Sandwich, 0.70 on Burger, 0.70 on Cheese Burger, and 0.30 on Double Cheese Burger, and the paper states that a turn-by-turn success-at-turn curve further confirms upward movement at every prefix length (Hu et al., 24 Sep 2025).

STEP addresses turn-wise optimization from a different angle. It defines a multi-turn trajectory

$m_{i,t}$4

where the final trajectory reward $m_{i,t}$5 denotes success or failure. For each task $m_{i,t}$6, STEP maintains a smoothed success-rate estimate $m_{i,t}$7 updated by

$m_{i,t}$8

where $m_{i,t}$9 is the number of sampled trajectories and $o_{s_t}\in\{0,1\}$0 the number that succeed. It then allocates sampling adaptively using

$o_{s_t}\in\{0,1\}$1

and computes a success-rate-weighted trajectory advantage

$o_{s_t}\in\{0,1\}$2

which is assigned to every step in the trajectory (Chen et al., 17 Nov 2025).

This is not a turn-indexed metric in the same sense as $o_{s_t}\in\{0,1\}$3 or $o_{s_t}\in\{0,1\}$4, but it operationalizes success information at the step level for policy optimization. On OSWorld, STEP achieves 62.5% success versus 48.4% for trajectory-level GRPO and 55.4% for GiGRPO; on AndroidWorld, it reaches 47.6% versus 33.3% and 39.2%, respectively, while speeding up training 1.74$o_{s_t}\in\{0,1\}$5 over T-GRPO (Chen et al., 17 Nov 2025). This suggests that Success-at-Turn can function not only as an evaluation quantity but also as a sampling and credit-allocation primitive.

5. Turn-aware credit assignment and turn-indexed attack surfaces

Multi-turn jailbreak RL exposes a specific credit assignment problem: trajectory-level outcomes are coarse, but turn-level contributions are non-uniform. TRACE addresses this by estimating turn-level contribution in successful trajectories through leave-one-turn-out masking. For a successful trajectory

$o_{s_t}\in\{0,1\}$6

it removes turn $o_{s_t}\in\{0,1\}$7, resamples the final response, and defines raw turn credit as

$o_{s_t}\in\{0,1\}$8

where $o_{s_t}\in\{0,1\}$9 is the judge’s harmfulness score. These credits are normalized to $o_{s_t}$00, clipped, and transformed into turn-level multipliers

$o_{s_t}$01

For failed trajectories, TRACE defines harmfulness and semantic relevance penalties, adds an optional refusal-aware local process penalty, and forms a turn-aware advantage

$o_{s_t}$02

The paper reports that, in raw trajectories, successful cases contain 47.1% attack-critical turns and 52.9% redundant turns, while failed trajectories contain 5.9% safety-critical turns and 94.1% neutral turns. On HarmBench, JailbreakBench, and WildJailBreak, TRACE(mix) reaches an average ASR@1 of 87.10 versus 69.97 for TROJail, described as about a 25% relative improvement (He et al., 9 May 2026).

A distinct turn-wise notion appears in structural backdoor analysis. TST defines the current turn index $o_{s_t}$03 as part of the trigger. Given a trigger set of turns $o_{s_t}$04, it defines

$o_{s_t}$05

The backdoored model is trained so that on trigger turns it emits a fixed payload, independent of user input, and evaluation uses a per-turn attack success rate

$o_{s_t}$06

With trigger turns chosen as every even turn $o_{s_t}$07, TST reports $o_{s_t}$08, $o_{s_t}$09, $o_{s_t}$10, overall average ASR $o_{s_t}$11, False-Trigger Rate on non-trigger turns $o_{s_t}$12, and Clean accuracy on non-trigger turns $o_{s_t}$13. Under five representative defenses, the average ASR remains approximately 98.04%, and per-turn ASR stays nearly flat within $o_{s_t}$14 percentage points across $o_{s_t}$15 (Lu et al., 20 Jan 2026).

Taken together, these results indicate two different meanings of turn awareness in adversarial work. TRACE treats turns as units of causal contribution inside a trajectory; TST treats the turn index itself as the adversarial signal.

6. Conceptual antecedents and recurring methodological issues

A formal antecedent to modern Success-at-Turn analysis appears in the adversarial Last-Success-Problem. There, two players observe independent Bernoulli variables $o_{s_t}$16 sequentially, and on observing $o_{s_t}$17 may either continue or pass the turn. If $o_{s_t}$18, the player must continue. Let $o_{s_t}$19 denote the probability that the player whose turn it is before observing $o_{s_t}$20 will eventually win. With boundary conditions $o_{s_t}$21 and $o_{s_t}$22, the recursion is

$o_{s_t}$23

The optimal rule is to pass if and only if $o_{s_t}$24, and the paper derives a threshold index

$o_{s_t}$25

This is a turn-based success process in which the relevant object is the win probability attached to the active turn, not a dialogue-quality score or a final-response classifier (Ribas, 2018).

Several recurring methodological issues emerge across the contemporary literature. First, Success-at-Turn is not standardized. Siren reports only aggregate success on the final turn and explicitly lacks per-turn success curves (Zhao et al., 24 Jan 2025), while Traver, TRACE, TST, SEMA, and the StrongREJECT-based jailbreak analysis all instantiate different turn-wise quantities (Wang et al., 18 Feb 2025, Yang et al., 11 Aug 2025, Lu et al., 20 Jan 2026, He et al., 9 May 2026, Feng et al., 6 Feb 2026). Second, coarse trajectory-level outcomes can obscure the causal role of individual turns. TRACE explicitly frames this as a credit assignment problem, arguing that uniform broadcast of outcome reward over-rewards redundant turns in successful trajectories and under-credits useful intermediate turns in failed ones (He et al., 9 May 2026). Third, greater multi-turn success does not necessarily imply sophisticated conversational steering: one empirical analysis concludes that automated multi-turn jailbreaks are approximately equivalent to simply re-sampling single-turn attacks multiple times, once refusal feedback is accounted for (Yang et al., 11 Aug 2025).

A common misconception is therefore to treat “Success-at-Turn” as synonymous with a single curve $o_{s_t}$26. The literature does not support that simplification. In some settings the turn index denotes a prefix of a conversation, in others an independent attempt, in others a minimal-horizon planning constraint, and in others a structural trigger or a turn-level causal weight. The unifying theme is narrower: success is measured, attributed, or optimized with explicit reference to turn position.