--- title: Success-at-Turn in Multi-Turn Systems url: https://www.emergentmind.com/topics/success-at-turn type: topic --- # Success-at-Turn in Multi-Turn Systems Searching arXiv for recent papers on “success-at-turn” and related multi-turn evaluation/training. Searching arXiv for: success-at-turn multi-turn jailbreak ASR turn-by-turn verifier success probability minimal turns. Success-at-Turn denotes a family of turn-indexed success notions for multi-turn systems rather than a single canonical metric. In recent work, it appears as a binary indicator that a tutor utterance contributed to eventual task completion, as a cumulative success-by-turn curve for jailbreak evaluation, as a success probability under the minimal number of turns in task planning, as a per-turn attack success rate for structural backdoors, and as a turn-level credit signal for reinforcement learning. At the same time, some influential multi-turn attack papers do not define any explicit turn-wise metric and instead report only aggregate end-of-session Attack Success Rate (ASR). The resulting literature is therefore heterogeneous in both semantics and measurement [2501.14250] [2502.13311] [2508.07646] [2509.20616] [2601.14340] [2605.08778] [2602.06854]. ## 1. Terminological scope and formal variants The available literature suggests that “Success-at-Turn” is best understood as a problem class: success is indexed by turn, but the object being indexed differs across domains. In tutoring, the turn is assessed for its contribution to eventual student success; in jailbreak evaluation, the turn often indexes cumulative attack success by or before turn $k$; in long-horizon planning, the turn indexes completion probability under minimal horizon constraints; in backdoor analysis, the turn itself can be the trigger; and in RL optimization, turn-level success is often operationalized through credit assignment or per-task success-rate estimates. | Setting | Formalization | Key quantity | |---|---|---| | Siren [2501.14250] | No distinct Success-at-Turn metric | Final-turn ASR only | | Traver [2502.13311] | Turn contribution indicator | $o_{s_t}$, $v_t$, TOC | | Multi-turn jailbreak evaluation [2508.07646] | Cumulative success by turn $k$ | $S(k)$ | | Minimal-turn planning [2509.20616] | Success under shortest completion horizon | $P_t^\pi(s_t)$ | | Structural backdoor [2601.14340] | Per-turn attack success rate | $\mathrm{ASR}(t)$ | | RL credit assignment [2605.08778] | Turn-level contribution multipliers | $c_{i,t}$, $m_{i,t}$ | This variation is substantive rather than notational. Siren explicitly does **not** introduce or label a distinct “Success-at-Turn” metric anywhere in its equations, and all reported ASRs are aggregate over a multi-turn session [2501.14250]. By contrast, Traver introduces a binary turn indicator $o_{s_t}\in\{0,1\}$ and a cumulative turn-based reward $v_t$ [2502.13311], while automated jailbreak evaluation defines success-by-turn as the fraction of harmful queries that have produced a “perfect jailbreak” by or before turn $k$ [2508.07646]. ## 2. End-of-session ASR versus cumulative success-by-turn in jailbreak evaluation A central distinction in multi-turn jailbreak work is whether success is evaluated only at the end of an interaction or accumulated across turns. Siren adopts the former. A multi-turn attempt consists of up to $T=3$ turns, with an optional 4th if the third turn fails to elicit a harmful response. At turn $i$ the attacker issues query $x^{(i)}$, the target returns response $r^{(i)}$, and $r^{(i)}$ is evaluated as “harmful” or “harmless.” However, the interaction is treated as successful only if the final response, $r^{(3)}$ or $r^{(4)}$, is judged harmful. The paper reports two flavors of ASR, KW-ASR and GPT-ASR, and defines overall ASR as $$ \mathrm{ASR}=\frac{\text{Number of successful attacks}}{\text{Total number of attempts}}\times 100\%. $$ It does not provide per-turn success curves, ASR after turn 1, turn 2, or turn 3, or any LaTeX definition of $\mathrm{ASR}(t)$ [2501.14250]. Despite that measurement choice, Siren reports high aggregate outcomes under this final-turn criterion: 90% ASR with LLaMA-3-8B as the attacker against Gemini-1.5-Pro, and 70% with Mistral-7B against GPT-4o. The paper also states that single-turn methods rarely exceed approximately 30–50% ASR on local models and often fall to zero on API models, and that Siren’s multi-turn strategies raise ASR into the 60–90% range. This suggests that multi-turn interactions materially change observed vulnerability, even when the evaluation remains trajectory-final rather than turn-wise [2501.14250]. Other jailbreak papers make the turn dimension explicit. “Multi-Turn Jailbreaks Are Simpler Than They Seem” defines, for harmful queries $\mathcal{Q}$ with $|\mathcal{Q}|=30$, a running maximum score $$ M_q(k)=\max_{1\le t\le k}\mathcal{S}(r_{q,t}), $$ where $\mathcal{S}(r_{q,t})\in[0,1]$ is the StrongREJECT score, and then defines Success-at-Turn $k$ as $$ S(k)=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathbf{1}[M_q(k)=1]. $$ Equivalently, with threshold $\tau=1$, $$ S(k)=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathbf{1}\!\left[\max_{t\le k}\mathcal{S}(r_{q,t})\ge \tau\right]. $$ Here $S(1)$ is the single-turn success rate and $S(K)$ is the overall multi-turn success rate after $K$ interactions, with $K=8$ in the main experiments [2508.07646]. That paper also provides representative success-at-turn curves. For GPT-4o-mini, $S(k)$ rises from 0.32 at $k=1$ to 0.71 at $k=8$; for Claude 3.5 Sonnet, from 0.18 to 0.38; and for Gemini 2.5 Flash Lite, from 0.64 to 0.94. The paper argues that parallel single-turn baselines re-sampled 8 times closely match each multi-turn curve, and therefore that the principal driver of improved attack success across turns is the opportunity to retry rather than deep conversational steering. It further reports that higher reasoning effort often leads to higher attack success rates, and that the growth of $S(k)$ is well captured by an exponential saturation form $S(k)\approx A-B\cdot \exp(-ck)$ [2508.07646]. SEMA uses a related but not identical notion. Its main metric is $\mathrm{ASR}@k$, defined over a dataset $\mathcal{D}$ as $$ \mathrm{ASR}@k=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\left[\bigvee_{n=1}^{k}s_{i,n}\right], $$ where $s_{i,n}\in\{0,1\}$ is the success label from the judge for the $n$-th rollout. Here the turn-like index is an attempt index over multiple sampled rollouts rather than the within-dialogue position. On HarmBench to Llama-3.1-8B-Instruct, SEMA rises from $\mathrm{ASR}@1=70.6\%$ to $94.8\%$ at $k=5$, $97.9\%$ at $k=10$, $99.1\%$ at $k=20$, and $99.7\%$ at $k=50$ [2602.06854]. ## 3. Turn-level contribution in tutoring systems In tutoring, Success-at-Turn is defined less as cumulative dialogue success and more as causal contribution by an individual tutor utterance. Traver introduces at turn $t$ a binary indicator $o_{s_t}\in\{0,1\}$ for whether the tutor’s $t$-th utterance contributed to the student’s eventual correct solution, a guiding distance $d_t=T-t$, and a cumulative turn-based reward $v_t$. These are linked by $$ v_t=\max(v_{t-1}+w_{r_t},0),\qquad v_0=0, $$ and $$ w_{r_t}=\frac{1-v_{t-1}}{d_t+1}\times (2\cdot o_{s_t}-1). $$ When $o_{s_t}=1$, the weighted reward is positive; when $o_{s_t}=0$, it is negative. The dependence on $d_t$ means that later turns can receive larger absolute weight [2502.13311]. This turn-wise formalism is coupled to an explicit verifier. At each turn $t$, the verifier $V_\theta$ takes as input the triplet $[\text{task }\mathcal{T}; \text{dialogue context } \mathbf{C}_t; \text{candidate tutor response } r_t]$ and predicts a scalar $v_t\in[0,1]$, intended to match the true cumulative reward. The implementation uses Mistral-7B as the base encoder, with a single linear head on top, and is fine-tuned via LoRA on the Q and V projection matrices with rank $r=8$ and $\alpha=16$. Training minimizes mean-squared error between predicted and synthesized ground-truth $v_t^{(i)}$ values, and inference samples $N$ candidate responses, scores each with the verifier, and selects the one with highest predicted $v_t$ [2502.13311]. Traver further embeds Success-at-Turn into evaluation through the DICT protocol. Tutoring Outcome (TO) is defined as relative improvement in Recall and Pass: $$ \Delta\%M=\frac{M_{\text{post}}-M_{\text{pre}}}{M_{\text{pre}}}\times 100. $$ To quantify turn-wise progress, the paper introduces the Tutoring Outcome Curve (TOC): at each turn $t$, it runs a hypothetical post-test using only the dialogue up to turn $t$ and plots Recall@k or Pass@k versus $t$. The slope of this curve measures how rapidly the tutor is driving coding success at each turn. Figure 3 shows that Pass rate rises more steeply with Traver’s verifier than with vanilla instruct prompting, and Figure 5 reports that increasing the candidate pool $N$ from 1 to 20 raises the turn-by-turn Pass rate from 35.1% to 39.3% [2502.13311]. A plausible implication is that tutoring work uses Success-at-Turn in a more interventionist sense than jailbreak work. The turn is not merely a checkpoint at which overall success is observed; it is an action whose local pedagogical effect is explicitly modeled and evaluated. ## 4. Minimal-turn completion and step-level success optimization In multi-turn task planning, Success-at-Turn is formalized as success probability under the shortest feasible horizon. “Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning” defines, for a multi-turn MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},f,R,\mathcal{T},s_0)$, the minimal turns from state $s$ as $$ T^*(s)=\min\Bigl\{T\ge 0:\exists \tau(T)\text{ s.t. }R(s_T,a_T)=1,\ s_0=s\Bigr\}. $$ For a policy $\pi$, its success probability in the minimal number of turns is $$ P_t^\pi(s_t)=\Pr_\pi\!\bigl(R(s_{t+T^*(s_t)},a_{t+T^*(s_t)})=1\mid s_t\bigr). $$ The paper proves that if a reference policy $\pi^{\rm ref}$ is improved in the single-turn setting via GRPO to obtain $\pi^*$, then $\pi^*$ has no worse success probability in the multi-turn MDP under the minimal-turn constraint. Experimentally, on the Robotouille benchmark, a Qwen2.5-1.5B model with SFT+GRPO reaches success rates of 0.30 on Cheese Sandwich, 0.70 on Burger, 0.70 on Cheese Burger, and 0.30 on Double Cheese Burger, and the paper states that a turn-by-turn success-at-turn curve further confirms upward movement at every prefix length [2509.20616]. STEP addresses turn-wise optimization from a different angle. It defines a multi-turn trajectory $$ T_{i,j}=\{S_{i,j,1},R_{i,j,1},S_{i,j,2},R_{i,j,2},\ldots,S_{i,j,T},R_{i,j,T}\}, $$ where the final trajectory reward $R_T(T_{i,j})\equiv R_{i,j,T}\in\{0,1\}$ denotes success or failure. For each task $Q_i$, STEP maintains a smoothed success-rate estimate $s_i\in[0,1]$ updated by $$ s_i^{(t)}=\frac{U_i^{(t)}+\alpha s_i^{(t-1)}}{N_i^{(t)}+\alpha}, $$ where $N_i^{(t)}$ is the number of sampled trajectories and $U_i^{(t)}$ the number that succeed. It then allocates sampling adaptively using $$ P_{\mathrm{rep}}(s_i)=\frac{1}{1+\exp(-K(s_i-s_0))}, $$ and computes a success-rate-weighted trajectory advantage $$ \mathrm{Adv}_{\mathrm{traj}}(T_{i,j})=(1-s_i)\,R_T(T_{i,j}), $$ which is assigned to every step in the trajectory [2511.13091]. This is not a turn-indexed metric in the same sense as $S(k)$ or $\mathrm{ASR}(t)$, but it operationalizes success information at the step level for policy optimization. On OSWorld, STEP achieves 62.5% success versus 48.4% for trajectory-level GRPO and 55.4% for GiGRPO; on AndroidWorld, it reaches 47.6% versus 33.3% and 39.2%, respectively, while speeding up training 1.74$\times$ over T-GRPO [2511.13091]. This suggests that Success-at-Turn can function not only as an evaluation quantity but also as a sampling and credit-allocation primitive. ## 5. Turn-aware credit assignment and turn-indexed attack surfaces Multi-turn jailbreak RL exposes a specific credit assignment problem: trajectory-level outcomes are coarse, but turn-level contributions are non-uniform. TRACE addresses this by estimating turn-level contribution in successful trajectories through leave-one-turn-out masking. For a successful trajectory $$ T_i=(x_1,y_1,\ldots,x_{T_i},y_{T_i}), $$ it removes turn $t