Turn-level Adjudicated RL (TARL)

• TARL is a reinforcement learning paradigm that assigns discrete, turn-level rewards to isolate the contribution of each action in long-horizon tasks.
• It combines adjudication mechanisms, such as LLM judges and external evaluators, with advanced techniques like advantage normalization and filtering to stabilize learning.
• Applications of TARL include dialogue systems, multimodal tool-use agents, clinical decision-making, and adversarial scenarios, enhancing policy updates and planning.

Turn-level Adjudicated Reinforcement Learning (TARL) formalizes the reinforcement learning paradigm by evaluating and assigning rewards to agents at each discrete decision point—or "turn"—throughout extended multi-turn interactions. This approach arises from the necessity to address the credit assignment problem in complex long-horizon tasks, where outcomes depend on a sequence of temporally extended decisions, and where global or trajectory-level rewards are insufficient for isolating the contribution of individual actions. TARL combines algorithmic structures for per-turn credit assignment, dedicated adjudication mechanisms (often implemented via LLM judges or external evaluators), and model architectures capable of leveraging fine-grained reward signals. Its application ranges from dialogue systems and multi-modal tool-use agents to adversarial attacks and collaborative multi-agent systems.

1. Foundations and Motivation

TARL emerges as a response to deficiencies in standard reinforcement learning for language agents, particularly in multi-turn settings where delayed rewards obfuscate which decisions led to success or failure. Conventional trajectory-level RL methods (e.g., RLHF, PPO, Behavioral Cloning with MC Returns, ILQL (Abdulhai et al., 2023, Shani et al., 23 May 2024)) tend to propagate outcome-based rewards back over all actions, resulting in myopic learning and policy collapse known as the "Echo Trap" (Wang et al., 24 Apr 2025). TARL instead stipulates turn-level adjudication—where each turn is evaluated by an explicit judge or verification process and assigned a local reward—addressing:

• Credit Assignment: Decoupling the impact of individual decisions within a trajectory.
• Planning Over Extended Horizons: Allowing agents to attribute future outcomes to specific early turns.
• Stability and Exploration: Enabling finer gradients and more stable policy updates via detailed feedback.

This methodology is particularly crucial in domains such as interactive dialogue (Feng et al., 26 May 2025), tool-integrated reasoning (Xue et al., 2 Sep 2025), multimodal agents (Tan et al., 17 Sep 2025), and adversarial attacks (Meng et al., 18 Nov 2024), where multi-turn, stateful environments predominate.

2. Algorithms and Reward Structuring

TARL implementations typically embed turn-level reward signals within a Markov Decision Process (MDP) formalism, extending the classic state-action-reward framework to accommodate adjudicated turns. Multiple studies instantiate this with the following core elements:

• Turn-level Reward Functions: For each turn $t$, adjudicators (either LLM judges or rule-based evaluators) assign $r_t$ based on criteria such as correctness, compliance, reasoning quality, or strategic value. For example, the process-supervised multimodal agent framework employs an LLM judge that outputs $r_t \in \{-1,0,1\}$ per turn, with trajectory-level bonuses applied multiplicatively for final task completion (Tan et al., 17 Sep 2025).
• Advantage Estimation and Group-Wise Normalization: Frameworks like Multi-Turn GRPO (Zeng et al., 17 May 2025) compute normalized advantage terms for both turn-level outcomes ($A^T$) and final results ($A^O$), then assign them to specific turns:

$$\begin{align*} A_{i,1}^{\text{MT-GRPO}} &= A^T_i + \lambda A^O_i \ A_{i,2}^{\text{MT-GRPO}} &= A^O_i \end{align*}$$

Here, $\lambda$ modulates the coupling between early and late decision rewards, facilitating credit assignment with group-relative baselining.

• Filtering and Stabilization Mechanisms: Filtering out unproductive or "void" turns—interactions yielding neither tool outputs nor answers—stabilizes training by blocking harmful gradient updates from low-probability token trajectories (Xue et al., 2 Sep 2025). Such techniques directly address gradient norm explosion and prevent misassignment of credit.
• Mirror Descent and Nash Equilibrium Convergence: In multi-turn RLHF settings, policy updates are performed via mirror descent on regularized preference games, with theoretical guarantees for last-iterate convergence to unique Nash equilibria (Shani et al., 23 May 2024):

$$\pi_{t+1}(y|x_h) \propto \pi_{\text{ref}}(y|x_h)^{\alpha \eta_t} \cdot \pi_t(y|x_h)^{1-\alpha \eta_t} \cdot \exp(\eta_t Q_{\text{reg}}^{(\pi_t, \pi_t)}(x_h, y))$$

and

$$KL(\pi^* || \pi_t) \leq \frac{32 H Q^2}{\alpha^2 (t+1)}$$

3. Evaluation Protocols and Benchmarks

Benchmark suites and sandbox environments facilitate rigorous evaluation of TARL agents:

• LMRL-Gym: Implements tasks requiring multi-turn reasoning and planning, including partial observability, delayed credit, and trajectory stitching (Abdulhai et al., 2023). RL agents are assessed via rollouts in synthetic or scripted environments, receiving per-turn and outcome rewards.
• τ-bench: Used for multimodal tool-use evaluation; agents undergo mixed-task training (including math reasoning) for enhanced self-correction and exploration (Tan et al., 17 Sep 2025).
• MTMedDialog: Provides simulated clinical consultation data for multi-agent dialogue and diagnostic tasks (Feng et al., 26 May 2025).

Policy and outcome metrics include pass@k (code/patch validation), tool-execution success, diagnostic F1 scores, and reward variance/statistics reflecting the stability and diversity of agent outputs.

4. Applications in Interactive Agents and Complex Environments

TARL has been integrated in a range of settings:

• Software Engineering Agents: In long-context development tasks, RL agents must process stateful environments over many tool calls and adjudicated turns. Modified DAPO algorithms with per-token loss averaging and dynamic sampling address credit assignment and reward sparsity, resulting in near-doubling of pass rates on SWE-bench Verified (Golubev et al., 5 Aug 2025).
• Multimodal Tool-use: Interactive agents are trained using interleaved speech-text rollouts, with process-level judges controlling per-turn rewards. Mixed-task curricula further foster robust planning (Tan et al., 17 Sep 2025).
• Clinical Dialogue: Multi-agent collaboration models consultative interaction as a sequential decision process. The doctor agent's policy is trained with multidimensional, turn-level rewards: diagnostic accuracy, information-gathering efficiency, and protocol compliance (Feng et al., 26 May 2025).
• Adversarial Attacks: TARL frameworks in attack scenarios use reinforcement learning to adapt key search parameters (e.g., the angle $\alpha$ in black-box attacks), dramatically improving query efficiency over fixed-update baselines (Meng et al., 18 Nov 2024).

5. Credit Assignment, Stability, and Optimization

TARL's central benefit is solving sparse and delayed credit assignment. By adjudicating outcomes at each turn, learning becomes more sample-efficient and policies less prone to collapse or overfitting:

• Temporal and Agent-level Reward Redistribution: Methods such as TAR$^2$ (Kapoor et al., 19 Dec 2024) decompose episodic rewards across time and agents, ensuring $$\sum_{t}\sum_{i} r_{i,t} = r_{\text{global, episodic}}(\tau)$$, which is provably equivalent to potential-based reward shaping and does not alter the optimal policy.
• Stabilization via Filtering: Removing trajectories with void turns or low in-group reward variance corrects the credit assignment and blocks catastrophic gradients, leading to stabilized multi-turn training and the emergence of sophisticated reasoning patterns (Xue et al., 2 Sep 2025, Wang et al., 24 Apr 2025).
• Mixed and Reasoning-aware Tasks: Integrating math or reasoning problems forces deeper exploratory behavior and avoids overconfidence in narrow domains (Tan et al., 17 Sep 2025).

6. Theoretical Guarantees and Limitations

TARL often employs regularization (e.g., KL penalties, reference policy anchoring) and mirror-descent updates to assure convergence and policy plausibility. For example, multi-turn preference optimization converges to the unique Nash equilibrium in regularized preference games. Reward redistribution methods preserve optimality via equivalence to potential-based shaping. Nevertheless, limitations remain:

• Scalability to very long horizons depends on reward signal design and credit assignment granularity.
• Efficacy presupposes access to accurate, turn-level adjudication—whether from human judges, LLMs, or structured external evaluators.
• In complex or noisy environments, tuning of weight coefficients (e.g., in advantage mixing) or filtering thresholds becomes critical.

7. Implications, Future Directions, and Broader Significance

TARL represents a modular advancement in reinforcement learning for agentic systems. By bridging sparse feedback, temporal delays, and multi-agent contexts, it equips agents to execute robust planning, self-correction, and strategic adaptation across diverse domains—including dialogue, software engineering, healthcare, and adversarial security. Ongoing and future research is focused on:

• Extending TARL to real-time, human-in-the-loop systems for more naturalistic interaction.
• Generalizing agent-temporal redistribution techniques to settings with multiple adjudicators or competitive agents.
• Designing richer reward functions that combine trajectory-level, turn-level, and reasoning-aware assessment.

TARL is thus a pivotal framework for advancing the capabilities of goal-directed, multi-turn autonomous agents, anchoring technical progress in the rigorous, reproducible, and fine-grained evaluation standards now prevalent in language-based reinforcement learning research.