TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoning

Abstract: LLMs increasingly rely on either reinforcement learning or multi-agent prompting to improve reasoning, yet these two paradigms remain difficult to combine. Directly applying single-agent reinforcement learning to multi-turn multi-agent systems faces following dilemmas: i) Sparse rewards, role-level free-riding and excessive training overhead. ii) Agents only imitate to collaborate. iii) Fixed collaboration protocol falls into oscillating local optimum. We introduce TRACER, a turn-level reinforcement framework for cooperative multi-LLM reasoning. TRACER separates collaborative decision making into a controller-regret layer, where controllers learn whether the agents should speak or skip the current round through regret matching, and a generation-credit layer, which optimizes proposer and reviewer utterances with role-specific GSPO rewards. This design i) assigns credit at the level of both action modes and generated utterances, thus avoiding free-riding and sparse rewards. We only expand the choices made by the controllers, thus greatly reducing computational cost of training. Moreover, ii) agents acquire collaborative capability as they learn when to utter and what to speak. Finally, iii) by designing binary actions ingeniously, we extend classical game theory established for finite action spaces to deep learning, thus achieving mathematically rigorous convergence. We train all local RL-style methods on the GSM8K training split and evaluate on held-out GSM8K, MATH500, and GPQA-Diamond to measure in-domain accuracy, cross-benchmark generalization, inference cost, and correction-preservation behavior. The resulting framework provides a compact and reproducible testbed for studying learned collaboration policies beyond fixed debate, voting, or aggregation protocols. Code is available at https://github.com/Shark-Forest/TRACER.

• The paper introduces a dual-layer regret matching framework that separates control and generation to address credit assignment challenges in cooperative multi-LLM reasoning.
• It employs GSPO for role-specific credit assignment, ensuring agents optimize utterance generation and decision making efficiently.
• Empirical evaluations on GSM8K, MATH500, and GPQA benchmarks show improved accuracy, reduced tokens, and stable convergence compared to existing methods.

TRACER: A Rigorous Two-Layer Regret-Matching Architecture for Cooperative Multi-LLM Reasoning

Motivation and Problem Formulation

Cooperative multi-agent LLM systems must address credit assignment, action selection, and training efficiency in collaborative reasoning tasks. Previous paradigms—pure RL (e.g., PPO, GSPO, GRPO) and multi-agent hand-crafted collaborations (debate, aggregation, voting)—each suffer inherent shortcomings. Applying single-agent RL to multi-LLM collaboration leads to sparse and confounded reward signals, role free-riding, and high sample complexity. Conversely, most MAS frameworks resort to fixed communication protocols and do not support learned, adaptive collaboration, resulting in missed opportunities for policy optimization and theoretical guarantees.

TRACER (Turn-level Regret Matching with Inner Reinforcement Credit) introduces a dual-layered framework that separates action-mode selection (controller-regret layer) from role-specific utterance optimization (generation-credit layer), solving free-riding, credit ambiguity, and inefficiency while providing game-theoretic convergence properties.

TRACER Architecture and Mechanisms

TRACER consists of two key components:

• Controller-Regret Layer: This consists of independent controllers (for the proposer and reviewer) specifying at each round whether the associated agent should "speak" or "skip." At each discrete phase—characterized by the pending answer’s score (votes by the reviewer) and the round index—the controller policy is updated through regret matching. The regret signal is derived from the difference between the value of a counterfactual action and the value under the current policy. This structure enables fine-grained turn-level credit assignment and circumvents sparse reward confounding typical in multi-agent RL settings.
• Generation-Credit Layer: Each agent’s utterance is optimized via GSPO (Group Sequence Policy Optimization), with separate updates for proposers (rewarded for correct numerical answers) and reviewers (rewarded based on the correctness of accept/reject judgments). Rewards are assigned based on direct correctness, enabling explicit, role-aligned credit.

  Figure 1: The TRACER architecture decouples speaking/skipping control (outer regret matching) from utterance optimization (inner GSPO) across multi-turn multi-agent reasoning loops.

This separation of decision layers (when to speak, what to say) diminishes over-sampling in multi-round training—since only controller-induced utterances enter the RL trajectory—and precisely attributes reward, suppressing free-riding and reducing sample complexity.

Theoretical Properties and Algorithmic Insights

By defining a binary action space (speak/skip) and constructing the round-wise interaction as a finite, perfect-information game, TRACER maps collaborative LLM reasoning to a framework compatible with counterfactual regret minimization (CFR). The controllers are updated via standard regret matching, with cumulative counterfactual regret computed in each phase bucket, yielding the following guarantees:

• Provable Convergence: The average regret for each controller vanishes as the number of training episodes increases, and the joint controller policy converges to a correlated equilibrium in the collaborative setting (see detailed proofs in the appendix).
• Rigorous Credit Assignment: The GSPO updates for each generation module guarantee role-aligned reward assignment, fully decoupled from the controller state.

TRACER’s protocol completes training without extra neural controller networks, relying solely on regret-matching updates, and all update steps are conducted in closed form.

Empirical Evaluation

Benchmark Results and Efficiency

TRACER is evaluated against both non-RL and RL-based multi-agent baselines on GSM8K (arithmetic), MATH500 (math transfer), and GPQA-Diamond (science multiple-choice) benchmarks, employing backbones such as Qwen2.5-7B-Instruct.

Notably, TRACER delivers a balanced accuracy-efficiency profile:

• GSM8K (in-domain): TRACER achieves 0.8901 accuracy (Qwen backbone), matching single-agent RL methods and surpassing most multi-agent prompt or RL competitors, while using only 1014 tokens and 3.02 model calls per task—significantly lower than most multi-agent debate/voting baselines.
• MATH500 and GPQA-D (cross-domain): TRACER preserves competitive transfer performance with substantial inference cost savings compared to heavy multi-agent rollouts.

  Figure 2: Radar plot contrasts TRACER’s balanced accuracy and cost profile with both RL and non-RL collaborative baselines, emphasizing consistent strength and reduced weaknesses across all axes.

Training Dynamics and Stability

TRACER features fast and highly stable convergence throughout training, as demonstrated in the stepwise training diagnostics. In contrast, prior RL-based MAS approaches exhibit oscillatory, unstable sample-wide curves, often requiring auxiliary fine-tuning.

Figure 3: GSM8K learning curves illustrate the rapid, monotonic convergence of TRACER relative to baselines.

Ablation and Component Analysis

Systematic ablation studies confirm that disabling either controller, removing explicit vote or phase state, or dropping GSPO role-specific updates leads to non-trivial drops in both accuracy and efficiency. The learning-enabled collaboration architecture is critical for both sample efficiency and final task generalization.

Implications and Future Directions

Practical Benefits

• Cost-Effective Collaboration: TRACER delivers multi-agent collaboration without the quadratic inference overhead of uncoordinated voting or debate, maintaining agent parsimony and minimizing unnecessary LLM invocations.
• Transparent Credit and Adaptivity: Role-aligned objective decomposition and dynamic, phase-conditioned action selection enable interpretable and adaptable collaboration policies, improving robustness over fixed protocol designs.

Theoretical Advancement

• Bridging Deep RL and Game Theory in LLMs: TRACER’s mapping of multi-agent reasoning to a finite-action, perfect-information game with provable CFR-style regret-matching advances the foundation for theoretically grounded learning in cooperative neural agents.
• Template for Generalized Multi-Agent LLM Systems: The architecture is modular and readily extensible to broader agent ensembles beyond dual-role systems by adding additional controllers, as well as to alternative backbone architectures.

Limitations and Future Work

TRACER’s present study is limited to training on arithmetic-style problems (GSM8K) and evaluates transfer on a select set of math and science reasoning domains. Further expansion to general domains, longer-horizon protocols, larger model variants, and empirical study of emergent collaborative protocols in larger agent societies represent promising future avenues.

Conclusion

TRACER introduces a unified, theoretically justified framework for cooperative multi-agent LLM reasoning based on two-layer regret-matching and explicit GSPO-based credit assignment. It achieves accuracy and sample efficiency competitive with both non-RL and RL state-of-the-art, with robust, stable learning processes and provable solution properties. By simultaneously addressing cost, stability, and collaborative policy learning, TRACER represents a scalable architectural and analytical foundation for next-generation learned cooperation in multi-agent LLM systems.