Bipartite Turn-Level Reward Assignment

• The paper demonstrates that bipartite matching algorithms can optimally map predicted actions to reference turns, yielding fine-grained reward assignment.
• It details rigorous similarity metrics and both hard (Hungarian) and soft (optimal transport) matching strategies to balance local and global policy objectives.
• Empirical results highlight improved sample efficiency and reduced redundant actions in domains like combinatorial bandits and tool-integrated reasoning.

Bipartite matching-based turn-level reward assignment refers to a class of methods that use bipartite graph matching algorithms to assign supervisory reward signals at the level of individual actions (“turns”) within sequential multi-turn tasks. Such strategies provide fine-grained, context-sensitive credit assignment by aligning predicted interaction elements—such as tool invocations or user-resource pairings—with reference (ground-truth or optimal) actions, thereby differentiating, at each decision point, between effective, redundant, or erroneous behaviors. This mechanism is essential for maximizing long-term objectives in domains ranging from combinatorial multi-armed bandits with Markovian rewards to tool-integrated reasoning with LLMs.

1. Formal Foundations: Bipartite Matching and Sequential Decision Frameworks

Consider an environment where, at each discrete time step or “turn,” an agent produces a set of actions, and where ground-truth reference actions per turn are available or can be defined. For instance, in combinatorial multi-armed bandits, users must be matched to resources, with rewards depending on partially observed Markov chains (Gai et al., 2010). In tool-integrated reasoning, LLM agents interleave natural language with tool calls, which can be mapped to canonical reference traces (Qu et al., 15 Jan 2026).

Let $\mathcal{U}$ and $\mathcal{R}$ (or, analogously, predicted calls $P$ and ground-truth calls $G$) denote the elements to be assigned (e.g., users to resources, or predicted tool calls to ground-truth calls). At each turn $t$, a bipartite graph is constructed where edges encode a similarity or reward metric between each predicted element and candidate reference element.

The core computational step is to solve a one-to-one or fractional assignment problem—a (possibly maximum-weight) bipartite matching—so as to optimally align and reward predicted actions in accordance with their correspondence to the reference.

2. Bipartite Matching Construction and Similarity Metrics

The bipartite matching is specified by building a similarity matrix $S \in \mathbb{R}^{m \times n}$, where $S_{ij}$ quantifies how well predicted element $i$ matches reference element $j$. In tool-integrated reasoning domains, similarity may combine:

• Tool name match: Binary indicator of action type correspondence.
• Parameter name Jaccard index: Degree of overlap in argument structure.
• Parameter content exact match: Token-wise parameter value agreement.

The overall similarity metric is normalized such that $0 \le S_{ij} \le 1$. In the combinatorial bandit problem, the “matching reward” for a user-resource allocation is determined by the stationary mean reward $\mu_{ij}$ of the corresponding Markov process, computed as a weighted sum over latent Markov states (Gai et al., 2010).

The assignment can be computed using:

• Hard (one-to-one) matching: Solve a maximum-weight bipartite matching, typically via the Hungarian algorithm. Each predicted element is matched to at most one reference, with unmatched predictions penalized.
• Soft (fractional) matching: Use optimal transport (e.g., Sinkhorn iterations) to distribute matching mass among multiple candidate reference elements, yielding a fractional credit assignment (Qu et al., 15 Jan 2026).

3. Turn-Level Reward Aggregation

Once the per-element (e.g., per-call or per-pair) reward is established, turn-level reward aggregation proceeds by averaging the individual credits for all agent actions at each turn: $r_t = \frac{1}{|P_t|} \sum_{p_i \in P_t} r_{p_i}$ where $P_t$ is the set of actions taken at turn $t$. This normalization regularizes agent behavior across turns with variable action counts, discourages superfluous action generation, and enables reward signals to capture true per-turn efficacy rather than dilute over long trajectories (Qu et al., 15 Jan 2026).

In more classical bandit settings, the total reward per slot is simply aggregated from the matched pairs: $Y(t) = \sum_{(i,j) \in A_{k(t)}} Y_{ij}(t)$ where $A_{k(t)}$ is the current matching (Gai et al., 2010).

Outcome-level rewards (e.g., F1 matching between final predicted and ground-truth answers) may further be included to balance local (turn-level) and global (trajectory-level) credit.

4. Dual-Level Advantage and Policy Optimization

To integrate local (turn-level) and global (trajectory-level) supervision for policy learning, dual-level advantage estimation is used in advanced frameworks such as MatchTIR (Qu et al., 15 Jan 2026). Specifically:

• Trajectory-level advantage: Each rollout’s overall return is normalized within a minibatch, providing a global signal.
• Turn-level advantage: For each turn, a discounted sum of immediate and future rewards is pooled and normalized across all rollouts at that turn index.
• Integrated signal: Each agent decision (token/action) receives an advantage signal $A_{i,j} = A^{\rm traj}_i + A^{\rm turn}_{i,t}$, where $j$ is a decision token within turn $t$.

This dual-level estimation ensures that policy updates are both locally sensitive (distinguishing high-quality from poor tool calls or assignments within a turn) and globally consistent with overall task success. Policy gradient optimization is then performed per token using objectives such as Group Relative Policy Optimization (GRPO), with explicit handling of turn-level masking and regularization (Qu et al., 15 Jan 2026).

5. Regret, Guarantees, and Computational Complexity

In the combinatorial multi-armed bandit setting, the matching-learning for Markovian rewards (MLMR) algorithm maintains per-edge counts and sample mean rewards, computes exploration-augmented indices for each user-resource pair, and solves a maximum-weight matching at every turn. Regret is analyzed with respect to the best static matching, with the main theorem stating: $$R(T) \leq O(M^{3} N L \ln T) + O(M^{2} N) + \text{constant}$$ where $M$ and $N$ are the number of users and resources, $L$ is a tuning factor involving mixing time and reward magnitude, and $T$ is the horizon. Storage and per-slot computational complexity are polynomial ($O(MN)$ for statistics and $O(M^2 N)$ for the matching step) (Gai et al., 2010).

In tool-integrated reasoning, the hard (Hungarian) matching algorithm has $\mathcal{O}(\max(m,n)^3)$ complexity per turn, while soft matching via optimal transport may be accelerated via Sinkhorn iterations. End-to-end, the aggregation and normalization steps scale linearly with rollout length and action count (Qu et al., 15 Jan 2026).

6. Empirical Findings, Applications, and Implications

Empirical results for bipartite matching-based turn-level reward assignment indicate improvements in both sample efficiency and fine-grained control. In tool-integrated reasoning, MatchTIR demonstrates that combining hard or soft bipartite matching for dense, turn-level supervision and a dual-level advantage scheme significantly outperforms trajectory-only or scalar turn-level reward approaches across diverse benchmarks. Benefits include higher task success rates, reduced tool invocation redundancy, and stronger scalability with task horizon and complexity.

A plausible implication is that such methods are particularly suited to domains with:

• High-action cardinality per turn (multiple simultaneous predictions)
• Long-horizon credit assignment and delayed outcomes
• Availability (or constructibility) of ground-truth interaction traces or oracle matchings.

Extensions and open research directions include efficient approximation for large-scale matching, adaptation to dynamic graphs, and integration with self-supervised or semi-supervised learning frameworks (Gai et al., 2010, Qu et al., 15 Jan 2026).