Reward-Driven Intention Reasoner

• Reward-Driven Intention Reasoner is a framework where reward signals guide intermediate reasoning, influencing planning, goal-setting, and decision making.
• It employs diverse approaches such as reinforcement learning on token trajectories, explicit intention extraction, and latent clustering to define and optimize intermediate computational paths.
• These systems have shown enhanced performance and interpretability in language, robotics, and web domains, while raising challenges in reward design and clustering quality.

Searching arXiv for the cited work and closely related papers on reward-driven intention reasoning. Searching arXiv for "reward-driven intention reasoner" and related terms. Reward-Driven Intention Reasoner denotes a class of systems in which reward does not merely score a final output, but shapes which intermediate reasoning traces, evaluative criteria, action abstractions, or plans are treated as worth pursuing. In current usage, the idea appears in at least four recurrent forms: reinforcement learning over token trajectories with critics or differentiable reward guidance; generative reward models that reason before judging; intention-space or reward-map abstractions that compress semantically similar actions or plans; and process-reward controllers that steer inference-time search in language, web, robotics, and motion domains (Hu et al., 31 Mar 2025, Chen et al., 5 May 2025, Yang et al., 31 May 2025, Pei et al., 16 Jul 2025, Zhang et al., 29 Jan 2026, Sohn et al., 10 Apr 2026).

1. Conceptual foundations

A central distinction in this literature is whether “intention” is implicit or explicit. In Open-Reasoner-Zero, intentions are not explicitly represented; instead, “the combination of reward and critic defines which reasoning trajectories are valuable,” so the model learns an internal preference for long, structured, and non-repetitive reasoning that reaches correct answers (Hu et al., 31 Mar 2025). In RM-R1 and WebArbiter, by contrast, intentions are externalized as rubrics, solutions, principles, and structured justifications, so the model first states what should matter and then evaluates candidate outputs against those criteria (Chen et al., 5 May 2025, Zhang et al., 29 Jan 2026). In “Inferring and Conveying Intentionality: Beyond Numerical Rewards to Logical Intentions,” intentions are logical task specifications in PLTL rather than scalar rewards, and inference over demonstrations is framed as recovering the formula that best explains observed behavior (Jha et al., 2022).

A second distinction concerns what reward is supposed to align. Some systems align reasoning with final correctness, as in outcome-only math or QA settings. Others align reasoning with pairwise preference judgments, safe web actions, tool-call arguments, or latent behavioral goals in traffic and robotics. ReCouPLe makes this point especially explicit: a rationale such as “avoids collisions” or “the cube is larger” is treated as the causal axis along which reward differences should be explained, rather than leaving the learner free to latch onto spurious correlates (Hwang et al., 5 Mar 2026). This suggests that “reward-driven intention reasoning” is best understood not as a single algorithm, but as a design pattern in which reward defines, selects, or regularizes the latent purpose of intermediate computation.

2. Reward-shaped reasoning inside base models

One major lineage uses reinforcement learning directly on a base model so that reward shapes internal reasoning trajectories. Open-Reasoner-Zero applies vanilla PPO with a learned critic to Qwen2.5-Base models. The policy samples long responses with > ... and <answer>...</answer>, receives terminal reward $R \in \{0,1\}$ based only on final answer correctness, and learns token-level value estimates $V_\phi(s_t)$. With $\gamma = 1$ and $\lambda = 1$, the advantage simplifies to

$\hat{A}_t = R - V_\phi(s_t),$

so every reasoning token shares credit for final correctness. A central empirical claim is that the learned critic assigns lower value to repetitive prefixes than to coherent ones, which makes PPO more stable than GRPO-style methods without a value network and suppresses pathological loops without any explicit repetition penalty or KL regularization (Hu et al., 31 Mar 2025).

A second lineage keeps the base model fixed but alters its next-token policy at inference time. $\nabla$-Reasoner treats per-token logits as continuous latent variables and performs test-time gradient descent on them to maximize a joint objective

$$\mathcal{L}(y) = -\lambda\, r(y \mid x) - \log \pi_{\mathrm{LLM}}(y \mid x),$$

where the reward model pushes toward high-value completions and the likelihood term keeps the optimized sequence near the base model’s distribution. The paper further shows that inference-time gradient descent in sample space is dual to KL-regularized reinforcement learning, making the method a “deamortized PPO” at test time rather than a conventional discrete search procedure (Wang et al., 5 Mar 2026).

Pixel Reasoner extends the same principle into multimodal tool use. Its policy interleaves textual reasoning with visual operations such as CropImage and SelectFrames. Because correctness-only RL quickly collapses to text-only behavior, the paper adds a curiosity-driven intrinsic reward

$$r'(x,y)=r(x,y)+\alpha r_{\mathrm{curiosity}}(x,y)+\beta r_{\mathrm{penalty}}(y),$$

where $r_{\mathrm{curiosity}}$ encourages pixel-space reasoning when the “Rate of Pixel-space Reasoning” is below a target $H$, and $V_\phi(s_t)$0 discourages excessive visual operations. The resulting policy is explicitly rewarded for intending to inspect pixels, but only up to a bounded operation budget (Su et al., 21 May 2025).

These systems differ in optimizer and modality, yet they share a common mechanism: the reward signal reshapes the probability of internal trajectories before or during answer generation. A plausible implication is that the minimal object of alignment in these methods is not the final string, but the policy over intermediate reasoning states.

3. Generative reward models and process evaluators

A second major lineage turns reward modeling itself into a reasoning problem. RM-R1 introduces Reasoning Reward Models, or ReasRMs, which generate a full judgment trace $V_\phi(s_t)$1 culminating in <answer>[[A]]</answer> or <answer>[[B]]</answer>. For reasoning tasks, the model first emits <solution>...</solution> and then evaluates candidate answers relative to that self-generated solution. For chat tasks, it emits <rubric>...</rubric> and <justify>...</justify>, assigning weights such as “Accuracy 40%” or “Comprehensiveness 30%” before entering <eval>...</eval>. Training combines distillation of high-quality reasoning traces and GRPO with a binary verifiable reward based on whether the final verdict matches the known preference label (Chen et al., 5 May 2025).

WebArbiter adapts the same reasoning-first principle to web process reward modeling. Instead of a scalar PRM score, it produces a structured justification with <State>, <Criteria>, <Analysis>, and <Answer>, where the criteria are principles such as goal alignment, task progress, correctness of reasoning, risks, or efficiency. Distillation teaches the model to induce such principles from interface state and task instruction; reinforcement learning then aligns the verdict with correctness through a binary reward and KL regularization against the distilled reference policy. The result is a step-level web reward model that reasons about what the agent is trying to achieve before deciding which action is most conducive to task completion (Zhang et al., 29 Jan 2026).

R-PRM applies an analogous idea to mathematical process supervision. Given a question $V_\phi(s_t)$2 and current step $V_\phi(s_t)$3, a generative verifier first writes a multi-part critique—covering previous steps, current step, data source, consistency, and calculation—and only then outputs a final “Yes” or “No” judgment. The step reward is the model’s probability that the verification token is “Yes,” and preference optimization is performed by constructing DPO pairs in which critiques whose verdict matches the human label are preferred over those that do not (She et al., 27 Mar 2025).

R2IF extends process-level reward design to interpretable function calling. Its composite reward combines hard format/correctness constraints, Chain-of-Thought Effectiveness Reward (CER), and Specification-Modification-Value (SMV) reward. CER measures whether a student model becomes more successful when conditioned on the generated reasoning prefix, while SMV checks whether parameter-level reasoning captures the relevant specification, modification, and value for each tool argument. The system is therefore rewarded not only for correct tool calls, but for producing reasoning that is both faithful to and effective for those decisions (Cheng et al., 22 Apr 2026).

Process Reward Agents push this family into knowledge-intensive search. PRA is a frozen-policy test-time method in which a separate reward agent decides whether to retrieve documents at a given reasoning step and assigns a step reward $V_\phi(s_t)$4. Beam search then ranks partial traces by cumulative process reward rather than waiting for a post hoc trajectory score. This decouples a generic reasoner from a domain-specific reward module and makes process reward available online during inference (Sohn et al., 10 Apr 2026).

Across these systems, reward modeling becomes a judgment policy over explicit intermediate representations. The important shift is that intentions are not only inferred from behavior; they are rendered as text, rules, or argument-level justifications and then optimized as objects in their own right.

4. Intention spaces, inverse reinforcement learning, and planning

A third lineage leaves token space altogether and introduces explicit intention spaces or reward fields. ARIA begins from the observation that language actions live in an exponentially large token space $V_\phi(s_t)$5, making sparse terminal rewards extremely high variance. It therefore projects actions and observations into a discrete intention space $V_\phi(s_t)$6 by clustering embeddings with hierarchical agglomerative clustering, and then replaces raw returns with intention-level averages

$V_\phi(s_t)$7

This aggregated return is used as the advantage in offline REINFORCE. Theoretical results show lower variance than raw per-action returns, while the SplitScore criterion chooses the granularity $V_\phi(s_t)$8 according to changes in reward structure rather than geometry alone (Yang et al., 31 May 2025).

InfoReasoner defines intention in epistemic terms. The latent object is a belief state $V_\phi(s_t)$9 over answer hypotheses, and the intrinsic reward is semantic information gain

$\gamma = 1$0

estimated from the model’s own output distributions after semantic clustering via bidirectional textual entailment. Telescoping additivity and non-negativity under ideal Bayesian updates make this reward a dense signal for information-seeking actions such as retrieval, allowing GRPO to optimize agentic reasoning around uncertainty reduction rather than final exact match alone (Hu et al., 31 Jan 2026).

In motion forecasting, “Foresight in Motion” introduces a reward-driven intention reasoner grounded in query-centric MaxEnt IRL. Scene context is encoded into a grid of BEV queries, a reward network maps each grid state $\gamma = 1$1 to $\gamma = 1$2, and policy rollouts under the IRL-derived policy generate Grid-based Reasoning Traversals (GRTs). These GRTs serve as explicit high-level intention paths before the trajectory decoder produces final multimodal forecasts (Pei et al., 16 Jul 2025).

A related driving paper, “Human-Like Implicit Intention Expression for Autonomous Driving Motion Planning,” constrains the trajectory space with human intention priors for yielding versus proceeding in unprotected left turns, then learns a linear reward $\gamma = 1$3 by Maximum Entropy IRL and selects among candidates with a Boltzmann distribution. Here intention is encoded not as text but as decision-conditioned trajectory manifolds and the reward weights attached to efficiency, comfort, and interaction safety (Liu et al., 2023).

VLA-Reasoner applies the same logic to robotics. A frozen VLA policy proposes an action, a world model simulates future observations, an offline reward model scores those imagined futures, and MCTS selects a reasoned action $\gamma = 1$4 that is blended with the base policy’s action. The node values $\gamma = 1$5 are updated from reward-backed future states, so test-time action selection becomes reward-driven reasoning over imagined trajectories rather than one-step imitation (Guo et al., 26 Sep 2025).

These methods make intentions structurally legible. Instead of “what token should come next,” the operative question becomes “which cluster, belief update, grid traversal, or imagined future should be treated as the current plan.”

5. Safety, causal alignment, and principle-guided control

Safety-oriented work reinterprets intention reasoning as the task of distinguishing benign purpose from harmful or causally spurious correlates. IntentionReasoner is a guard model trained on approximately 163,000 queries annotated with intent reasoning, four-way safety labels—Completely Unharmful (CU), Borderline Unharmful (BU), Borderline Harmful (BH), and Completely Harmful (CH)—and rewritten queries. Its RL stage uses a multi-reward objective

$\gamma = 1$6

where $\gamma = 1$7 rewards correct <thinking>, <label>, and <refined query> format; $\gamma = 1$8 rewards correct multi-level safety classification with partial credit for adjacent labels; $\gamma = 1$9 checks the safety of the rewritten query; $\lambda = 1$0 measures downstream utility; and $\lambda = 1$1 penalizes excessive length. The resulting safeguard reasons explicitly about good intention, harmful intention, and rewrite considerations before choosing whether to answer, rewrite, or refuse (Shen et al., 27 Aug 2025).

ReCouPLe addresses a different failure mode: causal confusion in preference-based reward learning. It treats each natural-language rationale as a projection axis $\lambda = 1$2 in a shared text-trajectory embedding space, decomposes trajectory representation into a rationale-aligned component $\lambda = 1$3 and an orthogonal residual $\lambda = 1$4, and then constrains preference differences to be explained by the rationale-aligned reward rather than spurious context. The framework uses a reason loss on $\lambda = 1$5, an orthogonal consistency loss, and a reward-ratio regularizer. This turns articulated reasons such as “avoids collisions” or “keeps firm grip” into reusable causal directions that define what the learner is intended to care about (Hwang et al., 5 Mar 2026).

This safety and causal-alignment branch broadens the notion of intention from “planned future action” to “stated purpose” and “causal explanation.” A plausible implication is that reward-driven intention reasoning becomes especially important when outcome labels alone are too sparse or too confounded to identify what should actually be optimized.

6. Empirical landscape, limitations, and open questions

Empirical results across domains show that intention-shaped reward designs often improve both performance and interpretability. Open-Reasoner-Zero-32B reports AIME24 48.1, AIME25 36.0, MATH500 92.2, and GPQA-Dia 55.5, while using only approximately $\lambda = 1$6 of the training steps of the DeepSeek-R1-Zero pipeline (Hu et al., 31 Mar 2025). RM-R1-Qwen-Instruct-32B reaches 92.9 overall on RewardBench, 83.9 on RM-Bench, and 0.730 on RMB (Chen et al., 5 May 2025). ARIA reports an average gain of 9.95% across four downstream tasks and shows 58.85 win rate on Bargaining and 47.16 on Negotiation at Iter2, with improved single-agent scores as well (Yang et al., 31 May 2025). WebArbiter-7B reaches 89.19% average pairwise accuracy and 74.60% average Best-of-N accuracy on WebPRMBench, outperforming GPT-5 by 9.1 points and surpassing the best prior WebPRM by up to 7.2 points in reward-guided trajectory search (Zhang et al., 29 Jan 2026). PRA achieves 80.8% accuracy on MedQA with Qwen3-4B and improves unseen frozen policies by up to 25.7% (Sohn et al., 10 Apr 2026). Pixel Reasoner reports 84% on V* bench, 74% on TallyQA-Complex, and 84% on InfographicsVQA (Su et al., 21 May 2025). IntentionReasoner-7B reaches average F1 99.4 with ASR 1.2% and ORR 0.0% on safeguard benchmarks (Shen et al., 27 Aug 2025).

The recurring limitations are equally clear. Outcome-only rewards remain narrow: Open-Reasoner-Zero is confined to auto-gradable tasks and does not directly address safety or truthfulness beyond correctness (Hu et al., 31 Mar 2025). Intention-space methods depend on embedding and clustering quality; poor clusters can smear together actions with different returns (Yang et al., 31 May 2025). IRL-based planners assume approximate rationality and pay a computational cost for inner-loop planning or world-model rollouts (Pei et al., 16 Jul 2025, Guo et al., 26 Sep 2025). Principle-guided reward models and web arbiters inherit teacher biases and can still rationalize incorrect verdicts post hoc (Zhang et al., 29 Jan 2026). Rationale-based reward learning assumes that provided reasons are themselves reliable causal explanations (Hwang et al., 5 Mar 2026). Test-time gradient or search-based methods remain bounded by the quality of the underlying reward model and by compute budgets (Wang et al., 5 Mar 2026).

Open questions therefore concentrate on three fronts. The first is reward design: how to move beyond correctness or pairwise preference toward richer notions such as cautiousness, honesty, fairness, or multi-objective tool use without making the reward easy to game. The second is representation: whether intentions should be modeled as value-weighted prefixes, explicit rubrics, latent clusters, logical formulas, belief states, reward maps, or hierarchical plans. The third is deployment: how to combine explicit reasoning traces with robust online control so that intentions remain interpretable, causally grounded, and stable under distribution shift. The literature to date suggests that no single representation dominates; rather, reward-driven intention reasoning is becoming a unifying abstraction for systems that learn not only what action to take, but what intermediate purpose is worth pursuing in order to take it.