Reward Evolution with Graph-of-Thoughts (RE-GoT)

• RE-GoT is a bi-level framework that integrates graph-structured reasoning with reward-driven optimization to iteratively refine reasoning graphs and reward functions.
• It employs modular techniques where an LLM constructs text-attributed graphs and an RL agent uses shaped rewards, yielding notable performance gains in robotics and symbolic tasks.
• Empirical studies demonstrate that adaptive reward evolution adjusts graph depth and operations based on task complexity and rollout feedback for enhanced task success.

Reward Evolution with Graph-of-Thoughts (RE-GoT) denotes a class of methods in which graph-structured reasoning and reward-based optimization are coupled so that reward signals alter either the reasoning graph itself or the reward function that supervises downstream reinforcement learning. In the strict sense of "Reward Evolution with Graph-of-Thoughts: A Bi-Level LLM Framework for Reinforcement Learning", RE-GoT is a bi-level LLM+VLM framework that decomposes robot tasks into text-attributed graphs and iteratively refines reward code from rollout feedback (Yao et al., 19 Sep 2025). In adjacent Graph-of-Thoughts literature, the same expression is also used more broadly for reward-driven adaptation of reasoning topologies, especially in "Reinforced Graph of Thoughts: RL-Driven Adaptive Prompting for LLMs", where an RL agent constructs a Graph-of-Operations online under shaped rewards (Riesen et al., 21 May 2026). Both senses inherit the core Graph-of-Thoughts premise that reasoning need not remain a linear chain: thoughts are nodes, dependencies are edges, and branching, aggregation, and revision become explicit computational objects (Besta et al., 2023).

1. Conceptual foundations

Graph-of-Thoughts (GoT) was introduced as a prompting framework in which LLM-generated thoughts are organized as an arbitrary directed graph rather than a single chain or a tree. In its canonical formalization, GoT is written as $$\text{GoT} = (G,\mathcal{T},\mathcal{E},\mathcal{R})$$, where $G$ is the current reasoning graph, $\mathcal{T}$ are thought transformations, $\mathcal{E}$ is an evaluation function, and $\mathcal{R}$ is a ranking function (Besta et al., 2023). That formulation already contains the basic ingredients needed for reward evolution: graph construction, evaluation, ranking, and pruning.

A second foundational line represented Graph-of-Thought as an encoded thought graph extracted from text, rather than only as a prompting controller. In that formulation, thought units are derived from Open Information Extraction triplets, organized into a graph $\mathcal{G}(\mathcal{N},\mathcal{E})$, encoded with a graph attention network, and fused with textual and visual representations. On ScienceQA, this encoder-based GoT raised T5-base accuracy from $85.19\%$ to $87.59\%$ over Multimodal-CoT (Yao et al., 2023). This established that graph structure can be useful both as an inference-time control object and as an internal representational prior.

Later taxonomic work reframed chains, trees, and graphs of thoughts as instances of a common notion of reasoning topology, with prompt execution described by preprocessing, context construction, LLM invocation, post-processing, and context update functions (Besta et al., 2024). Within that vocabulary, RE-GoT is most naturally a multi-prompt, explicitly serialized, graph-topology reasoning scheme in which evaluation is no longer merely diagnostic but acts as a control signal.

A crucial refinement appears in RGoT, which separates the Graph of Operations (GoO) from the Graph of Thoughts (GoT). The GoO is a task-level controller: a layered DAG $G=(V,E)$ with one source and one sink, depth and breadth constraints, and a human-defined operation set $\mathcal{O}$. The GoT is the concrete reasoning graph produced by executing that controller on an LLM (Riesen et al., 21 May 2026). This separation is important for RE-GoT because it makes explicit which object is being evolved: the graph of reasoning outputs, the graph of operations that generates them, or both.

2. Reward as a graph-construction signal

In the RGoT formulation, adaptive graph construction is modeled as an MDP $G$0. The action space is

$G$1

and the state tracks depth, breadth, global and local task complexity, a reduced GoO representation, previous actions, and previous score (Riesen et al., 21 May 2026). The controller policy $G$2 is trained with PPO using actor and critic networks with two hidden layers of size 64. This is the clearest formalization of reward-driven graph evolution in prompting: the agent does not merely score completed reasoning traces but decides, layer by layer, which structural operations to append, undo, or stop.

The reward function in RGoT is explicitly shaped to combine final correctness, intermediate progress, and structural regularization. Concrete components include a depth penalty, immediate $G$3 for invalid actions, $G$4 for useful backtracking after a failed scored operation, $G$5 plus depth penalty for valid intermediate operations, $G$6 for a final correct solution, and $G$7 plus depth penalty for final incorrect termination (Riesen et al., 21 May 2026). Because task complexity is part of the state and operation success probabilities decrease with complexity, the reward landscape itself changes across instances. This is the mechanism behind the claim that graph structures “evolve” under reward pressure: shallow near-IO graphs are favored for easy instances, while deeper and more branched graphs become advantageous for hard ones.

A complementary formulation appears in the Graph Reasoning Paradigm (GRP), where reasoning is explicitly emitted as a directed graph $G$8 with step-level cognitive labels such as Known, Aggregate, and Refine. PASC-GRPO then defines a graph reward

$G$9

combining label-format validity, connectivity, Effective Reasoning Subgraph ratio, reachability, and reverse-search reward (Liu et al., 19 Jan 2026). Its Stratified Clipping Advantage Estimation keeps these auxiliary structural rewards subordinate to correctness by allowing them to add only positive bonuses to already-correct trajectories and only negative penalties to wrong ones. In RE-GoT terms, this is a formal anti-reward-hacking device for graph-level supervision.

3. Bi-level RE-GoT for autonomous reward design

The explicit RE-GoT framework in robotics casts reward design as a bi-level optimization problem over an MDP $\mathcal{T}$0 with parameterized reward $\mathcal{T}$1 (Yao et al., 19 Sep 2025). The inner problem is standard policy optimization under $\mathcal{T}$2,

$\mathcal{T}$3

while the outer problem minimizes the discrepancy between task understanding encoded by an LLM and rollout behavior evaluated by a VLM: $\mathcal{T}$4

The graph object here is a text-attributed graph

$\mathcal{T}$5

where nodes are sub-goal stages, edges are transitions between stages, $\mathcal{T}$6 are node-level textual descriptions, and $\mathcal{T}$7 are edge-level behavioral descriptions (Yao et al., 19 Sep 2025). The LLM uses environment abstraction, task text, initial reward code, and available simulator APIs to build this graph in one pass. The graph is then used as structured context for reward generation and refinement.

The evolution loop has four recurrent phases. First, the LLM abstracts the environment and constructs $\mathcal{T}$8. Second, an RL agent is trained under the current reward. Third, rollout videos are evaluated by a VLM, which reports success, missed sub-goals, failure modes, and improvement suggestions. Fourth, the LLM updates reward structure and weights using the current reward code, component statistics, the graph, and VLM feedback (Yao et al., 19 Sep 2025). The procedure is gradient-free in practice even though the paper presents a conceptual gradient expression for $\mathcal{T}$9.

Empirically, this bi-level RE-GoT improves average task success rates on 10 RoboGen tasks by $\mathcal{E}$0, and on 4 ManiSkill2 tasks attains an average success rate of $\mathcal{E}$1 (Yao et al., 19 Sep 2025). Reported RoboGen examples include Close Dispenser Lid improving from $\mathcal{E}$2 to $\mathcal{E}$3, Turn On Lamp from $\mathcal{E}$4 to $\mathcal{E}$5, Rotate Safe Knob from $\mathcal{E}$6 to $\mathcal{E}$7, Close Window from $\mathcal{E}$8 to $\mathcal{E}$9, and Flush Toilet from $\mathcal{R}$0 to $\mathcal{R}$1. The same study reports that performance typically improves across iterations, with several tasks peaking around 8 reward-evolution rounds.

4. Adjacent formulations across modalities and agents

A closely related multimodal formulation is GoT-R1 for text-to-image generation. There, the intermediate reasoning object is a Generation Chain-of-Thought that can be viewed conceptually as a scene graph of objects, attributes, and spatial relations, and the policy is optimized with GRPO (Duan et al., 22 May 2025). Its reward is explicitly multi-stage: $\mathcal{R}$2 where $\mathcal{R}$3 scores prompt–reasoning semantic alignment, $\mathcal{R}$4 scores prompt–reasoning spatial alignment by rendering bounding boxes into an auxiliary image, $\mathcal{R}$5 scores prompt–image alignment, and $\mathcal{R}$6 averages IoU between planned and grounded boxes. This is reward evolution over a reasoning structure even though the structure is emitted as a sequence rather than an explicit graph.

In agentic RL, RewardFlow offers another graph-centric reward mechanism. It constructs a normalized state graph $\mathcal{R}$7 from multiple trajectories, runs inverse BFS from successful terminal states, and assigns state reward

$\mathcal{R}$8

with $\mathcal{R}$9 the shortest-hop distance to success (Feng et al., 19 Mar 2026). Per-action dense rewards are then defined as $\mathcal{G}(\mathcal{N},\mathcal{E})$0. Although formulated for agentic environments rather than prompting, this is directly compatible with RE-GoT if thoughts are treated as states and reasoning transitions as graph edges.

RouteGoT addresses a different axis of reward evolution: budget-aware control of graph execution. For each pending leaf node, it chooses among $\mathcal{G}(\mathcal{N},\mathcal{E})$1, $\mathcal{G}(\mathcal{N},\mathcal{E})$2, and $\mathcal{G}(\mathcal{N},\mathcal{E})$3 under a node budget

$\mathcal{G}(\mathcal{N},\mathcal{E})$4

and approximates

$\mathcal{G}(\mathcal{N},\mathcal{E})$5

(Liu et al., 6 Mar 2026). This is not reward evolution in the strict reward-code sense, but it makes the cost–accuracy trade-off explicit and turns graph expansion itself into an optimizable utility problem.

Two adjacent, non-RL lines also inform RE-GoT design. DGoT uses reward-like thresholds derived from ROUGE statistics and Gumbel modeling to decide whether to continue generation, aggregation, or improvement in abstract generation (Ning et al., 2024). Buffer of Thoughts instead stores reusable thought-templates $\mathcal{G}(\mathcal{N},\mathcal{E})$6 in a meta-buffer and updates that buffer by novelty thresholding, which can be read as evolution over a graph of reusable reasoning templates rather than per-instance thought graphs (Yang et al., 2024).

5. Empirical patterns across the literature

Across task-oriented prompting, RGoT shows that adaptive graph construction is viable under constrained operation sets. On five tasks—sum list, sort list, count keywords, intersect set, and merge docs—it reports an average solved rate of $\mathcal{G}(\mathcal{N},\mathcal{E})$7 within the training front and $\mathcal{G}(\mathcal{N},\mathcal{E})$8 across all complexities, including out-of-distribution settings (Riesen et al., 21 May 2026). The reported qualitative pattern is that operation count increases with task complexity, including beyond the training front, indicating that learned graph policies generalize by using deeper and more branched GoOs.

In visual generation, GoT-R1 reports substantial gains from reward-shaped reasoning. On T2I-CompBench, GoT-R1-7B improves over Janus-Pro-7B-GoT from $\mathcal{G}(\mathcal{N},\mathcal{E})$9 to $85.19\%$0 on Color, from $85.19\%$1 to $85.19\%$2 on Shape, from $85.19\%$3 to $85.19\%$4 on Texture, from $85.19\%$5 to $85.19\%$6 on 2D-Spatial, and from $85.19\%$7 to $85.19\%$8 on Complex (Duan et al., 22 May 2025). On GenEval, it reaches $85.19\%$9 overall, with $87.59\%$0 on Two Objects and $87.59\%$1 on Attribute Binding. Those results support the claim that process-level reward on intermediate reasoning structures can improve final generative fidelity.

For symbolic reasoning, GRP and PASC-GRPO report strong gains on both math and code. On Qwen3-8B, GRP-SFT lifts GSM8K from $87.59\%$2 to $87.59\%$3, MATH500 from $87.59\%$4 to $87.59\%$5, and AMC23 from $87.59\%$6 to $87.59\%$7; PASC-GRPO then further raises GSM8K to $87.59\%$8, MATH500 to $87.59\%$9, and AMC23 to $G=(V,E)$0 (Liu et al., 19 Jan 2026). Ablations show that removing reachability, reverse-search reward, or SCAE degrades performance, which directly links graph-structured reward components to outcome quality.

For agentic execution, RouteGoT reports an average $G=(V,E)$1 percentage point accuracy improvement and $G=(V,E)$2 output token reduction compared to AGoT (Liu et al., 6 Mar 2026). RewardFlow, in turn, reports large gains from topology-aware reward propagation without training a dedicated process reward model; one highlighted result is Sokoban with Qwen2.5-VL-7B improving from $G=(V,E)$3 under GiGPO to $G=(V,E)$4 under RewardFlow (Feng et al., 19 Mar 2026). Taken together, these studies indicate that the practical value of RE-GoT is not confined to a single formalism: graph-aware reward signals improve prompting, visual generation, agentic RL, and robotic reward engineering.

6. Limitations, controversies, and open directions

Several recurrent limitations appear across RE-GoT-style systems. In RGoT, reward shaping is explicitly described as non-trivial, with multiple reward versions tested before settling on Version 7; beyond the training front, reward becomes sparser and variance across seeds rises, while real LLM calls remain expensive and the graph search space grows combinatorially with operation count and depth (Riesen et al., 21 May 2026). In the robotic RE-GoT formulation, the method depends on the quality of both the LLM and the VLM, requires accurate environment abstraction and rollout rendering, and can regress when the VLM misanalyzes behavior, as reported for OpenCabinetDrawer after $G=(V,E)$5 steps (Yao et al., 19 Sep 2025).

Graph-structured RL formulations introduce their own tensions. GRP assumes a predefined cognitive label set and relatively static topology scheme, and its evaluation is concentrated on math and code rather than broader commonsense or interactive domains (Liu et al., 19 Jan 2026). RewardFlow assumes that trajectories can be normalized into discrete graph states and that some successful trajectories exist so that inverse BFS has anchors; without informative state representations or success states, the propagated signal weakens (Feng et al., 19 Mar 2026). RouteGoT relies on offline, static success and budget predictors, so misestimated node difficulty can lead either to wasteful over-decomposition or to under-allocation on genuinely hard nodes (Liu et al., 6 Mar 2026).

There is also a substantive design question about what exactly should evolve. One line evolves the reasoning controller or topology, as in RGoT. Another evolves dense reward code for a downstream RL policy, as in robotic RE-GoT. A third evolves process-level evaluators or state values over a fixed graph, as in RewardFlow. This suggests that RE-GoT is not a single algorithmic object but a design space organized around three coupled decisions: how thoughts are graphized, how rewards are assigned, and which layer—graph, policy, or reward function—is optimized.

Open directions stated or implied across the literature include richer multi-objective rewards, uncertainty-aware reward terms, relaxation of DAG constraints to allow feedback loops, integration with knowledge graphs, explicit evolutionary search over graph genotypes, and automated pipelines that estimate operation success probabilities and train adaptive controllers end to end (Riesen et al., 21 May 2026). A plausible implication is that future RE-GoT systems will combine several of these strands: graph-native process rewards, multimodal evaluators, cost-aware routing, and iterative reward-code refinement. The common premise is stable across papers: once reasoning is externalized as a graph, reward can be attached not only to outcomes but to structure, transitions, and subgoals, and that substantially enlarges the design space of reinforcement for LLM-centered systems.