Behavior Consistency Reward (BehR)

• Behavior Consistency Reward (BehR) is a metric that quantifies functional consistency by comparing the logged next action likelihoods between real and predicted states under a frozen reference agent.
• It shifts focus from surface-level state similarity to decision-critical fidelity by using per-token log-probability differences as a proxy for behavioral alignment.
• Integrated via GRPO in world-model training, BehR improves task-level consistency on benchmarks like WebShop and TextWorld, boosting Exact Match rates and reducing false positives.

Behavior Consistency Reward (BehR) is a step-level reward for training and evaluating text-based world models by measuring whether a predicted state preserves the action tendency of a downstream agent rather than merely matching the surface form of the real state. In the formulation introduced in "Beyond State Consistency: Behavior Consistency in Text-Based World Models" (Huang et al., 15 Apr 2026), BehR operationalizes functional consistency through the likelihood of a logged next action under a frozen Reference Agent, and is optimized through a GRPO-based post-training loop for world models. The same source positions BehR as an alternative to single-step metrics such as Exact Match, while related work frames behavior consistency more broadly as a reward-smoothing or decision-preservation principle in RLHF and multi-agent RL (Zhou et al., 10 Jun 2025, Lin et al., 2023).

1. Functional consistency and the definition of BehR

The ideal target underlying BehR is functional consistency. Let $\pi$ be a fixed downstream agent, and let $h_t = (s_0, a_1, s_1, \ldots, a_t)$ denote the history. A world model $\hat H$ is functionally consistent with the real environment if, for every step $t$,

$$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$

This condition states that replacing the real next state $s_{t+1}$ with the predicted next state $\hat s_{t+1}$ should leave the downstream agent’s action distribution unchanged (Huang et al., 15 Apr 2026).

In practice, the full distribution $\pi(\cdot \mid \ldots)$ is often inaccessible, especially when the downstream agent is a black box. BehR replaces this intractable objective with a tractable probe based on the logged next action $a^*_{t+1}$. A frozen Reference Agent $\pi_{\mathrm{ref}}$ is used to score that action under both the real state and the predicted state. Given context $h_t = (s_0, a_1, s_1, \ldots, a_t)$0 and logged next action $h_t = (s_0, a_1, s_1, \ldots, a_t)$1, the reference model returns the mean per-token log-probability

$h_t = (s_0, a_1, s_1, \ldots, a_t)$2

The real and predicted scores are then

$h_t = (s_0, a_1, s_1, \ldots, a_t)$3

and BehR is defined as

$h_t = (s_0, a_1, s_1, \ldots, a_t)$4

This reward lies in $h_t = (s_0, a_1, s_1, \ldots, a_t)$5 and is maximized at zero when the predicted state preserves the reference-agent likelihood of the logged action exactly. The paper also notes that a bounded transform such as $h_t = (s_0, a_1, s_1, \ldots, a_t)$6 can be used in practice so that $h_t = (s_0, a_1, s_1, \ldots, a_t)$7, while preserving the same core objective (Huang et al., 15 Apr 2026).

The central distinction is therefore between state similarity and behavioral faithfulness. BehR asks whether the generated state leads the agent to make the same decision, not whether the text simply looks similar to the real state. This suggests a shift from reconstruction-oriented training to decision-preserving training.

2. Derivation and intuition

The derivation of BehR begins from the ideal requirement that the downstream policy behave identically under the real and predicted next states. Because this full equality is generally unavailable when $h_t = (s_0, a_1, s_1, \ldots, a_t)$8 is a black box, the logged next action is used as a probe of the decision boundary. If $h_t = (s_0, a_1, s_1, \ldots, a_t)$9 preserves the likelihood of $\hat H$0 relative to $\hat H$1, then the world model has preserved at least one decision-critical aspect of the environment state (Huang et al., 15 Apr 2026).

The resulting quantity is dense and step-level. The paper contrasts this with binary task success and with token-level reconstruction metrics. Under this view, the absolute gap

$\hat H$2

measures how much the candidate state perturbs the downstream action preference of the frozen judge. Smaller gaps indicate greater behavioral faithfulness. A plausible implication is that BehR privileges latent state properties that matter causally for action selection, even when they are not the dominant contributors to surface-form similarity.

This design also clarifies what BehR is not. It is not a direct estimate of environment correctness in a semantic or factual sense, and it is not a full imitation of the downstream policy. Rather, it is a proxy for functional consistency defined through a single logged action under a surrogate agent. The paper explicitly motivates this substitution by the inaccessibility of the full action distribution and by the need for a tractable reward (Huang et al., 15 Apr 2026).

A related but distinct formulation appears in "Intra-Trajectory Consistency for Reward Modeling" (Zhou et al., 10 Jun 2025), where consistency is imposed over adjacent prefixes in autoregressive text generation. There, the Bayesian motivation is that as the generator probability of a continuation approaches one, the variance of the reward of a prefix around the reward of its extension vanishes; in words, “if the model is almost certain that extension $\hat H$3 follows $\hat H$4, then their rewards should be almost the same.” The paper enforces this locally between adjacent prefixes rather than over arbitrary continuations. This provides a closely related interpretation of BehR as a local smoothness principle specialized to sequence generation (Zhou et al., 10 Jun 2025).

3. Optimization with GRPO

BehR is integrated into an RL-style world-model training loop through Group Relative Policy Optimization (GRPO). The world model $\hat H$5 is treated as a policy $\hat H$6 over next states conditioned on $\hat H$7, and the training objective is to maximize expected BehR across an offline dataset of tuples $\hat H$8 (Huang et al., 15 Apr 2026).

The procedure described in the paper is as follows. For each minibatch of prompts $\hat H$9, the model samples $t$0 candidate next states $t$1 from $t$2 using temperature $t$3 to increase diversity, with the main experiments using $t$4 and $t$5. For each prompt, the real reference score $t$6 is cached to avoid duplicate calls, and each candidate state receives a BehR score

$t$7

Within each prompt group, the $t$8 rewards are normalized by subtracting the group mean and dividing by the group standard deviation. The GRPO objective is then

$t$9

where $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$0 is the normalized BehR, together with an auxiliary KL penalty $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$1 to prevent drift. The main experiments use $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$2 and run under the VeRL/GRPO framework with batch size $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$3, micro-batch per GPU, and FSDP plus tensor parallelism (Huang et al., 15 Apr 2026).

The paper reports that the Reference Agent $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$4 is frozen throughout training, with Qwen3-8B given as an example. This avoids a moving target. It also states that the exponential reward mapping $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$5 is used in the main experiments, while linear and Cauchy variants are also supported (Huang et al., 15 Apr 2026).

This optimization scheme positions BehR as a post-training signal layered on top of an SFT-initialized world model rather than as a replacement for supervised pretraining. A plausible implication is that BehR is intended to redirect generation toward decision-critical fidelity while preserving the broad linguistic and structural competence supplied by SFT.

4. Empirical findings in WebShop and TextWorld

The principal empirical study evaluates BehR on WebShop and TextWorld. WebShop is described as an e-commerce simulator where states are open-ended product catalog pages and actions are search[keywords] or click[ASIN]; success is purchasing the correct item within 50 steps. TextWorld is a structured text-adventure generator with rooms, objects, and subgoals; actions are natural-language commands such as “open chest” or “go north,” and success is completing all subgoals within 50 turns (Huang et al., 15 Apr 2026).

The base world models are Word2World SFT checkpoints on Qwen2.5-7B and LLaMA3.1-8B. BehR-WM denotes the SFT checkpoint followed by BehR-GRPO post-training. RL ablations include F1-WM, which uses GRPO with token-level F1 reward, FactR, which targets structured factual correctness, and BehR-WM. Evaluation agents include Qwen3-8B, Qwen3-32B, GPT-4o, and GPT-5, all at zero temperature (Huang et al., 15 Apr 2026).

On single-step prediction, BehR post-training preserves Exact Match on WebShop, with $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$6, and improves on TextWorld, with $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$7, in three out of four model-backbone setups. On task-level functional consistency, measured through the pairwise consistency ratio $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$8, BehR yields larger gains. For each held-out initial task, the agent acts in the real environment, then in the world model, and then the world-model-generated action sequence is replayed in the real environment; $$\pi(a \mid h_t, s_{t+1}) = \pi(a \mid h_t, \hat s_{t+1}) \quad \text{for all actions } a.$$9 is defined as the fraction of Real-successful tasks that remain successful under W2R. In WebShop, using the Qwen2.5-7B base world model and a Qwen3-8B agent, $s_{t+1}$0 rises from $s_{t+1}$1 for SFT to $s_{t+1}$2 for BehR. The paper reports similar gains for Qwen3-32B and GPT-4o. In TextWorld, the gains are smaller but consistent in non-ceiling regimes, with one reported setting improving from $s_{t+1}$3 to $s_{t+1}$4 (Huang et al., 15 Apr 2026).

The same study also examines calibrated surrogate evaluation. In offline “WM as evaluator” mode, W2W-WM is reported to greatly overestimate weak agents, with false positives up to $s_{t+1}$5 on TextWorld Qwen3-0.6B. BehR-WM reduces false-positive rate by $s_{t+1}$6–$s_{t+1}$7, raising task-level agreement between world-model and real-environment evaluation from approximately $s_{t+1}$8 to $s_{t+1}$9 in the weakest cases (Huang et al., 15 Apr 2026).

For inference-time lookahead planning with $\hat s_{t+1}$0, agents propose $\hat s_{t+1}$1 candidate actions, the world model simulates next states, and the agent picks the best outcome. Lookahead with SFT-WM already improves success rate by $\hat s_{t+1}$2–$\hat s_{t+1}$3 percentage points, and BehR-WM yields marginal extra gains, reported as $\hat s_{t+1}$4 percentage points in some cases. The paper characterizes these results as preliminary and points toward safer planning (Huang et al., 15 Apr 2026).

These findings support a narrow but important claim: optimizing a behavior-aligned signal can improve long-term alignment even when single-step metrics move only slightly. They do not establish that BehR dominates all alternative objectives in every regime; the source explicitly notes less movement in near-ceiling regimes (Huang et al., 15 Apr 2026).

5. BehR in relation to reward consistency in RLHF

A closely related notion appears in reward modeling for RLHF. "Intra-Trajectory Consistency for Reward Modeling" introduces an intra-trajectory consistency regularization that propagates response-level supervision across adjacent processes in a response trajectory using generation probabilities (Zhou et al., 10 Jun 2025). The generator’s next-token probability is written as

$\hat s_{t+1}$5

and the reward model’s calibrated output on the prefix $\hat s_{t+1}$6 is

$\hat s_{t+1}$7

Under a Bradley–Terry calibration, for a chosen response $\hat s_{t+1}$8,

$\hat s_{t+1}$9

with an analogous definition for a rejected prefix. The regularizer constructs cross-entropy terms over adjacent prefixes whose weights depend on both the next-token generation probability and the model’s confidence in the other prefix’s reward, and combines them with the standard Bradley–Terry loss through

$\pi(\cdot \mid \ldots)$0

The paper states that this is “exactly a data-driven instantiation” of the RLHF idea of augmenting scalar outcome reward with a local smoothness or consistency term over likely adjacent transitions, and summarizes the induced behavior-consistency reward as

$\pi(\cdot \mid \ldots)$1

Its practical take-aways are to weight the consistency penalty by next-token probability, calibrate process rewards, focus on adjacent prefixes, and tune the balance $\pi(\cdot \mid \ldots)$2 so as not to wash out the main preference signals (Zhou et al., 10 Jun 2025).

Empirically, that regularizer improves outcome reward modeling and downstream alignment. On RewardBench with a Gemma-2B-it backbone and 40K samples from Unified-Feedback, baseline GRM average accuracy rises from $\pi(\cdot \mid \ldots)$3 to $\pi(\cdot \mid \ldots)$4 with intra-trajectory consistency; with 400K samples, $\pi(\cdot \mid \ldots)$5. Using Skywork plus Unified-Feedback and Llama3-8B-instruct with an EMA of process rewards, GRM-average improves from $\pi(\cdot \mid \ldots)$6 to $\pi(\cdot \mid \ldots)$7. In RLHF, a DPO policy evaluated by a gold scorer improves from $\pi(\cdot \mid \ldots)$8 to $\pi(\cdot \mid \ldots)$9, and the “win” rate versus GRM rises from $a^*_{t+1}$0 to $a^*_{t+1}$1. In inference-time Best-of-$a^*_{t+1}$2 on math tasks with a Qwen-1.5B-Instruct reward model, Mistral v0.3 pass@$a^*_{t+1}$3 improves from $a^*_{t+1}$4 to $a^*_{t+1}$5, and Llama-3-8B-Instruct pass@$a^*_{t+1}$6 improves from $a^*_{t+1}$7 to $a^*_{t+1}$8 (Zhou et al., 10 Jun 2025).

A separate RLHF perspective on consistency appears in "The Trickle-down Impact of Reward (In-)consistency on RLHF" (Shen et al., 2023). That work defines response consistency and instruction consistency on a Contrast Instructions benchmark:

$a^*_{t+1}$9

$\pi_{\mathrm{ref}}$0

The same paper reports that a baseline LLaMa-7B reward model trained with the standard ranking objective is near random on this benchmark, with $\pi_{\mathrm{ref}}$1 and $\pi_{\mathrm{ref}}$2, while ConvexDA and RewardFusion improve those values modestly; humans score approximately $\pi_{\mathrm{ref}}$3 on both metrics. It also reports downstream RLHF gains, including human pairwise preference above $\pi_{\mathrm{ref}}$4 for the consistency-trained model and an increase in usefulness judgments from approximately $\pi_{\mathrm{ref}}$5 to $\pi_{\mathrm{ref}}$6 (Shen et al., 2023).

Taken together, these results suggest that BehR is part of a broader consistency-oriented family of objectives: one branch targets world-model state prediction through downstream action likelihoods, and another targets reward modeling through local reward smoothness or semantic coherence. The shared theme is preservation of decision-relevant structure rather than optimization of surface-form fidelity alone.

6. Broader consistency-reward formulations and open distinctions

The phrase behavior consistency is not unique to text-based world models. In "DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning" (Lin et al., 2023), behavior consistency is defined as the divergence in output actions between two agents given the same observation. If agent $\pi_{\mathrm{ref}}$7 observes $\pi_{\mathrm{ref}}$8 and produces action distribution

$\pi_{\mathrm{ref}}$9

and $h_t = (s_0, a_1, s_1, \ldots, a_t)$00 denotes the distribution that agent $h_t = (s_0, a_1, s_1, \ldots, a_t)$01 would produce on that same observation, then the inconsistency metric is

$h_t = (s_0, a_1, s_1, \ldots, a_t)$02

The dynamic consistency intrinsic reward is

$h_t = (s_0, a_1, s_1, \ldots, a_t)$03

and is combined with extrinsic reward through

$h_t = (s_0, a_1, s_1, \ldots, a_t)$04

Here the sign of $h_t = (s_0, a_1, s_1, \ldots, a_t)$05 determines whether the agent is rewarded for reducing divergence or increasing divergence, and a Dynamic Scale Network produces these coefficients from the joint observation (Lin et al., 2023).

This multi-agent formulation differs from BehR in target and mechanism. BehR compares real-state and predicted-state action likelihoods under a frozen Reference Agent. DCIR compares *